macOS Want to program something which can control keyboard/mouse

[hdawg87]

Hi,

I'm trying to build a program that can control the keyboard and mouse and take screenshots. The program would then use the screenshots to determine what to do next. I was hoping it could be done in Python but so far cannot find anything that would allow me to control a mouse and keyboard in Python, anyone have any idea where I could find some code or examples on how to do that? If it's even possible? Thanks!

 

[iSee]

Quartz event taps may allow you to do what you want.

This is a C API.

http://developer.apple.com/document...artzEventServicesRef/Reference/reference.html

 

[Consultant]

I don't really see a legit use for this except to make malware?

 

[iSee]

This *could* be an effort to script something that has no real interface to allow it. Not necessarily a bad thing.

What do you say, OP, do you want to fess up? What is this really for?
My guess is a game bot--a little bad but a really good programming challenge.

BTW, event taps in general are to enable assistive technologies.
I used an event tap to write a daemon that converts the useless enter key on my MBP to a right mouse button. I suppose that's a very limited assistive technology, though I have no particular disability.

 

[hdawg87]

This is more of a proof of concept than anything else. I was having a discussion with a friend the other day and we were talking about automation and whether you could program something that could operate a computer as a human does. Once a skeleton program was written, it could be tailored to fit many productive uses. For example, it could wait until something (video encoding, etc) is done and then open Mail to send you an email notifying you that it is completed. It could even play Minesweeper.

I do understand that there are simpler ways to do the things that I outlined above but want to reiterate that I have not come into this project with a set goal outlined. I essentially want to make something that can be used to perform any task that a human being could do, and for that I need to be able to control the keyboard and mouse. If it were to be implemented for me, I would most likely use it to to check on programs that are running and tell me when they are done. Since none of those programs have an interface that can be readily accessed via an Applescript, Automator, or something else simple I felt like it would be a good (and fun) idea to build something like this.

 

[iSee]

Fair enough! It is a really interesting problem.

If event taps won't work, another thing to look in to is writing a device driver as a way to implement a virtual device (mouse and keyboard).