macOS Applescript: determine RGB value at specific pixel on screen

[sammich]

Just wondering if there was some way to get the RGB value for a specific point on the screen using Applescript. I'm doing some (flash) game automating and sampling the color means I can check if the correct dialog has appeared before clicking.

I've googled around and found a couple of close hits, but no solutions:

- This post has the same idea as above, from 2006 and no solution
- This will get an RGB value from an image.

As the first link, I'd like to do:

if coloratpixel(x, y) is equal to {r, g, b} then
      click(x, y)
end

If there exists no solution, could someone point me in a direction where I can take selection screen shots (like command-shift-4) with AS or from the shell and I can use the 2nd link above to grab the RGB value. I know I can take full screen screenshots, but that would be unwieldy if I were to sample the screen several times a second.

Thanks in advance for any help rendered,
Sammich.

 

[Slymenstra]

I'm very new to applescript and I'm working on this same issue at the moment. I've figured out a functional solution to this problem, but I can't find the proper code to express the idea

My desire is to try to fetch the pixel color in this fashion:
display dialog Pixel(10, 10)

In doing further research this is the only solution I've found, which requires an image file and doesn't work on screen colors:

set imgpath to quoted form of POSIX path of (choose file with prompt "Please choose an image file:" without multiple selections allowed and invisibles)
set pixelposx to "1.0"
set pixelposy to "1.0"
set pyscriptpath to quoted form of POSIX path of (((path to desktop) as Unicode text) & "pixelcolor.py")
set command to "/usr/bin/python " & pyscriptpath & " " & imgpath & " " & pixelposx & " " & pixelposy
set output to do shell script command
set {red, green, blue} to words 2 through 4 of output

 

[Slymenstra]

Just wondering if there was some way to get the RGB value for a specific point on the screen using Applescript. I'm doing some (flash) game automating and sampling the color means I can check if the correct dialog has appeared before clicking.

I've googled around and found a couple of close hits, but no solutions:

- This post has the same idea as above, from 2006 and no solution
- This will get an RGB value from an image.

As the first link, I'd like to do:

if coloratpixel(x, y) is equal to {r, g, b} then
      click(x, y)
end

If there exists no solution, could someone point me in a direction where I can take selection screen shots (like command-shift-4) with AS or from the shell and I can use the 2nd link above to grab the RGB value. I know I can take full screen screenshots, but that would be unwieldy if I were to sample the screen several times a second.

Thanks in advance for any help rendered,
Sammich.

OK i think i've got it. This has some extra stuff, but you can trim out whatever you don't need. I've looked and looked for an answer to this, and finally found it by digging through commands.

The first thing, this solution requires Extra Suites. It's free, but it runs a popup when you run scripts unless you pay 10 USD.

tell application "Extra Suites" to ES move mouse {10, 120}
tell application "Extra Suites" to ES click mouse with control

set myLocation to {10, 120}
set myColor to {213, 248, 255}
tell application "Extra Suites"
	ES move mouse myLocation
	set coloratpixel to ES pixel color
	if (coloratpixel is equal to myColor) then
		ES click mouse with double click
	else
		set x to item 1 of coloratpixel
		set y to item 2 of coloratpixel
		set z to item 3 of coloratpixel
		

		display dialog y
		display dialog z
	end if
	
end tell

what this does is move your mouse to myLocation, and see if it's equal to myColor. if it is, it double clicks. If it's not, i'm echoing X Y and Z color values into a dialog

 

[jpyc7]

Have you seen Sikuli from MIT Media lab? It's a graphical scripting language where you can automate some GUI interaction. Basically, I think it does some sort of image comparison.

I'm not sure if it's a tool that can help with what you're doing or not. I've been told by a co-worker that it only works properly if the windows stay in the same location in the screen and that the screen resolution must not change.

Here's their website: http://groups.csail.mit.edu/uid/sikuli/

 

[sammich]

Have you seen Sikuli from MIT Media lab? It's a graphical scripting language where you can automate some GUI interaction. Basically, I think it does some sort of image comparison.

I'm not sure if it's a tool that can help with what you're doing or not. I've been told by a co-worker that it only works properly if the windows stay in the same location in the screen and that the screen resolution must not change.

Here's their website: http://groups.csail.mit.edu/uid/sikuli/

Wow. That's pretty much it. Thanks!

(had a quick look, and it's still pretty rough around the edges, but seriously the best damn thing by a long shot!)