macOS How can I make this script go faster?

[birdmadgirl7]

Hi there!

So, I made my very first ever script in order to create several thousand QR Codes for my boss. The script works (most of the time) but the problem I’m having is the speed. The script takes 4 seconds to create one code, which is fine except for the fact that we will need to create 75,000 codes at one time. This comes out to a total of 83 hours to create them all! I have a few delays in my script and I have tried taking them out and shortening them but any more alterationds causes the script to not work correctly. I’m sure that the script will look super newbish to pros like you guys but any suggestions would be grand.

The script:

on adding folder items to this_folder after receiving these_items
	repeat with an_item in these_items
		tell application "TextEdit"
			activate
			open an_item
			
			tell application "TextEdit"
				activate
			end tell
			tell application "System Events"
				keystroke "a" using command down
			end tell
			
			tell application "System Events"
				keystroke "c" using command down
			end tell
			
			tell application "System Events"
				keystroke "w" using command down
			end tell
			delay 0.25
			tell application "System Events"
				keystroke "q" using command down
			end tell
			delay 0.25
			tell application "System Events"
				keystroke "v" using command down
			end tell
			
			tell application "System Events" to keystroke return
			delay 0.25
			
			tell application "Preview"
				activate
			end tell
			delay 0.25
			tell application "System Events"
				keystroke "s" using command down
			end tell
			delay 0.25
			tell application "Preview"
				activate
			end tell
			delay 0.75
			tell application "System Events" to keystroke return
			
			delay 0.5
			tell application "System Events" to keystroke return
			delay 0.25
			tell application "System Events"
				keystroke "w" using command down
				delay 0.25
			end tell
		end tell
	end repeat
end adding folder items to

How it works:

The script relays on a couple factors. The “command q” shortcut triggers another script that makes the QR code, which I did not write:

set the theRes to "300"

display dialog "URL?" default answer "http://www.google.com"
set the theURL to the text returned of the result

set input to "http://chart.apis.google.com/chart?cht=qr&chs=" & theRes & "x" & theRes & "&chl=" & theURL

set input to quoted form of input

set temp_file to (path to temporary items)
set temp_name to do shell script "uuidgen"
set temp_file to (POSIX path of temp_file) & temp_name
set q_temp_file to quoted form of temp_file

set cmd to "curl -o " & q_temp_file & " " & input

do shell script cmd
set x to alias (POSIX file temp_file)
tell application "Finder" to open x

My script works with this one using a text file that is placed in a folder with the script linked to it. I hope this is making sense, I’m sure it seems backwards and weird. To make a long story short, I need to somehow make this work faster and I am willing to rework the entire thing or go with a new idea entirely. Thanks in advance!

-Erin

 

[chown33]

Is there a reason you're writing a script for this, rather than using a specialized QR-code making app?

Thread from 2011:
https://forums.macrumors.com/threads/1167658/


What have you tried?

More specifically, what have you tried other than writing a script?

Frankly, writing a script would be my last choice. A simple search using search terms make qr codes from mac shows a lot of results. Did you look at any of them? If so, why were those solutions rejected?

I also entered search terms qr code into the Mac App Store app, and it found a bunch of apps. The most expensive one costs about $20, but there are plenty of free or 99-centers, too. And some of them have decent ratings.

If your time in producing a working script isn't worth anything, then the $20 app might be too much. But I'd be checking out the free and 99-centers just to save myself the time in writing the script.

 

[birdmadgirl7]

Is there a reason you're writing a script for this, rather than using a specialized QR-code making app?

Thread from 2011:
https://forums.macrumors.com/threads/1167658/


What have you tried?

More specifically, what have you tried other than writing a script?

Frankly, writing a script would be my last choice. A simple search using search terms make qr codes from mac shows a lot of results. Did you look at any of them? If so, why were those solutions rejected?

I also entered search terms qr code into the Mac App Store app, and it found a bunch of apps. The most expensive one costs about $20, but there are plenty of free or 99-centers, too. And some of them have decent ratings.

If your time in producing a working script isn't worth anything, then the $20 app might be too much. But I'd be checking out the free and 99-centers just to save myself the time in writing the script.

The reason I wrote the script is because of the volume of codes we have to make. There are many fine options for making a couple codes but I have to make 75000 UNIQUE codes at one time and I have to do that several times in a year. Writing a script was not my first thought, I tried many other avenues and this is the only one I have found that will work out for me, short of paying someone to make them for me. The other option I spent the most time on was trying to create a photoshop action or variable but it wasn't possible. No program out there makes that many unique codes at once. Although, I did find that I really liked writing the script!

 

[chown33]

The first script appears to be doing a lot of keystroking to simulate various simple actions like open, select-all, copy to clipboard, close, etc.

The second script prompts for a string (URL), then does several things culminating in a 'curl' command that does the real work.

So if all the other steps could be condensed down to a "read line from file", and that line is then passed to a "run 'curl' with this line and the other QR-generating parameters", then the result would be a QR-code in a PNG file.

The other QR-generating parameters are pretty straightforward. Deconstructing the second script, it seems to build a URL like this:
http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl=http://example.com/

Here, I used the canonical example.com domain as the URL. The result is definitely a QR-code image-file. You can click on the above URL and see the resulting QR-code. I also tested it by running the 'curl' command, and it definitely downloads a PNG file.

I can't tell you what the QR-code represents, because I don't have a QR-reader, and I don't feel like getting one. You can check the accuracy yourself, which you should do anyway.


The point of all this is that most of the time in your script is probably being spent in all of the non-essential keystroking. There are far faster ways of reading a line from a file, and they don't involve AppleScript.

I could probably write a shell script, or a shell+awk script, that reads through as many files as you like, with as many lines per file as you like. It's not a difficult script to write (for me, or several other regulars here), but we need to know exactly what the input is.

That is, we need you to post some real examples of exactly the kind of file (or files) that need to be processed. They can contain real URLs, or fake ones, or whatever text you want. But they need to be actual valid examples of exactly the kind of data you'll use. That is, if you'll be using text that contains spaces, put that in your example. If none of your text will contain spaces, then don't put any in your examples. I use spaces as one example, because spaces need special handing in URLs. But there are other characters that also need special handling in URLs, such as quotes. The more accurate and more comprehensive your examples are, the fewer iterations we'll have to make.

You should also be preparing to run tests with any script that I or someone else posts. That means you need to be putting together test files, and also putting together a way to test that the QR-code image-files actually contain the correct QR-code. If you encounter a problem, you then need to post the exact URL or text that's causing the problem, and a description of exactly what the problem is, including any error messages.

The maximum speed is going to be limited by the speed of your internet connection, and the speed at which the Google-Charts API responds. You may hit some ceiling for the Google-Charts API, which is what's actually producing the QR-code PNGs. I haven't looked at it in a long time, but there may be an upper limit to the number of requests allowed in any 24-hour period. So you may have to do the production in stages. That will be something to consider in setting up your tests.

 

[lee1210]

My immediate thoughts are:
Turn this into shell. I'm just better with shell. The gist is take the URL in thousands of text files and generate a QR for them. This seems like a few lines of awk to me.
Second thought: divide and parallelize. 18,750 x 4 or 9,375 x 8, then have multiple instances running at once. I'm guessing most of the time will be grabbing the data with curl, so you don't have to wait in that long line.

AppleScript is cool, but when you don't need serious app interaction via an AppleScript dictionary I much prefer shell. I'd probably name the images the same as the text file names with the right extension. It's easier to reconcile than random names generated in the script above.

-Lee

 

[birdmadgirl7]

The idea is that each QR code is a unique URL that will be placed on a product. The customer will then be able to scan the QR code and the URL will take them to the serial of the product. Here is an example:

https://c.papergold.com/sn/53d90b020bf01876c780e110c1706990/


The URLs come to me in a spreadsheet from out client, with the accompanying serials. I will upload an example of what that looks like. I would much rather take the information from a spreadsheet directly but I didn't heave the skills to script that so I was trying to work around that in some way.

Thank you very much for your help and let me know if you need anything else from me!

 

[chown33]

The URLs come to me in a spreadsheet from out client, with the accompanying serials. I will upload an example of what that looks like. I would much rather take the information from a spreadsheet directly but I didn't heave the skills to script that so I was trying to work around that in some way.

Thank you very much for your help and let me know if you need anything else from me!

You should be able to export the XLS file as tab-separated text. Maybe that's a "Save as..." option, I really don't know since I don't use Excel. In any case, it's impractical to process an XLS file directly.

Comma-separated values would also work. That's a CSV file if you're an abbrev'er.

It's important that you do this exporting to produce a text file, because that way you can verify the data is correct. The XLS file is good enough to make small sample data files, but for real testing, you need to produce the text file yourself.


Also, I see that the first column in the XLS file is a short alphanumeric identifier of some kind. The obvious thing to do in producing the QR-codes in PNG files is to use that identifier as the name of the PNG file. This would be a lot more sensible than generating a random UUID, which is what the current script is doing. However, if that first column isn't unique, then will be a problem. So please confirm that the ID in the first column is unique, and is sensible to use for naming the PNG file.

Another possibility is to extract the final name of the URL. For example use 53d90b020bf01876c780e110c1706990 as the PNG filename.

Both options are feasible, so please choose which one you want.


Also, I agree with lee1210 on all points. Running scripts in parallel is pretty easy in shell. It might saturate your internet connection, depending on what the connection speed is, but parallelism is pretty simple to do.

 

[lee1210]

cat Serials.csv | awk -F"," '{print "curl -o "$1".png \"http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl="$2"\""}' | sh

So i saved your file as a csv. I took the trailing slash off because i didn't like it. Generally when i do awk i print the commands i want. Then if i want to save the result to one or more files to run in parallel it's pretty easy to do, and i can grab a sample and run it on its own. In this case, i just pipe it to sh to run the commands immediately.

This ran through these 400 in about 30 seconds. It should get through 75,000 in about an hour and 45 minutes it this rate continued. Again, parallelizing this and running 4 or 8 commands may cut that down.

-Lee

Edit:
Just for edification:

curl -o BR01856099B.png "http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl=https://c.papergold.com/sn/102c2ec01feb77cc304f58e909a65a5c"

is the format of the commands generated.

 

[chown33]

cat Serials.csv | awk -F"," '{print "curl -o "$1".png \"http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl="$2"\""}' | sh

Yup, that's almost exactly what I got, too.

The main change is I had awk emit an "echo "$1 to print the ID, and I added -s (silent) to the 'curl' command so it doesn't show the largely useless progress indicator. Minor cosmetics.

 

[mrichmon]

Again, parallelizing this and running 4 or 8 commands may cut that down.

Should be able to parallelize those operations using gnu parallel. (Install via fink for macports.)

The updated command is:

cat Serials.csv | parallel --jobs 8  awk -F"," '{print "curl -o "$1".png \"http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl="$2"\""}' | sh

This runs 8 jobs in parallel reading stdin from Serials.csv and invoking awk with a subset of the Serials.csv file for each job.

 

[chown33]

Should be able to parallelize those operations using gnu parallel. (Install via fink for macports.)

The updated command is:

cat Serials.csv | parallel --jobs 8  awk -F"," '{print "curl -o "$1".png \"http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl="$2"\""}' | sh

This runs 8 jobs in parallel reading stdin from Serials.csv and invoking awk with a subset of the Serials.csv file for each job.

Unless I'm misinterpreting it, this command runs awk in parallel. However, I don't think that's the expected slow part. It's the running of curl commands that is expected to be slower.

Since it's 'sh' that's reading those commands, and running them in sequence (as distinct from parallel), I would expect an insignificant, probably even unmeasurable, difference in speed here.

 

[mrichmon]

Unless I'm misinterpreting it, this command runs awk in parallel. However, I don't think that's the expected slow part. It's the running of curl commands that is expected to be slower.

You are correct. I had assumed that the print statement was just a debug placeholder before execing the curl command directly.

Based on the existing flow, use the earlier awk command to generate a list of curl commands, then have parallel split them out into separate jobs and execute the curl commands:

cat Serials.csv | awk -F"," '{print "curl -s -o "$1".png \"http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl="$2"\""}' | parallel -j 8

Running awk in parallel can significantly increase throughput more than you might think. You get benefit from parallel operation during blocking operations. But curl is the major hit in this problem.

 

[moonman239]

OP, I recommend you learn how to program in Ruby. It's a programing langue that's so easy to use that you might be able to learn how to make a program in Ruby in the time that it took you to learn how to program in AppleScript.

Here are two things you can do in Ruby:

1) Run shell programs. By invoking the osascript program, your Ruby script can have your computer run AppleScript commands.
2) Generate a QR code. You can find a "gem" that you can use to do the task. (A gem is a repository that contains a Ruby script or collection of them. In this case, the code in the gem is used for generating a QR code.)

 

[Red Menace]

I recommend you learn how to program in Ruby.

When the topic first appeared, that was my first thought, too - "if you want it to go faster, don't use AppleScript". Ruby is so much faster than AppleScript it isn't even funny (at least an order of magnitude), but those kinds of replies tend to get things off-topic.

Looks like most of the work can be done in the shell though, so the scripting language isn't that much of an issue. Make it a 2000 line or so Cocoa application, and I would definitely go with Ruby (and no, I don't care for Obj-C).