Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

rpmurray

macrumors 68020
Original poster
Feb 21, 2017
2,147
4,330
Back End of Beyond
I don't know if what I'm looking for is something already available and I'm just not finding it, or if no one has made anything like this.

My use case: I have a lot of reports that I must download from a specific website. This entails signing onto a single-sign on landing page, entering my username and password, clicking the tile for a specific web application, and when in the application click though various links and change various settings. Then submit and once the task runs take the resulting data and download it as a text file. And, yes, I have to do it this way. The third-party site will not generate and send me these files, or drop them to an FTP site that I can access and download them from.

I would need to do this for something in the neighborhood of twenty reports, sometimes more depending on the time of the year. The number of reports changes when done quarterly, semi-annually, and annually.

So what I'm basically looking for is a tool that will watch me go trough this process in a browser, recording the back and forth web interactions, generate a script from that, and then create a script application that will replay the whole thing. So in the end all I would need to do is double-click the application and it will run and drop the file(s) into a folder, just by double-clicking the application icon.

Now to be clear. I do not want to have it script the GUI, like an Applescript that controls a browser, where you have to keep your hands off the computer for fear of interrupting the various keyboard entries and mouse movements and clicks while it runs. I would like this to be like a bash script running in the terminal, while I could be doing other things on the computer while this runs.

Anyone know of such an application? My web developing skills are anemic. I'm basically a database programmer (Oracle) and spend all my time writing SQL, and one of my tasks is to run said reports and load the data into our various databases. I'd like to automate this whole process.
 
How much have you investigated this? I looks to me that the cost this type of application for Windows is typically in the tens of thousands of dollars.

As an alternative you could run the the type of GUI-controlling script you wanted to avoid in a VM, I suppose.
 
How much have you investigated this? I looks to me that the cost this type of application for Windows is typically in the tens of thousands of dollars.

As an alternative you could run the the type of GUI-controlling script you wanted to avoid in a VM, I suppose.
I wouldn't have thought this would be too expensive. Basically recording the GET, POST and any other requests, along with whatever header and cookie information, etc., gets sent and received and then replaying that in a pseudo web browser that doesn't have an interface.

I've done a fair bit of google searching but since my knowledge of web traffic is really sketchy I am not sure what exactly I should be using for search terms.

I did find an application called Fake that lets you do web automation but it's not exactly what I'm looking for. I've also played around a little with Wireshark and Charles to see if I could figure things out by looking at the underlying web traffic but this is a bit above my head. I have done some automation with JavaScript to control Safari, and wrote a nice little Applescript that could scrape the Geekbench website for the benchmark data of various Mac models using cURL, so I know just enough to know I don't know enough.

All the tools I've found aren't all that expensive, but they don't combine everything into a nice simple package that does it all. At the moment I'm trying to find out if there's anything already available so I don't have to build it myself (I'm kind of lazy that way) because I know I'd need to crack some books and spend quite a bit of time building and testing, if I do this from scratch, until I get something that works.

Your alternative doesn't work as I'd really like to eventually integrate this with the databases themselves. Perhaps have these run with a CRON job to drop out the data into a specific folder and then have the database slurp that up into the tables. Or, depending on the solution, have the database use it to directly load into tables without the need to drop the data into a file first. Also I've tried some Applescripts that control the browser, see above, but that solution is far too prone to breaking.
 
Last edited:
Use dev tools in chrome to identify the get requests.

Test it in postman or insomnia.

Build Python script to run through all you need to do.
 
  • Like
Reactions: rpmurray
Use dev tools in chrome to identify the get requests.

Test it in postman or insomnia.

Build Python script to run through all you need to do.
I'll give it a whirl. Is there any documentation or tutorials on how to identify the get requests and then test in postman or insomnia? Or maybe some examples of how this is done? Like I said, I'm kind of out of my depth.
 
I'll give it a whirl. Is there any documentation or tutorials on how to identify the get requests and then test in postman or insomnia? Or maybe some examples of how this is done? Like I said, I'm kind of out of my depth.
It’s just a case of going through the calls and checking the outputs.

If you get them working in insomnia. There’s an option to get insomnia to automatically generate code for python
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.