Other Aqua Proxy: Fix connection issues on OS X 10.6 – 10.9.

[Wowfunhappy]

You may have previously used my "Legacy Mac Proxy" package. However, I have completely recreated the software from scratch and given it a new name. I think starting fresh with a new thread will make things less confusing. So, without further ado...

---

Aqua Proxy​

Old versions of Mac OS X have trouble talking to the modern internet. Aqua Proxy can help.

Aqua Proxy is a type of software typically referred to as a "Man-in-the-Middle Proxy Server". It sits between you and the internet, capturing all of your computer's traffic and modifying it to be compatible with modern servers before sending it on its way. Aqua Proxy won't fix everything, but it will fix a lot of little things. For example, Aqua Proxy can fix problems with:

• Viewing remote images in the Mail app.
• Subscribing to calendars in the Calendar app.
• Playing live streams in QuickTime.
• Loading Help Center pages in various apps.
• Reading Wikipedia in the Dictionary app. (OS X 10.7+)

Aqua Proxy runs locally on your computer so that no unencrypted data leaves your network.

Download

Once downloaded, make sure to check out the readme for additional information, including how to set up IMAP email to use the proxy.

P.S. Aqua Proxy's uninstaller will also uninstall my old "Legacy Mac Proxy" package. If you previously used Legacy Mac Proxy, run Aqua Proxy's uninstaller first.

P.P.S. For now, I'm saying that Aqua Proxy is compatible with Mac OS X 10.6 – 10.9 Mavericks, because these are the four operating systems I have explicitly tested on. The package should also work on OS X 10.10 and 10.11, but you'll have to try it and let me know!

 

[Wowfunhappy]

.

 

[iPodNano3]

bravo

 

[GalacticStag]

You may have previously used my "Legacy Mac Proxy" package. However, I have completely recreated the software from scratch and given it a new name. I think starting fresh with a new thread will make things less confusing. So, without further ado...

---

Aqua Proxy​

Old versions of Mac OS X have trouble talking to the modern internet. Aqua Proxy can help.

Aqua Proxy is a type of software typically referred to as a "Man-in-the-Middle Proxy Server". It sits between you and the internet, capturing all of your computer's traffic and modifying it to be compatible with modern servers before sending it on its way. Aqua Proxy won't fix everything, but it will fix a lot of little things. For example, Aqua Proxy can fix problems with:

• Viewing remote images in the Mail app.
• Subscribing to calendars in the Calendar app.
• Playing live streams in QuickTime.
• Loading Help Center pages in various apps.
• Reading Wikipedia in the Dictionary app. (OS X 10.7+)

Aqua Proxy runs locally on your computer so that no unencrypted data leaves your network.

Download

Once downloaded, make sure to check out the readme for additional information, including how to set up IMAP email to use the proxy.

P.S. Aqua Proxy's uninstaller will also uninstall my old "Legacy Mac Proxy" package. If you previously used Legacy Mac Proxy, run Aqua Proxy's uninstaller first.

P.P.S. For now, I'm saying that Aqua Proxy is compatible with Mac OS X 10.6 – 10.9 Mavericks, because these are the four operating systems I have explicitly tested on. The package should also work on OS X 10.10 and 10.11, but you'll have to try it and let me know!

Fantastic! Managed to make Apple Mail work (after some setup issues), although for some reason, Safari doesn't work

 

[GalacticStag]

Fantastic! Managed to make Apple Mail work (after some setup issues), although for some reason, Safari doesn't work

After some testing, i've managed to see what works (at least for me):

-Some 2009/2010 browsers (Opera 10.6 and Chrome 7.0)
-iWeb '09 website uploading (though i assume that depends more on my ftp than the proxy considering it's an http website)
-Apple Mail (Gmail and iCloud Mail)
-Connection to TL Legacy servers (sorta...)
-iChat AIM using NINA's service (though that worked before)

And what doesn't work:
-Safari 5 (Any HTTPS website, with legacy mac proxy Safari was incredibly useable)
-iTunes Store (Again, legacy mac proxy made it useable after editing the hosts file)
-By extension, Podcasts don't work anymore thanks to that. :/
-App Store (But that didn't work before anyway lmaooo)
-Remote images on Apple Mail (?! how?)

I know it's an initial release, but idk how some things that worked before don't work anymore 0.o, hope that they get solved in an update!

 

[Wowfunhappy]

@GalacticStag Oh no, all of that stuff should work except for the App Store. There should not be any regressions from Legacy Mac Proxy.

Can you please:

• Share the output of running launchctl list | grep AquaProxy in the Terminal
• Show me some screenshots of System Preferences → Network and System Preferences → Network → [your active connection] → Advanced → Proxies
• Check Console.app for logs related to Aqua Proxy, particularly soon after rebooting and when a request fails, and share them with me
• Confirm that the Aqua Proxy certificate is installed and trusted in Keychain Access's System Keychain.
• Confirm that ISRG Root X1, USERTrust RSA Certification Authority, COMODO ECC Certification Authority, DigiCert Global Root G2, and DigiCert Global Root G3 are installed and trusted in Keychain Access's System Keychain.
• In Safari, if you try to view a page and click the lock icon in the upper right corner, what do you see?

This is every troubleshooting step I can think of right now.

Edit:

-iTunes Store (Again, legacy mac proxy made it useable after editing the hosts file)

What do you edit in the hosts file for the iTunes Store to work on Snow Leopard? This is probably something I can add to the proxy itself, since it can redirect arbitrary URLs.

Edit2: Also, stupid question, but before you try any of the above—have you tried restarting your computer? You shouldn't need to, but worth a try if you haven't.

 

[GalacticStag]

@GalacticStag Oh no, all of that stuff should work except for the App Store. There should not be any regressions from Legacy Mac Proxy.

Can you please:

• Share the output of running launchctl list | grep AquaProxy in the Terminal
• Show me some screenshots of System Preferences → Network and System Preferences → Network → [your active connection] → Advanced → Proxies
• Check Console.app for logs related to Aqua Proxy, particularly soon after rebooting and when a request fails, and share them with me
• Confirm that the Aqua Proxy certificate is installed and trusted in Keychain Access's System Keychain.
• Confirm that ISRG Root X1, USERTrust RSA Certification Authority, COMODO ECC Certification Authority, DigiCert Global Root G2, and DigiCert Global Root G3 are installed and trusted in Keychain Access's System Keychain.
• In Safari, if you try to view a page and click the lock icon in the upper right corner, what do you see?

This is every troubleshooting step I can think of right now.

Edit:

What do you edit in the hosts file for the iTunes Store to work on Snow Leopard? This is probably something I can add to the proxy itself, since it can redirect arbitrary URLs.

Edit2: Also, stupid question, but before you try any of the above—have you tried restarting your computer? You shouldn't need to, but worth a try if you haven't.

Ok, it seems to be working now, for whatever reason. VERY weird that yesterday it didn't work. I uninstalled LMP (legacy mac proxy) through the aqua uninstaller, didn't uninstall the certs, and then installed aqua proxy. Everything seems to work now! And btw, i did restart my computer yesterday when it initially didn't work. So for it to work now is quite strange, i have to say.


iTunes store works again too, after reinstalling the proxy. i added these lines, as per a reddit post, and the hotmail one fixes MSN 3.0 iirc.


76.217.61.232 phobos.apple.com

76.217.61.232 init.itunes.apple.com
76.217.61.232 ax.init.itunes.apple.com
143.198.4.104 messenger.hotmail.com

Anyways, NOW that everything seems to work, thank you for making this, and apologies if i gave you a little scare with my initial encounters!

 

[GalacticStag]

PS: Dictionary.app started crashing after my second install of aqua proxy. reinstalling it from my 10.6.3 install dvd via pacifist fixed it!

 

[Wowfunhappy]

Ok, it seems to be working now, for whatever reason. VERY weird that yesterday it didn't work. I uninstalled LMP (legacy mac proxy) through the aqua uninstaller, didn't uninstall the certs, and then installed aqua proxy. Everything seems to work now!

Phew! This was bothering me all day, so glad to hear it's working!

PS: Dictionary.app started crashing after my second install of aqua proxy. reinstalling it from my 10.6.3 install dvd via pacifist fixed it!

I'm not sure why it appeared to only happen after your second install, but this is (has to be) from Legacy Mac Proxy. You may notice that Aqua Proxy no longer allows users to install the Dictionary Patch on Snow Leopard. The uninstaller should have removed it, but I guess it didn't for you.

Now, here's the interesting thing—the Dictionary Patch actually works perfectly fine on Snow Leopard, insofar as it does what it's supposed to, it makes the Dictionary app respect the proxy settings in System Preferences and in so doing allows the app to actually connect to Wikipedia again.

The problem is that once the app can connect to Wikipedia, the data it gets back causes the app to crash! I wasted half a day figuring this out, I thought there was a problem with the patch, but the patch works perfectly!

The app is making requests to this endpoint (search for "earth"): https://search.wikimedia.org/?site=wikipedia&lang=en&search=earth&limit=31

Whatever API this is, Wikipedia must have turned it off, because now it just returns a web page. The Dictionary app, clearly not expecting to get back html data, completely chokes and actually crashes. Lion and above use a different API that still works.

 

[Wowfunhappy]

@Wowfunhappy, any chance of a 10.5.8/9 Leopard version of your proxy?

The problem is that AquaProxy is built using Go (because Go includes a standard library that does a lot of the work for me). As far as I know, there is no way to run binaries compiled by any version of Go on Leopard.

 

[rampancy]

I admit I haven't following the original thread on your original Legacy Mac Proxy project (for which I'm eternally grateful to, as it's been essential to help keeping my old Mac collection alive!), but what are the improvements/advantages/changes of Aqua Proxy over Legacy Mac Proxy? (Apart from the use of Go instead of Squid.)

 

[GalacticStag]

Phew! This was bothering me all day, so glad to hear it's working!


I'm not sure why it appeared to only happen after your second install, but this is (has to be) from Legacy Mac Proxy. You may notice that Aqua Proxy no longer allows users to install the Dictionary Patch on Snow Leopard. The uninstaller should have removed it, but I guess it didn't for you.

Now, here's the interesting thing—the Dictionary Patch actually works perfectly fine on Snow Leopard, insofar as it does what it's supposed to, it makes the Dictionary app respect the proxy settings in System Preferences and in so doing allows the app to actually connect to Wikipedia again.

The problem is that once the app can connect to Wikipedia, the data it gets back causes the app to crash! I wasted half a day figuring this out, I thought there was a problem with the patch, but the patch works perfectly!

The app is making requests to this endpoint (search for "earth"): https://search.wikimedia.org/?site=wikipedia&lang=en&search=earth&limit=31

Whatever API this is, Wikipedia must have turned it off, because now it just returns a web page. The Dictionary app, clearly not expecting to get back html data, completely chokes and actually crashes. Lion and above use a different API that still works.

Oh, i see! BTW, on the subject of wikipedia. The wikipedia widget crashes when browsing in english (it happened on LMP too, so yk. The API was probably causing it to crash too), but searches Wikipedia just fine when i change it to another language (for example, Spanish).

 

[Wowfunhappy]

I admit I haven't following the original thread on your original Legacy Mac Proxy project (for which I'm eternally grateful to, as it's been essential to help keeping my old Mac collection alive!), but what are the improvements/advantages/changes of Aqua Proxy over Legacy Mac Proxy? (Apart from the use of Go instead of Squid.)

Legacy Mac Proxy fixes lots of software, but it also *breaks* other software, including software that actually would have worked fine without a proxy! For example, the only reason Legacy Mac Proxy ever worked with iMessage was because I added a special rule to make the proxy exclude certain Apple domains. I used to think this was necessary because iMessage uses certificate pinning (which would mean it refuses to trust user-installed certificates), but it turns out that was completely wrong—there was never any certificate pinning, Squid was just breaking it.

Legacy Mac Proxy contained several exceptions like this for domains I knew were used with software that Squid would otherwise break. However, I of course can't predict all the software someone might use. I don't know all the types of connections that broke with Squid, but websockets in particular definitely did not work.

With this Go-based proxy, iMessage works fine, websockets were fine, and everything else I've tried seems to work fine. And this is when I explicitly set the proxy to decrypt and re-encrypt (or "mitm") all traffic for testing purposes.

Which brings me to the other great thing about this proxy! With Squid, all of your traffic gets mitm'd (except for connections to a limited number of specially-excluded domains, as mentioned earlier). By contrast, Aqua Proxy inspects your traffic to determine if it came from an app with modern HTTPS support. If Aqua Proxy is confident the connection will work without its help, it just sends the traffic unaltered instead of mitm'ing it. This makes it even less likely that Aqua Proxy will ever break something that otherwise would have worked on its own!

Basically, Legacy Mac Proxy made it impossible to follow a "first, do no harm" principle. Legacy Mac Proxy truly should never break anything, it will only ever help make stuff work.

 

[Wowfunhappy]

Oh, i see! BTW, on the subject of wikipedia. The wikipedia widget crashes when browsing in english (it happened on LMP too, so yk. The API was probably causing it to crash too), but searches Wikipedia just fine when i change it to another language (for example, Spanish).

Interesting, I can't replicate this with the Dictionary app. It's annoying because if I could see what data the app expects to get back, I could probably make a Cloudflare worker that puts things in the right format and then set up a URL redirect. Unfortunately, since the API is dead that doesn't work.

@f54da linked me to https://web.archive.org/web/2015091...wikipedia&search=Wikimedia_Foundation&limit=1 which would indicate it just wants langCode_pageName, but the Dictionary doesn't like that.

 

[GalacticStag]

Legacy Mac Proxy fixes lots of software, but it also *breaks* other software, including software that actually would have worked fine without a proxy! For example, the only reason Legacy Mac Proxy ever worked with iMessage was because I added a special rule to make the proxy exclude certain Apple domains. I used to think this was necessary because iMessage uses certificate pinning (which would mean it refuses to trust user-installed root certificates), but it turns out that was completely wrong—there was never any certificate pinning, Squid was just breaking it.

Legacy Mac Proxy contained several exceptions like this for domains I knew were used with software that Squid would otherwise break. However, I of course can't predict all the software someone might use. I don't know all the types of connections that broke with Squid, but websockets in particular definitely did not work.

With this Go-based proxy, iMessage works fine, websockets were fine, and everything else I've tried seems to work fine. And this is when I explicitly set the proxy to decrypt and re-encrypt (or "mitm") all traffic for testing purposes.

Which brings me to the other great thing about this proxy! With Squid, all of your traffic gets mitm'd (except for connections to a limited number of specially-excluded domains, as mentioned earlier). By contrast, Aqua Proxy inspects your traffic to determine if it came from an app with modern HTTPS support. If Aqua Proxy is confident the connection will work without its help, it just sends the traffic unaltered instead of MITM'ing it. This makes it even less likely that Aqua Proxy will ever break something that otherwise would have worked on its own!

Basically, Legacy Mac Proxy made it impossible to follow a "first, do no harm" principle. Legacy Mac Proxy truly should never break anything, it will only ever help make stuff work.

Is this why, for example, iMessage didn't work on 10.8 or certain third party minecraft launchers (like LauncherFenix) couldn't download their updates?

 

[Wowfunhappy]

Is this why, for example, iMessage didn't work on 10.8

iMessage domains were excluded from the old proxy, so unless 10.8 used different domains then 10.9, that's not it. Oh, unless iMessage on Mountain Lion actually needs a proxy to work in the first place, and couldn't because it was excluded—in that case, there's some possibility it'll work now with Aqua Proxy!

or certain third party minecraft launchers (like LauncherFenix) couldn't download their updates?

Possibly. I haven't tried it, but that's the sort of software I could imagine not working properly with Squid. You'd have to try it with AquaProxy and see if it works now!

 

[maverick28]

I want to chime in. I installed Aqua and rebooted several times. OS X Mavericks. Skimming through the previous posts, I noticed the certificate issue. I looked up the Keychain Access - ISRG Root X1 expired last year and was marked as untrusted. I "always-trusted" it, then rebooted, but no difference.

Wikipedia in the Dictionary doesn't display any content except "No entries found". It did just before the proxy installation.

The Console won't show anything; however, Charles will. This is the curl request sent.

curl -H "Host: lookup-api.apple.com" -H "User-Agent: AppleDictionaryService/208" -H "Proxy-Connection: close" http://lookup-api.apple.com/en.wikipedia.org/w/api.php?action=opensearch&search=benno_moiseiwitsch&limit=15

This is the response code from the overview tab of Charles Request Inspector:

Response Code: 301 Moved Permanently

Moved permanently in 2 minutes?

 

[Wowfunhappy]

I looked up the Keychain Access - ISRG Root X1 expired last year and was marked as untrusted.

...I don't know what is going on, but something is quite wrong on your system. ISRG Root X1 shouldn't expire until 2035. That's a real certificate, not something that gets generated, it should be exactly the same for everyone. If you truly have a rogue root certificate on your system claiming to be ISRG Root X1, that is potentially very scary. Does your copy of the certificate have a public key that starts with AD E8 24 73 F4 14 37 F3 9B 9E 2B 57 28 1C 87 BE DC B7 DF 38?

What about all the other certificates installed by Aqua Proxy? There's USERTrust RSA Certification Authority, COMODO ECC Certification Authority, DigiCert Global Root G2, and DigiCert Global Root G3. Let me know if you don't have certificates with these exact names.

Did you run the AquaProxy uninstaller to clean up Squid before installing AquaProxy?

And then, could you please share the results of the remaining troubleshooting steps I gave to GalacticStag?

• Share the output of running launchctl list | grep AquaProxy in the Terminal
• Show me some screenshots of System Preferences → Network and System Preferences → Network → [your active connection] → Advanced → Proxies
• Confirm that the Aqua Proxy certificate is installed and trusted in Keychain Access's System Keychain.
• In Safari, if you try to view a page and click the lock icon in the upper right corner, what do you see?

Edit: Also, are you sure the other network utilities you've apparently been using lately (stunnel for example) aren't interfering? I test my software on clean, unaltered systems, but if you've been messing with lots of stuff it gets very tricky for me to know what's going on. (This isn't a criticism, I just can't practically know what is happening on your computer.) Depending on how bad things have gotten you might consider an OS re-install.

 

[maverick28]

@Wowfunhappy

1. The Aqua cert is installed and universally trusted
2. In Safari, the root cert is the Aqua cert. Sites load normally.

3. certtool y x=a | egrep 'ISRG Root X1' -A 20 | awk '/Pub\ ?[Kk]ey\ ?[Bb]ytes/ {print $0}'
   --->  Pub key Bytes      : Length 526 bytes : 30 82 02 0A 02 82 02 01 ...
   Pub key Bytes      : Length 270 bytes : 30 82 01 0A 02 82 01 01 ...
   Pub key Bytes      : Length 270 bytes : 30 82 01 0A 02 82 01 01 ...
   Pub key Bytes      : Length 97 bytes : 04 D9 F1 9E 46 87 F8 21 ...
   Pub key Bytes      : Length 270 bytes : 30 82 01 0A 02 82 01 01 ...

4. sudo security find-certificate -a -c "ISRG Root X1" | grep keychain
   --> keychain: "/Users/me/Library/Keychains/login.keychain"

5. The following command generated the attached cert, which certtool missed (I fed it to qlmanage to peek immediately into the result upon the cert dump):
   
   

   sudo security find-certificate -a -c "ISRG Root X1" -p > ~/Desktop/'ISRG Root X1'.pem ; sudo qlmanage -p -c public.x509-certificate -g /System/Library/QuickLook/Security.qlgenerator ~/Desktop/'ISRG Root X1'.pem > /dev/null 2 &> 1

   
   This cert's pubkey starts with the hex digits you posted.
6. This is my network configuration regarding a proxy, as per the system config plist.
   
   

   PlistBuddy -c 'Print :NetworkServices:573403F3-F434-452D-BCFF-80A584C68374:Proxies' /Library/Preferences/SystemConfiguration/preferences.plist
   
   --> Dict {
   HTTPEnable = 1
   HTTPPort = 8888
   HTTPSProxy = 127.0.0.1
   HTTPSPort = 8888
   FTPPassive = 1
   HTTPProxy = 127.0.0.1
   HTTPSEnable = 1
   }

   
   

   PlistBuddy -c 'Print :NetworkServices:573403F3-F434-452D-BCFF-80A584C68374:Interface' /Library/Preferences/SystemConfiguration/preferences.plist
   
   --> Dict {
   UserDefinedName = Wi-Fi
   DeviceName = en1
   Type = Ethernet
   Hardware = AirPort
   }

7. Disabling Charles proxying falls back to localhost:6531, but makes no difference.

 

[Wowfunhappy]

In Safari, the root cert is the Aqua cert. Sites load normally.

Wait, so then Aqua Proxy is working...?

...I think I misunderstood, is it only the Dictionary app that doesn't work for you?

This is my network configuration regarding a proxy, as per the system config plist.

Well, that's the wrong configuration for Aqua Proxy. As the readme says, for Aqua Proxy, System Preferences should be set to localhost and 6531.

I understand you are using Charles Proxy. I've only ever used that on Windows and I don't know how it affects things on OS X. If you are having trouble, my first recommendation would be to change your proxy settings to point to Aqua Proxy directly, and potentially try uninstalling Charles Proxy altogether depending on how invasive it is.

 

[maverick28]

Yes, I have problems only with the Dictionary, see my earlier post in this thread for the curl response. I don't use stunnel, and it's not running in the background. As I said, disabling Charles fails to bring back the connection to the Dictionary Wiki, unless I have to reboot to effect every modification.

 

[Wowfunhappy]

see my earlier post in this thread for the curl response.

I don't actually understand what this is. The Dictionary app does not use curl, it uses Apple's networking APIs.

As I said, disabling Charles fails to bring back the connection to the Dictionary Wiki, unless I have to reboot to effect every modification.

But when you say CharlesProxy "falls back" to the standard AquaProxy settings (localhost:6531), is that CharlesProxy doing something automatically, or are you going into System Preferences and actually putting in the right settings? Basically, I don't trust CharlesProxy to not mess something up.

---

One other thing you could try is completely restoring the Dictionary app using a tool like Pacifist, and then reinstalling Aqua Proxy and its Dictionary patch, just in case something went wrong with the Dictionary patch. I don't really think this will work, but I'm otherwise out of ideas.

By the way, did you get iCloud mail working?

 

[maverick28]

No, I didn't say that Charles falls back to 6531. I said disabling Charles falls back to the default Aqua port, which is 6531. The curl form of the request is the way that Charles allows to send requests manually. As I understand it, apps send the same POST, GET, CONNECT, REPORT and other requests that curl does with the "-H" param.

I started using Charles when Squid was the SSL proxy, and the Dictionary Wiki worked flawlessly, which brings me to the conclusion that something went wrong with the binary. Perhaps, Dictionary.app on my system didn't need the tweaked version.

So, are we factoring out the ISRG cert, or do I need to get the new one?

 

[maverick28]

is Dictionary_backup the original pre-modified binary? Replacing Dictionary with Dictionary_backup (and renaming to the orig name, of course) crashed the app. Mind if I send the crash report?

 

[Wowfunhappy]

So, are we factoring out the ISRG cert, or do I need to get the new one?

If the signature matches it should be fine, although I don't understand how it could have expired. I'm sorry, but I'm out of ideas other than restoring the Dictionary app from a backup.

I said disabling Charles falls back to the default Aqua port, which is 6531.

Do you or don't you have your HTTPS proxy settings set to localhost and 6531 in System Preferences specifically? If it's set to anything other than those two things in the system preferences app, my recommendation is that you change it. Maybe I'm wrong and it won't do anything, but that is my advice, take it or leave it.

My second piece of advice is that you restore the dictionary app from a backup using Pacifist and then try reinstalling the patch with Aqua Proxy. If neither of those work I am out of ideas, I'm sorry.

is Dictionary_backup the original pre-modified binary?

Yes, but you can't just restore the backup binary, this will always cause the app to crash due to the modified info.plist breaking the code signature.

Aqua Proxy's uninstaller should be able to restore the app properly, but if something's not quite right... you should just restore the entire app from a backup or an OS X installer, it's safer.