Support of Wikipedia ended in Dictionary app

[maverick28]

It looks like just a couple of hours ago Wikipedia changed something on its end to drop support of several old versions, including Mavericks. I was able to read their articles as recently as today. Now, the window is empty when typing in search terms and keywords. The previous year they did that for anything older than Mavericks, and I lost the access to wiki-articles from within Dictionary in Lion. That's regrettable, since that was more convenient than opening a browser.

It would be a marvel if someone could hack or use another API scheme to revive the functionality just like it was done with the Weather widget for Dashboard.

 

[Wowfunhappy]

Well, this sucks.

It might be fixable. If you go into Contents/Resources, there's an editable WikipediaSupport.js file. The only problem is that Dictionary isn't logging anything to console, which makes it hard to figure out what's happening.

---

Update 1: Scratch that, all the JS/CSS files are the same between Mavericks and Yosemite, and in Yosemite Dictionary still works. So, what the heck actually broke?

---

Update 2: It would probably help if I was actually looking in the right place. /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/Resources/Wikipedia.wikipediadictionary

---

Update 3: This should be such an incredibly simple fix. The Wikipedia query URL is sitting right in info.plist, in plain text:

http://%@.wikipedia.org/w/api.php?action=opensearch&search=%@&limit=%d

Unfortunately, the query URL, like everything else, is completely identical between Mavericks and Yosemite. And yet in Yosemite it works, and in Mavericks it does not.

---

Update 4: This isn't a real fix, but I'm tempted to just add a line in my setup script which deletes Wikipedia.wikipediadictionary. This completely removes Wikipedia functionality from the Dictionary app, at which point I can look for a nice third party app and forget that this was ever an OS feature...

---

Update 5: Well this is interesting. Turn off Squid, and go to Wikipedia.org in Safari. You'll be redirected to https://www.wikipedia.org/sec-warning.

This is EXACTLY the type of problem that Squid is intended to fix. Unfortunately, I don't think that Dictionary.app is sending traffic through the proxy like it's supposed to—it never shows up in access.log. That would however make it the only Apple app that ignores System Preferences's proxy settings, which is extremely strange. Could be an Apple bug...

 

[Wowfunhappy]

Edit: Should not have posted too soon, still investigating....

 

[maverick28]

You were right. Added the exception in Network Settings-->Advanced and immediately (upon visiting) got the warning "Your browser is outdated". But maybe it's worth taking a closer look at that javascript file inside the app bundle? It mentions some proxy method, but if it's overriden then is it because of the certificate pinning? What certs does Wikipedia use?

NB. There's no good apps for Wikipedia, I checked today, virtually every app has the overwhelmingly bad ratings, and the only one that is said to do its job properly is Kiwix but it requires a separate download of the whopping GB-sized uncompressed file (one per language).

 

[Wowfunhappy]

I think that code comment is referring to a different sort of proxy—like how I set up the weather widget to "proxy" requests through Cloudflare. (I don't know exactly what it means though.)

if it's overriden then is it because of the certificate pinning? What certs does Wikipedia use?

No, it's not certificate pinning, it's just flat-out not sending traffic through the proxy. Like how you can configure Firefox to not send traffic through the proxy (although in Firefox's case, it's optional).

At least, that's what I initially thought. I'm seeing a few other things that make me think I might have been wrong... I think there's actually a chance the problem is caused by Wikipedia's warning message confusing the Dictionary app. If true (big if), the problem may go away on its own after November 1.

 

[maverick28]

But the fact it opened with Squid is telling. What's supposed to happen on Nov. 1?
I still hold that considering they cut off pre-Mavericks macOSes from Wikipedia it wouldn't be a bad thing to prepare for the inevitable, and maybe attempt to figure out with Wiki the same tricks like those that you've done with Squid and Weather. If within a realm of the possible, of course.
I too have noticed that Console doesn't output any records related to "Dictionary", I thought that was odd.

 

[Wowfunhappy]

What's supposed to happen on Nov. 1?

The message goes away, and the app starts working again because in actuality, Wikipedia still supports at least one compatible cipher suite, it's just that the dictionary app doesn't know what to do when it gets redirected. I think this is fairly unlikely, but possible.

See, the odd thing is that if you change the Wikipedia query URL to start with https instead of http, search results will start appearing again (but the articles themselves will not load, so this is not useful). This happens even without Squid. I don't know what's going on, but it's telling me that I haven't found the real problem. So at the moment, I'm stuck.

Now, if the above paragraph is just a red herring, and the problem can be boiled down to a combination of:

1. Wikipedia has dropped support for every Mavericks-compatible https cipher suite.
2. For some ludicrous reason, Apple programmed the Dictionary app to ignore the system proxy settings.
3. The Wikipedia query URL can be controlled via /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/Resources/Wikipedia.wikipediadictionary/Contents/Info.plist

Then, I would be able to fix the problem by modifying the query URL to the address of a Cloudflare worker, and programming that worker to:

1. Retrieve the list of Wikipedia results, modify the URLs to also get sent through the Cloudflare worker, and send those modified results back the app.
2. Retrieve actual Wikipedia articles, and send those back to the app as well.

I'm writing this up mostly as a note to self, in case I decide to work on it in the future. At the moment, I just don't want to—it's a massive hack that would leave me testing edge cases and tweaking stuff for a long time.

The reason I spent so much time compiling and configuring Squid is because it was a way to fix a lot of different problems at once—as well as guard against new problems going forward as more sites drop cipher suites. (Problems like this one, in fact—except that in this case, Apple appears to have done something unbelievably stupid.)

 

[maverick28]

One idea suddenly came to me, related to how the Wikipedia engine in Dictionary might connect to Wikipedia after I opened Keychain to look into details about one entry that was created by Xcode 5.1.1 on Mavericks. Xcode 5 syncs to online accounts such as AppleID, servers, and code repositories. I added links to various GitHub repositories to test the feature, the test was successful. The "Access" section of the corresponding keychain with my login info contained an item named com.apple.cfnetwork. That, suddenly, guided my thoughts in an unexpected direction: what if Dictionary uses this framework too? I then opened the built-in documentation (that is, within Xcode). In the introduction, Apple explains that

CFNetwork is a low-level, high-performance framework that gives you the ability to have detailed control over the protocol stack. It is an extension to BSD sockets, the standard socket abstraction API that provides objects to simplify tasks such as communicating with FTP and HTTP servers or resolving DNS hosts. CFNetwork is based, both physically and theoretically, on BSD sockets.

 

[Wowfunhappy]

CFNetwork is part of CoreFoundation, and used by lots of apps. It should respect the system proxy settings.

What might be interesting is if we could enable CFNetwork diagnostic logging. https://developer.apple.com/documen...ps_problems_with_cfnetwork_diagnostic_logging

I think it would require a SIMBL plugin...

Edit: Nope, much easier. https://lapcatsoftware.com/articles/logging-https.html Wish I knew about that sooner...
Edit2: Well that definitely didn't work.

 

[maverick28]

Just like I thought. My intuition didn't betray me. Dictionary entries in Console me searching Wikipedia.

CFNetwork SSL errors.

 

[Wowfunhappy]

Yep, started happening once Wikipedia took down the error page. I've also confirmed to my satisfaction that Dictionary isn't using the proxy (we basically new this, but I was seeing mirages that made me think it might be something else). Also happens in Snow Leopard and Mojave, so this is a very long-standing oddity.

The question is still what to do about it.

 

[maverick28]

In Mojave too? I fired up Dictionary as recently as a day ago, and it was alright.

 

[Wowfunhappy]

No, the Dictionary works in Mojave. But it ignores the system proxy settings.

 

[maverick28]

Dictionary makes only 1 connection (discovered using Mojave and HandsOff). I did that screenshot while reading a wiki article in Dictionary.

 

[Wowfunhappy]

Okay, at least I know for sure what's going on now.

The Dictionary app uses CFHTTPStream to access Wikipedia. And CFHTTPStream ignores the system proxy. https://developer.apple.com/forums/thread/69452

Thanks to 1110101001 for figuring that out: https://apple.stackexchange.com/que...ig-sur-dictionary-app-doesnt-use-system-proxy (read the comments on the question).

So it could be fixed with a SIMBL plugin that swizzled CFReadStreamCreateForHTTPRequest... ?

 

[Wowfunhappy]

Boo, never mind, it's too low-level for SIMBL. I'd have to use my original Cloudflare worker idea.

I basically have most of the code outlined. You'd edit the Wikipedia dictionary's URL to http://this-cloudflare-worker.workers.dev/search|||%@|||%@|||%d and then use this for the Cloudflare Worker:

addEventListener('fetch', event => {
    event.respondWith(handleEvent(event.request));
})

async function handleEvent(req) {
    let path = new URL(req.url).pathname.split("|||");
    if (path[0] === "/search") {
        let url = 'https://' + path[1] + '.wikipedia.org/w/api.php?action=opensearch&search=' + path[2] + '&limit=%d' + path[3];
        let response = forwardRequest(url);
        //APPEND http://this-cloudflare-worker.workers.dev/page|||original-url TO EVERY URL IN RESPONSE!!!
        return response;
    }
    else if (path[0] === "/page") {
        let url = path[1];
        let response = forwardRequest(url);
        return response;
    }
}

async function forwardRequest(url) {
    const init = {}
    let response = await fetch(url, init);

    return new Response(response.body, {
        status: response.status,
        statusText: response.statusText
    })
}

Since I can't actually run this there's probably a couple of obvious bugs somewhere, but it should basically work... except for one major piece. We need to append http://this-cloudflare-worker.workers.dev/page/original-url to every URL returned in the search results.

For html I would use Cloudflare's HTMLRewriter class, but Wikipedia returns search results as JSON. There's no reason rewriting JSON should be difficult, but I can't for the life of me figure out how to do it!

 

[Wowfunhappy]

I really can't take any credit for this, 1110101001 over on Ask Different—after already investigating the source of the problem—went ahead and wrote a library that makes Dictionary use the proxy server. Right now it requires running Dictionary with DYLD_INSERT_LIBRARIES, I'm going to add it into the binary with install_name_tool and then... I guess post a diff patch? I'll find some way to distribute it...

 

[maverick28]

Boo, never mind, it's too low-level for SIMBL. I'd have to use my original Cloudflare worker idea.

I basically have most of the code outlined. You'd edit the Wikipedia dictionary's URL to http://this-cloudflare-worker.workers.dev/search|||%@|||%@|||%d and then use this for the Cloudflare Worker:

addEventListener('fetch', event => {
    event.respondWith(handleEvent(event.request));
})

async function handleEvent(req) {
    let path = new URL(req.url).pathname.split("|||");
    if (path[0] === "/search") {
        let url = 'https://' + path[1] + '.wikipedia.org/w/api.php?action=opensearch&search=' + path[2] + '&limit=%d' + path[3];
        let response = forwardRequest(url);
        //APPEND http://this-cloudflare-worker.workers.dev/page|||original-url TO EVERY URL IN RESPONSE!!!
        return response;
    }
    else if (path[0] === "/page") {
        let url = path[1];
        let response = forwardRequest(url);
        return response;
    }
}

async function forwardRequest(url) {
    const init = {}
    let response = await fetch(url, init);

    return new Response(response.body, {
        status: response.status,
        statusText: response.statusText
    })
}

Since I can't actually run this there's probably a couple of obvious bugs somewhere, but it should basically work... except for one major piece. We need to append http://this-cloudflare-worker.workers.dev/page/original-url to every URL returned in the search results.

For html I would use Cloudflare's HTMLRewriter class, but Wikipedia returns search results as JSON. There's no reason rewriting JSON should be difficult, but I can't for the life of me figure out how to do it!

Sounds promising,
Could you, please, issue step by step instructions on what exactly should be inserted where? Most notably, where do I put the URL reference to my worker and do I need to edit lines or segments in the code above?

Sending GET while testing it on Cloudflare returns Internal server error 500

 

[Wowfunhappy]

The Cloudflare worker doesn't work because a major piece of the code is missing. And now that 1110101001 wrote code to make Dictionary use the system proxy, I won't be going that route anyway.

I'll do one better than providing instructions—I'm going to make a nice, proper pkg that both installs Squid _and_ the Dictionary fix!

 

[Wowfunhappy]

All done. Now integrated into the overall proxy server package. https://jonathanalland.com/downloads/wowfunhappy-https-proxy.dmg. It should install cleanly over an existing copy of Squid, except you may need to remove the current Squid certificate from Keychain Access and import the new one from Finder.

I'm sorry if following this thread was confusing, I was kind of using it as a place to leave notes to myself. It's also not lost on me that there's now quite a few "Mavericks Fixes" scattered across the forum, and it's probably getting hard to find them. At some point, I'm planning to put together a big page on my website with all of these tools, but, well, but I haven't had a chance yet.

(This was not supposed to consume my entire evening and I'm not particularly happy about it. Oh well...)

 

[maverick28]

Thank you, I'll try. Will it work in older macOS just like v.1?

 

[maverick28]

All done. Now integrated into the overall proxy server package. https://jonathanalland.com/downloads/wowfunhappy-https-proxy.dmg. It should install cleanly over an existing copy of Squid, except you may need to remove the current Squid certificate from Keychain Access and import the new one from Finder.

I'm sorry if following this thread was confusing, I was kind of using it as a place to leave notes for notes to myself. It's also not lost on me that there's now quite a few "Mavericks Fixes" scattered across the forum, and it's probably getting hard to find them. At some point, I'm planning to put together a big page on my website with all of these tools, but, well, but I haven't had a chance yet.

(This was not supposed to consume my entire evening and I'm not particularly happy about it. Oh well...)

You and these other people did it! I even don't know how to express my gratitude but any words are not enough. It may be placebo but with v.2 it feels faster. iTunes Store, App Store and Books Store connect and load the contents on the go. Additionally, an annoying pull-down error alert in MAS appearing me clicking on the "Updates" section is gone. It may be a coincidence but until I installed the version 2 I had iMessage and Facetime login problem after I changed my AppleID (that is no logging in at all): either Apple fixed that or Squid solved that too.
But Dictionary was the most jarring experience. Can't believe it's back.

 

[maverick28]

In macOS 10.7 it crashes with the report ("CFNetwork Library not loaded"). I attach the report. If not possible to fix then so be it - the main Dictionary functions will be enough without Wikipedia.

 

[Wowfunhappy]

In macOS 10.7 it crashes with the report ("CFNetwork Library not loaded"). I attach the report. If not possible to fix then so be it - the main Dictionary functions will be enough without Wikipedia.

That's interesting. I may need to compile ProxyFix on 10.6 to make it compatible with 10.6+. I thought for such a simple library it wouldn't matter, but clearly not.

For now, you can uninstall the Dictionary patch by:

1. In Dictionary.app/Contents/MacOS, rename Dictionary_backup back to Dictionary (delete existing).
2. Delete Dictionary.app/Contents/Frameworks/ProxyFix.dylib
3. In Terminal: sudo defaults delete /Applications/Dictionary.app/Contents/Info.plist LSEnvironment

It may be placebo but with v.2 it feels faster. iTunes Store, App Store and Books Store connect and load the contents on the go. Additionally, an annoying pull-down error alert in MAS appearing me clicking on the "Updates" section is gone. It may be a coincidence but until I installed the version 2 I had iMessage and Facetime login problem after I changed my AppleID (that is no logging in at all): either Apple fixed that or Squid solved that too.

Yeah, the speed is probably a placebo. I didn't update Squid itself in any way. The only reason I can think of for the other stuff getting fixed is the installer probably overwrote any changes you made to squid.conf.