El Capitan style adblocking?

[Somnesis]

Hey

We know that El Capitan and iOS 9 comes with native content blocking support for Safari which let's you write adblockers much more efficiently.

Are there any such extensions already made / in development?

 

[KALLT]

Several developers have expressed some disappointment and are in touch with Apple's developers to improve the content blocking APIs before release. I suppose the major adblock developers are not going to roll out new versions soon because of that. The old extensions do still work for the time being. There are some tutorials on the web that you can use to build one yourself though.

 

[rnbwd]

By 'several developers' are you referring to ABP? Despite objections on their blog, the new safari API threatens their business model. I've only seen critiques of the API's syntax a few days after the announcement, I'm legitimately curious how it works. The 'ad hoc' methods of using JS to block elements on a page via regex of URI's, is probably not going to be more effective than the methods exposed by safari to developers.

Most of the sites being blocked are from 3rd party forums, lists, etc. that have been compiled and updated for years. ABP is complaining that the 'syntax isn't standard' and they need to write 'special script to convert' - but - they're already doing that to the 3rd party lists so that it conforms to their syntax. It's just as simple to convert APB's lists (or the originals) into a json file. The only thing the new API threatens are the relevancy of every ad blocker currently in existence because they've been replaced with a JSON file - and their 'block lists' have and always will exist independent of them.

I haven't seen any in active development yet - which surprises me. I talked to support on the phone and they didn't even know what I was talking about.

 

[KALLT]

By 'several developers' are you referring to ABP? Despite objections on their blog, the new safari API threatens their business model. I've only seen critiques of the API's syntax a few days after the announcement, I'm legitimately curious how it works. The 'ad hoc' methods of using JS to block elements on a page via regex of URI's, is probably not going to be more effective than the methods exposed by safari to developers.

Most of the sites being blocked are from 3rd party forums, lists, etc. that have been compiled and updated for years. ABP is complaining that the 'syntax isn't standard' and they need to write 'special script to convert' - but - they're already doing that to the 3rd party lists so that it conforms to their syntax. It's just as simple to convert APB's lists (or the originals) into a json file. The only thing the new API threatens are the relevancy of every ad blocker currently in existence because they've been replaced with a JSON file - and their 'block lists' have and always will exist independent of them.

I haven't seen any in active development yet - which surprises me. I talked to support on the phone and they didn't even know what I was talking about.

If you check the WebKit mailing list, you can see that there are more people who think the blocklist format is a bit too limited. Apple is deprecating the previous JavaScript methods and the replacement isn't as powerful yet to encompass the whole range. For instance, I read that you can't selectively (un)block certain websites on the fly other than by disabling the blocker entirely or by recompiling the blocklist. This is certainly going to cause some problems for extensions like Ghostery as well.

Adblock Plus will hardly complain about the loss on the Safari front, the browser is not making a dent in global browser statistics and the competition is intense already. AdBlock (their competitors) had a decent extension even before ABP bothered to write one.

 

[nontroppo]

Do you guys *really* think the new API can replace an excellent blocker like uBlock₀?

https://github.com/gorhill/uBlock/wiki/Overview-of-uBlock's-network-filtering-engine

I haven't looked into it, but I assume the Safari API only covers the simplest of url filtering. How does it handle dynamic rules, or cosmetic filtering? A good blocker is much more than a fancy regex and a list of resources...

 

[KALLT]

Do you guys *really* think the new API can replace an excellent blocker like uBlock₀?

https://github.com/gorhill/uBlock/wiki/Overview-of-uBlock's-network-filtering-engine

I haven't looked into it, but I assume the Safari API only covers the simplest of url filtering. How does it handle dynamic rules, or cosmetic filtering? A good blocker is much more than a fancy regex and a list of resources...

That’s pretty much the criticism in a nutshell. So far the content blocker extension supports either blocking or hiding of elements and blocking of URLs. I suppose developers will have to come up with a dynamic approach that lets Safari do the heavy lifting and let the extension do the rest. The question is whether that will be possible in the future, as Apple is deprecating several methods from the extension framework altogether in anticipation of this, and whether it won’t defeat the purpose in the first place when extensions still need to execute an extensive amount of code. In the best case it requires at least a partial rewrite of the code, with the prospect of significant changes to the structure of the code, and the worst case that we end up with a much less effective adblocker or tracker blocker.

 

[rnbwd]

Do you guys *really* think the new API can replace an excellent blocker like uBlock₀?

https://github.com/gorhill/uBlock/wiki/Overview-of-uBlock's-network-filtering-engine

I haven't looked into it, but I assume the Safari API only covers the simplest of url filtering. How does it handle dynamic rules, or cosmetic filtering? A good blocker is much more than a fancy regex and a list of resources...

Ublock is my extension of choice - their wiki is my basis for understand how it all works. But if you're curious, I'd highly recommend watching the wwdc session on content blocking.

I was skeptical at first... until I built a small extension (based on this blog post) to test it out. I was pleasantly surprised - it decreased the load time of imore.com from 11s to 2s. It prevents network requests from occurring, like uBlock, but these requests don't show up in the console or even network timelines. It's potentially accessing system-level network proxies, focusing on privacy and performance. This might be rhetoric, but it's also trivially easy to develop. According to this article on Fortune (take it with a grain of salt):

Apple isn’t blocking ads itself. But they’ve done the next best (or worst) thing: They’ve given developers easy-to-use tools to build better, faster and more efficient ad blockers.

“I suspect there are going to hundreds if not thousands of ad blockers in the App Store from day one,” says Marco Arment, developer of Instapaper and Overcast. “It’s going to be a massive rush, because they’re just so easy to make.”

Asymco‘s Horace Dediu, who chooses his words carefully, calls ad blocking in iOS 9 “an atomic bomb.”

“This is a cataclysmic potential event that’s going to affect the whole ad industry.”

“iOS is a completely new place for us,” says Eyeo’s Ben Williams. He points out that Apple’s new APIs block ads not just from the browser, but form the OS itself, making it hard to do what Williams calls “sustainable” ad blocking.

“We know that ads fuel the Internet,” he says. “We’re trying to be the responsible ad blocker.” Adblock Plus (not to be confused with all the other Adblock) lets through what it deems acceptably non-intrusive ads as well as ads from big advertisers like Google and Microsoft that have paid to have their feeds “whitelisted.”

Based on Tim Cook's comments on privacy and encryption, and the resignation of an iAd exec because:

“I don’t believe they [Apple] are interested in this capability because they have a strict policy around what they do with user data,” Crawford said. “IAd has great assets and great capabilities, but they are going to follow Apple’s policy to the letter of the law.”

I think this looks promising. I'm a little disappointed that they're depreciating features in anticipation of this... but if this isn't just hype - YES, uBlock and most ad blocking extensions will become obsolete - at least for iOS and safari (not jailbroken). Out of the box, I expect it to be more effective at blocking ads and trackers than what was possible via extensions. But that's just speculation..

Here's ABP's blog post (with responses) for reference.

 

[nontroppo]

Well, at least ABP now claims the Safari API is insufficient for ABP style filter lists:

https://lists.webkit.org/pipermail/webkit-help/2015-June/003918.html

And uBlock does a lot more than ABP does (see the filter engine diagram). uBlock has an excellent visual logging table (advanced user mode), and ensures background requests, not only page requests are caught, along with ensuring any privacy-leaking settings are disabled. Dynamic rules are much more powerful addition to the filter lists of ABP. Does the API enable DOM inspector visual blocking?

I still can't see this API doing much on OS X (perhaps it can be used to reduce CPU for some extensions). iOS is a different matter as getting any extension working is currently painful or not possible. This is where this API will probably come in — are Apple opening extensions fully for iOS 9?

 

[rnbwd]

It's the same extension API for iOS9 and OSX 10.11 - uBlock does do a lot more than ABP, specifically with privacy leaking and background requests (all network requests in general). ABP focuses more on the superficial aspects of hiding elements on the DOM, while uBlock focuses more on network requests for privacy and performance gains. The safari JSON api is not just meant for superficial element hiding - which is the bulk of ABP's complaint - it's stopping network requests (including background requests) for increased privacy / performance of page loads.

The source code for uBlock is all up on github and you can check it out in the console if you have it installed right now. It's not trivial - but the new content blocking json API is trivial - providing easy access to high level network blocking - which is much more effective and meaningful than hiding elements on a page. Plus it's shared between iOS and OSX - write it once and you're done. (I'm also pretty certain the rules can be changed in an app fairly easily)

Edit: After reading through the ABP email carefully - their main complaint appears to be that the safari API will block too much content and that their rules for unblocking and allowing content will be more difficult. The complaint is that Safari's blocking is too powerful.. API is too simple to encompass the full spectrum of features that ABP (and similar blockers) provide. The options are to block an entire domain OR use a less encompassing method to block elements - but IMO blocking requests is by far the most powerful / performant / private feature you could provide in a content blocker (vs loading the resource and then hiding elements)

 

[soy]

Gonna bump because I'd love for someone to try their hand at this.

I downloaded the BlockParty source code from https://github.com/krishkumar/BlockParty and compiled it for my phone. It's amazing how much faster everything loads!

Tried to then use the json file from BlockParty to build a Safari Extension, but no luck - while Xcode 7 allows us to build apps and load them on our on phones for testing, you need to be part of the paid developer program in order to build Safari Extensions and run them on your own system.

Any paid developers want to try and post a content blocker?

 

[rnbwd]

> you need to be part of the paid developer program in order to build Safari Extensions and run them on your own system.

How are you compiling apps on your phone if you're not in the paid dev program? I think you just need to download the certs for safari extensions, but I need to try it out first. I actually found one lib on github using the extension for osx:

https://github.com/dgraham/Ka-Block

I also found this library which is making strides converting the easylists used by abp into the safari format: https://github.com/TheFuzzball/EasyListSafari

The blockerList.json has the same syntax as ka-block

edit: diving into the source of ublock - there are regexes parsings out all the urls to modify what's happening on the page and define the syntax - those regexes can be used to generate static JSON files in the Safari format..

 

[KALLT]

How are you compiling apps on your phone if you're not in the paid dev program? I think you just need to download the certs for safari extensions, but I need to try it out first.

Apple lets you compile for and run iOS apps on your own devices with Xcode 7. They also discontinued the Safari dev programme and merged it with the new combined developer programme. You can’t compile any Safari extensions anymore without it. All existing extensions will continue to work, but can’t be updated anymore without it. Eventually, developers will have to sign up and submit their extensions to Apple’s gallery, otherwise they don’t get automatic updates anymore.

 

[soy]

Compiling for phone can be done through Xcode 7 beta. It's available for free through Apple, and it allows you to compile apps an run them on your phone. It no longer requires a paid membership.

Sadly, nothing so far allows for Safari (desktop) extensions to be built without the paid dev certificate.

(side-note: tried to compile the GBA4iOS source as well, but it gives build errors that I can't figure out)

As to the two links, I had found those too. Ka-Block works, but it leaves giant gaps where ads were meant to be. On my phone with BlockParty it doesn't do that, so I don't know for sure if that's how the desktop versions will act, or whether that will be resolved before release. Ka-Block is also hybrid old-style adblocking for older Safari, and new-style for Safari 9, but I can't tell from the source code whether it actually switches to new-style reliably or not. A new-style only extension would be good so we can see it in action (and so we can tell whether or not the annoying gaps remain).

The easy list is just a json, which I could use to update the list in BlockParty if I want to (and might once the whole list is converted, it's only just over halfway done), but again can't turn into a .safariextz file.

 

[KALLT]

diving into the source of ublock - there are regexes parsings out all the urls to modify what's happening on the page and define the syntax - those regexes can be used to generate static JSON files in the Safari format..

It’s still strict, however. There are additional problems, like: there is a limited amount of rules a blocker can have and Apple also said that WebKit may ignore certain complex rules.

 

[SG-]

follow this guide to create an iOS 9 Safari blocker extension, and simply paste in the JSON from any of the other projects. there is zero code to do here other than filling in a few fields and pasting in that large JSON chunk:

https://www.hackingwithswift.com/safari-content-blocking-ios9

 

[rnbwd]

Weird - I guess a lot of things changed with xcode 7. Regarding limitations in the amount of rules - I think it's a trade off in complexity. It's difficult maintaining an ad-blocking extension - and this new JSON API could account for ~ 80% of the codebase for ABP or uBlock. I recognize that new JSON API isn't as flexible as the configuration files you can put into uBlock or ABP - but now the difficultly level of building a content-blocking extension is equivalent to configuring a content blocker.

I had to delete ad-blocker when I started web-dev because there was an error in the console associated with jQuery - which I hadn't included in my project - and that's when I realized that these ad-blockers had full access to the DOM, they were doubling the memory footprint of my webpage. It's not efficient to block resources with what's available via the Javascript API, and it's also a potential invasion of privacy. The Safari JSON API is more efficient, easier to maintain, and less invasive than an equivalent extension that's directly manipulating the DOM.

With that being said - it's not standard - and it's weird. It's annoying for maintainers of content-blocking exts. - and it's even more annoying for the billions of dollars of revenue that are potentially going to be lost over the next few years. But there's also no reason for the maintainers of ABP or uBlock or (whatever) to even maintain a safari app at this point because literally anyone can do it now - so I think that's awesome. Chrome / Edge will NEVER adopt an API like this, mozilla *might* - but they probably won't. Safari's on it's own.

 

[KALLT]

With that being said - it's not standard - and it's weird. It's annoying for maintainers of content-blocking exts. - and it's even more annoying for the billions of dollars of revenue that are potentially going to be lost over the next few years. But there's also no reason for the maintainers of ABP or uBlock or (whatever) to even maintain a safari app at this point because literally anyone can do it now - so I think that's awesome. Chrome / Edge will NEVER adopt an API like this, mozilla *might* - but they probably won't. Safari's on it's own.

I think everyone agrees that its a good initiative. It is simply puzzling, however, that Apple deprecates the previous methods entirely. By terminating the developer programme and making it paid, they will likely scare off many hobbyists and destroy the wealth of extensions that we currently have if those require modifications. Think of content blockers for particular sites or with particular goals other than broad ad blocking. Also, YouTube ads are notoriously slippery, so I wonder whether extension developers can continue to tackle those effectively. Since the content blockers are compiled once and not dynamically, Google could make this a lot trickier.

I personally don’t care about the business models of these adblockers. They can still take on donations (AdBlock), or sell white list spots (AdBlock Plus) or sell other professional software (AdGuard). I am not at all concerned about that. Personally I am rooting for AdGuard to come up with a good extension soon, as they develop apps for OS X and mobile devices too. This is an opportunity for them.

But there's also no reason for the maintainers of ABP or uBlock or (whatever) to even maintain a safari app at this point because literally anyone can do it now - so I think that's awesome.

You can’t write your own Safari extensions for free though. Only for iOS will this work.

Chrome / Edge will NEVER adopt an API like this, mozilla *might* - but they probably won't. Safari's on it's own.

Nonsense, they could follow. If Safari keeps a promise of efficient blocking, I’m sure others will follow at some point. Browsing speed in particular is a hard-fought goal.

 

[rnbwd]

off many hobbyists and destroy the wealth of extensions that we currently have if those require modifications. Think of content blockers for particular sites or with particular goals other than broad ad blocking. Also, YouTube ads are notoriously slippery, so I wonder whether extension developers can continue to tackle those effectively. Since the content blockers are compiled once and not dynamically, Google could make this a lot trickier.

I had no idea that API's were being depreciated before I joined this thread. I would like to see the content.JSON as syntactic sugar over what already exists - best of both world It makes no sense from a dev point of view - it's purely financial and political decision on their part.

The new API is legit - but why does it exist? It's the most *powerful* one out there (not necessarily flexible) - it also has the potential for be censored easier. IMO they're going to have apps like 'news', etc. be their source of ad revenue. They're intentionally trying to **** over google and FB (their competition) - and the difference between the companies is that Apple sells $10,000+ watches, $2000 computers, and they don't need ad money - while google / fb rely almost entirely on ad money.

It's competition (apple doesn't need ad revenue) - and MOBILE APPS ARE THE WORST FREAKING THING TO EXIST ON THE INTERNET - it's a huge win for apple. Plus they have so much control over the app store... at least I can make my own

 

[PowerBook-G5]

So . . . El Capitan has been released and I've heard nothing about new Content Blockers. Does anyone have any info on any?

 

[Somnesis]

Yeah I'd love to hear about this. Seems like the world went on fire when the iOS content blockers came out. Not a peep for the Mac. Typical

 

[KALLT]

Check out the Safari extensions gallery, that’s the likely place where you will find them.
https://safari-extensions.apple.com/?category=mostrecent

I just had a look at some of the recent adblockers in the Safari extensions gallery. The following extensions use the new content blocker APIs (I’ve had a glance at the source code):

• Adamant
• Ka-Block!
• Roadblock
• Wipr
• NoThirdParty

AdBlock, Adblock Plus, Adguard and Ghostery don’t use the new method yet.

 

[Wheelie4]

If I'm not mistaken don't all the security Extensions have to be adapted to use the new content blocker APIs to make it on their new extension page now? If so Adblock, Adblock Plus and Adguard are included.

When I checked earlier today Adblock was listed on the most recent page and Adguard was nowhere to be found on the safari extension site at all. Now Adguard is on the most recent page and Adblock is under the security category.

 

[KALLT]

If I'm not mistaken don't all the security Extensions have to be adapted to use the new content blocker APIs to make it on their new extension page now? If so Adblock, Adblock Plus and Adguard are included.

When I checked earlier today Adblock was listed on the most recent page and Adguard was nowhere to be found on the safari extension site at all. Now Adguard is on the most recent page and Adblock is under the security category.

No, the previous mechanism is deprecated but it still works. They will have to update eventually, but I don’t think that Apple will be as strict as with the App Store. As long as extensions don’t do anything weird, they will be distributed. Without the gallery, big extensions like Adblock Plus won’t be updated automatically, so the gallery is the only option to make the transition seamless for users.

 

[Wheelie4]

No, the previous mechanism is deprecated but it still works. They will have to update eventually, but I don’t think that Apple will be as strict as with the App Store. As long as extensions don’t do anything weird, they will be distributed. Without the gallery, big extensions like Adblock Plus won’t be updated automatically, so the gallery is the only option to make the transition seamless for users.

Ahhh. Gotcha. I had Adguard extension already installed when I updated to El Capitan and it still seems to be doing its job so I guess I can wait patiently until they get their new extensions ready.

 

[Garussell]

does anyone know an adblocker that works on safari that blocks espn ads and youtube adds?