A "tally" of positive matches are kept, and only once a certain threshold of positive matches are found will the image be uploaded to the iCloud server with a voucher attached to it that basically notifies Apple of the offending image, which is then reviewed by a human.
I don't really see this as an invasion of privacy if Apple literally can't know about the image unless it is something noteworthy, and only after a specific threshold is crossed.
The way some people are describing it is that it's like some tool that gives Apple complete freedom to access their device and snoop all the photos/data, which it's not.
That's not entirely accurate. The device you own (MacBook, iPad, iPhone) performs the image hashing and checks it against the database, uploading the match result with the image. The server keeps track of how many positive vouchers there are (sometimes throwing false ones in so the actual number is off), and if a threshold of positive matches is reached, the server then tries decrypting the matched vouchers. Images with positive matches are uploaded after the voucher is assembled, not after a threshold of positive vouchers.
The problems myself and many others have with this approach are:
1. Apple has to wait until accounts get flagged and go to human review to ensure the database sources aren't abusing Apple's system. Apple backed themselves into a corner with this system, and I don't trust that the database won't be seeded by the providers. My only problem with Apple is that they decided to build and (try to) implement this hashing system.
2. The database and hashing program lives ON OUR DEVICES. Among power and performance concerns, and concerns about the database living on our devices, this is a dangerous precedent Apple just set that you know the rest of the tech industry will follow without the same level of care. The fact that NCMEC was so ecstatic about this is alarming. Our entire iCloud Photo library can be decrypted by Apple AT ANY TIME they wish, with or without CSAM detection. It is not E2E encrypted, and no intention of making it so was announced. This system provides absolutely no increase in privacy, and could simply be done entirely on-server for SHARED photos only (technically, "hosting" means sharing with other people).
3. Other companies, such as Dropbox, already perform this scan on-server. Their human review team gets to see the full image and not a "visual derivative", which could actually REDUCE the number of false reports. As someone who had bathtime photos accidentally stored on Dropbox for months as part of a drive backup years ago, nothing came of it.
4. With part of the scanning system already on-device, it's not hard to see a future update moving the ENTIRE system on-device, nullifying the opt-out of "Don't use iCloud". The opt-out at that point would be, "Don't use iPhone", followed by "Don't use smartphones" within 2 years. Slippery slope argument? How about stepping off the edge of the Grand Canyon rim?
TL
R:
1. Apple has little way to verify database integrity.
2. Crossing the on-device line will forever tip the balance of power away from users.
3. Server-only scanning of non-E2EE content works for the rest of the industry and would be fine for Apple, too.
4. iCloud's role could only be temporary as detection and reporting move completely on-device. Rest of the industry follows suit.
My stance:
I would be okay with server-only scanning of photos shared with other people. I could even be persuaded into entire library scanning only done on-server (again, I use other cloud storage that does this). Absolutely, DO NOT put any part of an illegal-content scanning system ON MY DEVICE. I would much rather continue to use and enjoy my Apple products (the iPhone makes many things easier when other people are involved), but I am prepared to leave behind my Apple and other "smart" devices if need be. I am careful to only purchase items that, at most, have an app that is optional for extended functionality but not required for basic use. I personally have paused my exit at this point, but have some replacements already in-place.