> As we’ve explained in Deeplinks blog posts, Apple’s planned phone-scanning system opens the door to broader abuses. It decreases privacy for all iCloud photo users, and the parental notification system is a shift away from strong end-to-end encryption. It will tempt liberal democratic regimes to increase surveillance, and likely bring even great pressures from regimes that already have online censorship ensconced in law.
What’s clear is the potential for future abuse mandated by various gov (though that could easily be the same even without all these CSAM measures.. not a new threat). However, the other things are erroneously lumped in with it. It only weakens privacy for iCloud photos if you have CP or photos in a CP database, and it doesn’t prevent E2E with texting… it just gives an additional parental control in addition to the many numerous parental controls (with them kids have no privacy already), and is totally unrelated to the CSAM effort.
I admire the EFF in many ways, but I wish they’d be more exact here given their access to expert knowledge.
> (though that could easily be the same even without all these CSAM measures.. not a new threat)
Apple has historically avoided being pressured by governments to allow this kind of surveillance by arguing they can't be forced to create functionality that doesn't exist, or hand over information they don't have. It's the argument they made in the San Bernadino case.
If they release this, it'll be much harder to avoid giving governments that existing ability for other, non-CSAM uses.
Ehh. I'd argue Apple has a mixed record. It's well known that iCloud backups are unencrypted and often handed over to authorities. In Jan 2020[1], it was reported they planned to encrypt backups, but dropped the rollout due to pressure from the FBI. I'm surprised they don't get called out because the difference between this and San Bernadino makes sense to me from a technical standpoint, from a practical standpoint and to the laymen it seems hypocritical.
In this case, I actually kind of buy Apple's argument that this will make it harder to cowtow to governments (assuming they encrypt iCloud photos at some point). Right now they can scan iCloud data and hand over photos and accounts without user's knowledge. They can do that without informing users (like they currently do with backups). With this in place the database and logic ships with the OS. They would have to implement and ship any changes world-wide. Users will have to install that update. An alternative is Apple silently making a server-side change affecting specific customers or countries. With that said, I do understand people's concern over the change.
iCloud backups _are_ encrypted, but have an HSM-escrowed key process on court order.
People like to pretend crypto is always E2E or nothing, but escrowed keys do mean that the process of checking cloud-backed data has an auditable release process, that the keys to your data are outside the cloud-hosted infrastructure, and that there is no support for blanket data scanning.
> assuming they encrypt iCloud photos at some point
The request from the FBI in the San Bernardino case was to change a passcode limit constant and retry timeouts. Those are about as trivial to implement as any of the convoluted government coercion database attacks against CSAM detection being proposed here.
The difference here is, that apple would have to develop a new feature for them, test it, and waste millions in lawers to protect themselves from accusations of tampering with the evidence (which a software update definitely is, and who knows what FBI wanted in that software update, maybe even to insert a fake sms to the sms database, or many other things a good defense lawyer could bring to the jury).
Here, it's different.. let's say there's a new wikileaks, photos of secret documents... and again, first a few journalists get the data, start slowly writing articles, and FBI just adds the hashes to the database, and they can find out who has the photos, with metadata even who had the first, before the journalist, and they can find a leak.
Has anyone seen something like this in the other direction? Something that walks through “the FBI would do this, then this, etc. and now they’ve coopted Apple’s system”?
But the FBI can’t just add the hashes to the db. That’s why it’s the intersection of two dbs in two jurisdictions… to prevent exactly that kind of attack. Then they need to pass a human reviewer as well.
You are suggesting that another country could launder the request on behalf of the USA in order to circumvent the 4th Amendment. Okay then, let's play that out.
Australia contacts Apple and demands they augment CSAM detection so that every iPhone in the USA is now scanning for image hashes supplied by Australia.
“ The Australian Telecommunications and Other Legislation Amendment (Assistance and Access) Act 2018 (TOLA) allows government agencies to issue “technical assistance” and “technical capability” notices to providers of communications services. The notices require that the providers give the authorities help in conducting criminal enforcement intercepts, and that they make changes in their systems to ensure that they can give that help.”
…
“In any event, the FBI chose a curiously roundabout way of getting access to ANOM messages. For ANOM devices operating outside of the United States, “an encrypted [blind carbon copy or BCC]” of each message that a user sent was transmitted to a server located outside of the United States, which then decrypted and reencrypted the message with an encryption key known to the FBI. Those reencrypted messages were sent to another server that was owned by the FBI, outside of the United States.
In the summer of 2019, the FBI started negotiating to build the legal structure that would make this technical architecture work. In essence, the FBI went looking for a third country that would host the BCC server and could lawfully accept all of the decrypted messages and send the copies to the FBI. As the affidavit notes, “Unlike the Australian beta test, the third country would not review the content in the first instance.” This would have been a fascinating negotiation. Both participants wanted to make criminal cases and avoid privacy scandals. The U.S. would want to be sure that the third country had full legal authority to intercept the contents of every ANOM message, and that the country was also willing to share the full ANOM take with the U.S. in something like real time.”
None of that explains how Australia could ever successfully coerce Apple into performing widespread surveillance of US citizens. The fact that Australia is a "five eyes" country doesn't make it any more plausible than if the demand came from China or Russia.
It explains how Australia could coerce Apple into performing widespread surveillance of Aus citizens. Then it’s trivial for the US to coerce Apple into switching that functionality on in the US.
Any widespread warrantless surveillance of the private physical property of US citizens, performed at the direction of the US Government, would be an absolute clear-cut unambiguous breach of the 4th Amendment.
I'm not saying the US Government wouldn't care that it's unconstitutional—we know they'd ignore the constitution when they can get away with it. But they'd also have to convince Apple's lawyers to go along with unconstitutional surveillance. You don't think Apple wouldn't be itching for another opportunity to prove their strength against a Government? Especially now? Apple would love nothing more than to have more opportunities like they got with the San Bernardino iPhone.
You haven't heard of how GCHQ would happily hand over intelligence they had on U.S. Citizens?
I know for a fact there are efforts at creating fusion centers across national boundaries. The question you need to ask is not if but what will make you interesting enough to mobilize against.
Thou shalt not build the effing Panopticon, nor it's predecessors. Is that so hard to not do.
I've no doubt that GCHQ would hand whatever they like to whomever they like. But why would Apple comply with a demand from the GCHQ to spy on US citizens?
They will come up with some kind of fuckery to get around the rules, like some crypto ridiculousness that makes files that match a certain hash go to servers in a different jurisdiction? Far-fetched, I know...
The passcode limit constant is enforced by the secure enclave. I don't know if it's been proven that the secure enclave component of the device can be changed without the device being unlocked. I'm not even sure it possible for any operating system updates to occur on a device which is locked.
Not on the iPhone 5C which was the phone used by the terrorist and did not have an SE. Locked iPhones can be updated from DFU, but I think SE firmware can’t.
The technical feasibility of the FBI’s request was never the question, nor the basis of Apple’s objection.
Of course it’s even easier for Apple to say “no” to the government if they literally cannot do what the government is asking.
That’s the basis of the EFF’s objection to Apple’s plans: they think that by implementing this CSAM system, Apple will turn an impossibility into a possibility.
We were never in a world where Apple was unable to do what the government was asking. Nothing impossible has been made possible, and Apple has made a stand against small changes - like changing constants - before.
It's not "impossibility". It's cost. The FBI cannot force Apple to do free work to circumvent security. Can they force them to add one more hash? Is that work?
I’m not a lawyer, but at the very least Apple would have to train its reviewers to recognize new image types, redact or update all statements where they explicitly drew a line in the sand about what the feature searched for, likely produce a custom build of iOS, update its knowledge base to distinguish between the root hash of the CSAM database and FBI Imagery database, perhaps re-present consent, add code to target this new database (or subset of the full database) to US users, etc. I’m not a product manager either, but superficially simple changes are often quite complex!
I think once Apple has established a pipeline for adding content in a known pattern and at a known cost, it lowers the legal bar (although I am not a lawyer). The government can take your stuff, they just have to pay you for it. I'm not sure if adding the N+1 image to the next dataset is similar to that or not.
No. Because the hash causes searches to be performed on citizens' private property, this would be an unambiguous, indisputable, clear-cut violation of the 4th Amendment.
Apple simply tells the DOJ if any non-CSAM content is added surreptitiously to the DB, Apple will drop CSAM scanning. Also, if DOJ makes any request to scan for non CSAM by court order or warrant, Apple will in kind drop support for the technology.
Apple is making the good faith effort here. If DOJ makes a bad faith effort, Apple is in now way required to continue to participate.
Apple will see the results of the matched images however. Given that Apple believes their algorithm will generate an impossibly small quantity of false positives, if they start to see political images, sensitive documents etc. show up, that tips them off the DB is corrupt.
All of these processes will be subject to discovery in the very first trial generated by this technology. If there is evidence that governments are polluting the DB with non-CSAM content, then warrants issued on evidence from the DB can be overturned. Prosecutors in DOJ and FBI etc. have a strong incentive to ensure that DB doesn't turn into a free-for-all dragnet because it could work to invalidate their ability to use evidence from legitimate CSAM cases.
You’d have to not only have over 30 hash collisions, but also have it collide with another secret hash function, and then also have a human look at it and agree it’s CP.
So what’s the actual realistic issue here? This keeps getting thrown around as if it’s likely, yet not only are there numerous steps against this in the Apple chain, this would already be a huge issue with Dropbox, Facebook, Microsoft, Google, etc who do CP scanning according to all of the comments on HN.
The supposed other 'secret' hash function cannot be secret from the state actors generating the databases.
Also, if it has a similar structure/training, it's not that unlikely that the same images would collide by chance.
> also have a human look at it and agree it’s CP
That's straight-forward: simply use nude or pornographic images which looks like they could be children or ones where without context you can't tell. It's a felony for them to fail to report child porn if they see it in review, and the NCMEC guidance tells people when in doubt to report.
Besides, once someone else has looked at your pictures your privacy has been violated.
If this really was such a problem, then as I said, we’d have been getting reports of this over the past 10+ years it’s already been in place at big cloud providers. So where is all of this ruining of peoples lives by uploading CP on their devices?
Also if you’re a gov actor trying to frame someone, why bother with a pre-image when you could put the real images on it?
None of that is new today — all that’s new is Apple is joining the effort to scan for CSAM, and instead of doing it on server they’re doing it on device right before you upload in a way that attempts to be more secure and private than other efforts.
What do you mean? People are arrested all the time for having CP on their machines, I see it in the news frequently. Impossible to know how many of them could have just been framed, no one is giving the benefit of the doubt to an accused pedo. And it never goes to trial due to the possibility of enormous prison sentences. If you’re innocent would you risk 100 years in federal prison going to trial or plead guilty and only face a few years?
Many people seem to miss that the automated scanning makes framing much more effective.
Say I sneak (psedo-)child porn onto your device. How do I get authorities to search you without potentially implicating myself? An anonymous tipline call is not likely to actually trigger a search.
With automated mass scanning that problem is solved: All users will be searched.
> it’s already been in place at big cloud providers.
I think the big cloud providers scanning your private data is of dubious ethics, but it's like complaining that your mail carrier is reading the content of your postcards.
So long as you send unencrypted data to a third party your privacy will be limited, regardless of what our laws or norms say. People usually know this, and so many do avoid uploading things to these places or encrypt what they upload.
When its your device itself doing the scanning, ahead of any encryption-- then that protection goes out the window.
Sometimes the same violation of privacy is made more acceptable by a clear boundary that you can stay on one side of to protect your privacy. Your devices vs someone elses devices is the most clear historical boundary in this case, and apple is breaking it.
I don't think it's unreasonable to expect the erosion of the private boundary to have an effect. And we can't say that the scanning by providers does nothing, -- the convictions prove otherwise. We can only hope that all those convictions were deserved, the nature of this crime is such that its hard to prove someone wasn't framed.
> So where is all of this ruining of peoples lives by uploading CP on their devices?
It's already happening. Except we just choose to SWAT people instead, since it's faster, easier, and there's effectively no liability on the behalf of the caller.
> So where is all of this ruining of peoples lives by uploading CP on their devices?
Once the capability is in place on everyone's devices, how are we supposed to guarantee it will never be used maliciously? Just say no to the capability.
> Also if you’re a gov actor trying to frame someone, why bother with a pre-image when you could put the real images on it?
Because the capability for this is now built-in in everyone's phones.
> That's trivial. If the attacker can get one image onto your device they can get several.
At which point everything you brought up about attacks on the hash function is completely irrelevant because the attacker can put actual child porn from the database on your device.
The apple system is a dangerous surveillance apparatus at many levels. The fact that I pointed out one element was broken in a post doesn't mean that I don't consider others broken.
My primary concern about its ethics has always been the breach of your devices obligation to act faithfully as your agent. My secondary concern was the use of strong cryptography to protect Apple and its sources from accountability. Unfortunately, the broken hash function means that even if they weren't using crypto to conceal the database, it wouldn't create accountability.
Attacks on the hash-function are still relevant because:
1. the weak hash function allows state actors to denyably include non-child porn images in their database and even get non-cooperating states to include those hashes too.
2. The attack is lower risk for the attacker if they never need to handle unlawful images themselves. E.g. they make a bunch of porn images into matches, if they get caught with them they just point to the lawful origin of the images. While the victim won't know where they came from.
Yours is a very fair negative reaction. The information in the EFF’s includes a portion where it seems to be concerned only about alerting parents[1]. I think many parents would find that reasonable. However, the fact that the information will also be sent to the government [2] is just plainly an abuse of privacy, goes outside of the relationship between parent and child, and I do not imagine that parents would find that reasonable.
> [1] Moreover, the system Apple has developed assumes that the "parent" and "child" accounts involved actually belong to an adult who is the parent of a child, and that those individuals have a healthy relationship. This may not always be the case; an abusive adult may be the organiser of the account, and the consequences of parental notification could threaten the child’s safety and wellbeing. LGBTQ+ youths on family accounts with unsympathetic parents are particularly at risk. As a result of this change, iMessages will no longer provide confidentiality and privacy to those users through an end-to-end encrypted messaging system in which only the sender and intended recipients have access to the information sent.
> [2] When a preset threshold number of matches is met, it will disable the account and report the user and those images to authorities.
Apple announced two separate things in one press release: a CSAM-scanning system, and a parental control that uses AI to attempt to detect nude pictures in iMessages and alert the parents. The latter system does not send any info to Apple or any authorities.
Which database? I get the impression that people think there is a singular repository for thoroughly vetted and highly controlled CP evidence submission. No such thing exists.
It’s an intersection between a US db and a not yet chosen non-US db, which then will have a human reviewer verify its CP before sending off to the authorities.
An independent audit for both the secret secondary perceptual hashing algorithm and the chain of custody policies/compliance for the "US db" and the disconcertedly open ended "not yet chosen non-US db"?
What's the point of that? If you don't trust Apple, why would you use Photos.app in the first place? They already have 100% control over that, and can spy as much as they want to. No need to go by way of the CSAM database, that would be absurd.
I've never been a customer of Apple but I'll try and imagine the experience... I might trust them to assemble hardware and write software for my consumer needs - but that doesn't mean I trust them to competently reason about me potentially being a pedo. That is only a small part of a much larger point, but it is reason enough alone.
Apple's responsibility ends with notifying law enforcement, at which point presumably there would be a subpoena for evidence and a trial.
The concern people have is logic on their phone snitching on them, with scenarios based on authoritative regimes setting the baseline of what is scanned/reported.
Apple is not serving as judge, jury and executioner (unless there is an electric shock delivery system being added to the iPhone 13)
That could be the most obnoxious nitpick I've ever heard, do you honestly think anybody is worried about Apple doing anything beyond submitting a false report - and that you are being helpful by pointing out that they can only do that very thing? Do you think that might be why I said "competently reason about" instead of "send the Apple genius death squad"?
My point is that people seem to think that this feature somehow makes it easier for Apple to spy on you.
In fact it doesn’t make any difference at all, since they already have full access to everything you do on the phone, so if you don’t trust them you shouldn’t use an iphone.
Neither Google nor Microsoft scan pictures people have on their devices running Android or Windows. I'm not sure how that's even applicable to Facebook and Dropbox.
You’re asking for an auditing chain presumably due to concerns about governments putting in photos of things that aren’t CP. Apple is only doing this for photos that get uploaded to iCloud, with this neural hash that gets uploaded with it. The actual verification of a CP match occurs on the server due to the hashes being blinded. So in many ways, it’s very similar to what these other cloud providers effectively do — search for CP matches on uploaded data.
If you’re concerned about non-CP being scanned for, then you should already be concerned about that with everyone else. Thus if you’re asking for auditing of Apple, then you should widen your request. If you do, then sure, I can understand that. If you’re not concerned about the existing system, then I think you’re being non-consistent.
Most people in comments to me seem to be non-consistent, and are being overly knee jerk about this… myself included initially.
Microsoft, Google, Facebook, Dropbox, etc. all scan photos which are cloud hosted for CSAM. Apple's new system scans only photos which are cloud hosted for CSAM.
This would be the source of the inconsistency - that the code to do scanning is on the client side of the uploader rather than the server side of the uploader does not change any abuse scenarios, but exclusively serves "slippery slope" arguments of full on-device surveillance.
It absolutely does. When the scanning is server-side, companies can only scan the files you choose to send to their servers but with a client-side system in place, all it takes is a change in company policy and a few simple directory config changes to result in ANY file on your device being scanned.
Slippery slope or not, the client-side system massively lowers the bar for abuse and that cannot be ignored.
Sure, but I don't expect it from third party cloud platforms - in the same way I wouldn't expect accountability from a garbage man who reports to the police after finding evidence of crime in my garbage. Apple is, for some insane reason, trying to establish the precedent that the contents of your Apple product are now part of the public space - where expectation of privacy isn't a thing.
And all it would take is a confidential change in policy before all photos are scanned, and the same "save the children" argument would be used to save face if the decision ever leaks.
"Pedophiles are taking pornographic photos of kids on their phones and then sharing them outside of iCloud. But wait, you don't want us to compare each photo you capture against ML models capable of identifying child porn? Why are you defending and siding with pedophiles?"
The only surefire way to avoid a slippery slope is to keep away from the edge.
Lie? I don't take kindly to such words, because you're ascribing malicious intent where there is none. Please check your tone... HN comments are about assuming the best in everyone.
The hash matching is occurring on device, but only for iCloud photo images:
> Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the database of known CSAM hashes. This matching process is powered by a cryptographic technology called private set intersection, which determines whether there is a match without revealing the result. The device creates a cryptographic safety voucher that encodes the match result. It also encrypts the image’s NeuralHash and a visual derivative. This voucher is uploaded to iCloud Photos along with the image.
Read that PDF. You'll see everything in it is designed for iCloud photos only.
It is not a lie. The scanning is done on device, but photos are not scanned unless they are going to be uploaded to iCloud. Apple has explicitly stated this.
Oh, well if Apple says... I'm sure their statement somehow completely aligns with all the potentially conflicting interpretations one can draw from their PR, their stated objectives, and the implementation details observed, and it always will - forever.
Apple has released fairly detailed technical summaries of their system, far beyond what could be hidden behind "conflicting interpretations" of material written by a PR department. Have you read them? Are you claiming that Apple is lying?
If your contention is that Apple is lying now, then you have no reason to think Apple—or any other corporation for that matter—hasn't been lying about your data security for the past decade. Who knows, maybe Google Chrome is sending everyone's passwords in plain text to the NSA.
If your contention is that Apple might turn evil in the future, that charge could be levied against against any other company at any time. It's functionally unfalsifiable.
> Apple has released fairly detailed technical summaries of their system...
... that don't address implementation details - like what background processes get hooked into for hash generation and under what circumstances. They are also relying very heavily on secrecy, the secret local db being a good example. Why does that matter? Because their assurances against false positives depend on you assuming that the threshold counter only applies to content being flagged with a reasonably low rate of false positives and subsequently stored on their cloud, which permits additional safety assuring verification steps - and I don't see any reason why you should assume that.
> Have you read them?
I have.
> Are you claiming that Apple is lying?
Yes, but not about their intent (which I don't really care about, and is immaterial anyway) - they are lying about their ability to execute the program as they've described. Take for example their hybrid perceptual algorithm approach, which supposedly provides some increased measure of protection against adversarial attacks. We know for a fact that their primary algo is hopelessly vulnerable to hash length extension attacks, which makes the generation of false positives trivial. The second algo that supposedly addresses that is a secret, which should immediately raise red flags for anyone familiar with infosec. But I wouldn't be surprised if that safety turns out to be Microsoft's PhotoDNA - because it is already commonly used in the CP cataloging realm, and Apple would have more than one reason to not want to advertise something like that. First, PhotoDNA is a blackbox that has no independently conducted research available for public scrutiny. Second, it would mean they designed their system totally backwards - as PhotoDNA employs a high pass filter to guard against extension attacks, but at this point in the flagging process (as Apple has described) that filtering protection can't be employed to guard against extension attacks... so it provides no additional protection to speak of. Third, it was invented by a competitor.
> ...hasn't been lying about your data security for the past decade.
You are forgetting about all the cries for not ascribing malice to stupidity, and how this case involves a very different kind of cover for action. When calc.exe sends tiny encrypted fragments to telemetry.microsoft.com and it has no means of using a hidden channel to receive anything outside of itself - I'm irritated, but not alarmed. When PhotoAgent is chilling in the background - occasionally opening a RW handle to some persistent encrypted db, a db that also gets opened by another process prior to establishing a network connection to thinkofthechildren.apple.com, I become suspicious.
> If your contention is that Apple might turn evil in the future...
My contention is that they can't avoid making mistakes, and that assurances addressing concerns related to consequence of said mistakes depend entirely upon secrecy. History has shown how relying on secrecy and infallibility plays out, Apple's defenders are ignoring that.
> which then will have a human reviewer verify its CP
No, it won't. The human reviewer only sees very low resolution thumbnails, to check there's a "match". The content is not verified, so the two DBs could contain anything.
> It only weakens privacy for iCloud photos if you have CP or photos in a CP database
Who controls what is in the database? What independent oversight ensures that it’s only CSAM images? The public certainly can’t audit it.
What is stopping the CCP from putting pro-Uighur or Xi Winnie the Pooh images into the database? Or from the US using this to locate images that are interesting from an intelligence perspective (Say for example pictures of Iranian Uranium centrifuges)? Apple says they will only add images that more than one government request? All it would take is a few G men to show up at Apple with a secret court order to do this no?
Their use of a highly vulnerable[1] "neural" perceptual hash function makes the database unauditable: An abusive state actor could obtain child porn images and invisibly alter them to match the hashes of the ideological or ethnically related images they really want to match. If challenged, they could produce child porn images matching their database, and they could had these images to other governments to unknowingly or plausibly denyably include.
...but they don't have to do anything that elaborate because Apple is using powerful cryptography against their users to protect themselves and their data sources from any accountability for the content of the database: The hashes in the database are hidden from everyone who isn't Apple or a state agent. There is no opportunity to learn, much less challenge the content of the database.
They have to come from the intersection of two databases from two jurisdictions. So already that’s out as you suggest. Then you’d have to match _nearly exact photos_, which isn’t a vector for general photos of some random minority. Then you’d need 30 of such specific photos, a match with another secret hash, and then a human reviewer at Apple has to say yes it’s CP before anything else happens.
I think there are plenty of reasons to be concerned about future laws and future implementations, but let’s be honest about the real risks of this today as it’s currently implemented.
Every step you've described is unfalsifyable: You just have to blindly trust that Apple is doing these things, and that e.g. authoritarian regemes haven't compromised Apple staff with access to the data.
> They have to come from the intersection of two databases from two jurisdictions.
My message directly answered that. A state actor can modify an apparent childporn image to match an arbitrarily hash and hand that image to other agencies who will dutifully include it in their database.
> Then you’d have to match _nearly exact photos_
It's unclear what you mean here. It's easy to construct completely different images that share a neuralhash. Apple also has no access to the original "child porn" (in quotes because it may not be), as it would be unlawful to provide it to them.
> but let’s be honest about the real risks
Yes. Lets be honest: Apple has made a decision to reprogram devices owned by their customers to act against their users best interest. They assure us that they will be taking steps to mitigate harm but have used powerful cryptography to conceal their actions and most of their supposed protections are unfalsifable. You're just supposed to explicitly take the word of a party that is already admittedly acting against your best interest. Finally, at best their protections are only moderate. Almost every computer security vulnerability could be dismissed as requiring an impossible series of coincidences, at yet attacks exist.
> A state actor can modify an apparent childporn image to match an arbitrarily hash and hand that image to other agencies who will dutifully include it in their database.
Even if a state actor constructs an image that is an NeuralHash collision for the material they wish to find, that only gets them through one of the three barriers Apple has erected between your device and the images being reported to a third party. They also need to cause 30 image matches in order to pass threshold secret sharing, and they need to pass human review of these 30 images.
Arguably investigation by NCMEC represents a fourth barrier, but I'll ignore that because it's beyond Apple's control.
> You just have to blindly trust that Apple
This has been true of all closed source operating systems since forever. Functionally, nothing has changed. And whatever you think of the decision Apple has made, you can't argue that they tried to do it in secret.
The invocation of 30 images like it's a barrier confuses me. I created a bunch of preimages posted on github, I could easily create 30 or 3000 but at this point all I'd be doing is helping apple cover up their bad hash algorithm[1].
I pointed out above that the attacker could use legal pornography images selected to make it look like child porn. This isn't hard. Doing it 30 times is no particular challenge. I didn't use pornographic images in the examples I created out of good taste, not because it would be any harder.
If someone is trying to frame a known individual, the 30 image threshold may not be a significant barrier, I'll grant you that. But if you're enlisting Apple's algorithm to perform a dragnet search of the citizenry (e.g. leaked state secrets) then this mechanism cannot be effective unless the material in question is comprised of at least 30 photographs.
I have some residual nitpicks, on that point: many leaked data troves are much larger than that, though it is a material restriction.
The 30 threshold isn't leakless. Say you only have one hit, it still gets reported to Apple. The software also emits a small rate of "chaff", fake hits to help obscure the sub-threshold real hits. But it could still be used to produce a list of possible matches, including anyone with targeted material plus people who emitted fake matches, producing a list of potential targets much smaller than the whole population, for enhanced surveillance.
This still relies upon some degree of compliance by Apple in order to acquire the stream of vouchers. Or alternatively a working security breach of Apple's systems. Either way this represents an additional, non-trivial barrier to overcome.
It would be interesting to know what the "chaff" rate is and whether any intelligence agency could stomach that amount of surveillance, particularly since it's by no means certain that any of them are real. In fact it seems to me that it's very unlikely indeed, especially if the material is in any way radioactive. After all, finding a match this way requires quite a few assumptions:
1. The target owns an iPhone;
2. The target has enabled iCloud Photo Library;
3. The target has a photo library small enough, or is paying for sufficient iCloud storage space, that the flagged images are included in the (sub)set stored in the cloud;
4. The target has imported the flagged images into their photo library rather than to iCloud Drive or any third party app like SpiderOak, Mega or Tresorit.
> Every step you've described is unfalsifyable: You just have to blindly trust that Apple is doing these things, and that e.g. authoritarian regemes haven't compromised Apple staff with access to the data.
Better than not, but how much can you trust the word of a third party auditor in a world where government intelligence agencies conspire to spy on each other's citizens because their laws expressly prohibit on spying on their own... where the (one time) worlds largest supplier of cryptographic hardware was covertly owned by the CIA and shipped backdoored units for decades, where our national standards bodies make standard backdoored random number generators, and where the US government used a sham vaccination campaign to collect genetic samples from a whole community in order to locate and assassinate a single terrorist.
When it comes to keeping data private we face adversaries whos only rules seem to be what they can get away with. Against that, full transparency when it comes to the construction of our security infrastructure really needs to be the starting position.
>and where the US government used a sham vaccination campaign to collect genetic samples from a whole community in order to locate and assassinate a single terrorist.
National Center for Missing and Exploited Children intersected with a not yet determined db in another jurisdiction, and then human moderators at Apple.
So what you’re describing is something that is a concern for the future that could exist anyway (and maybe already does at places like Google that have _zero_ auditable processing of data that already scan for CP from the same database above). But let’s not pretend that’s today.
> It only weakens privacy for iCloud photos if you have CP or photos in a CP database
Or if someone hacks your device and uploads one of those photos to your iCloud.
(I still have no idea why people aren't pointing this out more aggressively -- phones get hacked probably every minute of every day, and acquiring those actual photos isn't that difficult once you're willing to commit felonies -- hash collisions are a distraction)
Because that then would already be a problem for Facebook, Google, Microsoft, etc that host photos that hacked phones could be uploading today.
And we’re just not seeing that being the case. Because all these providers have been doing this for so many years, including the nearly 17 million photos identified by Facebook last year, you’d figure there would be a lot more noise if this was really going on.
In fact, I would venture to say it’s far easier to hack a Facebook account than it is iCloud that has many on-device-based security protections that a cloud login without mandatory 2FA (and often SMS when it is used).
We had SWAT teams for a long time before SWATing became popular. The publicity that this has gotten is only going to increase the chances that all these services start getting abused. And who is to say that it hasn't happened already and been entirely successful, but nobody believed the victim.
Suspected CSAM is always reviewed—the actual files, not a hash or reduced-resolution version—by members of law enforcement before an arrest warrant is issued.
Police and prosecutors have to do that because they have to attest to the judge that it is actually CSAM. And, unlike any private party, law enforcement is legally authorized to possess and review CSAM, so they don’t run any risk (aside from the risk of seeing horrifying images).
Unlike SWATing, the police don’t have to go to a person’s house to review suspected CSAM that is submitted by a service provider like Google or Apple. So it’s possible that there are existing collisions for PhotoDNA that make innocent files trigger an alert. But no one would know externally because once law enforcement reviews the file and sees it is a false positive, they just ignore it. The account owner would never know. The service provider might do forensics if they interpret the incident as an attempted attack.
I'm honestly surprised that's not a much more common way of griefing (either specific or random targets) or disabling accounts to deny a victim access to their e-mail after an attacker has gotten access, used it to reset passwords, and is now abusing the linked accounts (with the added benefit that the victim may be too busy being arrested to deal with the fraud).
Probably because the group that has/is willing to handle CSAM is small enough that it doesn't overlap much the other groups (e.g. account hijackers, or people who want to grief a specific person and have the necessary technical skills and the patience to actually pull it off). For criminals it may not be worth the extra heat it would bring, but from what I've heard about 4chan, I'm surprised there is not a bigger overlap among "for the lulz" griefers.
Yes. I would bet the large majority of people here who are now quick to point out that other cloud providers have been doing this for years didn't know that fact a month ago. We're now well armed with that information due to arguing about this Apple issue. The fact that you're so quick to inform me of the facts is precisely why I think its more likely to happen.
What’s clear is the potential for future abuse mandated by various gov (though that could easily be the same even without all these CSAM measures.. not a new threat). However, the other things are erroneously lumped in with it. It only weakens privacy for iCloud photos if you have CP or photos in a CP database, and it doesn’t prevent E2E with texting… it just gives an additional parental control in addition to the many numerous parental controls (with them kids have no privacy already), and is totally unrelated to the CSAM effort.
I admire the EFF in many ways, but I wish they’d be more exact here given their access to expert knowledge.