illustration of the a strutless dyadic partitioning of the unit
squareGreg Maxwell / Apple Neuralhash

First neuralhash second-preimage

Image of Kat Walsh from the 1980s, altered to have a neuralhash matching the dog image. Image of a Dog
$ ./ model.onnx neuralhash_128x96_seed1.dat 129860794-e7eb0132-d929-4c9d-b92e-4e4faba9e849.png
$ ./ model.onnx neuralhash_128x96_seed1.dat 130296784-bdf329d0-06f6-454e-ba87-d83b010614a7.png
Standard test image Lena adjusted to match standard test image barbara. Standard test image barbara.
$ ./ ./model.onnx ./neuralhash_128x96_seed1.dat 130310372-d6b2c633-c5d3-4b3e-bc12-d3b2f3104ec6.png
$ ./ ./model.onnx ./neuralhash_128x96_seed1.dat 130310383-9fcc80ba-117e-4383-9177-0df24bc99f9a.png
In August 2021 Apple computer announced that they intended to use their power to control the software on customer owned devices that they manufactured to deploy software changes which would make user's devices test their users files against a database and then report matches to governments without user's knowledge or consent with a stated purpose of scanning for images of child abuse. These databases were to be sourced from largely unaccountable quasi-government organizations.

Apple promoted the system as having powerful technical and procedural privacy protections but these protections weren't protections for the users they were protections from the users: Their proposed system would have used powerful cryptography— yes, but it's purpose was to conceal what images were being matched— even though the scanning was being performed on the user's own computer and in doing so shield Apple and their data sources from accountability over the kinds of content they are matching. Apple promoted their review process as protective but it was structured to bypass American users fourth amendment protection against government search and seizure by having an apple employee technically performing the search, protecting governments from user's civil rights.

An automated search of your person (cellphone) and effects (desktops) without any suspicion based on an secret list maintained by a governmental and unaccountable quasi-governmental entities. What could possibly go wrong? I don't use apple products, but I share a world with people who do.

I got involved in the public discussion arguing against the technology in principle. I found that some people argued that while some of the risks I suggested were theoretically possible, they weren't practically possible. For example: people could have their privacy invaded due to lawful images that were rigged to match against images in the database, or due to images in the database surreptitiously rigged to match against lawful images connected to targeted races, religions, and political views. Many people were basically willing to regard the technology as magic and on that basis assume it did what Apple said and only what they said. But almost every story about 'magic' teaches the opposite lesson, one that's apt for new and poorly understood technology.

By this point people had already extracted the scanning software from Apple computers, and had used it to construct matches between real images and obviously fake noise images. When these came up in discussion they were dismissed on the basis that people wouldn't be easily tricked into downloading noise images or that they wouldn't be entered into the secret databases. I disagree, but it also misses the point: the whole idea of the system was wrongheaded and vulnerable and attacks only get better. So I analyzed their scheme and discovered a way to turn any image into a false positive matched for almost any other image, without even seeing the other image only its fingerprint used for matching. — a very flexible second preimage attack, which is the most severe kind of attack on a hash function.

Unfortunately, I found that posting these examples only sent people down a path of trying to suggest band-aids to fix the specific examples. Without examples my principled concerns were too theoretical to some, once there were examples others became too tunnel visioned about just those specific examples. As a result I stopped posting examples, not wanting to contribute to improvements that only cover up the flawed concept.

In my view, your personal computing device is a trusted agent. You cannot use the internet without it, and even outside of lockdown (as this was in 2021) most people can't realistically live their lives without use of the internet. You share with it your most private information, more so even than you do with your other trusted agents like your doctor or lawyers (whom you likely communicate with using the device). Its operation is opaque to you: you're just forced to trust it. As such I believe your device ethically owes you a duty to act in your best interest, to the greatest extent allowed by the law. — not unlike your lawyers obligation to act in your interest.

A lot of ink is being spilled these days around "AI ethics" but it seems to me that the largest technology companies today struggle to behave ethically with respect to simple computing technology we've had since the 80s. If we can't even start from a first principle that a computer's first objective is to serve their user what can any kind of computing ethics be except an effort to, ultimately, exert control against other people under the guise of merely controlling machines.

The existing mass scanning of users files onto commercial services— where it has fewer ethical complications, since many users know they lose privacy when they entrust their files to third parties—, in spite of supposedly getting millions of hits per year has resulted in a comparatively negligible number of convictions much less any evidence of actually protecting children from harm.

All too often we deploy harmful non-solutions in the sake of urgency. "Something must be done, this is something, therefore we must do it." —politician's fallacy.

Fortunately, many technology luminaries adopted nuanced positions on the risks and limitations of Apple's scheme and published position papers arguing against it. Apple was convinced to abandon the program. But, unfortunately, not before giving ideas to governments that don't consider civil liberties particularly fundamental and inspiring a number of bad legislative proposals which will periodically repeat for decades to come.

My technical efforts had the amusing result of giving my partner the ability to say that she appears (literally) in a paper by a variety of respected figures. (page 29 of the above paper).

How my second-preimage attack was constructed:

Apple used a hash, a kind of digital fingerprint, to match images. But to keep the hash consistent even if images were slightly altered they built it from a neural network trained to do just that. But neural network and hash are words that obviously don't belong together if you understand anything about hashes and neural networks.

Neural networks are designed to be differentiable: They're a mathematical function structured so you can efficiently compute how changing the inputs will change the outputs, and one where changes usually have a more or less smooth effect. This is necessary for trainability. Cryptographic hashes, on the other hand, are designed to be maximally non-linear, maximally non-differentiable-- any change should have a complex and difficult to predict effect on the output.

So the obvious thing to do is run the hash, compute a direction to change the image that will bring the output closer to the desired output and keep repeating. That will produce a match but the result won't look like a sensible image anymore. The solution is just to introduce a second objective that continually biases the process back towards something that looks reasonable, shaping the noise through negative feedback— a standard digital signal processing technique. It was literally the first thing I tried, and I spent more time figuring out how to use the machine learning tools, since I'd never used them before, then I did working on the actual problem. (though my latter images looked somewhat better than my earlier ones as I twiddled parameters)

As I write this in 2023, I think this sort of solution would be found even faster now that diffusion generative image models are so popular, as they work through a similar (but much more powerful) iterative denoising process and could probably by used to produce attack images that look even more natural than the ones I produced.

Additional posts by me on the subject: