Why? Don’t do this. Society is built on an implicit assumption of trust. You will erode the foundation that enables any success you might have in the short term.
IMHO you cannot base your trust on something that is so easy to work around.
Trust should be between people, not a person and some data.
So showing how easy to generate "false" data, this makes it more obvious for people focus on other people. Trusting people makes life much easier in my experience, while focusing on data, again in my experience, is a game of cat and mouse.
> Trusting people makes life much easier in my experience
Sure, but how do you apply that to a society at large where powerful people are interested in making everybody distrust all reliable sources of information?
AI watermarks are no panacea, but at least they are a clear signal of what not to trust.
You can’t, it’s an inherent contradiction. Human social structures have sophisticated and robust evolved mechanisms for establishing and maintaining trust. These dynamics are not one option among many, they are the optimum. By their definition they don’t scale to strangers around the planet. This is an immutable factor in why we have spam, bank fraud, etc. We want the benefits of trust without the cost of local constraints but wishing doesn’t make it so.
> Sure, but how do you apply that to a society at large where powerful people are interested in making everybody distrust all reliable sources of information?
Isn't that the scenario the watermarks are useless against? Adversarial governments or anyone with enough money will be the ones who can generate images without watermarks even if you force them on the proles.
> AI watermarks are no panacea, but at least they are a clear signal of what not to trust.
Which seems like it only makes the actual problem worse? If most of them have watermarks, that only encourages people to put more trust in the ones that don't, even though those are the ones "powerful people" can still forge to manipulate everyone. What good is something that increases the credibility of adversarial government forgeries?
Again, just because something is possible doesn’t mean we should make it broadly available. Sure the CIA can manufacture poison to assassinate someone, that doesn’t mean we should make it broadly available.
That being said, I’ll make a counter argument. To the extent that deep fakes need to be so wide spread that everyone becomes skeptical and there is no other way to trigger broad diligence then maybe there is a case to be made. But I am concerned that this will be worse than a more broadly trusted image environment
> Sure the CIA can manufacture poison to assassinate someone, that doesn’t mean we should make it broadly available.
Poisons are widely available to everyone. People commonly have liters of them under their kitchen sink or in their garage. Many of them are also simple compounds that anyone can make, e.g. "cyanide" is just CN (carbon and nitrogen) typically attached to H, Na or K. Every one of those elements is required for human life, but combining them in specific ratios under well-known conditions creates a deadly poison. It's so simple to do that hobbyist chemists have to be careful not to create it by accident.
> But I am concerned that this will be worse than a more broadly trusted image environment
There are two different metrics of trust here.
The first is, should you trust an unauthenticated image you see on the internet? Now more than ever before, the answer to that question is no. Watermarks can't fix that because the people you least want to be trusting are the ones who can still forge images without any watermarks. You can't undo this by putting watermarks on other images.
The second is, do people trust random images from strangers on the internet? The ostensible purpose of doing watermarks is to get more people to do that. But when the answer to the first question is no, getting more people to do that is bad, because you're encouraging people to trust something that is not trustworthy.
What you want is for undetectable forgeries to be difficult/impossible in general, e.g. to have something that can tell anyone if an image is machine-generated regardless of whether the machine added a watermark. But watermarks don't give you that since they don't all add them or can feasibly be removed, and a generic algorithm that can detect even novel/unknown image generation methods may not even be possible.
You are over estimating your specialized knowledge. This statement requires more knowledge than I think you realize.
“ Every one of those elements is required for human life, but combining them in specific ratios under well-known conditions creates a deadly poison. It's so simple to do that hobbyist chemists have to be careful not to create it by accident.”
I agree: don’t trust an unauthenticated image on the internet”. But surely a watermark is a helpful feature in confirming its AI so my doubts are removed no?
AI watermarks empower elite / those with resources and disempowers the common person.
Only people with resources will be allowed to make content that is AI generated passed off as real.
Pandora's box is open. Instead of making a multi-tiered privileged society, we need to fundamentally restructure society to adapt.
Before that restructuring occurs it is critical to keep the playing field level. These are not tools that should be controlled by a minority authority, they are far too dangerous.
That’s a reasonable argument. But why should we tolerate the two tier system? Why not enforce a restriction on all images? Why do we have to accept someone is above the law?
Because it is technologically impossible to enforce this on the owner class. The people who control compute control the rules. Without proper audit trails it becomes impossible to prove or enforce.
The only ones, then, who could afford to break the law are also the ones who make the law and own the law. Everyone else is subordinate in that model.
That's why technology like watermarking doesn't work and will never work. Implicitly it creates a minimum of a two-tiered society and no amount of "law" will change the technological reality of this.
The people with compute only control the rules society allows them to control. It isn’t law of physics. We have export restrictions on hardware. The government could put restrictions on image generation just like the Movie, TV and music rating system or the water marks in printers that prevent counterfeit money printing.
All the things you mentioned were massive creative endeavors demanding the talents of countless humans with real-world stakes, moral compasses, and sovereign autonomy. These creatives had rights, they had families, they had parents, and they had a place in society.
Now the things they produce can be done in a fraction of the time without any of the stakes, nor any of the moral society-integrating foundational encoding.
Instead, a single billionaire with an army of GPUs can ask for a thing, and that thing will be produced near instantaneously.
The old way of regulating depended on PEOPLE creating things at PEOPLE speeds in order for the laws to work. That simply is not the case any more.
The Kentucky primary had $1.7 million spent on deepfake political ads which were seen 49 million times. Don't know how much it effected the result, but it's not a good sign of where things are headed.
Except how do you have trust between people when deep fakes and voice clones are trivial to spin up? Are we going to have to go back to doing business physically in person?
If I got a picture of my mom being held hostage it sure would be nice to have a watermark confirming in was an AI generated image. So again, why make this broadly an easily removable?
Information is default low-trust unless you have reason to extend trust to the source and that's been the case for thousands of years, if not the entirety of human existence.
We now have the tools to increase trust in specific information, for example: by signing images that need high trust for things like news reporting using camera hardware root of trust with time and geo stamping. If signatures are removed, that's back to a default low-trust state.
I don’t want kidnapping scammers to have easy to fake image generators that are untraceable. I don’t want eBay scammers to have that (sometimes I will ask the seller to take a picture with specific information to confirm a listing is credible). I could site lots more examples
It is best not to extend blanket trust to a specific source at all.
That is how we ended up with the situation where "reputed" media organizations peddle daily lies or selective truths that are useful to their benefactors.
How is a hidden watermark that only police and providers can read better than a hidden watermark that the people can also read if they go out of their way to setup some project like this? Also your argument makes zero sense because bad guys will do this anyway. At least there's also an open source project for normal people, and it's interesting from a stenography standpoint.
Watermarks exists to create a chain of custody or attribution that can be used to establish culpability. Just like the government requires printer companies to add a dot matrix watermark to printed pages to prevent currency forgery.
Making untraceable assets means parties can’t be held liable for harms.
Just because a bad actor might poison a toddler doesn’t mean you should sell them the arsenic.
Just because the effects of arsenic on the neurons of a brain are interesting means you should feed them to a toddler and watch the effects unfold.
I support the legal freedom to pursue any idea, but we should also mentor our colleagues about the consequences of our projects and avoid unnecessary harm.
Is an ikea knife infinitely replicable, distributable and capable of undermining the entire social fabric of a community? No. so these are not the same category of threat potential.
Further, an ikea knife used in a crime can be traced back to the criminal through other evidence like finger prints, surveillance cameras, purchase receipts etc.
Arguments that have a emotional component, especially related to our evolutionary instinct to take care of kids has a way of avoiding the rationalizations people adopt to justify otherwise indefensible behavior.
You also dont do yourself any favors in a bad argument that lumps me in with some unknown group of people rather tan responding to my argument.
The distortion is shockingly visible in images. Especially with any amount of generational iteration https://streamable.com/9x3s4r
That said, this tool is incredibly lossy, garbling text, completely changing shapes. It also fails to remove the new gemini spark / placement that moved since yesterday.
>Why? Don’t do this. Society is built on an implicit assumption of trust. You will erode the foundation that enables any success you might have in the short term.
would you say the same thing about a ROT13 breaker, or would you recognize how laughably naive that sounds?
I have no idea what a Rot13 breaker is and dont care that you think advocating for an ethical posture in what the tech industry builds is naive. Its naive to think bullies wont be in a kids playground. Its also naive to think we just need to accept that rather than creating norms around the type of play that is acceptable so that the playground functions.
AI generated images have never existed before, they will break our ability to use the digital tools we have built society on if we let them. Ensuring they can be identified and have attribution tracing data embedded is a reasonable step to prevent abuse.
I'd like to frame this differently: watermarking is a (weaker) form of DRM, and DRM has never worked in favor of the users.
I know of at least one music GenAI service whose ToS forbid me from using their productions in ways that are incompatible with my local rights. If Google decides, as they did with YouTube, that they'll enforce this company's watermark even though I have a right to use their results then I've passively accepted that a (foreign) company decides which of my legal rights I can exercise.
For a more pessimistic outlook: for the GenAI companies that work for the military, are those pictures also watermarked? Because if watermarks only apply to one group then the "implicit trust" argument doesn't work.
I'm in favor of watermarking the same way I am in favor of paying artists for their work. But just as with youtube-dl and DVD decryption libraries, tools like these are necessary to level the playing field.
I don’t think the military should be allowed to generate fake images. I do think there should be international standards on AI because the social harm is universal and dangerous. Just like we don’t allow chemical weapons etc.
Would you prefer if someone did the same thing and kept it to themselves (or sold it to the highest bidder)? I think knowing it exists is better than not knowing it exists.
No. I would prefer we, as a community, don’t build this. I am under no illusion that I will persuade all people but I feel clarity that it is morally and practically unwise to build a tool like this at all and especially unwise to publish it as a freely available open source project
Nothing morally wrong about finding an exploit in a system, it's what allows you to make it more secure in the future. Perhaps the most ethical course of action would have been to disclose this to Google/OAI first (which I don't know whether or not has happened), but I find that optional in this case since this isn't really a vulnerability in the conventional sense.
Finding an exploit with the intent to patch it is different that finding the exploit and using it for personal gain which, in turn, is different than finding the exploit and publishing it for open source ecosystem use.
It’s kind like if I took a picture off of your facebook with your keychain, and used it to make a copy of your house key. You’d probably prefer I reached out and told you to take down the picture instead of creating a template of the key for anyone to download and make a copy along with your home address.
The electricity used to send the http request, run the lisp code on the server, save it to whatever database Paul Graham used to build HN and render it on my display could have been put to better use that your comment. Or mine. :)
