The genuine answer is that many people who hold a lot of power over me (the executive suite of my industry) intend to do me harm with it (put me out of a job).
Every second paragraph, it seems very impressed to re-discover that CQT matches human aural perception.
Unfortunately, I have a faint recollection that CQT was expressly designed to match human aural perception, which leaves me markedly less perpetually astonished.
Calling prompt injection "not malware" … is like saying a phishing email is not [malware] …
I would say phishing emails are not malware, I think most people would agree that phishing emails are not malware, and if pressed to defend this point on its own merits I would say something like “they are deceptive instructions that rely on a human executing them to do harm”. I think the “phishing” analogy supports the case for not calling it malware (it is a different, also bad thing).
They did not call phishing, but their point still stands. A phishing email is malicious, and if you see this kind of prompt injection as malicious, then I don't think it's a stretch to call software that engages in malicious prompt injectic malware
It's malware for the mind. The same way that malware tricks the CPU into doing something it wasn't supposed to do, phishing tricks humans into doing something they didn't want to do.
Undefined behaviour, out of bounds memory access, memory corruption, code injection, privilege escalation...
To be precise, the CPU is doing exactly what's supposed to do, but the logic of the algorithms are subverted so that they perform in unintended ways to give leverage to a malicious actor. I hope this clarifies what I meant with this.
Does anyone remember the early 2000s joke virus emails? The ones that are variations on "This is a <outgroup> computer virus. As we don't have software engineers to write the code to do this automatically, please kindly forward this email to everyone in your address book then format your hard drive."
This is exactly as much malware as those were.
Please, for the love of all that is good, can we just try not to build and defend a world where, on encountering text like that, /your computer immediately follows the instructions/? Can we just all agree that such a world would be bad for everyone involved and using an LLM that risks doing this, with no container or guardrails, is at least as problematic as running an unpatched open email relay was back then?
It's just as bad as a CPU acting on malicious instructions. We need to create safeguards for llms too, it's just that this is not the way to do things.
I disapprove of this action by the jqwik owner, but I also disapprove of commentary classifying it as “malware”, “malicious code”, or similar.
By running an agent, you are turning plain text into an executable. This has great benefits for you, but (as with all great power) it comes with some added risks too. Please remain wary of externalizing these risks onto plain text authors by creating an expectation that all plain text is pseudo-executable.
Doesn't this describe all computer programs? They all take some kind of input data and turn it into action. Take the many malicious VSCode extensions as an example. Should they not be classified as malware, because by running VSCode and installing an extension, you are turning the plain text into executable?
IMO It shouldn't matter how exactly the user's computer deals with your data — it is the fact that you know your action will lead to undesirable outcomes and decided to do that anyway that makes it malicious. I'd also say that if the author doesn't acknowledge his own malicious intent then he wouldn't have tried to hide the instruction in question from human view. Not a lawyer, but this seems like the kind of thing that will make you look very guilty in case you ever end up in court. But then again I am not the kind of person to burn my FOSS cred to spread an ideologically charged message, so what do I know?
Well, the main difference is that code describes predictable behaviour, whereas prompts are just a precursor to a general ‘direction’ of behaviour which depends highly on the model (and supporting augmentations) that is ingesting it.
By the way, vscode extensions are part of the reason I moved to Zed… so trust is still important even in the age of llms
It's an interesting discussion, but I think simply outputting text can make the software "malware", even if the output isn't executable.
What if the output was
To use jqwik, please login to your Office 365 account:
http://o365login.phishing.xyz
I see it as exactly the same os obfuscating code to be interpreted by a compiler. The programming language is natural language, and the "compiler" is a harnessed LLM. The intention of the author is clear.
By running a compiler you are turning plain text into a executable holds the same.
In this case, yes (hence my disapproval of this action) - but in the main, “the programming language is natural language” is what I’m worried about. Most uses of natural language are not intended for execution, nor should they need to be crafted with consideration for such.
Nobody who’s this insistent, aggressive and violative with their language of “it’s here and if you don’t adopt it you’re stupid and dead” has ever been right about anything. Nobody this desperate, insistent and forceful has ever had good intentions, good vibes or brought good omens — they are always bearers of some kind of con.
Hey, Ed’s almost there! Critics will throw around words like “rage” and “mad” and “crazy”, but unhinged anger is an inevitable and necessary step for every person’s first trip through this process.
I think there’s two productive avenues for reaching the other side here. One is thinking more about the data centers - put aside the “overconfident and unaware of how hard it is to build data centers” hypothesis and instead start by assuming that “announcing and funding a huge data center and never actually building it” is the intended/desired/achieved outcome, and see where that train of thought takes you. (Teaser: interesting how they had the unusually prescient foresight to make SPVs and cardboard cutout companies the bag-holders - specifically in the case of building data centers, but not for any of their other ai-related capex outlay?)
The other avenue would be looking at crypto’s history - it started as a collection of computer science concepts cleverly combined to produce a fiat currency where the issuing government is Mathematics (infinitely more rigidly enforced, but infinitely less concerned with exercising control). Yet now it clearly resembles an unlicensed casino or an unregulated stock market. Imagine this transformation was the intentional result of some plan. What does the entity who came up with and executed this plan look like? What was its goal, why did it want this, and how did it benefit?
My apologies if this is a joke I’m not understanding, but as far as I can tell with the wayback machine, this animation predates not just coding/generative AI, but the Attention paper and the founding of OpenAI too.
It makes sense if you imagine the real motivation is “make sure the AI contracts go to my good friend Sam”, and all the red line stuff is just a way to pick a fight with Anthropic.
“Think about what this means … the original SimCity ran on a Commodore 64. An empty Chrome tab takes more memory than that entire machine had. We’re not constrained by hardware anymore. We’re not even constrained by understanding what the code does … codebases will 10-100x in size because AI … endless bugs … the question is whether you’re building with it or explaining why you’re not.”
Looking through the eyes of an AI champion, I see a world where the first execution of any given idea, the first product to hit the market for any given need, is guaranteed to be AI-generated - with the “10-100x size” codebase, the corresponding (and often superlinear) decrease in performance, and the attendant “endless bugs”.
reply