There's still the question of access to the codebase. By all accounts, the best LLM cyber scanning approaches are really primitive - it's just a bash script that goes through every single file in the codebase and, for each one and runs a "find the vulns here" prompt. The attacker usually has even less access than this - in the beginning, they have network tools, an undocumented API, and maybe some binaries.
You can do a lot better efficiency-wise if you control the source end-to-end though - you already group logically related changes into PRs, so you can save on scanning by asking the LLM to only look over the files you've changed. If you're touching security-relevant code, you can ask it for more per-file effort than the attacker might put into their own scanning. You can even do the big bulk scans an attacker might on a fixed schedule - each attacker has to run their own scan while you only need to run your one scan to find everything they would have. There's a massive cost asymmetry between the "hardening" phase for the defender and the "discovering exploits" phase for the attacker.
Exploitability also isn't binary: even if the attacker is better-resourced than you, they need to find a whole chain of exploits in your system, while you only need to break the weakest link in that chain.
If you boil security down to just a contest of who can burn more tokens, defenders get efficiency advantages only the best-resourced attackers can overcome. On net, public access to mythos-tier models will make software more secure.
On that latest episode of 'Security Cryptography Whatever' [0] they mention that the time spent on improving the harness (at the moment) end up being outperformed by the strategy of "wait for the next model". I doubt that will continue, but it broke my intuition about how to improve them
This is basically how you should treat all AI dev. Working around AI model limits for something that will take 3-6 months of work has very little ROI compared to building what works today and just waiting and building what works tomorrow tomorrow.
This is the hard part - especially with larger initiatives, it takes quite a bit of work to evaluate what the current combination of harness + LLM is good at. Running experiments yourself is cumbersome and expensive, public benchmarks are flawed. I wish providers would release at least a set of blessed example trajectories alongside new models.
As it is, we're stuck with "yeah it seems this works well for bootstrapping a Next.js UI"...
This assumes AI model improvements will be predictable, which they won’t.
There are several simultaneous moving targets: the different models available at any point in time, the model complexity/ capability, the model price per token, the number of tokens used by the model for that query, the context size capabilities and prices, and even the evolution of the codebase. You can’t calculate comparative ROIs of model A today or model B next year unless these are far more predictable than they currently are.
Chinese AI vendors specifically pointed out that even a few gens ago there was maybe 5-15% more capability to squeeze out via training, but that the cost for this is extremely prohibitive and only US vendors have the capex to have enough compute for both inference and that level of training.
I'd take their word over someone that has a vested interested in pushing Anthropic's latest and greatest.
The real improvements are going to be in tooling and harnessing.
> The real improvements are going to be in tooling and harnessing
I don't have any special knowledge here, but the guy in the podcast (who worked/works with one of the big AI firms) is the one who made the claim. In the future when (if?) the speed of development slows I agree it would no longer be true
It's wild to me that a paragraph or 7 of plain English that amounts to "be good at things" is enough to make a material difference in the LLM's performance.
As the base is an auto-regressive model that is capable of generating more or less any kind of text, it kind of makes sense though. It always has the capabilities, but you might want it to emulate a stupid analysis as well. So you're leading in with a text that describes what the rest of the text will be in a pretty real sense.
I read once (so no idea if it is true) that in voice lessons, one of the most effective things you can do to improve people's technique is to tell them to pretend to be an opera singer.
I think you took away the wrong lesson from that podcast:
I think there is work to be done on scaffolding the models better. This exponential right now reminds me of the exponential from CPU speeds going up until let’s say 2000 or something where you had these game developers who would develop really impressive games on the current thing of hardware and they do it by writing like really detailed intricate x86 instruction sequences for like just exactly whatever this, like, you know, whatever 486 can do, knowing full well that in 2 years, you know, the pen team is gonna be able to do this much faster and they didn’t need to do it. But like you need to do it now because you wanna sell your game today and like, yeah, you can’t just like wait and like have everyone be able to do this. And so I do think that there definitely is value in squeezing out all of the last little juice that you can from the current model.
Everything you can do today will eventually be obsoleted by some future technology, but if you need better results today, you actually have to do the work. If you just drop everything and wait for the singularity, you're just going to unnecessarily cap your potential in the meantime.
The problem, though, is that this turns "one of our developers was hit by a supply chain attack that never hit prod, we wiped their computer and rotated keys, and it's not like we're a big target for the attacker to make much use of anything they exfiltrated..." into "now our entire source code has been exfiltrated and, even with rudimentary line-by-line scanning, will be automatically audited for privilege escalation opportunities within hours."
Taken to an extreme, the end result is a dark forest. I don't like what that means for entrepreneurship generally.
This is a great example of vulnerability chains that can be broken by vulnerability scanning by even cheaper open source models. The outcome of a developer getting pwned doesn't have to lead to total catastrophe. Having trivial privilege escalations closed off means an attacker will need to be noisy and set off commodity alerting. The will of the company to implement fixes for the 100 Github dependabot alerts on their code base is all that blocks these entrepreneurs.
It does mean that the hoped-for 10x productivity increase from engineers using LLMs is eroded by the increased need for extra time for security.
This take is not theoretical. I am working on this effort currently.
It's great news for developers. Extra spend on a development/test env so dev have no prod access, prod has no ssh access; and SREs get two laptops, with the second one being a Chromebook that only pulls credentials when it's absolutely necessary.
Yes, having a good development env with synthetic data, and an inaccessible, secure prod env just got justification. I never considered the secondary SRE laptop but I think it might be a good idea.
The value-add is having a workstation that's disconnected from work that would be susceptible to traditional vectors that endpoints are vulnerable to. For example, building software that pulls in potentially malicious dependencies, installing non-essential software, etc. The "SRE laptop" would only have a browser and the official CLI tools from confirmed good cloud and infrastructure vendors, e.g. gcloud, terraform.
I think that such a posture would only be possible in a mature company where concerns are already separated to the point where only a handful of administrators have actual SSO or username/passphrase access to important resources.
It's not a joke. Supply chain attacks are a thing, but Google Chromebooks are about the most trustable consumer machine you can run custom code on short of a custom app on an iPad. The Chromebook would only ever have access to get the root AWS (or whatever) credentials to delete, say, the load balancer for the entire SaaS company's API/website. If my main laptop gets hacked somehow, the attacker can't get access to the root AWS credentials because the main laptop doesn't have them. The second laptop would only be used sparingly, but it would have access to those root credentials.
since the suggestion is that the new security bug finding LLMs will increase protection because it will have access to the full source code then, the dark forest fear would be, if it is possible for an attacker to get all the source the attacker will be in a better position.
This seems wrong however, as it ignores the arrow of time. The full source code has been scanned and fixed for things that LLMs can find before hitting production, anyone exfiltrating your codebase can only find holes in stuff with their models that is available via production for them to attack and that your models for some reason did not find.
I don't think there is any reason to suppose non-nation state actors will have better models available to them and thus it is not a dark forest, as nation states will probably limit their attacks to specific things, thus most companies if they secure their codebase using LLMs built for it will probably be at a significantly more secure position than nowadays and, I would think, the golden age of criminal hacking is drawing to a close. This assume companies smart enough to do this however.
Furthermore, the worry about nation state attackers still assumes that they will have better models and not sure if that is likely either.
Any single company might be able to proactively defend themselves from attackers, but will companies invest the tokens in this? Most people simply don't care until it's too late.
And in a world where companies begin to suffer from attacks as a result - can the ones who are willing to invest in security defend themselves, not just against cyberattackers, but against a broader investor and customer backlash that believes that startups that build their own technology stacks are riskier due to perceptions about cybersecurity?
An angel investor or LP who sees news articles and media about cyberattacks, then has a portfolio company get hacked in a material way, may simply decide the space has become too risky for further investments, no matter how much prospects get on better security footings.
The dark forest hypothesis, at its core, is about a decision of whether to put your neck out in the universe; if the weapons and countermeasures being used are too horrifying to fathom, the risks unquantifiable, one chooses not to extend one's neck. And that is how an industry begins to dry.
The pressure by internal auditors and cyber insurance providers to implement these programs will be strong. I have been at organizations where EDR was added only due to the board of directors following the recommendation of 3rd parties. Of course, there will be new companies that haven't achieved the maturity to have had these pressures. But new companies being thoroughly compromised is hardly a recent phenomenon.
I guess the connection would be human history, a dark forest is a scene of lawlessness and violence and danger in much of that history - at least where stories are concerned.
In the use of the phrase Dark Forest to explain the Fermi paradox it suggests that alien civilizations have kept themselves dark out of fear that the rest of the forest is actually lawless and violent.
In this case though we are entering a dark forest, like Hansel and Gretel, supposedly defenseless against the monsters that lurk in there, but really - they weren't that defenseless were they? I don't think the phrase that apt.
> In the use of the phrase Dark Forest to explain the Fermi paradox it suggests that alien civilizations have kept themselves dark out of fear that the rest of the forest is actually lawless and violent.
It's more complicated.
For the Fermi paradox version of the 'Dark Forest' to work, you need civilisations to actively go out and destroy any other form of life they find announcing themselves:
> The "dark forest" hypothesis presumes that any space-faring civilization would view any other intelligent life such as theirs as an inevitable threat and thus destroy any nascent life that makes itself known. As a result, the electromagnetic radiation surveys would not find evidence of intelligent alien life.
Wikipedia has a section on game theory etc.
Without this additional element (basically the version you describe), the dark forest theory doesn't explain the Fermi Paradox: it's just another filter that might perhaps exclude 90% of civilisations, but many civilisations would still be dumb enough to announce themselves. Humans certainly did and keep doing so: it only needs a some people to send a message, and near unanimity to not send anything.
(And that's completely ignoring that our very atmosphere with its chemical imbalance has been sending a strong message of "there's probably life here" for billions of years now. Even our own technology, still in its infancy, is increasingly able to pick up clues about the chemical composition of the atmosphere of exoplanets ever further away from us. And we are still getting better quickly.)
If you add the element that other civilisation are hiding, but come out of hiding just to strike, that breaks down as soon as you have more than two players. Or even just the faint possibility of more than two players.
When you know there are only at most exactly to players, and you are the lurker and find someone else being 'noisy': yes, you have an incentive to strike. When there might be other third parties lurking, you better stay quiet, lest you invite a strike by a third party against you.
My point is that the cost for the attacker is higher than the cost for the defender, if the attacker has to spend tokens probing for vulnerabilities against a system which has little know about it, while the defender spends tokens on a system they have the full source to.
That is not at all relevant to "security via obscurity" or similar arguments: having the source in the open may (eventually) be more secure, but it lowers the token-spend for the attacker.
> By all accounts, the best LLM cyber scanning approaches are really primitive - it's just a bash script that goes through every single file in the codebase
What accounts are these?
I've seen some people use this but I cannot imaging that anyone thinks this is the best.
For example I've had success telling LLMs to scan from application entry points and trace execution, and that seems an extremely obvious thing to do. I can't imagine others in the field don't have much better approaches.
Indeed, all the hot security scanning vendors are using custom prompts to capture a more holistic approach. There are of course plenty of legacy scanners that still focus on OS package versions and static configs, but the parts of the industry leaning into LLMs have genuine value to add.
I don't expect Claude Code Review to be a replacement for a good vendor's solution.
This feels pretty fertile atm to me, because it has been prohibitively expensive to do. I expect there is a ton of low hanging fruit. Why not in the age of AI?
One defender, many attackers, I don't see how the economy of scale can be positive for the defender.
Assuming your code is inaccessible isn't good for security. All security reviews are done assuming code source is available. If you don't provide the source, you'll never score high in the review.
I think automated scanning can be positive for the defenders, when the rate of introducing new vulnerabilities vs fixing old ones is < 1 (detection rate + infra is a factor too ofc). In that case, AI can become the many eyes to check FOSS and those projects will eventually reach a "secure" state.
Yes, and it apparently burns lots of tokens. But what I've heard is that the outcomes are drastically less expensive than hand-reversing was, when you account for labor costs.
Can confirm. Matching decompilation in particular (where you match the compiler along with your guess at source, compile, then compare assembly, repeating if it doesn't match) is very token-intensive, but it's now very viable: https://news.ycombinator.com/item?id=46080498
Of course LLMs see a lot more source-assembly pairs than even skilled reverse engineers, so this makes sense. Any area where you can get unlimited training data is one we expect to see top-tier performance from LLMs.
My own experience has been that "ghidra -> ask LLM to reason about ghidra decompilation" is very effective on all but the most highly obfuscated binaries.
Burning tokens by asking the LLM to compile, disassemble, compare assembly, recompile, repeat seems very wasteful and inefficient to me.
That matches my experience too - LLMs are very capable in "translating" between domains - one of the best experience I've had with LLMs is turning "decompiled" source into "human readable" source. I don't think that "Binary Only" closed-source isn't the defense against this that some people here seem to think it is.
> Has anyone used an LLM to deobfuscate compiled Javascript?
Seems like a waste of money; wouldn't it be better to extract the AST deterministically, write it out and only then ask an LLM to change those auto-generated symbol names with meaningful names?
yes, but it requires some nudging if you don't want to waste tokens. it will happily grep and sed through massive javascript bundles but if you tell it to first create tooling like babel scripts to format, it will be much quicker.
Yeah, it's token intensive but worth it. I built a very dumb example harness which used IDA via MCP and analyzed/renamed/commented all ~67k functions in a binary, using Claude Haiku for about $150. A local model could've accomplished it for much less/free. The knowledge base it outputs and the marked up IDA db are super valuable.
I did something similar using ghidramcp for digging around this keyboard firmware, repo contains the ghidra project, linux driver and even patches to the original stock fw. https://github.com/echtzeit-solutions/monsgeek-akko-linux
Another asymmetric advantage for defenders - attackers need to burn tokens to form incomplete, outdated, and partially wrong pictures of the codebase while the defender gets the whole latest version plus git history plus documentation plus organizational memory plus original authors' cooperation for free.
Prediction 1. We're going to have cheap "write Photoshop and AutoCad in Rust as a new program / FOSS" soon. No desktop software will be safe. Everything will be cloned.
Prediction 2. We'll have a million Linux and Chrome and other FOSS variants with completely new codebases.
Prediction 3. People will trivially clone games, change their assets. Modding will have a renaissance like never before.
Prediction 4. To push back, everything will move to thin clients.
I think if prediction 1 is true (that it becomes cheap to clone existing software in a way that doesn't violate copyright law), the response will not be purely technical (moving to thin clients, or otherwise trying to technically restrict the access surface to make reverse engineering harder). Instead I'd predict that companies look to the law to replace the protections that they previously got from copyright.
Obvious possibilities include:
* More use of software patents, since these apply to underlying ideas, rather than specific implementations.
* Stronger DMCA-like laws which prohibit breaking technical provisions designed to prevent reverse engineering.
Similarly, if the people predicting that humans are going to be required to take ultimate responsibility for the behaviour of software are correct, then it clearly won't be possible for that to be any random human. Instead you'll need legally recognised credentials to be allowed to ship software, similar to the way that doctors or engineers work today.
Of course these specific predictions might be wrong. I think it's fair to say that nobody really knows what might have changed in a year, or where the technical capabilities will end up. But I see a lot of discussions and opinions that assume zero feedback from the broader social context in which the tech exists, which seems like they're likely missing a big part of the picture.
You cannot get away with „well no one is going to spend time writing custom exploit to get us” or „just be faster than slowest running away from the bear”.
I've actually had pretty good results from doing exactly that. There was one FP when it tried to be Coverity and failed miserably, but the others were "you need to look at this bit more closely", and in most cases there was something there. Not necessarily a vuln but places where the code could have been written more clearly. It was like having your fourth grade English teacher looking over your shoulder and saying "you need to look at the grammar in this sentence more closely".
And using an LLM to audit your code isn't necessarily a case of turning it into perfect code, it's to keep ahead of the other side also using an LLM. You don't need to outrun the bear, just the other hikers.
> By all accounts, the best LLM cyber scanning approaches are really primitive - it's just a bash script that goes through every single file in the codebase and, for each one and runs a "find the vulns here" prompt
You can do a lot better efficiency-wise if you control the source end-to-end though - you already group logically related changes into PRs, so you can save on scanning by asking the LLM to only look over the files you've changed. If you're touching security-relevant code, you can ask it for more per-file effort than the attacker might put into their own scanning. You can even do the big bulk scans an attacker might on a fixed schedule - each attacker has to run their own scan while you only need to run your one scan to find everything they would have. There's a massive cost asymmetry between the "hardening" phase for the defender and the "discovering exploits" phase for the attacker.
Exploitability also isn't binary: even if the attacker is better-resourced than you, they need to find a whole chain of exploits in your system, while you only need to break the weakest link in that chain.
If you boil security down to just a contest of who can burn more tokens, defenders get efficiency advantages only the best-resourced attackers can overcome. On net, public access to mythos-tier models will make software more secure.