The way Claude/Codex behave is entirely consistent with how every vibe coded project (of mine) has ended up so far. I bet those guys have no idea what's going on and are taking guesses because no one understands the thing they've made.
Instead of asking the model: "Here's this codebase, report any vulnerability." you ask. "Here's this codebase, report any vulnerability in module\main.c".
The model can still explore references and other files inside the codebase, but you start over a new context/session for each file in the codebase.
Honestly, that's the only way I've ever been able to trust the output. Once you go beyond the scope of one file it really degrades. But within a single file I've seen amazing results.
Are you not supposed to include as many _preconditions_ (in the form of test cases or function constraints like "assert" macro in C) as you can into your prompt describing an input for a particular program file before asking AI to analyze the file?
Please, read my reply to one of the authors of Angr, a binary analysis tool. Here is an excerpt:
> A "brute-force" algorithm (an exhaustive search, in other words) is the easiest way to find an answer to almost any engineering problem. But it often must be optimized before being computed. The optimization may be done by an AI agent based on neural nets, or a learning Mealy machine.
> Isn't it interesting what is more efficient: neural nets or a learning Mealy machine?
...Then I describe what is a learning Mealy machine. And then:
> Some interesting engineering (and scientific) problems are: - finding an input for a program that hacks it; - finding a machine code for a controller of a bipedal robot, which makes it able to work in factories;
I've noticed even experienced engineers have started overestimating how long things would take to build without AI. Believe it or not we coded before AI and not everything took years all the time.
We’ve all worked on projects where it took months to get requirements from the business. Sometimes to see the project cancelled after months of sitting around waiting for them to decide on things.
Coding has never been the roadblock in software. Indeed don’t we experience this now with ai? Vibe code a basics idea then discover the things we didn’t consider. Try to vibe that and the code base quickly gets out of hand. Then we all discover “spec driven development” SDD and in turn discover thinking of specifying everything our selves is an even bigger of PITA?
Is all of it shit, or you just can't find the good stuff? "The struggle will completely shift to how to get traffic" is from the business side, and you're experiencing it from the customer side.
Employees are the ones with the real power to make this hurt. The customers switching over are easily offset by the DoD contract. But losing talent over this, and having a harder time to attract future talent? That could hurt them
Sam probably expects to solve this by just offering more money. It worked in the past
Yeah, they will have to raise salary by 10% to attract people. This will no doubt hurt their bottom line. Poor starving SV text workers will have no choice but to accept working for them, lest they starve.
Maybe my sarcasm is not justified, but I don't think most people care that that work for a company that does unethical things. In fact I think all large companies are more or less immoral (or rather amoral) - that's just how the system is built.
I do this all the time but then you end up with really over engineered code that has way more issues than before. Then you're back to prompting to fix a bunch of issues. If you didn't write the initial code sometimes it's difficult to know the best way to refactor it. The answer people will say is to prompt it to give you ideas. Well then you're back to it generating more and more code and every time it does a refactor it introduces more issues. These issues aren't obvious though. They're really hard to spot.
reply