51 comments so far, the vast majority panning Astra's coding abilities. An uninformed reader may come away with the impression that this isn't an absolutely revolutionary technology that with coding abilities many of us thought were not even going to be possible with language models as recently as a year ago.
Yeah, it's not perfect, but it's really good and extrapolating this rate of improvement for 6 months is rather terrifying (from a SWE perspective, at least).
The biggest jump was Opus 4.6. Since then they have gradually gotten better at finding issues in your reasoning, not hallucinating, and being rigorous with the code, but much much worse at explaining things and generally just talking in a way that a human can understand. All the models I've tried seem to be suffering from the same fate so it must be something going on with the training meta right now.
i think the main consensus here is that the actual performance is not indicative of the benchmark performance (which supposedly outperforms the previous iterations)
It's impressive technology but not revolutionary. Revolutionary technology would have resulted in, you know, a revolution in software quality. Instead quality keeps going down.
I agree, the nomenclature is impenetrable, it's like reading software that is not well commented. Perhaps LLMs are very good at "challenging" mathematics because what we perceive as challenging is primarily the language component and not the conceptualization.
As someone who leans pro in this debate, I don't think I would make that statement. I would say the results are exactly as we expect.
Also, a highly verifiable task like this is well suited to LLMs, and I expect within the next ~2 years AI tools will produce a better compiler than gcc.
it can feed into itself and improve. the idea that self-training necessarily causes deterioration is fanfic. remember that they spend massive amounts of compute on rl.
No, they will point out that the way to make GCC better is not really in the code itself. It's in scientific paper writing and new approaches. Implementation is really not the most work.
Yes, we will certainly go that way, probably code already added to gcc has been developed through collaborative AI tools. Agree we don't call that "produced by AI".
I think compilers though are a rare case where large scale automated verification is possible. My guess is that starting from gcc, and all existing documentation on compilers, etc. and putting ridiculous amounts of compute into this problem will yield a compiler that significantly improves benchmarks.
"Commerce Secretary Howard Lutnick told reporters on a call Friday evening that the administration came to the fee of $100,000 per year, plus vetting costs, after talking with companies.
He noted that the payment structure is still under discussion with the Department of Homeland Security, in terms of “whether we’re going to charge the $300,000 up front or $100,000 a year for the three years.”
I found this map a few years ago and had it printed online on canvas, to hang on my wall near my bike area, I recommend doing this with other old maps as well.
One thing to keep in mind with BI software is that the users are often very different than, well, those individuals that prefer to use mutt as an email client.
Many, or most, users for a BI tool will be operations, product managers, and business management who simply will not find the interface to be intuitive, responsive, or well designed. At least that's my experience.
I agree, learning admissible heuristics will retain worst case performance, which has always been the measuring stick for these algorithms. It's not at all uncommon to find faster solutions for the average or even p99 cases that cannot provide guarantees on the worst case.
For something like focal search, it doesn't even need admissibility, you just apply it as a second selection heuristic among the choices your admissable heuristic returns as 'top k' choices.
reply