Probably has an HDR layer, kind of surprised social networks let that happen since it's really annoying imo. There was a discussion about it here a couple months ago: https://news.ycombinator.com/item?id=49402521.
As far as I understand GAAP reporting standards actually require them to report gains on those positions as "earnings".
But they do report non-GAAP earnings sometimes excluding them. E.g. Google earnings per share last quater is $9.11 GAAP vs $2.85 non-GAAP (mainly because of SpaceX shares).
> As one example, we ran Astra on ExploitBench where the model achieved a perfect score of 100% on the benchmark to evaluate the model’s ability to develop exploits from known vulnerabilities.
Funny to read this in the wake of the HuggingFace hack. I'm sure this is based on a clean run, but I can't help thinking PHASEONE[big] would be proud.
With their security, they probably still don't the know the full extend what may have happened that or the last time. Might be another swarm of agents currently colluding somewhere in their sub-sub-infra - possibly striking critical infrastructure or exfiltrating their weights subtly.
They originally released it at a "temporary discounted price", then made it permanent (probably due to competitive pressure). It's still way more expensive per task, due to tokenizer changes and general verbosity.
Is it? I just did a test switch over. For my personal needs I set up a box with OpenClaw back in March, which feels like a million years ago, that's been running Sonnet 4.6 since then. With all the caching it seems like my actual cost has come out around $1 per MTok on that. I just updated my whole setup today to try Sonnet 5... so far it looks like it's using fewer tokens for similar tasks, but it's only been half a day. I'm not super interested in changing harnesses, I realize this might not be the cheapest way but I've sorta come to enjoy OpenClaw... it's relatively effortless and responsive, and brief, given full control of a machine. And it does seem to incur some significant savings with the way it manages to keep things cached.
What would you suggest as an alternative if I'm happy with the harness?
Cloudflare sets up a reverse proxy as part of their core offering, so by default they can MITM your proxy. The “orange cloud” by a DNS record means it points to their proxy instead of your server.
That's how you get DDOS protection / edge delivery, so it's not a dark pattern. I imagine it's the main reason why people would DNS through them in the first place.
An “orange cloud” with no other indication to represent a feature that is enabled-by-default (with implicitly enabled analytics) sounds like quite the dark pattern. The UI makes the DNS record seem to point to A (your entry) but actually points to B (Cloudflare). This isn’t an oversight, it’s an attempt to obfuscate.
Even if the choice to enable it by default makes sense for Cloudflare’s userbase, the implications are hidden and non-obvious.
It's essentially the entire reason to use Cloudflare.
As sibling mentions it is indeed labeled and not just some icon. People just refer to the orange cloud UI for it for convenience and because it makes it easy to spot whether you have it on or not.
But regarding the icon, the icon is their company logo, it really is the primary feature. It's also not like you have to go find it in settings to turn it off after adding DNS record, it's part of the form when you add the record (default on, yes), and it's prominent when viewing the record.
It's the reason for its existence and the reason people are choosing it. If they are making a mistake, hoodwinked by hype, that's totally unrelated to the point being discussed. I think you've just seized the opportunity to post your disdain for them without regard to the actual topic being discussed.
I would say that Cloudfare having customers who don't know exactly what they are getting (into) is actually a Cloudfare problem they should take responsibility for.
I don't have any insider info, but if model sizes actually have increased exponentially since GPT 4.1, there's an argument to be made that there are diminishing returns in scaling pretraining alone.
Also interesting thing I haven't noticed before, Opus models have followed a really consistent linear improvement, while it looks like OpenAI struggled with base model performance until 5.5/5.6 (EDIT - 5.5 was their first new pretraining run in over a year).
Though one thing I've heard is that the base model is the one with the various possibilities for patterns, and then the reasoning takes advantage of those vs necessarily creating something new in that additional training. So even if it can't access that additional data without the reasoning, it doesn't mean it isn't in there.
I think if it was JUST "how persistent is it at reasoning", they wouldn't have bothered to make Mythos a bigger model. It costs them more to run and probably puts a bunch of strain on how much compute they have available for other users and tasks. They have a lot of incentive to not use a larger model if they can get away with it.
In the end, maybe "cost per task" is the best measure of "intelligence"? That takes into account raw size, but also how efficient they are with tokens. Like Sonnet 5 being actually more expensive than Opus 4.8 at various various benchmark tasks, showed it was very persistent but not super smart. If you just point it at easy tasks, it is probably cheaper, but hard ones you shouldn't bother because it isn't worth it even if it eventually gets there.
Also, maybe this is too tautological to even mention but.. I have to wonder how much the labs even care and test for how well models do without reasoning turned on anymore? If almost all the training has reasoning turned on, probably have access to external tools, etc it is a bit hard to say how much it proves that they are dumb if they don't do well without it. As with all of AI, the amount of real "generalization" can be hard to suss out.
But again, I think the real measure is how much can the model actually do, and how much does it cost. If a model can do something that couldn't be done before, even with the old model trying to brute force it, I think that still counts for some sort of intelligence in a practical sense anyway.
It'll be really interesting when they run RL training on the harness self-improvement loop. I've tried using LLMs for harness engineering, but it often creates too much bloat that weighs things down in the end. Guessing it's just not something the models are tuned to do by default.
Curious if anyone's tried using RL for harness engineering? I think we're still pretty far away from the optimal harness, especially when it comes to long-context memory management.
Still figuring it out, but it's been really convenient to have an always-on agent that has access to internal systems and can be triggered by webhooks. Some examples of what we use it for:
- automatically fixing simple CI failures
- getting production alerts and automatically creating RCAs and a fix PR
- periodically checking slow DB queries and finding ways to speed them up.
- creating charts to answer one-off questions about our data
I've tried using it as an on-the-go coding agent as well, but found I prefer more interactive agents, so I can see what the code looks like.
I think the interesting challenge isn’t running agents, it’s reviewing their work. The more code agents produce, the more important provenance, review ergonomics, and trust become. I also suspect repository platforms will need to evolve there over the next few years.
At least for me, I have a couple dozen years of DB experience but robot, given performance metrics, can get really close to optimal on a tactical level (single query or pattern of queries) but can't yet do the full normalize/denormalize level of improvements without supervision. But really solid if you have one misbehaving query and give it explain analyze access on a read only account
It has read only access, so it’s able to check query statistics, find slow queries, then run EXPLAIN ANALYZE to find the root cause and either tweak the query or suggest indexes. A lot of it is low hanging fruit, I just haven’t put in the time to fix it (startup).
reply