Excellent breakdown. The title seems misleading since we don't even know it's AI. I very much dislike when folks state speculation as fact.

> Amazon created an AI agent to look at every account and, instead of flagging them for any potential violations, had them canceled outright.

“What cannot be known hollows the mind. Fill it not with guesswork.”

I can say I have seen a lot of cases where someone who was flagrantly guilty of abuse complained loudly that their account at some big tech company was unfairly canceled. I cannot say that's what is going on here, and I can also say I've seen plenty of cases where it was unfair and there was no due process.

It's not Constitution-free, but CBP does claim a lot of extra rights within that zone. It covers about two-thirds of the U.S. population.

https://www.aclu.org/know-your-rights/border-zone

It's like when you're playing a video game and you accidentally press the wrong button and shoot a guy in the face instead of healing him.

The idea that you could accidentally open a webpage instead of launching a program is a UX embarrassment.

Right, but when humans are writing the code, they have learned to focus on putting downward pressure on the complexity of the system to help mitigate this effect. I don't get the sense that agents have gotten there yet.

Big business LLMs even have the opposite incentive, to churn as many tokens as possible.

At least tokens are equivalent to measuring 'thinking'... I wouldn't mind if it burned 100k tokens to output a one line change to fix a bug.

The problem is maximizing code generated per token spent. This model of "efficiency" is fundamentally broken.

Debugging would suffer as well, I assume. There's this old adage that if you write the cleverest code you can, you won't be clever enough to debug it.

There's nothing really stopping agents from writing the cleverest code they can. So my question is, when production goes down, who's debugging it? You don't have 10 days.

Rather than lying, I think of it more as financial dead reckoning.

I wonder what percentage of services run on the Internet exceed a few hundred transactions per second.

I’ve seen multimillion dollar “enterprise” projects get no where close to that. Of course, they all run on scalable, cloud native infrastructure costing at least a few grand a month.

> a few grand a month.

A negligible cost for a successful tech business that also works when your requirements exceed the capabilities of a single VPS.

I agree. But these are projects that barely get any requests.

I think the better question to ask is what services peak at a few hundred transactions per second?

This is a nice point that I haven't seen before. It's interesting to regress AI to the simplest form and see how we treat it as a test for the more complex cases.

I trend libertarian because I have a strong anti-authoritarian streak. I used to think of myself as closer to the Republicans, but these days I mostly only agree with the Democrats. Weird times.

Been running lemonade for some time on my Strix Halo box. It dispatches out to other backends that they include, like diffusion and llama. I actually don't like their combined server, and what I use instead is their llama CPP build for ROCm.

https://github.com/lemonade-sdk/llamacpp-rocm

But I'm not doing anything with images or audio. I get about 50 tokens a second with GPT OSS 120B. As others have pointed out, the NPU is used for low-powered, small models that are "always on", so it's not a huge win for the standard chatbot use case.

Even small NPUs can offload some compute from prefill which can be quite expensive with longer contexts. It's less clear whether they can help directly during decode; that depends on whether they can access memory with good throughput and do dequant+compute internally, like GPUs can. Apple Neural Engine only does INT8 or FP16 MADD ops, so that mostly doesn't help.