This is going to be very geographically dependent, surely?

Some of the inference engines can process multiple requests in parallel more efficiently than doing them sequentially. Not sure of the exact mechanism but e.g. llama.cpp's llama-server can do this (you tell it the number of slots to have when starting, then fire HTTP requests at it and it batches them together when it can).

Waiting for the hooman (or tool calls) won't help either, of course.

The mechanism is that generating tokens (the "decode" phase) in an LLM is limited by memory bandwidth for the weights, so computing multiple streams amortizes the bandwidth over streams as long as you can keep the contexts in RAM. This is most true for dense models and the always-on expert in MoE models, or when you have significantly more streams than experts for MoE models.

In contrast, prompt prefill is more easily compute-bound, so there are interesting trade-offs for latency of decode vs prefill when the LLM utilization is high.

They have a presence in London; have met someone who works there. Sounds like there is an office too

There’s also a presence in Dublin. Coworking space, last time I checked.

Disk space is one thing, but the actual download size is higher than some people's data allowances altogether! It baffles me that a lot of people don't seem to be aware of this

Disagreed; it's not a download you'd expect and it's also at least an order or two of magnitude than you'd expect to find reasonable for browsing a page.

I have a 2GB mobile data plan. If I was using Chrome, then some site triggers the Prompt API, that will cause Chrome to not only wipe out my data plan, but need 2 of my data plans. I don't find this reasonable.

This is exactly a consent problem, because I'm not denying it might be a useful feature, but it should be at the user's own informed choice. The fact that Chrome developers don't appear to see this might be due to them living in a bubble where they've never had to think about the costs.

Worth noting that NVIDIA confidential computing and similar schemes have been compromised and shouldn't be relied upon if it really matters. See https://tee.fail/ and similar.

I was interested in trusted execution environments and how safe they were. If you look on google scholar and start reading, they seem super vulnerable. The feeling is that the industry has no better option and that they are a way to tell customers they are safe when they're not

with physical access right?

> the function signature is what they read, but the scars are what they need.

This feels like a golden quote. Don't know if you intended for it to rhyme, but well done :D

thanks. honestly didn't catch the rhyme, accidental aphorism :D

A while ago I checked this out and the homepage looked like it had fallen to the 'AI hype' trend, you know like how everything was 'AI-native XYZ for Autonomous Agents' at the time. I'm not seeing that now though.

Am I thinking of someone else or did you reverse on that?

Yes, that was us. And yes, we reversed on that. The feedback from our community was quite clear :)

they don't restrict you to using their opencode agent, you can use go in any other agent

Weird; I clicked through out of curiosity and didn't get any corruption of the sort in the end result.

I also asked it some technical details about how diffusion LLMs could work and it provided grammatically-correct plausible answers in a very short time (I don't know the tech to say if it's correct or not).