The threshold for a surveillance system to affect societal norms is not necessarily "legal event", and potentially even lower than any observable reaction (from self-censorship).
Just consider how algorithmic moderation can shape language (=> people using weird constructs like "to unalive", or weird metaphors in chinese), even in contexts where it would technically be unnecessary.
A close US equivalent is the "cant google that, I'll end up on a list" notion. This is all quite undesirable from my point of view.
Perhaps, depending on specific intent, credibility, and the nature of harm threatened.
But since this is about surveillance, I hope that detection of verbal threats is not a goal of government surveillance because it's difficult to imagine how that could be accomplished without significant loss of privacy or other liberties.
I can see it in court now. Our AI monitoring system did indeed know about the threat to the building where 800 people died on Sunday.
It says: "
Agent: Voice to text detected: I have everything ready - all the XXX chemicals are ready in the van and I'm going to park in the 900 S Crap St now"
Agent: Thread Level HIGH.
Agent: Looking up local codes.
Agent: Mayor signed SB-1238 in 2026 - no surveillance devices may be used for audio threat determination.
Agent: Threat silenced, but logged.
Judge: Oh, that makes sense. Make sure to bag and tag and bill the families for the bags.
City Employee: We also know who parked the van, should we arrest them.
Judge: No it looks like SB-1238 would forbid us from using this data for the purposes of arrest. I guess send them a thank you letter for testing our laws.
Oh, only 800? Maybe you can pick a larger imaginary number to make me feel really guilty about not wanting to give up my rights to live free of surveillance.
They wrote a 600 page report about it and it included a ton of recommendations. Not many people remember at this point, but for months and even a few years after, the entire country was on edge about it happening again, in different means (trains, car wrecks on purpose, shootings). There is a reason they have called this a post-911 world ever since. That hasn't ended.
Appreciate the pushback, saltyoldman. Yes, we want to respond to credible threats. And, as always, courts and law enforcement can invade privacy when there's reason to believe someone is worth surveilling. But we're talking here about widespread, extremely cheap, technically easy surveillance of potentially everyone at all times. That's the endgame that some commercial and government interests have in mind.
Would you agree that sometimes an uptick in theoretical safety is not worth a downtick of definite lost liberties?
I used to be that way. However more recently I have come to prefer security over privacy, at least where I live. I do want to make sure human, drug and weapon traffickers are not exiting off my freeway ramp. I do get the issues with what you're saying, but let's think of ways to have both. The existence of a surveillance net with safegards. In other words yes let's have the conversation to make our country secure and also prosecute sherrifs spying on their girlfriends, make sure no API hole exists and some company isn't selling billions worth of data to China.
There is no way to have both. Surveillance is power and it corrupts in the same way as any other form of power. It's not just about patching some individual holes. You can't have too much of it for the same reason why you can't have a cop stationed at every single building in your city. For sure, doing that would make some people feel safer, but it would also make anyone doing something legal but disfavored by their government terrified, increase prosecutions for frivolous infractions and open the door for a future government to swoop in and make great use of all that free power lying around.
Besides, even if it was possible to do both (it's not), do you think this would ever actually happen? When it comes to surveillance, they only take and take and take and never give anything back, further encouraged by a terrified populace that wants more safety in a safer-than-ever world. It's a ratchet that only goes one way because it greatly benefits anyone vying for power in governments and businesses alike. Once you let them have it, you're not getting anything back.
I'm in Seattle and everyone knows exactly where human trafficking is happening and the police are doing nothing about it. Teenagers are being pimped out all along Aurora and literally nothing is happening despite literally years of public outcry.
The pimps get arrested again and again and then released without charges being filed.
The interesting thing is how I was making a very contained point pertaining to cameras, and how cameras, which we were talking about in this thread, seeing a verbal confrontation, could not and should not make a call, because a verbal confrontation is not a legal event. You then took this into a totally different case involving ... what? hypothetical recording of a conversation between two hypothetical terrorists? To prove ... what? My point is that it is not a shortcoming of the camera that it is not making a judgement call on the thing OP was originally talking about. A verbal altercation between two people. I was not talking about a hypothetical bombing. I was not citing a specific law, I was not advocating that there should be a law, I was not advocating anything about whether or not we should ban collection of existing evidence. I was not making any of these moves. I was saying simply: a camera looking at two people in a verbal argument from far enough away that it cannot hear the conversation is not a failure of the technology. Not every negative interaction between two human beings is criminalizable.
You received a straw man and decided to engage it. You fell for the trap, and have already been put into a losing position. How are you supposed to recover from engaging this straw man.
alternatively, it turns out the voice to text ended up picking up on dialog from a movie the suspect was watching, and he opens the door to a SWAT team thinking that's his pizza being delivered.
I don't think you're advocating to have our personal conversations continuously monitored whenever outside, but in the context of this thread, that's what it sounds like.
No, in the context of the thread it sounds like they're illustrating myrmidon's point about how the selective enforcement of crimes that are easy to catch on camera means that the police have less time (and less inclination, training, norms) for addressing more serious crimes, like interpersonal violence.
More broadly, they're not saying that we should make the cameras better to catch more crime, they're saying that when you make cameras the main way you catch crime, you shift the social definition of what crime is to "what cameras can catch".
When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?"
Dario's point, and the point of the people actually trying to solve the problem, is that AI is not just Anthropic and OpenAI. It's the knowledge that you can put more compute in, and get more capability out.
It is a technology now. It exists, in the world. Wishing will not make it go away. Being angry at it will not make it go away. Lying about how much water it uses will not make it go away. If Anthropic and OpenAI Shut down tomorrow, Accenture will not say "oh guess that llm thing won't work, let's go back to hiring humans!"
It is a truth that you can multiply matrices and get something that is economically useful. We cannot un-know this.
Physics allows it, so it will happen. So we should probably figure out what the heck to do about it. If your answer is something along the lines of "restrict it" then 1. let me know how that goes when other people don't, and 2. I really would rather prefer a world where we have the machines do the work the machines can do, not a world where we have human makework. If this means we need to figure out redistribution, let's talk about redistribution!
Physics allows this, and actually taking advantage of it requires billions of dollars of unprecedented infrastructure buildout that is already destabilizing the power grid.
The only reason that infrastructure buildout is happening at all is the ideological capture of a small handful of obscenely wealthy people, who are fueling this buildout by spreading the extreme paranoia you’re echoing here.
I do not understand why no one else can see the circularity of this reasoning. There is nothing inevitable about tying up all of this productive capital in the pursuit of AI. There are many, many other projects requiring similar capital and human effort, with much more obvious payoffs, such as decarbonizing the world’s energy systems.
“It’s physically possible to provide abundant electricity without burning fossil fuels” is more provably true than any of the insane science fiction bullshit that undergirds the AI buildout, and yet, the entire clean energy industry is still having to build insane financial Rube Goldberg contraptions to make incremental progress.
“Inevitability” is a lie, period. This entire thing is extremely historically contingent, and we could easily stop this train tomorrow.
> This entire thing is extremely historically contingent, and we could easily stop this train tomorrow.
So, the Baruch Plan?
The Manhattan Project was $~2B in 1945 dollars, and a national-scale industrial mobilization. Now North Korea has the bomb. That's with nuclear material, which doesn't get easier and easier and easier to work with every year.
Compare to the price to train GPT-2 in 2019 ($43,000), and in 2026 ($73) [0].
Since nobody uses GPT-2 any more it's more informative to compare the price to train GPT-2 in 2019 and the price to train GPT 5.5 in 2026. Unfortunately that cost is not disclosed but it's probably in the billions.
The point being: the price to train frontier models isn't coming down, nor is it going to come down because for models to remain on the frontier they have to keep getting bigger and bigger (and trained on more and more data).
> There is nothing inevitable about tying up all of this productive capital in the pursuit of AI.
In the US capitalist context, it's certainly inevitable, because AI is the biggest and most attractive source of profit and power out there right now. In that context, the broad strokes of what's happening currently, including the financial bubble, are predictable and inevitable.
What are the concrete steps which would allow us to "easily stop this train"? And why haven't we used steps like that to stop other cases where obscenely wealthy people have screwed everyone else over to increase their wealth? Is public control of the means of production involved, perhaps? If so, your definition of "easily" and mine are incompatible.
I grew up in a union household, and my dad and my grandfather fighting for better wages, healthcare and working conditions are the reason why I got a good education and work in Silicon Valley surrounded by Stanford assholes.
All of us who work for a paycheck can get together and say, “no, we will not allow you to record keystrokes and mouse movements to train our replacements. No, we will not have our performance or future employment based on an AI leaderboard.”
Previous generations fought and died for our right to do that, but in 2026 we just sit on our hands and complain on this forum. We can and should do better.
The U.S. is absolutely on fire right now with opposition to data centers. We, collectively, can extract concessions or ban their construction altogether.
These things aren’t “easy”. They are also eminently possible.
I think you're in denial about the reality of the situation.
The reality is that the US has been a story of increasing concentration of wealth and power. The people who "fought and died" bought some important (from a human rights perspective) but ultimately minor (from a capitalist perspective) concessions from the capital class. The battle you describe is one of defense of rights, not of gaining control.
The overall system of capitalist control remains unaffected, and it's why the buildout of AI is, in fact, inevitable under the current system.
You're essentially saying no, you want public control of the means of production instead. That might be great, if even a sizable fraction of the US population agreed with you. But due in no small part to decades of propaganda, they don't.
A US-China AGI ban treaty could prevent superintelligence indefinitely. Data centers are hard to hide. Have fun buying GPUs when you're cut off from all global payments. America would have to make some unpleasant concessions but that seems like a solid trade for preventing a wide variety of nightmare futures.
I don't think we can stop it, but the people saying "this is happening and we should probably figure out what to do about it" are doing their best to accelerate it, which gives us less time to figure out what to do about it.
The people saying this is happening are competing with each other for resources, so there's no way for one of them to hold back without losing out to the others. We see this with people dropping Claude subscriptions in favor of ChatGPT because codex 5.5 > Claude Opus 4.6/7/8. Anthropic is losing money by not releasing Mythos.
Aidenn0 said they're doing their best to accelerate it, and I'm saying "they're" isn't a single monolithic entity and that they're in competition with each other so they're incentivized to go as fast as possible, so it would be hard to hold them back.
> Aidenn0 said they're doing their best to accelerate it, and I'm saying "they're" isn't a single monolithic entity and that they're in competition with each other so they're incentivized to go as fast as possible, so it would be hard to hold them back.
Maybe it would run afoul of antitrust regulations, but it's totally realistic for all those competitors to get together and say "hey, we could really fuck up society in our race to get rich with this tech, lets all slow down." And if these companies are run my mature people who don't subordinate every consideration to greed, they'd to it.
I would propose that it is very likely to have zero effect. Your argument supposes they are all working together, like many connected computers calculating primes.
It only takes one of them to do it and they are not sharing information. If the 1 you remove from N is the one that will discover it, then it will dramatically affect when AGI happens. If it is not, then it will have zero effect.
> When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?"
I still believe Dario asks these questions in good faith. Nobody believes that about e.g. Sam Altman or Elon Musk. They compared themselves to Oppenheimer because it helped them get attention. When it started an actual regulatory conversation, they were suddenly less worried.
Of course he doesn't, and of course you cannot find a single person at Anthropic who cares about this, and of course you are just looking for gotcha points. But even with that. Can we please try and couple to reality just a little bit?
I personally know anthropic researchers who cared deeply about roko's basilisk. Go to an EA meetup in the bay if you'd like to meet them yourself. Sure, theyve moved past it at this point, but they still care deeply about AI x risk, and many of them do already believe that their AI is sentient. And before you claim its all a psyop to prop up AI hype these people were AI doomers before openAI and anthropic existed, they had minimal financial incentive at that point to behave that way.
Even if your core offering disappears you can do the same thing that every other SMS-sending thing can do?
I also notice you answered the question, but not in the way anyone who needs to depend on this service would want to hear. So yeah you're doing the Mac Mini thing.
I'm with landl0rd. This service should not exist, you should feel bad for creating it, and every time I get a spam iMessage I will think about you and curse your name. Hope the money's worth it.
Not even the worst reading of their reply would lend to that implication.
It’s pretty obvious that they meant that anyone who depended on their service would/should probably run away kicking and screaming if they were looking for a dependable service that will do what they claim to in the long term.
If they were Apple sanctioned, then at least you’d have some reassurance that the service won’t die randomly one day when Apple has had enough, à la Beeper.
First, that's pretty obviously not what I said. Two things can be true. This is bad, and also if I were evaluating it for use in my business, it is obviously not something I can rely on.
But then just ...Um yes? I trust Apple to keep a handle on their iMessage network. Citation: having used iMessage for ~15 years. This would mean things like ensuring that I didn't get spam. Ensuring actual company identity (does anyone remember Messages for Business?) &c. This is pretty obvious and I am trying to understand your comment?
Curse you for this ear worm of a song, but also my brain keeps wanting to interpret this as "duck... You're bad at your job and will never amount to anything. Your mother is ashamed of you and you're the reason your father is an alcoholic." hahaha
The strange part is how moral responsibility somehow always lands on the builders... the people with the least leverage... while the funders get to ask the ethical questions. Weird!
No, we don't have to take the funders' money; that's what having professional standards means. Nobody would excuse a doctor performing unsafe procedures because they "needed the money". Engineers were jailed for the Volkswagen emissions tampering scandal and nobody would excuse them for needing to take funders' money.
I agree that holding only the concrete implementers responsible would be inappropriate. However, I don't believe that distinction is made. One says "I am building a house" even if they are completely contracting the job out. I'd suggest the greatest responsibility lays with the funders and that the Pope would agree.
The funders are among the principal builders in this context. This is addressing the people who have a say in what is built, and how it is built. Much of that belongs to the executive and ownership class, but not all of it.
Who do you consider to be by-and-large, overall, more intelligent? Who then is more capable of exercising intelligence towards figuring out what's actually good?
I’m only through the introduction but this encyclical makes clear everyone bears some responsibility to act, including but not limited to the builders.
So is it that humans are inherently creative, machines could never do what we do? Or is it that humans will only replicate our training data, and so we have to ensure that machines don't bound our training data? Or are you going meta and gently pointing out the absurdity? (I hope it's this one!)
I think I have an answer. Human's don't have "training data" in the same way we think of LLMs, yes you can walk outside your house and quantify every electromagnetic pulse, random pertubation etc and then "train on it". But that isn't how people process information. We have the ability to process our entire "existence" if that makes sense, which means the density is much higher.
The LLM is bounded by it's training data, and relying on it means we are as well.
A bug existing or not for a person is a statement about that person's knowledge of the bug.
Is your assertion that, since you specifically didn't know about the bugs that nobody, not in Russia or anywhere else did?
Obviously if bugs are out there existing in software and you don't know about them, or the CVE system doesn't know about them, or whatever ... this does not preclude bad guys from knowing about them. In the era of agents, knowing the bug exists is equivalent to having a PoC, so the distinction completely collapses.
Two people arguing in public, words only, is close to a legal non-event in the US. So I would hope so?
reply