The arms race is an intrinsic game theoretical property of a multi-adversarial-actor scenario involving exponential growth of a universally potent technology. It's almost certainly winner-take-all, on a global scale, which behooves everyone to participate.
The actors here are states or corporations embedded in societies that risk growing popular backlash against the technology.
The other factor is that if it is truly an "alien mind", racing incurs risks to all players. In game-theoretic terms it may be more like a stag hunt than a prisoner's dilemma. In which case cooperation is an equilibrium.
This is not an impulse reaction, as I've thought deeply about this for many years, and I'm quite resolute in holding the following view:
Quibbling over the academic nature/definition of "true intelligence" is a terrifically useless endeavor from the lens of evaluating "practical impact to the world".
Regardless of whether or not one academically disagrees that these systems are intelligent, they are clearly already capable of permanently displacing a portion of human-based economic value. A small portion currently, but it's very clear that most computer-bound domains are imminently at-risk.
And that will profoundly affect the world, regardless of whether or not they're "truly intelligent" as per your personal definition.
> Regardless of whether or not one academically disagrees that these systems are intelligent, they are clearly already capable of permanently displacing a portion of human-based economic value.
You say that as if the intelligence con trick were not a major driver of that displacement.
A dimwitted bot does not need intelligence to convince a dimwitted human beings it is intelligent.
Look up "mechanistic interpretability" in the context of LLMs. The next token prediction machinery is just a foundation for a higher order learned structure that appears to encode specific concepts, regardless of input language.
The analogy to humans is that the human brain is "just atoms bouncing around", but there's unquestionably something "more" going on that just that.
Until ais get better from feeding on their own output the way humans do there is nothing to question.
That will probably be the fundamental indicator that something other than repeating things some human previously created is going on.
So far, ais only get worse from feeding on their own output. Meanging/including the output of other ais not a single ai feeding on it's own output. Also bear in mind that so far even the output of ais is 100% the downstream of a human command. No ai has persued it's own curiosity that didn't result from a human asking a question or giving a command. That is input which is different from a human taking in their environment even though our limited language can call those both the same word input.
The fact that humans also repeat and remix things, and humans also produce essentially procedurally generated empty output like corporate-speak etc, is an irrelevant distraction in the same way that both a human and an electric motor can both perform the same simple mechanical task.
In areas of applicability where GenAI to now considered to be good/great, the pattern I've observed is a frighteningly rapid progression from "abjectly terrible", to "middling and mostly more trouble than it's worth" and then "good/great".
Agentic coding and image generation are probably the best examples of this.
There's no guarantee that this trend will be seen with virtual assistants, but if it does, we're on the brink of incredible. Dedicated personal assistants, that actually know and understand one's context and work well together, are unarguably hugely appreciated by those who can afford them.
Can’t say you’re the only one but I certainly want an assistant.
I’ve been putting off some dental work for a year or more because I have an implant that’s come lose, and the two dentists I saw about it said they’d have to remove the crown, send pictures to some lab that might be able to identify the manufacturer (or maybe not), maybe get the right tooling, maybe have to replace the entire thing (thousands of dollars, removing and resetting a post in my jaw), etc etc. all because I didn’t know the manufacturer of an implant I got 20 years ago.
Claude cowork was able to understand the problem, interview me, find dental insurance paperwork in my email, identify the prosthedontist used, find that that office had closed years ago, find where the practitioner moved to, their new practice, and get me a number to call.
Within a couple of ours I had the manufacturer, procedure notes, and X-rays from the installation.
Is that WALL-E style laziness? Maybe. But I’m good with it, as I have line of sight to solving an annoying problem that seemed insurmountable.
I'm just not that annoyed by having to search for a flight, or train, or bus. Or make plans. Or find a restaurant. I'm not annoyed enough by these trivial tasks to want to outsource them.
I don't want to ask Siri to book me a restaurant, when the alternative is a quick google search.
Is it possible they have families, aging relatives to care for, health issues or other reasons that time is precious and “spend an hour correlating amazing restaurants with their available bookings against my crazy schedule” is not a good solution?
IMO the greatest failure of the HN-style tech community is complete lack of empathy, or even awareness, for people whose lives have different shapes.
The conclusion of this blog post is a bit hysterical. The intent of this steg is excruciatingly clear (identifying usage by Chinese firms that may be conducting model distillation). It's unclear on how this "punishes normal developers" in any shape or form.
So block people, instead of having false positives be secretly fucked over, and having them pay for the pleasure?
Given the hidden model degradation of fable and now this, what makes you think this is where it stops? That's just what we know about and there's clearly a long-standing and deeply rooted malicious intent here.
I've had Claude fuck over clean well documented code-bases for no reason, and there's a good chance this is due to some faulty trigger. Luckily I don't trust these things one bit, and claude only ever runs in an isolated VM, however, I am pissed I am being made to pay for their errors in detection and waste my time fixing things I apparently paid to have fucked up.
That's unacceptable conduct. It's witch-hunting. Punishment and attacks on you for things without real proof. That isn't right.
To be fair to Anthropic, [they're trying very hard to do that.](https://www.anthropic.com/news/detecting-and-preventing-dist...). The attacks are sophisticated and difficult to detect. I don't accept that if this fails, their only option is to just accept Chinese companies stealing IP.
They're not "attacks." Anthropic calling them "attacks" doesn't make them attacks. Companies are collecting transcripts of conversations with Anthropic, and giving users a discount to share them.
As many have pointed out, they're collecting data in ways much less aggressive than Anthropic itself about what Anthropic does.
Anthropic doesn't like it, but I don't see this as "Chinese companies stealing IP," any more than if Google tried to ban competitors from seeing how Google Docs or an Android phone behaves, or Ford trying to ban anyone from Toyota from seeing what their car looks like.
Please stop calling them "attacks." It's distillation training. It's looking at what Anthropic does -- as a block box -- and trying to duplicate or beat it. It's how progress happens.
Stealing IP is an attack. You might feel that Anthropic should just give away hundreds of billions in IP and capex, but that's not the way the world works, and I don't agree with you. They're taking enormous risk to build these models. They should be fairly compensated.
And I don't think anyone in their right mind would argue the AI industry isn't being fairly compensated.
To go further: most people in the world wouldn't feel bad at all if we found a way to slow what's likely the biggest socio-polical-economic change in human history down a bit.
Them strugglig to prevent others from doing what they themsleves do, is my problem to a degree where I must pay for the pleasure to getting sabotaged? Are you fucked in the head? In what world are we "being fair" by claiming that? It takes a problem anthripic has hallucinated, and makes its consequences mine for no reason. Anthropic chose to create both this problem and it's consequences, why am I wearing it?
If I decide I dislike words starting in S, despite myself being a prolific user of words starting in S, and smack some child who'd never even heard the rules, simply for saying "sorry", simply because some people say "shit" and it makes me mad, despite it being my most used word, are we "being fair" by saying "to be fair, managing curse words is a difficult problem"
> I've had Claude fuck over clean well documented code-bases for no reason
How exactly do you define "fucking over", and why do you suspect this "fucking" was done as a result of a faulty trigger as opposed to the inability of LLMs to write maintainable, extensible code?
"Never attribute to malice what can adequately be explained by stupidity."
Changing well documented requirememts and functioning code to do subtly incorrect things, over multiple areas, for no reason. When confronted you get the usual, "youre right to push back, there was no reason to touch that and it did make everything worse"
Why do I suspect faulty trigger? The things wrecked weren't wrecked by even much less capable, including local models, while reverting everything to pre fucking up and asking claude again led to similar results. Once on whatever shitlist that was, claude also failed tasks it previously aced when given the exact same prompt and project files. I attributed it to opus 4.6 being a downgrade which people always pushed back on, claiming they thought it was better, but i had empirical proof, it couldnt do tasks 4.5- could do quite well. Now all of this? It's clicking all of a sudden that its quite plausible i got flagged somehow and ended up with an intentionally degraded service.
So, claude is off table for me these days, and deepseek gets very deep git commit read-throughs with every file even being read being carefully monitored. This obviously ruins the "agentic" promise, but the reality is we cannot trust these fucks (being the companies). The irony is deepseek now queries claude on problems its stuck on for input via openrouter, with mild success (the gap really isnt all that big, if its even there), before escalating to me for input on direction on solving a given problem. So now chinese models get more claude training data, not less. In fact, they get training data on frontier ML stacks and problems. Anthropic did that to themselves.
Half of those don't actually require proxying Claude. Also, Claude has made it apparent time and time again that it does not want people using Claude Code as a "tool" in a workflow. If you want to select a model dynamically based on the prompt difficulty, Anthropic wants people to use the API for this. It was the whole issue Claude had with OpenClaw.
> Also, Claude has made it apparent time and time again that it does not want people using Claude Code as a "tool" in a workflow.
Why would Anthropic get to dictate how someone uses a "tool" (that's literally what Claude Code is... a tool in a workflow)
They're swimming upstream. Trying to maintain a rapidly shrinking moat and not being very creative about it. Making enemies of your users is often a failing strategy.
> Why would Anthropic get to dictate how someone uses a "tool" (that's literally what Claude Code is... a tool in a workflow)
This is a direct conflict in framing. They clearly do not see Claude Code as a "tool in a workflow" but instead as a service that will eventually replace all programmers.
I think the self-evident quality of the various parts of the Claude Code universe is a pretty obvious indicator of the problems with that approach. It is still important to understand a party's thinking if you want to understand their position.
> They're swimming upstream. Trying to maintain a rapidly shrinking moat and not being very creative about it. Making enemies of your users is often a failing strategy.
Time will tell, but I agree that they are indeed in a tough spot. Probably not for the reasons that they think.
I guess I can see why they might nerf detected clients server side, but without evidence I would not assume it. Could also be so that 1) they can identify sus client IPs, 2) do a statistical analysis on distilled models to prove that their system prompts were clearly using unique tokens from Anthropic’s API.
They offer (extremely) discounted Claude prices but you have to go through their gateway. They subsidize part of that, and they get the low price by reselling unused Max capacity, there's been a few posts on that in the past months. People are apparently getting 90% discounts on their claude use this way, tradeoff is that you have two companies learning from your data, instead of just one.
So people use the same tools they use normally, but get it for a lot cheaper
> hysterical. The intent of this steg is excruciatingly clear
Even good goals do not excuse malicious or reckless execution. The ends do not always justify the means.
Whether or not it harmed you this time, it's a violation of trust and autonomy.
Surely you'd be angry if someone secretly installed a rootkit onto your computer, even if--at least for now--it only had code to try to detect and snitch on Public Enemy #1.
What do you see as malicious or reckless here, exactly?
This seems to be a VERY low resolution, functionally anonymous, bit of info, probably related to protecting their IP from bad actors breaking the TOS.
This looks like it's covered in the second bullet point of the "Personal data we automatically receive", that you consented to:
> Usage Information: We collect information about your use of the Services, such as the dates and times of access, browsing history, search, information about the links you click and about third-party applications, services, and content you integrate or interact with, pages you view, and other information about how you use the Services, and technology on the devices you use to access the Services.
What do you see as malicious or reckless here, exactly?
If some other AI company wants a highly compressed collection of everyone else's IP, they can get it by themselves - not from Anthropic pre-compressed =)
I don't want my harness doing sneaky stuff like this. I don't want my harness data mining me. I want my harness to implement the agentic loop and I want it to be transparent.
You may be ok with the harness doing 100 things that are not what I am using it for. But none everyone is, and it’s hardly hysterical. Perhaps you are simply careless.
Is it completely clear to you what the purpose of this is? It isn't completely clear to me but it's very likely it's an issue with me. I feel that it can be part of some larger counteroffensive against certain actors in China in a way that is more than just the signalling here--like maybe there's much more we haven't seen. In any case, it certainly shows their willingness to use less conventional tactics against those they view as adversaries.
oh no, the terms of service how dare people break those. you don't get to claim fair use while CFAAing everyone's actual IP then whine about the tos, and then when called out on spying on users point to it as if it being in the tos somehow justifies it
a lot of other malware has a tos too but we still call it for what it is
basically the point is that it's not protected because it doesn't fall under any classification of IP (it's not copyright or patent since it's mathematical outputs of a mechanical system, it's not trademark because obvious, and it's not a trade secret because it's not a secret)
Are you honestly surprised that roughly 0 HN users read that, or that they are loudly complaining about this, likely without even reading beyond the headline of this post?
> Surely you'd be angry if someone secretly installed a rootkit onto your computer
I surely would. What does that have to do with this scenario.
Note that the SW running on your machine is not doing anything malicious. The service is the thing that behaves in ways you want like - and that service is not running on your device.
There is no comparison with rootkits here. This is the equivalent of Google giving you a CLI to make searches easier, and that tool decides to just Rickroll you randomly. Annoying, yes. A security concern? No.
The article is really quite reasonable and calmly presented, actually.
Your claim re. the intent of the fingerprinting is a guess.
Normal developers are the users that aren't taking steps to avoid being flagged by this system.
The software is written in a deliberately obtuse way, presumably in service of some (unknown to us) goal. This is a deceptive and anti-social thing to do, it is by nature an adversarial stance to adopt. An already adversarial actor may be "punished" by this, but in such a relationship, hostility can be expected. A non-adversarial actor -- a normal developer / user -- is being harmed by this because the software is treating them as an adversary.
Further, lets assume your guess is correct and, in addition, that Anthropic elects to alter/downgrade/poison their service[0] for users that fit a particular pattern of markers.
It's obvious how this system would "punish normal developers" (i.e. not the intended target/victim) that happen to fit those patterns.
[0] to some extent, the service already has been altered as its behaviour depends on the prompt text
Copying over my comment from elsewhere in this post:
Anthopic choosing to delay their models' invevitable distillation by competitors is their prerogative.
That they choose to implement it by fingerprinting my access patterns without first disclosing is where they shit the bed. It isn't "sneaky" it's straight up sneaky (and dishonest and unscrupulous while we're at it). That this particular instance is harmless doesn't give me much comfort. Who's to say they aren't harvesting PII?
That their actions make sense for their business isn't any reason for people to accept their deceitful, customer-hostile decisions.
Are you implying that Anthropic lawyers wrote the user agreement to protect users out of the goodness of their hearts? That's how they receive their big fat paychecks? Very funny.
We all know user agreements exist to strip users of their rights and to absolve companies of wrongdoing. We know that corporate apologists here are also well aware of this fact. When user agreements explicitly grant companies the right to screw over users, apologists are quick to make excuses about how it's all standard operating procedure and accuse people of being uncharitable for doing a plain reading of the text [1]. Yet when people are actually screwed over by companies, those very same people blame users for accepting the user agreement. It's a bad faith system.
User agreements are nothing more than power plays by exploitative companies.
And even if that's true (and it frequently is!), detractors usually miss the underlying and immense impact of "sleeping dad capability" equivalent artificial systems.
Horizontally scaling "sleeping dads" takes decades, but inference capacity for a sleeping dad equivalent model can be scaled instantly, assuming one has the hardware capacity for it. The world isn't really ready for a contraction of skill dissemination going from decades to minutes.
> Not "hidden", but probably more like "no one bothered to look".
Well yeah. There weren't enough "someones" available to look. There are a finite number of qualified individuals with time available to look for bugs in OSS, resulting in a finite amount of bug finding capacity available in the world.
Or at least there was. That's what's changing as these models become competent enough to spot and validate bugs. That finite global capacity to find bugs is now increasing, and actual bugs are starting to be dredged up. This year will be very very interesting if models continue to increase in capability.
I was just thinking about this and what it means for closed source code.
Many people with skin in the game will be spending tokens on hardening OSS bits they use, maybe even part of their build pipelines, but if the code is closed you have to pay for that review yourself, making you rather uncompetitive.
You could say there's no change there, but the number of people who can run a Claude review and the number of people who can actually review a complicated codebase are several orders of magnitude apart.
Will some of them produce bad PRs? Probably. The battle will be to figure out how to filter them at scale.
It's doom and gloom because the underlying game theory forces all state actors into an unbound and irresponsible arms race, consequences be damned.
AI development game theory is extremely similar to the game theory behind nuclear arms development, but worse (nuclear weaponry was born from Human General Intelligence, and is therefore a subset of the potential of AI development). Failing to be the most capable actor could put one in a position of permanent loss of autonomy/agency at the whims of more capable actors.
Unfun thought experiment: In an AGI scenario, what lever of power do the UBI recipients have to force what you've described?
Historically, humanity's biological monopoly on the fundamental resource of "general intelligence" has always been that lever. Looking at the world today, it's pretty clear that democracies are just a temporary balance between the general contempt of the powerful towards commoners, and the fact that the powerful begrudgingly need our economic utility, which is ultimately based on our general intellect. Even callous dictatorships had to exercise partial restraint on violence and murder, to retain a pool of general intellect.
AGI will be a multiple-whammy here:
- most people will become economically worse-than-useless, they will be a total liability on the rich and powerful
- those same people are unlikely to be given access to any levers of power or influence, because they no longer have anything of value to provide to the rich and powerful
- AGI in combination with robotic platforms, after a certain threshold, will permit for insurmountable policing. The "rise up against the robots" cliches we see in film simply become impossible after a certain point.
I desperately hope we end up wielding AI to usher in a post-scarcity utopia, but looking at the types of people who own the world...we'll get what we're allowed to get.
The people in power (whether a government/king, or company) will have the incentive to shrink the population. In the past, the economy grew with the number of people, but now only the costs would grow. There is basically a non-zero chance that our future contains laws limiting who is allowed to have kids or how many. And that's not even the worst action they could take.
Excellent comment. From a pure resource allocation perspective, walking the current path is like looking down the barrel so to speak. I guess our fate is already in the hands of the decision makers. Hopefully theres enough conscience up there.
That said, I would expect that if such a position were taken, it would be accomplished via a mass sterilisation campaign rather than direct extermination. More justifiable amongst other decision makers. Less moral burden.
The lever is that whenever somebody tries to give you a scarcity-only token like USD or BTC you say "that's no good here, come back with something that proves to me that you didn't make it by making life worse for the people around you". We (the masses) still control consensus on what counts as valuable, and if were going to break from scarcity as the basis for that (and we should) then it has to be a proper break.
Play by rules that have something to do with our collective values or we're not going to acknowledge your stores of "value". "I got this from my grandpa who was a successful swindler" needs to stop being a source of money that you can expect the children of the swindled to accept. We need to drop the idea that scarcity implicitly creates meritocracy and instead deal in verifications of merit.
If they're really ready to go it alone in their bunkers with robots for friends then I guess we won't be seeing each other anymore, but I kinda doubt that'll work for them in the long run.
No, that's a protocol that were going to have to build.
I think CirclesUBI gets it partly right, re: different dynamics between parties based on their connectedness on a trust graph. But I think we'll want this on a CRDT, not a blockchain, and I think we'll want more nuance than their treat-others'-tokens-as-equal-in-value-to-your-own approach. But it's a decent start.
I used to think that "AI operating in meatspace" was going to remain a tough problem for a long time, but seeing the dramatic developments in robotics over the last 2 years, it's pretty clear that's not going to be the case.
As the masses fade into permanent unemployment, this will likely coincide with (and be partially caused by) a corresponding proliferation in intelligent humanoid robots.
At a certain point, "turning on them" becomes physically impossible.
What development have happened in robotics over the last 2 years, that changed your mind?
I am familiar with general industry trends in electronics over the last two decades, as well as the 1 decade in machine learning.
And no, I don't think it will end well.
reply