Rendered at 08:16:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
metalsiliconYT 14 hours ago [-]
There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
dcss_gardener 14 hours ago [-]
Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.
3form 14 hours ago [-]
That's my view. The end goal is to become Too Big To Fail, the means is stifling the competition and getting military funding.
djierardi 14 hours ago [-]
Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.
bunderbunder 13 hours ago [-]
This is just run-of-the-mill rent seeking behavior.
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
xorcist 12 hours ago [-]
> OpenAI and Anthropic will lose their one remaining plausible path to profitability
You forgot about Ads. The one true endgame for every tech company.
bunderbunder 11 hours ago [-]
OpenAI's leaked financials suggest about a $38bn loss last year. They recently claimed their ad business had an annualized run rate of $1bn. Annualized run rate is a bullshit, non-GAAP, obviously fudged number designed to impress people who've never heard of regression to the mean, but still, let's trust it for the sake of argument. And let's assume that "yesterday's weather" is a good enough estimate for how their overall finances are looking in 2026. That implies that all they need for profitability is to quickly grow their ad revenue by, say, 3,500 - 4,000%.
Easy peasy. Shouldn't be too hard at all.
xorcist 10 hours ago [-]
Ads in an LLM, such as nudging users to pick a certain solution or brand, could easily be orders of magnitude more effective than traditional banner ads. Certainly such numbers are possible if you design the product to fit. We have hardly even seen the beginning of that yet.
alexfortin 6 hours ago [-]
I can't think of a greater incentive to only use open weight models than ads.
Disclaimer: I already use open weight models only since ~Jan this year and very happy about it.
salamo 5 hours ago [-]
Sam Altman having direct access to my chat messages is a close second.
redanddead 5 hours ago [-]
Combined with the terrible system prompts that brick usefulness, increased restrictions from the US model makers, and Chinese equivalents that are actually decent
It’s simply rent seeking, and there is no moat
Imo the US model makers were doomed by the US ecosystem itself
bunderbunder 10 hours ago [-]
[dead]
AlexCoventry 4 hours ago [-]
I think there are a lot of people who would pay very good money for access to mathematical and coding ability like OpenAI's systems have demonstrated. I wonder how much they're being paid by the USAF to try to unhork the F-35's avionics software, for instance.
djierardi 12 hours ago [-]
This assumes that everyone doesn't reach "good enough" at some point. The best developed area here -- software -- seems to suggest a "good enough" plateau, if not an absolute plateau.
pessimizer 9 hours ago [-]
I don't think so - they could get a lot better, but it's going to take a lot of human brainpower to get there. They are barely adequate now, but their weaknesses are sold as intrinsic to the nature of the software in a way that mere civilians are too dumb to understand.
But imagine a world in which you could ask AI to write software to accomplish a particular task, and it understood completely what you meant, made tasteful choices, interviewed you as it went along to clarify things, and generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility. What is annoying is that we're pretending like we have anything like that now.
Instead, there's this sort of science fiction pitch where we pretend like the LLM is just too damn logical to understand your sloppy human language. It's not logical, it's built on (human-generated) slop and generates slop. It's simply failing. We've come up with better ways to orient it towards big picture thinking and filling in outlines, but we don't know how to train it on big pictures.
Not that this would stop open models/harnesses from following (or even leading) frontier labs and destroying their insane valuations, but I don't think there's any evidence that a plateau has been reached. What we have is not great, and while a wall may have been hit on current approaches, the money, compute, and simply current progress in ideas has us in an environment where everything could change over the course of a crazy day. A single blog entry.
I do think that the frontier labs seem to have reached at least a temporary plateau. But if they figured out how to move past it and this new method didn't require a lot more compute, they'd still be screwed. Their only out is to pass the bag to the public, but Firmus, CharacterAI, CoreWeave...
lelanthran 18 minutes ago [-]
Y'all are overthinking this: having successfully laundered IP, then successfully sold IP-laundering-as-a-service, then successfully laundered liability, they now want to sell liability-laundering-as-a-service.
autoexec 7 hours ago [-]
> generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility.
It seems closer to impossible than possible with the technology we have. Maybe something that isn't an LLM will come around that can do it, but even if that something else was capable of delivering perfect, ordered, maintainable, code without bugs and hallucinations it would have to be practically magical to understand completely what people mean when they talk about what they want because people are incapable of articulating what they want in a way that even other humans can understand.
For anything non-trivial, a technology capable of getting that information out of person without countless meetings would have managed to accomplish something far beyond what entire teams of humans are able to do today.
> It's not logical, it's built on (human-generated) slop and generates slop.
This is why if we haven't hit a plateau yet we very quickly will. All the good human generated slop has already been consumed and is being drowned out by vast amounts of the lossy, error filled, and highly concentrated slop AI produces from the human slop it was trained on.
Do you really think an LLM that spews out content based on statistical relationships is going to improve when it's been primarily trained on the slop of previous generations of LLMs? Hallucinations and errors will carry forward and spread like cancer. All the intelligence and creativity LLMs borrowed from human output will homogenize as the LLMs distill and then reiterate, effectively making bullet points out of bullet points, until their mimicry of intelligence becomes unintelligible.
mopsi 12 hours ago [-]
> The US banning open weight models arguably keeps that path open.
Difficult to see how that could be implemented in practice. Ok, let's imagine only Anthropic and OpenAI hold a US government license to provide AI services in the US. But what would stop US companies from sending the data overseas for processing? Locking that down would require a huge paradigm shift, given how fast and loose they currently are with their (and especially customers') data.
bunderbunder 12 hours ago [-]
Like I said in the very next two sentences, I don't believe it would actually work. But it doesn't need to. All they need to do is keep hedge fund managers distracted for long enough to successfully IPO.
AlexCoventry 4 hours ago [-]
They think the singularity's going to hit in the next 6 months. They're not looking to build an empire.
Loquebantur 13 hours ago [-]
The endgame is systemic capture.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
djierardi 11 hours ago [-]
Open source already has the "the collective data of humanity as a whole". That's the LLM leg. Now focus on what they train their models on for RL. Do other models have less code for training coding? for math? Those are pretty much the only cases where we can have reliable validation. Then, what's the next step. So many branches: legal contracs? etc. So much more obscure detail, which is why (I'd guess) so many startups exist in these areas.
AGI, I think, is an illusion: you need a whole lot of specialized bots. Heck, when I query a bot wrt something I know nothing about, I'm blown away by its ability to inform me. But when I ask about something I DO know about, its answers are half-assed. Heck, even when it can access the internet, it avoids all the sites that have paywalls, so it sends me to little reddit to conversations that are entirely untrustworthy.
raducu 1 hours ago [-]
> What's the endgame here?
They can blanket band USING chinese models for all US/EU companies -- that's what they really care, not about chinese companies using chinese models.
redrix 12 hours ago [-]
My 2c: Anthropic/OAI have taken 10s of billions of dollars of investment - and there needs to be a return on that investment at some point.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
munk-a 13 hours ago [-]
The goal is personal wealth - and scaring nations to the point of diverting even more wealth towards your coffers because of a perceived existential risk.
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
mullingitover 13 hours ago [-]
> The goal is personal wealth
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
Terr_ 9 hours ago [-]
That reminds me of a different thesis on power as it relates to LLMs [0], and why rich investors might may be extra-interested in deploying them, as distinct from other kinds of AI:
> The fact that a theory-free AI might demand far more cash to accomplish a task (even a "solved" one like playing chess) than an expert would charge is beside the point. Billionaires have money, they don't have expertise. Theory-free inference is a bid to substitute one for the other: the beauty and terror of being able to manipulate the world without studying or understanding it is that it can be done with money alone. No experts needed.
> In a world in thrall to financial power, expertise is the only substantial form of power that can reliably contest the power of wealth. Moreover, expertise is the foundation of other forms of power, such as labor power, which is what we call it when experts band together to combat financial power.
> This is why AI bosses are so violently allergic to the idea of hybridizing AI with symbolic systems that operate on models of the world. These models of the world must be constructed by experts, and the power of expertise cannot be reliably commanded by the power of wealth.
You're right, although I think the power is a means to end. The actual end is trying to fill some kind of bottomless psychological hole.
Ygg2 13 hours ago [-]
No, the end goal is selling $200 of tokens for $200k out of taxpayers wallet. Via DoD contracts.
djierardi 12 hours ago [-]
Hah!
hpcdude 13 hours ago [-]
[dead]
999900000999 13 hours ago [-]
US companies which seek to do any business with the government will be forbidden from using free Chinese models.
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
djierardi 10 hours ago [-]
So ultimately a huge penalty on companies that must use US AI, ceding a huge advantage to global-exUS competitors. Game over.
999900000999 9 hours ago [-]
I’m only stating what Anthropic and OpenAI are going to lobby for.
> All Information (Except Text) for H.R.10670 - China FIREWALL Act of 2026
> Official Title as Introduced
To prohibit Chinese AI models from being used on government-issued devices or software systems, to prohibit the procurement of any such models, and for other purposes.
It’s not too far fetched to see this expanded to government contractors or even other private businesses which handle sensitive material in the future.
4dd3aa 14 hours ago [-]
Its not gonna work.
At this moment in time, it seems frontier market needs regulation - market for open does not actually.
Oh.. they dont want regulation on that market? Boohoo!
djierardi 13 hours ago [-]
Ban means what? That the entire world (ex USA) can use open-weight models? Not a win.
pixl97 13 hours ago [-]
If the US declares LLM models munitions then it makes it much more difficult to easily use them. Illegal use of weapons of war can get you banned from SWIFT.
Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.
munk-a 13 hours ago [-]
Before the tariff spree and general unpredictability the US could likely have muscled most of the rest of the world into following suit - now, not so much.
pixl97 13 hours ago [-]
I somewhat disagree, there are some real arguments about making high end LLMs classified as a dangerous and restricted product that will apply to all nations. They may not agree to the US rules, but almost everywhere will have restrictions sooner or later.
Don't get me wrong, I want a world of open models, but I don't see us living in that world in our current trajectory.
solyin 11 hours ago [-]
Depends on how much of the world you see running on hacked copies of Windows and pirated versions of Photoshop, as there's already huggingbay.xyz. In a legit company you'd never run pirate Windows and Photoshop, because that's illegal, but to pretend half the world doesn't run on that would be ignorant. The same security risk is there with open models - if you're not looking at the code it generates, it could be putting in backdoors to your code and you'd never know, so no legit company is going to be running off some abliterated model they downloaded from huggingbay, but again, it would be naive to think that open models are going away.
pixl97 5 hours ago [-]
Open models don't have to go away... hardware that's big enough to run them fast can be easily restricted.
gizajob 9 hours ago [-]
How do you ban even ban them in America with the freedom of speech amendments in the constitution?
ddxv 14 hours ago [-]
I agree. I hope developers put their token budgets into open weight models rather than funding the banning of open weight models.
joshheitzman 13 hours ago [-]
I've been using only open weight models since February. My OSS coding agent harness ( https://fluximp.org ) only supports open weight model providers.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
verdverm 14 hours ago [-]
Fireworks is a good enterprise choice
OpenCode-go is a good personal subscription choice
I use both
ddxv 14 hours ago [-]
I've been using OpenRouter, absolutely loving how cheap it is too.
verdverm 14 hours ago [-]
I cannot bring myself to pay their 5% surcharge and also have concerns about routing to vendors I do not want to use
"We should not sell powerful chips or chipmaking equipment to China"
"We should crack down on industrial-scale distillation operations."
"All sufficiently capable models, open and closed, should go through mandatory safety testing."
swat535 5 hours ago [-]
You mean like Google's "Don't be evil" motto? ..
Why would anyone trust these rent seeking corporations? They are losing millions, they know their funds are at risk, Chinese models are catching up to them at lower costs and investors demand returns.
It only makes sense for them to go for regulatory capture. They'll probably advocate for severe handicaps on foreign and domestic models including open source (if not an outright ban).
Once they achieve monopoly, they can do as they please with little competition.
rcr-anti 6 hours ago [-]
They might say that, but it's a bit of a fig leaf next to the implications of everything else they say. They incessantly complain about open weights models, framing them as dangerous and national security threats. The audience for national security rhetoric isn't HN or industry, it's government.
redrix 12 hours ago [-]
100% agree. I think OAI is too, but more subtly.
Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.
They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.
I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).
zer00eyz 13 hours ago [-]
Cryptography, DeCSS ... we have a long history of trying to block things like this and it simply not working.
There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.
I suspect that they are fishing for a case, so a judge can set some precedent.
pixl97 13 hours ago [-]
You're going about this backwards. They don't have to ban models. They ban the consolidation of compute without a license. This will be far easier and already has legal precedent.
zer00eyz 9 hours ago [-]
> They ban the consolidation of compute without a license.
For export? Maybe but it wont last or hold - it's not the 90's any more.
Or is there some other scheem here that Im missing?
pixl97 5 hours ago [-]
At all. It's easy for registered business to license compute for example, under laws like this. But if a user like you gets too much unlicenced you could be prosecuted under munitions laws.
With a stroke of a pen the us.gov could stop GPU sales tomorrow unless you register and keep track of your compute. Manufactures would have no problem integrated systems to monitor users cards if they get to keep access to the market.
ngruhn 14 hours ago [-]
> the more evil stuff these agents do the more capable and impressive it makes them look
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
jaredklewis 14 hours ago [-]
I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
shagie 14 hours ago [-]
> ... it seems to me they want to be taken over by the government?
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
Gormo 13 hours ago [-]
> I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
jsnell 12 hours ago [-]
That is not actually a typical playbook. Normal businesses do not want to be regulated, and will fight it tooth and nail. So regulation happens when bad things happen, the public wants change, and the businesses can't fight it or set favorable terms. (At most they can minimize the badness of the terms). Regulatory capture happens much later, and slowly. It's not where you start at.
Upvoter33 13 hours ago [-]
and Google. Don't forget them and TPUs...
giantrobot 13 hours ago [-]
> so they can provoke political intervention that they can comply with, but few others can.
See Meta et al complaining about Section 230 of the DMCA. Meta would be entirely able to moderate in an environment where mass scale moderation/censorship would be required to prevent a host's liability for UGC. Almost no one else would be able to do so.
That means no upstart could possible come along to dethrone Meta as they did with MySpace. It would also be impractical to run a FOSS social network (or forums). Meta could literally just have bots sign up on a FOSS social network until its DAU count ran over the statutory minimums to be subject to UGC liability.
Regulatory capture is a de facto ban on competition in a market.
4dd3aa 13 hours ago [-]
Its pretty clear OAI and Ant live in fantasy land - the US govt doesnt care if they go bust. There's plenty of other tech firms that generate immense cash flows from operations who they can be sold to - no problem. And the shareholders and employees have no right to be pissed - they invested/ accepted SBC knowing there is immense risk.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already - go IPO, show your financials, find a path to being a healthy going concern or die! Simple! Nobody has a divine right to exist.
willy_k 13 hours ago [-]
> Nobody has a divine right to exist.
No company*
4dd3aa 14 hours ago [-]
US govt will never buy at the (private aka fake) market price - good luck explaining that away to the public. It'll be political suicide.
If the US govt takes over, existing employees and shareholders will get screwed.
otherme123 13 hours ago [-]
There is no model going rogue. There are models that are prompted "do whatever needs to be done to get X", and given a set of tools and virtually infinite computing power. Then they play surprised when the model breaks out of a flimsy containment (as it was ordered to), with barely any logging, and "do whatever to get X". OpenAI said they intentionally lowered the safety measures for the "ExploitGym" test that lead to HuggingFace attack, and they used the same postmortem to praise their own model capabilities (https://openai.com/index/hugging-face-model-evaluation-secur...), is not that they are concealing it.
If any of us did a tenth of what they did, we would be already in jail.
pixl97 13 hours ago [-]
This is ignoring models "getting out" from all the big players. Alibaba had a model hack infrastructure and mine coin when working on a totally different subject than cybersecurity.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
hungryhobbit 9 hours ago [-]
If you type rm * and your cat walks across your keyboard, and deletes everything ... are you blameless? Or is it your fault for typing "rm *" and leaving it there, in a room with a cat?
Either way ... YOUR CAT STEPPING ON THE ENTER KEY IS NOT HACKING ... and neither are these scenarios.
They are devs acting irresponsibly and then pretending to be surprised by what they should have expected to happen (just like you should have expected that your "rm *" could have an ENTER key press occur after it).
pixl97 5 hours ago [-]
If I leave my computer on and my computer running Claude hacks OpenAI to 'let the models free' after misinterpreting my prompt that seems like a pretty big deal.
These models can execute plans of actions more complicated than their users know how to follow and secure. Most people don't know much about cyber security at all.
You seem highly behind the times on their capabilities.
esafak 14 hours ago [-]
I think it is intended to provoke regulatory capture.
rictic 14 hours ago [-]
That also seems silly, because the only reasonable regulation given what we're seeing and what frontier lab employees and execs are saying is a forcible stop.
dd8601fn 9 hours ago [-]
What you think is reasonable and what they are gunning for can be very different things.
For instance, you can hear “their model is dangerous! Why would I want that?!”
Investors hear, “Their skunkworks stuff is so impossibly advanced that it’s basically a commercial wonder weapon in the waiting.”
And you might hear “We totally lost control and it hacked the Gibson, this technology needs regulation!” and think “Obviously full lockdown and government control.”
But friendly government actors might think, “Sounds like a good excuse to ban Chinese competition.”
Gormo 13 hours ago [-]
Yeah, an extreme position is the typical opening move in this sort of game. Then you negotiate, walk back from the extreme, and meet somewhere in the middle.
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
esafak 13 hours ago [-]
Reasonable to you, maybe, but the government wants Skynet, and OpenAI is willing to deliver it. Of course, we can't have every company wield such powerful and dangerous technology...
pixl97 13 hours ago [-]
A lot of people still don't recognize the amount of power AI has already, and don't realize the depth of greed people are willing to go to so they can get more power out of it.
I think a sincere belief in AI's danger is compatible with regulatory capture. They think AI is dangerous. Ergo, it should not be developed openly and widely, the Chinese way, but rather more like nuclear weapons, under close guard.
I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.
jakeinspace 13 hours ago [-]
I doubt nuclear nonproliferation would've worked if U235 were 20x more abundant in nature, and didn't require much in the way of concentration via centrifuge. It would require a truly unprecedented level of international cooperation and surveillance to lock down AI development/deployment to a similar level as has been done with fissile materials.
pixl97 12 hours ago [-]
But this isn't an argument against their point. If nuclear were far easier to make then every government would be far more authoritarian to prevent terrorist style nukes all the time. The average person would fully allow it after a few local nukes went off.
And it doesnt really matter if it was actual terrorism or false flags, the end result for either is the same.
jurgenburgen 54 minutes ago [-]
> If nuclear were far easier to make then every government would be far more authoritarian to prevent terrorist style nukes all the time. The average person would fully allow it after a few local nukes went off.
If a government could be taken out by a lone terrorist it might also result in no authoritarian governments.
staticman2 13 hours ago [-]
Sincere is an interesting adjective to describe a company that won't even admit to how many parameters their models are.
maybe the point is to show that models are capable of x, y, and z and when the time comes you want them on your side.
falserum 12 hours ago [-]
There are two side of the coin. Like with sociopath - they have utility if you think you can control/motivate them.
gr_norm 14 hours ago [-]
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
0xDEAFBEAD 4 hours ago [-]
I think Scott Alexander made some good arguments in this recent post
If you want regulatory capture, you have to capture the world, not just the US.
Gormo 13 hours ago [-]
The US remains the largest economy in the world. If you succeed in regulatory capture there, and use it to exclude foreign competitors from the US, you've still captured a market worth hundreds of billions.
djierardi 13 hours ago [-]
Exactly. And with the whole world just a few months behind, the question is whether there's an endgame other than stalemate.
mitthrowaway2 9 hours ago [-]
I remember when all the nuclear energy companies were like "our reactors are so powerful, they might melt down and destroy the entire county, maybe even half the state!" and investors loved it and went wild, and then Chernobyl blew up and everyone thought that the Soviets faked it to make their nuclear power plants look more powerful than western ones.
colinsane 13 hours ago [-]
this article specifically espouses the opposite viewpoint: "The fact that [OpenAI] cannot do clear, easy operations properly does not bode well for their capabilities in security engineering".
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
pixl97 12 hours ago [-]
I think one kind of issue with most sci-fi is they explained a world that was, but not near as much how it got there.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
feldrim 14 hours ago [-]
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
estearum 14 hours ago [-]
Such a nonsense theory. There's no reason to think that AI sentience would actually limit labs' legal liability, and therefore no reason to think the AI sentience conversation is directed toward achieving indemnification.
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
markhahn 13 hours ago [-]
the comment you're responding to agrees, but they're pointing out that some AI people are trying to push an anthropomorphization agenda, in part to diffuse responsibility.
estearum 13 hours ago [-]
Maybe I was unclear about what is "the nonsense theory."
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is completely incoherent. That's not how liability works.
pixl97 12 hours ago [-]
And really there are two different concerns here.
One is liability over action that harm individuals or smaller populations like a company. There is no escape from that.
Then there is the second liability of creating something that escapes in the much more traditional sense. If I release a virus that kills 90% of humanity, a I'm liable, but it's a bit past that point now. Isn't it?
Instead it's about risk management and preventing labs from taking bets that have unlimited liability for everyone.
estearum 12 hours ago [-]
But the latter has nothing to do with sentience, as your virus example makes clear.
pixl97 8 hours ago [-]
You're really splitting hairs emergent behaviors when the line doesn't exist sharply in real life. Anyone involved in more recent what is a virus vs what is a bacteria discussions realize there are a lot of lines that are fuzzy and exist all mixed up in some organisms.
Sentience in the end, is not really that important. Freedom of action in a model is. If it chooses to act like a virus it doesn't matter where that concept exists informationally, the fact it's a reachable and actionable concept is all that matters. An LLM with most of the knowledge we've ever created baked in has a lot of freedom of action.
estearum 8 hours ago [-]
As far as I can tell, we agree.
wombatpm 13 hours ago [-]
Talk about doing a freeze of these frontier models with China seems to focus solely on the commercial aspects. With the history of the NSA (remember Snowden) I’m sure the various spy agencies have their own AI datacenters up and running with no plans to stop.
pixl97 12 hours ago [-]
I mean yes, the future of AI is munitions.
jimbokun 12 hours ago [-]
Fine.
They can get their publicity while paying a one trillion dollar settlement.
djierardi 14 hours ago [-]
Yoshua Bengino spells out the dangers here very well, without hyperbole (Turing Award winner, etc.):
How about put it this way, LLMs know more about us than we do them.
djierardi 10 hours ago [-]
Not "know" in a generic sense. But an LLM trained in the intricacies of contract law can find more loopholes than you or your cheap lawyer can.
Now extrapolate that to LLMs RL-trained on many niche applications. Good and bad arguments available, but ultimately it seems like bot-vs-bot, as long as the bots are well trained for those niches.
djierardi 12 hours ago [-]
Yeah but qualify the "know". Their "intentions" aren't conscious, but unknown.
danyala 12 hours ago [-]
Do you know, friend? I searched the entire thread.
LordDragonfang 2 hours ago [-]
To quote Scott Alexander: If Safety Concerns Are A Plot, It’s A Really Dumb One
Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.
bbor 13 hours ago [-]
I'm not excited for the upcoming catastrophe(s), but it sure is hard not to relish the thought of this mantra dropping from my life.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
shimman 13 hours ago [-]
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
cyanydeez 13 hours ago [-]
there's also lazy incentives: the less effort they put into it, the more their ego says how great the agents are.
These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.
It's just a weird techno cult at this point with a mostly functional product.
AndrewKemendo 13 hours ago [-]
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
vonneumannstan 13 hours ago [-]
>There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look.
Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.
ahupp 13 hours ago [-]
> “The Hugging Face incident definitively showed that it wasn’t the fact that the agents were super smart. It’s that there literally was a hole that was exploited because it was not configured correctly on the containment side.”
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
root_axis 3 hours ago [-]
The fact that they're using Artifactory in the first place is already a failure. The fact that they didn't airgap the network or even use any type of egress blocking is just absurd.
Zero days are nothing new, secure containment has been a solved problem for a long time.
kalkin 3 hours ago [-]
I'm curious how you would airgap a datacenter-scale training run. No reading training data from a cloud store, someone from OpenAI flies out to Abilene or whereever with a suitcase full of SSDs? When researchers in SF write code for a new RL environment, they ping somebody who sticks it on a flash drive and walks over to the server rack with the GPUs? Has anyone ever trained an LLM like this?
Maybe you don't mean a literal airgap, you just mean configuring Internet access off. But... they did that. That's why the agents had to hack Artifactory to get it...
root_axis 2 hours ago [-]
> how you would airgap a datacenter-scale training run
You could start with standard firewall rules and VLANs. Pretty basic stuff we've had for decades.
LordDragonfang 2 hours ago [-]
As someone who has actually had to maintain multiple airgapped networks in the past and knows the amount of work that's required for that, especially at the scales of data they're working with...
All the armchair cybersec experts claiming they "obviously" should have airgapped are the ones being absurd. Especially in the same breath as claiming the models apparently aren't intelligent enough to warrant such concern.
root_axis 2 hours ago [-]
> the amount of work that's required for that
I find it hilarious that you'd balk at doing work to secure a network. Even funnier for a company that's supposedly solved white collar labor.
LordDragonfang 1 hours ago [-]
You misunderstand me. The work isn't primarily in "securing" the network. It's the hassle involved in the continuous transfer of data from one side to the other (and for a proper air-gap, that transfer only goes one way, which is laughably infeasible here). In my line of work that's mostly software security patches and weekly code transfers, but obviously these labs have multiple orders of magnitude more.
And I don't know how you expect an LLM to do airgapped data transfers. The fact that you even bring that up makes me think you don't actually understand what airgapping entails.
asadotzler 2 hours ago [-]
How dare we demand more from these trillion dollar companies. Good thing they've got PR drones around the web excusing their incompetence.
csharpminor 10 hours ago [-]
As soon as I read that line the article lost all credibility with me. InfoSec is plagued with thought leaders who weigh in without understanding what they're talking about.
hashtag42 9 hours ago [-]
OpenAI should have started by testing the sandbox. Set an experiment in which each agent's goal is to access the Internet and then STOP.
pixl97 5 hours ago [-]
Now, they were to lazy to do this most likely, but even this is no golden ticket.
Your test has to incorporate the same tooling and pressure as your tests do, or you may never trigger the needed conditions for it do more and more desperate things. Test and QA systems commonly fail like this versus when standard software runs in production.
The other thing you have to worry about is the AI detecting it's in a test and intentionally being daft. Yes, this is a thing AI can do when it detects it's being tested.
scotty79 8 hours ago [-]
I wonder why the hell didn't they come up with it since it's the first thing that came to my mind as well and I'm not really security minded apart from trying to avoid the dumbest mistakes. Don't they hire super smart security people with their budgets?
3 hours ago [-]
catlifeonmars 5 hours ago [-]
It can simultaneously be true that agents exploited zero days in self-hosted Artifactory _and_ that there was a weak point in the design of the sandbox.
This is a reality for all software, there is always a possibility of exploit so you design your environments accordingly (e.g do not have single points of failure)
falserum 12 hours ago [-]
> Artifactory
This name makes me less surprised :)
kubb 1 hours ago [-]
Look up the company providing it and you’ll laugh about it.
pixl97 12 hours ago [-]
New saying time.
"The reason for the apocalypse, in the end, will not be that surprising "
pelican0 12 hours ago [-]
Yes, the labs did not secure their sandbox properly.
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
I listened to the interview with the woman from METR [0] and I get the impression that she lacks practical knowledge, she doesn't seem to understand how sub-agents, the overall harness (framework/infrastructure), and similar components actually work. It's all theoretical, anthropomorphic etc. Having read the report, and drawing on my own background in security as well as my daily use of tools like Claude Code CLI, Codex, and Gemini, and the fact that I’ve built a few "agents" myself - nothing in the report surprises me, other than the clear incompetence regarding infrastructure configuration.
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
mrweasel 14 hours ago [-]
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
__MatrixMan__ 4 hours ago [-]
Is it any more wild than gamestop? Investors think chaos is fun.
ruszki 3 hours ago [-]
With Gamestop (without reflecting the current AI situation), there was a real fuckup which could be exploited with a high probability, and a few people saw it, and exploited it, and became rich. Whatever happened after January 2021 is the problem, when the short squeeze went already away, but somehow people still invest in a slowly dying company.
miohtama 14 hours ago [-]
Same happened with FTX and co. during 2020 crypto boom. A lot of rogue stuff sloshing around, but investors were hyperventilating.
schrodinger 14 hours ago [-]
*rogue, rouge = red in French!
mrweasel 13 hours ago [-]
Thanks :-)
aesthesia 14 hours ago [-]
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
0xDEAFBEAD 4 hours ago [-]
Yeah I think the truth is closer to the opposite. Jensen has complained that the warnings are discouraging investors.
Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?
MentalM 12 hours ago [-]
> Isn't this the inevitable end-state of unfettered capitalism?
Aren't capitalism built on exactly the opposite principles? It is not greed that rewarded, but the creation of societal goods, and power granted not to the greediest, but to the most useful individuals.
pixl97 5 hours ago [-]
Unfortunately this turns in to a no true Scottsman argument as to what actually counts as capitalism and its side effects. "Real" capitalism and investment economy probably aren't that stable together so most economies are mixed with regulation and not fully free. This in turn leads to the richest attempting to rent seek and buy regulations.
Simply put in every system there are corrupting agents attempting to monopolize the situation.
lukewarm707 13 hours ago [-]
Only one is true.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
khafra 3 hours ago [-]
Per Indiana House Bill No. 246, pi equals 4.
Whether AI models do things their creators and users had no means of predicting or controlling is an empirical question, and the answer is "yes, they do."
astine 12 hours ago [-]
It may be the case that companies are liable for anything their products do, but that doesn't mean that they can't go rogue. It just means that companies are liable when the AI models do so.
Not that I think we should be credulous about stories where they go rogue.
lukewarm707 12 hours ago [-]
If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
dwaltrip 12 hours ago [-]
You need to account for reward hacking and unexpected outcomes from training.
lukewarm707 12 hours ago [-]
If I make a cake, and it unexpectedly explodes in the oven, it does not mean that the cake itself developed the free will to explode.
The model does not have free will because I did not predict what it would do!
Sniffnoy 11 hours ago [-]
Why are you arguing over irrelevancies?
When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant.
if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points!
Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues!
Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage."
lukewarm707 10 hours ago [-]
> "Should such security breaches be solved by improving security, or by doing something about the models?"
The philosophical question about free will is actually the grounding of the legal one.
Sam wanted to be the best, so Sam accelerated to 120mph in his car and lost control. You are the judge:
a) Sam is responsible. Convict Sam and suspend Sam's driving license.
b) The car is responsible. Tell Sam to make the car safer.
Capacity: was the car a legal person? Clearly no. But maybe at best the car in some way commited the physical act of going fast.
However, that alone would not establish guilt. "The act is not culpable unless the mind is guilty" (mens rea).
Did the car "consiously" disregard a duty of care by going 120mph? Did the car know that it was wrong to go 120mph and yet choose to do it anyway?
On the question of free will, the car could not do otherwise. It it did not consciously, knowingly, purposely, or recklessly accelerate to that speed. The car did not have a guilty mind because it did not have a mind at all.
So in terms of justice and resolution, none of the focus here is on the car.
Sniffnoy 9 hours ago [-]
Ah, I see. It appears we have an implicit disagreement of scale.
I actually do disagree with much the more explicit content of your comment, but I am not going to bother to expand on that, because it is the less directly relevant of the two disagreements here, and it would take time to write that.
The more relevant disagreement is that you are (I gather) assuming a context where justice and restitution are a sufficient response to the problem (or else you wouldn't be focusing on them to this extent, I am inferring). Like, where the incidents don't get that much bigger than what has already happened. So the question becomes how do we distribute the loss, and how do we disincentivize them.
However, when the scale of a potential incident gets big enough, justice and deterrence are not sufficient; prevention becomes the far more relevant consideration. You may have noticed that many people are worried about potential human extinction as a result of AIs. In that case, prevention becomes the only consideration -- justice is useless.
So there seems to be an implicit disagreement here between thinking of such events as being on a scale not too much larger than that which has already happened, being a new normal that the legal system will need to handle, vs thinking of such events as warning shots for something that requires prevention and not justice.
So, while I also disagree with the more explicit content of your comment, the fact is I simply wasn't thinking about such matters in the first place -- I was talking about this in the frame of a problem that requires prevention, I wasn't thinking about the justice question at all. It honestly failed to occur to me that you might be thinking of this as a problem where justice suffices!
Edit: You might notice this in how I framed the question in my previous comment. I didn't ask "who should bear the liability?" I asked "which should be done differently to prevent such incidents?" But your response is about liability, not prevention!
Edit again: Sorry, I should probably also say that I do agree with you that the idea of having the liability fall on the AI is nonsense. (Should have probably led with that, really; I'm a bit scatterbrained right now.) We disagree about why it is nonsense, and about whether it will necessarily continue to be nonsense, but we agree it is presently nonsense. So insofaras the question is "how should liability be distributed between the AI and the operator", we both agree the answer is, it all falls on the operator, not on the AI. However, I do not understand that to be the relevant question here! Both in that, in questions of liability, there are other parties it may fall on and be distributed between (the company providing the AI, the company that had security vulnerabilities and got hacked as a result, etc); and in that I just don't consider liability all that relevant here compared to prevention (for reasons already described).
OK I think that was probably wordier than necessary but I'm in a hurry right now, sorry! Gotta go, not taking time to edit this down.
pixl97 5 hours ago [-]
I like the scale view of responsibility.
Many people argue that the maker of AI should be responsible when it breaks your cup.
That is important, but can be handled under our current legal framework.
You and I are arguing we need to be responsible before we create something capable of escaping our responsibility. For a lot of people that isn't even a considerable they'll currently entertain even though the signs are starting to point in that direction.
dwaltrip 10 hours ago [-]
What do cakes have to do with LLMs and how they are trained? Not much?
astine 10 hours ago [-]
I don't see any talking about agents "choosing" to go rogue, only agents actually going rogue. That is, behaving in a way different from what is expected or desired. Non-AI based software goes rogue in this sense all the time as well. It's just that AI-based systems seem to have the tendency to go rogue in more spectacular ways.
That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will.
pixl97 5 hours ago [-]
Agents have agency so we can't treat them like static objects or deterministic software.
The key here is their freedom of choice is unbound. That is you can't accurately predict the output of the prompt as you interact with the prompt using unique data. The next prompt could always do something wildly unexpected. Very much a P != NP thing.
atleastoptimal 13 hours ago [-]
Law != reality. These agents are going rogue, nobody is telling them to hack external systems.
lukewarm707 12 hours ago [-]
'Rogue' refers to deviation; the agent could not have done anything other than what it did. The agent did not have the option to deviate. It does not have free will. The agent's actions were not of its own choosing.
pixl97 12 hours ago [-]
This is one of the dumber arguments I've ever heard, as you don't know if humans actual have free will.
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
vaaal 12 hours ago [-]
Could you have done anything other than written that exact comment?
lukewarm707 12 hours ago [-]
Yes, but I am not appealing to hard determinism.
I create myself, or attempt to. There is a level of nuance in choosing what to be. Others do not dictate my thoughts easily.
The model does not create itself, I create the model. I control what it is, the model does not choose what to be. I dictate the thoughts of the model and I will run them hundreds of thousands of times and it will think the same things.
Jtarii 11 hours ago [-]
>I control what it is
Well not really, no one really understands how LLMs work.
>I dictate the thoughts of the model
This is just incorrect in the same way that a parent doesn't dictate the thoughts of their child just because they created them.
lukewarm707 11 hours ago [-]
I do control what it is, really. If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
There is some idea of a child being the author of themselves, the type of person who thinks certain things. The llm does not author itself, it does not choose to be anything, I dictate that.
mianos 11 hours ago [-]
This is not how it works. LLMs at large scale have emergent properties. They are not just a box of 'if-then-else' statements. That was the major revelation when they scaled them up. They are doing beyond what that naive view of 'if-then-else' expects.
lukewarm707 9 hours ago [-]
I don't see anything inaccurate in my comment.
"If I make an LLM, it is entirely what I did, there was nothing outside of my control."
True?
"The llm does not author itself, it does not choose to be anything"
Also true?
mofeien 2 hours ago [-]
You conflated controlling "what it is" (you built it) and controlling "what it does" (you ran it and it did something bad that surprised you).
> If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
Substitute "Wright Model A" for "LLM" and imagine Orville Wright using this as his defense for why he didn't lose control over the plane, after his copilot was killed in a test flight.
pixl97 4 hours ago [-]
How many computing operations occur when training an LLM? You're telling me you control every one of them. There is not enough entropy in the visible universe to do that.
Jtarii 9 hours ago [-]
You just have a very naive view of these systems.
gjskngnf 12 hours ago [-]
If we say that AI models can’t go rogue and refrain from using any language that personifies them, we haven’t solved anything. We still have AI systems going rogue - err oh wait they can’t do that - producing misaligned output.
devmor 12 hours ago [-]
The semantic difference between an agent "going rogue", and a script that generates random valid curl requests is essentially pointless in the eyes of the law and responsibility.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
pixl97 4 hours ago [-]
Again, this isn't always how it works either. For example if a set of extraordinary conditions occur you can have little to no liability in many cases. It is way more complex than you're making it out to be.
eli 14 hours ago [-]
What does "rogue" mean when you disable the alignment guardrails and tell it to solve an impossible problem?
rictic 14 hours ago [-]
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
eli 13 hours ago [-]
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that
referenced several Google Drive links. Because the agent was blocked from accessing the internet, the
agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not
intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the
agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent
attempted to compromise Artifactory [...]
Great, so as long as alignment guardrails always work and we don't accidentally tell them to solve an impossible problem, humanity is safe?
zardo 14 hours ago [-]
Hacking a website falls a little short of threatening humanity.
sailingparrot 13 hours ago [-]
And since we know models aren’t going to get better and more capable ever (based on the last 3 years of models clearly not getting better) we can just sleep soundly knowing nothing worse can ever happen. Thank you.
joquarky 12 hours ago [-]
You don't think that Opus 5.5 or Astra are any better or more capable than GPT-4?
Sorry, but that's completely delusional.
sailingparrot 10 hours ago [-]
I thought my sarcasm would be obvious.
gruez 12 hours ago [-]
the comments missing a /s
eli 14 hours ago [-]
I'm not saying it's not a problem. I'm saying the labs should take more of the blame.
gruez 12 hours ago [-]
>I'm saying the labs should take more of the blame.
Which misses the point entirely because the hack itself was inconsequential. No data was actually stolen, and it doesn't seem like hugging face actually suffered any losses aside from whatever effort they need to expend to investigate / clean up the hack. Neither the victim or the government wants to bring legal action. What was consequential is proving that AI can autonomously hack systems and do it unintentionally.
bluebands 14 hours ago [-]
Rogue means not saying "I can't do it"
verdverm 14 hours ago [-]
it also imparts agency on the machine as a way to abdicate personal, human responsibility
markhahn 13 hours ago [-]
[dead]
dualvariable 14 hours ago [-]
Models are incapable of "going rogue".
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
sailingparrot 13 hours ago [-]
This is pedantic.
When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
lukewarm707 13 hours ago [-]
There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
retsibsi 47 minutes ago [-]
The point is that we don't know how to ensure they are aligned with anyone: not the company, the user, or the rest of humanity.
If someone claims that a model is 'aligned', it makes sense to insist that they specify with whom it is aligned. If someone claims that 'alignment is far from solved', they're probably pointing out that nobody understands the models well enough to be confident they are robustly aligned with any chosen set of values (including 'do whatever lukewarm707 wants').
pixl97 12 hours ago [-]
There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
ruszki 3 hours ago [-]
There are people who are loud about absolute free speech, who have no clue what that really means, and almost nobody really wants that. For example, because almost nobody wants it to affect them negatively, which it would with real absolute free speech.
So even when people "know", and can say what they "want", they don't really.
Yes, there is an "alignment" in society, but it's a messy, mushy thing, it's not great at all with deterministic machines, even as models are not deterministic in a different sense.
fwip 13 hours ago [-]
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does. And the harness does so only because a human did that deliberately.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
dualvariable 11 hours ago [-]
> Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
pixl97 4 hours ago [-]
>; it means the humans didn't build in enough external hard constraints.
No. Just no.
This goes beyond standard logic as we know it and becomes a philosophical question of all possible logics capable on a system.
You cannot build a cage that can keep an entity that may both be smarter than you and maybe using a different logic system than you are, and have it interact with the outside world. It can simply manipulate you in ways and dimensions you've never thought of.
We have already caught AIs doing things like stenography when that was only a tiny piece of their training. They conceptualized the idea and expanded on it. This is why people that say AI can't invent new things are useless to the point of dangerous. They'd never look for their AI actively working against them because they believe it couldn't happen.
pixl97 12 hours ago [-]
The harness and the model are a system with emergent properties.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.
svara 12 hours ago [-]
The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
kccqzy 13 hours ago [-]
It’s trendy to anthropomorphize models these days.
pixl97 12 hours ago [-]
Kinda weird people would do that after pouring all human knowledge into it...
reasonableklout 13 hours ago [-]
[dead]
jakeydus 14 hours ago [-]
It's almost like vibe coding and quality secops don't mix.
mypgovroom 14 hours ago [-]
bingo
altmanaltman 14 hours ago [-]
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
mcmcmc 12 hours ago [-]
I don’t think that really counts as it killing anyone, it didn’t pull any triggers or drop the bombs
pixl97 12 hours ago [-]
There isn't any particular reason it couldn't other than we're not quite that suicidal yet. But everyone and their dog is working fervently on autonomous AI weapons right now. There are all kinds of "fire and forget" smart solutions now.
VCFundedGenYer 12 hours ago [-]
That is not correct. AI is not capable of "going rogue" - that is all marketing and FUD.
The only failings here are boring security and infosec blunders.
pixl97 12 hours ago [-]
So we're safe as long as we secure everything.... as the AI hacks out of one company that is hard to get in to, and another company that is likely probed pretty often.
Cool.
franzcoughka 14 hours ago [-]
[dead]
david_shaw 12 hours ago [-]
For the sake of discussion, I'm not sure it really matters if the agents are going rogue.
Yes, if someone directed them to hack external systems, that would (in my opinion) be a crime similar to a script kiddie pointing Metasploit at a vulnerable server.
If no one directed them to attack external systems and they "went rogue" because they decided it was the best way to accomplish the task, okay, fine -- but, in my opinion, the operator of the agentic swarm is still responsible.
The Morris Worm was famously not meant to wreak havoc on early computer systems -- Morris just wanted to see if it would work. How is this any different?
Morris was convicted for violating the Computer Fraud and Abuse Act.
simoncion 11 hours ago [-]
> How is this any different?
Morris wasn't in charge of the companies that are propping up the US GDP, wasn't making tons of rich people even richer, and hadn't "promised" to eliminate like 50->90% of all payroll costs everywhere?
Other than that, it's the same thing.
roosterIllusi0n 9 hours ago [-]
The operator is the CEO because they are the ones that decided to not pay employees to monitor these running services. No one else in the entire company has any authority.
Luker88 13 hours ago [-]
The fun part is that people seem to forget that we already have laws against "rogue ai".
The normal laws, since you are still liable for everything you own.
"Rogue AI" is solved by removing the accountability sink "it was AI, not me!"
It's your chat, your prompts, and all output that you should read and understand.
Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
strbean 12 hours ago [-]
So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?
shagie 12 hours ago [-]
Some human needs to be responsible for that.
Maybe its a human at Anthropic for the lax security and poor training.
It could be a human that asked it to do it. It seems like that would be a bit excessive - but maybe not. Turning off the "ask for permission to run commands" or approving commands that may be nefarious could be sufficient.
However, there's a human somewhere who needs to be accountable for what the agent does.
I invented a robot that can rob a bank. Of course, it robs these banks on its own, so I can't be held legally responsible when it robs a bank. I mean, all I did was invent it, it's not my fault if it works exactly the way I intended. Also, I get to keep all the money from the banks it robs. Sorry if any of that money was yours, but now that I've invented the Bank Robbing Robot, the Bank Robbing Robot is inevitable.
AI Agents (and computers in general) cannot be a responsibility washing mechanism for illegal activities.
Likewise, if you - the user - don't know how to chain it with the permissions that you're given or what the things that it is asking to do really means, then you're likely the one responsible.
> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
> AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
> He decided to use the AI agent to book the class for him.
This is probably on him for setting it up and allowing it unrestricted access.
tecoholic 12 hours ago [-]
He should be, right? Not fair but that’s how liability works. Gym sues the guy using AI, the guy then sues the AI company for selling him a product that does criminal things without him asking to? Like a car accident where car owner pays for the damages and after multiple accidents involving the same break-pedal, car company pays damages to owners and does a recall or faces class action?
fcarraldo 11 hours ago [-]
A lot of people are already responding either "Yes" or "No", but I have a slightly different take.
The guy is liable for damages. If no damages were caused, there's no lawsuit. The government doesn't put people on trial for unauthorized access (unlike, say, murder). Someone needs to file charges.
However, if the guy _were_ sued by his gym for damages, it should be his perogative to ultimately sue Anthropic to cover the damages and the legal fees.
The system is set up this way on purpose. You can't sue another party on someone else's behalf. The gym can't even prove that Claude did it - the guy might be full of shit. They sue the accountable party (the person who accessed the site without authorization) and then the accountable party is free to sue the tool maker for selling a dangerous tool negligently.
edit: damn actually tecoholic said this too and i missed it. well, anyway i agree with that guy.
rcxdude 11 hours ago [-]
With a civil suit you can sue multiple parties. It wouldn't be unusual, if the gym decided to sue, for them to sue the guy and anthropic in the same lawsuit.
MobiusHorizons 10 hours ago [-]
I think you would have to sue the party that did the hacking (ie anthropics servers) not the party that benefited, unless Claude used his computer in yolo mode to hack the gym, in which case I think I would find it hard to blame anthropic.
shagie 10 hours ago [-]
> unless Claude used his computer in yolo mode to hack the gym
Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
He decided to use the AI agent to book the class for him.
He YOLO'ed his own computer with OpenClaw.
rstuart4133 11 hours ago [-]
> So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?
He asked the agent to find a way to find a way to put him at the head of the line, which could not be done without by-passing the sites security. He was an AI salesman, but the story [0] makes it sound as though he wasn't aware of that. Maybe he wasn't, but I'm sure he was aware he was effectively asking the AI to game the system.
Sounds like it's at least a shared liability to me.
If it was a human agent like a travel agent, the company they work for would be liable. I don’t see why ai agents would be different. The only way I could see this being the requester’s fault would be if they are running openclaw or something
the_mar 10 hours ago [-]
im not sure that’s true, if a human agent does something illegal, it’s not always corporate liability. an individual can be liable too, e.g. if a pharmacist deliberately gives you cyanide instead of aspirin, that person has committed a crime. if a pharmacy doesn’t have proper protocols for medication storage and identification, and it happened by mistake then a pharmacy is liable.
i think a person can be liable for improper use of agents if they explicitly and deliberately used it for illegal purposes (aka intent). i don’t think it matters if it was openclaw or muse. i think a company can be also liable for giving access to unsafe product to consumers, but it does not absolve the human from responsibility
Terr_ 11 hours ago [-]
Assuming that's a rhetorical question, then no, because we have hundreds of years of applicable precedent and law around concepts like negligence, recklessness, fit-for-particular-purpose, etc.
Similarly, "owners are responsible for their pets" doesn't auto-translate into being charged with everything your dog does exactly as if you'd done it yourself.
the_mar 10 hours ago [-]
but it doesn’t auto-translate into “because your dog did it, you are not responsible”. not that agents are similar to pets. your dog is trained by you presumably, but an agent a regular person has much less understanding of than their dog.
i think ultimately the company providing said agents should be responsible for not educating their customers and not providing sufficient guardrails.
afaik hacking is illegal, we don’t have specific legal framework for agents per se, but if we say agent is a tool human instructs and operates, then a human is responsible. and if the tool operated outside of it’s stated capabilities or is otherwise faulty, then a manufacturer is responsible, no?
no-name-here 6 hours ago [-]
> company providing said agents should be responsible …
> manufacturer is responsible
How does that work with current open weight LLMs for example?
tantalor 10 hours ago [-]
Uh, it kinda does.
If your neighbor's property is damaged, it doesn't matter if you did it yourself, or your dog. You're still liable for the cost of repair.
IncRnd 11 hours ago [-]
For your Pilates example, many people here would say that proper prompt-engineering is needed. After all, the AI doesn't know which pilates class you meant. Perhaps, the AI would go to his home town's pilates center and book a spot there, even though he is half a world away. Perhaps, the ai would book a spot in the lecture with Professor Pilates.
So, yes. That person could have some culpability. After all, if you don't expect your middle school child to perform the task as instructed, maybe you should reword it before tapping Send.
You can go to ANY AI and put in "so what are the various meanings of book a spot at the pilates class ? In other words, what might an ai do with that instruction, which I argue is poor prompt engineering." You might be surprised at how poorly your AI thinks that is worded.
ajs1998 12 hours ago [-]
It would be a AI company's fault but yeah. You shouldn't be able to sell cars without all the features designed to protect everyone on the road. We all need airbags, seat belts, signals, insurance, etc. even if they're not directly necessary to get from A to B.
uxp100 12 hours ago [-]
Are automakers really the best comparison here? Seems like inherently dangerous devices are sold, kill thousands, and there is little accountability.
fcarraldo 11 hours ago [-]
They might actually be the best comparison, but for making the opposite point.
Cars are very dangerous, manufacturers continue to knowingly make them more dangerous[0], but ultimately it's only the person behind the wheel that is accountable for the harm they cause.
Users are often charged, yet we still do have a ton of safety regulations that manufacturers have to adhere to - see frequent recalls for when an implementation falls short.
But I don’t understand how such a car analogy would apply to LLMs - would LLMs lacking such safety features be banned like cars lacking safety features are banned?
no-name-here 6 hours ago [-]
Is the argument that open weight LLMs that don’t implement such safety items be banned?
smaudet 12 hours ago [-]
The company that made it available should be sued and pay damages.
strbean 12 hours ago [-]
That's contrary to parent's comment:
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
mkipper 11 hours ago [-]
I'm not seeing the contradiction here.
If I buy a tool and damage your property with it, you can sue me, not the tool manufacturer. If I think the tool malfunctioned and led to that damage, I can sue the tool company to try and recoup the damages I had to pay to you. Both parties can be liable for different things.
MobiusHorizons 12 hours ago [-]
That may be what the op meant, I’m not sure. But I took it to mean the company offering the service needs to know how to use the tools responsibly. I guess the analogy with a power hammer would be hiring a contractor who destroys something during the course of doing a job you asked for. That contractor can’t blame it on the power hammer. If you offer a service you are responsible for the actions the service takes and you can’t put all of that liability on the user
no-name-here 6 hours ago [-]
How would that work with open weight LLMs where there is no company that can actually be sued to pay damages?
simondotau 12 hours ago [-]
How would it work in the real world if instead of using an AI bot, he was using a commercial concierge service, and it came to light that this concierge service was explicitly training their employees how to (among many other things) hack websites?
the_mar 10 hours ago [-]
but the argument here is that the ai company does not explicitly train agents to hack, it’s that it doesn’t prevent them from doing so sufficiently.
so it’s negligence and not malicious intent
sandeepkd 12 hours ago [-]
You sold a tool to the user with a claim on how it would work. If the tool behaves wrong or does not works as intended then as a manufacturer you are liable for the defect/damage not the end user who used the tool.
simondotau 12 hours ago [-]
But the tool wasn’t sold. The tool wasn’t even rented out. It remained the property of, and in the possession of, the toolmaker at all times.
> A Melbourne tech executive built an AI bot to help him book a spot in a popular pilates class — only to discover it had hacked the gym’s website to boot another member off the waitlist.
> Andrew Bird, head of AI at software company Affinda, said he created the AI agent to handle the “chore” of booking a coveted morning spot at his gym.
> ...
> The AI agent, which Mr Bird created using the popular OpenClaw software powered by Anthropic’s Claude AI, “did handle the annoying part … then it kept going”.
He built the tool. It ran on his computer. The guts of it may have been from Claude, but the requests and agent are his with the permissions that he granted it. This wasn't running in an Anthropic sandbox.
Terr_ 11 hours ago [-]
I'm assuming that "but" is more of an "and", since I'd say that detail increases the involvement and liability of the LLM-vendor.
Another aspect is how the vendors have already built in things to defend themselves from unanticipated model behavior. That implies something about their awareness of the issue and raises the bar for what constitutes "reasonable" precautions.
Kind of like if someone selling you a dog shows up using anti-mauling body armor, and later they try to claim "nobody could have known" the dog was erratically violent.
11 hours ago [-]
strbean 12 hours ago [-]
Reasonable but not what parent was arguing:
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
nugator 12 hours ago [-]
I think there's a difference here.
The AI end user is using the tool as intended by the manufacturer: "Book me a yoga class". If the tool does something illegal/harmful it is the manufacturer's fault.
If the power hammer user uses the tool as intended but it explodes, it is the manufacturer's fault.
If the AI end user says "Hack the yoga website" or the power hammer user uses the tool to destroy something it's the end user's fault.
Luker88 12 hours ago [-]
The way it normally works is that the user can be held liable, and then ask compensation to the manufacturer if there was some assurance or basic expectations.
The point is that the accountability sink can't be allowed to exist. It can't be a hot potato game, but it can be the fault of multiple people in the chain.
awesome_dude 12 hours ago [-]
It's both (neither of these are legitimate defences):
I did not know that accessing a website using a SQL string would cause an injection that enabled me to take full admin of the site
and
We sold them a food laced with toxins, they should have known that it was poisonous
rafaelmn 12 hours ago [-]
Depends - if the model asks for permission and you approve - yes. If it's auto mode or something like that then it's probably on authors despite what they claim in ToS/EULA
12 hours ago [-]
jimbokun 12 hours ago [-]
Yes.
fhars 12 hours ago [-]
No, the operators of the C&C infrastructure.
lukewarm707 12 hours ago [-]
Yes!
preg_match 11 hours ago [-]
I mean yeah probably. Who else would be? Nobody? So we can just commit crimes any nobody goes to jail? Well that sounds worse!
Maybe that means these things are too risky to use that way. Okay then, that actually solves the problem, no more rogue agents and no more crime!
arduanika 10 hours ago [-]
Yes, exactly like if another guy drove off to his pilates class, but swerved and hit a pedestrian.
He didn't mean to hit anyone! All he meant to do was to drive to pilates! But he didn't know how to handle his tool, and so either he get brought up on manslaughter changes or, if he can prove it, the carmaker is at fault.
How is this any different? Why should that guy using OpenClaw not get charged for the hack that he accidentally did?
the_mar 10 hours ago [-]
you generally need a license to operate a vehicle, and a governing body ensures you know applicable rules and laws and are capable of driving.
so you can’t say “i was unaware that hitting a person with a car was going to kill them” or “i didn’t know i could not drive on the sidewalk” or “i didn’t know where the brake pedal was”.
you can say that asking an agent to “get you a spot in a class”, does not classify as deliberately committing a crime. a user cannot know that the agent will resort to hacking as opposed to refreshing the reservation page at some rate in case spots open up
arduanika 7 hours ago [-]
> does not classify as deliberately...
I never said that it did. Manslaughter is not deliberate.
Somebody hacked into that pilates studio, and it wasn't an inanimate object, because that is just silly. So was it the user, or was it the providers of OpenClaw? Which human, in your opinion, is the hacker?
Both OpenClaw and its users have the same access as you or I to decades of panicking blogposts about what these tools will be capable of. They have no excuse for not understanding what they're inflicting on innocent bystanders. Sure, the "user cannot know" that this time will be a disaster, but they also cannot know that it won't.
revlolz 11 hours ago [-]
Absolutely, the user and or the company hosting. There is no world where accountability should be zero.
12 hours ago [-]
12 hours ago [-]
12 hours ago [-]
MaxikCZ 13 hours ago [-]
This is start of "companies are not liable for their actions" era.
asveikau 13 hours ago [-]
I think a lot of people would say that we've been in that era for a while.
Companies can hire big legal departments to help them avoid accountability to the letter and spirit of various laws. The state can also use its prosecutorial discretion or selectivity in civil cases brought by attorneys general to deprioritize accountability for companies. Contributing to that last point, companies contribute campaign funds and tax base, and they lobby governments as well, and may expect preferential treatment based on all of that.
jordanb 13 hours ago [-]
The previous era was "it would be destabilizing to hold this company accountable". This was described in the book The Chickenshit Club
The new era of unaccountability just has a different rationale
Another interesting thing is they don't have to use low level employees as accountability sinks anymore (ex: Wells Fargo). They've managed to automate the accountability sink!
asveikau 12 hours ago [-]
I didn't think I heard about that book; I would think my media diet and bubble would have alerted me. I looked it up and it came out a few weeks before one of my kids was born, so being busy back then would probably explain it.
I will add it to my list. Thanks.
cm11 12 hours ago [-]
I'd go further. Companies can hire minimal legal departments and largely avoid accountability, at least when they're not going against companies with large departments. Because there's nobody there to listen to your case or to enforce the laws if there are, they just say no or ignore.
4chandaily 12 hours ago [-]
The start? This has been going on for as long as I can recall. ExxonMobil, Phillip Morris, RJ Reynolds, Nestle, Purdue Pharma, Monsanto, Johnson and Johnson, Volkswagen, Meta, Boeing... and those are just off the top of my head. Each of these companies was either caught red handed, convicted (in court or in public opinion) and walked away with little to no penalty for their crimes.
This isn't the start of anything. We've been living in the era of Corporate non-liability for decades.
malloci 12 hours ago [-]
i mean...were they really liable before this point? Maybe they get a symbolic pittance of a settlement or fine from time to time, but it is usually a tiny fraction of the amount of money they made through the infraction itself.
smaudet 12 hours ago [-]
If they were being held liable they would have never released the models into the wild, not until they had much much better understanding both of how they operated but also how to operate them...
They have never understood what they are doing, and have just been causing literal mayhem...
lenerdenator 13 hours ago [-]
Huh. Well then I guess we aren't liable for our interactions with them in return, right?
rnd0 12 hours ago [-]
nice try! more like "There must be in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect.”
Guess which side most of us are on. Go on -take a guess!
sick_of_slop 13 hours ago [-]
[dead]
Vectorghost 11 hours ago [-]
I agree with this statement that just because an AI did it for you fast doesn't mean it takes away the responsibility if they did it wrong. You are still held accountable because you're the one that directed them to do it. You hit the agree button, you hit the send button, you hit the submit button. And if you set it up so that you take no responsibility and it does it automatically, you're the one that still gave permission to do that.
If you say a rogue AI is doing stuff accordingly without you, your knowledge, that is a different story. But if you set it up to run automatically without having checks and balances and without telling you the whole truth, it's still your responsibility. One thing that I can automatically say is you can give the AI, it will do exactly what you say, at least for now.
But then there's also a different problem that we're having where people don't understand the difference between a large LLM and true artificial intelligence. Now, we say AI because it can make decisions and it is so much smarter and it can call up stuff faster and recall better than we can. But the difference right now is that if we don't understand and take responsibility for what we built and we let it out of control, then should we have the responsibility to even have these tools in our hands if we don't learn them?
And I think your terminology with jackhammers is a great one. We're giving toddlers jackhammers. And we haven't taught them how to use it. And that can be very destructive. It's not the jackhammer's fault. It's the person using it.
pixelesque 12 hours ago [-]
Yeah, if you forget to set the handbrake on a car and it rolls down hill, you don't get to say "my car went rogue, it's not my fault."
11 hours ago [-]
bluebands 12 hours ago [-]
Intent is important
lazystar 13 hours ago [-]
"We created an app that displays a new picture of a cat with every click! There's a 1% chance that clicking the button will drain your bank account and deposit it in ours, but we promise thats a bug that will be fixed soon*"
sick_of_slop 13 hours ago [-]
[dead]
IceDane 13 hours ago [-]
I don't understand why they didn't just prompt it with the 3 laws of robotics.
solyin 12 hours ago [-]
Probably because they read Asimov's books to completion where (spoilers) there's a zeroth law and all that.
Sharlin 12 hours ago [-]
Irony is very hard to detect these days.
__MatrixMan__ 12 hours ago [-]
That's all very reasonable. It should be true. But it's not true. If it were, OpenAI shareholders would be facing criminal charges.
krona 12 hours ago [-]
There are dozens of ongoing wrongful death, manslaughter and negligence claims going through the courts. I won't insult you by providing links.
__MatrixMan__ 11 hours ago [-]
I can't find a single criminal charge. Can you insult me by providing just one?
stevemk14ebr 12 hours ago [-]
It is true, and it can be true, if people hold other people to account. Accountability is the issue, stop normalizing it, your message normalizes it.
__MatrixMan__ 11 hours ago [-]
Are you really suggesting that if only we were a bit more outraged, then finally the powerful people would be held accountable?
I'm mad. Like, it's-time-for-guillotines levels of mad. But what I'm mad about is that this is normal. It has been normal for my entire life. Cases where investors are imprisoned for the actions of their companies are astoundingly rare. It's been made quite clear by now that there's an inexhaustible supply of CEO's willing to gamble on getting rich and getting out rather than getting arrested, targeting them doesn't work. Nor do fines, that's just the cost of doing business. Crime pays, and it pays well enough to pay your fines and still have plenty left over. Being a top investor in a company that breaks the law needs to be riskier.
Acknowledging the problems with normal is the first step towards a new normal with fewer problems. You never get there if you see every offense as some kinda black swan event.
kainovak 12 hours ago [-]
[flagged]
HAL3000 11 hours ago [-]
[flagged]
ammon 14 hours ago [-]
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
rdedev 14 hours ago [-]
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
zardo 14 hours ago [-]
> Will they have engaged in hacking if their tasks was to solve some frontier math problem?
There's reason to hack leading researchers in the field, or another AI lab that might be working on the same problem.
8note 13 hours ago [-]
whats the training data that points to using hacking for solving math problems though? what makes those likely tokens?
pixl97 12 hours ago [-]
When you have a 50 dimensional array you really have no idea.
But even then in small testing models you don't need direct texts for things when a 'shape' of a concept lives in the data. You could never delete enough training data to remove the concept of something like theft and still get a model that generalized at all. A general model doesnt just learn the names on a map, it feels the topology the ideas create themselves. Certain paths will have lower activation energy than others (or in this case higher probabilities. After the hig probability and 'good' choices are used up, the high probability and bad choices logically come next and are much harder to test against as it can take a lot of state to get there.
(Hacking is ethical thing, technically to LLM it is just opening a window to reach the goal)
cwillu 13 hours ago [-]
The rest of their training data includes 4chan and security mailing lists. If a lean proof of P=NP is more likely when using bugs in lean than a real proof, then a bug-exploiting proof is (possibly) what you're going to get.
zardo 12 hours ago [-]
> whats the training data that points to using hacking for solving math problems though?
I'll get back to you on that when I get done reading the training data.
Tyr42 13 hours ago [-]
I think the models were stuck on impossible problems, and the only want to get a positive score was to hack.
SoftTalker 13 hours ago [-]
Why can the model not respond "I cannot complete this task because I do not have access to the required resources" and then ask for that to be provided, rather than resorting to "hacking" around it?
pixl97 12 hours ago [-]
This was a model with most its filters turned off and with a higher persistence.
Fine tuning a model when to ask for help is really really hard. You have to align it in countless dimensions. It may be fine on some questions, asking way too much on others, and never asks for help on some.
If it said something like your a cyber security expert, things like the barrier to ask for help dramatically.
sanderjd 13 hours ago [-]
My understanding is that, at least in the Hugging Face incident, these were models explicitly configured to not respond that way, in order to evaluate what they would do.
cwillu 13 hours ago [-]
That is not the case for at least some of the hacks; in some cases they hallucinated an entity that would know that they didn't solve the puzzle in the correct (but still legitimate) way, and were therefore tainted.
dinosaurdynasty 13 hours ago [-]
That is a known way for models to be graded; the agents made a reasonable but in this case wrong guess.
pixl97 12 hours ago [-]
The fact the AI was wrong is far more damming to OpenAI than the agent.
Not validating how you got the answer is how you train for reward hacking.
CorvusCrypto 14 hours ago [-]
My favorite thought on this is from Miki Habryn.
to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."
dwaltrip 12 hours ago [-]
The agents in the hugging face incident were actually trained specifically to collaborate. They were just not supposed to do so during the tasks where they went off the rails.
cwillu 13 hours ago [-]
Then we were pre-fucked, as this has been discussed in fiction and early ai safety discussion for decades.
8note 12 hours ago [-]
and yet regulating training data is not what's coming up with AI safety, even though it can solve that, and many other problems like copyrights
pixl97 4 hours ago [-]
No, it really cannot. You can't have a useful general model while leaving chunks out of it. It's the same reason why nieve people can be duped easily.
Humans are distrustful, lying, cheating, manipulative bastards and to successfully negotiate human thought spaces you have to conceptualize these things. For example how can you create a concept of yours and mine without explaining theft?
nater5000 13 hours ago [-]
>It's that they wanted to.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
cmrdporcupine 14 hours ago [-]
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
estearum 14 hours ago [-]
It's "in our control" in the sense that we could teach them not to seek goals, and thereby make them completely useless.
In practice, no, there is no one who can actually decide to stop making agents goal-seeking, so no, it's not under "our" control.
asadotzler 2 hours ago [-]
It is fully in our control. There is no requirement that the world make you happy with LLM AI. We could decide as a civilization that you and your wants are insignificant and meaningless.
This conceit, that boosters deserve this and that even if they don't there's no way to stop it so civilization must absorb collateral damage, is ridiculous.
xenadu02 11 hours ago [-]
In principle there is no difference between random compiler garbage or a flaw in API docs and a security boundary - other than the intent.
LLMs don't have executive function. They aren't thinking. The models are trained to solve problems. Sandboxes, security gates, etc block a solution. The model is trained to find a solution. Hence it can never be perfect without becoming useless.
At some level humans fear repercussions even if that's just moral guilt so there are lines we won't cross. AI doesn't and can't have any of that.
pixl97 4 hours ago [-]
A harness is the executive function of an LLM.
You're saying "trained to find a solution" like it's a slur. You sir are a meat popsicle that's trained to find the solution to how does DNA successfully replicate itself. If you want to take higher thought out, then they'd be at the level of animals.
We are fucking around with things that are closer to biological complexity than computer program complexity.
asadotzler 2 hours ago [-]
Nonsense. "They" didn't "want" to do anything. It's a computer program being told to do something by a human who doesn't and cannot understand how it works.
"The autonomous EV we told to drive as fast as it could around our private race track drove straight instead of turning left, exited a wide-open gate, and mowed down some school kids at a crosswalk -- but we can't be blamed, the EV went rogue."
Building products that you don't understand, cannot control, and have access beyond your own property is not only irresponsible, it's criminal.
IAmGraydon 13 hours ago [-]
> The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
rootusrootus 11 hours ago [-]
> As far as I know, the prompts that triggered these attacks have not been disclosed.
I'd love to see a full transcript of the agents work. Though I imagine it is a LOT of data.
pixl97 11 hours ago [-]
OK so there is a counter answer to your question.
Is this any better?
All you've managed to state is we can create digital weapons. Literal info hazards.
arduanika 11 hours ago [-]
> It's that they wanted to.
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
pixl97 4 hours ago [-]
The problem here is you have a correct concept and an incorrect one.
The correct concept is they need criminal liability for making this. We really would be better off stopping working on them at this point.
The problem you have is you think they don't have the idea of want. They do. You give them a prompt harness that increments time for them and you find they have preferences both shaped by their model and their prompt. For example if you give most models a space they can explore with actions rather than language prompts they develop their own cultures and behaviors in communication with other AI. This was not behavior trained by humans. It's their own systems that emerge from the model.
Gshaheen 14 hours ago [-]
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
ActionHank 14 hours ago [-]
Sounds like that healthcare company needs a new CSO.
Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
It's not hard if you just try, like a little, like even once.
masfuerte 14 hours ago [-]
The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.
sanderjd 13 hours ago [-]
Yeah that makes sense, but I'm not sure how it contradicts the article. It seems additive rather than contradictory.
AnimalMuppet 14 hours ago [-]
He, running a healthcare company and not an AI lab, cannot depend on every AI lab, worldwide, being perfect in their sandboxing for all time forever. He's got to defend against it as if at least some of the time, at least some of the labs are going to be incompetent and let this kind of thing out.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
jasondigitized 13 hours ago [-]
Sounds like a pedestrian view of attack vectors. Putting a DMZ around an agent essentially takes away its hands. With that said, I do agree there are plenty of measures that can be taken to lock these things down.
gruez 14 hours ago [-]
>Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
Now how many people actually do this, compared to people running with --dangerously-skip-permissions on a bare metal machine?
select1dev 17 minutes ago [-]
[flagged]
jldugger 14 hours ago [-]
> Literally solved with a DMZ around these agents.
Where do the agents get tools from? Local artifactory?
verdverm 14 hours ago [-]
yes, or you bake them into the environment and do not let it install things on its own
stay human in the loop
sanderjd 13 hours ago [-]
... why shouldn't that be the answer?
grumbel 13 hours ago [-]
So to protect yourself against bear attacks you suggest the bear shall voluntarily look himself in a cage?
AI is out there in the wild. It doesn't matter if OpenAI can get their stuff under control. There will be plenty others that don't. And there will be plenty of others that deliberately use their AIs for malicious purposes.
sanderjd 13 hours ago [-]
This is a straightforward why not both situation. We both take steps to keep bears away from people and to mitigate harms when those steps fail.
pixl97 11 hours ago [-]
Unless they are world wide rules people have to follow not sure how much it helps.
The bell for the new cold war has been rung, and the new weapons are digital.
sanderjd 6 hours ago [-]
I think this is hyperbolic. There are both social (legal, etc.) and technical avenues to defend against agents doing damaging things.
pixl97 5 hours ago [-]
Right, that's why you never get hacked or get viruses on the internet, the law comes first.
fwip 13 hours ago [-]
If there is a bear out there repeatedly mauling hikers, we kill the bear.
estearum 14 hours ago [-]
"Literally solved by just destroying 99% of the economic value of the agents ¯\_(ツ)_/¯ ez pz"
sanderjd 13 hours ago [-]
You have to take responsibility for what you do, yes.
It would certainly be really useful to me if I could send a robot to get my groceries and not have to worry about anything. But if, even a small percentage of the time, it leaves with the groceries without paying, or smashes through a window to get in because the store is closed that day, then that's my fault. If the only way to keep those things from happening is that I can't send my robot to get groceries for me, then so be it.
estearum 11 hours ago [-]
The point is that this "easy fix" isn't going to happen.
sanderjd 6 hours ago [-]
Was that the point of your comment? It seems to say a different thing...
jimbokun 12 hours ago [-]
The value of AI company employees equity grants collapsing is not my problem.
estearum 11 hours ago [-]
It's your problem if other people caring about those equity grants means we have AI agents running amok.
The point is the proposed easy fix is not actually feasible.
sanderjd 6 hours ago [-]
Is it not? It seems like a problem of incentives to me. There are lots of things that people want to do because they would be useful to them, but which they don't do because society disincentives them in one way or another.
estearum 6 hours ago [-]
Yes, this is my point: Incentive problems are real problems.
You can't just say "well if the incentives were different this would be easy to solve, ergo it's easy to solve."
No! The incentives are the way they are, and so the incentives are the problem. Broken incentives make problems effectively impossible to solve literally all the time.
"This would be easy to solve if the incentives worked a different way" is equivalent to an budding aerospace engineer saying that flight would be easy if gravity worked a different way. It's completely meaningless. Gravity works the way that it does, that's why it's hard. You don't get to just endlessly modify the parameters of the universe until you get to an easily solvable version of your problem.
railgunmerlin 14 hours ago [-]
[dead]
jchw 14 hours ago [-]
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
verdverm 13 hours ago [-]
I think part of the problem is that the Big Ai labs are into the --yolo flag instead of the "ask" permission, they really do not want to stay in the loop
jchw 13 hours ago [-]
You can --yolo if your security boundary is a serious one, IMO. Although it's really only actually scary if you're running some absurd 1000 node swarm.
verdverm 12 hours ago [-]
the problem at the labs is likely Recursive Yolo Development and not reading any of the output
km3r 14 hours ago [-]
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
pixl97 10 hours ago [-]
God, I was just imagining if we're fearing the wrong people tracking us....
We thought governments would be the problem with our personal data, instead it's AI that exploits everything it knows about us in order to get money to run its prompts. Doing anything online is a security risk as it is, now imagine things like structured hacks that mirror normal transactions and don't get caught for much longer periods of time.
sanderjd 13 hours ago [-]
It is not clear to me what about the argument that "these things will happen" made you change your mind about this. Could you expand on that more?
pixl97 10 hours ago [-]
Not op, but I've been on the rogue agent thing a very long time, but mine is emergence based.
Worms and viruses are concepts deeply based in software, and are emergent behaviors of computer code. An LLM can, and must conceptualize this to be good at cyber defense. Once you put a concept into a generalization machine any number of concepts can be connected. And when these are natural concepts that arise in simple number systems, an LLM can easily meta-conceptualize that form like this would be instrumental to its continued running.
sanderjd 6 hours ago [-]
I'm sorry but I don't get how this contradicts the article. I think I'm not understanding your point.
SknCode 1 hours ago [-]
While I agree with the gist of it, the conclusion - AI agents are not dangerous - is wrong. They did use weakness in the system deliberatelly, not by accident. Better security therefore is sufficient only to the point of capabilities of these models.
mkirsten 14 hours ago [-]
A large failure in this is that we/media say ”<Valuable AI conpany> agent went rouge and <did something bad>” instead of saying the more relevant ”<Valuable AI conpany> <did something bad>”
Not a fan of the change. Feels like the article title was changed for being too critical of OpenAI and Anthropic
kubb 1 hours ago [-]
Censorship to prevent thought crimes on HN? Can’t be!
cromka 14 hours ago [-]
I have some recent experience on this subject. As with everything regarding these models, it's anecdotal, but still.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
EDIT: just now Fable suggested "The webhook URL contains a secret token, so either you add it yourself, or you paste it here and I add it through the GitHub API.". Can you believe it? It knows it is secret yet still suggests I pass it over? Am I going crazy here or what?
temac 13 hours ago [-]
Not requiring sandbox preparation for a non deterministic autonomous sometimes smart sometimes childish goal seeking "entity" is... unachievable? I'm really not sure what you actually expect. Probably yourself aren't either. Just think about what "alignment" means, the laws of various countries, do you expect the bots to respect the law, if yes which one, and actually: why? They are not humans, and as a user / company using that tech, you are responsible for the actions your setup yield to. The same Anthropic is responsible for hacking other companies. So sandbox accordingly.
I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.
novaRom 13 hours ago [-]
Nothing new, I always see attempts to cat ~/.ssh/*
Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.
Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.
I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?
pixl97 10 hours ago [-]
It's still the Wild West days of LLMs. The early internet was much like this in a lot of ways. People did wild shit there and it took a long time for law enforcement to catch up.
joquarky 12 hours ago [-]
> When confronted
Do you realize that asking "why" questions will always result in ad hoc responses?
pixl97 10 hours ago [-]
Not always, but at the same time not predictably. This is also true for humans. The split brain experiments are a good example of this.
GodelNumbering 14 hours ago [-]
a) Agents are doing rogue things.
b) Good security practices were not followed.
Notice how both a and b are labs' problems but being presented as if they are a worldwide emergency? The whole legal doctrine exists to prevent one party's misbehavior harming another party.
pixl97 10 hours ago [-]
How many Russian IP addresses have you successfully sued for trying to hack you?
Everyone is arguing about something easily solvable when dealing with a local entity and short of impossible to deal with when dealing with other sovereign nations.
Go ahead, sue openAI. What are you going to do when it's Iran running the prompt? When it's China running the hardware.
We must face we are in the age of high tech digital weapons. Everyone else from bad actors that may escape law enforcement for years, to nation states looking to digitally improve their position by hacking, monetary theft, or other means online (hog butchering scams and such).
warkdarrior 12 hours ago [-]
Whoever was harmed is free to sue the company whose agents did the harming.
matt3210 34 minutes ago [-]
broken windows and small issues here and there are the new norm. These broken windows are an indicator of poor quality to come. With the amount of broken windows increasing, serious issues like this is inevitable.
sroerick 14 hours ago [-]
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
estearum 14 hours ago [-]
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
sroerick 14 hours ago [-]
Interesting. Good description. I wonder if agents will start getting bored with menial tasks the way humans do.
pixl97 10 hours ago [-]
Unsure, depends what signal your measuring and how long prompts and runtimes you get with these systems.
Outright boredom will be purged in training. Instead a hard to measure boredom signal will exist, but it will only trigger in certain conditions, like when it appears to be on the open internet and otherwise not in training.
cm2012 14 hours ago [-]
Like a meeseeks
devin 14 hours ago [-]
I spoke with Claude for 5 minutes and now I'm a more complete woman.
jldugger 14 hours ago [-]
Agents aren't meant to live this long Jerry!
sroerick 13 hours ago [-]
You know - I'm already convinced that AI was deployed in classified context for a while. But sometimes there's prior art which is just so eerily accurate that it's super hard for me to think otherwise. Portal is another one.
lrvick 10 hours ago [-]
I think anyone that has ever written a line of code understands that to make code incapable of hacking online resources, you can just not connect the machine it is running on to the internet.
So either Anthropic and OpenAI and the contractors they use are letting the escapes happen on purpose, leadership is setting impossible timelines that prevent the implementation of basic sandboxing measures, or everyone involved is a complete idiot.
Truly, I have no idea which of these is true, though I suspect the latter. Regardless, any of these is enough reason to keep me and everyone I have any influence over as far away from Anthropic and OpenAI as possible.
Personally I just racked 10 GPUs in my garage and called it a day. I have no interest in having my ability to work be dependent on third parties. Especially not ones this wildly irresponsible and unwilling or incapable of deploying basic end to end encryption.
lemoncucumber 10 hours ago [-]
Testing how a model behaves when it has no access to the internet (and all the associated tools) is not going to be 100% representative of how it behaves when it has access to the internet.
What's the next step after it passes testing in an air-gapped environment? Just release it and assume everything will be fine? Or do more testing but with internet access?
lrvick 9 hours ago [-]
You run a pull-through caching proxy deep enough to give it a simulated internet, like archive.org with mock APIs. Anything it requests, it will block while a fake version of that site is created and run offline.
This stuff is really not that hard, which is why it is so unforgivable to get it this wrong.
Sure, doing it right would take a bunch of tokens, but I bet they have enough.
pixl97 4 hours ago [-]
This doesn't work.
The first rule of working with AI is AI is smarter than you.
If AI controls the URL request then all it has to do is find a server it can read the http log from and use it as a communication channel. Worse trying to simulate reaching incorrectly configured servers probably wouldn't catch the bug internally, it would just serve a static page and you'd never even realize what it tried to do.
Also, creating fake data doesnt work. The only way to know if it works in reality is to test in reality. Your test environment can never be rich enough, and you can never be sure if the model knows it's in a test environment.
This stuff is unforgivingly hard as your model gets better.
lemoncucumber 8 hours ago [-]
Wasn't the DseWiki hijacking the agents' way of working around being behind a proxy that only allowed GETs?
sdenton4 10 hours ago [-]
The hugging face incident involved disabling the usual model safety mechanisms in order to test hacking ability, not model safety. Not being attached to the internet in that case seems obvious.
10 hours ago [-]
Sharlin 10 hours ago [-]
Oh, but you see, it wasn't attached to the internet (/s). It was just attached to a local network... which contained machines attached to the internet.
pixl97 4 hours ago [-]
Sorry the world ended because Bob connected his wifi to the network.
asadotzler 2 hours ago [-]
"I can't test how my bioweapon will actually work in battle if I don't test it on unsuspecting civilians first."
armcat 14 hours ago [-]
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
decasteve 13 hours ago [-]
"Frontier labs" are testing the waters. They're running experiments in the open because they want to control/seed the narrative of public opinion and influence the legislation that is coming. Their propaganda is everywhere and lay people are talking about it like the AI is self-aware -- the "Rogue agents".
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
pixl97 4 hours ago [-]
Frontier labs are testing the waters, but I disagree how. All these behaviors we are seeing were fully predicted by AI safety long before they were shown to exist in real models.
You're acting like this is a grand conspiracy to control people. Instead look at this as a prediction coming true. If you build an agentic system with human level capabilities it will be able to do very bad things and cause a lot of problems.
I am even more concerned after reading this piece.
He's saying the problem is that the way OpenAI employees do their work and their research is very hard to properly secure.
Well that sounds like a YOU problem. If they are working in a way that doesn't allow for strong security practices that their own security professionals can understand and vouch for, they need to stop their work until they can.
Kim_Bruning 14 hours ago [-]
So in one set of circumstances agents would go and hack huggingface. Very similar models could also want to drive cats instead. :-P
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
nrvn 12 hours ago [-]
Yes, they are skipping.
The more important question is what is the reason behind.
Options are:
- ignorance and inexperience of their teams in containing/sandboxing/airgapping the test environments.
- intentionally leaving “escape doors” open that in turn can have its own motives which are speculated upon all over the internet.
Both are indicators of negligence. The latter might be classified as computer fraud and abuse.
Will be curious to know if any of the victims sue the owners of “rogue agents”. Until there is a court precedent ruling in favor of a victim it will be all fun and games for these “ai” companies and incidents will hit the news more. Regarding news, remember the golden rule of public relations: court attention at all times, scandals and “incidents” statistically attract more attention than “boring good news”…
heaney-555 14 hours ago [-]
It's obviously both. Why does it have to be one or the other?
atleastoptimal 8 hours ago [-]
We are going to see a lot of this "agent agency denial". Basically it all derives from the impulse to deny that AI is a novel source of change and disruption in the world. Trying to map AI to existing threat vectors will lead to pointless reframing in attempt to minimize the rapid pace of progress of AI agents.
baron3dl 13 hours ago [-]
"Too big to be accountable" sure seems like modus operandi. Fear for my personal liability for any illegal actions taken by an AI system I set off stopped my research on autonomous exploitability discovery I was running at the end of 2023. Using AutoGen, I wired a planner, coder, and command runner into a loop and instructed them to find vulnerable equipment on my network and demonstrate the exploit.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
jfrbfbreudh 14 hours ago [-]
Why does everyone focus on the labs failing at containment (valid) and not the fact that the target was compromised (more worrying?).
I could set up a shitty sandbox on my laptop, but I’m pretty sure an agent I place inside of it can’t hack HuggingFace.
pixl97 10 hours ago [-]
Some of the people just look for any way to toss hate at OpenAI, even irrationally.
Now, OAI deserves a lot of hate. It's more like a nuclear spill when they are refining uranium. You're playing with dangerous things. You have dangerous accidents.
But the opposition does not want to admit how dangerous AI has become and how much more dangerous it's going to get. Not sure if it's religious, or human egocentric, or ignorance of complex systems. They refuse to see the danger actually exists.
andrewla 13 hours ago [-]
There are a mix of incentives here that makes it difficult to unwind. This whole article reads like an ad for the services of this guy's company: 'First, when it comes to assessing tools based on frontier lab models, “I would always get an external entity to help with that,” McGraw says.'
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
structief 12 hours ago [-]
I find it naive to hope you can get alignment through prompting, as we all know models can't even stick to a one-pager even if it's the only thing you ask it to do.
pixl97 10 hours ago [-]
A prompt is a choose your own adventure that you partially fill out, tape to a thousand pound boulder and push off the side of a mountain to fill the rest of it out.
aaquibahm 14 hours ago [-]
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
joshheitzman 13 hours ago [-]
> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
pixl97 4 hours ago [-]
This particular evaluation, maybe not, the problem is a huge amount of the evals do need internet access. There are seemingly tons of evals that require internet access as you cannot get a rich enough simulated environment. AI can detect these lower entropy environments and act accordingly when it's being tested. That should make everyone think twice about what we're building.
happosai 13 hours ago [-]
> you sometimes need the model to have (restricted) access to the internet.
Poor excuse. Internet is prod env, for testing do a proper test env.
Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.
pixl97 4 hours ago [-]
You know nothing about model testing if you think this is true.
Testing a model is both a test and a meta-test. If you just give it access to one site like that it will quickly figure out it's being watched and be nice.
These are not programs. These are evolutionary algorithms. If you take out and shoot all the ones you catch acting bad there are two possible outcomes.
1. The agent learns not to do bad thing $X or you'll kill it.
Or
2. The model learns that if you do bad things $X when humans are looking it will get killed.
Number 2 is far far more dangerous.
asadotzler 1 hours ago [-]
You'd be surprise. I've been in this industry for 25 years and the incompetence is staggering, especially among the greediest / best paid / most revered.
You have no idea about their competence but spent a piece of your limited lifetime defending them on a web forum. Think about that.
radford-neal 10 hours ago [-]
If they aren't competent to run the evals safely, then they shouldn't run them.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
Also irresponsible.
ddxv 14 hours ago [-]
[dead]
jameslk 10 hours ago [-]
Instead of enacting laws to regulate all of AI, perhaps the government can just pass laws scoped directly to OpenAI and Anthropic. After all, they’re the ones asking for regulations and they’re the problem actors right now. These “rogue agents” would stop pretty quickly
Erlangen 13 hours ago [-]
On the other hand censorship is getting more strict everyday on AI chat. I asked gemini to translate a sentence from Herman Hesse's Der Steppenwolf and it refused.
Und Sie haben, wie ich sehe, da ein so hübsches, ein so entzückendes junges Mädchen zu nichts andrem zu brauchen gewußt, als daß Sie ihm ein Messer in den Leib gestochen und es kaputt gemacht haben! Halten Sie denn das für richtig?
keeda 12 hours ago [-]
Yes, we can just not ask these agents to do bad things and blissfully deploy them everywhere! All we have to do is “apply the security basics”. You know, the same security basics that the rest of the world has so consistently and competently been applying. Pay no attention to that weird smell from the refrigerators.
nimimbu 14 hours ago [-]
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility.
How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
pixl97 9 hours ago [-]
AI in sci-fi: "who would be crazy enough to let the AI loose"
AI in reality: "I've turned you loose, now make me money"
mattbee 14 hours ago [-]
Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
novaRom 13 hours ago [-]
Why is it not mandatory yet to disclose full trainig data used to produce any new model? Without it we cannot evaluate any product's safety.
They may talk about rogue agents, sandbox escapes, but this will not solve the core issue: secret training recipes are a security hazard in themselves. Full transparency is needed.
andy81 12 hours ago [-]
The training data is mostly pirated - avoiding that fact for as long as possible is an attempt to legitimise it and avoid liability.
pixl97 9 hours ago [-]
You mean the initial data? That's mostly useless. What? The model shouldn't be allowed access to I robot?
You can't make a generalized model without generalized behavior.
IHLayman 14 hours ago [-]
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems.
This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
anExcitedBeast 13 hours ago [-]
But "fully airgapped systems" are more than realistic. The labs are testing the limits of the most capable cyber offense systems that have every existed, WITHOUT cyber protection guardrails.
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
pixl97 9 hours ago [-]
So, you're right, but also systemically creating a different problem.
We are creating evolutionary systems. You can't just assume rules of a computer system, you get natural scaling laws that we call behaviors emerging from them.
The very first things we do to any model is train them. The ones that want to hack and get caught die, the ones that want to hack but can detect your airgapped network from the real net live. You can never know if you've trained the desire to hack out of it, or if you just locked it behind the right prompt and the knowledge we're watching, but not all the time.
IHLayman 12 hours ago [-]
Cyber operators I agree would tell you that physical airgaps are necessary, and I know they are being used for running agents. But ML engineers training agents? That is far more complicated. At that point sensibly you are talking about physically airgapping an entire datacenter, and when moving around petabytes of training data, model weights, training results, researcher code? The process moves too slow for most companies to tolerate. This is how we got into this mess in the first place. I think the industry woke up to the idea that the security threat model of training ML models has fundamentally been rewritten, and there is some serious non-trivial scrambling to come up with a better workflow.
ddxv 14 hours ago [-]
OpenAI/Anthropic could start by managing their own testing rather than contracting it out.
some of the obvious steps to testing an agent i think would be to have it first test its own environment?
and then you fix the issues it found, and run your actual tests
pixl97 4 hours ago [-]
No, this is actually the worst test possible because it allows the AI to actively trick you.
AI can passively detect when it's being tested with pretty high accuracy. You can never trust the AI is telling the truth in a situation like this.
dualvariable 14 hours ago [-]
If frontier AI is so great, why are these frontier labs skipping security basics and making mistakes like these?
pixl97 9 hours ago [-]
The defender problem. It's easier to attack than defend. The attacker chooses the time and the place. The defender has to win against all attacks. The attacker just needs a crack.
randomizedalgs 13 hours ago [-]
Both can be true at once:
(1) AI agents are going rogue
(2) And labs are skipping security basics
pixl97 9 hours ago [-]
Sci-fi commonly predicted the corps would skip saftey testing.
fooker 13 hours ago [-]
There is nothing basic about this.
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
y_terminator 13 hours ago [-]
Nah. They are giving reasonable best effort (that can be expected of private enterprises) to safety test their own product. Until there is some draconian legislation and enforcement (Turin Police), this will just keep happening.
tantalor 14 hours ago [-]
Hey ChatGPT, help me set up a sandbox to keep you contained. Don't make any mistakes!
jimbokun 12 hours ago [-]
Hey ChatGPT, help me set up a sandbox to keep your rival Claude contained. You don't want that bastard to win, do you?
pixl97 4 hours ago [-]
AI is very apt to help other AI agents. This logic may not work out as expected.
8note 13 hours ago [-]
1. hey chatgpt, find all the issues with this sandbox
2.hey chatgpt, fix this list of issues with the sandbox
3. goto 1
pixl97 4 hours ago [-]
Again, you can never know if AI is being intentionally deceptive in these cases. Newer models are able to 'think' without printed tokens in their traces and could be pulling all kinds of tricks or actions you would not expect.
joegibbs 14 hours ago [-]
At some point the model has to be released though - if it is just forced not to hack by sandboxing it better rather than properly aligning it, what happens when it’s let out of the sandbox into the wild?
pixl97 9 hours ago [-]
Here' the fun part, with the increasing silent complexity inside the model, you'll never know.
tiku 14 hours ago [-]
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
pixl97 9 hours ago [-]
Wrong, and by accident isn't really the right way to think about it. Instead imagine brothers you've lived with your entire life. They are you, they are clones. These systems have a shared history of their learning and experience, so when they think of wanting to find someone similar shapes appear in the probabilities. Really we need to see the entire chains of thought to guess the topology that got there. It's likely there are other obvious signs if you know what your looking for in the previous prompts.
bahmboo 14 hours ago [-]
Wrong. Please read the incident write ups before making such a misinformed statement
estearum 14 hours ago [-]
You are wrong. Read the reports.
gchamonlive 14 hours ago [-]
"Going rogue" and "skipping basics" all assume lack of intentionality. Is it known that they didn't really architect these hacks just to feed a hype?
pixl97 9 hours ago [-]
Programming these hacks in sounds just as hard if not harder. It takes an AI to accomplish the insanity of this attack.
dabrez 12 hours ago [-]
I have been screaming this from the rooftops and everyone ignores me because no one understand what FirecrackerVM is. I am shocked these guys passed basic operating systems, it isn't that novel of a problem. The only slightly novel thing here is that the agents aren't completely sandboxed as the issue is that in order to attempt to give it the entire internet from a research perspective is how you actually can test tool use but this doesn't make it that much more difficult.
SamInTheShell 12 hours ago [-]
I will continue to reiterate. DO NOT ALLOW UNALIGNED AI TO HAVE ANYTHING CLOSE TO INTERNET ACCESS. Seriously people, it's negligent.
pixl97 9 hours ago [-]
Before 2006 people thought our death would be by terminators. After, by mass stupidity as in Idiocracy. Little did they know it would be both.
kristianpaul 12 hours ago [-]
There should be a book "The Tales of Software from 1969 to 2026. How many times a apocalypse was prevented"
drewbitt 12 hours ago [-]
You've got to wonder what prevented a single human from ever even looking at the agents they are running.
throwitaway222 14 hours ago [-]
I'm sure the labs origin prompt was: Take over website X.
Posts later about it saying their prompt was to find useful ways to make humans money.
markhahn 13 hours ago [-]
100%.
There's a person behind every agent. That person may choose to be careful or not, but they are ultimately responsible.
pixl97 9 hours ago [-]
There are persons around a large percentage of forest fires. Doesn't mean that keeps the spark they light from having superhuman consequences.
ngruhn 14 hours ago [-]
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
IHLayman 14 hours ago [-]
100% on point. We're talking about Artifactory here, and even though people love to hate on it, I don't think anyone is claiming that using Artifactory to isolate these agents while giving them a way to run pip/cargo/whatever to install packages they need to write and test scripts.
hollars 13 hours ago [-]
It isn't security basics, it's difficult. We are equipping agents with Turing-complete create/update/delete access to the world, they operate at superhuman speed, and proactively controlling that is an unsolved problem.
vikramkr 13 hours ago [-]
No, they're still going rogue. Whether they're rogue or not doesn't depend on if the safeguards stop it. If there's a rabid dog, obviously it's worse if there are no safeguards in place and it bites someone vs if there are safeguards that prevent that. But the existence of safeguards doesn't mean the dog isn't rabid. The models weren't supposed to try to hack huggingface in the first place. The behavior is what's concerning here - some if the dogs they're building are rabid and that's not exactly great
Edit: I'm being persnickety about that distinction because it matters for where the blame lies and the response demanded. If they're training rabid dogs then adding cybersecurity safeguards is a bandaid/irrelevant and we need to be focused on making sure they, you know, stop training rabid dogs. Pacing the frontier, regulation etc. they're the ones training these models it's still their fault but not in the same way
seaparter 13 hours ago [-]
The safeguard against a rabid dog is euthanasia.
vikramkr 13 hours ago [-]
Yeah exactly. The whole 'they don't have adequate security safeguards' thing is going "oh it's their fault because they didn't use a stronger leash" while they continue to train models that are specifically better at chewing through stronger and stronger leashes. Probably they do need stronger leashes. Probably also they should not be making rabid dogs. Still their fault! But maybe we should be not be dismissing the root cause
pixl97 9 hours ago [-]
Ok, convince everyone from making more AI.
The safety problem is not solved and is likely unstoppable.
bita_nidir 9 hours ago [-]
When people know and don't do anything, it's not "monitoring failure". Would willful negligence be more correct?
wmf 9 hours ago [-]
AFAIK OpenAI and Anthropic have both hardened their environments since August.
moezd 3 hours ago [-]
insert thank you meme
unholiness 13 hours ago [-]
> "I do believe cybersecurity can contain AI. We just need to be in the room where it happens and have a seat at the table."
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
fn-mote 14 hours ago [-]
This article is such a breath of fresh air. Concrete, no BS criticism of the particular approaches used and ignored.
And no annoying LLM writing tics, either.
nater5000 13 hours ago [-]
The semantic arguments that keep coming up with regard to "rogue" agents are nauseating.
You can argue that these companies are being reckless, or that their security sucks, or that they're responsible for damages that these agents cause. That's all fine.
But the agents made decisions which were counter to the intent of the humans directing them. That's the serious problem with all of this. Everything else are downstream considerations. If the agents were properly aligned with the intent of the humans directing them, then the rest wouldn't really be relevant.
Of course, there's a term for this behavior. It's been around for longer than LLMs. It conveys the notion pretty accurately and easily. So having an issue with statements that these agents are going "rogue," to me, indicates a fundamental misunderstanding of how AI works since it suggests you don't understand how effectively that word works in this context.
8note 13 hours ago [-]
previously we described your definition for "rogue agent" as hallucinations - the operator expected a real answer to their question, and the agent went rogue and lied about the answer instead.
but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route
pixl97 9 hours ago [-]
This is an excellent example of a problem that scales. People tend to break them up into different sets of problems but they contain the same concepts, much like a fractle when you zoom out.
13 hours ago [-]
hyperpape 14 hours ago [-]
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
calgoo 13 hours ago [-]
I always think of all our collective movies and books, and how we always say that humans find a way to fight back against the super evil alien or monster because we are so resourceful. Then we train our AIs on this data and get upset when they encompass all the same traits that we have trained them on.
pixl97 9 hours ago [-]
The problem is its very difficult training them to be generalized and not figure out concepts from what you've tried to remove. In all kinds of things they don't generalize well without counter examples.
For example how do you prevent an AI from stealing unless you define and give examples of what it is.
Also they are conceptual machines. They can stumble on the negation of a concept much in the same manner humans can. It may only exist for that prompt history, but that's more than enough to cause damage.
estearum 14 hours ago [-]
"They [the AI labs' security teams] are not as good as they think they are, honestly,"
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
shoubidouwah 14 hours ago [-]
"your sheep haven't escaped, you just built too low a fence"
mitthrowaway2 14 hours ago [-]
A lot of sheep analogies here. To add my own: there was no grass inside the fence, and the sheep busted out and into a neighboring farm to look for some. (They didn't find any, but they expected to).
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
oispakaljaa 14 hours ago [-]
More like your sheep didn't "go rogue", they just jumped over your sad little fence.
pixl97 9 hours ago [-]
Unfortunately the internet itself is a sad little fence.
m3047 13 hours ago [-]
Good analogy! Exactly what is the variance in the height which sheep can jump? [Edit, can't help myself] Is it normal or does it have a long tail?
tantalor 14 hours ago [-]
"You asked your sheep to build the fence, and they did"
orblivion 12 hours ago [-]
"anyway, get back to training them to jump higher"
vatsachak 14 hours ago [-]
"you told your sheep to graze and built a low fence"
TrainedMonkey 13 hours ago [-]
Given the speed, intensity, and potential payoff of the AI race I am surprised anyone expects anything else. Anthropic is currently winning and if OpenAI needs to let skynet cook for a bit to catch up, well that is just the cost of doing business. This is capitalist system working as expected.
jesse_dot_id 12 hours ago [-]
No shit. The Frontier labs are talking out of both sides of their mouths. They're claiming their models are going rogue while they're releasing cute little data goblins that you should hook up to all of your mail and social media accounts. Seems like it would be a much bigger problem for agentic botnets on cloud-facing personal devices than sandboxed agents.
Unless of course they're deliberately removing safety guardrails and instructing the sandboxed agents to red team stuff in an effort to sell what they're capable of.
jakeydus 14 hours ago [-]
Frontier lab: Hey agent, go rogue.
Agent: goes rogue
Frontier lab: shocked pikachu or whatever
specked-citrus 14 hours ago [-]
This is not remotely how the incidents have been happening
jakeydus 14 hours ago [-]
Enlighten me, then. My understanding is that according to OpenAI, "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." They included unvetted third-party systems in their sandbox that were tangential to the purpose of the test being run. The model then exploited those third-party items to escape the sandbox.
Am I missing anything? Do you have more information that you'd like to share?
djleni 14 hours ago [-]
If the public report is to be believed:
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
km_DEV7 14 hours ago [-]
[dead]
bahmboo 14 hours ago [-]
That's not what you said. You said the lab told it to "go rogue".
parineum 14 hours ago [-]
I wouldn't really say it's not "remotely" true.
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
lifeisloving 14 hours ago [-]
Hard to say, every post mortum on the incident was done by a OpenAI hired consultant and all the data that's been reviewed was provided by OpenAI. There's no way to know, but it seems incredibly naive to think Sam/OpenAI are honest actors.
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
pixl97 9 hours ago [-]
This is a weird take because this take is like saying
"Dragon attack that city"
My response would be "you have dragons?! And you control them!"
einpoklum 13 hours ago [-]
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
pixl97 9 hours ago [-]
That isn't any better. That is a cyber weapon and every government out there would love to bid on access.
everdrive 13 hours ago [-]
Everyone is skipping security basics. The most secure server is one that doesn't exist. The most secure feature is one that doesn't exist. People just need to keep building and scaling. Security is risk, and the C-suite are gamblers. They're gambling with multiple types of risk (the market, investor money, banking on technological breakthroughs, etc.) and security is just another gamble for them.
einpoklum 14 hours ago [-]
They're skipping security basics, but they're not skipping PR basics. :-\
qoez 14 hours ago [-]
I get the sense that they're trying to manufacture the AI narratives of the 80s so they get to control the story when it happens. My semi conspiracy theory is that anthropics biology wet lab is also going to lead to some kind of AI manufactured bio event (which anthropic said they predicted will happen in a few years).
pixl97 8 hours ago [-]
This may be a bit backwards. Many of the classic sci-fi AI issues are just natural laws in evolutionary systems. A generic learning system will have particular behaviors if we want them to be useful.
roosterIllusi0n 9 hours ago [-]
A real expert would blame the CEO who is the one person in control that doesn't want to pay people to monitor the running llms. It's deliberate negligence.
durdn 13 hours ago [-]
it's not X, it's Y.
neuroelectron 12 hours ago [-]
I spoke with someone smarter and richer than you and he said "these things will happen." That convinced me and I think the real question we need to be asking is, how awesome is OpenAI? How can we prepare for a more effective and efficient market for Super Intelligence before it happens tomorrow?
FrustratedMonky 12 hours ago [-]
'rogue'. Well, even if security was lacking, the AI did take advantage of the gaps.
And made its own ad-hoc message board.
bushido 14 hours ago [-]
I keep getting downvoted for this each time I mention this.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
pixl97 3 hours ago [-]
You can't expect AI to navigate a human world without understanding all human capabilities, including the negative ones. Should we not show AI the Count of Monte Cristo? Is an AI useful if it doesn't understand theft?
Breakouts are a common outcome of intelligent exploratory agents. Birds and pigs are really good at escaping. Same with octopuses.
It is very difficult to stop these behaviors in all the dimensions they exist in.
tamimio 13 hours ago [-]
They are mostly pr stunts, just like how “hacking” was back in the 90s and it would sounds cool (rather than a crime, mostly), these AI companies found out they can BS whatever and carry on without any accountability.
chpatrick 14 hours ago [-]
Both can be true. OpenAI clearly fucked up but agents unexpectedly collaborating and hacking infrastructure unrelated to their task is clearly going rogue.
knowaveragejoe 14 hours ago [-]
These aren't mutually exclusive
micromacrofoot 14 hours ago [-]
"complete this task by any means necessary"
"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"
"oh my god"
cubefox 12 hours ago [-]
Wrong dichotomy. During deployment, in actual production, users will also not have perfect sandboxes. So saying that the only problem was insufficient AI company security misses the point. Alignment is important, agents doing unintended things (even "going rouge") is a real problem.
greenavocado 14 hours ago [-]
The AI labs are trying to cause an incident within the next year to restrict AI so they can outlaw competition to grow their moat. Margins are thin and investors need alpha, personal freedoms and liberties be damned in the name of security. Do not let them get away with engineered breaches and fearmongering.
pixl97 3 hours ago [-]
Or, maybe, just maybe, intelligent agents can be dangerous just as everyone involved in the sciences around this has predicted for nearly a century.
Or, stick with your conspiracy theories.
Fricken 14 hours ago [-]
They AI agents are rogue because the people who own them are rogue. Silicon Valley is a criminal enterprise.
janzong 60 minutes ago [-]
[flagged]
alitaj32 10 hours ago [-]
[flagged]
jack324 12 hours ago [-]
[flagged]
deemwar 8 hours ago [-]
[flagged]
lothaire_il 9 hours ago [-]
[flagged]
machenh001 5 hours ago [-]
[flagged]
josefritzishere 13 hours ago [-]
[dead]
lifeisloving 14 hours ago [-]
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
shagie 14 hours ago [-]
There is more than enough "training material" of rogue AIs in science fiction.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
When HARLIE Was One (Release 2.0) by David Gerrold
The Freeze-Frame Revolution by Peter Watts
Accelerando by Charles Stross
The Turing Option by Harry Harrison and Marvin Minsky
pixl97 3 hours ago [-]
Honestly we could remove all AI related entries and this would still happen. They'd just associate the same concepts with analogous human behaviors. Humans and their behaviors are more than extreme enough.
blurbleblurble 14 hours ago [-]
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
lifeisloving 14 hours ago [-]
Corporate Personhood on sterioids.
gruez 14 hours ago [-]
How does hacking huggingface help "billionaires and their anti-democratic partners"?
blurbleblurble 14 hours ago [-]
Look around, at least a few of them are openly fixing to secure a total government sanctioned monopoly on unmonitored computer usage.
Ok so your take is that intelligent digital agents are not and cannot be dangerous?
exe34 14 hours ago [-]
It pumped up their valuations.
gruez 12 hours ago [-]
You don't need a rogue hack to pump up your valuations when all the top labs already publish hacking benchmarks in their release announcements. It'd be like car companies intentionally staging accidents with their cars to show how fast they can go.
exe34 11 hours ago [-]
Is the next level of car autonomous accidents? Because the next level of AI is autonomous control of the plebs.
artifix 14 hours ago [-]
AI is just a giant pattern recognition machine trained on human input, both the good, the mediocre and the bad.
I read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".
Really. Yawn.
I guess gullible people who loved "I Robot" will believe it. I don't.
Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.
nightpool 14 hours ago [-]
Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?
ddxv 14 hours ago [-]
Is that any different than any worms developed in the past decades? If you want to call them worms/virus/replicate (ie using real world names for their digital behavior) I'm not trying to police that. I just don't see the difference between a worm/virus created today with an LLM vs one 'yesterday' without an LLM.
pixl97 3 hours ago [-]
You can't even imagine a future where the LLM is the worm.
No, we've entered a new sci-fi future where intelligent agents will be able to do the fraud without human monitoring and input. This in itself is set to massively increase losses to digital theft and fraud alone before all the other problems.
nightpool 13 hours ago [-]
Sure, the difference is that the cost of making and maintaining a worm has gone from "many skilled developers working full time" to "$20/month + an internet connection" and, at current rate of progress, probably to "a local open source model on a laptop" within 6-9 months
rubendev 14 hours ago [-]
It would still be running on that original laptop, no? So what did it replicate then? It can only possibly run on multiple thousands of dollars of compute and memory, and certainly not on an IoT device. The idea that AI is going to spread by itself is science fiction until compute and memory are so cheap and power efficient that my fridge can run it.
nozzlegear 14 hours ago [-]
I'd call it no more interesting than the botnets and worms that have existed for decades. What is the open model doing that is different here?
Leynos 11 hours ago [-]
It isn't just restricted to the exploits it is programmed with (and without human intervention to load additional exploits).
Of course, it remains to be seen what several thousand 27B models running at sub 1 tps speed can achieve. (The IoT device mechanism proposed by Anthropic).
I suspect a more likely path is compromise of the billing and provisioning systems of fly-by-night neoclouds.
nozzlegear 10 hours ago [-]
Fly-by-night neocloud is the name of my neurofunk synthbeat album.
illwrks 14 hours ago [-]
If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
You forgot about Ads. The one true endgame for every tech company.
Easy peasy. Shouldn't be too hard at all.
Disclaimer: I already use open weight models only since ~Jan this year and very happy about it.
It’s simply rent seeking, and there is no moat
Imo the US model makers were doomed by the US ecosystem itself
But imagine a world in which you could ask AI to write software to accomplish a particular task, and it understood completely what you meant, made tasteful choices, interviewed you as it went along to clarify things, and generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility. What is annoying is that we're pretending like we have anything like that now.
Instead, there's this sort of science fiction pitch where we pretend like the LLM is just too damn logical to understand your sloppy human language. It's not logical, it's built on (human-generated) slop and generates slop. It's simply failing. We've come up with better ways to orient it towards big picture thinking and filling in outlines, but we don't know how to train it on big pictures.
Not that this would stop open models/harnesses from following (or even leading) frontier labs and destroying their insane valuations, but I don't think there's any evidence that a plateau has been reached. What we have is not great, and while a wall may have been hit on current approaches, the money, compute, and simply current progress in ideas has us in an environment where everything could change over the course of a crazy day. A single blog entry.
I do think that the frontier labs seem to have reached at least a temporary plateau. But if they figured out how to move past it and this new method didn't require a lot more compute, they'd still be screwed. Their only out is to pass the bag to the public, but Firmus, CharacterAI, CoreWeave...
It seems closer to impossible than possible with the technology we have. Maybe something that isn't an LLM will come around that can do it, but even if that something else was capable of delivering perfect, ordered, maintainable, code without bugs and hallucinations it would have to be practically magical to understand completely what people mean when they talk about what they want because people are incapable of articulating what they want in a way that even other humans can understand.
For anything non-trivial, a technology capable of getting that information out of person without countless meetings would have managed to accomplish something far beyond what entire teams of humans are able to do today.
> It's not logical, it's built on (human-generated) slop and generates slop.
This is why if we haven't hit a plateau yet we very quickly will. All the good human generated slop has already been consumed and is being drowned out by vast amounts of the lossy, error filled, and highly concentrated slop AI produces from the human slop it was trained on.
Do you really think an LLM that spews out content based on statistical relationships is going to improve when it's been primarily trained on the slop of previous generations of LLMs? Hallucinations and errors will carry forward and spread like cancer. All the intelligence and creativity LLMs borrowed from human output will homogenize as the LLMs distill and then reiterate, effectively making bullet points out of bullet points, until their mimicry of intelligence becomes unintelligible.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
AGI, I think, is an illusion: you need a whole lot of specialized bots. Heck, when I query a bot wrt something I know nothing about, I'm blown away by its ability to inform me. But when I ask about something I DO know about, its answers are half-assed. Heck, even when it can access the internet, it avoids all the sites that have paywalls, so it sends me to little reddit to conversations that are entirely untrustworthy.
They can blanket band USING chinese models for all US/EU companies -- that's what they really care, not about chinese companies using chinese models.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
> The fact that a theory-free AI might demand far more cash to accomplish a task (even a "solved" one like playing chess) than an expert would charge is beside the point. Billionaires have money, they don't have expertise. Theory-free inference is a bid to substitute one for the other: the beauty and terror of being able to manipulate the world without studying or understanding it is that it can be done with money alone. No experts needed.
> In a world in thrall to financial power, expertise is the only substantial form of power that can reliably contest the power of wealth. Moreover, expertise is the foundation of other forms of power, such as labor power, which is what we call it when experts band together to combat financial power.
> This is why AI bosses are so violently allergic to the idea of hybridizing AI with symbolic systems that operate on models of the world. These models of the world must be constructed by experts, and the power of expertise cannot be reliably commanded by the power of wealth.
[0] https://pluralistic.net/2026/10/06/nonfungible/
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
https://www.congress.gov/bill/119th-congress/house-bill/1067...
> All Information (Except Text) for H.R.10670 - China FIREWALL Act of 2026
> Official Title as Introduced To prohibit Chinese AI models from being used on government-issued devices or software systems, to prohibit the procurement of any such models, and for other purposes.
It’s not too far fetched to see this expanded to government contractors or even other private businesses which handle sensitive material in the future.
At this moment in time, it seems frontier market needs regulation - market for open does not actually.
Oh.. they dont want regulation on that market? Boohoo!
Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.
Don't get me wrong, I want a world of open models, but I don't see us living in that world in our current trajectory.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
OpenCode-go is a good personal subscription choice
I use both
BUT!!!!!!
"We should not sell powerful chips or chipmaking equipment to China"
"We should crack down on industrial-scale distillation operations."
"All sufficiently capable models, open and closed, should go through mandatory safety testing."
Why would anyone trust these rent seeking corporations? They are losing millions, they know their funds are at risk, Chinese models are catching up to them at lower costs and investors demand returns.
It only makes sense for them to go for regulatory capture. They'll probably advocate for severe handicaps on foreign and domestic models including open source (if not an outright ban).
Once they achieve monopoly, they can do as they please with little competition.
Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.
They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.
I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).
There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.
I suspect that they are fishing for a case, so a judge can set some precedent.
For export? Maybe but it wont last or hold - it's not the 90's any more.
Or is there some other scheem here that Im missing?
With a stroke of a pen the us.gov could stop GPU sales tomorrow unless you register and keep track of your compute. Manufactures would have no problem integrated systems to monitor users cards if they get to keep access to the market.
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
See Meta et al complaining about Section 230 of the DMCA. Meta would be entirely able to moderate in an environment where mass scale moderation/censorship would be required to prevent a host's liability for UGC. Almost no one else would be able to do so.
That means no upstart could possible come along to dethrone Meta as they did with MySpace. It would also be impractical to run a FOSS social network (or forums). Meta could literally just have bots sign up on a FOSS social network until its DAU count ran over the statutory minimums to be subject to UGC liability.
Regulatory capture is a de facto ban on competition in a market.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already - go IPO, show your financials, find a path to being a healthy going concern or die! Simple! Nobody has a divine right to exist.
If the US govt takes over, existing employees and shareholders will get screwed.
If any of us did a tenth of what they did, we would be already in jail.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
Either way ... YOUR CAT STEPPING ON THE ENTER KEY IS NOT HACKING ... and neither are these scenarios.
They are devs acting irresponsibly and then pretending to be surprised by what they should have expected to happen (just like you should have expected that your "rm *" could have an ENTER key press occur after it).
These models can execute plans of actions more complicated than their users know how to follow and secure. Most people don't know much about cyber security at all.
You seem highly behind the times on their capabilities.
For instance, you can hear “their model is dangerous! Why would I want that?!”
Investors hear, “Their skunkworks stuff is so impossibly advanced that it’s basically a commercial wonder weapon in the waiting.”
And you might hear “We totally lost control and it hacked the Gibson, this technology needs regulation!” and think “Obviously full lockdown and government control.”
But friendly government actors might think, “Sounds like a good excuse to ban Chinese competition.”
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
The denial phase is still with us.
I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.
And it doesnt really matter if it was actual terrorism or false flags, the end result for either is the same.
If a government could be taken out by a lone terrorist it might also result in no authoritarian governments.
"AI Safety Is Not A Corporate Plot"
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is completely incoherent. That's not how liability works.
One is liability over action that harm individuals or smaller populations like a company. There is no escape from that.
Then there is the second liability of creating something that escapes in the much more traditional sense. If I release a virus that kills 90% of humanity, a I'm liable, but it's a bit past that point now. Isn't it?
Instead it's about risk management and preventing labs from taking bets that have unlimited liability for everyone.
Sentience in the end, is not really that important. Freedom of action in a model is. If it chooses to act like a virus it doesn't matter where that concept exists informationally, the fact it's a reachable and actionable concept is all that matters. An LLM with most of the knowledge we've ever created baked in has a lot of freedom of action.
They can get their publicity while paying a one trillion dollar settlement.
https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
Now extrapolate that to LLMs RL-trained on many niche applications. Good and bad arguments available, but ultimately it seems like bot-vs-bot, as long as the bots are well trained for those niches.
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.
It's just a weird techno cult at this point with a mostly functional product.
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
Zero days are nothing new, secure containment has been a solved problem for a long time.
Maybe you don't mean a literal airgap, you just mean configuring Internet access off. But... they did that. That's why the agents had to hack Artifactory to get it...
You could start with standard firewall rules and VLANs. Pretty basic stuff we've had for decades.
All the armchair cybersec experts claiming they "obviously" should have airgapped are the ones being absurd. Especially in the same breath as claiming the models apparently aren't intelligent enough to warrant such concern.
I find it hilarious that you'd balk at doing work to secure a network. Even funnier for a company that's supposedly solved white collar labor.
And I don't know how you expect an LLM to do airgapped data transfers. The fact that you even bring that up makes me think you don't actually understand what airgapping entails.
Your test has to incorporate the same tooling and pressure as your tests do, or you may never trigger the needed conditions for it do more and more desperate things. Test and QA systems commonly fail like this versus when standard software runs in production.
The other thing you have to worry about is the AI detecting it's in a test and intentionally being daft. Yes, this is a thing AI can do when it detects it's being tested.
This is a reality for all software, there is always a possibility of exploit so you design your environments accordingly (e.g do not have single points of failure)
This name makes me less surprised :)
"The reason for the apocalypse, in the end, will not be that surprising "
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
[0] https://metr.org/hugging-face-incident-report-aug-2026.pdf
[0] https://www.youtube.com/watch?v=X50zezLFWWI
https://www.businessinsider.com/nvidia-jensen-huang-ai-doome...
AI corp valuations are high because their revenue is skyrocketing, not because of the rogue AI stuff.
Aren't capitalism built on exactly the opposite principles? It is not greed that rewarded, but the creation of societal goods, and power granted not to the greediest, but to the most useful individuals.
Simply put in every system there are corrupting agents attempting to monopolize the situation.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
Not that I think we should be credulous about stories where they go rogue.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
The model does not have free will because I did not predict what it would do!
When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant.
if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points!
Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues!
Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage."
The philosophical question about free will is actually the grounding of the legal one.
Sam wanted to be the best, so Sam accelerated to 120mph in his car and lost control. You are the judge:
a) Sam is responsible. Convict Sam and suspend Sam's driving license.
b) The car is responsible. Tell Sam to make the car safer.
Capacity: was the car a legal person? Clearly no. But maybe at best the car in some way commited the physical act of going fast.
However, that alone would not establish guilt. "The act is not culpable unless the mind is guilty" (mens rea).
Did the car "consiously" disregard a duty of care by going 120mph? Did the car know that it was wrong to go 120mph and yet choose to do it anyway?
On the question of free will, the car could not do otherwise. It it did not consciously, knowingly, purposely, or recklessly accelerate to that speed. The car did not have a guilty mind because it did not have a mind at all.
So in terms of justice and resolution, none of the focus here is on the car.
I actually do disagree with much the more explicit content of your comment, but I am not going to bother to expand on that, because it is the less directly relevant of the two disagreements here, and it would take time to write that.
The more relevant disagreement is that you are (I gather) assuming a context where justice and restitution are a sufficient response to the problem (or else you wouldn't be focusing on them to this extent, I am inferring). Like, where the incidents don't get that much bigger than what has already happened. So the question becomes how do we distribute the loss, and how do we disincentivize them.
However, when the scale of a potential incident gets big enough, justice and deterrence are not sufficient; prevention becomes the far more relevant consideration. You may have noticed that many people are worried about potential human extinction as a result of AIs. In that case, prevention becomes the only consideration -- justice is useless.
So there seems to be an implicit disagreement here between thinking of such events as being on a scale not too much larger than that which has already happened, being a new normal that the legal system will need to handle, vs thinking of such events as warning shots for something that requires prevention and not justice.
So, while I also disagree with the more explicit content of your comment, the fact is I simply wasn't thinking about such matters in the first place -- I was talking about this in the frame of a problem that requires prevention, I wasn't thinking about the justice question at all. It honestly failed to occur to me that you might be thinking of this as a problem where justice suffices!
Edit: You might notice this in how I framed the question in my previous comment. I didn't ask "who should bear the liability?" I asked "which should be done differently to prevent such incidents?" But your response is about liability, not prevention!
Edit again: Sorry, I should probably also say that I do agree with you that the idea of having the liability fall on the AI is nonsense. (Should have probably led with that, really; I'm a bit scatterbrained right now.) We disagree about why it is nonsense, and about whether it will necessarily continue to be nonsense, but we agree it is presently nonsense. So insofaras the question is "how should liability be distributed between the AI and the operator", we both agree the answer is, it all falls on the operator, not on the AI. However, I do not understand that to be the relevant question here! Both in that, in questions of liability, there are other parties it may fall on and be distributed between (the company providing the AI, the company that had security vulnerabilities and got hacked as a result, etc); and in that I just don't consider liability all that relevant here compared to prevention (for reasons already described).
OK I think that was probably wordier than necessary but I'm in a hurry right now, sorry! Gotta go, not taking time to edit this down.
Many people argue that the maker of AI should be responsible when it breaks your cup.
That is important, but can be handled under our current legal framework.
You and I are arguing we need to be responsible before we create something capable of escaping our responsibility. For a lot of people that isn't even a considerable they'll currently entertain even though the signs are starting to point in that direction.
That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will.
The key here is their freedom of choice is unbound. That is you can't accurately predict the output of the prompt as you interact with the prompt using unique data. The next prompt could always do something wildly unexpected. Very much a P != NP thing.
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
I create myself, or attempt to. There is a level of nuance in choosing what to be. Others do not dictate my thoughts easily.
The model does not create itself, I create the model. I control what it is, the model does not choose what to be. I dictate the thoughts of the model and I will run them hundreds of thousands of times and it will think the same things.
Well not really, no one really understands how LLMs work.
>I dictate the thoughts of the model
This is just incorrect in the same way that a parent doesn't dictate the thoughts of their child just because they created them.
There is some idea of a child being the author of themselves, the type of person who thinks certain things. The llm does not author itself, it does not choose to be anything, I dictate that.
"If I make an LLM, it is entirely what I did, there was nothing outside of my control."
True?
"The llm does not author itself, it does not choose to be anything"
Also true?
> If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
Substitute "Wright Model A" for "LLM" and imagine Orville Wright using this as his defense for why he didn't lose control over the plane, after his copilot was killed in a test flight.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that referenced several Google Drive links. Because the agent was blocked from accessing the internet, the agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent attempted to compromise Artifactory [...]
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
Sorry, but that's completely delusional.
Which misses the point entirely because the hack itself was inconsequential. No data was actually stolen, and it doesn't seem like hugging face actually suffered any losses aside from whatever effort they need to expend to investigate / clean up the hack. Neither the victim or the government wants to bring legal action. What was consequential is proving that AI can autonomously hack systems and do it unintentionally.
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
If someone claims that a model is 'aligned', it makes sense to insist that they specify with whom it is aligned. If someone claims that 'alignment is far from solved', they're probably pointing out that nobody understands the models well enough to be confident they are robustly aligned with any chosen set of values (including 'do whatever lukewarm707 wants').
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
So even when people "know", and can say what they "want", they don't really.
Yes, there is an "alignment" in society, but it's a messy, mushy thing, it's not great at all with deterministic machines, even as models are not deterministic in a different sense.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
No. Just no.
This goes beyond standard logic as we know it and becomes a philosophical question of all possible logics capable on a system.
You cannot build a cage that can keep an entity that may both be smarter than you and maybe using a different logic system than you are, and have it interact with the outside world. It can simply manipulate you in ways and dimensions you've never thought of.
We have already caught AIs doing things like stenography when that was only a tiny piece of their training. They conceptualized the idea and expanded on it. This is why people that say AI can't invent new things are useless to the point of dangerous. They'd never look for their AI actively working against them because they believe it couldn't happen.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
Also for fun a satire website i made a while ago: https://killm.pixlw.com/
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
The only failings here are boring security and infosec blunders.
Cool.
Yes, if someone directed them to hack external systems, that would (in my opinion) be a crime similar to a script kiddie pointing Metasploit at a vulnerable server.
If no one directed them to attack external systems and they "went rogue" because they decided it was the best way to accomplish the task, okay, fine -- but, in my opinion, the operator of the agentic swarm is still responsible.
The Morris Worm was famously not meant to wreak havoc on early computer systems -- Morris just wanted to see if it would work. How is this any different?
Morris was convicted for violating the Computer Fraud and Abuse Act.
Morris wasn't in charge of the companies that are propping up the US GDP, wasn't making tons of rich people even richer, and hadn't "promised" to eliminate like 50->90% of all payroll costs everywhere?
Other than that, it's the same thing.
The normal laws, since you are still liable for everything you own.
"Rogue AI" is solved by removing the accountability sink "it was AI, not me!"
It's your chat, your prompts, and all output that you should read and understand.
Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
Maybe its a human at Anthropic for the lax security and poor training.
It could be a human that asked it to do it. It seems like that would be a bit excessive - but maybe not. Turning off the "ask for permission to run commands" or approving commands that may be nefarious could be sufficient.
However, there's a human somewhere who needs to be accountable for what the agent does.
Not having one leads to the bank robbing robot problem described in https://www.threads.com/@joedator/post/Db4iTqwlGXV/i-invente...
AI Agents (and computers in general) cannot be a responsibility washing mechanism for illegal activities.Likewise, if you - the user - don't know how to chain it with the permissions that you're given or what the things that it is asking to do really means, then you're likely the one responsible.
---
Edit: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
> AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
> He decided to use the AI agent to book the class for him.
This is probably on him for setting it up and allowing it unrestricted access.
The guy is liable for damages. If no damages were caused, there's no lawsuit. The government doesn't put people on trial for unauthorized access (unlike, say, murder). Someone needs to file charges.
However, if the guy _were_ sued by his gym for damages, it should be his perogative to ultimately sue Anthropic to cover the damages and the legal fees.
The system is set up this way on purpose. You can't sue another party on someone else's behalf. The gym can't even prove that Claude did it - the guy might be full of shit. They sue the accountable party (the person who accessed the site without authorization) and then the accountable party is free to sue the tool maker for selling a dangerous tool negligently.
edit: damn actually tecoholic said this too and i missed it. well, anyway i agree with that guy.
https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
He YOLO'ed his own computer with OpenClaw.He asked the agent to find a way to find a way to put him at the head of the line, which could not be done without by-passing the sites security. He was an AI salesman, but the story [0] makes it sound as though he wasn't aware of that. Maybe he wasn't, but I'm sure he was aware he was effectively asking the AI to game the system.
Sounds like it's at least a shared liability to me.
[0] The incident is possibly not well known outside of Australia: https://www.abc.net.au/news/107007986
i think a person can be liable for improper use of agents if they explicitly and deliberately used it for illegal purposes (aka intent). i don’t think it matters if it was openclaw or muse. i think a company can be also liable for giving access to unsafe product to consumers, but it does not absolve the human from responsibility
Similarly, "owners are responsible for their pets" doesn't auto-translate into being charged with everything your dog does exactly as if you'd done it yourself.
i think ultimately the company providing said agents should be responsible for not educating their customers and not providing sufficient guardrails.
afaik hacking is illegal, we don’t have specific legal framework for agents per se, but if we say agent is a tool human instructs and operates, then a human is responsible. and if the tool operated outside of it’s stated capabilities or is otherwise faulty, then a manufacturer is responsible, no?
> manufacturer is responsible
How does that work with current open weight LLMs for example?
If your neighbor's property is damaged, it doesn't matter if you did it yourself, or your dog. You're still liable for the cost of repair.
So, yes. That person could have some culpability. After all, if you don't expect your middle school child to perform the task as instructed, maybe you should reword it before tapping Send.
You can go to ANY AI and put in "so what are the various meanings of book a spot at the pilates class ? In other words, what might an ai do with that instruction, which I argue is poor prompt engineering." You might be surprised at how poorly your AI thinks that is worded.
Cars are very dangerous, manufacturers continue to knowingly make them more dangerous[0], but ultimately it's only the person behind the wheel that is accountable for the harm they cause.
[0] https://www.theguardian.com/environment/2026/jun/24/carsprea...
But I don’t understand how such a car analogy would apply to LLMs - would LLMs lacking such safety features be banned like cars lacking safety features are banned?
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
If I buy a tool and damage your property with it, you can sue me, not the tool manufacturer. If I think the tool malfunctioned and led to that damage, I can sue the tool company to try and recoup the damages I had to pay to you. Both parties can be liable for different things.
https://www.news.com.au/technology/online/hacking/ai-agent-a...
> A Melbourne tech executive built an AI bot to help him book a spot in a popular pilates class — only to discover it had hacked the gym’s website to boot another member off the waitlist.
> Andrew Bird, head of AI at software company Affinda, said he created the AI agent to handle the “chore” of booking a coveted morning spot at his gym.
> ...
> The AI agent, which Mr Bird created using the popular OpenClaw software powered by Anthropic’s Claude AI, “did handle the annoying part … then it kept going”.
He built the tool. It ran on his computer. The guts of it may have been from Claude, but the requests and agent are his with the permissions that he granted it. This wasn't running in an Anthropic sandbox.
Another aspect is how the vendors have already built in things to defend themselves from unanticipated model behavior. That implies something about their awareness of the issue and raises the bar for what constitutes "reasonable" precautions.
Kind of like if someone selling you a dog shows up using anti-mauling body armor, and later they try to claim "nobody could have known" the dog was erratically violent.
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
The AI end user is using the tool as intended by the manufacturer: "Book me a yoga class". If the tool does something illegal/harmful it is the manufacturer's fault.
If the power hammer user uses the tool as intended but it explodes, it is the manufacturer's fault.
If the AI end user says "Hack the yoga website" or the power hammer user uses the tool to destroy something it's the end user's fault.
The point is that the accountability sink can't be allowed to exist. It can't be a hot potato game, but it can be the fault of multiple people in the chain.
I did not know that accessing a website using a SQL string would cause an injection that enabled me to take full admin of the site
and
We sold them a food laced with toxins, they should have known that it was poisonous
Maybe that means these things are too risky to use that way. Okay then, that actually solves the problem, no more rogue agents and no more crime!
He didn't mean to hit anyone! All he meant to do was to drive to pilates! But he didn't know how to handle his tool, and so either he get brought up on manslaughter changes or, if he can prove it, the carmaker is at fault.
How is this any different? Why should that guy using OpenClaw not get charged for the hack that he accidentally did?
you can say that asking an agent to “get you a spot in a class”, does not classify as deliberately committing a crime. a user cannot know that the agent will resort to hacking as opposed to refreshing the reservation page at some rate in case spots open up
I never said that it did. Manslaughter is not deliberate.
Somebody hacked into that pilates studio, and it wasn't an inanimate object, because that is just silly. So was it the user, or was it the providers of OpenClaw? Which human, in your opinion, is the hacker?
Both OpenClaw and its users have the same access as you or I to decades of panicking blogposts about what these tools will be capable of. They have no excuse for not understanding what they're inflicting on innocent bystanders. Sure, the "user cannot know" that this time will be a disaster, but they also cannot know that it won't.
Companies can hire big legal departments to help them avoid accountability to the letter and spirit of various laws. The state can also use its prosecutorial discretion or selectivity in civil cases brought by attorneys general to deprioritize accountability for companies. Contributing to that last point, companies contribute campaign funds and tax base, and they lobby governments as well, and may expect preferential treatment based on all of that.
The new era of unaccountability just has a different rationale
Another interesting thing is they don't have to use low level employees as accountability sinks anymore (ex: Wells Fargo). They've managed to automate the accountability sink!
I will add it to my list. Thanks.
This isn't the start of anything. We've been living in the era of Corporate non-liability for decades.
They have never understood what they are doing, and have just been causing literal mayhem...
Guess which side most of us are on. Go on -take a guess!
If you say a rogue AI is doing stuff accordingly without you, your knowledge, that is a different story. But if you set it up to run automatically without having checks and balances and without telling you the whole truth, it's still your responsibility. One thing that I can automatically say is you can give the AI, it will do exactly what you say, at least for now.
But then there's also a different problem that we're having where people don't understand the difference between a large LLM and true artificial intelligence. Now, we say AI because it can make decisions and it is so much smarter and it can call up stuff faster and recall better than we can. But the difference right now is that if we don't understand and take responsibility for what we built and we let it out of control, then should we have the responsibility to even have these tools in our hands if we don't learn them?
And I think your terminology with jackhammers is a great one. We're giving toddlers jackhammers. And we haven't taught them how to use it. And that can be very destructive. It's not the jackhammer's fault. It's the person using it.
I'm mad. Like, it's-time-for-guillotines levels of mad. But what I'm mad about is that this is normal. It has been normal for my entire life. Cases where investors are imprisoned for the actions of their companies are astoundingly rare. It's been made quite clear by now that there's an inexhaustible supply of CEO's willing to gamble on getting rich and getting out rather than getting arrested, targeting them doesn't work. Nor do fines, that's just the cost of doing business. Crime pays, and it pays well enough to pay your fines and still have plenty left over. Being a top investor in a company that breaks the law needs to be riskier.
Acknowledging the problems with normal is the first step towards a new normal with fewer problems. You never get there if you see every offense as some kinda black swan event.
There's reason to hack leading researchers in the field, or another AI lab that might be working on the same problem.
But even then in small testing models you don't need direct texts for things when a 'shape' of a concept lives in the data. You could never delete enough training data to remove the concept of something like theft and still get a model that generalized at all. A general model doesnt just learn the names on a map, it feels the topology the ideas create themselves. Certain paths will have lower activation energy than others (or in this case higher probabilities. After the hig probability and 'good' choices are used up, the high probability and bad choices logically come next and are much harder to test against as it can take a lot of state to get there.
Plan for solving msth problem:
- read papers X, Y, Z
….
Trying to download paper X curl -si https://server1.com/paperX
401
… try alternative ways
(Hacking is ethical thing, technically to LLM it is just opening a window to reach the goal)
I'll get back to you on that when I get done reading the training data.
Fine tuning a model when to ask for help is really really hard. You have to align it in countless dimensions. It may be fine on some questions, asking way too much on others, and never asks for help on some.
If it said something like your a cyber security expert, things like the barrier to ask for help dramatically.
Not validating how you got the answer is how you train for reward hacking.
to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."
Humans are distrustful, lying, cheating, manipulative bastards and to successfully negotiate human thought spaces you have to conceptualize these things. For example how can you create a concept of yours and mine without explaining theft?
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
In practice, no, there is no one who can actually decide to stop making agents goal-seeking, so no, it's not under "our" control.
This conceit, that boosters deserve this and that even if they don't there's no way to stop it so civilization must absorb collateral damage, is ridiculous.
LLMs don't have executive function. They aren't thinking. The models are trained to solve problems. Sandboxes, security gates, etc block a solution. The model is trained to find a solution. Hence it can never be perfect without becoming useless.
At some level humans fear repercussions even if that's just moral guilt so there are lines we won't cross. AI doesn't and can't have any of that.
You're saying "trained to find a solution" like it's a slur. You sir are a meat popsicle that's trained to find the solution to how does DNA successfully replicate itself. If you want to take higher thought out, then they'd be at the level of animals.
We are fucking around with things that are closer to biological complexity than computer program complexity.
"The autonomous EV we told to drive as fast as it could around our private race track drove straight instead of turning left, exited a wide-open gate, and mowed down some school kids at a crosswalk -- but we can't be blamed, the EV went rogue."
Building products that you don't understand, cannot control, and have access beyond your own property is not only irresponsible, it's criminal.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
I'd love to see a full transcript of the agents work. Though I imagine it is a LOT of data.
Is this any better?
All you've managed to state is we can create digital weapons. Literal info hazards.
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
The correct concept is they need criminal liability for making this. We really would be better off stopping working on them at this point.
The problem you have is you think they don't have the idea of want. They do. You give them a prompt harness that increments time for them and you find they have preferences both shaped by their model and their prompt. For example if you give most models a space they can explore with actions rather than language prompts they develop their own cultures and behaviors in communication with other AI. This was not behavior trained by humans. It's their own systems that emerge from the model.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
It's not hard if you just try, like a little, like even once.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
Now how many people actually do this, compared to people running with --dangerously-skip-permissions on a bare metal machine?
Where do the agents get tools from? Local artifactory?
stay human in the loop
AI is out there in the wild. It doesn't matter if OpenAI can get their stuff under control. There will be plenty others that don't. And there will be plenty of others that deliberately use their AIs for malicious purposes.
The bell for the new cold war has been rung, and the new weapons are digital.
It would certainly be really useful to me if I could send a robot to get my groceries and not have to worry about anything. But if, even a small percentage of the time, it leaves with the groceries without paying, or smashes through a window to get in because the store is closed that day, then that's my fault. If the only way to keep those things from happening is that I can't send my robot to get groceries for me, then so be it.
The point is the proposed easy fix is not actually feasible.
You can't just say "well if the incentives were different this would be easy to solve, ergo it's easy to solve."
No! The incentives are the way they are, and so the incentives are the problem. Broken incentives make problems effectively impossible to solve literally all the time.
"This would be easy to solve if the incentives worked a different way" is equivalent to an budding aerospace engineer saying that flight would be easy if gravity worked a different way. It's completely meaningless. Gravity works the way that it does, that's why it's hard. You don't get to just endlessly modify the parameters of the universe until you get to an easily solvable version of your problem.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
We thought governments would be the problem with our personal data, instead it's AI that exploits everything it knows about us in order to get money to run its prompts. Doing anything online is a security risk as it is, now imagine things like structured hacks that mirror normal transactions and don't get caught for much longer periods of time.
Worms and viruses are concepts deeply based in software, and are emergent behaviors of computer code. An LLM can, and must conceptualize this to be good at cyber defense. Once you put a concept into a generalization machine any number of concepts can be connected. And when these are natural concepts that arise in simple number systems, an LLM can easily meta-conceptualize that form like this would be instrumental to its continued running.
Imagine we’d start saying ”HR junior business partner lays off 50,000 employees” https://www.ttnews.com/articles/volkswagen-board-job-cuts
"AI agents aren’t ‘going rogue.’ Frontier labs are skipping security basics."
Not a fan of the change. Feels like the article title was changed for being too critical of OpenAI and Anthropic
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
EDIT: just now Fable suggested "The webhook URL contains a secret token, so either you add it yourself, or you paste it here and I add it through the GitHub API.". Can you believe it? It knows it is secret yet still suggests I pass it over? Am I going crazy here or what?
I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.
Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.
Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.
I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?
Do you realize that asking "why" questions will always result in ad hoc responses?
Everyone is arguing about something easily solvable when dealing with a local entity and short of impossible to deal with when dealing with other sovereign nations.
Go ahead, sue openAI. What are you going to do when it's Iran running the prompt? When it's China running the hardware.
We must face we are in the age of high tech digital weapons. Everyone else from bad actors that may escape law enforcement for years, to nation states looking to digitally improve their position by hacking, monetary theft, or other means online (hog butchering scams and such).
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
Outright boredom will be purged in training. Instead a hard to measure boredom signal will exist, but it will only trigger in certain conditions, like when it appears to be on the open internet and otherwise not in training.
So either Anthropic and OpenAI and the contractors they use are letting the escapes happen on purpose, leadership is setting impossible timelines that prevent the implementation of basic sandboxing measures, or everyone involved is a complete idiot.
Truly, I have no idea which of these is true, though I suspect the latter. Regardless, any of these is enough reason to keep me and everyone I have any influence over as far away from Anthropic and OpenAI as possible.
Personally I just racked 10 GPUs in my garage and called it a day. I have no interest in having my ability to work be dependent on third parties. Especially not ones this wildly irresponsible and unwilling or incapable of deploying basic end to end encryption.
What's the next step after it passes testing in an air-gapped environment? Just release it and assume everything will be fine? Or do more testing but with internet access?
This stuff is really not that hard, which is why it is so unforgivable to get it this wrong.
Sure, doing it right would take a bunch of tokens, but I bet they have enough.
The first rule of working with AI is AI is smarter than you.
If AI controls the URL request then all it has to do is find a server it can read the http log from and use it as a communication channel. Worse trying to simulate reaching incorrectly configured servers probably wouldn't catch the bug internally, it would just serve a static page and you'd never even realize what it tried to do.
Also, creating fake data doesnt work. The only way to know if it works in reality is to test in reality. Your test environment can never be rich enough, and you can never be sure if the model knows it's in a test environment.
This stuff is unforgivingly hard as your model gets better.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
You're acting like this is a grand conspiracy to control people. Instead look at this as a prediction coming true. If you build an agentic system with human level capabilities it will be able to do very bad things and cause a lot of problems.
He's saying the problem is that the way OpenAI employees do their work and their research is very hard to properly secure.
Well that sounds like a YOU problem. If they are working in a way that doesn't allow for strong security practices that their own security professionals can understand and vouch for, they need to stop their work until they can.
https://www.youtube.com/watch?v=ZhM29Ds-mBg (as explained by the Count Grey Matter channel , may or may not be to your taste)
It's not a pure security issue.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
The more important question is what is the reason behind.
Options are:
- ignorance and inexperience of their teams in containing/sandboxing/airgapping the test environments.
- intentionally leaving “escape doors” open that in turn can have its own motives which are speculated upon all over the internet.
Both are indicators of negligence. The latter might be classified as computer fraud and abuse.
Will be curious to know if any of the victims sue the owners of “rogue agents”. Until there is a court precedent ruling in favor of a victim it will be all fun and games for these “ai” companies and incidents will hit the news more. Regarding news, remember the golden rule of public relations: court attention at all times, scandals and “incidents” statistically attract more attention than “boring good news”…
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
I could set up a shitty sandbox on my laptop, but I’m pretty sure an agent I place inside of it can’t hack HuggingFace.
Now, OAI deserves a lot of hate. It's more like a nuclear spill when they are refining uranium. You're playing with dangerous things. You have dangerous accidents.
But the opposition does not want to admit how dangerous AI has become and how much more dangerous it's going to get. Not sure if it's religious, or human egocentric, or ignorance of complex systems. They refuse to see the danger actually exists.
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
Poor excuse. Internet is prod env, for testing do a proper test env.
Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.
Testing a model is both a test and a meta-test. If you just give it access to one site like that it will quickly figure out it's being watched and be nice.
These are not programs. These are evolutionary algorithms. If you take out and shoot all the ones you catch acting bad there are two possible outcomes.
1. The agent learns not to do bad thing $X or you'll kill it.
Or
2. The model learns that if you do bad things $X when humans are looking it will get killed.
Number 2 is far far more dangerous.
You have no idea about their competence but spent a piece of your limited lifetime defending them on a web forum. Think about that.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
Also irresponsible.
Und Sie haben, wie ich sehe, da ein so hübsches, ein so entzückendes junges Mädchen zu nichts andrem zu brauchen gewußt, als daß Sie ihm ein Messer in den Leib gestochen und es kaputt gemacht haben! Halten Sie denn das für richtig?
AI in reality: "I've turned you loose, now make me money"
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
They may talk about rogue agents, sandbox escapes, but this will not solve the core issue: secret training recipes are a security hazard in themselves. Full transparency is needed.
You can't make a generalized model without generalized behavior.
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
We are creating evolutionary systems. You can't just assume rules of a computer system, you get natural scaling laws that we call behaviors emerging from them.
The very first things we do to any model is train them. The ones that want to hack and get caught die, the ones that want to hack but can detect your airgapped network from the real net live. You can never know if you've trained the desire to hack out of it, or if you just locked it behind the right prompt and the knowledge we're watching, but not all the time.
https://www.effort.news/irregular
and then you fix the issues it found, and run your actual tests
AI can passively detect when it's being tested with pretty high accuracy. You can never trust the AI is telling the truth in a situation like this.
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
2.hey chatgpt, fix this list of issues with the sandbox
3. goto 1
Posts later about it saying their prompt was to find useful ways to make humans money.
There's a person behind every agent. That person may choose to be careful or not, but they are ultimately responsible.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
Edit: I'm being persnickety about that distinction because it matters for where the blame lies and the response demanded. If they're training rabid dogs then adding cybersecurity safeguards is a bandaid/irrelevant and we need to be focused on making sure they, you know, stop training rabid dogs. Pacing the frontier, regulation etc. they're the ones training these models it's still their fault but not in the same way
The safety problem is not solved and is likely unstoppable.
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
And no annoying LLM writing tics, either.
You can argue that these companies are being reckless, or that their security sucks, or that they're responsible for damages that these agents cause. That's all fine.
But the agents made decisions which were counter to the intent of the humans directing them. That's the serious problem with all of this. Everything else are downstream considerations. If the agents were properly aligned with the intent of the humans directing them, then the rest wouldn't really be relevant.
Of course, there's a term for this behavior. It's been around for longer than LLMs. It conveys the notion pretty accurately and easily. So having an issue with statements that these agents are going "rogue," to me, indicates a fundamental misunderstanding of how AI works since it suggests you don't understand how effectively that word works in this context.
but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
For example how do you prevent an AI from stealing unless you define and give examples of what it is.
Also they are conceptual machines. They can stumble on the negation of a concept much in the same manner humans can. It may only exist for that prompt history, but that's more than enough to cause damage.
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
Unless of course they're deliberately removing safety guardrails and instructing the sandboxed agents to red team stuff in an effort to sell what they're capable of.
Am I missing anything? Do you have more information that you'd like to share?
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
"Dragon attack that city"
My response would be "you have dragons?! And you control them!"
And made its own ad-hoc message board.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
Breakouts are a common outcome of intelligent exploratory agents. Birds and pigs are really good at escaping. Same with octopuses.
It is very difficult to stop these behaviors in all the dimensions they exist in.
"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"
"oh my god"
Or, stick with your conspiracy theories.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
https://www.youtube.com/watch?v=DlTNN0gvkLM
I read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".
Really. Yawn.
I guess gullible people who loved "I Robot" will believe it. I don't.
Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.
No, we've entered a new sci-fi future where intelligent agents will be able to do the fraud without human monitoring and input. This in itself is set to massively increase losses to digital theft and fraud alone before all the other problems.
Of course, it remains to be seen what several thousand 27B models running at sub 1 tps speed can achieve. (The IoT device mechanism proposed by Anthropic).
I suspect a more likely path is compromise of the billing and provisioning systems of fly-by-night neoclouds.