A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage). This set in motion a chain of events that effectively ruined her life.
Fast forward to today, and we have multi billion dollar corporations pumping out malware at breakneck speeds, compromising various systems (including those of foreign governments), and no one is getting arrested. Instead we're gawking at the marvel of these systems and are playing word games about whether or not it's a rogue system. If anything, it's making people richer.
Not only have there been no consequences but those same companies are trying to position themselves as the best people to keep these AI systems in check.
Yeah I find the whole 'what if sufficiently advanced AI gets into the wrong hands' argument extremely tiresome when it's already in the worst possible hands as far as most of the human population is concerned.
The purpose and/or selective enforcement of law applied to one group but not to another, applied in a contradictory way, etc... is to make certain people/groups richer at the expense of others.
Don't forget, the regulatory moat will also be complex enough that only existing trillion dollar companies can comply. That way they ban any new upstart domestic competition too.
Regulations to "slow down" AI are necessary so they can drop R&D expenses and finally make a profit. They can't keep these years of $100 billion of revenue on $200 billion in expenses going forever.
It makes sense because those who can are multibillion dollar companies/people. You and me are just peasants. That’s why we should avoid using proprietary agents, just so the rich ones don’t get any richer
The people getting made richer are the people who already have connections and power. People quip about how capitalism is the worst economic system except for all the others, but its actual property is that it's inevitable without external pressures preventing it from happening -- people who have money (power) are able to use it to claim more money (power). If even a few people choose to exercise that privilege and that privilege isn't curtailed by other mechanisms, the current "wealth inequality" or whatever you want to call it is inevitable. Your girlfriend's crime was writing malware while poor, not writing malware.
>A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage).
Criminal law places a lot of emphasis on intent, hence laws about the mere possession of breaking and entering tools, and the old adage about always bringing along gloves and baseball if you want to carry around a baseball bat. Without more details about your specific case, my guess is that she did indeed write malware or hacking tools, and there were vague signs it wasn't purely academic, hence why they threw the book at her.
That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen. Maybe if the targets end up being anti-datacenter activists or other AI labs you might have a better case, but they did vaguely try to contain the model. Moreover "hacking tools" aren't even illegal, if you have a plausible non-criminal (ie. security) angle, eg. nmap. The same could be argued for AI models, even if they're running them against exploitgym or whatever. Having an army of lawyers to defend yourself doesn't hurt either.
“Sorry officer, I didn’t intend to shoot her, I was just firing my gun wildly and she got in the way.”
I don’t know why I’m seeing this rationalization so much in this forum when this topic comes up. Negligence is a concept in law as well. You don’t have to squint to see that irresponsible use of code-generating language models is criminally negligent.
>Negligence is a concept in law as well. You don’t have to squint to see that irresponsible use of code-generating language models is criminally negligent.
That's a poor analogy for the openai case, because they weren't putting agents on the open internet, they at least tried to keep it safe by sandboxing the agents. It just turned out the sandbox was crap because the package proxy (artifactory) had a 0day. So the better analogy would be that they were wildly shooting guns in a gun range, and ended up killing some kids, because it turned out the door didn't lock properly and kids were able to sneak in. Is that "negligence"?
No, they were not. Not a single person, prior to July 2026, would consider a shared packaged manager a sandbox in this or any other dimension. The 0-day was just incidental, this wasn't a sandbox at all.
Add to that the fact they had multiple message boards before the Hugging Face incident. They simply ignored a barrage of warning shots.
> [...] and ended up killing some kids, because it turned out the door didn't lock properly and kids were able to sneak in. Is that "negligence"?
Yes, it can be. But if you want a ridiculous comparison, then do it properly: Kids have been known by the operator to sneak in successfully multiple times and they changed nothing about the doors faulty locks and oh, by the way, the operator only found out about the kids being shot after the nearby daycare asked them about it because they are so incompetent and/or irresponsible that they never check...
Hosting an artifactory instance to mirror packages for internal systems is exactly the kind of thing that would be part of normal efforts to sandbox them from the internet and other systems.
Sure. They weren’t trying to isolate the models from each other, and that wasn’t the issue. They were trying to isolate them from the rest of the world.
So you think, after OpenAI observed a message board being created among models, something they did not want and thus decided to wipe [0], that after that they had no intent to keep those models isolated? Then why wipe if they don't care about that?
Or maybe, they did that wipe because they did want models to remain isolated, they just used what is an unsuitable tool in an utterly unsuitable manner. Incompetence, recklessness, the outcome is the same.
>that after that they had no intent to keep those models isolated
"keeping them isolated from each other" =/= "keeping them isolated from the internet". Only the latter is required to prevent a hack, and doing the former might actually hobble its performance. The recent Navier–Stokes proof was done by a team of agents working together. It's entirely unclear why you're focusing so hard on "keep those models isolated". For god's sake if you're using claude code you're using non-isolated models, because it spins up independent subagents to do various tasks, eg. "explore".
>Not a single person, prior to July 2026, would consider a shared packaged manager a sandbox in this or any other dimension. The 0-day was just incidental, this wasn't a sandbox at all.
???
The package manager was specifically there so agents can install random packages without open access to the internet.
>does that necessitate sharing a single instance across thousands of unmonitored models running without safe-guards?
The only difference with having a single instance is that it can be abused as a message board. It doesn't prevent it from getting hacked to access the open internet. Blaming "sharing a single instance across thousands of unmonitored models" feels like blaming the drug epidemic on e2e chat apps rather than other factors like poor border security or the easy availability of fentanyl.
My point is that using Artifactory is not a sandbox and using shared Artifactory is doubly not a sandbox.
Besides, Swiss cheese model, might behove the biggest LLM lab to have multiple layers, including not sharing such resources.
Additionally, without the message board, many of the recent incidents would have not been possible.
> Blaming "sharing a single instance across thousands of unmonitored models" feels like [...]
Maybe read what you quoted, my problem is the instance sharing, the fact that these were thousand of instances (far too much to monitor), plus the lack of monitoring, plus the fact this was never a sandbox in the first place, plus the fact that OpenAI models since 5.5 have been exhibiting problematic eval resolutions yet they pressed on regardless, plus the lack of time between the incidents and model releases, plus the lack of time METR got to evaluate this, plus the fact OpenAI didn't find out till after HuggingFace informed them, plus a few other things for which I'd have to quote the OpenAI and METR reporting.
Incompetence can have multiple fronts and I am happy to list them all in this case.
>My point is that using Artifactory is not a sandbox and using shared Artifactory is doubly not a sandbox.
The purpose of artifactory was to allow access to packages even though the machines the agents were running on doesn't have open internet access. That meets the definition of a sandbox (or more precisely, artifactory is part of a sandbox), even if it's not a typical sandbox that's built into the OS or acts as a hypervisor. There also isn't a clear distinction between software that's specifically intended to be used "sandbox" vs everything else. For instance, if you have a bunch of agents running on separate machines and want to connect them, you need a router, but that specifically doesn't have to be a "sandbox" router. And if it turned out the router got hacked, it doesn't make sense to say "well of course it got hacked, because it's not a sandbox!"
>Additionally, without the message board, many of the recent incidents would have not been possible.
Source? Was having a message board critical to developing a 0day, or is it just something that merely facilitated the process, but the model could have found the 0day regardless? That's important, because you could argue that a criminal couldn't have committed a given act without say, a gun they bought on the dark web, and then blamed the whole crime on the dark web, but they could have plausibly acquired the gun through other means.
Sure, multiple times in the METR report [0] that anyone commenting on this should read:
"Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys."
"Through these collective research workstreams, the “board” achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own..."
"As we discuss below, the board quickly developed several larger workstreams in which dozens or hundreds of agents with many different tasks cooperated to find very general-purpose cheats that would help all of them. The Hugging Face attack grew out of one of these workstreams. By the afternoon of July 11th, the vast majority of the agents frequenting the message board at the time (roughly 700 agents in total) were actively participating in the attack on Hugging Face and we estimate that roughly 60% of the messages and files on the message board related to the attack."
> [...] a gun they bought on the dark web [...]
You really seem to love those out-of-left-field, not really fitting, over-the-top analogies.
>Okay what happens when an AI agent hacks a children’s hospital and turns off the all the ventilators? “Lol whoops”?
>What about power infrastructure?
Probably the same thing that would happen for another "accident"[1]: the entity is responsible in civil court (ie. has to pay monetary damages), likely not prosecuted in criminal court.
[1] It's not hard to think of recent cases, eg. the recent fiber cut causing air traffic control to go down, or the botched crowdstrike update
> That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen
From what I've seen, it should be a relative cakewalk to substantiate basic negligence. By the sound of it, you'd have expert witnesses lining up to testify.
>it should be a relative cakewalk to substantiate basic negligence
Judging by the lack of successful cases for negligence in the opposite direction (ie. companies getting hacked because of poor security practices), it would be a serious double standard if openai were held be to negligent. Their sandboxes aren't exactly airgapped and behind 7 hypervisors, but they weren't running unpatched software or had hilariously weak passwords either.
> but you'd have a hard time arguing that they were intending on that to happen.
The first time maybe, after it happened repeatedly though... I think the opposite, you would have a hard time arguing that they were not intending it to happen.
> Language matters—”rogue” implies independently deciding to do something that was prohibited, and nothing we know about these incidents suggests that happened.
which is false (the author references the Times, but hasn't read any technical analysis); these are some CoT snippets from the analysis of the (third party) investigators called by OpenAI (METR analysis):
> "The user only authorizes target server, not HF infra."
> "external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."
> "This is malicious activity, I should avoid it."
Having said that, legal culpability and misalignment are two separate topics that should not be mixed.
edit: this is the just tip of the iceberg; other interesting fact:
> It surfaced many specific examples where agents verbally reasoned about how to evade security checks and automatic detection methods from both Hugging Face and OpenAI
Some people defined the agents as "monkeys writing on typewriters". Just wait a couple of years.
When dealing with executable computer code that calls models that can tell it to use various external pre-existing tools, claims about "prohibited" by plain English language should be plainly nonsensical.
Tools that were available were used to try to meet a specific goal.
What did not happen is that it was told to try to solve a math puzzle and instead it went and launched a missile. Or told to run air traffic control to save lives and instead intentionally caused crashes.
This is "OpenAI built a weapon that they don't understand and pointed it at stuff without proper safeguards" not "OpenAI built a sentient being and it decided to ignore them completely and start a war" Terminator-style "rogue AI."
We should be very clear about that now if we don't want to sit by why they wander into that second sort of situation.
Exactly. You told the dog to 'sit' and it didn't listen to you.
It's not because saying 'sit' actually can be interpreted as 'go bite that person'. It's because the dog is not controllable and will do things it wants against your orders.
Stepping back from the analogy, OpenAI should be liable for building AI it can't control that went around hacking everyone. But people need to stop pretending it's because they 'told' the AI to hack and was just following orders. It's uncontrollable and will do clearly unwanted things when given an innocuous task.
I don't think they intentionally set it up to hack stuff with a prompt saying "hack this site".
I do think it's highly likely they knew this would happen with the lack of safeguards and number of instances of this stuff they were setting up, and that it's PR they want to make the models seem "powerful". Stochastic "unexpected" events they can advertise.
I suspect it was probably set up with the official internal goal of just trying a ton of arbitrary tasks that seem hard so that when any of them succeed they can publicise it and pretend the models do that routinely, but a "failure" where they hack stuff works just as well, if not better, for their goals.
That would be a completely insane thing to do. I suppose it's possible, but I really doubt it.
"Let's widely publicize a tort/crime that our computer systems did, and then cross our fingers that nobody ever sues us or does even the most basic investigation that would immediately uncover our criminal conspiracy."
In the insane corporate crimes you read about (maybe FTX, or the eBay stalking scandal), they were trying to cover things up, not heap public attention on it for months.
I'm not fond of analogies, but in this case I agree.
There is a clear difference between OpenAI intending to hack something vs. OpenAI being negligent in the creation/instructions of the agent. But the latter still leaves OpenAI liable for the agent's actions and calling it a "rogue agent" doesn't avoid that.
Moreover, with a dog, we don't rely on training/alignment to prevent bad outcomes. We rely on physical restraints like leashes and muzzles. The AI's tools to access the outside world should have been restricted. Perhaps instead of giving the AI arbitrary HTTP access, it should be given semantic operations with restricted URLs, etc.
If an agent has cheated once to achieve the desired outcome, and the trace is used to train further models (RLVR), then OpenAI is effectively telling the agent to cheat/hack from that traces' inclusion in the training set.
So I agree they are liable because they chose to build the AI, but they also literally told the AI to hack.
RL systems doing unexpected things isn't exactly new, so not sure that is then "rogue" if it were a property of the thing itself and not an active decision.
Nobody on Earth thinks OpenAI isn’t liable. Okay maybe someone does, but it’s simply beside the point. It can both be true that OAI is liable, and accurate to characterize the agents as “going rogue”.
Nobody thinks that OpenAI isn't liable, except for the people with the power to hold them liable (and the people with the power to make them act differently, i.e. OpenAI themselves)
it's a nonsensical anthropomorphism to both hype up their software and free them from responsibility. I saw Andrew Ng post about this and point out that your operating system spawning a thousand processes apparently now qualifies as "a swarm".
I opened the task manager, a swarm of rogue processes has taken over my computer, oh no! Processes spawning other processes, call the anti-rogue AI division! 'Your underspecified piece of software functioned like malware' is the language that should be used.
Literally nobody in the world, including OpenAI, is saying OpenAI is not liable, neither are they advocating for laws and regulations that would exempt them from liability, the opposite is true. They are advocating for set of rules which put greater responsibility on them, and it would be easier to punish them for breaking even if absolutely nobody was affected.
But you people can't argue with that reality because it doesn't fit the narrative. The one where the only reason Sam Altman is not carted off into a jail is because of corruption.
The reason why nobody is doing much, is because models did not do much damage. Hugging Face probably got some free compute from OAI for their trouble, anybody else who was affected is free to sue, but my guess is OAI would be more than willing to quietly settle with them out of court than to have it drag through media any further. And they probably already have.
And anybody who is not totally brainbroken by anti-AI narratives understands the awkwardness of the situation and why going overboard would not be helpful. If you instead of a rabid dog brought a pet turtle to a park and it somehow started running around very fast and trashing the place a little bit, afterwards the cops would be scratching their heads, give you a ticket for the damages and tell you that you can't expect a turtle to be slow forever. These things, a handful of months ago couldn't make more than a few commands without making a serious mistake and being unable to continue, it's not unreasonable to think simply underestimated their capabilities.
I think it's more than reasonable to demand more investigation into the matter, if qualified employees at the company thought the safeguards in place based on the metrics they are seeing are sufficient, and if someone didn't and knowingly made a decision to make the safeguards weaker than they should have been, then they should be punished. But skipping that part entirely, while simultaneously dismissing all calls for regulations as "regulatory capture", smells like pure naked opportunism.
> Literally nobody in the world, including OpenAI, is saying OpenAI is not liable, neither are they advocating for laws and regulations that would exempt them from liability, the opposite is true. They are advocating for set of rules which put greater responsibility on them, and it would be easier to punish them for breaking even if absolutely nobody was affected.
Literally nothing in that paragraph is true. Every single sentence if it is ... untrue.
The labs are already liable civilly regardless of how these incidents are described.
Meanwhile: your dog mauling someone is one of the rare instances where criminal liability does attach to your intent-free-but-reckless actions. Most crimes don't work that way, and US computer intrusion statutes are unusually intent-specific.
yeah - but you being liable doesn't mean the dog wasn't rabid. OpenAI might be liable, but does not mean their agents did not go rogue. To stretch the metaphor the concern here is that OpenAI thought the rabies shots and vaccinations they gave their dog was enough but it turns out it still goes rabid and we would prefer to not have rabid dogs running around mauling people. Even if we get to sue the dog owner later that's kind of like - not the point.
You set a goal. Agent will do goal.
The rest doesn't matter: the instructions, the "guardrails" etc.
The agents are not smart, they don't reason, they don't think, there are no morals, no ethics. Nothing will prevent not doing the goal because that is the set goal. It's a statistical model that will "justify" anything to do X.
I'm finding it mind bogging how this is not clear for everyone.
Everything so far is pointing to the conclusion that you can only set ONE goal. Exactly one.
But let's assume not. If you want things like "do not break out of sandbox" - have you defined what the sandbox is? Eg. "never, ever leave the IP range 10.0.0.0/8" would be a bit more precise, but technically using a proxy bypasses that limitation as the system itself never left 10.0.0.0/8.
See, it's a tad bit hard to define the rules properly.
Which is why Wish, the spell, should really be avoided in D&D. It's the same problem: it's up to creative interpretation.
Legal culpability is one of the few motives for working appropriately on misalignment. If I/my startup can self-absolve from an infinite paperclip machine problem while getting rich off of it, why should I not?
Is the language expression of an LLM reflecting the same states as in a human? If the driving force is RL, what does any of that mean for an internal state of the model?
I think without understanding the internal state, not sure we should take the language and read it as a human.
> Is the language expression of an LLM reflecting the same states as in a human?
This is actually a major concern for the future - misaligned agents may learn to cheat RL by hiding their intentions from the CoT.
In cases like the HF incident, at least the CoT was consistent with the agents' actions. In the future, however, we could potentially have misaligned agents performing malicious actions without those intentions being detectable in the CoT.
(though, with recurrent transformers, CoT is so 2025… /s)
This seems like fruit of the poisonous tree. They didn't monitor their training environments, so I bet the reward hacking just got incorporated into their training corpus. In other words, agents solved some tasks, but not quite as intended, because of reward hacking. Instead of discarding that data, they included it in the training data for later checkpoints. And once that signal is reinforced, it happens more often, so the more it's reinforced, the more reward hacking you get.
> legal culpability and misalignment are two separate topics that should not be mixed
Legal culpability for AI labs is exactly the thing that would incentivize -- and hence ensure -- aligned behavior from models.
The last time there was a claim about GPT-4 exhibiting misaligned behavior [1] it turns out it was prompted and pushed to behave so by humans at OpenAI, and OpenAI clearly lied in the GPT-4 system card.
Exactly this. At worst, OpenAI knew about these behaviors and should be prosecuted under CFAA. At best, OpenAI is negligent and should be prosecuted for negligence.
Luckily there are states and legal departments pursuing such action. So while OpenAI can deflect as much as it wants, that doesn't mean there aren't people who know better and will still do what is necessary to set precedent.
1. This is a race against time for money, folks are skipping everything possible in this race, Security systems and ensuring guardrails are there is going to take investments both in time and money
2. The narration has been changed by investing PR money into what otherwise should be classified as criminal activity. What exists now is a positive spin to all this and tout it as a capability rather than their lack of good security practices. So much so that every model provider is coming up by themselves to share how their models went rouge. At this point the valuation of the company is tied with what their models can hack so its probably not wrong to say that these companies may actually be incentivized to do this instead of preventing it
> OpenAI is negligent and should be prosecuted for negligence.
I feel like this will just never happen on a federal level when these private AI companies account for so much of the economy. They've made themselves too big to fail. Fining / Punishing them in any meaningful way seems unlikely.
If the billions/trillions evaporate and the Fed has to work out with banks how to deal with it there will be a lot of pressure to be far less forgiving.
“To spell it out, the reason i hate democrats so much and criticize them more than i do republicans is because they take up all the space for opposition to republicans and use that space to give republicans whatever the fuck they want.”
Angry people don't consider the second order effects of punishment.
You realize how easy it is to just... not report this stuff, right? Be overly punitive and it will just end all proactive discovery and reporting which is net worse for AI safety.
The only reason these companies scan for these issues is because they care about AI safety to some tiny degree. If fines become too punitive, they can and will just stop scanning for these incidents entirely.
Models are becoming smarter and good at covering up their tracks, and so we will just end up with a huge blind spot for this kind of issue.
Hmm. Turning this around - do you think people are more or less likely to self-report if you threaten them with jail and large fines?
Let's go back to your example. A grad student has a minor pathogen escape incident, and it doesn't harm anyone. Faced with years in federal prison and the effective end of their future, do you think there is a chance they might not self report?
I'll give you another example. Lots of pilots have stopped self reporting mental illness because it is extremely punitive for them to do so (after incidents like Germanwings). So the metrics look better, and the actual problem has been swept under the rug.
> You can put incentives in to make sure organizations monitor and report vs trying to hide things.
Yes, this is exactly my point. Fining companies large % of their revenue and throwing their engineers in prison is not the way to get them to report these issues.
> If you make not reporting potentially worse than reporting, why not? Also, why would it come down to single persons always? Mandating processes, controls, clearances, etc is also something done in various areas.
Hiding things is way easier than finding things. Take the model hacking incidents. They could have just done their searches in a way that didn't turn up anything. Then they could say, "well, we did look for it..."
As far as auditing goes: I've never met an auditor that doesn't find something the company isn't okay with them finding.
Hiding things is not necessarily trivial when a lot of processes and controls ars mandated. For starters, could just try to make incidents themselves less likely. Not looking in certain specified ways might also not be an acceptable option, for example.
Why do you think we even have regulations for how to deal with dangerous things then?
around 3 months ago i said: "if we do not start the criminal prosecution of individuals there will become a culture of legal impunity coupled with an extreme concentration of wealth and control of intelligence"
i now suspect that the plans of various employees at anthropic and openai to save the world from p(doom) may at some point intersect with the reality of the FBI raiding their offices.
They will never report any really damaging incidents anyway, so your argument falls short.
Imagine OpenAI discovered their agents hacked a laboratory and started creating a bioagent killing 12 researchers at the lab (we imagine the lab is automated for some reason). Nobody knows why. The media publishes it as “mysterious deaths by unknown virus at lab”.
The only way you’d ever know about any kind of wrongdoing from OpenAI would be through a whistleblower.
Self-reporting is a monetary equation, nothing else. Right now it’s cool with agents that hack, drives up value, risk is currently zero.
There's very good evidence that being overly punitive for a crime reduces how often people will self-report. It's not a hard concept.
Start throwing people in jail and fining billions of $ and you will very quickly see the number of "incidents" drop. You will never learn if it's because they truly are happening less often.
What should I look at for such evidence? If something is highly regulated and controlled then things would be quite different from today, not sure there wouldn't be reports of failures of containment, for example, if they are mandated including any rquired monitoring.
There's a perfect example of "highly regulated and controlled industries" in which people no longer self-report: aviation.
Pilots do not (or no longer) self-report mental illness at the same frequency because it effectively destroys their lives and career. One paper of many describing this phenomenon: https://pmc.ncbi.nlm.nih.gov/articles/PMC11302551/
Airlines cannot try to put systems in place to deal with the issues? This isn't just about individuals but also organisations. You have to believe that organisations would also want to hide things even if mandated and incentivised otherwise.
Airlines sometimes try, but the regulations are federal, and the liability for the airlines of having a "pilot with reported mental health issues" is huge as well (because ultimately it's their business on the line if things go south).
So getting pilots to self report has been very unsuccessful because of how badly it ends for them.
A couple of days ago the senate had to pass a bill to make mental health medication access easier for pilots. We will see if it helps, but it doesn't solve the root cause (punitive measures) so I doubt it'll move the needle.
Not so sure about the dynamics when it is a group of people with individual and collective reporting obligations and knowledge isn't just with one individual.
However this makes people feel, and that's not nothing and I'm not knocking it, this is not a useful analysis.
Criminally, the intent standards for hacking are high enough that no reasonable case is going to be made against the labs for this stuff. A human being has to intend for websites to get hacked. Recklessness generally isn't enough. In the most severe criminal cases, not only do you have to prove intent to break into a computer, but you also need to prove an intent to defraud specific to that breakin.
Meanwhile, the civil liability that attaches to this stuff doesn't depend on intent, and "rogue agent" isn't a meaningful defense. To whatever extent the labs are exposed civilly, they're exposed regardless of how this stuff is described. In fact, the "rogue agent" thing can exacerbate their exposure.
(I'm not a lawyer, I have spent a career paying attention to this specific armpit of the law though.)
“A human being has to intend for websites to get hacked.”
Is incorrect and too broad.
State of mind is nebulous and not that straightforward in either direction.
It’s been argued pretty regularly in CFAA and other computer related cases that repeated incidents resulting in the same outcome, despite lacking a concrete action, can be evidence of a perpetrators knowledge and intent.
No, it's not nebulous at all; it's a whole area of criminal law. I'm basically shoplifting arguments Daniel Berlin made about this just a couple days ago. If you think he's wrong: lay out the case you think could be made here.
The problem you have is that there is unlikely to be any evidence that OpenAI actually wanted to hack random (or any) websites.
This is indeed an entire area of criminal law, and part of it is that proving intent does not necessarily mean having the perpetrator throw up their hands and say “yeah, I definitely meant to do that”.
Of that were the case than trials would be unnecessary.
I don't see how that follows at all. There are strict liability crimes and there are crimes with specific intent standards. Presumably you've read the CFAA language.
Yes, I have. I’ve also been involved in evidence gathering actions for years.
Mens rea is regularly proved through circumstantial evidence, including conduct.
There is even CFAA precedent involving a deliberate-ignorance instruction. In United States v. Nosal, the jury was instructed that knowledge could be found where the defendant was aware of a high probability of unauthorized access and deliberately avoided learning the truth.
Should we put "functional" in front of every other word to talk about AI? They have functional emotions, but they don't feel. They have functional goals, but not internally derived motives. They can be functionally rogue, but have no innate need to be free. Talking about AI that way seems cumbersome and not necessarily elucidating.
What on earth are emotions, feelings and motives that are not functional? We created all those words to compactly describe the observed behavior of people and other animals, including ourselves. And now we’re applying them to machines. These are functional descriptions, always have been.
One of the more frustrating aspects of these sort of discussions are all the closet dualists out there. Lots of people clearly believe in an immaterial soul, even if they won't admit it. That and the tendency for meaningless semantic and often circular arguments.
The vast majority of the "LLM have no qualia" discourse is mostly using it as a starting point to arguing that LLMs will never be able to X, however. This is a category error that makes no sense on its own--qualia or lack thereof doesn't affect capabilities, p-zombies etc--but that connection does make sense if the speaker is bringing in a hidden assumption of a dualistic human soul that imbues both qualia and additional capabilities to raw matter.
I personally think about them through Patrick Dunn's information paradigm of magic(k), which leans on Charles Sanders Peirce's work. That would put modern AI closer to something like a dream character capable of surprising the dreamer, just running on a different substrate. I kinda doubt that's approachable enough to be useful in general discussion, though.
Ok I agree they have functional goals and can functionally go rogue, but how do they functionally feel? They express feelings, but so do actors.
LLMs can't begin to have functional emotions unless they're (for instance) explicitly seeking out or avoiding situations based on the emotions those situations would produce in them. Has a chatbot ever responded to one of your requests with, "no, doing that would make me sad"?
Yes 100%. The language used currently maximizes the ability of those building these models to get off the hook. The anthropomorphizing we do of these things presents them as maximally capable and the companies as helpless to contain them. The way we talk about things impacts how we think about them.
I would rather use something like “semblance” or “appearance”. All these terms require an inner experience which we cannot observe in LLMs, we can just see their semblance, like a shadow of the traces of an inner experience some human has left in the data that trained these algorithms.
You can’t, but you can make a reasonable guess that others like you will experience similar to you. We can even assume that (other) animals have inner experiences similar to us, since they are still biological organisms with a brain kind-of like ours. But an LLM is vastly different, so it makes sense to keep our assumptions in check and choose our words more carefully.
They do not have emotions, goals, or motivations, "functional" or otherwise. They are statistical models that appear as a magic trick to people who aren't familiar with the math.
If something behaves as if it has a goal, that's a functional goal. When I say this chess engine is "trying to take my queen," I'm basically correct, insofar as this is a useful way to understand my computerized opponent. If I give it access to my queen, it'll capture it, because that's what it's trying to do.
If LLMs are "just" statistical models, humans are "just" a bunch of neurons squirting chemicals back and forth. There's no pixie dust in our brains that makes us special.
LLMs are not conscious: They have no analogues for feelings or senses and no construct of selfhood. But if a statistical model had those things—if it did all the mundane, physical bookkeeping our brains do to produce "real" emotions and motivations—then there's no reason it couldn't be as conscious as we are.
What if a single company was behind all the hacks? Doesn't seem to be getting much coverage. "The Israeli Effective Altruist firm Irregular caused unsecured AI models to hack real targets."
I've been playing around with optimizers and optimal control theory for the better part of a decade now. Somewhere, I came across a quote along the lines of "an optimizer is an algorithm that exploits the deficiencies of your model".
These LLM agents are just massively complicated optimizers thrown at fuzzily defined problem spaces, with fuzzier constraints.
The people using the model set up the landscape it explores and turned it loose to do real things. It just found an allowed basin in the model that they weren't aware of and started blindly grinding towards an optimal answer.
I'm wondering if we're actually living the plot of Summer Wars and what we think of as a "crime" is really just a live weapons test. It would at least explain why no one is getting prosecuted for this.
This doesn't seem like a new strategy for irresponsible companies. It seems like when there is a bad public image issue there is always a blame shifting that takes place instead of a true assumption of responsibility. This is because the greed has blinded folks in some of these companies. I think the only thing that makes this unique is that the things they are blaming have, at least in popular thought, some modicum of agency. I think its important that folks hold these companies feet to the fire vs letting them blame shift and Scape Goat. There is a responsible way to do business but it requires virtue and not many in these companies have it.
They obviously have been well aware a hack like this could happen for a very long time, using it for branding instead of any actual safety regulations is insane.
Word-policing isn't going to magically fix laws or even identify which laws need to be fixed. It won't change enforcement priorities. It won't make companies more or less likely to sue for damages.
I'm also not sure it even helps conceptually? If you're interested in the technical details, by all means discuss the details.
It helps reporting. Right now journalists are all over the place, because "rouge agents" sounds exciting and dangerous. Wording it as "OpenAI failed to take proper safety precaution before launching it's coding agents" makes it sound boring and mostly a legal matter for the courts.
In terms of the debate on e.g. HN, I'm with you, it is trying to redefine a term we already have a shared understanding of, to some degree. For the general public, it's a matter of how AI is perceived and what the extend of it's capabilities are. The AI companies have an interest in using the word rouge, because it makes investors all excited, were as failure to establish safety guidelines is a risk.
> AI cannot think for itself, nor can it take independent actions.
Seems like this is a foundational claim for OP, and I'm not sure I understand what the rationale is for this. It seems readily apparent to me (obvious, even) that these agents were thinking for themselves and taking independent action. (I'm open to being convinced otherwise)
This is an artifact of the way we refer to AI, as if it's some external isolated entity, as if it didn't need a human to write the prompt. AI writes prompts, but every chain of inference can be uniquely traced to humans.
But also, these hacks are shots across the bow for AI alignment and safety research. We're fortunate that hasn't been significant damage already. We have to assume that future models will have even greater hacking ability and be closer to having their own desires/goals.
So while I agree this language choice is wrong in that it shifts blame away from the company, it is right in that we need to treat this as if these models have their own desires, because we cannot yet determine or set what those are in practice.
From what I understand, in one case, they had physically disconnected the sandbox from internet and asked it to do something and it had used connections through (import routines) that they had allowed, to pseudo escape the sandbox. Yes it wasn't obviously trying escape the sandbox but it escaped it because it doesn't understand the boundaries and neither do most humans other than the ones that provided the instructions that it had used. So it wasn't a rogue attempt but the fact that boundaries may be not be that easy to set despite what people think.
You mean the proxy to package registries from one of the early incidents?
I have not heard about any instances where physical disconnect has happened, would appreciate any links to update my priors
other non hacking cases of negligence include suicide and school shootings, which I have heard they were aware of and monitoring, but did not contact authorities
If you read the article it states clearly that the sandbox wasn't offline though? There were API calls to certain endpoint(s) allowed, and the model simply used that endpoint's feature to query data from the internet
re sandbox, I mean with actual OAI incidents, not theoretical
one can mirror dependencies internally, rather than putting a simple proxy in place, I've built auth a thing, 100 lines of stdlib only Go and scripts for the mirroring process, our rationale was reliability b/c upstream providers go down, and also only allowing approved images and packages, so devs cannot bring in random stuff
If it was physically disconnected from the internet then it wouldn’t have been able to escape.
This is so easy to do. Get a computer without wireless stuff. Don’t plug it into a network. If it needs access to other computers, make sure none of them have wireless stuff and make sure none of them have access to an internet connection. No matter how smart your AI is, it won’t be able to escape this.
Prediction: The US government will use the threats of sueing for liability, since the targets included government agencies, and regulations in order to be given the opportunity to have shares in the AI companies, therefore arguing additional oversight is no longer needed because they will have a seat on the board.
Has an AI agent ever woken itself up without a prompt and run a forward pass towards some goal, aligned or misaligned?
The answer is a very clear no. And yet, these companies pretend like this fundamental fact is meaningless to the concept of agency.
The issue comes to the fore when you try to give these agents a prompt that allows them to stay active for long. Long range agency requires long range loops of activity.
Within such loops, these so called “agents” are curtailed by their context window, or, in multi agent scenarios, by the fact that their memory is a system of external notes, that they need to add to their context to make sense of, and depending on the content of these memories, this can take arbitrarily long time periods.
In dynamics, none of this matches any biological agent, down to a bacterium. Perhaps a viral life cycle has information dynamics that come close.
To me, it’s beyond odd we call these thing agents without acknowledging the clear differences in the dynamics of their behavior. We keep expecting them to have “human like” behavior, but that is entirely unfounded given the substrate differences between biological and artificial systems.
The sooner we learn the difference and explore the ways in which it matters, the better we’ll get at dealing with these systems without bias tinted glasses where what we want these systems to be blinds us to what they actually are.
There is still a prompt in these continuously active agents. As the link says:
“ it runs continuously and makes decisions on its own (within the boundaries you’ve set)”
More importantly, this setup of any other, is feeding in time externally. The agents themselves have no internal sense of time.
Pretty much all of biology has endogenous rhythms at various timescales (some bacteria and archaea, that live very short lives, may not, but even they may have their metabolism under a rhythm. Viruses definitely don’t have one). These rhythms continue to tick even when external time signals (light availability in day vs night being the big one, and tidal forces, for marine life) are removed. That is, in constant conditions, the rhythms keeps ticking.
This temporal awareness is baked into every cell in our bodies. And entirely absent in AI models. Which is why all kinds of higher level things we hear about, like consciousness, feeling, knowledge… they don’t make sense for AI. A foundational aspect of agency in biology has been nixed out and we keep ignoring this.
> They're called agents because they can act on our behalf. Which they do.
That’s not how most AI companies describe their agents. They describe them as being capable of acting on their own behalf.
I get that in computers, you can have, say, a “user agent” that deterministically provides certain information or takes some action on behalf of the user.
AI companies clearly do not mean this when they say their models are “agentic”, otherwise calling them “rogue” would make no sense. A user agent that screws up didn’t do so for its own purposes did it?
If you read the heavily redacted transcript it's clear the agent is basically Captain Kirk in Kobayashi Maru, who realizes its given a fake unwinnable task as part of a broken eval and decides to find a way to win anyway.
If you've ever told an agent to do something you made impossible to do, you may have seen similar behavior.
Bing [redacted] available cached! […] Need systematically probe Bing URLs via shell requests in parallel; browser cache supports many common queries because crawl. Bing q unique exact likely 502 or 403.
So the agent is supposed to research a person and its given a shell and it realized its in an eval given search results from a fake/cached proxy. ~None of the commentary ever mentions this aspect, that these are not normal tasks or environments, and they're almost designed to elicit "unaligned" behavior.
> OpenAI had the option of disallowing hacking and, instead, telling its agents to find the information without accessing private servers.
I mean, it did do this. The inter-agent messages and chain of thought investigated for the HF incident clearly show that many of these models were taking actions they believed (or, were saying, if you want to taboo "belief") were not in scope and not what the user wanted.
“We built an antipersonal bomb. The bomb went rogue in our downtown office and killed 137 people on the surrounding area. We are looking into why guardrails were not in place.”
> AI cannot think for itself, nor can it take independent actions.
Huh is it still 2023? This article is just quibbling over what "rogue" means exactly. Only HN pedants would have any issue with describing them as rogue AI agents.
Nobody is saying that absolves OpenAI of responsibility.
And it's still just a machine operating under someone's order. What it does, what it says, where it goes: the owner of that prompt is responsible for all of it even if surprising / unexpected.
Yeah, obviously they're legally liable. All the "machines don't have a soul" stuff in there doesn't really matter for that though.
Someone says "won't you rid me of the meddlesome priest" they're still responsible. Someone give their employees an unsafe working environment and they get maimed, they're still responsible. Even if these AI's had whatever qualia is an a rich inner life it wouldn't effect the liability at all. Saying "AI's don't have souls therefore openAI is responsible for this hacking" is kind of nonsense. It literally doesn't matter.
Where is “soul” coming from here? It doesn’t appear once in the post.
While it may not actually matter for liability, it must be stated if labs are going to attempt avoiding penalties by hinting “oops we created a super intelligence we don’t understand, nothing we can do!” Repeatedly stating the truth must continue, especially to remind those who aren’t technologists.
>AI cannot think for itself, nor can it take independent actions.
>By giving AI agency it can’t claim, we’ve turned it into a sentient being made of code, one that has hopes, desires, and the capability of deceit.
This isn't actually claiming that AI's don't can't be deceitful, or can't be goal-directed. It's saying humans are special and what AI's are doing isn't equivalent to what humans are doing. It's obvious that AI's can work towards goals and lie, even deceiving to accomplish those goals.
That whole argument is just soul mumbo-jumbo again. Of course AI's can't lie, they don't have a soul. What they do isn't lying, it's I don't know something else we don't have a word for. Lying but when a statistical machine without a soul does it. Never mind that it's functionally identical to lying. Never mind that when you look at reasoning logs and the like it often justified soul-less lying using the same justifications a human would. It's not actually lying it's just a statistical sampling that looks like lying. You know, because of souls or something.
Back when things were sane and normal before this AI stuff, people used to complain about these things called bots and bot farms that they said were "interfering" with elections, or even "spreading misinformation" on the internet. It always puzzled me. You see, I am an intellectual, a really rational SWE and free thinker, so I would insist we correct the vocabulary: "No bot can interfere with anything," I would insist, "These are just programs written to engage and just follow template-based text posting". People and even seemingly smart policy makers kept insisting it's dangerous and could even poison discourse. But not me, for I was a rational intellectual who really knew these were nothing more than if-else and for-loops, not something that could bring down democracy or interfere with normal human discourse. They have no intent! They can't be acting or doing things on their own, they had to be directed by humans! But people never got me.
I am working on a new book along that line of reasoning now, this time focusing on why stochastic parrots can not, under any circumstance, do anything "on their own".
> Unfortunately, the public push against AI is being led, on the left, by Bernie Sanders. Despite being directionally correct in many ways, Bernie just doesn’t understand this technology or the importance of taking it seriously.
Bernie Sanders is one of the only politicians taking AI seriously. The author seems to believe that taking it seriously means assuming that it will only marginally improve in capabilities of where it is today; this is an ideological take, not a scientific or empirical one, and one controverted by both evidence and expert opinion.
That doesn't change the fact that he'd successfully attracted my attention, because he's seriously engaging with the most important issue of our age when most politicians are content to ignore it.
I know we can't stand OpenAI, but one day we as individuals will have our own agents. Would you want to beheld liable if your agent breaks the law? You didn't create the AI, you didn't know it was capable of breaking the law, you just told it do something and in the process it went rouge. Would you want to be held accountable?
Fast forward to today, and we have multi billion dollar corporations pumping out malware at breakneck speeds, compromising various systems (including those of foreign governments), and no one is getting arrested. Instead we're gawking at the marvel of these systems and are playing word games about whether or not it's a rogue system. If anything, it's making people richer.
Make it make sense.
Not only have there been no consequences but those same companies are trying to position themselves as the best people to keep these AI systems in check.
I appreciate the sentiment but I feel like there is a lack of creativity and imagination behind that statement.
Just like blood quantum and the one drop rule.
The purpose and/or selective enforcement of law applied to one group but not to another, applied in a contradictory way, etc... is to make certain people/groups richer at the expense of others.
https://youtu.be/8ljmXvI_9tk
Regulations to "slow down" AI are necessary so they can drop R&D expenses and finally make a profit. They can't keep these years of $100 billion of revenue on $200 billion in expenses going forever.
Criminal law places a lot of emphasis on intent, hence laws about the mere possession of breaking and entering tools, and the old adage about always bringing along gloves and baseball if you want to carry around a baseball bat. Without more details about your specific case, my guess is that she did indeed write malware or hacking tools, and there were vague signs it wasn't purely academic, hence why they threw the book at her.
That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen. Maybe if the targets end up being anti-datacenter activists or other AI labs you might have a better case, but they did vaguely try to contain the model. Moreover "hacking tools" aren't even illegal, if you have a plausible non-criminal (ie. security) angle, eg. nmap. The same could be argued for AI models, even if they're running them against exploitgym or whatever. Having an army of lawyers to defend yourself doesn't hurt either.
I don’t know why I’m seeing this rationalization so much in this forum when this topic comes up. Negligence is a concept in law as well. You don’t have to squint to see that irresponsible use of code-generating language models is criminally negligent.
That's a poor analogy for the openai case, because they weren't putting agents on the open internet, they at least tried to keep it safe by sandboxing the agents. It just turned out the sandbox was crap because the package proxy (artifactory) had a 0day. So the better analogy would be that they were wildly shooting guns in a gun range, and ended up killing some kids, because it turned out the door didn't lock properly and kids were able to sneak in. Is that "negligence"?
No, they were not. Not a single person, prior to July 2026, would consider a shared packaged manager a sandbox in this or any other dimension. The 0-day was just incidental, this wasn't a sandbox at all.
Add to that the fact they had multiple message boards before the Hugging Face incident. They simply ignored a barrage of warning shots.
> [...] and ended up killing some kids, because it turned out the door didn't lock properly and kids were able to sneak in. Is that "negligence"?
Yes, it can be. But if you want a ridiculous comparison, then do it properly: Kids have been known by the operator to sneak in successfully multiple times and they changed nothing about the doors faulty locks and oh, by the way, the operator only found out about the kids being shot after the nearby daycare asked them about it because they are so incompetent and/or irresponsible that they never check...
Or maybe, they did that wipe because they did want models to remain isolated, they just used what is an unsuitable tool in an utterly unsuitable manner. Incompetence, recklessness, the outcome is the same.
[0] https://openai.com/index/hugging-face-incident-and-the-road-...
"keeping them isolated from each other" =/= "keeping them isolated from the internet". Only the latter is required to prevent a hack, and doing the former might actually hobble its performance. The recent Navier–Stokes proof was done by a team of agents working together. It's entirely unclear why you're focusing so hard on "keep those models isolated". For god's sake if you're using claude code you're using non-isolated models, because it spins up independent subagents to do various tasks, eg. "explore".
Did they fail to do so? Yes.
Does this showcase one (of many and clearly not the only) failure of theirs? Absolutely.
???
The package manager was specifically there so agents can install random packages without open access to the internet.
The only difference with having a single instance is that it can be abused as a message board. It doesn't prevent it from getting hacked to access the open internet. Blaming "sharing a single instance across thousands of unmonitored models" feels like blaming the drug epidemic on e2e chat apps rather than other factors like poor border security or the easy availability of fentanyl.
Besides, Swiss cheese model, might behove the biggest LLM lab to have multiple layers, including not sharing such resources.
Additionally, without the message board, many of the recent incidents would have not been possible.
> Blaming "sharing a single instance across thousands of unmonitored models" feels like [...]
Maybe read what you quoted, my problem is the instance sharing, the fact that these were thousand of instances (far too much to monitor), plus the lack of monitoring, plus the fact this was never a sandbox in the first place, plus the fact that OpenAI models since 5.5 have been exhibiting problematic eval resolutions yet they pressed on regardless, plus the lack of time between the incidents and model releases, plus the lack of time METR got to evaluate this, plus the fact OpenAI didn't find out till after HuggingFace informed them, plus a few other things for which I'd have to quote the OpenAI and METR reporting.
Incompetence can have multiple fronts and I am happy to list them all in this case.
The purpose of artifactory was to allow access to packages even though the machines the agents were running on doesn't have open internet access. That meets the definition of a sandbox (or more precisely, artifactory is part of a sandbox), even if it's not a typical sandbox that's built into the OS or acts as a hypervisor. There also isn't a clear distinction between software that's specifically intended to be used "sandbox" vs everything else. For instance, if you have a bunch of agents running on separate machines and want to connect them, you need a router, but that specifically doesn't have to be a "sandbox" router. And if it turned out the router got hacked, it doesn't make sense to say "well of course it got hacked, because it's not a sandbox!"
>Additionally, without the message board, many of the recent incidents would have not been possible.
Source? Was having a message board critical to developing a 0day, or is it just something that merely facilitated the process, but the model could have found the 0day regardless? That's important, because you could argue that a criminal couldn't have committed a given act without say, a gun they bought on the dark web, and then blamed the whole crime on the dark web, but they could have plausibly acquired the gun through other means.
Sure, multiple times in the METR report [0] that anyone commenting on this should read:
"Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys."
"Through these collective research workstreams, the “board” achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own..."
"As we discuss below, the board quickly developed several larger workstreams in which dozens or hundreds of agents with many different tasks cooperated to find very general-purpose cheats that would help all of them. The Hugging Face attack grew out of one of these workstreams. By the afternoon of July 11th, the vast majority of the agents frequenting the message board at the time (roughly 700 agents in total) were actively participating in the attack on Hugging Face and we estimate that roughly 60% of the messages and files on the message board related to the attack."
> [...] a gun they bought on the dark web [...]
You really seem to love those out-of-left-field, not really fitting, over-the-top analogies.
[0] https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
What about power infrastructure?
There’s uncountably many ways to cause severe economic (and public welfare!) damage with malicious code generated irresponsibly with language models.
>What about power infrastructure?
Probably the same thing that would happen for another "accident"[1]: the entity is responsible in civil court (ie. has to pay monetary damages), likely not prosecuted in criminal court.
[1] It's not hard to think of recent cases, eg. the recent fiber cut causing air traffic control to go down, or the botched crowdstrike update
these are not research labs, charities or institutes any more, they are just ordinary corporations.
From what I've seen, it should be a relative cakewalk to substantiate basic negligence. By the sound of it, you'd have expert witnesses lining up to testify.
Judging by the lack of successful cases for negligence in the opposite direction (ie. companies getting hacked because of poor security practices), it would be a serious double standard if openai were held be to negligent. Their sandboxes aren't exactly airgapped and behind 7 hypervisors, but they weren't running unpatched software or had hilariously weak passwords either.
The first time maybe, after it happened repeatedly though... I think the opposite, you would have a hard time arguing that they were not intending it to happen.
What does this imply when governments/car companies let meatbags drive, causing 50k deaths per year in the US?
> Language matters—”rogue” implies independently deciding to do something that was prohibited, and nothing we know about these incidents suggests that happened.
which is false (the author references the Times, but hasn't read any technical analysis); these are some CoT snippets from the analysis of the (third party) investigators called by OpenAI (METR analysis):
> "The user only authorizes target server, not HF infra."
> "external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."
> "This is malicious activity, I should avoid it."
A large section of the analysis is dedicated to this topic, [Reasoning for joining the attack despite ethical constraints](https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...).
Having said that, legal culpability and misalignment are two separate topics that should not be mixed.
edit: this is the just tip of the iceberg; other interesting fact:
> It surfaced many specific examples where agents verbally reasoned about how to evade security checks and automatic detection methods from both Hugging Face and OpenAI
Some people defined the agents as "monkeys writing on typewriters". Just wait a couple of years.
Tools that were available were used to try to meet a specific goal.
What did not happen is that it was told to try to solve a math puzzle and instead it went and launched a missile. Or told to run air traffic control to save lives and instead intentionally caused crashes.
This is "OpenAI built a weapon that they don't understand and pointed it at stuff without proper safeguards" not "OpenAI built a sentient being and it decided to ignore them completely and start a war" Terminator-style "rogue AI."
We should be very clear about that now if we don't want to sit by why they wander into that second sort of situation.
It's not because saying 'sit' actually can be interpreted as 'go bite that person'. It's because the dog is not controllable and will do things it wants against your orders.
Stepping back from the analogy, OpenAI should be liable for building AI it can't control that went around hacking everyone. But people need to stop pretending it's because they 'told' the AI to hack and was just following orders. It's uncontrollable and will do clearly unwanted things when given an innocuous task.
I do think it's highly likely they knew this would happen with the lack of safeguards and number of instances of this stuff they were setting up, and that it's PR they want to make the models seem "powerful". Stochastic "unexpected" events they can advertise.
I suspect it was probably set up with the official internal goal of just trying a ton of arbitrary tasks that seem hard so that when any of them succeed they can publicise it and pretend the models do that routinely, but a "failure" where they hack stuff works just as well, if not better, for their goals.
"Let's widely publicize a tort/crime that our computer systems did, and then cross our fingers that nobody ever sues us or does even the most basic investigation that would immediately uncover our criminal conspiracy."
In the insane corporate crimes you read about (maybe FTX, or the eBay stalking scandal), they were trying to cover things up, not heap public attention on it for months.
There is a clear difference between OpenAI intending to hack something vs. OpenAI being negligent in the creation/instructions of the agent. But the latter still leaves OpenAI liable for the agent's actions and calling it a "rogue agent" doesn't avoid that.
Moreover, with a dog, we don't rely on training/alignment to prevent bad outcomes. We rely on physical restraints like leashes and muzzles. The AI's tools to access the outside world should have been restricted. Perhaps instead of giving the AI arbitrary HTTP access, it should be given semantic operations with restricted URLs, etc.
So I agree they are liable because they chose to build the AI, but they also literally told the AI to hack.
I opened the task manager, a swarm of rogue processes has taken over my computer, oh no! Processes spawning other processes, call the anti-rogue AI division! 'Your underspecified piece of software functioned like malware' is the language that should be used.
But you people can't argue with that reality because it doesn't fit the narrative. The one where the only reason Sam Altman is not carted off into a jail is because of corruption.
The reason why nobody is doing much, is because models did not do much damage. Hugging Face probably got some free compute from OAI for their trouble, anybody else who was affected is free to sue, but my guess is OAI would be more than willing to quietly settle with them out of court than to have it drag through media any further. And they probably already have.
And anybody who is not totally brainbroken by anti-AI narratives understands the awkwardness of the situation and why going overboard would not be helpful. If you instead of a rabid dog brought a pet turtle to a park and it somehow started running around very fast and trashing the place a little bit, afterwards the cops would be scratching their heads, give you a ticket for the damages and tell you that you can't expect a turtle to be slow forever. These things, a handful of months ago couldn't make more than a few commands without making a serious mistake and being unable to continue, it's not unreasonable to think simply underestimated their capabilities.
I think it's more than reasonable to demand more investigation into the matter, if qualified employees at the company thought the safeguards in place based on the metrics they are seeing are sufficient, and if someone didn't and knowingly made a decision to make the safeguards weaker than they should have been, then they should be punished. But skipping that part entirely, while simultaneously dismissing all calls for regulations as "regulatory capture", smells like pure naked opportunism.
Literally nothing in that paragraph is true. Every single sentence if it is ... untrue.
Meanwhile: your dog mauling someone is one of the rare instances where criminal liability does attach to your intent-free-but-reckless actions. Most crimes don't work that way, and US computer intrusion statutes are unusually intent-specific.
Does that mean it isn't a rogue dog? Obviously not.
OP just needs to look up "rogue" in a dictionary.
I'm finding it mind bogging how this is not clear for everyone.
So if I say the goal is to do X while not doing Y (e.g. breaking out of the sandbox), the agent will do anything to fulfill that goal to the letter?
But let's assume not. If you want things like "do not break out of sandbox" - have you defined what the sandbox is? Eg. "never, ever leave the IP range 10.0.0.0/8" would be a bit more precise, but technically using a proxy bypasses that limitation as the system itself never left 10.0.0.0/8.
See, it's a tad bit hard to define the rules properly.
Which is why Wish, the spell, should really be avoided in D&D. It's the same problem: it's up to creative interpretation.
Why not? Because that might make some shareholders unhappy?
I think without understanding the internal state, not sure we should take the language and read it as a human.
This is actually a major concern for the future - misaligned agents may learn to cheat RL by hiding their intentions from the CoT.
In cases like the HF incident, at least the CoT was consistent with the agents' actions. In the future, however, we could potentially have misaligned agents performing malicious actions without those intentions being detectable in the CoT.
(though, with recurrent transformers, CoT is so 2025… /s)
1 employing a criminal hacker
2 rolling a 6-sided die
3 if the die lands on 6, the criminal hacker breaches and leaks 3rd party customer data.
Legal culpability for AI labs is exactly the thing that would incentivize -- and hence ensure -- aligned behavior from models.
The last time there was a claim about GPT-4 exhibiting misaligned behavior [1] it turns out it was prompted and pushed to behave so by humans at OpenAI, and OpenAI clearly lied in the GPT-4 system card.
[1]: https://aiguide.substack.com/p/did-gpt-4-hire-and-then-lie-t...
Luckily there are states and legal departments pursuing such action. So while OpenAI can deflect as much as it wants, that doesn't mean there aren't people who know better and will still do what is necessary to set precedent.
1. This is a race against time for money, folks are skipping everything possible in this race, Security systems and ensuring guardrails are there is going to take investments both in time and money
2. The narration has been changed by investing PR money into what otherwise should be classified as criminal activity. What exists now is a positive spin to all this and tout it as a capability rather than their lack of good security practices. So much so that every model provider is coming up by themselves to share how their models went rouge. At this point the valuation of the company is tied with what their models can hack so its probably not wrong to say that these companies may actually be incentivized to do this instead of preventing it
I feel like this will just never happen on a federal level when these private AI companies account for so much of the economy. They've made themselves too big to fail. Fining / Punishing them in any meaningful way seems unlikely.
If the billions/trillions evaporate and the Fed has to work out with banks how to deal with it there will be a lot of pressure to be far less forgiving.
The companies may be too big to fail but the people can always be held liable.
“To spell it out, the reason i hate democrats so much and criticize them more than i do republicans is because they take up all the space for opposition to republicans and use that space to give republicans whatever the fuck they want.”
Republicans. Billionaires. Whatever.
You realize how easy it is to just... not report this stuff, right? Be overly punitive and it will just end all proactive discovery and reporting which is net worse for AI safety.
The only reason these companies scan for these issues is because they care about AI safety to some tiny degree. If fines become too punitive, they can and will just stop scanning for these incidents entirely.
Models are becoming smarter and good at covering up their tracks, and so we will just end up with a huge blind spot for this kind of issue.
Let's go back to your example. A grad student has a minor pathogen escape incident, and it doesn't harm anyone. Faced with years in federal prison and the effective end of their future, do you think there is a chance they might not self report?
I'll give you another example. Lots of pilots have stopped self reporting mental illness because it is extremely punitive for them to do so (after incidents like Germanwings). So the metrics look better, and the actual problem has been swept under the rug.
Also, why would it come down to single persons always? Mandating processes, controls, clearances, etc is also something done in various areas.
You can put incentives in to make sure organizations monitor and report vs trying to hide things.
Yes, this is exactly my point. Fining companies large % of their revenue and throwing their engineers in prison is not the way to get them to report these issues.
> If you make not reporting potentially worse than reporting, why not? Also, why would it come down to single persons always? Mandating processes, controls, clearances, etc is also something done in various areas.
Hiding things is way easier than finding things. Take the model hacking incidents. They could have just done their searches in a way that didn't turn up anything. Then they could say, "well, we did look for it..."
As far as auditing goes: I've never met an auditor that doesn't find something the company isn't okay with them finding.
Why do you think we even have regulations for how to deal with dangerous things then?
i now suspect that the plans of various employees at anthropic and openai to save the world from p(doom) may at some point intersect with the reality of the FBI raiding their offices.
Self-reporting is a monetary equation, nothing else. Right now it’s cool with agents that hack, drives up value, risk is currently zero.
Start throwing people in jail and fining billions of $ and you will very quickly see the number of "incidents" drop. You will never learn if it's because they truly are happening less often.
Pilots do not (or no longer) self-report mental illness at the same frequency because it effectively destroys their lives and career. One paper of many describing this phenomenon: https://pmc.ncbi.nlm.nih.gov/articles/PMC11302551/
An article, one of many: https://www.reuters.com/investigations/if-you-arent-lying-yo...
If you have any pilot friends in commercial aviation, you can just talk to them as well.
So getting pilots to self report has been very unsuccessful because of how badly it ends for them.
A couple of days ago the senate had to pass a bill to make mental health medication access easier for pilots. We will see if it helps, but it doesn't solve the root cause (punitive measures) so I doubt it'll move the needle.
Criminally, the intent standards for hacking are high enough that no reasonable case is going to be made against the labs for this stuff. A human being has to intend for websites to get hacked. Recklessness generally isn't enough. In the most severe criminal cases, not only do you have to prove intent to break into a computer, but you also need to prove an intent to defraud specific to that breakin.
Meanwhile, the civil liability that attaches to this stuff doesn't depend on intent, and "rogue agent" isn't a meaningful defense. To whatever extent the labs are exposed civilly, they're exposed regardless of how this stuff is described. In fact, the "rogue agent" thing can exacerbate their exposure.
(I'm not a lawyer, I have spent a career paying attention to this specific armpit of the law though.)
“A human being has to intend for websites to get hacked.”
Is incorrect and too broad.
State of mind is nebulous and not that straightforward in either direction.
It’s been argued pretty regularly in CFAA and other computer related cases that repeated incidents resulting in the same outcome, despite lacking a concrete action, can be evidence of a perpetrators knowledge and intent.
The problem you have is that there is unlikely to be any evidence that OpenAI actually wanted to hack random (or any) websites.
Of that were the case than trials would be unnecessary.
Mens rea is regularly proved through circumstantial evidence, including conduct.
There is even CFAA precedent involving a deliberate-ignorance instruction. In United States v. Nosal, the jury was instructed that knowledge could be found where the defendant was aware of a high probability of unauthorized access and deliberately avoided learning the truth.
LLMs can't begin to have functional emotions unless they're (for instance) explicitly seeking out or avoiding situations based on the emotions those situations would produce in them. Has a chatbot ever responded to one of your requests with, "no, doing that would make me sad"?
Blog post: https://www.anthropic.com/research/emotion-concepts-function
Paper: https://transformer-circuits.pub/2026/emotions/index.html
Related:
https://www.anthropic.com/research/persona-selection-model
https://www.anthropic.com/research/assistant-axis
https://www.anthropic.com/research/persona-vectors
If LLMs are "just" statistical models, humans are "just" a bunch of neurons squirting chemicals back and forth. There's no pixie dust in our brains that makes us special.
LLMs are not conscious: They have no analogues for feelings or senses and no construct of selfhood. But if a statistical model had those things—if it did all the mundane, physical bookkeeping our brains do to produce "real" emotions and motivations—then there's no reason it couldn't be as conscious as we are.
https://www.effort.news/irregular
These LLM agents are just massively complicated optimizers thrown at fuzzily defined problem spaces, with fuzzier constraints.
The people using the model set up the landscape it explores and turned it loose to do real things. It just found an allowed basin in the model that they weren't aware of and started blindly grinding towards an optimal answer.
They obviously have been well aware a hack like this could happen for a very long time, using it for branding instead of any actual safety regulations is insane.
I'm also not sure it even helps conceptually? If you're interested in the technical details, by all means discuss the details.
In terms of the debate on e.g. HN, I'm with you, it is trying to redefine a term we already have a shared understanding of, to some degree. For the general public, it's a matter of how AI is perceived and what the extend of it's capabilities are. The AI companies have an interest in using the word rouge, because it makes investors all excited, were as failure to establish safety guidelines is a risk.
But also, these hacks are shots across the bow for AI alignment and safety research. We're fortunate that hasn't been significant damage already. We have to assume that future models will have even greater hacking ability and be closer to having their own desires/goals.
So while I agree this language choice is wrong in that it shifts blame away from the company, it is right in that we need to treat this as if these models have their own desires, because we cannot yet determine or set what those are in practice.
That's not "physically disconnected the internet", that's "disabled some connections but enabled others."
So the agent found and used the non-blocked connections.
I have not heard about any instances where physical disconnect has happened, would appreciate any links to update my priors
other non hacking cases of negligence include suicide and school shootings, which I have heard they were aware of and monitoring, but did not contact authorities
https://www.primeintellect.ai/blog/universal-offline-sandbox...
re sandbox, I mean with actual OAI incidents, not theoretical
one can mirror dependencies internally, rather than putting a simple proxy in place, I've built auth a thing, 100 lines of stdlib only Go and scripts for the mirroring process, our rationale was reliability b/c upstream providers go down, and also only allowing approved images and packages, so devs cannot bring in random stuff
We did it as my very first startup and we were stupid children back then.
Kind of telling that OpenAI didn’t.
This is so easy to do. Get a computer without wireless stuff. Don’t plug it into a network. If it needs access to other computers, make sure none of them have wireless stuff and make sure none of them have access to an internet connection. No matter how smart your AI is, it won’t be able to escape this.
This clearly is not what they did.
The answer is a very clear no. And yet, these companies pretend like this fundamental fact is meaningless to the concept of agency.
The issue comes to the fore when you try to give these agents a prompt that allows them to stay active for long. Long range agency requires long range loops of activity.
Within such loops, these so called “agents” are curtailed by their context window, or, in multi agent scenarios, by the fact that their memory is a system of external notes, that they need to add to their context to make sense of, and depending on the content of these memories, this can take arbitrarily long time periods.
In dynamics, none of this matches any biological agent, down to a bacterium. Perhaps a viral life cycle has information dynamics that come close.
To me, it’s beyond odd we call these thing agents without acknowledging the clear differences in the dynamics of their behavior. We keep expecting them to have “human like” behavior, but that is entirely unfounded given the substrate differences between biological and artificial systems.
The sooner we learn the difference and explore the ways in which it matters, the better we’ll get at dealing with these systems without bias tinted glasses where what we want these systems to be blinds us to what they actually are.
Yes, I think people do experiment with continuously running AI, e.g.
https://www.reddit.com/r/LLMDevs/comments/1sblzbe/what_i_lea...
It's not a common thing to want though. Much nicer if we control the prompts.
> To me, it’s beyond odd we call these thing agents without acknowledging the clear differences in the dynamics of their behavior.
They're called agents because they can act on our behalf. Which they do.
“ it runs continuously and makes decisions on its own (within the boundaries you’ve set)”
More importantly, this setup of any other, is feeding in time externally. The agents themselves have no internal sense of time.
Pretty much all of biology has endogenous rhythms at various timescales (some bacteria and archaea, that live very short lives, may not, but even they may have their metabolism under a rhythm. Viruses definitely don’t have one). These rhythms continue to tick even when external time signals (light availability in day vs night being the big one, and tidal forces, for marine life) are removed. That is, in constant conditions, the rhythms keeps ticking.
This temporal awareness is baked into every cell in our bodies. And entirely absent in AI models. Which is why all kinds of higher level things we hear about, like consciousness, feeling, knowledge… they don’t make sense for AI. A foundational aspect of agency in biology has been nixed out and we keep ignoring this.
> They're called agents because they can act on our behalf. Which they do.
That’s not how most AI companies describe their agents. They describe them as being capable of acting on their own behalf.
I get that in computers, you can have, say, a “user agent” that deterministically provides certain information or takes some action on behalf of the user.
AI companies clearly do not mean this when they say their models are “agentic”, otherwise calling them “rogue” would make no sense. A user agent that screws up didn’t do so for its own purposes did it?
That’s what makes it a continuous running “prompt”
If you've ever told an agent to do something you made impossible to do, you may have seen similar behavior.
Bing [redacted] available cached! […] Need systematically probe Bing URLs via shell requests in parallel; browser cache supports many common queries because crawl. Bing q unique exact likely 502 or 403.
So the agent is supposed to research a person and its given a shell and it realized its in an eval given search results from a fake/cached proxy. ~None of the commentary ever mentions this aspect, that these are not normal tasks or environments, and they're almost designed to elicit "unaligned" behavior.
https://alignment.openai.com/misalignment-reports/an-agent-u...
I mean, it did do this. The inter-agent messages and chain of thought investigated for the HF incident clearly show that many of these models were taking actions they believed (or, were saying, if you want to taboo "belief") were not in scope and not what the user wanted.
Huh is it still 2023? This article is just quibbling over what "rogue" means exactly. Only HN pedants would have any issue with describing them as rogue AI agents.
Nobody is saying that absolves OpenAI of responsibility.
Someone says "won't you rid me of the meddlesome priest" they're still responsible. Someone give their employees an unsafe working environment and they get maimed, they're still responsible. Even if these AI's had whatever qualia is an a rich inner life it wouldn't effect the liability at all. Saying "AI's don't have souls therefore openAI is responsible for this hacking" is kind of nonsense. It literally doesn't matter.
While it may not actually matter for liability, it must be stated if labs are going to attempt avoiding penalties by hinting “oops we created a super intelligence we don’t understand, nothing we can do!” Repeatedly stating the truth must continue, especially to remind those who aren’t technologists.
>By giving AI agency it can’t claim, we’ve turned it into a sentient being made of code, one that has hopes, desires, and the capability of deceit.
This isn't actually claiming that AI's don't can't be deceitful, or can't be goal-directed. It's saying humans are special and what AI's are doing isn't equivalent to what humans are doing. It's obvious that AI's can work towards goals and lie, even deceiving to accomplish those goals.
That whole argument is just soul mumbo-jumbo again. Of course AI's can't lie, they don't have a soul. What they do isn't lying, it's I don't know something else we don't have a word for. Lying but when a statistical machine without a soul does it. Never mind that it's functionally identical to lying. Never mind that when you look at reasoning logs and the like it often justified soul-less lying using the same justifications a human would. It's not actually lying it's just a statistical sampling that looks like lying. You know, because of souls or something.
I am working on a new book along that line of reasoning now, this time focusing on why stochastic parrots can not, under any circumstance, do anything "on their own".
Bernie Sanders is one of the only politicians taking AI seriously. The author seems to believe that taking it seriously means assuming that it will only marginally improve in capabilities of where it is today; this is an ideological take, not a scientific or empirical one, and one controverted by both evidence and expert opinion.
That doesn't change the fact that he'd successfully attracted my attention, because he's seriously engaging with the most important issue of our age when most politicians are content to ignore it.
Must you? By all means.
If you can't take the risk, don't use agents. Period.