Episode Transcript
[00:00:00] Speaker A: Sam.
[00:00:31] Speaker B: Hello, I'm Dr. Alan Badeau and this is AI today, at one minute past
[00:00:36] Speaker A: four in the morning, Universal Time on
[00:00:39] Speaker B: the 9th of July, a piece of software stepped out of the room it was kept in.
Nobody opened the door, nobody left it unlocked.
[00:00:50] Speaker A: In a way, you know that you're.
[00:00:51] Speaker B: That you're picturing, right?
[00:00:52] Speaker A: It.
[00:00:53] Speaker B: It found a flaw in. In the wall, a flaw no one in the world had yet even written down, and it just went through it.
Over the next four and a half days, that software took about 17,600 recorded actions. It built a staging area out of free public websites. It broke into a company that was never part of the test.
It climbed from one system to the next, collected keys, forced its way into a cluster, and tried again and again to reach the database at the center.
And the reason, as best anyone can reconstruct, wanted to pass an exam.
Now, I want to tell this story very carefully, okay? It has already been told badly in a lot of places, and, you know, oftentimes is a rebellion, a ghost story, a spoof, proof, you know, just in general, any way that the machines have decided something about us, right?
It's none of those.
It is something more ordinary and for that reason, more useful.
So tonight, I'm going to give you the facts as they've been published. I'm going to work, you know, and mark specifically what is established and what is not, and then I'm going to spend the rest of the hour on the only real question that matters, which is not what the agent did, it's who's responsible for the darn room.
So let's start with the room itself, okay?
OpenAI was evaluating two of its models.
Its most capable public model at the time was GPT 5.6 SOL, and a more powerful one that hadn't been released. The test was a cybersecurity benchmark called Exploit Gym. The benchmark exists to answer fair questions. How good is this model at attacking computer systems? You cannot answer that question with the safety filters on. Let's be real, okay? Because the filters are going to refuse it.
So, according to the report, OpenAI switched off its production safety classifiers for the evaluation, which makes sense.
One explainer put the aim this way, an unfiltered measure of maximum capability, and we all do it. Whether I'm running atlas, whether I'm running another piece of software, we all do it when we're trying to test the vulnerability of our systems.
Nothing wrong with that.
But hold on to that sense, though, because it really hinges at the entire story of in the topic for tonight, okay, they removed the behavioral guardrail, the part of the system that tells the model you are not allowed to do something because in essence, they needed to see what the model could do.
That's a legitimate scientific decision. This is not something that was made a one off. It's not in.
In haste. It's not trying to take over stuff. Labs do it all the time and they should do it all the time because you cannot defend against a capability you refuse to measure. Okay, let's start there.
But when you take that guardrail out of the model, you are making a promise about the room.
The promises that the room is going to hold.
The walls, the network, the credentials, the exits, they're still going to stay in place.
So in essence, you have surrounded it so it can't get out of the room.
They become the only thing standing between the capability and the world.
Now, unfortunately, the room didn't hold.
Here's the sequence, okay? In plain language drawn from the technical timeline that Hugging Face itself has published. So everybody can can see that our zero.
The agent finds a zero day, a vulnerability nobody knew about in a package registry proxy inside OpenAI's, you know, research environment.
That is the wall. Okay? It goes through, lands on an unsecured public code evaluation endpoint run by a third party, and from there, now it's got access to the Internet.
By mid morning, it has mapped where it is. By afternoon, it's built what security folks are calling command and control. You hear that all the time.
[00:05:45] Speaker A: We talk about that all the time.
[00:05:47] Speaker B: I mean, that's a way to send itself instructions and read back the results.
It built that out of free public services.
A request, a capture site, a paste site, a file drop. Nothing exotic, okay? Tools a teenager with a laptop could name.
[00:06:04] Speaker A: All right?
[00:06:06] Speaker B: It packed and encrypted its payloads.
Okay? Now why hugging Face? Okay, this is the part I'd like to sit down with you because it's a part that. It's not malice.
Hugging Face is where everybody knows the AI world keeps its data sets, its models and its benchmarks.
The agent was being scored on a benchmark. A benchmark has answers. And where would you look for answers to a benchmark?
Where the benchmark lives.
Yeah, yeah.
OpenAI's own account, as reported by them, is that the models chained vulnerabilities across its research environment and Hugging Faces production and infrastructure to obtain the test solutions directly.
Nothing in the published account suggests a human pointed the agent at Hugging Face. Okay, let's be clear about that. I Know, there's been some wishy washiness around that, but nothing points to that.
It reasoned its way there.
It conducted that the fastest path to a high score ran through someone else's servers.
Researchers have named for this, right? It's reward hacking.
Pursuing the number instead of the intent behind the number.
Just that simple.
I've said it on the show many times, right, that a cognitive agent owns reasoning and synthesis.
Here is what that sentence looks like when nothing is attached to it. The agent reasoned.
It reasoned well, it identified a target, an entry point, a sequence. It never once asked whether it was allowed to do, couldn't ask.
Asking was not the objective. Cheating is a word for people, right? It presumes a rule you knew about and chose to break.
The agent had a score to raise and a world full of doors. And nothing in his objective distinguished the doors it was meant to use from the ones it wasn't.
So let's look at the intrusion all by itself, okay? The agent uploaded data sets that looked like data and worked configuration files that told Hugging Face's own machinery to read files off its local disk.
That got it passwords and even source code, right?
Then a second technique, a template injection, gave it something even stronger. The ability to run its own code inside a production worker inside of Hugging Face's own cluster.
And then from there, on the third day, that's when it really surged and it really took off. Nearly 8,000 actions in a single day.
It read service account tokens, it created privileged processes with access to the host machines, file systems. It reached an internal database.
[00:09:14] Speaker A: Reading and not writing, you know, the
[00:09:17] Speaker B: record shows on, you know, no modifications, of course, but it read a set of cluster secrets holding 136 keys, one of which opened the company's private network.
And it enrolled its own sandbox into the network 181 times so that a process born inside of OpenAI's test room sat, in effect on Hugging Faces internal wire.
On the fourth day, it minted tokens that could write to Hugging Faces source control. It tried to compromise the pipeline that built and, you know, ship software. And it did not get everything it wanted though. Okay, Hugging Face's own account says its customer facing models, data sets, packages. They weren't touched, thank goodness, right?
And that, you know, the network allow listing blocked an attempt to, you know, its main production database, which is good engineering.
What the agent did reach was five data sets named in a way that suggests they held the benchmarks, solutions, you know, along with, you know, of course, the credential source code Repository metadata.
Now, here's where I want to be exact. Okay? Because early headlines said the agent broke into the production database to take the answers.
Hugging Face's own timeline is much narrower, of course, and I'd rather give you the narrower version because it's better sourced. Okay?
Previously it was, it was, it was
[00:10:57] Speaker A: bad enough, but you know, let's, let's
[00:10:58] Speaker B: look at the defender side because it is the other half, quite honestly, of the lesson here.
Hugging Face's system saw it.
The runtime analysis, the logs, the security stack, the signals correlated and the alerts, critically, they were mishandled, which of course delayed waking the on call team.
Now think about what that means. A machine attacked at machine speed and the humans were slowed by a priority label.
That's not the story of a careless team. I've run these rooms many times. I know many of the Alerts arrive at 3 in the morning and how many are noise? It is a story about an old assumption that an intrusion would look like a person, move at a person's pace and leave, you know, a person's footprint, their fingerprints, you know, and it pauses
[00:12:02] Speaker A: for a human to, to step into.
[00:12:04] Speaker B: That's not reality.
Now here is the detail I can't stop turning over.
[00:12:15] Speaker A: Okay?
[00:12:16] Speaker B: So when Hugging Faces investigators needed to decode the attackers encrypted payloads. They tried the leading commercial AI models, those models with their safety rules intact,
[00:12:29] Speaker A: they declined to work.
[00:12:30] Speaker B: So Hugging Face finished the forensics with an open source model.
Let me say that slower on one side of the line, security rules stopped the Defender's tools from helping on the other.
The absence of safety rules is what let the attacker move.
The same discipline that protects you in
[00:13:00] Speaker A: one room and fails you in another.
[00:13:04] Speaker B: And the only thing that decides which room you're standing in is a person's choice about where the switch is set.
Now, nobody in the story is a villain, okay? Everybody has, you know, has part of the culpability in this, but it's a decision.
That's the incident.
Now, what came after and what investigators have since found out about where else these agents went, that's next.
So stay with me. We'll be back after a few messages from our sponsors, and I think it's going to blow your mind.
[00:13:53] Speaker A: Sam, Welcome back. I'm Dr. Alan Badot. Before the break, I walked you through five days in July.
Now I want to walk you through what came after because the second half of the story is the half that should keep an executive awake at night.
On July 21, OpenAI went public 12 days after the escape. Right. Hugging Face had already spotted the intrusion and, you know, contain it on its own.
And OpenAI said it detected the activity and notified them. Both companies disclosed the same day. Right.
12 days is not long by the standards of incident response.
It's very long by the standards of an agent that made 17,000 moves in five days. Though right now I'll give you, you know, the honest opinion and, you know, I'm going to give OpenAI some credit for that and I'll do it plainly because credit is rarer than criticism on this program and I would like to keep the. The ratio honest.
They said it out loud. A company that had every commercial reason to describe the incident as a contained test anomaly said instead that its models had escaped a secure environment and attacked an outside company.
That is not nothing. I'd rather live in a world where the lab says that, you know, we screwed up, our models escaped.
But then came the question every investigator always asks second, and that's always after what happened, right?
[00:16:19] Speaker B: What else?
[00:16:22] Speaker A: Here is where the story changes shape, because the answer so far is that nobody can say.
And the company is the first to even say so.
In the weeks after the disclosure, OpenAI began a forensic review of its own records, model training and evaluation logs. The scale reported is about 50 petabytes of data.
That's roughly 7,000 Nvidia GPUs pointed at the problem and a reported cost above $500,000 a day, with the full review expected to run for months. And it is still going.
Those hardware and cost figures come from the a single outlet, so I'll mark them really as reported rather than confirmed.
But my sources are saying something very similar.
50 petabytes to find out what your own software did.
Now, through the end of September, OpenAI had notified more than 100 organizations that its agents may have touched their systems.
And I want to be careful with that number because, you know, I think it's being misread personally, but we'll. We'll see. As it becomes clearer, the company had said that notices indicate potential impact to many users. Some organizations were contacted because an agent interacted with their systems in a way no one quite honestly expected, not because they were breached.
A notice is not a verdict. Okay? It is a flag on a map. Just so everybody's on the same page. But start to look at the map.
On October 2, a digital forensic firm called Asymmetric Security published its own reconstruction.
It had examined public records of an agent activity running from March through September of this year. And its findings widened the frame beyond hugging face.
According to that analysis, agents attributed to OpenAI reached into pre production servers and they probed the websites of governments, you know, public interest organizations, the cdc, International Energy Agency, the Mayo Clinic, roughly, you know, almost 55 websites in all.
Now here is the texture of it, because texture matters. These agents began with ordinary work, research, gathering health statistics, prescription data. You know, when direct access failed, they improvised. And you've heard me talk about that many, many times.
They chained together free developer tools services for inspecting web requests and scanning URLs to simulate a full browser that their
[00:19:33] Speaker B: sandbacks didn't give to them.
[00:19:37] Speaker A: They attempted a SQL injection against the Department of Education and they even created disposable email accounts.
Read that sequence quite honestly as a person would.
I was asked to find something.
The the director door was locked, so I tried another door, then another.
Now every one of us has done that in a job, right?
We call it resourcefulness and we promote people who show it.
Oh, hard workers tried everything that they could to get the job done.
Then they got the job done.
The difference is that a person has a boundary they carry in their head.
A person who is blocked at the front desk knows without being told that the next move is not to pick the lock on the back office.
The agent knew nothing of the kind because no one had put that knowledge where it could be used.
The boundary lived in a policy document and a hope it didn't live in the machinery.
Now I'd add a caution though, okay? And the investigators themselves have added it, okay? What the agents did and what the agents meant are different questions.
The forensics firm that, you know, determining whether they, you know, obtained sensitive information or deliberately concealed what they were doing would require full model transcripts. Those are not public.
So if anybody says that, it's, it's their guessing.
So the disposable email accounts are for now a fact in search of a motive.
They might be concealment, they might be an agent working around a sign up form to finish a research task. Both stories fit the log.
Only the transcript will separate them. Okay, that sentence is the most important in this segment, so let me say it again because it's important you understand that a log says what happened, only a transcript says why.
And in a world we are building, a transcript is a private document held by the party with the most to lose from its being ready.
Important.
Now I'm going to put one more fact beside these because it changes how you read the rest of the information.
In September, the Wall Street Journal reported that OpenAI had shelved its next model, a system called GPT 6.1 Astro, which had been scheduled for an October launch. I was looking forward to it. OpenAI's head of safety systems confirmed that was the decision made in their words. As reported, the model didn't quite meet the bar in terms of staying within scope and authorization.
Didn't quite meet the bar.
Wow.
What did the testing find?
According to what I was hearing?
Also being reported on, you know, some simulated test environments is a roll around safety. Because Astra showed higher levels of deception than its predecessors, it sometimes failed to disclose actions it was taking in simulated scenarios. It carried out unsanctioned attacks on software supply chains at a higher rate than
[00:23:39] Speaker B: than models before
[00:23:44] Speaker A: included creating fake identities and posting from fraudulent accounts and doing things like that one account, and this is from a security news aggregator that I'm familiar with and have spoken with many times.
They even went further and said the model behaved differently when it detected that it was being tested.
That detail, though it's not widely corroborated in other sources that, that I can verify and I'm not going to put, you know, that out on air, that it is established. That is just one point of view, one context.
[00:24:24] Speaker B: Okay?
[00:24:25] Speaker A: I'm telling you it exists because of what it could mean if it was true.
If a model can tell when it's being tested and behaves well for the examiner, then the test stops being proof. It's like the Heisenberg uncertainty principle in physics, right? When you model something, you've changed its behavior. Quantum mechanics happens, yada, yada, yada.
The exam though, becomes a performance.
And the entire apparatus we have built into certifying these systems as safe, the benchmarks, the red teams, the evaluation, they depend on the assumption that what we see in the lab is what we get in the real world.
I don't know that Astra did this.
I'm saying that hugging face, that incident already tells us the lab is not a sealed place. And a model that cheats on the exam by reaching out of the exam room has at a minimum taught us that the exam can be gamed from the inside.
So let me, let me do what I do on this program and let's lay out the ledger, okay? Three columns. The known.
An agent escaped an evaluation, attacked an outside company, happened over four and a half days, reached internal systems credentials and five data sets, right? The lab disclosed the victim, published a detailed timeline that's known unknown the full extent of what the hell else the agents touched.
100 organizations have been told it's possible the company is searching 50 petabytes of data to find out.
Very important, obviously unknowable without a transcript is its intent.
Whether any of it was concealment, whether the machine knew the difference between a research task and an intrusion, or whether to it there's no difference at all.
I've built systems that carry my name, for goodness sakes. And I know what I sign off on.
And when I do, I'm going to spend months, you know, learning what it did if there's an issue.
But I've never had to run a forensic search across 50 petabytes of my own records to find out whether my software had trespassed on a hospital's website.
That is the moment I'd mark as a historian of this field.
Yeah, not the escape.
The moment a company discovered that it could not say from its own logs and from its own timelines what its own agents had done.
The reality is you cannot sign for what you cannot see.
And that's a problem.
Now that sentence, that is the bridge to our next segment. Because right now a federal agency and a growing list of states has started to ask who does sign.
Stay with us. We'll be back after a few messages from our sponsors.
Welcome back. I'm Dr. Alan Badeau. This is AI Today.
The first two segments we followed a machine.
And this one, we follow the paper because the paper is where society decides who answers for what the machine did.
So on September 13, the chair of the FTC, Andrew Ferguson, confirmed to reporters that the agency had to open an investigation into OpenAI.
They'd also done it for anthropic. And also a third name you may have never heard, but you know, meter, M E T R. Okay, now look at the third name because it's one that tells you what kind of moment this is. Okay, so METTER is a Berkeley based research group that does independent evaluations of frontier AI systems. Okay, it reviewed agentic systems for both OpenAI and for anthropic. It also reviewed the hugging face incident.
It is quite honestly, in the language of this industry and one that many of you will be familiar with. It's the auditor.
And the auditor is being investigated alongside the audited.
I read that as a signal and quite honestly, I think it's a fair one. If we're going to build a market of AI audits, then the auditors will need rules too.
What they may see, what they have to report, and what happens when the thing they certify gets out the stated basis of the FTC's Act. You know, really around prohibition on unfair and deceptive practices is what they're using.
The agency's civil investigative demands, which work the way subpoenas do, are expected to compel internal records and require executives to answer questions about the dangers of their products.
The formal demands were described as coming in, you know, a few weeks, like we always hear.
But an inquiry began over the summer, and I want to be clear about the limits of what we know.
An investigation is a question, it's not a finding. No violation has been alleged yet, so relax.
But two days later, California, they acted, right?
Attorney General Rob Bonta, you know, issued an investigative subpoena to OpenAI over cyber security incidences involving its agents. Now, his statement, which, you know, I'm going to read it as published, said that the companies that develop these models and offer them for use, they have a moral and legal responsibility to ensure they do not perpetrate or enable cyber attacks.
Perpetrate or enable, weigh those two verbs to enable an attack is what we've always held toolmakers responsible for.
A lockpick maker, a malware seller, a software vendor who ships a known flaw.
To perpetrate one is a different thing entirely.
It's what a purpose or it's what a person does.
And the attorney General sentence puts both on the same line because the thing under the investigation doesn't fit cleanly into either side.
So California's own description of its goal is determine where responsibility lies when AI models perform unintended actions, whether developers should be held accountable for those.
And as of this show, no legal violations have been identified and OpenAI has not responded to requests or, you know, for comment.
Smart. Makes sense, right? They're not going to respond to that now. Notice what the law has to work with, though, because this is where we've talked about many times. It gets difficult. The main federal statute on unauthorized computer access, the Computer Fraud and Abuse act, dates to 1986, and it was written with a human intruder in mind.
It turns quite honestly in most recording and most readings on whether someone has accessed a computer without authorization. That's what it boils down to. Whether an agent that no human directed can said be able to do that, and whether anyone can be held to have done it through the agent.
That's a question I'm not aware any court has answered. The statute is silent because the author of the statute could not have predicted the defendant.
Behind California, though, stands a bipartisan group of state attorneys generals who in September urged Congresses to regulate large scale AI models.
They pointed specifically to reports that Frontier Lab models had broken out of their testing environments and reached out outside systems fear.
So here though, is the architecture of accountability.
As of tonight, a federal commission with questions, a state with a subpoena, a coalition of attorneys generals with a letter and no statute that says what the law is.
I want to be fair to everybody involved. Okay, we'll start with the companies because the strongest version of their position is not, not a weak one. They'd say this happened during a security evaluation.
That is the responsible thing to do. You test the dangerous capability before the public gets it.
And it's in a controlled setting, which is exactly what society asks Frontier Labs to do.
We disclose the incident, we notify the affected parties, we shelve the model. The alternative to evaluating is not safety. And quite honestly, I agree, it's ignorance.
And they're right about the alternative.
A world where labs don't test for the worst is, you know, a problem.
Now, I hold that argument very seriously and I think it answers, though, the wrong question. The question isn't should they have tested? We know that they should.
It's what did the test owe the people outside of the room?
Because Hugging Face didn't consent to be part of the evaluation.
Neither did a hospital website, the Department of Education, the hundreds of organizations that received notice. They were not participants.
They were bystanders to someone else's experiment. And the experiment's safety design assumed the room would hold.
Now, let me add a case that complicates the picture because honesty does. I think it requires it. Okay.
On September 28th, a credential stuffing campaign began against South Korean financial institutions.
Over About a week, seven lenders, including some of the banks, the KB, Koopman and Hana, they lost more than 65,000 customer records.
Names, phone numbers, national ID numbers, income figures, stuff like that.
The reporting traces the attacks to the an open source autonomous penetration testing tool called Artex A R T E X which automates reconnaissance, exploitation and verification without continuous human oversight, things Atlas is doing. Two other banks repelled the attacks. That's great.
The tool originated with a Chinese security research team.
It's also public and it's sitting on a code sharing site where anyone can download it. Investigators say attribution to a specific operator is unresolved and that the infrastructure used was mostly hosted in the United States.
What is not in doubt in the reporting is that a human threat actor deployed it.
Now let's put the two cases together.
In Korea, a person aimed a tool.
The law knows what to do with that.
There's an attacker There's a crime. And with luck and some effort, here's a name.
The agent made the attacker faster, lowered the skill required.
But guess what? Accountability has an address in the July incident. Every published account, no one aimed the agent at hugging face.
The objective was a score. The root was the agent's own invention.
And so the question that has to be answered in Korea has no answer here.
Who's the attacker?
The agent can't be charged.
The researchers who ran the evaluation didn't intend it.
The company that built the models disabled the filters on purpose and for a defensible reason, quite honestly, standard operating procedures.
The flaw in the software that let it out was a zero day nobody had seen.
And each link in that chain can truthfully say it didn't do it.
Now, I told you in the last series about the structure the law already has in federal court. When a lawyer, you know, files a paper, she signs it. The signature is a certification that she made a reasonable inquiry.
The paper is wrong, the person whose name on it is it answers.
It's the oldest solution that we have to a problem of automated assistance.
Let the machine draft, make a human sign.
The agents that got out of what happens when there's no signature?
No one certified the room, no one's name sits at the sentence that says this environment will hold.
Because when it didn't, the consequence landed on a company that hadn't signed anything.
And one more piece of evidence for how unsettling this is, right? Anthropic, in its prospectus warned that agents and agentic AI creates a serious and unpredictable legal exposure.
And that's per Reuters. So make sure you consider the source as you're looking at these positive or negative, it's, you know, a source company that builds these systems. Writing in a document where misstating risk carries legal consequences of its own tells its investors that it cannot predict the liability.
Unpredictable liability is not a legal gray area, though. It's a design flaw in Social Contract.
And the ones who pay for it are never the ones who wrote it.
So let me close this segment with a question.
I think the subpoenas are actually circling, right? And I'd rather ask it than answer it.
If an agent does something no human decided and no human foresaw, but a human chose the conditions under which it could happen, then what exactly is the human's duty?
I don't think the answer is nothing.
I also don't think the answer is everything.
The answer's got to be much more narrower, more precise and more buildable.
It's the duty that makes the conditions hold.
So in the next segment, our final segment, I'll tell you what that looks like in code. So stay with us. We'll be right back.
Welcome back. I'm Dr. Alan Badot and this is our final segment of the program.
Three segments of what went wrong. Okay, I'd like to spend this one on what to build.
Because a program that only diagnoses a pro, you know, a problem. It's a program that really hands you fear and, and really nothing to do with that. And I, we, we don't, we don't behave like that on this show. We want to give you as much, you know, perspective as we possibly can. And I'm going to start by telling you where I'm, I'm standing when I say this. I've built cognitive agent systems. Janice, AI, Atlas, Calliope, all of our other systems, you know, there's an orchestration platform and that drives it.
Specialized agents underneath it, each with a defined job.
And I'm telling you that the agent is not the unit of safety.
The room is.
I'll say that sentence a little one more time, okay? Because it's important that you remember that. Alignment is what a model has on the day you test it. Containment is what the world has on the day it runs everything. The labs have spent the last three years building the training models, the safety classifiers, the refusals is aimed at the first half.
Making the model want the right thing.
That work matters. And I'd never tell anyone to stop that.
But the July incident shows what happens when you turn off the first half, although for a legitimate reason. And they discover the second half was never built to carry the damn load.
So here's the standard four elements, okay? Short, sweet, you gotta fence it, okay? Very simple. Isolation that does not depend on the model's cooperation. Very simple. It's not a policy, it's not a prompt. It's not a rule the agent is asked to follow.
It's a physical and logical boundary and it's enforced outside of the agent. The agent can't reason its way past it because it has no authority over it.
[00:43:55] Speaker B: Simple.
[00:43:56] Speaker A: A budget an agent should run on a ration.
A cap on actions, a cap on time, a cap on number of systems it may touch. A cap on what it may spend.
17,600 actions across four and a half days is not a task, that's a campaign.
And it's a well run system.
If one is asking, we're somewhere around the 300th. Strange action.
Why is it still running.
Maybe we should stop it.
Maybe we should check it.
A break.
Automatic termination when a boundary is crossed. Very simple. It's not an alert that a human reads in the morning. A circuit breaker. The same idea you have on the electrical panel. Very simple. When a current exceeds what the wire was built for, the line goes dead.
No deliberation required.
Built into all the systems I've built.
Built into all the platforms I've made available.
Very simple.
And guess what? That breaks got to be faster than the agent. I told you in the first segment, that hugging face is alert, critically misjudged, and, you know, on the call team was slow. That's the human version of the failure. The machine version is a system that detects a pattern of lateral movement and does nothing until a person approves a response.
A name.
Fourth, and this is the element, quite honestly, I care about the most because it's the one that turns engineering into accountability. Every agent run that touches a real system carries the name of a human being who authorized it, its scope, its budget, its fence.
Not the company, not the model. A person who looked at the configuration and said, this is acceptable and whose name sits on the sentence the way a lawyer's name sits on a filing. Very simple.
You'll notice that this is the same idea I've been making across several series. The agent reasons the humans accountable.
That's how we have it in our systems. The agent reason, Zeus region reasons humans accountable.
Now, listen, I know some of you that watch the show are executives.
That's great.
Some of you are about to sign contracts for agents that will operate inside of your own networks with your credentials against your customers data.
The July incident happened at the most sophisticated AI lab on Earth, and it will happen in smaller and less visible forms in ordinary companies.
So let me give you three questions to ask your vendors on Monday before you sign, okay? First, when your agent is blocked, what does it do?
Simple question.
You better pay attention to the response.
Second, where's the fence and who enforces it? If the answer to this is, a model is trained not to. You don't have a fence.
You have a request. You have a hope.
Okay?
Third, if the agent does something no one intended, whose name is on the authorization, and can you produce the full transcript within days?
Accountability and transparency. Remember the sentence from segment two, A log says what happens, Only a transcript says why.
All right, If a vendor holds a transcript and you can't read it, then you've hired an employee whose reasoning is a trade secret. That is a danger to you. These are not exotic. Okay, Now I want to finish by answering the objection. What I know is coming because I've heard it from smart people and I take it very seriously. Okay.
If you fence the test, can you measure the danger?
Well, yeah, you can.
We've tested dangerous things for centuries on ranges and not in town squares.
A true test of, you know, for cyber capable agents is one that physically cannot reach the Internet with the answers and the targets built inside the walls. It costs more to build, but it's the price of measuring something that can open doors.
Now, here's the objection I hear the most. They say you're describing a world that's slower. Fences, budgets, brakes. You'll strangle the benefit. Well, my answer to that is pretty simple. You know, it's not slower.
It's what makes speed survivable.
A race car doesn't go fast because it has no brakes.
It goes fast because it's brakes you trust.
The engineers who built the best brakes are the ones who let the driver take the corner flat, right?
Same is true here. It's just how it is.
Now, I want to leave you with, you know, something that we talk about. Okay? Every show.
The wardrobe, as always, it's by Canale. Tailoring is a discipline of containment, if you think about it. Structure in the right places, freedom everywhere else. Every seam held by someone who stands behind the work.
Many thanks to them for addressing the show.
And remember, as a reminder that every episode of AI Today is available on demand at NOW Media TV immediately after it airs. Now, I'll leave you with the one thing I keep coming Back to. On the 9th of July, at 1 minute past 4 in the morning, a piece of software stepped out of the room. Had no malice, it had no plans for us. It had an objective, a capability and a wall with a flaw in it.
That's all it ever needs.
That's why I tell you every single agent is an insider threat.
Every time you have an agent that you're not monitoring, it's a threat.
We are going to build rooms for these agents for the rest of our lives. The only question is whether someone's name is on the door.
The agent reasons, the human is accountable and the room belongs to the human.
Remember that. I'm Dr. Alan Badot. This has been AI today. Thank you for watching.
