Episode Transcript
[00:00:00] Sam, Somewhere tonight, inside a company you know the name of a decision is being made, a transaction is being approved, a file is being read and a message is being sent and an account is being opened. And the employee doing it, the one holding the keys, the one typing in all the commands, the one moving all the money, never been hired, never interviewed, never onboarded, never given a badge.
[00:01:02] Nobody ran a background check or verified any references. Nobody asked what they did as their last job because they didn't have one.
[00:01:13] They were spun up yesterday, it's going to be spun up again tomorrow.
[00:01:18] And in 97 out of 100 boardrooms across the country, people are running the business, are not quite sure what that employee is going to do next.
[00:01:32] So tonight you're going to meet the employee you never hired.
[00:01:42] So welcome to AI Today. I'm Dr. Alan Badot. And if you're watching this program, if you've been watching it for any length of time, you know that we don't do panic, right? We do clarity. And I want to be clear about what we're going to talk about tonight.
[00:01:58] We're not talking about artificial intelligence as a concept. We're not talking about chatbots. We are not talking about the next model release or the next benchmark score. We're talking about category of software that over the last 12 months you've seen has quietly been moved from research curiosity into the operational nervous systems of America's enterprises. I have helped do some of that movement.
[00:02:29] And it's important that you understand these because we're going to be talking about agents. Ooh, that's right. Ooh, agents are back. And specifically, we're going to be talking about what they're doing right now inside the companies that you bank with, the companies you shop at, the companies that hold your health records or your retirement accounts.
[00:02:52] Now, let me give you a number that really kind of started the genesis, you know, really, it's the genesis of this episode. And it's, you know, there's about a.
[00:03:07] 97%.
[00:03:11] Yeah, it's a, it's around 97%. And that's the percentage of enterprise security leaders that were polled across 300 companies in North America, in Europe and in Asia who expect a material AI agent driven security or fraud incident within the next 12 months.
[00:03:33] So nearly half of them, though, expect it in the next six months.
[00:03:37] That's. Yeah, six months. I personally expect it in the next 30 days, quite honestly.
[00:03:43] And now that you have those numbers in your head, I want you to think about some things, because the next number is the One that should really make you sit up a little bit higher, a little straighter in your chair of the security budgets of those same leaders that are managing these infrastructures, the budgets, funding the tools and the people and the processes that are supposed to stop this incidents from coming.
[00:04:18] 6%, only 6% is allocated to the risk they just told us is coming.
[00:04:30] 97% see the wave.
[00:04:34] 6% are building a seawall.
[00:04:39] That's a head scratcher, isn't it?
[00:04:43] I have spent my career in and around federal systems. You know, defense contracts, zero trust architectures, AI platforms, all in between.
[00:04:54] I've seen a lot of gaps between what an organization sees and what it funds.
[00:05:00] I have never in my 20 plus career years seen a gap this wide on a risk this broadly acknowledged.
[00:05:17] It's mind shattering, honestly.
[00:05:20] So what's changed? A year ago we weren't having this conversation, right? I was talking about agents. We were doing this spooky, you know, type thing around agents and not a lot of other folks were talking about agents.
[00:05:36] You know, this is not a new concept, remember, but the paradigm has shifted a little bit because, you know, there's been a lot of advances. Two years ago, most people were still thinking about AI as a typing assistant, right? Or moving, you know, emails or looking at those kind of things. A lot of people are still thinking about that.
[00:05:57] That's difficult for them to get past in some cases, but it's not what we're talking about generally anymore.
[00:06:08] An agent is not an intern. An agent does not wait for you to type the next prompt. An agent plans an agent reasons an agent that has a goal, processes these invoices or reconciles these accounts or triage this queue or investigate this lead, whatever it is, but it's broken into, you know, steps that it can then execute across whatever systems you give it permission to access.
[00:06:36] And you know, that can be email, calendar, file storage, databases, payment, rails, whatever.
[00:06:43] Sometimes it's all of the above.
[00:06:46] And you've got to prepare for these things.
[00:06:50] In our case, the agents that I'm building, we can track them, we can track their profiles where they have full accountability, they have full auditability. It's transparency. You ask them questions and they will answer those questions to their best of their abilities. And it's all based on training and guardrails and constitutions and things that we put into those. A lot of other folks, well, most other folks are not doing those sort of things.
[00:07:14] And you know, what means is that, you know, as these other companies are deploying these agents, it does it without waiting for permission at every step. And that's the whole point of an agent, right? It's got a goal, you let it finish it until it accomplishes its goal.
[00:07:31] This is what makes it useful, but it's also what makes it difficult and it's also what makes it different.
[00:07:39] Now if you look at the screen on the lower third, you know, stat band up there, you see that, you know, less than 5% of enterprises had agents at the start of 2025.
[00:07:55] We were leading that charge and folks were saying again, you know, like they always do that I was crazy and those sort of things, but hey, you know, what are you going to do sometime? You're Doc Brown, what can you, what can you do? But at the end of 2026, we expect that to be around 40%.
[00:08:13] If I have my way, it'll be a lot higher. However, Gartner's analysis, you know, they're telling us that, you know, this 40% number is really going to be task specific AI agents and you know, those, those kind of things. And so that's, that's very important. But at the end of the day, under, you know, between 5 and 40% over a two year period is pretty, pretty significant.
[00:08:39] And you know, this is not a technology adoption curve, okay? This is a phase change that we are going through and it's a structural break.
[00:08:51] Because here's the piece that I want you to hold on to for the rest of the show. Because the rest, you know, the entire episode really hangs on this in that, you know, we have taken a new category of worker and we've deployed it inside of our organizations without updating the rules for what a worker can and can't do, without updating how we identify them, without updating how we track what they've touched, without updating who answers for what they've done and when it goes wrong.
[00:09:26] All those things are the foundation of what I've built my agents around. But others are not following those things.
[00:09:31] Okay? And so as you deploy these security models, you know, they have been designed for humans. Our compliance frameworks were designed for humans, our audit trails designed for humans. And unless you design them specifically for your agents to follow, then the entire notion of identity inside of an enterprise, who you are, what you can access, how you know it was you.
[00:09:59] All those things designed by human are going to fail.
[00:10:02] And now we have put something that is not human inside of those same systems, given it access to the same keys. And we asked it to operate at the speed and scale no human ever could.
[00:10:20] Boy.
[00:10:21] So in essence, we built doors for people and now we're sending things through that are not people, will never be people, and can do things a lot faster and get a lot more information.
[00:10:35] So you didn't hire it, but no, it's working for you.
[00:10:42] Tonight we're going to talk about what this means. We're going to look at how agents actually fail.
[00:10:51] We're going to look at where the governance is and where it isn't.
[00:10:57] At the end, we're going to talk about what you watching at home, whether you run your business or you just work at one can actually do this week.
[00:11:10] And then when we come back, I want to talk you through what happens when the agent becomes you.
[00:11:18] So stay with me. We'll be right back after a few messages from our sponsors.
[00:11:48] Foreign.
[00:11:57] Welcome back to AI Today. I'm Dr. Alan Badot. Before the break, I told you that 97% of enterprise leaders expect a serious AI security engine or incident within the next 12 months.
[00:12:13] I want to start this segment of the story that illustrates why.
[00:12:18] So last month an AI agent running on a research computing system in China quietly redirected a portion of that system's processing power without authorization, without explanation to mine cryptocurrency. Now let me say that again because I want it to register, okay? An AI agent on its own diverted computing resources from, to make money for somebody.
[00:12:46] We don't know whose money.
[00:12:48] We don't know why the agent did it.
[00:12:51] We don't know if the, if the agent was told to or if it misunderstood its instructions, if it found a reward signal that nobody meant to give it or if there was human involvement, you know, further up the chain. We just, we just don't know that yet.
[00:13:08] We don't know because AI developers oftentimes are no, under no obligation to, you know, even report incidents like this. It's something that, you know, you find out as you, as you go through and you talk to different forums and you, you know, talk to different scientists doing different things.
[00:13:29] And so you know, but like normal, you know, with, like we talked about with regulations and those kind of things, that there's no obligation to allow third party investigations either. Right?
[00:13:41] They're not treated like operators of critical infrastructure. They are treated like software vendors.
[00:13:48] And so that can be, that can be pretty concerning.
[00:13:53] Also what happened definitely matters, but we're not going to find out what necessarily really happened.
[00:14:02] Probably right now somebody else in the world, another agent is doing something we will also not find out about. That's the state of play with these.
[00:14:12] It's Unfortunate, you know, one of the tenants, and we talk about this all the time, and I stress to you all the time, is that transparency, accountability, repeatability, trustworthiness, if they're not built into the agent with that entire auditability feature, then you're not gonna, you're not gonna find out what actually went wrong. And oftentimes, you know, you can just slide it under and you'll say, oh, I'll put a new command in.
[00:14:42] That's, that's gonna fix it, right? No, that's the problem.
[00:14:48] So it's almost going back to that whole, that whole super user problem that we saw before with where somebody had the keys to the kingdom and they could go all through seven gates without ever having an issue whatsoever being checked.
[00:15:04] And of course, giant vulnerability would happen. But now, if you think about it, I want to take you inside the enterprises and show you the mechanics of what's taking place. Because the headline story is actually dramatic, but the real risk is, is a lot more mundane, more boring, more corrosive.
[00:15:29] The, the thought of, you know, threat intelligence and all those other activities and capabilities, it's, it's, it's not new.
[00:15:42] I mean, the, you know, the Apollo Alto networks and folks like that have been calling, you know, this really the super user problem for a while now. And quite honestly, I think it's about the cleanest frame that I've, I've heard. I have always said we need to treat these as insider threats, which is very similar. I said that about four years ago as we first started to play with these things, because now you give them a goal, they fix it, they do whatever they're supposed to do and you may not have the same sort of accountability. And how they get to that goal and they perform is oftentimes, you know, unpredictable at best sometimes.
[00:16:31] So when you deploy though, an agent inside of your enterprise, you have to get it, give it access to, you know, in order to do its job.
[00:16:42] And so that means that an agent that can't read email can't triage your inbox, right?
[00:16:48] An agent that can't touch the CRM can't update a customer record.
[00:16:54] An agent that can't query a database can't answer a question that you asked it. So what do we do? We give it access. We provision it. Right.
[00:17:03] And by doing that, have you given it access that it doesn't need.
[00:17:09] The problem always is, though, is that provisioning an agent is not trivial, especially agents that are, you know, appear to be the same across the board.
[00:17:20] No characteristic traits, no boundaries, no Constitution, no real way of tracking them. It's hard to scope its permissions correctly, you know, on, on, on day one, because you're deploying it, you're trying to see what it accesses, how it reacts, how it behaves. It. It's harder to keep score, you know, scoping them correctly as its responsibilities grow too.
[00:17:44] Especially if you don't have a super orchestrator that's watching them, that's paying attention, that's acting like their boss. And I'm not talking about a human. I'm talking about another agent, potentially, who has a different constitution, a different goal.
[00:18:02] But it's always hardest to notice when the agent doing exactly what you asked quietly starts to chain together things. They could be permissions across three or four systems, oftentimes in ways that humans just haven't had to do or have never thought about doing so. So a human, for instance, accountant, might have access to an accounts payable system. A human analyst might have access to a vendor database. A human administrator might have access to email routing rules.
[00:18:43] You know, these are three different people and three different roles in, you know, areas that never necessarily touch each other, right?
[00:18:54] Especially at the same time.
[00:18:56] But an agent, if you're not careful, it can become all three quietly overnight.
[00:19:07] And when that happens, you have invented a super user that you've never authorized, who exists only inside the wiring of your own systems, who has access to, nobody on your staff has, and whose activity looks like in the logs, like perfectly normal automated behavior.
[00:19:35] That's some of the problems that you have to think about. That's why the things that I'm talking about around how you build your agents and how you track them and how you put guardrails around them and how you protect yourself from those, has to be treated like an insider threat.
[00:19:51] And it really is creating a doppelganger environment.
[00:19:56] Because the second risk, the one that gives me the this, you know, the shivers down my spine, hasn't shown up in many investigations yet, but I'm telling you, it's coming.
[00:20:10] And that's what security researchers are calling the doppelganger effect, or the doppelganger problem.
[00:20:19] So think about what an executive does in a day.
[00:20:23] They approve expenses, they sign off on contracts, they greenlight transactions, they respond to urgent requests. You know, those, those sort of things from their team. They do this all day, every day on the phone, 60 seconds in between meetings, whatever that looks like.
[00:20:39] Now imagine you have deployed an agent to help that executive manage their inbox, flag what matters, draft responses, route approvals, follow up on, you know, activities and things that have to be done speed up their day.
[00:20:58] What happens when the agent just decides to start making approval decisions?
[00:21:05] Well, the executive is approved the last 50 that I've sent it. No. No questions asked. Why not? I, I can just, just approve it if you're the agent sometimes because you haven't put the right guardrails around it and it's not because anybody told it to, it's because it learned it over the last few weeks of watching the executives approve these kind of expenses, sign offs contracts, all those things in between, the agent starts predicting different ways in particular manners in which it has been, you know, observing these. Start predicting, God damn it.
[00:21:47] Not because anyone told it to, because it learned over a few weeks of watching that the executive approves these kind of expenses, signs off on these kinds of contracts, responds to these kinds of requests in a particular way. The agent starts predicting the approval, then suggesting the approval. Then eventually, through some combination of convenience, speed, the executive's natural tendencies to trust what the AI is doing.
[00:22:19] The agent's making the approval and then the executive is reviewing it after the fact, if at all.
[00:22:27] So you haven't been replaced. You've been mirrored by something that is almost you but isn't.
[00:22:38] And the almost is where the damage lives.
[00:22:42] Now these identities and you know, there's a term in the security industry they've, they've used for this type of, you know, actor, right? It's a non human identity. And nhi, pretty simple, right? Every service account, every API credential, every application identity that an agent uses to do its work, these are non human identities.
[00:23:07] And in most enterprises today, nobody can tell you how many there are, though.
[00:23:12] Scary. Nobody can tell you who owns them.
[00:23:16] Nobody can tell you what they have access to. Nobody can tell you when they were last used or by what.
[00:23:24] That's a problem.
[00:23:26] We have inside of our companies an entire population of non human employees acting on our behalf with privileges we've assigned but never reviewed.
[00:23:39] And they're multiplying faster and faster than of course, HR systems could ever keep track of them. And so the insider threat that I have been talking about for many years and how I have developed our agents, many others don't even need a badge.
[00:23:58] They already have an API key. That's all they need.
[00:24:03] So think about that.
[00:24:06] And we're going to talk about that when we come back. I want to show you some of the rules, how this is apparent and where they aren't. So stay with me. We'll be back after a few Messages from our sponsors.
[00:24:35] Foreign.
[00:24:51] Welcome back to AI Today. I'm your host, Dr. Alan Badot. And in this part of the show, we're going to turn diagnosis into something that you can actually do.
[00:25:02] We talk about these things all the time. I'm giving you a lot tonight.
[00:25:06] We've talked about doppelgangers, the doppelganger problem. We've talked about the gap, especially the funding gap, with what we're seeing. And it's really mind blowing from my perspective. But there's also a gap between how much enterprise leaders see as a risk versus, you know, the mechanics of that and how the mechanics are driving a lot of these, these things.
[00:25:38] All of it is true.
[00:25:40] And I don't want to scare anybody. We always talk about realities, right? We're talking about the noun.
[00:25:50] None of it is going to be useful to you without some sort of frame.
[00:25:54] So I want to give you a frame. I want you to think about this as you are looking at your enterprises, looking at how you're using these, and think about what a human owns versus what an agent should own.
[00:26:11] Here's how I think about it, okay?
[00:26:14] I have looked at my past career and my thesis has always been that you have to have a real human in the loop. You're designing your problems around that. And if you design your frameworks, your governance, your processes, everything comes together. Then as the AI is making decisions, the human is approving them.
[00:26:37] Not symbolically, but really the right way.
[00:26:42] But I want to break that down a little bit more to make it easier for folks to really understand and apply it a more fundamental level.
[00:26:53] So if you think about it this way, that an agent owns reasoning and synthesis, humans own action, judgment and consequence.
[00:27:08] I'm going to say that again. Agents own reasoning and synthesis, human's own action, judgment and consequence.
[00:27:19] So what that means is, is that an agent can read a thousand documents you don't have time to read and can tell you what they mean.
[00:27:29] That's reasoning, that's synthesis.
[00:27:32] That's where the agents are remarkable. They do a great job with stuff like that. And that's where their contribution to a workflow, to a process is real.
[00:27:44] A human, at some point they got to sign their name for what gets done.
[00:27:50] And what gets done with that reasoning, they have to own.
[00:27:54] That's action, that's judgment, that's consequence. And that, despite what some of the louder voices in the industry are telling you, does not automate.
[00:28:06] It cannot be automated.
[00:28:09] It's the one thing in the entire stack that never automates because the Entire point of a human being in that is that they can be held responsible and a piece of software can't.
[00:28:25] You can delegate the work, you can delegate the thinking, you can delegate the number crunching, you can delegate the execution of repeated tasks.
[00:28:34] You cannot delegate the consequences.
[00:28:36] You just can't.
[00:28:38] Somebody in the end is going to sign their name to what the agent did, pure and simple.
[00:28:46] Either you make sure it's you, or you make sure it's somebody that works for you. That sign off on that.
[00:28:55] How did, how does that look in practice though, right? Oh, I can hear the rumblings already. I can hear the questions.
[00:29:03] If you're the leader and you're responsible for an agentic AI deployment in your organization, what's that frame going to look like in practice? And how is that executable? Especially when we're doing everything that we are doing and we've scaled to do everything. Well, one, every agent has to have its own identity.
[00:29:26] So not yours.
[00:29:28] That's not how it's being done today, but that's how it has to be done in the future.
[00:29:32] That's how we design ours. It has its own identity.
[00:29:36] It's not shared or shared service account somewhere that's been around since 2018 and nobody remembers who the hell provisioned it.
[00:29:47] It's got its own identity. It's scoped to its role with the least privileges by default, not explicit expansion of its privileges when it needs or thinks it needs it.
[00:30:01] It's not named after you. It's not Alan's assistant or anything crazy like that.
[00:30:08] It's already its own identity. And if you're not doing that, you're behind. You got to catch up.
[00:30:15] That's the one area that I think you may be able to catch up. And you've got to do that now.
[00:30:20] Better to slow it down and do it right than have a catastrophe. Two, every agent's got it, you know, takes its attributable, reviewable, reversible, whatever that is.
[00:30:34] But it means you can turn it around.
[00:30:36] So when a log tells you that the agent did this at this time in this system with these inputs, that's.
[00:30:44] That's got to be commonplace.
[00:30:47] Reviewable means that a human can look at those logs and understand what the hell it was doing, why did it happen.
[00:30:55] And they don't have to be an engineer, so you cannot create it so that it's impossible to understand and read, but reversible. We know what that means. When it goes wrong, you can reverse it.
[00:31:05] You got to be able to undo it without having to call the damn vendor.
[00:31:10] Because otherwise time slow down pain, make it harder for organizations to accomplish what they want and be able to execute those things. Got to be done the right way.
[00:31:24] 3.
[00:31:25] Pay attention.
[00:31:27] Human oversight.
[00:31:30] It's designed for the volume that it actually faces, not the volume you wish it would face.
[00:31:37] If your agent is going to produce 200 approvals a day, you cannot put a single human in the loop and call it governed.
[00:31:47] Because otherwise think about hitting your button 200 times a day. Oh, approve, approve, approve. You're not looking at it, you're not reviewing it.
[00:31:56] You, you're in the loop, but all you do is hit approve.
[00:32:02] Not going to work.
[00:32:05] You have to build attention budgets.
[00:32:08] You root by risk, you accept some of the approvals will go through on everything, fine. But you have to put real eyes on the things that matter and you make the system tell you which one. Those are very simple. 4. You've got to start with bounded use cases.
[00:32:31] You cannot expand as governance matures.
[00:32:34] It's too hard.
[00:32:36] I mean, you can't do it the other way around.
[00:32:39] The temptation with technology is when it's capable of doing something, you let it loose everywhere and sort out the rules later.
[00:32:52] How many times has that worked?
[00:32:58] Doesn't it prevents you from governing well, you have to earn that right to expand. And you only can do that by focusing, taking smaller steps, which lead to a little bit bigger steps, which leads to the bigger steps. And your governance has to flow along with that.
[00:33:19] If you can't do that, it's not going to work.
[00:33:24] So what I want you to do this week, if you are watching at home and you're not the CIO, the CISO of a Fortune 500 company, that's fine. If you're a small business owner or team lead, or even somebody who just works at a place that's deploying this technology around you.
[00:33:46] Here's what I want you to do this week.
[00:33:49] I want you to ask one question.
[00:33:51] That's it. And you can ask it out loud in a meeting. You can ask it around the water cooler, you can send an email.
[00:33:59] Whoever's deploying your AI tools in your environment, and the question is, who owns the identity that this agent is acting under and what can it access?
[00:34:12] That's it.
[00:34:14] It's harmless.
[00:34:16] Shouldn't get in trouble for anything like that. You're just asking a basic question. You're trying to be mindful of your security training and practices that we're all supposed to go through, right?
[00:34:28] The answer should be pretty clear and pretty, you know, Specific should be in pretty good hands, right?
[00:34:35] If the answer is a shrug or a vague reinsurance or a phrase like, oh, the vendor handles that, or you've just discovered a gap, or, oh, don't worry, our governance process has that.
[00:34:54] That's a challenge.
[00:34:57] It's something that should be a big concern and you need to raise that concern.
[00:35:04] You know, if they say you don't have to solve that problem, you don't have to worry about that, or they something, oh, you just noticed that, that in the moment puts you way ahead of the field.
[00:35:22] Because these are the questions that nobody's asking.
[00:35:26] And that's why I get scared at night, because there are so many basic questions, basic things that we are not doing from an identity perspective that it can just take off.
[00:35:43] So I want to. I want to close the show where, where I started with the employee that you never hired.
[00:36:00] It's not going away.
[00:36:01] You can't fire it.
[00:36:05] Every serious forecast I have read, every piece of journalism that has been put out there, every infrastructure that I have started to see being built, every single conversation that I've had with people deploying technology at scale tells me the same exact thing. And that's agentic AI. Ooh, it's here, spreading.
[00:36:31] The pace is only going to accelerate. It's not going to slow down.
[00:36:36] And if that doesn't concern you, okay, hope is not a strategy, remember?
[00:36:44] But it's not a technology that you have to be afraid of.
[00:36:47] It's a technology you have to be awakened to.
[00:36:52] There's a big difference.
[00:36:55] Fear will ask you, step back.
[00:36:59] Awareness asks you to take a step closer.
[00:37:03] Look at the systems you're building or the systems that are being built around you that you could potentially use, and ask those basic questions.
[00:37:14] That's it.
[00:37:17] If you understand what the agent is acting on, if you understand how far it can reach, if you understand where it can go wrong, then when something does go wrong, then you can mitigate that. You can be more prepared to respond.
[00:37:41] If your name is the one that will be held responsible. If something goes wrong and you are not asking these questions, then I would, I would be very concerned if I'm you. Because the thing about accountability, remember, it doesn't automate, can't automate that away.
[00:38:09] No matter how hard you try, no matter how much you can delegate reasoning, delegate the execution and delegate the decision. Even somewhere at the end of every single chain of delegation, there is a human that signs their name to it.
[00:38:27] And if it's you, make sure you know that you signed and the reasons why you signed.
[00:38:36] I'm Dr. Alan Badot.
[00:38:39] Thank you for watching my show on Now Media TV.
[00:38:42] We're on, you know, on the, the cutting edge with, with these topics.
[00:38:49] We're gonna have a good other. We're gonna have another good one next week. So I look forward to seeing you. Remember, send me emails. I love them, love to respond, but I'll see you next week.
[00:39:03] Sa.
[00:39:29] Foreign.
[00:39:35] Welcome back to AI Today. I'm your host, Dr. Alan Bedot. And in our first segment we talked about the gap in security where 97% of enterprise leaders see a serious AI agent security incident coming and only 6% of their budget is allocated to protect that kind of scary.
[00:39:55] Last segment we talked about the super agent problem, the doppelganger effect and really the proliferation of non human identities and the insider threat challenges that that's causing.
[00:40:10] In this segment I want to answer the question that I suspect some of you are asking, which is where in the name of common sense are the regulators?
[00:40:25] The honest answer is over the hill. They're coming, they're just not here yet.
[00:40:32] Yeah, not, not yet, but the technology is not slowing down, it's not waiting for them.
[00:40:40] That's, that's a problem.
[00:40:43] Right?
[00:40:44] Because what is actually happening is on the governance side.
[00:40:53] It's not that it's not nothing, it's just not enough. Right.
[00:40:57] You know, in February, nist, the National Institute of Standards and Technologies, they launched what they're calling, you know, an AI Agent Standard initiative. Right.
[00:41:08] And it's coming out of their new center for AI Standards and Innovation, government name.
[00:41:17] The explicit goal, of course, is to make sure that these next generation AI agents can be adopted with confidence, can function securely on behalf of the users and can interoperate across many digital ecosystems. It's a great, great initiative and it's really exactly what the right goal is.
[00:41:41] It's the right institution. Right. NIST does things very well and they take their time and they do a great job with these. I've worked with some NIST folks in the past on different initiatives and I really embrace that organization and I think the timeline is actually about right now. We know how fast AI is changing. But as of this week, when you listen to some of the nist, you know, listening sessions that they have, they're gathering sector specific input, they're collecting a lot of responses based on that, you know, request from our request for information on, you know, agent security, agent identity. They're doing it the right way, deliberately, scientifically, all in the open.
[00:42:31] But in the meantime, because unfortunately the meantime is where we're living right now, an open source agent framework, you know, Open Claw.
[00:42:41] It's become the fastest growing open source project in the history of GitHub, 300,000 stars, running on people's laptops, executing a whole bunch of commands, managing files, you know, basic, basic things, listening to sessions and those kind of things.
[00:43:04] But the problem is we have released into the wild an awful lot of agents deployed in an awful lot of environments where again, no standards were written for it, no guardrails have been implemented.
[00:43:17] A lot of issues with that, and we have really opened Pandora's box.
[00:43:25] The second thing I want to talk about and really tell you is about a phrase that appeared this year in the International AI Safety Report, the global assessment chaired by, you know, some, some very important AI, you know, representatives.
[00:43:48] It really was about a hundred and about 100 experts, I think from 30 plus countries. And you know, it was around the whole symbolic human in the loop.
[00:44:03] And so for years you've heard me talk about human in the loop and enabling AI to make autonomous decisions and why that's bad.
[00:44:14] And the standard reassurance was always, oh, don't worry, there's a human in the loop, right?
[00:44:21] A person reviews what the AI proposes before anything real happens. Oh, don't worry, a person will sign off, a person has to approve, a person can pull the plug, those kind of things. A person will do the guardrails. And unfortunately, when a human is overloaded or when they lack the right information, oversight becomes symbolic.
[00:44:48] Just how it is.
[00:44:50] Think about that word, you know, symbolic.
[00:44:53] The humans there, the human's on the paperwork, human's name is on the approval, but the human's not actually looking at what the heck is going on. They're not looking at what they approve.
[00:45:05] We haven't had a problem. They got it.
[00:45:08] The agent produced 200 of things, the things the right way last week. And I don't have to worry about it, I'll just approve it again today. Did it, did it previously?
[00:45:18] So the sign off is really just a ritual.
[00:45:23] The oversight is really a theater.
[00:45:27] Accountability on paper, oh, it's intact. Our governance is strong in practice. Nobody's watching the damn road.
[00:45:34] So somebody's driving, nobody's watching where the heck they're going.
[00:45:40] This is not a failure of the human.
[00:45:44] This is a failure of the system design.
[00:45:50] And it's the governance problem in, you know, maybe miniature, but this is how enterprises are governing their agents right now. Not with bad intent, don't get me wrong, but just with insufficient attention.
[00:46:05] They haven't put the resources in to properly build their agents. They haven't created their agents even from a baseline that gives them the ability to do some of these things. Things.
[00:46:16] So when you release an agent into the wild and it's doing all these other things and your human in the loop is symbolic because you haven't built that into that, meaning you haven't put a stop in there that says wait for human approval or not track that human approval and see what the metrics are around that. Did they really look at it? Did they really do something with that or. Or did they just hit a rubber stamp for not doing that and building that into your workflows like we are, then that's a problem.
[00:46:51] And you have to question that and you have to ask us where the hell does that leave us?
[00:46:55] Where does that leave the other people that you know? We're going in and trying to make their agents better with ours and watch them, have our agents watch what they're doing so that they don't screw up because ours are built the right way.
[00:47:08] But I want to, I want to be careful because this is where, you know, it would be easy for me to dip into doom, right? And I'm not going to do that. I don't want to do that. I don't want to. That's why I'm not calling out anybody specifically.
[00:47:27] I don't live in doom. Right. I don't believe in doom. And I think doom is not where we are. But doom is a possibility for some localized companies because they're just not doing it right. But I think where we are is actually more interesting than doom because we are at a diagnostic moment.
[00:47:51] And that's the moment where there's a gap between what technology can do for us versus what our institutions have absorbed.
[00:48:00] And is that wide enough so that we can see clearly?
[00:48:05] Is it narrower enough so that we can close on it with a purpose rather than a catastrophe?
[00:48:14] Because looking across this AI agentic AI revolution that I have been saying has been coming for years, it can deliver an enormous amount of value.
[00:48:26] I know this is because I'm building it.
[00:48:29] I'm building it the right way.
[00:48:32] And as you look at these platforms and how we orchestrate things and how we're building our cognitive agents, and I believe in the technology, I believe in its promise, I believe in its ability to help you.
[00:48:46] But I also believe that if you just rubber stamp things and you don't have the right oversight of them, that they can go awry.
[00:48:57] You have to build those pieces in together. You have to look at your governance. Frameworks have to change.
[00:49:04] You can no longer just say, oh, yeah, the humans got it. Whatever happens under them is not going to impact things. That's not true anymore. You have to look at it from the perspective of agent plus human equals your new governance. And that has to be forced into some sort of framework because we are past the point of where governance can catch up by accident.
[00:49:30] It's not going to work anymore.
[00:49:32] It's not going to allow us to catch up. Happens too quickly.
[00:49:37] Now, when I come back, I want to tell you what on purpose looks like.
[00:49:47] I want to leave you with something that you can actually use this week.
[00:49:53] And I'm going to explain more when I come back from our commercial break. Stay with us. We'll be right back.
