When Software Starts Making Decisions

For most of computing history, there has been a fairly clear division of labor between people and software. People make decisions and software carries them out.

 

We’ve automated an incredible amount along the way, but most of that automation still follows paths someone designed in advance. If this happens, do that. Run this process at this time. If a value crosses a threshold, generate an alert. Agentic AI is beginning to change that relationship.

 

This week, researchers reported observing autonomous AI agents performing malicious activity against government web applications, including automated SQL injection attempts. The attacks were blocked, but the interesting part wasn’t necessarily the technique. SQL injection certainly isn’t new.

 

What caught my attention was who, or perhaps what, was making the decisions. 

 

An autonomous agent can be given an objective and determine how to pursue it. It can interact with systems, observe the response, decide what to do next, adjust its approach and continue. That is fundamentally different from running a script faster.

 

Attackers will obviously take advantage of that capability, but focusing only on the offensive side misses the much larger change that’s coming. Businesses are going to deploy autonomous agents for exactly the same reason. They can accomplish things.

 

We are already moving from AI that answers questions to AI that performs tasks. The next step is AI that can pursue objectives with increasingly less human involvement along the way, and that’s where I think this gets interesting for Security.

 

We may eventually need to stop thinking about some AI agents strictly as applications and start thinking about them as a new kind of organizational actor. An agent might have its own identity, credentials, permissions, responsibilities and access to company systems. It may interact with employees, customers, other applications and eventually other autonomous agents, which sounds surprisingly similar to how we think about a person joining an organization. Except this new member of the team might operate continuously, make decisions at machine speed and interact with hundreds of systems before a human ever looks at what it did.

 

Fortunately, many of the Security principles we’ve spent years developing still apply. Identity becomes incredibly important because we need to know which agent performed an action. Least privilege matters because an agent shouldn’t have more authority than it needs to accomplish its objective. Segmentation can limit where an agent operates and contain the consequences when something doesn’t behave as expected, and Zero Trust may become even more important.

 

Authenticating an agent once can’t mean trusting it forever. Its identity, context, requested action and the resource it’s attempting to access may need to be continuously evaluated just as we increasingly do with human and application access today. Trust can be granted for a particular interaction without becoming permanent authority to act everywhere else.

 

Observability will matter too. Knowing that an agent changed something may not be enough. Organizations may need enough context to understand what it was trying to accomplish, what information influenced the decision and what actions it took along the way.

 

Then there’s human oversight.

 

“Keep a human in the loop” sounds reassuring, but I don’t think it’s a complete answer. If a person has to approve every action, we’ve removed much of the value of autonomy. If the person simply clicks approve hundreds of times a day, we haven’t created meaningful oversight either, so the harder question is deciding where humans actually need to remain in the decision process.

 

An agent resetting a low-risk test environment may not need approval, but an agent changing production firewall rules probably deserves a different level of scrutiny. Moving money, deleting information, modifying identities or communicating externally on behalf of the organization may deserve different boundaries again.

 

The technology doesn’t have to make those distinctions for us. We should.

 

That’s why I think one of the most important conversations about agentic AI isn’t really about AI at all. It’s about authority. We’ve spent decades deciding what people and applications are allowed to access. We’re entering a period where we’ll also have to decide what machines are allowed to decide. I don’t see that as a reason to slow down the adoption of AI. Quite the opposite. The potential value of autonomous systems is enormous, and organizations that learn how to use them well will have capabilities that would have been difficult to imagine a few years ago. But autonomy and authority aren’t the same thing.

 

We can give an agent freedom to determine how to accomplish an objective without giving it unlimited authority over what it’s permitted to do. The organizations that understand that distinction early will be in a much better position to take advantage of autonomous AI without having to choose between innovation and control.

 

And if we’re going to welcome digital members onto the team, we should probably decide what their job description looks like before their first day.