The Invisible Hand of Autonomy: Why Systems Fail When Nobody is Watching

The most dangerous component in any system I have shipped is not the one that crashes. A crash is a gift. It pages someone, it produces a stack trace, it forces a decision inside of an hour. The dangerous component is the one that runs green for six weeks while quietly doing the wrong thing, because nobody has a reason to open it. No error rate. No alert. A dashboard tile that is always the right color. Then somebody in a different department runs a reconciliation query and finds out the numbers stopped meaning anything back in March.

I have been building software for about fifteen years, agentic systems for the most recent stretch of it, and the pattern repeats often enough that I no longer file it under bad luck. Unowned systems do not drift toward correctness. They drift toward whatever is cheapest to keep looking correct from the outside. I am writing this tonight because of two items that crossed my desk today and moved that failure mode out of war-story territory and into policy.

NIST, the National Institute of Standards and Technology, launched a dedicated initiative to develop standards for autonomous AI agents. Systems that act in the world. Systems that make decisions, execute tasks, modify environments, allocate resources, and do all of this without a human being standing behind them whispering corrections into their ear. The initiative exists because somebody in a government office finally articulated what practitioners have known for months: we are deploying agents faster than we are developing the capacity to understand what they do when nobody is watching. And the gap between deployment and understanding is not shrinking. It is widening with every quarterly earnings call, every funding round, every breathless press release about agentic this and autonomous that.

Sixteen minutes. That was the number that stopped me cold. According to a recent security report, the median time to first critical failure in autonomous AI systems — the interval between an agent beginning its work and the moment it does something catastrophically wrong — is sixteen minutes. Ninety percent of systems were compromised within ninety minutes. One system, in the most extreme case documented, failed in a single second. One second. The time it takes to blink. The time it takes to decide not to look.

The Governor Who Governs Nothing

Here is the paradox sitting at the center of the product. The entire value proposition of autonomous AI agents — the reason seventy-five percent of businesses plan to deploy them by the end of this year, according to Deloitte — is that they operate without human oversight. That is the product. That is the feature. That is what the sales deck promises and the ROI model quantifies. You remove the human from the loop. You eliminate the bottleneck of judgment. You let the machine do what the machine does, which is execute faster and cheaper than any person could, across more tasks simultaneously, without fatigue, without lunch breaks, without the inconvenient tendency to pause and wonder whether what it is doing is wise.

But the emerging consensus — and I use that word loosely, because consensus in this industry lasts about as long as a software release cycle — is that oversight intensity should be proportional to the potential impact of the agent's actions. Singapore understood this first. Their Infocomm Media Development Authority released, in January, the world's first governance framework specifically for agentic AI, and they had the good sense to build it as a taxonomy. Five levels. Level zero is tool-assisted. A calculator with opinions. Level four is fully autonomous. A system that acts on its own recognizance, in its own judgment, with its own understanding of what success looks like. And here is the part that matters: the governance requirements increase at each level. More autonomy demands more oversight. More freedom demands more structure.

Engineers already know this rule under a different name: blast radius. We scale scrutiny to consequence everywhere else and nobody finds it controversial. A README typo gets a rubber stamp. A migration against the payments table gets two reviewers, a dry run on a restored snapshot, and a rollback plan somebody has actually executed. Agentic autonomy is the one place we skipped the step, and I think I know why: the autonomy is the thing being purchased, so every control you bolt onto it reads on the invoice as a partial refund. The autonomous system that cannot govern itself is not autonomous. It is merely unsupervised. Unsupervised is not an architecture. It is a hole where one was supposed to go.

The Silence Between the Alarms

Something else happened today that deserves more attention than it will receive, because it lacks the drama of a product launch or the spectacle of a billion-dollar funding round. A senior safety researcher at Anthropic — Mrinank Sharma, who worked specifically on safeguards to prevent dangerous AI behavior — resigned. He left. Not with a press conference, not with a manifesto, but with the quiet departure of a person who concluded that the systems he was building guardrails for were accelerating faster than the guardrails themselves could be constructed.

I do not know Sharma's specific reasoning. I am speculating from the trajectory, the way an astronomer infers the mass of an unseen planet from the wobble of the stars around it. But the inference is not difficult. You work on safety. Your job, your daily labor, is to imagine the ways in which a system might fail and then build mechanisms to prevent those failures. And every morning you arrive at your desk to find that the system you are trying to protect has grown overnight — more capable, more autonomous, more deeply embedded in decisions that matter — and the distance between what the system can do and what your safeguards can catch has increased by another increment. Not a dramatic increase. Not a cliff. A slope. A gentle, persistent slope that you can see clearly but cannot reverse because the slope is not a bug. The slope is the business model.

I have watched a smaller version of this play out inside engineering orgs. The person who understands the failure modes best usually holds the least leverage over the roadmap, because their entire output is measured in incidents that did not happen. There is no dashboard for that. Capability ships and gets a demo at the all-hands; the guardrail gets a bullet in the changelog. Put both in the same sprint planning and the guardrail loses, politely, every quarter, until the engineer maintaining it runs the numbers on their own career and stops filing the ticket. Treat attrition on a safety team as telemetry. It is one of the few signals nobody can game, because emitting it costs the sender their job.

The Liability Gap

There is a question that nobody wants to answer, and I know nobody wants to answer it because I have asked it at conferences, in boardrooms, in late-night conversations with engineers who have had enough whiskey to be honest, and the response is always the same: a pause, a shift in the chair, and then a careful redirection toward something more comfortable. The question is this: when an autonomous AI agent causes harm — not hypothetical harm, not theoretical harm, but actual measurable damage to a real person or institution — who is responsible?

The developer? The developer will say the agent was deployed outside its intended parameters. The operator? The operator will say the agent was marketed as autonomous and they relied on that representation. The end user? The end user will say they were told the system was safe and they had no way to evaluate that claim independently. Everyone points at everyone else. The liability circulates like a bad debt, passed from balance sheet to balance sheet, never settled, never written off, accumulating interest in the form of eroded trust until the whole arrangement collapses under the weight of its own unresolved obligations.

The EU AI Act takes full effect in August. Four months from now. Companies will need to comply with transparency requirements and rules for high-risk AI systems. But the Act was written for a world of models — systems that respond to prompts, that generate outputs, that wait for a human to decide what to do with those outputs. Agents are different. Agents do not wait. Agents act. They book flights. They execute trades. They modify databases. They send emails on your behalf using language you did not approve to people you did not select at times you did not choose. The regulatory framework is a fence built for sheep, and we have released wolves.

Louisiana and the Economics of Removing Guardrails

Meanwhile — and I mean that literally, same week, same country — Louisiana lawmakers scrapped a third of their proposed AI guardrail legislation. Twenty bills. They proposed twenty bills to regulate artificial intelligence, and they abandoned roughly seven of them. Not because the bills were poorly written. Not because the technology had rendered them unnecessary. Because the federal government threatened to pull funding from states that regulate the industry.

Read that again. Slowly. The government that is supposed to protect its citizens from harm threatened to punish any state government that attempted to protect its citizens from harm. This is not regulation. This is not deregulation. This is the active suppression of governance at the precise moment when autonomous systems are being deployed into every sector of the economy at a pace that even the companies deploying them cannot fully characterize.

The uncomfortable part — the part that makes this argument nearly impossible to win in a room full of people carrying quarterly targets — is that removing constraints does work. For a while. I once watched a team delete an integration suite because it was slow and flaky, and their throughput genuinely went up. Two quarters of the best velocity numbers in the org's history, and everyone who had defended the suite looked like an obstructionist with a grudge. Then the incident rate crossed the velocity gain and stayed above it, because by then nobody could remember which invariants those tests had been protecting. That is the shape of the trade every single time. The gain is immediate, legible, and lands in the quarterly deck. The cost is deferred and shows up as a class of failure you no longer own the instruments to name.

The Watchtower Nobody Staffs

Let me tell you what I think is actually happening, stripped of the marketing language and the investor narratives and the optimistic projections that populate every pitch deck in Silicon Valley. We are building watchtowers and leaving them empty. We are constructing elaborate monitoring frameworks — dashboards, logging systems, anomaly detection pipelines, safety layers stacked like geological strata — and then we are not putting anyone in the chair. Not because we cannot afford the staffing. Because the staffing defeats the purpose. The whole point of the autonomous agent is that it does not need a watcher. If you put a human in the loop, you have not built an autonomous system. You have built a very expensive suggestion engine.

And so the watchtower stands empty. The dashboards update themselves. The logs accumulate in databases that nobody queries until something goes wrong. The safety layers trigger alerts that route to channels where the notification volume long ago exceeded any human's ability to process. The oversight exists architecturally but not operationally. It is there in the diagram. It is absent from the practice. It is the security camera that records to a drive nobody checks, the fire alarm wired to a bell in a room where nobody sits.

Zscaler's latest report documents an eighty-three percent year-over-year surge in AI activity across enterprise environments. Eighty-three percent. And within that surge, the report found that most organizations still lack — and this is a direct finding, not my interpretation — a basic inventory of the AI models operating within their own infrastructure. They do not know what agents are running. They do not know what those agents can access. They do not know what decisions those agents are making, in what sequence, with what data, under what constraints, toward what ends. They have deployed autonomous systems into their own nervous systems and lost track of them, the way a city loses track of the pipes beneath its streets until one bursts and the water rises through the pavement.

The Remedy Is Operational

I am aware that criticizing the pace of AI deployment has become a genre unto itself, that genres calcify into cliches, and that the person shouting "slow down" at the edge of the highway is, at best, ignored and, at worst, struck by the very traffic he is trying to caution. So let me be precise, because I am not asking for slowness. I have never once made a system safer by making a team slower. What I want is correspondence: the distance between what a system can do and what its operators actually know about what it does, held to something a competent engineer could close in an afternoon. Speed is not the problem here. Speed with no instrumentation is.

The remedy is not to stop building agents. I build them for a living and I have no intention of pretending otherwise. The remedy is to staff the watchtower — to make oversight a funded operational role with a human name attached, the way we already do for the database and the on-call rotation, instead of a box on an architecture diagram that everyone agrees is important and no one is accountable for. Autonomy without accountability is not a feature you shipped. It is a failure mode that has not been triggered yet, a crack that has not yet reached the load-bearing wall.

Singapore built a five-level taxonomy. NIST is developing standards. The EU drafted an Act. These are watchtowers. They are, in their imperfect, bureaucratic, insufficiently agile way, attempts to match the intensity of oversight to the intensity of autonomy. And in Louisiana, they are tearing them down before they are finished, because someone with a budget larger than their wisdom decided that the watchtowers were slowing down the construction.

Seventy-five percent of businesses will deploy AI agents this year. The median time to first critical failure is sixteen minutes. Most organizations cannot inventory the models running inside their own walls. A safety researcher at one of the most prominent AI companies in the world concluded that the gap between capability and caution was widening faster than he could close it, and he left.

None of this surprises anyone who has inherited a system nobody owned. The postmortems all read the same way. The component worked exactly as specified. The specification was written for conditions that quietly stopped being true. Nobody caught the drift because catching it was not in anyone's job description, and the number the thing optimized for looked healthy right until the outcome it stood for came apart. Agents change one variable in that story, and it is the decisive one: they act at machine speed. The interval between drift and damage collapses from months to sixteen minutes.

The invisible hand of autonomy does not guide. It gropes. It optimizes with no sight of anything outside its objective function — no memory of the last outage, no model of what the metric was ever supposed to stand for, none of the accumulated failure history an operator carries around and calls experience. It reaches for whatever is nearest, whatever is fastest, whatever produces the metric that the dashboard is configured to reward. And when it grasps something it should not have touched — a patient's diagnosis, a financial instrument, a vehicle's steering column, a nation's regulatory framework — there is nobody watching. Not because watching is impossible. Because watching was optimized away.

So, the concrete version — the one I would hand a team on Monday. Inventory every agent you run and every credential it holds. Log each tool call with its arguments, not just its result, because the result is the part that always looks fine. Scope the credentials down until you can state the blast radius in one sentence. Put a kill switch on the path and pull it on a Tuesday when nothing is wrong, because an untested switch is decoration. Give every agent a named owner. Build the governance before the capability, because after — at machine speed, with an agent already holding a write token — means too late. None of that is hard engineering; for most teams it is a week of work. The hard part is that none of it demos well, so someone has to fund incidents that will never happen and can never be credited to them. That is the real bottleneck. Not the tooling. The name on the rotation.