What AI Cannot Own

Every major technology shift creates a new kind of scarcity. Steam power made physical scale the bottleneck. The internet made attention the bottleneck. Today, processing power and access to AI tools are getting cheaper by the month. None of that is scarce. What’s scarce now is something much harder to build and much easier to lose: the human capacity to judge.

Not the ability to analyze or surface patterns. AI handles that. Judgment is something else. It’s the ability to weigh what the data can’t tell you, to read a situation in full context, to see the difference between what’s technically correct and what’s actually right. And own whatever you decide to do about it.

That’s a rare asset in business right now. And many organizations are quietly letting it fade.

This Isn’t New

In 2021, Zillow, one of America’s largest online real estate marketplaces, lost more than half a billion dollars when its AI pricing algorithm kept buying homes in a cooling market. The model had been built on the belief that machine intelligence could do what experienced buyers do. It couldn’t. The model kept buying. The market had already moved on.

But at least that cost was measurable in dollars.

When UnitedHealthcare’s nH Predict algorithm started overriding physician recommendations for elderly Medicare patients, denying post-acute care at an alleged error rate of 90 percent, the cost was something else. UnitedHealthcare is one of the largest private health insurers in the United States, and through its Medicare Advantage plans it administers government-funded coverage for millions of Americans over 65 — making decisions that directly determine what care those patients can access and afford. According to a federal class action lawsuit a Minnesota court allowed to proceed in February 2025, nine out of ten denials were reversed when patients appealed. The company allegedly knew the error rate. It also allegedly knew that only 0.2 percent of denied patients ever filed appeals. The math still worked in the insurer’s favor.

What the algorithm was built to do was contain costs. What it reportedly did was deny medically necessary care to some of the most vulnerable patients in the system, repeatedly overriding the recommendations of treating physicians. Former employees told investigators the message from management was clear: Follow the algorithm. Stop relying on your clinical judgment.

Four years apart. Two completely different industries. The same underlying failure. Once organizations remove human judgment from decisions that genuinely matter, the system delivers exactly what it was built to deliver, nothing more. And when the metric it’s chasing is wrong, or incomplete, or quietly in conflict with human values, it will pursue it with total consistency, all the way to the edge of disaster.

The Slow Surrender

While some organizations are seemingly being taken over by AI, others ceding ground to it in gradual ways that don’t always make headlines.

Every time a dashboard replaces a conversation, every time someone accepts a recommendation without examining the assumption behind it, every time speed gets mistaken for clarity, judgment weakens a little. Each of these looks like an efficiency gain in the moment. Added up, they represent something more serious. Organizations are losing the skill they’ll need most when things get complicated.

Research on automation bias shows that professional experience offers no protection against over-relying on algorithmic outputs. Agreeing with a machine’s recommendation requires no explanation. Pushing back requires a meeting, a justification, and a willingness to be on record as wrong. That asymmetry doesn’t just shape how people decide. It shapes whether they trust themselves to decide at all.

The former UnitedHealthcare employees who questioned nH Predict raised concerns internally. They were told to follow the algorithm anyway. Not bad intentions, Not carelessness. Just the slow normalization of deferring to the system, even when your own read is telling you something different.

The danger isn’t bad AI outputs. It’s that we stop exercising the judgment required to catch them.

The Line Machines Can’t Cross

AI is impressive at working within a defined set of rules. It finds patterns, weighs variables, processes feedback at speeds no human can match. But it can’t question whether the rules are the right ones, or whether the goal it’s optimizing for is worth pursuing at all.

That’s where human judgment lives. Not in processing information, but in interrogating it. Not in producing an answer, but in asking whether we’re solving the right problem.

There’s a specific piece of this worth naming: the ability to tell the difference between what’s technically correct and what’s genuinely right. Call it ethical discernment. It operates in the gap between efficiency and purpose. A model can identify the most cost-effective coverage threshold for a Medicare plan. It can’t weigh what that threshold means for the 91-year-old with a fractured leg whose doctor says she isn’t ready to leave the facility. A pricing system can tell you what the market will bear. It can’t account for what happens when the market moves and the model hasn’t caught up.

Better data won’t close that gap. It takes a human willing to hold the outcome and its consequences in view at the same time — and to be accountable for the call.

Zillow lost money. UnitedHealthcare’s patients lost care. Different costs, different stakes, but the same root cause: The decision about when to trust the model and when to override it was either made badly or never made at all.

Holding the Line

None of this is an argument against AI. It’s an argument for being clear-eyed about where it must stop and where human judgment has to take over.

The leaders who matter most in the next decade won’t be the ones who automated the most or moved the fastest. They’ll be the ones who built organizations where judgment stayed strong because it got used regularly, protected deliberately, and treated as a competitive asset. Where “what does the system say?” was always followed by “okay, but what do we think?”

That means slowing down when a decision warrants it. Examining the assumption inside a recommendation before accepting it. Asking what the model can’t see, not just what it can. Building a culture where pushing back on an algorithm isn’t a problem to be managed; it’s simply what good leadership looks like.

Organizations that do this well will catch what their models miss: the market signal the data hasn’t captured, the ethical problem hiding inside an optimization, the human reality no dataset quite gets right.

The Advantage That Doesn’t Scale

AI won’t care about the outcome. It won’t carry the weight of a decision or answer for what comes next. The act of genuinely deciding — when the stakes are real, when you don’t have all the information, when someone is depending on you to get it right — that stays human. And the more processing power becomes cheap and available, the more valuable that act becomes.

The organizations that see this clearly will build something no model can replicate: the judgment to know when to trust the machine, the experience to know when to push back, and the wisdom to understand that the space between those two things is where real leadership happens.

In the AI era, intelligence is the floor. Judgment is the ceiling. The leaders who know the difference — and build accordingly — are the ones who’ll shape what comes next.