Don't Let the Agent Drive Alone: The Safety Case for Hybrid AI Workflows in Healthcare

Imagine an AI agent managing a patient's transition from the ICU to a general ward. It reads the chart, synthesizes the notes, orders appropriate medications, schedules follow-up labs, flags nursing tasks, and drafts the handoff communication. Each step looks impressive. Each step is probabilistic. And by the time the seventh step completes, the probability that everything has gone exactly right is alarmingly low.

This is the central tension in clinical AI design today. Autonomous agents—systems that can read, reason, plan, and act across multiple sequential steps—promise to transform healthcare workflows. They can handle complexity and novelty that rigid rules cannot. But their power comes with a dangerous property: error compounds. In a seven-step workflow where each step is 95% accurate, there is only a 70% chance the entire workflow completes without error. At ten steps, that drops to 60%. At twenty, to 36%.

In a quiz, a wrong answer is a learning opportunity. In clinical care, it can be a missed medication, an incorrect diagnosis, or an adverse event. Healthcare can tolerate neither the arithmetic nor the consequences of purely probabilistic multi-step workflows.

The answer is not to abandon AI agents—they offer genuine and important capabilities. Nor is it to retreat entirely to rigid rule-based systems that brittleness eventually breaks. The answer is a deliberate hybrid: rules-based guardrails protecting the high-stakes, deterministic decisions; agents handling the flexible, reasoning-heavy tasks where rules can't generalize. The question for every clinical AI design team should not be "agent or rule?" but "which decision warrants which approach—and how do we combine them safely?"

The Rules-Based Tradition: Powerful, Brittle, and Still Necessary

The history of clinical decision support is largely a history of rules. Edward Shortliffe's MYCIN system, developed at Stanford in the early 1970s, is the canonical example: a rule-based expert system encoding roughly 600 if-then rules for antibiotic selection in bacterial infections. MYCIN's performance in blinded evaluations matched or exceeded infectious disease specialists, and it introduced a generation of clinicians and informaticists to the potential of computational decision support.¹

Rules-based systems have endured for good reason. They are deterministic: given the same input, they produce the same output every time. They are auditable: every decision traces back to an explicit logical chain that clinicians, regulators, and liability counsel can inspect. And they are predictable: a drug-drug interaction check that fires correctly 100% of the time—because it is deterministic—protects every patient, without fail.

But rules-based systems have equally well-documented limitations. They are brittle. Every clinical scenario not anticipated by the rule engineers falls through the cracks. They require enormous expert effort to author and maintain. They do not generalize—each new drug, each new clinical pathway, each new variant of a familiar presentation requires new rules. Over time, large rule libraries become internally inconsistent, generating contradictory recommendations and, eventually, the phenomenon that has arguably done more damage to clinical decision support than any algorithmic flaw: alert fatigue.

Studies have documented override rates for clinical decision support alerts exceeding 90% in some systems.² When rules fire too often, too indiscriminately, or for low-stakes situations, clinicians learn to dismiss them reflexively—and a genuine critical alert fires into the same cognitive environment as its 47 predecessors that day. Rules, deployed without discipline, can paradoxically reduce patient safety by training clinicians to ignore warnings.

The Appeal—and the Risk—of Pure Agent Workflows

Into this landscape arrived large language models and autonomous agents, offering something rules never could: the ability to reason about novel situations, integrate heterogeneous information, and adapt to context without requiring explicit programming for every scenario. An LLM-based agent can read an unstructured clinical note, extract the relevant history, compare it against treatment guidelines it has internalized during training, and produce a contextualized recommendation—all without a single hard-coded if-then statement.

This flexibility is genuinely valuable. Clinical reality is messy, polysemous, and rarely conforms to the clean categorical logic that rules require. Agents can handle that mess. The question is at what cost.

The cost is probabilism. Every inference an LLM makes is a probability distribution over possible outputs, not a deterministic computation. The model may hallucinate—generating plausible-sounding but factually incorrect outputs with high confidence. Ji and colleagues conducted a comprehensive survey of hallucination in neural language generation, documenting that hallucination occurs across all major model architectures and task types, including in domains requiring factual precision such as medical question answering.³ Even the best current models produce factual errors at measurable rates, and those rates do not fall to zero as models scale.

In a single-step application—a model generating a clinical summary that a physician reviews before acting—probabilistic error is manageable. The human is the verification layer. But in multi-step agentic workflows, where the output of one step becomes the input of the next without human review at each stage, errors do not stay isolated. They propagate, amplify, and compound.

The Mathematics of Compounding Failure

The reliability mathematics of sequential probabilistic systems are straightforward and sobering. If a single agent step succeeds with probability p, then a workflow of n independent steps succeeds with probability pⁿ. The table below illustrates what this means in practice:

A care coordination agent that performs ten steps—reading the chart, identifying follow-up needs, checking formulary, ordering medications, scheduling labs, updating the problem list, drafting the handoff note, notifying the receiving team, flagging unresolved issues, and documenting the encounter—needs each step to perform at 99.9% accuracy just to achieve 99% end-to-end reliability. Current LLM systems cannot guarantee that for clinical tasks. And unlike software bugs, which are deterministic and can be fixed once identified, probabilistic errors in language models are stochastic: the same workflow may succeed today and fail tomorrow without any change to the system.

Two further aggravating factors make this worse than the simple mathematics suggest. First, in agentic workflows, errors compound semantically: a misinterpreted medication name in step 2 doesn't produce a blank result or a system error—it produces a plausible-looking wrong value that becomes the input to step 3, which acts on it confidently, propagating the error forward. Second, the longer the chain, the less interpretable the eventual output: tracing a final error back to its origin in a ten-step workflow is far harder than catching an error in a single-step system, precisely when accurate attribution is most needed for patient safety investigation.

Case Studies in Pure-Approach Failure

IBM Watson for Oncology

Perhaps no single case better illustrates the risks of deploying AI agents into high-stakes clinical workflows without sufficient deterministic guardrails than IBM Watson for Oncology. Launched with considerable fanfare as a system capable of recommending cancer treatment plans, Watson was deployed across multiple health systems globally. Internal documents later revealed that Watson had been generating treatment recommendations that oncologists described as "unsafe and incorrect"—in some cases recommending chemotherapy regimens that contradicted institutional protocols or that no oncologist would plausibly endorse.⁴

The root failure was architectural. Watson had been trained substantially on synthetic clinical vignettes authored by physicians at a single elite institution, not on the real-world diversity of actual patient cases. More critically, it operated without rules-based guardrails that could have enforced hard constraints—formulary restrictions, contraindication checks, allergy alerts—before surfacing recommendations. The agent was permitted to generate outputs that deterministic rules would have blocked before they reached a clinician. Several institutions ultimately discontinued the program.

The Epic Sepsis Model

At the other end of the complexity spectrum, the Epic Deterioration Index—widely deployed as an AI-based sepsis early warning system—illustrates how even a well-resourced, single-step predictive model can underperform simple rules-based criteria when taken outside its validation environment. Wong and colleagues conducted an external validation study of the Epic sepsis model at a major academic medical center, finding an area under the receiver operating curve of just 0.63—barely better than chance—at the alert threshold used in clinical practice.⁵ Simpler rule-based screening criteria (SIRS, qSOFA) performed comparably while being fully transparent and auditable.

The lesson is not that machine learning cannot improve on rules—in many contexts it can. The lesson is that probabilistic models degrade unpredictably when deployed outside their training distribution, and that this degradation is invisible until someone does the kind of external validation study that should be mandatory before deployment—but often isn't. Rules degrade visibly and predictably: you know what a rule does and when it fires. An opaque probabilistic model can fail silently for months while appearing to function normally.

Why Rules Remain Irreplaceable for High-Stakes Decisions

The case for retaining rules-based logic in clinical AI is not nostalgia for older technology. It is grounded in fundamental properties of deterministic systems that probabilistic models cannot replicate.

Determinism matters for medication safety. A rule that checks for penicillin allergy before allowing an amoxicillin order either fires or it doesn't. Its behavior is perfectly predictable across every patient, every time, regardless of how recently the model was updated, what data was in the training set, or how the clinical note was phrased. An LLM checking the same allergy may phrase the question differently, interpret a note ambiguously, or—in the context of a long multi-step workflow—simply miss it. This is not a hypothetical concern; it is the documented failure mode of systems where language-based reasoning substitutes for deterministic data retrieval.

Auditability matters for liability and learning. When an adverse event occurs, the clinical and legal system requires an explanation of why the system recommended what it did. A rules-based decision support system can produce a complete logical trace. A multi-step agentic workflow—particularly one involving LLM reasoning—often cannot. Seshia, Sadigh, and Sangiovanni-Vincentelli have argued that verifiable AI requires the ability to formally specify and certify system behavior, something that is tractable for rule-based components and remains largely unsolved for neural models at the reasoning layer.⁶

Checklists—the simplest form of rules—save lives. Pronovost and colleagues famously demonstrated that a five-item checklist for central-line insertion—a purely rules-based intervention with no AI component whatsoever—reduced catheter-related bloodstream infections in Michigan ICUs from a baseline of 2.7 infections per 1,000 catheter-days to essentially zero over 18 months.⁷ The reliability that made this possible was the rule's determinism: either the checklist was followed or it wasn't, with no probabilistic degradation. The lesson is that for procedural safety—where failure is catastrophic and the required actions are known in advance—deterministic rules outperform probabilistic systems not because they are smarter, but because they cannot be wrong in the specific way that matters.

The Architecture of Hybrid Workflows

The productive frame is not "rules versus agents" but "rules as guardrails, agents as reasoners." The two approaches are complementary when applied to the right layers of a clinical workflow.

Topol's foundational articulation of human-AI partnership in medicine provides the right intuition: neither pure human judgment nor pure algorithmic decision-making is optimal; the combination outperforms either alone.⁸ The same principle applies within AI systems themselves. Rules provide the structural backbone—the non-negotiable safety checks, the hard eligibility criteria, the deterministic data lookups. Agents provide the reasoning layer—the interpretation, the synthesis, the generation of recommendations in context that rules could never anticipate.

A Framework for Allocating by Risk and Reversibility

Maddox, Rumsfeld, and Payne identified a critical design question for healthcare AI: what is the appropriate level of human oversight for a given decision?⁹ We can extend this into an explicit framework for hybrid workflow design:

High-risk, irreversible decisions → deterministic rules with mandatory human review. Medication orders, critical value alerts, allergy checking, eligibility for a clinical trial, contraindication screening. These must fire correctly every time. Even a 99% reliable agent leaves 1% of patients exposed to the error, which is unacceptable when the consequence is a fatal drug interaction. Rules handle these; agents do not.

Medium-risk decisions with contextual complexity → agent reasoning bounded by rules. Differential diagnosis generation, discharge planning recommendations, prior authorization narratives, treatment plan summaries. These benefit enormously from agent flexibility—clinical context varies, language matters, and rigid rules cannot generalize. But they should operate within a rules-based envelope: the agent generates the narrative, but the system first checks that no hard constraints have been violated before surfacing the output. The rules are the floor; the agent builds above it.

Low-risk, reversible tasks → agent-led with lightweight review. Administrative scheduling, clinical note summarization, patient education draft generation. Here the consequences of occasional errors are low, the output is reviewed before action, and the flexibility of agents provides clear value with acceptable risk.

The Aviation Parallel

Aviation offers the most instructive analogy. Modern commercial aircraft fly on autopilot for the majority of each flight—a sophisticated agent-like system that adjusts trajectory, altitude, and speed based on sensor data and routing logic. Yet modern aircraft are also loaded with hard-coded, deterministic safety systems: terrain proximity warnings, traffic collision avoidance, stall protection—systems that cannot be disabled by the autopilot and that fire with complete reliability regardless of what the autonomous systems are doing.

The result is remarkable safety. Not because the autopilot is infallible, but because the hard safety systems are. When the autopilot errs, the deterministic envelope catches the error before it becomes catastrophic. Healthcare AI should be designed with exactly this architecture: autonomous agents for the intelligent, contextual, adaptive functions; deterministic rules for the non-negotiable safety envelope. Neither is sufficient alone. Together, they achieve what neither can accomplish independently.

Char, Shah, and Magnus framed the core ethical challenge in implementing machine learning in healthcare as a question of accountability: when an algorithmic system causes harm, who is responsible, and how do we design systems that preserve the ability to answer that question?¹⁰ Hybrid systems, where rules-based logic is explicitly documented and agent reasoning is logged and bounded, preserve accountability in a way that pure agent workflows—where a chain of probabilistic inferences produces an outcome untraceable to any single decision—fundamentally cannot.

---

Conclusion: Design for the Spectrum, Not the Extremes

The debate between rules and agents is often framed as a generational conflict—old, rigid expert systems versus new, flexible neural reasoners. That framing obscures the real design question, which is not which paradigm is better but which is appropriate for which layer of a clinical workflow.

Pure rules-based systems are brittle, require enormous maintenance effort, and cannot handle the genuine complexity of clinical reality. Alert fatigue is a real cost of rules deployed without discipline. But pure agent-based systems are probabilistic, opaque, and mathematically guaranteed to fail with increasing frequency as workflow complexity grows. The compound probability problem is not a flaw that better models will eventually eliminate—it is a structural property of any system composed of uncertain sequential steps. No model, however large or carefully trained, achieves 100% accuracy, and no multi-step pipeline built from uncertain components achieves end-to-end certainty.

Healthcare is not a domain that tolerates frequent errors gracefully. The appropriate response is a deliberate hybrid: rules as the deterministic skeleton, agents as the flexible reasoning layer operating within that skeleton's constraints. The clinical AI systems that will earn—and deserve—adoption are those whose designers asked not "how much can we automate?" but "for which decisions can we tolerate uncertainty, and for which must we guarantee correctness?" Answering that question honestly, and building the architecture to match, is the work that makes clinical AI safe.

---

References

1. Shortliffe EH. Computer-Based Medical Consultations: MYCIN. New York: Elsevier/North Holland; 1976.
2. Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med. 2020;3:17. doi:10.1038/s41746-020-0221-y
3. Ji Z, Lee N, Frieske R, et al. Survey of Hallucination in Natural Language Generation. ACM Comput Surv. 2023;55(12):248. doi:10.1145/3571730
4. Strickland E. How IBM Watson Overpromised and Underdelivered on AI Health Care. IEEE Spectrum. 2019;56(4):24-31. doi:10.1109/MSPEC.2019.8678513
5. Wong A, Otles E, Donnelly JP, et al. External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Intern Med. 2021;181(8):1065-1070. doi:10.1001/jamainternmed.2021.2626
6. Seshia SA, Sadigh D, Sangiovanni-Vincentelli A. Toward Verified Artificial Intelligence. Commun ACM. 2022;65(7):46-55. doi:10.1145/3503914
7. Pronovost P, Needham D, Berenholtz S, et al. An Intervention to Decrease Catheter-Related Bloodstream Infections in the ICU. N Engl J Med. 2006;355(26):2725-2732. doi:10.1056/NEJMoa061115
8. Topol EJ. High-Performance Medicine: The Convergence of Human and Artificial Intelligence. Nat Med. 2019;25(1):44-56. doi:10.1038/s41591-018-0300-7
9. Maddox TM, Rumsfeld JS, Payne PRO. Questions for Artificial Intelligence in Health Care. JAMA. 2019;321(1):31-32. doi:10.1001/jama.2018.18932
10. Char DS, Shah NH, Magnus D. Implementing Machine Learning in Health Care — Addressing Ethical Challenges. N Engl J Med. 2018;378(11):981-983. doi:10.1056/NEJMp1714229