The Best Documentation Request Is the One Nobody Has to Make

A recent study looked at what clinicians actually ask a hospital AI assistant to do. The answer was not what most AI benchmarks measure. Across 127,833 conversations from 6,342 physicians, advanced practice providers and nurses in 35 specialties, 36.2% of requests were about documentation and administration. Another 28.9% were knowledge retrieval. Diagnosis, the task most medical AI benchmarks are built around, made up 3.7% (1). David Talby summarized the lesson neatly: evaluate clinical AI on the tasks clinicians actually give it (2). That is a very different picture from the media message that AI is going to replace doctors.

I agree, and I want to push the point a step further. Most of that documentation work is predictable. It follows events we can see in the record, and much of it can be done before the doctor has to ask.

What clinicians ask for versus what we test

The study, by Vishwanath and colleagues, followed an institutional assistant over the first eight months of its roll-out at a large academic health system (1). The authors also assembled 58 public clinical benchmarks, drawn from the major evaluation suites and from frontier model reports. The median benchmark contained no documentation requests at all, and only 13 of the 58 reached even 5%. On average, a benchmark's task mix overlapped only 31% with what clinicians actually asked. Benchmarks designed to resemble clinical practice were no closer to real use than the others. MedQA, one of the most quoted, is 48% diagnosis and 0.1% documentation.

That gap matters for model selection. If we pick models by their scores on diagnostic reasoning, we are optimizing for a few percent of the real workload. We are also nudging ourselves toward large, generative, reasoning-heavy models, because those are the models that win diagnostic leaderboards.

Most documentation requests ask for text, and the text has decisions inside it

I expected that much of this documentation work would turn out to be simple decisions in disguise. The data say otherwise, at least on the surface. Of the 44,807 documentation requests, 95.6% asked for generated text. Notes, plans and instructions made up 46.4% of them, chart summaries 19.4% and replies to patient-portal messages 6.6%. Billing and coding, the most obviously decision-shaped work, was only 1.0% of all requests. Across every task type, requests whose intent was to decide or verify something made up 13.4% (1).

So clinicians mostly ask for documents, and generative language models are the right tool for writing them. But a clinical document is not just prose. A discharge plan contains a set of choices: which medications continue, whether follow-up imaging is needed, which service sees the patient next, which instructions apply. A chart summary contains a choice of which problems are active and which findings matter. Those embedded choices are where errors do the most harm, and they are decisions with a small, known set of answers.

That suggests a division of labor. A System 1 decision model, which is given the question and the allowed answers and returns a probability for each answer in a single pass (3, 4), makes the embedded decisions. A generative model writes the prose around them. The decisions stay bounded, calibrated and auditable, and the writing stays fluent. This is cheap: in our recent radiology study, a 3-billion-parameter decision model extracted 18 findings from a CT report in about a third of a second, with an AUROC of 0.95 and well-calibrated confidence (5).

Documentation can be anticipated

The more interesting point is about timing. Two-thirds of documentation requests (68.9%) concerned a specific patient (1). The patient's record already exists, and most documentation is triggered by predictable events in it:

• A clinic visit is scheduled, so a chart summary will be wanted.
• A report is signed, so the follow-up and critical-result questions arise.
• An encounter is closed, so the note and the coding questions arise.
• A discharge is ordered, so the instructions, medication list and follow-up plan are needed.

Chart summaries are the clearest case. They made up nearly a fifth of documentation requests, and the clinic schedule tells us days in advance which summaries will be needed. If we know which event calls for which document, nobody needs to type a request. The workflow can look like this:

1. A deterministic trigger fires on the event. This part is ordinary automation: a rule that says "when a discharge is ordered, prepare the discharge packet." There is nothing probabilistic about it, and there shouldn't be.
2. A decision model answers the embedded questions. One pass over the record returns a probability for every answer to every question the document depends on.
3. A generative model drafts the document around those answers.
4. Calibration decides what happens next. Confident decisions are pre-filled for one-click confirmation. Uncertain ones are flagged for a person, together with the evidence the model used.

This is the pairing of deterministic automation and probabilistic AI I described in an earlier post (6). Rules decide when to act, models decide what the answer is, and the confidence of the answer decides who acts on it.

Anticipation also helps with a problem the study found. More than a third of requests (36.1%) could not be answered well as posed, and 22.2% needed patient or local information the clinician had not supplied (1). A system that starts from an event in the record doesn't depend on the clinician remembering to paste in the right context. It already has the patient, the encounter and the reason the document is needed.

The economics depend on the decisions being cheap. Running a large model with long reasoning on every scheduled visit, signed report and discharge "just in case" would be expensive. A small decision model on the same events costs very little; in our study, a single workstation could keep up with a department's report volume comfortably (5).

An example from practice

A common task in radiology that is more safety than summarization is done before CT or MR scans. If IV contrast is to be administered, it is critical to know if the patient has a history of sensitivity to contrast media, and if so, the type of contrast administered and the nature of the reaction. Unfortunately, there is not a single place to go looking for this and so best practice is to do an exhaustive search in the EHR and also across radiology exams where contrast was administered. Renal function is also of interest in determining if and how to administer IV contrast.

MRI also has implanted device concerns ranging from dental implants to cardiovascular devices, nerve stimulators, cochlear implants, and the list goes on. Each device has a separate MRI compatibility profile so one must know the manufacturer and specific model number AND if the device is still implanted. This again requires an exhaustive search of procedure notes, radiology reports and clinical notes that may document non-electronic information.

Both of the above would benefit from automation which would improve efficiency and patient safety.

What could go wrong

Anticipation carries risks, and I would want any implementation to address three of them.

Rubber-stamping. A pre-filled answer with a "confirm" button invites people to click confirm. Automation bias, the tendency to over-rely on automated advice, is well documented. A systematic review of 74 studies found it is made worse by high workload, task complexity and time pressure, which describes most clinical documentation (7). Care providers are not immune. In a reader study, 27 radiologists rated mammograms with suggestions from a purported AI system. When the suggestion was wrong, the share of correctly rated mammograms fell from about 80% to 20% for inexperienced readers, and from 82% to 46% even for very experienced ones (8). The same review found that showing confidence levels with each output was one of the design features that reduced the effect (7). That is something decision models provide naturally. Pre-fill only above a threshold set on local data, show the confidence and the evidence, audit a random sample of confirmed answers, and track how often people override the model. A confirmation rate close to 100% is a warning sign, not a success.

Doing work nobody needed. Not every event leads to every document. If the system prepares five documents and the clinician needed one, the other four are clutter, and clutter becomes alert fatigue. Query logs are valuable here because they show which requests clinicians actually make after which events, so the trigger list can be built from real use instead of guesswork.

Moving the error rather than removing it. When a clinician asks for something, they at least know it was asked. When the system acts silently, a wrong answer can sit in the record unnoticed. Every anticipated document and decision needs a visible record of what was decided, by which model, with what confidence, so it can be found and corrected later.

The evidence we still need

The Vishwanath study tells us what clinicians ask for. It does not tell us whether those requests could have been predicted from the state of the record at the moment they were asked. That is a measurable question. With timestamped queries matched to record events, one could estimate what fraction of documentation requests follow a recognizable trigger, and how long clinicians waited between the event and the request. If that fraction is large, the case for anticipation becomes concrete, and health systems would have a way to rank which triggers are worth building first.

That is also a better kind of benchmark. Instead of asking how well a model answers questions we invented, we would ask how much of the real workload it can take off clinicians' hands, and how reliably.

What this means

The study's main conclusion was that benchmark scores say little about how clinical AI performs on most of the work it is actually given (1). I would add a corollary. The real workload is mostly documentation, documentation is mostly predictable, and the decisions inside it are the work System 1 models do best. Much of it doesn't need a chatbot waiting for a prompt. It needs a trigger, a small calibrated model for the decisions, a generative model for the prose, and a clear rule for when a person takes over.

The goal of clinical AI shouldn't only be to answer clinicians faster. It should be to make many of their requests unnecessary.

References

1. Vishwanath K, Lin H, Alyakin A, et al. Clinician use of language models diverges from how the models are evaluated. arXiv:2610.11069; 2026. https://arxiv.org/abs/2610.11069
2. Talby D. Substack note, November 11, 2026. https://substack.com/@davidtalby/note/c-378549929
3. Erickson BJ. Fast Thinking for AI. MDSynapse.org. https://mdsynapse.org/blogposts/fast-thinking-for-ai
4. Erickson BJ. Can System-1 Models Reduce p(Doom)? MDSynapse.org. https://mdsynapse.org/blogposts/can-system-1-models-reduce-pdoom
5. Erickson BJ. Any Language Model Can Be a Decision Model. MDSynapse.org; November 5, 2026. https://mdsynapse.org/blogposts/any-language-model-can-be-a-decision-model
6. Erickson BJ. Deterministic Automation and Probabilistic AI: Designing for Complementarity. MDSynapse.org. https://mdsynapse.org/blogposts/automation-and-ai-optimal-synergy
7. Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2012;19(1):121-127. https://doi.org/10.1136/amiajnl-2011-000089
8. Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation bias in mammography: the impact of artificial intelligence BI-RADS suggestions on reader performance. Radiology. 2023;307(4):e222176. https://doi.org/10.1148/radiol.222176