Any Language Model Can Be a Decision Model

What a radiology benchmark taught me about small System 1 models and about getting decisions out of ordinary language models

In two recent posts I argued that System 1 decision models, which return a probability for each of a fixed set of answers instead of writing free text, are a better fit for many clinical tasks than the large language models (LLMs) we usually choose. We tested this using three open models zero-shot (with no task-specific training) on seven radiology tasks covering 4,435 cases. Two results stood out. A small, 3-billion-parameter decision model extracted findings from radiology reports almost perfectly, at a fraction of the cost of the larger models. And a widely used general-purpose language model, Qwen3.8, worked as a decision model with no retraining at all, simply by changing how its output was read. This second result suggests the pool of models that can serve as System 1 models might be very large.

A quick recap: what is a System 1 decision model?

Kahneman described two modes of human thinking. System 1 is fast and cheap; System 2 is slow and deliberate. Most of the decisions in a radiology department fall in the first group. Is this series a FLAIR or a T2? Is the endotracheal tube in good position? Does this report describe a pleural effusion? Is this implant safe to scan? Each question has a small, known set of answers, and each is asked thousands of times a day.

A generative LLM answers such questions by writing text, one token (roughly one word) at a time, often after a stretch of "reasoning." That is slow and expensive, and the answer can drift outside the allowed choices. A System 1 decision model is given the question and the list of allowed answers. In one pass it returns a probability for every answer, for example T1 0.02, T1c 0.03, T2 0.78, FLAIR 0.17 (1). Because the answer set is fixed in advance, the output is always one of the choices the designer planned for. The probabilities can be calibrated against outcomes, and an application can act on confident answers and send uncertain ones to a person. I described this architecture in more detail in Fast Thinking for AI and its safety implications in Can System-1 Models Reduce p(Doom)?

What was tested

Our study (3) followed the zero-shot protocol of a companion evaluation of System 1 models across medicine by Yang, Vrettos and colleagues (2), but focused on radiology. The seven tasks came from public datasets and were chosen to span the work of a radiology department:

• finding 18 abnormalities in chest CT reports (CT-RATE) and 20 findings in Spanish-language chest radiograph reports (PadChest);
• finding the same 18 abnormalities on the chest CT images themselves;
• judging the MRI safety of implanted devices from their FDA registry records (GUDID);
• checking tube and line positions on chest radiographs (RANZCR CLiP);
• grading motion artifact on brain MRI (MR-ART); and
• identifying the MRI sequence (T1, T1 with contrast, T2 or FLAIR) of brain tumor scans (BraTS 2021).

Three open models ran locally on a single NVIDIA DGX Spark. Two were System 1 decision models: Clef, a 27-billion-parameter model from Cloudflare (4), and d1, a 3-billion-parameter model from Liquid AI (5). The third was Qwen3.8-27B (6), a general-purpose vision-language model that is also the backbone Clef was built from. That pairing allowed a direct comparison of the same weights with and without a decision head. The main measure was the area under the receiver operating characteristic curve (AUROC) on each test.

The full results, including the image tasks, are in our paper on arXiv (3). Here I want to focus on two findings that I think matter most in practice.

Finding 1: a small decision model reads reports very well

Pulling structured findings out of radiology reports is one of the most common jobs we ask of AI. It feeds registries, quality programs, follow-up tracking and the labeling of images for training other models. It is also a job where the answers are almost always on a fixed list: the finding is present or it is not.

On this task the small model did remarkably well. For the 1,000 English CT reports, d1 reached an AUROC of 0.951 across the 18 findings, and its top answer was correct 92% of the time. On the 1,000 Spanish chest radiograph reports its AUROC was 0.957. It had no special training for Spanish. Its probabilities were also honest. Keeping only the half of d1's answers it was most confident about, 99% were correct for the CT reports and 98% for the radiograph reports. Its calibration error was small (0.04 or less on both report tasks), meaning that when d1 reported 90% confidence, it was right about 90% of the time.

The 27-billion-parameter models were better, with AUROCs of 0.988 to 0.992 but at the cost of higher computation.

A separate timing run let me separate two sources of that cost: the decision head and the size of the model. The head is isolated by comparing Clef with its own backbone, which has the same size and weights. On a CT report with 18 questions, Clef answered all of them in one pass in a median of 1.5 seconds, using about 140 joules of GPU energy. Qwen3.8-27B, asked one question at a time, needed 4.2 seconds and about 375 joules, roughly three times more of each. That advantage comes from answering many questions at once. On the device records, where there is only one question, Clef was actually slower than Qwen (0.40 versus 0.21 seconds). Size is isolated by comparing d1 with Clef, since both are System 1 models. d1 answered the same CT report in 0.32 seconds with about 20 joules, about five times faster than Clef with seven times less energy, and across all seven tasks it used 3 to 9 times less energy per request. The two effects multiply, which is why d1 was about 13 times faster than Qwen3.8-27B per report and used nearly 20 times less energy. These figures come from different serving software and numeric precision, so the exact ratios should not be over-read; the orders of magnitude are the point. A model of this size runs comfortably on modest hardware, and a department could process its entire report archive overnight on a single workstation.

d1 did not do well on the image tasks. On the tube and line radiographs, for example, it was close to chance. I do not see that as a strike against it so much as a statement of fit: a 3-billion-parameter model is a good tool for reading text, and a poor one for interpreting images zero-shot. Choosing the right-sized model for each decision is exactly what a System 1 architecture makes easy.

Finding 2: an ordinary language model can be used as a decision model

The second result surprised me more. Most open models are generative: they are built to write text, not to return a probability for each answer. Purpose-built System 1 models are still rare. If only those few models can be used in a decision role, the approach is limited by whoever chooses to build them.

To compare Clef with its own backbone, I needed a way to get decision-style probabilities out of Qwen3.8-27B. The approach was simple. Each question was posed as a multiple-choice item with lettered options, with the instruction "Answer with one letter." Its internal "thinking" mode was turned off. The model was then allowed to produce exactly one token, its first. Language model servers can report how likely the model considered each possible first token. I kept only the probabilities of the option letters (A, B, C or D) and rescaled them to sum to one. The result is a probability for every allowed answer from a single generated token, which is the same kind of output a System 1 model produces. Letters were used rather than the answer words themselves because answers often share a first word (all three MRI safety classes begin with "MR"). In a separate check of the same prompts, the model returned exactly one option letter every time.

This worked better than I expected. On the report tasks, Qwen3.8-27B used this way was the best of the three models, with AUROCs of 0.992 on the CT reports and 0.989 on the Spanish radiograph reports. It was also among the best calibrated, with a calibration error of 0.015 on both. On MRI sequence identification, its probabilities ranked the four sequences almost perfectly (AUROC 0.989), matching Clef.

Because Clef is a fine-tuned version of the same model, the comparison also shows what a purpose-built decision head adds. Figure 1 shows the differences. On the report tasks, the device records and the MRI sequence task, Clef and its backbone ranked cases essentially equally well. The head made a real difference on some image tasks, and a larger difference in how the probabilities turned into decisions.

Figure 1. Change in AUROC for each task compared with Qwen3.8-27B used as a one-token decision model. Blue diamonds: Clef 27B, a System 1 model fine-tuned from the same Qwen3.8 weights. Orange triangles: d1 3B. Vertical lines are 95% confidence intervals from 2,000 paired bootstrap resamples; a line that crosses zero means no clear difference. The shaded area marks the report-text tasks.

That last point deserves a word of caution, and it is the main limitation of the trick. A single-token readout inherits the model's preferences among the answer letters. On the MRI sequence task, Qwen3.8-27B ranked the sequences almost perfectly, yet it never once chose FLAIR as its top answer. It labeled most FLAIR scans as T2, because its probabilities were consistently tilted toward the other options. Ranking well and deciding well are not the same thing. In practice the fix is straightforward: set the decision thresholds for each answer on a few hundred local cases rather than simply taking the highest probability. The dedicated decision head in Clef largely removed this bias, which is a good argument for purpose-built models where they exist. But the bias can be corrected, and the underlying ranking is clearly there.

This raises the question of zero-shot use versus fine-tuning. Our study was deliberately zero-shot: no model saw a single labeled example from these tasks, because the goal was to measure how the models behave out of the box. Had we fine-tuned Qwen3.8-27B on even a modest number of labeled MRI series, I expect the FLAIR problem would have largely disappeared. The near-perfect ranking shows that the model already distinguished FLAIR from T2; what it lacked was a correctly placed boundary between them, and a few hundred labeled examples of each sequence are what would teach it where that boundary lies. Clef offers indirect support for this. Its decision head is itself the product of additional training on top of the same Qwen3.8 weights, and it correctly identified 90 of 100 FLAIR volumes, where its backbone identified none.

Fine-tuning has costs, though. It requires labeled cases, computing time and expertise. It produces a separate model for each task that must be validated, versioned and maintained, and it can erode the general abilities that made the model useful in the first place. Zero-shot use avoids all of that, which is what makes drawing on thousands of off-the-shelf models attractive. Between the two lie lighter options: recalibrating the decision thresholds on local cases, as described above, which leaves the model untouched, and small adapter layers (such as LoRA) that adjust a model with far less data and compute than full fine-tuning. The right choice depends on what the zero-shot model gets wrong. When its ranking is already good, as it was for MRI sequences, recalibration should be enough. When the ranking itself is weak, as it was for grading motion on brain MRI, only fine-tuning or a different model is likely to help.

The implication is broad. The same readout works with any open model that a standard server such as vLLM can host and that reports token probabilities, and that is most of them. Thousands of open models, including specialized medical and multilingual models, can therefore be used in a System 1 role today without waiting for someone to build a decision version. This approach keeps the important properties of a decision model: the answer is always one of the planned options, every option gets a probability that can be checked and calibrated, and no time is spent generating text or reasoning. It does give up some efficiency. Each question needs its own request, whereas a true System 1 model can answer many questions about the same input in one pass. That one-pass design is what made Clef about three times faster than its own backbone on the 18-question CT reports.

What this means

I came away from this study more convinced that decision models deserve a central place in clinical AI, and with a better sense of where they fit.

First, many of the tasks that matter most are decisions, not essays. Report abstraction, protocol checks, device safety screens and worklist triage all reduce to choosing among known options. For these jobs, a model that returns a probability for each option is not a lesser tool than a chatbot. It is the right tool. Its outputs are bounded, so the application can define in advance what happens for each answer. Its outputs are probabilities, so they can be measured, calibrated and audited, and uncertain cases can be routed to a radiologist.

Second, the efficiency is real and it compounds. A 3-billion-parameter model that reads a report in a third of a second with a small energy budget can be run on every report, every day, on hardware a department already owns. Generating text and reasoning for each of those questions would cost at least an order of magnitude more for little or no gain in accuracy.

Third, and most important to me, decision models are no longer limited to the handful that have been purpose-built. A one-token readout turns an ordinary open language model into a usable decision model, as long as its thresholds are set on local data. That widens the choice from a few models to thousands. It lets us pick a small, fast model for text, a larger one for images, and a domain-specific one where it helps, all behind the same interface and with the same well-characterized probability outputs.

Decision models are not the only kind of System 1 model. Cloudflare has recently released Clairic, which it describes as a System One extraction model (7). Like a decision model, it answers in a single pass without generating text. Instead of choosing from a list, however, it returns typed values: a span of the input text, a number, a date or a duration, each with a confidence score and a probability that the value is absent. Much of radiology needs exactly this kind of output, such as lesion measurements, dates of prior studies and follow-up intervals. The one-token readout described above does not extend to extraction, because a date or a measurement cannot be listed in advance as a few lettered options. Getting one from a general language model means generating several tokens, which gives up much of the efficiency and the clean probabilities. Cloudflare reports that a pipeline using Clairic for extraction and Clef for the decision took 0.46 and 0.22 seconds for the two steps, compared with 16.1 and 21.9 seconds for a general LLM. I have not tested extraction models, but they are the natural next step for radiology, and together with decision models they could cover most of the structured work that now goes to large language models.

None of these models will interpret images unsupervised, and none should be deployed without local validation. But for the large share of radiology work that is really a sequence of well-defined decisions, the building blocks are already here, and there are many more of them than I thought.

References

1. Typesafe. Introduction. Documentation. docs.typesafe.ai/introduction
2. Yang X, Vrettos K, Ben Chaaben Z, et al. The Value of System 1 Models in Medicine. arXiv:2610.31315, 2026. arxiv.org/abs/2610.31315
3. Erickson BJ, Salehi S. Zero-shot evaluation of open vision-language models as System 1 decision models on radiology tasks. arXiv:2611.01631, 2026. arxiv.org/abs/2611.01631
4. Cloudflare. Clef 27B decision model (Ollama library). ollama.com/library/clef:27b
5. Liquid AI. d1 decision model (Ollama library). ollama.com/library/d1
6. Qwen Team. Qwen3.8-27B model card. huggingface.co/Qwen/Qwen3.8-27B
7. Cloudflare. Introducing Clairic, a new System One class of extraction models. blog.cloudflare.com/clairic-extraction-model