Sixty-Seven Cents of Reasoning

A small transformer, trained from scratch at inference time for ninety minutes on a single consumer graphics card, scored 44% on ARC-AGI-1. The result is interesting as an possible evidence for the value of sample, a question that is critical to the future of AI in medicine.

The work in question is the third post in a series by Mithil Vakde describing his approach to the Abstraction and Reasoning Corpus.¹ ARC was proposed by Francois Chollet as a measure of skill-acquisition efficiency rather than of accumulated skill: each puzzle supplies three to five input-output demonstrations of a rule the solver has never seen, and asks for the rule to be applied to a held-out case.² The benchmark is deliberately hostile to the strategy that has driven most recent progress, which is to pre-train on a corpus large enough that few tasks are genuinely novel.

Two qualifications belong at the front. ARC puzzles are small colored grids with discrete, noise-free ground truth, which is not at all similar to medical imaging. And the conclusions below are drawn from a single well-documented set of experiments by an individual investigator, not from a peer-reviewed multi-site study. What follows should be read as a hypothesis about mechanism, not as evidence of clinical benefit. That said, I believe that in medicine, the value of a small number of carefully curated training examples is underestimated relative to the return on a large number of examples of variable quality, and this work bears directly on why that might be true.

What Was Actually Done

The method trains a small transformer, scaled to just eight layers, with random initialization at test time. There is no offline pre-training and no synthetic pre-training corpus. Puzzles from the training and evaluation sets are converted to token sequences and trained on autoregressively, with the evaluation test labels strictly withheld. Training runs approximately 1.5 hours on an RTX 5090.

Three design choices carry most of the weight. Spatial structure is supplied through 3D rotary position embeddings, extending the rotary scheme of Su and colleagues to represent the two-dimensional grid rather than a one-dimensional sequence.³ Each puzzle receives its own additive learned embedding, which separates task identity from task content and permits the shared weights to carry structure common across puzzles. Loss is computed on output tokens only. The remaining components are conventional modern practice: SwiGLU activations, RMSNorm, flash attention with variable-length batching, and color and dihedral augmentation with majority voting over inverse-augmented predictions at inference.

The ablations are the part worth noting. Removing 3D RoPE drops the score from 44% to approximately 25%. Removing the per-task embeddings drops it to approximately the same place. Substituting one-dimensional positional encoding yields approximately 24%. Training each puzzle separately in the unsupervised, per-task style of CompressARC yields approximately 18%. Nearly half the performance of this system is attributable not to capacity, data, or optimization, but to how the problem is represented to the model.

For calibration: the reported 44% on ARC-AGI-1 is comparable to the 45% reported for the 7M-parameter Tiny Recursive Model⁴ and sits in the same neighborhood as the Hierarchical Reasoning Model,⁵ both of which use similar batched test-time dynamics. It is below the 53% reported by Akyürek and colleagues for test-time training applied to an 8B-parameter pre-trained language model,⁶ which is the relevant comparison for anyone tempted to read this as evidence that pre-training is unnecessary. It is not. It is evidence that pre-training is not the only route to sample efficiency.

Where the Author Is Appropriately Cautious

Several limitations are acknowledged in the original and should be carried forward rather than discarded in translation.

The augmentation strategy is described by the author as an "anti-bitter-lesson cheat" that he has not found a way to remove without a substantial cost in accuracy or compute. Color and dihedral permutations are hand-specified invariances--essentially, hand-specified domain knowledge. This is not a criticism so much as a clarification of what is being claimed.

The switch to computing loss on output tokens alone worsened test loss while improving the final score, from 40% to 44%. The author admits that he does not understand why. An unexplained inversion between the optimized quantity and the reported metric is the kind of finding that should lower confidence in any single number derived from the system.

The system did not generalize well, scoring 7% on ARC-AGI-2. Whatever is being learned transfers poorly to a harder distribution of the same general kind, which is precisely the failure mode that matters clinically.

Finally, all evaluation tasks are trained on simultaneously, which amortizes compute across tasks in a way that does not correspond to solving one problem at a time. The author accepts that this criticism has merit, and now reports total lifetime compute rather than cost per task, restricting direct comparison to systems with comparable batch dynamics.

Why This Might Matter in Medicine

The data availability and annotation bottleneck is a structural constraint, not a temporary one

In medical imaging, the limiting factor is expert time. Ground truth for rare pathologies limits samples, and expert annotation and data sharing challenges further limits dataset size. Willemink and colleagues describe the curation and labeling pipeline in enough detail to make clear that this cost is not incidental to the process but constitutes most of it.⁷ Active learning and human-in-the-loop approaches exist precisely because the bottleneck is structural.⁸ A demonstration that a novel rule can be acquired from a handful of examples, provided the architecture already encodes the relevant structure, is therefore directly relevant, even if the demonstration domain is toy.

The complementary point concerns label quality. Karimi and colleagues show that deep networks trained on medical images are substantially degraded by label noise and that mitigation is an active problem rather than a solved one.⁹ Where truth is a matter of expert judgment, as it frequently is in imaging, more labels are not automatically more information. Five carefully adjudicated cases and five hundred rapidly annotated ones are not interchangeable, and an architecture that requires only the former is worth more than a comparable margin of accuracy.

Test-time adaptation maps onto the domain-shift problem

Distribution shift across institutions is another important challenge. AlBadawy and colleagues showed that brain tumor segmentation performance degrades measurably when training and testing institutions differ.¹⁰ Zech and colleagues showed that a pneumonia classifier's performance varied by site and that models exploit site-specific confounders available in the image.¹¹ The conventional responses are broader training distributions, harmonization, or federated training across institutions.^12,13

Test-time training suggests a possible different arrangement: adapt to a hospital's data, using the unlabeled images actually present at that site, plus a small number of locally annotated examples. This is not a new idea in medical imaging. Karani and colleagues demonstrated test-time adaptable segmentation networks intended for exactly this problem, in which a normalization module is adapted per-scan while the segmentation network is held fixed.¹⁴ The contribution of the ARC work is not the mechanism but the accounting: it makes visible how much of the achievable adaptation is attributable to representation rather than to data volume, and it does so at a compute cost that a department could absorb without discussion.

Representation engineering translates more readily than the architecture does

The single most transferable finding is the ablation result. Encoding the geometry of the problem in the positional scheme was worth nearly twenty percentage points. Medical images have far more exploitable structure than ARC grids do: three-dimensional volumes with known and often anisotropic voxel spacing, physical units, standard orientations, established anatomical priors, and known acquisition physics. A model that has been given this structure does not need to spend training examples discovering it.

This is also the least surprising claim in the discussion, and it should be recognized as such. nnU-Net has been the most reliable segmentation baseline in the field for several years largely because it configures preprocessing, spacing, and patch geometry correctly for the dataset in front of it, without architectural novelty.¹⁵ The ARC result is consistent with that experience and quantifies it in a setting where the ablation is clean.

Local compute is a governance argument as much as a cost argument

A method that trains in 1.5 hours on a single consumer GPU can run inside an institutional firewall. Transferring identifiable images to an external inference API raises privacy, contractual, and regulatory questions that many institutions resolve by declining. Federated approaches were developed to address the same constraint from the training side.¹² A model small enough to be trained or adapted locally sidesteps a portion of the problem rather than solving it, but sidestepping is frequently sufficient.

Test-Time Training Compared with Error-Driven Selection

Test-time training and uncertainty-driven methods such as active learning, hard-negative mining, and focal loss share an objective, which is to extract more from fewer examples. Their mechanisms differ enough that they should not be treated as alternatives to one another.

Active learning is selective and inductive. It filters for the examples nearest the decision boundary, requests labels for them, and thereby concentrates optimization where the model is least certain. It assumes the target distribution is fixed and refines the boundary within it. It is correspondingly sensitive to label noise, since an irreducibly ambiguous case is indistinguishable from an informative one by the uncertainty criterion alone.

Test-time training is comprehensive and transductive. It uses all available target inputs to construct a shared representation, discards nothing as too easy, and derives its efficiency from priors supplied before any data is seen. Its exposure is different: it depends on the target inputs being available in advance and on the structural priors being correct, and it offers no protection when they are not.

Stated as questions, active learning asks which examples will teach the model the most, and test-time training asks how the model can reconfigure itself around the specific case in front of it. In a clinical setting where the relevant regularities vary by scanner, protocol, and population, both questions have answers worth having, and they are not the same answer. The combination — locally adapted, with expert annotation directed to the cases where the adapted model is least confident — is the arrangement we would expect to be most efficient with expert time, and we are not aware of it having been evaluated as such.

What Would Make This Actionable

The claim being made here is narrow. A 44% score on grid puzzles is not evidence about segmentation, and the 7% on ARC-AGI-2 is a reminder of how quickly this class of result degrades under a harder distribution. What would move the argument forward is a study in a real modality that reports the ablation as carefully as this one does: hold the architecture and data fixed, vary the spatial and acquisition priors supplied to the model, and report how many annotated cases are required to reach a fixed performance target under each condition. Sample efficiency reported as a function of the priors supplied, rather than accuracy reported at a fixed dataset size, is the number that would inform how an institution should spend its annotation budget.

Meanwhile, the practical inference is unglamorous and probably correct. Before requesting another thousand annotations, it is worth establishing how much of the current performance gap is attributable to the model not having been told what the images are: their geometry, their spacing, their orientation, their physics. Based on the ablations reported here and on the sustained performance of methods that get this configuration right,¹⁵ that fraction is frequently larger than expected.

References

1. Vakde M. 44% on ARC-AGI-1 with a small transformer trained from scratch at test time. 2026. mvakde.github.io/blog/44-on-arc-1
2. Chollet F. On the Measure of Intelligence. arXiv. 2019;1911.01547. arXiv:1911.01547
3. Su J, Ahmed M, Lu Y, Pan S, Bo W, Liu Y. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing. 2024;568:127063. doi:10.1016/j.neucom.2023.127063
4. Jolicoeur-Martineau A. Less is More: Recursive Reasoning with Tiny Networks. arXiv. 2025;2510.04871. arXiv:2510.04871
5. Wang G, Li J, Sun Y, et al. Hierarchical Reasoning Model. arXiv. 2025;2506.21734. arXiv:2506.21734
6. Akyürek E, Damani M, Qiu L, Guo H, Kim Y, Andreas J. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning. arXiv. 2024;2411.07279. arXiv:2411.07279
7. Willemink MJ, Koszek WA, Hardell C, et al. Preparing Medical Imaging Data for Machine Learning. Radiology. 2020;295(1):4-15. doi:10.1148/radiol.2020192224
8. Budd S, Robinson EC, Kainz B. A survey on active learning and human-in-the-loop deep learning for medical image analysis. Med Image Anal. 2021;71:102062. doi:10.1016/j.media.2021.102062
9. Karimi D, Dou H, Warfield SK, Gholipour A. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Med Image Anal. 2020;65:101759. doi:10.1016/j.media.2020.101759
10. AlBadawy EA, Saha A, Mazurowski MA. Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Med Phys. 2018;45(3):1150-1158. doi:10.1002/mp.12752
11. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLoS Med. 2018;15(11):e1002683. doi:10.1371/journal.pmed.1002683
12. Rieke N, Hancox J, Li W, et al. The future of digital health with federated learning. NPJ Digit Med. 2020;3:119. doi:10.1038/s41746-020-00323-1
13. Sheller MJ, Edwards B, Reina GA, et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci Rep. 2020;10(1):12598. doi:10.1038/s41598-020-69250-1
14. Karani N, Erdil E, Chaitanya K, Konukoglu E. Test-time adaptable neural networks for robust medical image segmentation. Med Image Anal. 2021;68:101907. doi:10.1016/j.media.2020.101907
15. Isensee F, Jaeger PF, Kohl SAA, Petersen J, Maier-Hein KH. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods. 2021;18(2):203-211. doi:10.1038/s41592-020-01008-z