How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Room to Learn Without Forgetting

By Joshua Bolick, Awareness Software Group ~18 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: Which parts of a model actually matter? – The parts that move most are not the parts that matter, and no single component is required -- the model routes around every freeze. What a part is worth shows up only when you remove it.

In plain English

What we asked. When you teach a trained model something new, it tends to forget what it already knew. A paper published this week proposes a neat fix: find the internal directions the old skill actually uses, and let the new learning happen only in the directions it leaves free, so the old skill's calculations cannot be disturbed. We tested that idea on a small model, against the plainest alternative there is: mixing a few old examples back in.

What we found. The idea works. Taught the ordinary way, our model kept a tenth of its first skill; restricted to the unused directions, it kept six times as much, and picking the same number of directions at random protected nothing. But mixing back in the same few dozen old examples the method itself needs kept even more of the old skill and learned the new one better, at no extra cost. We had also predicted in advance that the method would struggle when the new skill uses the same inputs as the old one, and it did.

Why it matters. Two practical lessons. First, when a clever method needs a sample of old data anyway, try simply training on that sample before reaching for anything cleverer. The clever method still has a real advantage where old data cannot be kept or reused, which this test did not price. Second, and new for this project: measuring how much of the model the old skill occupied told us in advance where the method would run out of room. It is the first time here that measuring a model's internal shape predicted something useful, rather than just describing it.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, six seeds, about fifteen minutes. The design and kill test were committed (7a56fcc) before any measured run; calibration used seeds disjoint from these and is disclosed in the program doc.

Program v2 Bucket S, item S1. Decisive computation: analysis/null_basis_adaptation.py. Output: analysis/null_basis_adaptation.json. Reproduce with python analysis/null_basis_adaptation.py; --reuse re-derives every endpoint from the saved curves, and does so identically.

The question

Reasoning-Preserving Fine-Tuning of Post-RL LLMs with Null-Basis LoRA (arXiv 2609.25618, 2026-09-22) observes that a capability's activations occupy a low-dimensional subspace, estimates that subspace from a modest sample, and constrains adapter updates to act only through its complement -- the null basis. A layer's output on the old capability's activations then cannot change. It reportedly matches ordinary LoRA on the new task while keeping reasoning near its pre-fine-tuning level.

**Teaching a model something new without wiping out what it knew.** A small model learned one skill, then was adapted to a second, different one using a handful of small add-on weights. Taught the ordinary way it forgot almost all of the first skill. A recent paper proposes letting the add-ons act only through the directions the first skill does not use. The last bar is a plain alternative: mix in the same few dozen examples of the first skill that the paper's method needs. Restricting the change to the unused directions really works: it keeps six times as much of the first skill, and choosing directions at random keeps nothing, so which directions matters. But simply replaying the same small sample of old examples keeps more still and learns the new skill better, at no extra cost. The clever method is real; here the obvious one beats it.

**Protecting more of what a model knows leaves less room to learn.** The method protects the directions the first skill uses and lets the new skill use the rest. How big the protected region is can be dialled. This measures the result at three settings, and counts how many of the model's 48 internal directions each one leaves free. The first skill already fills most of this small model: at the default setting only about eight of 48 directions are free. Protect more and almost nothing is left, so the new skill stops being learned while the old one is protected no better. Measuring how much of the model a skill occupies told us in advance where the method would run out of room.

The maintainer's report put the idea more sharply: can a model's representation be partitioned into protected and adaptable capacity? And it connects to this programme directly. Program v1 found that forcing representations into a subspace does nothing for learning. This asks whether the subspace a model already uses is worth protecting.

Kill test, fixed before execution: on the new-domain task, null-basis LoRA's retained task-A accuracy does not exceed both ordinary LoRA's and random-subspace LoRA's, paired over six seeds with intervals excluding zero.

Prediction, fixed before execution: on the same-domain task, null-basis LoRA adapts worst of the LoRA arms, because a task whose inputs live in task A's subspace leaves the constraint nothing to work with.

Prior recorded in advance: good on the kill test (one calibration seed had shown it); moderate to low against replay from the same sample, which was named as the headline comparison.

Design

A width-48 GRU learns task A -- delayed copy, lag 4, tokens 0-31 of a 48-token vocabulary -- for 800 steps. Every base model reaches at least 0.973 on it (mean 0.979). It is then adapted for 600 steps, with rank-8 adapters on all three of its linear maps (input-to-hidden, hidden-to-hidden, readout), to one of two new tasks:

New task	What it is	Why
new-domain	delayed copy, lag `3`, on tokens `32-47`	inputs the base model has never seen, so they can live outside task A's subspace
same-domain	delayed copy, lag `7`, on task A's own tokens	inputs inside task A's subspace: the constraint should leave little room

The null basis. For each matrix, the top eigenvectors of the uncentred second moment of that matrix's inputs on 64 task-A sequences, cut at 99% of energy; the adapter B A becomes B A P, with P the projector onto the complement. Uncentred, because what must be preserved is W x.

Arm	What it is
full	every parameter trained
LoRA	ordinary rank-8 adapters
null-basis LoRA	adapters see only the complement of task A's inputs
random-subspace LoRA	a random subspace of the same dimension removed -- the report's control
principal-basis LoRA	adapters see only task A's subspace -- the opposite constraint
replay, same sample	ordinary LoRA, a quarter of each batch drawn from the same `64` sequences the basis came from
replay, unlimited	ordinary LoRA, a quarter of each batch from a fresh task-A stream -- a ceiling, not a competitor

Why replay from the same sample is the price. NB-LoRA's selling point is that it needs only a modest sample of the old capability. Replay can use exactly that sample. That makes it the cheapest ordinary alternative with the same data in hand, which this programme's rules require any efficiency claim to beat.

Preconditions, in code, both met: every base reaches 0.90 on task A; ordinary LoRA forgets -- it loses 0.879 of task A's accuracy, far past the 0.20 floor, so there is something to protect.

Result: the kill test does not fire

Null-basis adaptation protects the old task, and the protection is specific to the null basis.

New-domain task, final accuracy (mean of the last three evaluations), six seeds:

Arm	New task	Task A kept
full	`1.000`	`0.203` `[0.143, 0.263]`
LoRA	`1.000`	`0.100` `[0.052, 0.148]`
random-subspace LoRA	`0.984`	`0.033` `[0.009, 0.058]`
principal-basis LoRA	`0.999`	`0.045` `[0.033, 0.058]`
null-basis LoRA	`0.955` `[0.925, 0.985]`	`0.600` `[0.479, 0.720]`
replay, same sample	`0.999`	`0.768` `[0.729, 0.808]`
replay, unlimited (ceiling)	`0.999`	`0.965`

Paired over seeds:

Contrast	Task A kept	New task
null-basis minus LoRA	`+0.500` `[+0.395, +0.604]`	`-0.045` `[-0.075, -0.014]`
null-basis minus random subspace	`+0.566` `[+0.459, +0.674]`
null-basis minus replay, same sample	`-0.169` `[-0.297, -0.041]`	`-0.044` `[-0.075, -0.014]`

Removing a random subspace of the same size protects nothing (0.033, worse than ordinary LoRA), so the effect is about which directions are removed, not how many. That is the report's own control, and it is decisive.

The paper's claim reproduces in miniature. Adaptation matched to within about five points on the new task, retention up from 0.10 to 0.60.

Result: the prediction holds

Same-domain task, new-task accuracy by LoRA arm:

LoRA	principal	random	null-basis
`0.700`	`0.696`	`0.566`	`0.394`

Null-basis adapts worst, by a wide margin, exactly as predicted: a task whose inputs already live inside the protected subspace gives the constraint nothing to work with. The corollary is the useful part: **NB-LoRA is a tool for adding a different skill, not for modifying the protected one.** A fine-tune that needs to change how the model handles the same inputs is the case it cannot serve.

The headline: replay from the same 64 sequences beats it on both axes

Replaying the same sample the null basis was estimated from keeps more of the old task (+0.169) and learns the new task better (+0.044), both intervals excluding zero. It costs nothing extra in compute (a quarter of each batch is old data rather than new, at the same batch size) and needs no eigendecomposition.

So in this setting the answer to "does NB-LoRA work?" is yes, and the answer to "is it the cheapest way to get what it gives?" is no. The paper positions NB-LoRA against experience replay with lower overhead; at this scale the overhead of replay is zero, and it wins.

Where the free capacity actually is

The null basis is only as useful as the dimensions it leaves free. How many task A occupies, out of 48, by threshold (mean over seeds):

Energy kept	input-to-hidden	hidden-to-hidden	readout
`90%`	`19.3`	`25.2`	`23.0`
`99%`	`28.8`	`40.3`	`39.0`
`99.9%`	`32.0`	`46.3`	`45.3`

The recurrent state space is nearly full. At 99%, task A uses 40 of the 48 hidden-to-hidden directions, leaving eight. At 99.9% it uses 46. (The input side stops at exactly 32 because there are 32 task-A tokens and their embeddings span 32 dimensions; the 16 directions left over are exactly where the new tokens' embeddings can differ.)

And the threshold is a dial between protecting and learning, not a free lunch:

Threshold	New task	Task A kept
`90%`	`0.996`	`0.463`
`99%`	`0.955`	`0.600`
`99.9%`	`0.705`	`0.588`

Tightening to 99.9% costs a quarter of the new task and protects no better, because the constraint has run out of room: with two free recurrent directions, the adapter cannot learn and the remaining 0.1% of energy still leaks through the recurrence. The measured dimensionality of the old skill predicted where the method would stop working -- which is the first direct evidence in this archive for the report's proposed reading of geometry as a diagnostic of available capacity, as opposed to a target to optimise (program v1's closed question).

What stands

The kill test does not fire. Null-basis adapters keep 0.600 of the old task against 0.100 for LoRA and 0.033 for a random subspace of the same dimension.
The prediction holds. When the new task shares the old task's inputs, null-basis adaptation learns worst (0.394 against 0.700).
Replay from the same 64 sequences beats it on both axes, at no extra compute. In this setting NB-LoRA is a real effect that a simpler method dominates.
The old skill's measured dimensionality told us where the method would saturate. A measurement of geometry was useful here as a diagnostic -- the first such case in this archive.

Limits

Tiny recurrent model, synthetic tasks, one width. The paper's setting is post-RL LLMs with reasoning spread across many layers; here task A fills most of a 48-dimensional state. In a model with far more spare dimensions per layer the null basis has more room, and the comparison with replay could change. That is the scale question S7 would need a GPU to answer.
The replay comparison assumes the sample can be kept. In the paper's setting, retaining and training on reasoning data may be restricted or costly in ways that are not compute -- privacy, licensing, distribution shift from the original RL data. There NB-LoRA's advantage is that it needs the sample once, to estimate a basis, and never trains on it. That is a real advantage this experiment does not price.
One adapter rank and one learning rate. Null-basis LoRA might close its five-point new-task gap with a longer budget or a larger rank; replay was not tuned either.
Same-domain retention is similar across arms (0.52-0.74), because a lag-7 copy partly overlaps a lag-4 copy on the same tokens. The prediction was about adaptation, where the separation is wide; it is not a claim about retention there.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
adapter: A small set of extra weights attached to a trained model and trained on a new task while the original weights stay fixed.
batch size: How many examples the model looks at before updating itself once. Bigger batches give a steadier but more expensive update.
calibration: Working out an instrument's settings from runs whose answer you already know, so it can be used on a run whose answer you do not.
fine-tuning: Continuing to train an already-trained model on a new, usually narrower, task.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
LoRA: Low-rank adaptation. A cheap way to teach an already-trained model something new: its original weights are frozen and a small set of add-on weights is trained instead.
null basis: The directions inside a layer that a particular skill does not use. A change made only along them cannot disturb that skill's calculation at that layer.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
replay: Mixing a few examples of an old task back into training on a new one, so the model is reminded of what it already knew. The standard defence against forgetting.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
subspace: A slice of a model's internal space: some directions kept, the rest ignored.
vocabulary: The set of distinct symbols a model can read and produce.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.