How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

How Fast It Learned

By Joshua Bolick, Awareness Software Group ~11 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. When you start a new model from a piece of an existing one, you have to choose which existing one. A free check tells helpful candidates from harmful ones. In our last experiment we noticed, after the fact, that among helpful candidates from other kinds of task, how quickly each had learned its own task seemed to rank them better than the check did. Something noticed after the fact is a lead, not a result.

What we found. So we tested it properly: new training runs, with the rule for which candidates count written down before any data existed. It held. The advantage was smaller than first seen, which is normal, but clearly real.

Why it matters. That gives a simple, free recipe: use the quick check to throw out the candidates that would hurt, then rank the rest by how fast they learned their own task, which you already know because you trained them. Our best guess is that learning speed is standing in for how closely related the tasks are, but the recipe works either way.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study in the formal sense of this archive, but its design, band and kill test were fixed and committed (9787823) before any data from these seeds existed, to retest a finding R13 made post hoc. Local CPU, six fresh seeds, R6's full grid.

Program v2 Bucket R, item R17. Decisive computation: analysis/donor_learning_time.py. Output: analysis/donor_learning_time.json. Reproduce with python analysis/donor_learning_time.py (about an hour on a throttled laptop CPU); --reuse re-derives every endpoint from the saved runs.

The question

R13 asked what should rank the donors Q14's free screen keeps. With its band fixed in advance -- every helpful donor except the receiver's own task -- nothing beat the screen. Post hoc, with R6's own honest convention of removing the receiver's whole task family, the donor's own transition time ranked far better: +0.862 against +0.429, paired +0.434 [+0.067, +0.801]. A band chosen after seeing the data cannot decide a kill test, so R13 generated this.

**How fast a donor model learned ranks its usefulness better than a quick check.** Ranking candidate models to start a new one from, among the helpful candidates from other task families. 'Check' is a free one-pass test on the new task; 'speed' is how fast the candidate learned its own task. First noticed after the fact, then tested on new runs with the rule written down in advance. The speed signal held up when tested properly: smaller than first seen, as expected, but still clearly better than the quick check. It costs nothing, because every candidate's training record already exists. Use the check to throw out the harmful candidates, then rank the rest by how fast they learned.

Design, fixed before execution: R6's pool, receivers and budgets on six fresh seeds disjoint from R6's; worth read at R6's committed levels, so the yardstick cannot move with these data; the helpful band fixed as donors outside the receiver's family with negative mean worth on the fresh seeds; agreement per seed and receiver (R15's unit).

Kill test, fixed before execution: transition time's agreement does not exceed zero-shot accuracy's, paired over fresh units, with an interval excluding zero.

Prior recorded in advance: moderate -- post hoc effects usually shrink, and R8 suggests learning time may be a proxy for relatedness.

Anchor, in code, passed exactly: R6's first seed's control on delayed-copy-4 re-runs to 90.0, its committed value.

Result: the kill test does not fire

The fixed rule, applied to the fresh data, chose the same helpful band R13 used post hoc: delayed-copy-4 gets noisy-copy-15, noisy-copy-35, dispatch-8lag and dispatch-4lag; dispatch-8lag gets delayed-copy-4, delayed-copy-7 and both noisy copies.

Score	Agreement with worth, 12 fresh units
zero-shot accuracy (the screen)	`+0.290` `[-0.217, +0.798]`
the donor's own transition time	`+0.621` `[+0.348, +0.893]`
transition minus screen, paired	`+0.330` `[+0.085, +0.576]`

The lead replicates. It shrank, as the prior expected -- +0.434 post hoc on R6's seeds, +0.330 fixed in advance on fresh ones -- and its interval still excludes zero. Among the helpful donors from other task families, how fast a donor learned its own task ranks its usefulness better than how well it already does yours.

A practical rule, and its cost

The screen and learning time do different jobs, and together they make a two-stage rule:

Reject with the screen. One forward pass on the receiver's task separates helpful donors from harmful ones perfectly (Q14, R6).
Rank the survivors by how fast each learned its own task. This costs nothing: every donor's training curve already exists, because the donor had to be trained.

Within a single family the screen still wins (R13's pre-specified band): a same-family donor is both the most useful and the most obviously related, and zero-shot accuracy sees that directly. The learning-time rule is for choosing among donors from other families, which is the case R6 called honest because near-identical donors make any rule look good.

What it might be measuring

R8 found that a donor task's difficulty does not order transfer once relatedness is held constant. A donor's transition time is close to its task's difficulty, so the most economical reading is that learning time works here as a proxy for relatedness, not that easy tasks make better donors. The rule does not care which: it is free, and it ranks. Whether it survives on a pool where difficulty and relatedness are deliberately untied is the natural next test.

What stands

The kill test does not fire. Preregistered on six fresh seeds, the donor's own learning time ranks helpful out-of-family donors better than the screen, +0.330 [+0.085, +0.576].
R13's post hoc lead replicated, smaller, as expected.
A free two-stage rule: screen to reject, learning time to rank, both already paid for.
Probably a relatedness proxy (R8), which does not stop it being useful.

Limits

Two receivers, four helpful donors each, so each unit's rank correlation rests on four points; the interval is over twelve units.
One pool of nine configurations, which R6 built; a pool with difficulty and relatedness untied is where the proxy question would be settled.
Worth at R6's level, inherited from R6 by design.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
post hoc: Worked out after the fact, rather than decided in advance. We report such checks separately and never let them decide a result, because it is far too easy to find a pattern once you already know the answer.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
settled: A run has settled when it has stopped improving. Measurements anchored to a run's own best score are unreliable until it has, because that best score is still moving.