How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Nobody Tuned the Rate

By Joshua Bolick, Awareness Software Group ~12 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: How do we know our own results are real? – Repeatedly, the control rather than the measurement decided the result -- and several striking findings dissolved when the right comparison was finally run.

In plain English

What we asked. We had just found that one group of our results was really explained by a setting, the learning rate, that nobody had tuned. So we checked every other result where we claimed something made a model learn sooner, by reading the code that produced it.

What we found. None of them had been measured at a tuned learning rate. But for most of them the setting we used happened to be close to the best one, so the rate could explain only a small part of their effect. The group that fell apart had used a setting far from the best, which is why it fell apart. And the data showing how far had been sitting in our own archive for a month, collected for a different purpose.

Why it matters. Before trusting a speed-up, check that the thing you compare it to is set up as well as it can be, and look at the data you already have first.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. No training: the audit imports committed pilots to read the rate and task each claim's training loop actually used, and reads committed sweeps. The audit script was committed (effcca8) before its recorded run.

Program v2 Bucket R, item R18. Decisive computation: analysis/untuned_rate_audit.py. Output: analysis/untuned_rate_audit.json. Reproduce with python analysis/untuned_rate_audit.py (a few seconds).

Two changes after the committed script's first run, both reported, neither deciding anything: a column giving the headroom to O16's tuned rate (the fastest that keeps final accuracy), added once O16 reported, because the preregistered comparison is against the fastest rate at which runs merely still learn; and each claim's published size in steps, cited from its record, for the figure.

The question

O15 showed a four-record thread (J8, O3, O13, O14) had been compensating for a learning rate nobody had tuned. Which other published speed-up claims were measured at a rate slower than their substrate tolerates?

**How much of each speed-up could a better learning rate have bought instead?.** Each pair is one published claim that something made a small model learn sooner. The first bar is the saving the claim reported; the second is how many steps simply tuning the learning rate would have saved on the same setup. Two claims (J8 and O14) were measured at a rate so slow that tuning it would have saved several times their effect, and they turned out to be explained by it. The rest used a rate close to tuned, so most of their savings are far larger than the rate could explain. O10 is the exception.

Thirteen standing or recently standing claims that an intervention shortens the time to the transition or to a criterion are listed in the script with the pilot whose training loop produced them. The script imports that pilot and reads the rate and task it trained with -- following imports, since most pilots take their loop from another -- and compares against every committed rate sweep of the same task and width: critical_slowing.py's phase diagram (committed 2026-08), O15 and O16. All three cover one substrate: delayed copy, lag 4, width 48, batch 64.

Kill test, fixed before execution: no standing claim outside J8's own thread is EXPOSED (a committed sweep of its substrate shows a faster rate at which every run still learns) or UNTESTED (no committed sweep covers its substrate).

Result: the kill test does not fire

Claim	Record	Rate	Task	Verdict	Headroom to fastest learning rate	Headroom to tuned rate
P1 head start (`35.8` steps)	machinery-transplant	`0.005`	delayed copy	EXPOSED	`29.9`	`8.5`
P6 bias share (`12.5`)	carried-subspace	`0.005`	delayed copy	EXPOSED	`29.9`	`8.5`
P7 six numbers	synthetic-bias	`0.005`	delayed copy	EXPOSED	`29.9`	`8.5`
P8 break-even (`25.0`)	donor-economics	`0.005`	delayed copy	EXPOSED	`29.9`	`8.5`
P11 matched ladder	matched-difficulty	`0.005`	ladder, widths 24-96	UNTESTED
Q4 second task	transfer-across-tasks	`0.005`	dispatch copy	UNTESTED
R2 donor screen (`16.9`)	donor-screening-economics	`0.005`	Q14's pool	UNTESTED
R12 donor-length optimum	unscaled-ladder	`0.005`	dispatch copy	UNTESTED
O10 early freeze (`5.8`)	early-gate-freeze	`0.005`	delayed copy	EXPOSED	`29.9`	`8.5`
K2 freeze saving (withdrawn by Q5)	freeze-after-transition	`0.005`	delayed copy	EXPOSED	`29.9`	`8.5`
J7 batch cut (reversed by Q1, Q11)	spend-the-lead	`0.005`	delayed copy	EXPOSED	`29.9`	`8.5`
J8 own direction (`9.27`)	direction-transplant	`0.002`	delayed copy	EXPOSED	`97.9`	`76.6`
O14 late boost (`11.6`)	momentum-push-schedule	`0.002`	delayed copy	EXPOSED	`97.9`	`76.6`

Headroom is the control's time to the transition at the claim's rate (interpolated in log-rate along the sweep) minus the time at the comparison rate: the steps the rate alone would have saved.

Outside J8's thread: 7 EXPOSED, 4 UNTESTED, 0 TUNED of 11. No claim in this archive was measured at a rate anyone had tuned for time to the transition.

What the headroom column says

The verdict is binary; the headroom is not, and it splits the archive in two:

J8's thread ran at 0.002, 76.6 steps from its substrate's tuned rate -- eight times the size of the effects it reported. That is how four records could be explained by the rate.
Everything else ran at the house rate 0.005, 8.5 steps from the tuned rate 0.006. The house rate was never tuned, but it happens to sit close. Against 8.5 steps, P1's 35.8-step head start is four times larger than anything the rate could explain, and R19 has since re-run it at the tuned rate: it survives (-30.8). O10's 5.8 steps is smaller than the headroom and is the claim most exposed; it is also already confined to one task.
Four claims sit on substrates no sweep covers (dispatch copy, Q14's pool, the matched ladder). Their exposure is unknown, not zero.

Why this took so long to find: critical_slowing.py swept the rate on this exact substrate a month ago and showed the transition falling from 160 to 74 steps between 0.002 and 0.008. Its purpose was to give early-warning detectors room to calibrate, which is why 0.002 was chosen -- a sensible reason for an instrument study that nobody re-examined when the substrate was later used for speed-up claims. The data answering O15 was in the archive the whole time.

What stands

Kill test does not fire: every speed-up claim outside J8's thread is EXPOSED (7) or UNTESTED (4).
The house rate 0.005 is 8.5 steps from its substrate's tuned rate; J8's 0.002 was 76.6.
Generates R19 (P1 at the tuned rate; since run: the head start survives) and names O10 as the exposed claim smallest against its headroom.

Limits

One swept substrate. Claims on other tasks are UNTESTED, and the sweep's own seeds differ from each claim's.
Headroom is a scale, not a correction: a speed-up and a rate change need not add.
The claim list is hand-built from a scan of records for speed-up language (sooner, head start, saves, faster); a claim phrased otherwise could be missing.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
momentum: Keeping part of each previous training step in the next one, so training keeps moving in a consistent direction instead of zig-zagging.
phase diagram: A map showing what a system does across combinations of settings, so you can see where behaviour changes rather than testing one point at a time.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
subspace: A slice of a model's internal space: some directions kept, the rest ignored.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.