How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

A Window That Parks

By Joshua Bolick, Awareness Software Group ~12 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. Our last experiment found that an automatic teacher, choosing what a small model practises, beat every fixed plan we had written, and that what it found was really a plan: practise what is just becoming learnable, and retire what is learned. So we tried writing that plan down by hand, as a simple rule with two settings, to see whether the teacher was needed at all.

What we found. The hand-written plan beat the old fixed plans and got most of the way to the teacher, but not all of it; with six runs we cannot say for sure whether a gap remains, though it leans that way. The breakdown shows why. Our rule moved on to the hardest version of the task halfway through training and stayed there, so it mastered the hardest version and let the middle ones slip. The teacher kept practising whatever was not finished yet.

Why it matters. The practical lesson: most of the benefit comes from something free and simple, which is to stop drilling what has been learned. The rest comes from not abandoning what is only half-learned, and that is the part a simple rule missed.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, six seeds plus S2's screening seed. The design, kill test and anchor were committed (bce0717) before any run.

Program v2 Bucket S, item S8. Decisive computation: analysis/sliding_window_curriculum.py. Output: analysis/sliding_window_curriculum.json. Reproduce with python analysis/sliding_window_curriculum.py (about an hour on a throttled laptop CPU); --reuse re-derives every endpoint from the saved curves.

The question

S2 found the first adaptive mechanism in this programme to beat the best fixed schedule it was tested against: a free bandit teacher reached family accuracy 0.830 against 0.791 for the best of four easy-to-hard ramps. Its replayed control then matched it -- another seed's teacher schedule played back blind reached 0.821 -- so the teacher's value was a schedule, and the schedule looked like a sliding window: practise the band that is currently learnable, retire what is solved.

**A hand-written practice plan, against the teacher that inspired it.** Three ways of choosing what a small model practises across seven versions of a task, all with the same amount of training: a fixed easy-to-hard plan, a two-setting sliding window written by hand, and the adaptive teacher that inspired the window. The window moves on to the hardest version halfway through and stays there, so it masters the hardest version and lets the middle ones slip. The teacher keeps practising whatever is not finished yet. Retiring what is learned gets you most of the gain; not abandoning what is half-learned gets the rest.

The question S2 left: is the teacher needed even once? A window is two constants. If one written down by hand matches the teacher, the teacher is a way of discovering something you could have written yourself.

Kill test, fixed before execution: the best hand-written window's family accuracy is below the teacher's, paired over S2's six seeds, with an interval excluding zero.

Prior recorded in advance: good that the window matches.

Anchor, in code, passed: S2's teacher re-run on its first seed reproduces the committed family accuracy exactly (0.8420).

Design

The window's centre slides from the easiest lag to the hardest, reaching it at fraction reach of the run; each lag's weight is a Gaussian bump around the centre, exp(-((i - centre) / width)^2), over a 3% floor. Six combinations were screened on S2's screening seed, exactly as S2 screened its ramps:

`reach` / `width`	`0.75`	`1.5`
`0.5`	`0.825`	`0.828`
`0.7`	`0.823`	`0.810`
`0.9`	`0.821`	`0.790`

Every window but one beat S2's best ramp on the same seed (0.803). The best, 0.5 / 1.5, then ran on the six measurement seeds.

Result: the kill test does not fire, and the window does not match either

Arm	Family accuracy	Paired, window minus
S2's best ramp	`0.791`	`+0.016` `[-0.002, +0.034]`
hand-written window	`0.807` `[0.792, 0.821]`
S2's teacher, replayed	`0.821`	`-0.014` `[-0.031, +0.004]`
S2's teacher	`0.830`	`-0.024` `[-0.055, +0.008]`

The window sits between the ramps and the teacher, and six seeds cannot separate it from either. The kill test as written does not fire -- the interval against the teacher includes zero -- but that is not the same as matching: the point estimate is two and a half points short and the interval reaches five. The honest reading is undecided, leaning to the teacher. The window is fastest to a family accuracy of 0.60 (300 steps against the teacher's 333).

Why: the window parks on the hardest task

Final accuracy by lag:

Arm	2	3	4	6	8	10	12
best ramp	`1.000`	`0.994`	`0.972`	`0.816`	`0.644`	`0.582`	`0.532`
window	`0.968`	`0.901`	`0.830`	`0.686`	`0.639`	`0.758`	`0.866`
teacher	`0.981`	`0.958`	`0.935`	`0.848`	`0.752`	`0.682`	`0.656`

The window is U-shaped. Screening chose a centre that reaches the hardest lag at half the run, and from then on the window sits on lags 10 and 12 -- which it learns far better than the teacher does (0.866 against 0.656 at lag 12) -- while lags 4, 6 and 8 decay. The teacher never parks. Late in training it spreads its practice across lags 6 to 12 in proportion to how far each is from 70%, so a lag that slips back into the band is, by the teacher's own rule, practised again (the rule guarantees that; this record did not trace individual revisits). A single sliding window has no way to do that: once the centre has passed a task, the task is on the floor.

What the teacher knows that the window does not is which finished-looking tasks are slipping. That is responsiveness -- the thing S2's replay said was not carrying the gain. The two results are consistent: S2's replayed schedule was another seed's teacher schedule, which contained that seed's revisits, and revisits generalise across seeds because every seed forgets the middle lags in roughly the same way. The teacher's schedule is transferable; it is not a two-constant window.

What stands

The kill test does not fire, and the result is undecided: -0.024 [-0.055, +0.008] against the teacher. A two-constant window gets most of the way from the ramps to the teacher.
Almost any window beats the equal-weight ramps on the screening seed (five of six), which confirms S2's diagnosis of what the ramps got wrong: keeping solved tasks at full weight.
The window parks. Moving the centre to the hardest task halfway through buys the hardest task and loses the middle ones. The teacher's advantage is in re-practising what slips.
The practical reading: retiring solved tasks is most of the gain and is free to write down; the rest needs either a recorded teacher schedule or a rule that revisits what is decaying.

Limits

Six seeds, and a single screening seed. The screening seed chose reach 0.5 by 0.003 over the next candidate; a different screen could have picked a window that parks later. The kill test is about the window the screen chose, as fixed.
One window family. A window whose centre stops at the learnable frontier rather than at the hardest lag, or one with a revisit rule, was not tried. That is the natural next test, and it is a small change to this script.
One task family, one width, as in S2.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
replay: Mixing a few examples of an old task back into training on a new one, so the model is reminded of what it already knew. The standard defence against forgetting.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.