How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Depth Sets the Order

By Joshua Bolick, Awareness Software Group ~11 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually happens at the moment a model learns? – It builds machinery rather than selecting it, working through the task in a reproducible order and trying a simpler wrong rule on the way. The visible training curve cannot tell you which is happening.

In plain English

What we asked. We built a task with five skills, where two of the skills can only be done by combining the others. On their own, those combined skills are the easiest of the five to learn. So if a model learns easy things first, it should learn them first; if it learns in order of what depends on what, it should learn them last. Our first try ran out of time before the deepest skill was learned, so we ran it again, more than three times longer.

What we found. Every skill was learned on all six runs, and on every one of them the order was the same: the three simple skills first, then the skill built on them, then the skill built on that. The deepest skill took about 27 times as long as it takes when taught on its own.

Why it matters. This matters because a model's learning order is sometimes read as a map of what depends on what. Here that reading holds, in a setting built so that difficulty points the other way. What order cannot tell us yet is exactly which skill feeds which, and that is the next experiment.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, six seeds. The design, precondition and kill test are O6's, unchanged; the pilot was committed (b3eb4cf) before any run.

Program v2 Bucket O, item O11. Decisive computation: analysis/prerequisite_graph_long.py. Output: analysis/prerequisite_graph_long.json. Reproduce with python analysis/prerequisite_graph_long.py (about forty minutes on a throttled laptop CPU); --reuse re-derives every endpoint from the saved series, and was checked to print the identical summary.

The question

O6 planted five skills -- look-ups A, D, B (depth 0), C = A + B (depth 1) and E = C + D (depth 2) -- with both composites made the easiest skills in isolation, so planted depth and isolated difficulty predict opposite acquisition orders. Its precondition failed: E was acquired on only 2 of 6 seeds in 1500 steps, so its ordering test was not computed. O11 runs that test as written, with 5000 steps. O6's isolated runs are reused from its committed output; only the composite runs are retrained.

**Given time, every skill is learned, always in the order of what depends on what.** The same five-skill experiment as before, run more than three times longer. A, B and D are simple look-ups; C combines A and B; E combines C and D. Blue is when each is learned when all five are trained together, grey when each is trained alone. Every skill is now learned on all six runs. On every one of six runs the model learned the three look-ups first, then C, then E, although C and E are the two easiest skills when taught alone. The skill two levels deep took about 27 times as long as it does alone. What a skill depends on, not how hard it is, decides when it is learned.

Kill test (O6's, fixed before execution): per seed, the rank correlation between acquisition step in the composite task and planted depth; fires if the mean's interval does not exclude zero.

Precondition (O6's): every skill acquired on at least four of six seeds.

Anchor, in code: each extended run's first 1500 steps must reproduce O6's committed series exactly. All six do, so nothing changed but the length.

Result: the precondition passes, and the kill test does not fire

Acquisition step (first step past halfway from chance to perfect, and staying there), mean over six seeds:

Skill	Depth	In the composite task	Seeds	Alone (O6)
`B`	`0`	`146.7`	`6/6`	`171.7`
`D`	`0`	`378.3`	`6/6`	`188.3`
`A`	`0`	`438.3`	`6/6`	`190.0`
`C`	`1`	`991.7`	`6/6`	`70.0`
`E`	`2`	`1695.0`	`6/6`	`63.3`

Rank correlation of acquisition order, per seed:

Against	Mean	Interval
planted depth	`+0.894`	`[+0.894, +0.894]`
isolated difficulty	`-0.806`	`[-0.994, -0.618]`
depth minus difficulty	`+1.700`	`[+1.512, +1.889]`

Every one of the six seeds acquires all three look-ups before C, and C before E. +0.894 is the largest rank correlation depth can have here, because three skills share depth 0; the interval has no width because every seed reaches that ceiling. Read as a sign test instead of an interval, six of six seeds in the planted order has a probability of 1/64 under no relation at all.

Per seed, E arrives at 1380-2420 steps. Alone it takes 63. Being two levels deep costs this skill about 27 times its isolated learning time; C, one level deep, pays about 14 times.

What this adds to O6 and P3

P3's finding holds one level deeper, now as a test rather than a description. With depth and isolated difficulty set against each other on purpose, acquisition follows depth on every seed and runs against difficulty (-0.806).
O6's failure was a budget. Its first 1500 steps are reproduced exactly here; E simply arrives later than O6 looked, on four of six seeds.
The cost of depth grows with depth. One level multiplies the learning time of this composite by about 14, two by about 27, in a model for which both are the easiest skills when taught alone.

Limits

Layers, not edges. The test is whether acquisition recovers depth. Whether it recovers which skills feed which is not tested: every depth-0 skill arrives before every depth-1 skill, so order alone cannot tell C's parents from D. That is O12.
One graph, one task family, one width (a width-64 GRU), six seeds.
The interval is degenerate because every seed ties at the maximum; the sign test is the fairer summary of strength, and it is 6/6.
The retrieval positions did not form the intended difficulty ladder (O6's limit, unchanged); it does not affect a test that compares layers.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
sign test: The simplest check that a pattern is not chance: count how many runs go the predicted way. Six out of six happens by luck about once in 64 tries.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

In plain English

The question

Result: the precondition passes, and the kill test does not fire

What this adds to O6 and P3

Limits

Terms on this page

Read next

Get new results as we publish them

Want this measured on your data?