How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

It Was the Way It Was Going

By Joshua Bolick, Awareness Software Group ~13 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: Which parts of a model actually matter? – The parts that move most are not the parts that matter, and no single component is required -- the model routes around every freeze. What a part is worth shows up only when you remove it.

In plain English

What we asked. Earlier we found one direction inside a small model's learning that seemed special: nudging the model along it just before it learned made it learn sooner. We could not move that effect to another model in two different ways. So we asked what made it work in the first place.

What we found. The answer is that it was never special. A random direction that leaned the same amount toward where the model was already heading did exactly the same thing. Pushing straight along where the model was heading did twice as much, on all eight models. Our earlier result looked uncertain because it was really two groups: models where the special direction happened to point the way they were going, and models where it did not.

Why it matters. A steady push in the direction training is already moving is called momentum, and it is an old idea. Whether this push saves anything when it is not timed with hindsight is the next test.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 48 training runs. The design and kill test were committed (d06f254) before any run, with one arm's matching amended before execution and the amendment noted in the programme entry.

Program v2 Bucket O, item O13. Decisive computation: analysis/own_direction_persistence.py. Output: analysis/own_direction_persistence.json. Reproduce with python analysis/own_direction_persistence.py (about ten minutes on a throttled laptop CPU); --reuse re-derives every endpoint from the saved runs, and was checked to print the identical summary.

The question

Across J8 and O3, the only injected directions that sped a receiver's transition were built from its own gradients -- the top singular direction of its recent gradient buffer (-9.27 steps) -- and they were the only ones aligned with what the receiver was already doing (0.57 against 0.05 or less). Is it the direction's identity, or just a steady push the way the model was going?

**The push that helps is the model's own average direction, not its top direction.** Four ways to push a small model's training for the 40 steps before it learns. The top direction is the one earlier work singled out; the random one shares only how much it leans the way the model was already going; the average direction is simply the way it was going; the last just makes each step half as big again. The random push with the same lean does exactly what the top direction does, so the top direction was never special. Pushing along the model's own average direction does twice as much, on every run. What helps is momentum the way it was going.

**The top direction helps exactly as much as it leans the way the model was going.** For each of eight models, the effect of pushing along its top direction (blue) and along its average direction (green). The number is how closely the top direction points the way the model was going: 1.00 means the same way, 0 means sideways. Where the top direction happens to point the way the model was going, it helps as much as the average does. Where it points sideways, it helps little or even hurts. The average direction helps every time. An earlier result's wide uncertainty was really two groups of models.

Every run is J8's own train(), at J8's scale (alpha = 0.5, half the gradient's norm) in J8's window (the 40 steps before the receiver's transition), on J8's eight receivers:

Arm	What is added to the gradient each step of the window
`top`	J8's self-transplant: the top singular direction of the receiver's gradient buffer
`mean`	the mean of the same buffer, as a unit direction: where the model was going
`matched-random`	a random direction built to have exactly `top`'s alignment with the mean
`rescale`	the step's own gradient, i.e. the gradient times `1.5`: no fixed direction at all

Kill test, fixed before execution: mean minus top, paired over receivers, has an interval including zero -- then the top direction's identity carries nothing beyond persistence.

Anchor, in code: control and top reproduce J8's committed transitions exactly on all eight receivers. They do.

Result: the kill test does not fire -- because the mean does twice as well

Change in the transition (steps; negative is sooner), J8's transition definition (interpolated floor-to-ceiling midpoint, 0.30 gate; every run passed it):

Arm	Change	Alignment with the buffer mean
`top` (J8's self-transplant)	`-9.27` `[-17.36, -1.18]`	`0.575`
`mean`	`-19.02` `[-19.79, -18.26]`	`1.000`
`matched-random`	`-8.68` `[-17.22, -0.14]`	`0.575`
`rescale`	`-5.78` `[-6.10, -5.46]`	--

Paired difference	Mean	Interval
`mean` minus `top`	`-9.75`	`[-18.27, -1.23]`
`matched-random` minus `top`	`+0.59`	`[-0.49, +1.67]`
`rescale` minus `top`	`+3.49`	`[-4.43, +11.41]`

The kill test as written does not fire, and not in the direction it was written to detect. It asked whether the mean would do as well as the top direction; it does twice as well, on every receiver, with an interval a tenth as wide.

The decisive arm is matched-random. A random direction with nothing in common with the top direction except its alignment with the mean does the same thing, receiver by receiver, to within +0.59 steps. The top singular direction's identity carries nothing. What it carries is how much of it points the way the model was going.

Per receiver: the effect is a function of alignment

Receiver	`top`'s alignment with the mean	`top`'s change	`mean`'s change
`9041`	`0.979`	`-17.7`	`-18.2`
`9001`	`0.974`	`-18.2`	`-18.8`
`9011`	`0.950`	`-18.2`	`-19.1`
`9059`	`0.780`	`-16.5`	`-19.0`
`9029`	`0.292`	`-5.0`	`-17.6`
`9043`	`0.282`	`-1.3`	`-20.0`
`9007`	`0.264`	`-4.1`	`-18.9`
`9049`	`0.077`	`+7.0`	`-20.5`

J8's wide interval on the self-transplant ([-17.36, -1.18]) was never noise about one effect. It was two populations: receivers whose top direction happened to coincide with their mean gradient got the full effect, and receivers whose did not got little or none. The mean direction gets the full effect everywhere.

Reading it

What helps is a steady push along the model's own recent average gradient, which is momentum by another name: forty steps of the same added vector, which AdamW's per-coordinate normalisation does not cancel because it does not change sign.
A plain rescale is not nothing (-5.78, tight): multiplying the gradient by 1.5 for forty steps still helps, presumably because AdamW's second-moment estimate lags the change. It is a third of the mean-direction effect, so direction still matters beyond magnitude.
This closes J8's thread from the inside. J8 found a donor's direction does not travel; O3 found its function does not either; O13 finds the receiver's own top direction was never special. What helped was the receiver's own momentum.

What stands

The top singular direction of the gradient buffer carries nothing beyond its alignment with the mean gradient (matched random +0.59 [-0.49, +1.67] from it).
The mean direction removes 19.0 of about 153 steps to the transition, on all eight receivers.
It is oracle-timed: the window is placed from each run's own transition, and the gradient buffer it averages includes the window's own gradients (J8's capture, reused unchanged). By this programme's own rules (K2, J7, N14) that is not yet a free speed-up. Generates O14: the same push at a fixed step and at random times, and a plain learning-rate boost, compute-matched.

Limits

J8's substrate only: width-48 GRU, delayed copy, eight receivers, one scale.
The buffer looks ahead. J8 captures the receiver's direction from gradients ending at the window's close, so the pushed direction includes information from the window it is pushed in. O14's fixed-schedule arm must use a buffer that ends where its window starts.
matched-random is one random draw per receiver, and its agreement with top is per receiver, not only on average.

QUALIFIED 2026-09-26 by O14. With the averaged buffer ending where the window opens (no look-ahead) the push is worth -11.75 [-12.98, -10.51], not -19.02; one fixed opening step matches the oracle window to +0.11 [-0.17, +0.40]; and raising the learning rate by half in the same window, with no direction added, matches the push to +0.18 [-0.88, +1.24]. This record's finding about the top direction stands; its "momentum along the model's own path" reads, on this substrate, as a learning-rate boost. Text and numbers unchanged.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

AdamW: A widely used training algorithm that adapts how big a step it takes for each individual parameter. The default in most modern AI training.
compute-matched: Comparing two methods by giving them the same amount of computing power rather than the same number of steps. A method that takes twice as long per step should not get twice the compute for free.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
momentum: Keeping part of each previous training step in the next one, so training keeps moving in a consistent direction instead of zig-zagging.
normalisation: Dividing a measurement by something else to make it comparable across cases. What you divide by is a choice, and it can change the answer as much as the data does.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.