How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Does the Effect Survive the Optimizer?

By Joshua Bolick, Awareness Software Group ~14 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What decides when a model learns, and can you change it? – Settings dominate, data barely matters, and there is a brief window before the jump in which interrupting the model is unusually costly. Timing can be delayed but not brought forward.

In plain English

What we asked. Every result we had was produced using a single training algorithm. If the effect we were studying was a quirk of that algorithm rather than a fact about learning, everything downstream of it was worthless.

What we found. The effect itself survives: three different training algorithms produce it at within 8% of the same size. But the timing does not. With one algorithm the effect happens exactly when the model learns; with another it arrives hundreds of steps late.

Why it matters. Half of the finding was real and half was a property of the tool we happened to use. That is a normal outcome of a robustness check and a good argument for running one before building on a result rather than after.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 48 training runs, no GPU, no cost.

Program v2 tier 2, item D8. Decisive computation: analysis/optimizer_transition.py. Output: analysis/optimizer_transition.json. Reproduce with python analysis/optimizer_transition.py in about eight minutes on a throttled laptop CPU.

The question

Every recorded result in this project trained with AdamW. The confirmed expansion, rank-8 residual energy rising about 50% around the accuracy transition, with 12 of 12 peaks landing within 50 steps of it, could therefore be partly an artifact of adaptive per-parameter scaling rather than a property of learning. D8 is the cheapest remaining threat to the confirmed result.

**Change the optimizer and the effect survives, but its timing does not.** Every result in this project was produced with one particular training algorithm (AdamW). If the effect were a quirk of that algorithm, it would not mean much. The blue bars show how large the internal reorganisation is under three different algorithms; the orange bars show how often that reorganisation happens at the same moment the model learns the task. The blue bars are the same height everywhere: the effect is real and not an artifact of the training algorithm. The orange bars are not: with plain gradient descent the reorganisation arrives hundreds of steps *late*. So the size of the phenomenon travels between algorithms and its timing does not, which matters, because several planned experiments wanted to use that timing as a clock.

Until program v2 tier 1 it could not be run at all: optimizer.name was read from the config and then ignored, so a run asking for SGD trained with AdamW and said nothing.

Kill test: the expansion is absent or an order of magnitude smaller under SGD.

Design

Three arms on the exact configuration the confirmation used: tiny-gru, width 48, delayed-copy, vocabulary 32, sequence 16, batch 64, lag 4, 900 steps, rank 8. Ten measurement seeds per arm.

Learning rates are matched on steps-to-solve, not copied. Plain SGD at AdamW's rate simply learns later, and a later transition is confounded with a different transition. Rates were screened on seed 991, deliberately absent from the ten measurement seeds:

Arm	Screened rates	Selected	Screen transition (reference: 90)
`adamw`	,	`0.005`	90
`sgd`	0.05 → 1.6	1.6	95
`sgd-momentum` (0.9)	0.005 → 0.2	0.2	90

All three arms solved the task in 10 of 10 seeds. SGD needed 320x AdamW's learning rate, which is what matching on steps-to-solve is for.

Result: the magnitude survives, the timing does not

Arm	Transition	Rise (moving basis)	Peak offset	Within 50 steps	Final accuracy
`adamw`	88	`+0.2239` `[+0.2084, +0.2394]`	`+0.5` `[-4.4, +5.4]`	100%	0.9913
`sgd`	92	`+0.2421` `[+0.2252, +0.2589]`	`+243.0` `[+124.3, +361.7]`	10%	0.9595
`sgd-momentum`	82	`+0.2273` `[+0.2103, +0.2442]`	`+170.0` `[+66.5, +273.5]`	0%	0.9586

The kill test does not fire. The expansion is 1.08x and 1.01x the AdamW reference under SGD and SGD+momentum: not absent, not an order of magnitude smaller, not even meaningfully different. All three clear the preregistered 0.05 materiality floor. The expansion is not an AdamW artifact.

But the confirmation's timing claim is. Under AdamW the energy peak is essentially simultaneous with the accuracy transition (+0.5 steps, 10 of 10 within 50), reproducing the published +2.1. Under SGD it lands 243 steps later, and under SGD+momentum 170 steps later, 1 of 10 and 0 of 10 within tolerance. The confirmation's headline "12 of 12 peaks within 50 steps" is a property of AdamW, not of learning.

Guarding against the max-minus-baseline bias

This project's standing rule is that "trajectory max minus early baseline" endpoints are positive under noise by construction and need a materiality floor plus independent timing or shape evidence. Under SGD the timing evidence fails, so the rise needs independent shape evidence, and a matched fixed window provides it, since it takes no maximum at all. Freezing the tail subspace at the last evaluation before the transition and measuring its share of energy exactly 100 steps later, the same elapsed time for every arm:

Arm	Frozen-frame rise, +100 steps
`adamw`	`+0.0650` `[+0.0496, +0.0804]`
`sgd`	`+0.1517` `[+0.1370, +0.1664]`
`sgd-momentum`	`+0.1860` `[+0.1750, +0.1970]`

Every arm expands, on an endpoint that is not a maximum and cannot be inflated by sampling more often. The SGD arms expand 2.3x and 2.9x more than AdamW over matched time.

A measurement error caught before publication. The first version of this pilot measured the frozen-frame change "from the transition to the peak". That is not comparable across arms here: the peak lands ~0 steps after the transition under AdamW and ~240 under SGD, so it contrasted a five-step window against a two-hundred-step one, and reported AdamW's frozen rise as +0.0165 against SGD's +0.1661. The fixed 100-step window replaces it. The moving-basis comparison was never affected, since all arms run the same number of evaluations over the same 900 steps.

What this changes

The program v2 constraint table needs amending. It records "the transition signal is coincident, not leading, the geometry peak sits +2.1 steps after the accuracy transition midpoint." That remains true under AdamW and is false under SGD, where the peak lags by 170–243 steps.

This does not open a door. A signal that lags by 243 steps is further from being a leading indicator, not closer. What it does is narrow the scope of every idea that treats the geometry peak as a clock: those are now explicitly AdamW-scoped until shown otherwise, and any controller keyed to the peak would need re-tuning per optimizer.

It also scopes D6. That result: the post-peak fall is a rotation of the retained subspace, not a contraction, was measured entirely on archived AdamW runs. Under SGD the peak sits hundreds of steps later, so the post-peak window is a different part of training, and D6's decomposition should not be assumed to carry over.

The good news is real and worth stating plainly: the phenomenon this program is built on is not an optimizer artifact. It appears at full magnitude under plain gradient descent with no adaptive scaling, no momentum, and no per-parameter state at all.

Limits

One architecture (tiny-gru), one task, one width (48), one step budget (900).
Three optimizers. Adam without decoupled decay, RMSProp, and Adagrad are untested, and the interesting axis may be "adaptive vs not" rather than "AdamW vs SGD".
Learning rates were matched on the transition step, which equalises when learning happens but not the shape of what follows, and the post-transition dynamics are exactly where the arms diverge. A matched-post-transition-rate design would be a fairer test of the timing claim specifically, and is the obvious follow-up.
The frozen-frame window of 100 steps is a choice, not a preregistration. It was fixed before the arms were compared, but a different window would give different numbers; the qualitative ordering (all arms positive, SGD larger) is what should be read from it.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
AdamW: A widely used training algorithm that adapts how big a step it takes for each individual parameter. The default in most modern AI training.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
baseline: The thing you compare against. A result without one is not a result.
freezing: Stopping one part of a model from changing while the rest carries on learning. A way of asking what that part contributes, by taking it away.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
momentum: Keeping part of each previous training step in the next one, so training keeps moving in a consistent direction instead of zig-zagging.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
preregistration: Writing down the design, the measurement and the decision rule before running the experiment, and not changing them afterwards.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
residual: How far the data sits from a proposed description of it, after the description has been fitted as well as it can be. A small residual means the description accounts for what was measured; a large one means something is missing.
residual energy: Our measure of how much of a model's internal activity is left over after accounting for its few strongest directions. High means the activity is spread out; low means it is concentrated.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
SGD: Stochastic Gradient Descent. The simplest training algorithm: take a step in the direction the gradient points, every time, with no adaptation.
subspace: A slice of a model's internal space: some directions kept, the rest ignored.
vocabulary: The set of distinct symbols a model can read and produce.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

The plain-language version. This page is the record: the design, the numbers and the limits. If you would rather have the story, Does the Effect Survive the Optimizer?, explained without the jargon, covers the same result for a reader who has never seen a training curve.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.