Open research: how AI models actually learn

What this is for. Training an AI model is expensive, and most of that cost is spent before anyone knows whether it was necessary. This is our ongoing search for ways to train models more cheaply, more efficiently and with fewer wasted runs, and for measurements that tell you early which approaches are worth paying for. It feeds directly into the models we build for clients: what we find here is what we do, or deliberately stop doing, on real work.

We run it in the open, using models small enough that anyone can reproduce every number on a laptop in minutes, and we publish the results and the data behind them here, including the ones that did not go the way we expected.

How to read the language in these records. These pages are a working laboratory notebook: the design, the numbers, the controls and the limits. Because that is what they are, you will read that an approach did not work, that a result did not hold up under a harder test, or that one method was worse than another. Those are findings, not admissions. Ruling an approach out is how the search narrows, and these are the pages that have taught us the most: a technique that looks promising and then fails a control tells you exactly where the next idea has to be different. Nearly everything we now rely on came out of understanding why something else fell short. If you would rather have the story than the record, the blog carries a plain-language version of most of these, written for a reader who has never seen a training curve.

Start with a question, not a record. Each card below is one question we set out to answer, with where it currently stands and the results behind it. Within a question the records are in the order we ran them, which is the order they are best read in: most build on the ones before, and several exist only because an earlier result made a new question askable.

Why we publish it rather than keeping it. We build custom AI models that you own and run yourself, using our own platform, and a model is only worth what its evaluation is worth. So we publish the whole record, not the flattering half: if a supplier cannot show you the experiments that went against their own idea, you have no way to check the ones that went their way. Holding our own work to that standard in public is the strongest evidence we can offer that we will hold yours to it too.

Ongoing research. These are experimental results from active work, not settled conclusions. The numbers are what we measured and the methods are described so you can judge them, but the programme is still running: later experiments here have already overturned earlier readings more than once, and several pages record exactly that. Expect this section to change as the work moves forward.