How do we know our own results are real?

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Most of what goes wrong in this kind of work is not the experiment; it is the comparison. These are the times our own checks overturned our own headline.

Where this stands

Repeatedly, the control rather than the measurement decided the result -- and several striking findings dissolved when the right comparison was finally run.

If you are evaluating whether someone would catch a problem in your models, this is the group to read. It is the most transferable output the project has.

Still open · 36 published results bear on this question.

What this does not settle yet

Stated plainly, because the gaps are as much a part of the record as the answers:

• The programme still has no unlearnable-data control anywhere except one record (I6).

How these results fit together

Ten records that are all about this project checking its own work, and they are the most transferable thing here. Several overturned our own headlines. The largest is that a pattern we had described as the model expanding and then consolidating is substantially a *measurement* effect: we were re-fitting the coordinate frame at every step, and the apparent contraction is the frame rotating rather than the model unwinding. Any quantity compared across training steps now has to say which frame it was measured in. A companion record does the opposite service -- certain quantities depend only on eigenvalues, which are unchanged by rotation, so that critique cannot reach them, and one of those still peaks on the event. We also audited how many of our own results are close calls, found that eight runs is not enough for several of them, and published that. The habit that produced most of these: re-read the whole record after every new result, not just the item in front of you. The number of times the mistake was in the *analysis* rather than the experiment is the single most useful statistic on this site.

The results

Each of these is a self-contained record: what we asked, what would have proved us wrong, what we found, and what it does not show. They open with a plain-language summary before any of the technical detail.

Back to all research questions