How much model does a task need, and what changes when it has more?

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Does a bigger model learn differently, or just better? And what does the extra capacity buy?

Where this stands

The abrupt jump is what spare capacity buys -- it fades smoothly as the model shrinks, long before the model stops working. Capacity and task difficulty act separately, not as a ratio.

It is the closest this project has come to a scaling-shaped statement, and it reframes the central phenomenon as something oversized models do rather than something tasks force.

Still open · 6 published results bear on this question.

What this does not settle yet

Stated plainly, because the gaps are as much a part of the record as the answers:

• Everything here is tiny models on a laptop. The scale question is the standing limit on the entire project.
• Architecture and objective are perfectly correlated across every result (I5).

How these results fit together

Four records asking what size of model a task needs, and how that changes what we see. The finding that shapes the rest is that the sharpness of the jump depends on having capacity to spare: bigger models transition more abruptly, smaller ones more gradually, and the effect is strong enough to beat seed noise several times over. That reframes the whole phenomenon. The sudden jump is not a universal fact about how neural networks learn -- it is what a model with room to spare does on a task that does not need all of it. The phenomenon is also architecture-dependent: it appears reliably in gated recurrent models and does not appear in the ungated and attention-based models tested, even when those models learn the task perfectly well. We also tried to compress size and difficulty into one number, on the theory that only the ratio matters, and that failed cleanly: three settings at an identical ratio spanned a factor of two, because the two axes have genuinely different strengths. Difficulty is not simply capacity in reverse.

The results

Each of these is a self-contained record: what we asked, what would have proved us wrong, what we found, and what it does not show. They open with a plain-language summary before any of the technical detail.

Back to all research questions