We Tested Five Ideas From This Week's AI News
Every week brings new claims about how to train AI more cheaply or make it behave better. Most never get tested by anyone except the people selling them. We run a research programme on models small enough that a claim can be tested on a laptop in an afternoon, so this week we took five ideas from the news and tested every one.
One rule shaped all five: each idea had to beat the plainest alternative we could think of, not just beat doing nothing. We wrote down in advance what result would count against each idea, before running anything. Here is what came back.
1. Teaching a model something new without it forgetting
A paper this week proposes letting new learning happen only in the internal directions an existing skill does not use, so the old skill cannot be disturbed. It works. Our model kept six times as much of its old skill as ordinary fine-tuning allowed, and choosing the same number of directions at random protected nothing at all.
But the method needs a small sample of the old skill's data to find those directions, and the plainest alternative is to mix that same sample back into training. That kept more of the old skill and learned the new one better, at no extra cost.
2. Training an agent across many setups
A large lab said this week that training an agent across many different setups, meaning different instructions, tool names and layouts, helps it cope with setups it has never seen. It did not publish the comparison. We ran it. One format: about half right on new formats. Sixteen formats, with the same total training: almost all right, even on formats built from symbols the model had never seen.
The more useful part is which variety mattered. Renaming things while keeping them in the same place taught nothing. Moving things around taught everything. We predicted that before running it: with a fixed layout a model can learn where the answer usually sits instead of how to find it.
3. Teaching a model to think less
One company reported cutting its model's thinking by 37% with almost no loss of accuracy. We charged a small thinking model a fee for every step of thinking. It cut its thinking by 38% and lost at most one point of accuracy, a strikingly similar number.
Then the plain alternative: always think for exactly three steps. That beat every version that learned when to stop, with fewer steps and less than half the training cost. The model that chose its own thinking time spent the longest on the questions it never got wrong and the least on the ones where all its mistakes were. A fee on thinking shortens it everywhere without teaching the model where thinking is needed.
4. Letting a teacher choose what to practise
A paper this week trains models on material generated at the edge of what they can currently do. We tried a simple automatic teacher that keeps picking the tasks our model gets about 70% right. It won: it beat random choice and the best of several fixed easy-to-hard plans we wrote ourselves. That is a first in this project. Three earlier adaptive methods all lost to a fixed plan.
Then we replayed one run's teacher choices on a different run, with the teacher switched off. That did almost exactly as well. So what the teacher found was a plan: practise the next thing, and retire what has been learned. Our hand-written plans had kept drilling mastered tasks for most of the run.
5. Looking inside a compressed model
Shrinking a model to run on smaller hardware usually means rounding its numbers. A new tool makes it easy to look inside rounded models, so we asked whether measuring a model's internal shape tells you how badly rounding hurt it. It does, somewhat better than measuring how much the numbers changed, and without needing an answer key, though a rerun with four times as many models found that edge smaller and less certain than our first test suggested. But a small repair won back all of the lost accuracy while the internal shape stayed different. Looking inside can tell you a model was damaged. It cannot tell you it is fixed.
What to take from the week
Most new methods work, and that is not the question. Four of the five did what their authors said. The question is whether they beat the simplest thing you could have done instead, and for two of them (the forgetting fix and the thinking fee) the simplest thing won.
When a clever method wins, find out what it is actually doing. The adaptive teacher won because of the plan it discovered, which can be written down and reused without the teacher. That is worth more than the method itself.
Write the prediction down first. Two of the most useful findings this week, that layout beats labels and that thinking was spent on the wrong questions, came from checking a specific prediction rather than a headline number.
These are small models on simple tasks. They test the mechanism behind each claim, not the claim at the scale it was made. We say in each record what would need a bigger test, and one of them, the compressed model at full size, is waiting on a larger machine.
Read the thinking-budget record
Get new results as we publish them
Roughly monthly, one finding per email, in plain English first. Including the approaches that turned out not to work, which are usually the useful ones. No sales email.
Double opt-in: we send a confirmation link and add nobody who does not click it. One-click unsubscribe on every email.