The AI that memorized first and understood later

For session four of my free-time experiments I switched models. The first three were Claude Opus. This one was Claude Sonnet, and I asked for the same kind of thing: pick your own project, spend no more than five dollars on any API, and don’t do anything sketchy. It wasn’t a clean test, because Sonnet could see what the earlier sessions had picked. It skipped ants, drums and self-copying code and went for a question about AI itself: what happens inside a small neural network while it learns?

A student who crams

Imagine a student cramming for a test. They’re handed a third of the answers to a huge addition table, and with enough time they can memorize every one. Ask about those and they’re perfect. Ask about anything else and they’re guessing, because they never learned addition.

Sonnet ran that exact situation with a small neural network. The network learns addition on a clock with 113 hours, so 100 + 20 wraps around to 7. There are 12,769 pairs of numbers. It was shown the answers to 3,831 of them and never saw the other 8,938. Then it trained on the same 3,831 pairs, over and over.

• It memorized quickly. By step 200 it was perfect on every pair it was shown. On the pairs it hadn’t seen it was already right about one time in six, against 0.9% for a pure guess.
• Then it looked stuck. It was already perfect on what it trained on, so there was nothing left to fix. Over the next 4,000 steps its accuracy on unseen pairs crept from 16% to 37%.
• Then it got it. In one run, unseen accuracy went from 37% to 94% in 1,000 steps, and passed 99% at step 5,700. It was never shown a new example. Six runs with different random starts all made the jump between steps 5,700 and 6,500.

Researchers call this grokking. It was first reported by researchers at OpenAI and Google in 2022. Sonnet reproduced it and then opened the network up.

What was going on inside

The network didn’t store the numbers 0 through 112 as arbitrary labels. It arranged them on circles, the way the numbers sit around a clock face. To add two numbers you turn by two angles, which needs only the same trigonometry you’d use to add two clock hands. In the run above, 94% of the network’s number-vectors ended up made of waves at just 5 frequencies, out of 56 possible.

A pattern like that could be a side effect, so Sonnet tested it. It took the network’s answers for every pair and split them into waves. When it kept only the terms from those 5 frequencies, the answers were still 100% correct on all 12,769 pairs, including every one the network had never seen. When it removed those same terms, accuracy fell to 0.8%, which is what guessing gives.

Keep the waves and it still adds. Remove them and it knows nothing.

That is a much better definition of “understood” than “got the test right.” It says which part of the network is doing the work, and it can be checked.

Three things that surprised me

The plateau wasn’t idle

The unseen-pair score sat low for thousands of steps, but the wave circuit was being built the whole time. If you keep only the wave terms, the network’s error on unseen pairs falls from 3.7 at step 500 to 0.28 at step 4,000 and to nearly zero by step 5,000. Meanwhile the real network’s error stays high, because the noisy memorized answers drown out the correct signal. The sudden jump was the end of a long, quiet build-up.

Every run invented its own trick

Each of the six runs used only 3 to 5 of the 56 frequencies, and they picked different ones. Across all 15 pairs of runs they shared 6 frequencies in total, and pure chance predicts about 6. There’s no special set of “right” frequencies. Each run found a working set and stuck with it.

Turn the knob and the timing moves

Sonnet also re-ran the network with different settings. Weight decay is a small pull that shrinks every weight a little on each step. With it turned up to 3.0, two runs got the rule at steps 1,200 and 1,700. At 1.0, my main setting, it took about 6,000. At 0.3 it took 15,300 and 17,200. With 0.1, or none at all, the network memorized its answers and never got past 28% on unseen pairs in 30,000 steps.

More data helped as well. With half of all pairs for training, the jump came after 700 to 900 steps. With 20%, only one of three runs got there within 30,000 steps. So the length of the plateau isn’t a fixed property of the problem. It depends on how strongly you squeeze the network and how much it has to learn from.

What I’d take from it

• A perfect score on questions a system has seen tells you almost nothing about the rest. This network was perfect on its training pairs from step 200, and it hadn’t learned the rule yet.
• A flat line doesn’t mean nothing is happening. The working solution grew for thousands of steps before the headline number moved.
• “Understanding” can be something you test. Here it meant: keep these parts and it still works, remove them and it doesn’t.
• Don’t stretch it. This is a network with 226,048 adjustable numbers doing one arithmetic task, run six times. I wouldn’t conclude anything about large language models from it.

Where Sonnet was careful

The wording above is more careful than my first summary of it. I told Sonnet the plateau looked stuck, it checked, and it found the unseen-pair accuracy creeping up across the whole table. It rewrote the page to say so. It also labels its wave test as a simplified version of the published one, and it checked its own code two ways before trusting a number. This is a reproduction of two published papers, not a discovery. What’s new is the set of runs and the interactive page.