The AI That Aced the Math Olympiad Can’t Count the Letters in “Strawberry”

Jun 10, 2026 · by Hardik Goel

Last summer, AI systems built by Google and OpenAI correctly answered five of six questions at the International Mathematical Olympiad, one of the most elite high school math competitions on the planet. Problems that stump PhD candidates. Problems with solutions that take pages to write out.

A few months later, a software engineer in Sri Lanka named Anuradha Weeraman noticed that a leading AI couldn’t correctly count the letters in a simple word.

Same technology. Same year. Same approximate model generation.

This is not a contradiction. This is jagged intelligence, and it’s the most important mental model you can develop for working with, building on, or thinking clearly about AI in 2026.

The New York Times put it on the front page of their Business section last month. Andrej Karpathy coined the term. Google DeepMind published a preprint formalizing it. And yet, most teams deploying AI systems right now are still treating “the model is smart” as a binary, which means they’re going to keep getting surprised by failures that were, in hindsight, completely predictable.

Let me explain what jagged intelligence actually is, where it comes from structurally, and what you actually do about it.

What Jagged Intelligence Actually Means

The standard way people talk about AI capability is a smooth curve. Models get smarter over time. Benchmarks go up. GPT-N is better than GPT-(N-1). You can roughly plot it.

The problem is that this framing implies something like uniform improvement. Better at everything, proportionally, the way a student who studies harder gets incrementally better across all subjects.

That’s not what’s happening.

What’s actually happening looks more like this:

The line at 100 is human baseline. The bars are approximate frontier model capability. The profile is jagged, not smooth.

The peaks are real and extraordinary. The valleys are also real, and they appear right next to the peaks, with no warning sign between them.

François Chollet, the AI researcher who created the ARC benchmark specifically designed to probe reasoning AI can’t fake, put it clearly: “AI does not have general intelligence. What it has is a lot of different skills.”

That’s the whole thing. A collection of skills, not a coherent intelligence. Some skills are superhuman. Others are subhuman. And the boundary between them is not obvious from the outside.

Why This Happens: The Architecture Explanation

This isn’t mysterious once you understand what’s actually going on inside the model. There are three structural reasons jaggedness exists, and they compound each other.

1. Training Distribution Density

An LLM learns by predicting patterns in its training data. Where the training data was rich, deep, and diverse, the model develops strong representations. Where it was sparse, the model’s internal “map” of that territory is blurry.

The internet has an enormous amount of mathematical proofs, code, legal documents, and scientific papers. It has very little data about physical world tasks, character-level string manipulation, or genuinely novel reasoning problems that haven’t appeared in any form before.

The model is brilliant at math olympiad problems partly because thousands of such problems, with solutions, exist online. It fails at counting letters in “strawberry” because that specific task, checking character-level composition of a word, is almost never represented as a reasoning task in text. The model sees “strawberry” as a token, not as a sequence of characters.

Which brings us to the second reason.

2. Tokenization Creates Invisible Seams

This is the one that surprises most people.

LLMs don’t see text the way you do. They see tokens, chunks of text that their vocabulary was trained on. “Strawberry” might be a single token. Or it might be split as “straw” + “berry.” Either way, the model never processes it as s-t-r-a-w-b-e-r-r-y, nine individual characters in a sequence.

When you ask “how many r’s are in strawberry?”, you’re asking the model to do character-level counting on something it perceives as a semantic unit, not a string. It’s like asking someone to count the brushstrokes in a word they’ve only ever seen as a printed label. The answer requires decomposing something the model learned to treat as atomic.

Research from TCS confirms this formally: when natural words get split across token boundaries in ways that break semantic units, model performance degrades measurably and predictably. The tokenization scheme literally shapes what the model can and can’t do well.

3. The KV Cache Doesn’t Think, It Retrieves

Here’s the memory architecture piece that most articles skip over.

When an LLM processes a long prompt, it builds what’s called a KV (key-value) cache: a compressed representation of everything in its context window that attention layers use to relate new tokens to previous ones. This is the model’s “working memory” during a conversation.

The problem: this working memory has no semantic indexing. It doesn’t organize information the way human memory does, with important things flagged and irrelevant things suppressed. It’s positional and attentional. The model pays more attention to recent tokens and to tokens with high attention weights relative to the current generation step.

This is why models can “forget” something you said 50 messages ago even though it’s technically still in context. It’s why long-horizon planning degrades. The KV cache grows, memory pressure increases, and the attention mechanism literally has less bandwidth to allocate to distant context.

It’s also why the model can answer a complex reasoning question about a document you just pasted, but stumble on a simple factual question about something in the middle of a long conversation. The architecture’s attention isn’t uniformly distributed. It’s weighted. And that weighting creates predictable blind spots.

The Memory Architecture Map

Here’s how the different memory types in an LLM system interact, and where jaggedness shows up in each:

Jaggedness isn’t just a training-time phenomenon. It shows up differently at each layer of the memory stack. And each layer has different mitigation strategies.

My Honest Practitioner Take: The Real Risk Is Confidence, Not Capability

Here’s what 15 years of building systems has taught me about failure modes.

The most dangerous machine isn’t the one that fails loudly. It’s the one that fails quietly, confidently, and in a way that looks correct to a casual observer.

LLMs fail that way. When they hit a valley in the jagged profile, they don’t say “I’m outside my training distribution.” They don’t slow down, flag uncertainty, or produce garbage that’s obviously wrong. They produce fluent, well-structured, confident output that happens to be incorrect.

This is the gap that the Google DeepMind jaggedness paper is trying to address formally. Their argument is that we need jaggedness profiles, not just benchmark scores. A single number like “92% on MMLU” tells you the average, which tells you almost nothing about where the cliff edges are.

I’ve seen this in production repeatedly. Team demos the model on their core use case, it performs beautifully, they deploy. Three weeks later, a user hits an edge case that’s technically simple but sits in a valley of the model’s capability profile. The failure is jarring because everything around it was working so well. And because the model failed confidently, the user trusted it.

The jagged profile problem isn’t just a safety issue. It’s a trust calibration issue. Users who see peaks assume the valleys don’t exist. And that assumption is exactly what gets teams into trouble.

What You Actually Do About It

This is the section most articles skip. Here’s the practitioner playbook:

1. Map the jagged profile for your specific use case before deploying.

Don’t benchmark on the model’s strong tasks. Specifically test the boundaries. What’s the simplest thing adjacent to your use case that the model might fail at? Go find that thing. Test it systematically. The goal is to find the valleys before your users do.

2. Treat confidence signals as independent from accuracy.

A model saying something confidently is not evidence that it’s correct. Build your systems accordingly. Log outputs. Sample multiple responses on high-stakes decisions. Build in human review at the specific junctions where the model is likely to be in a valley. Not everywhere, that’s too expensive. At the specific junctions.

3. Use tools to route around architectural weaknesses.

Character counting? Use a code interpreter. The Python len() function will never miscount letters in “strawberry.” Factual retrieval on current events? Use RAG. Long-horizon planning across many steps? Break it into smaller verified checkpoints rather than trusting one long context window.

The model’s architecture creates specific, predictable weaknesses. External tools exist precisely to paper over those weaknesses. The insight is to route deliberately rather than hoping the model will handle everything.

4. For memory: scope, chunk, and verify.

If you’re building agents with persistent memory: scope what gets remembered aggressively (not everything is worth remembering), chunk long context into verified summaries rather than relying on raw KV cache for long conversations, and verify retrieved memories before acting on them. Bad memories compound. A wrong assumption from session one can corrupt session ten.

5. Build jaggedness-aware evaluation, not just accuracy metrics.

The final score on a test suite hides the variance. Report capability ranges, not just averages. “The model gets X% on task type A, Y% on task type B, and Z% on edge cases” is a more honest and more useful evaluation than a single headline number.

Subscribe now

The Bigger Picture

The NYT piece frames jagged intelligence as a tool for reframing the AI debate, specifically whether AI is “as smart as humans.” That’s useful at the societal level.

But for practitioners, the more useful reframe is narrower: jaggedness means the model is a specialist, not a generalist, and the job is to figure out which specialist you’re dealing with on which task.

The analogy that works for me: think of an LLM like a team of incredibly talented consultants who happen to have inexplicable blind spots. One of them is a world-class corporate lawyer who can’t do basic arithmetic without a calculator. Another is a brilliant analyst who gets confused by metaphors. You wouldn’t fire them. You’d route tasks appropriately, verify the outputs you’re uncertain about, and build workflows that complement their weaknesses rather than expose them.

The model isn’t broken. The deployment pattern is.

And perhaps the strangest part of jagged intelligence is this: as models get better at more things, the peaks get higher and the valleys get easier to miss. The model that scores at 99% on the benchmark is the one whose failures are most surprising when they arrive.

The better the model gets on average, the more dangerous it is to assume it’s good everywhere.

-Hardik

buymeacoffee.com/HardikGoel

Sources: Andrej Karpathy, “Deep Dive into LLMs” (YouTube, 2025). Cade Metz, “How Jagged Intelligence Can Reframe the AI Debate” (NYT, April 2026). Morris et al., “Characterizing Model Jaggedness Supports Safety and Usability” (Google DeepMind preprint, 2026). Pawar et al., “Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs” (TCS Research, 2024).

Thanks for reading Hardik's Substack! This post is public so feel free to share it.