Models that skim books super carelessly.

Did you know about this?

Most of you are probably familiar with techniques where LLMs manage their own context. The approach MemGPT proposed is the representative one. It pulls in information chunk by chunk through tool calls, archives older content with a sliding window, and has the LLM itself summarize and compress the conversation. The tools that summarize long conversations all grew out of these ideas.

But I had this thought. (This is not AI, it's really my own idea.)

GIF

"If the model marks unimportant tokens with a special token, wouldn't it be nice if the serving engine detected that and quietly removed them?"

The reason is that LLMs are smart enough to recognize their own reasoning mistakes. I'll show you the evidence in a bit.

But what's really ridiculous is that this is already an actively researched field. I was already too late. Still, there's something you need to know first.

The first token, which seems meaningless, is the one that never dies

For an LLM, the first token doesn't carry much meaning. It's usually an input token, and most of the time it's the first character of the system prompt. Some models explicitly insert a start token, but these days the trend seems to be using an input token instead, and it can also be modified through fine-tuning.

But this token absorbs an enormous amount of attention. Researchers call this an attention sink.

GIF

Apparently, when the softmax must force a sum of 1 but no value has diverged strongly, or when the top-k candidates are all similar, the weight gets piled onto the first token. Isn't that fascinating?

The important thing is that if you evict this sink token in a long conversation, the model's output falls apart. This is what StreamingLLM uncovered. That's why serving engines never discard this seemingly useless token. If they did, everything would break.

I'll save the mechanism for a separate explanation. It's fun to dig into.

Put simply, the dumbest token is the one that survives the longest.

GIF

Research that keeps only important tokens and evicts the rest is already active

H2O: keeps only a small number of heavy hitters that receive lots of attention and discards the rest of the KV cache
SnapKV: scans the prompt once and compresses only the important parts

For reference, in these eviction studies, keeping sink tokens as an exception has already become the standard practice.

DeepSeek goes even further and only looks at the tokens it needs to see

This is DSA, introduced starting with DeepSeek-V3.2-Exp. A small module called the lightning indexer picks the top-k at every query, saying "you only need to look at these tokens," and the engine skips the attention computation for tokens that weren't selected. What's truly shocking is that it only looks at the top 2,048. If you think about it simply, it's like summarizing a book by reading only 2,048 important words from the entire book. You might wonder how that's even possible, but the indexer that selects those 2,048 is also built through very rigorous training. This is what makes ML so fascinating.

GIF

This is the closest thing to "discarding unnecessary tokens," but precisely speaking, it's not removing them, it's not looking at them. It's a trick that dramatically cuts the cost of long contexts. It's a more sophisticated method than the idea I had.

So I tried it myself

Talking only about model layers feels like chasing clouds, so let me introduce two open-source projects I built around the same problem.

inline-agent: an advanced sliding window

At every step, an agent re-reads the entire conversation history. According to research, 99% of Claude 4 Sonnet's 100 billion tokens per day are not generation but re-reading (Xiao et al., FSE 2026). Is that too obvious? On SWE-bench, 63% of the average 48.4K tokens are tool results, and most of them are useless in the next step: passing test logs, ls output, and so on.

But agents can't actively erase their context. Even when AgentDiet gave the model an erase tool, Claude 4 Sonnet just did the original task. So inline-agent handles the cleanup in a different layer so the agent doesn't need to spend effort on it.

Keeps the most recent N turns in their original form, and replaces older turns with command-result pairs (removing most of the text)
Always preserves errors, failures, and diffs; deletes passing tests and directory listings (the criteria are somewhat ambiguous, but it uses keyword search)
Zero LLM calls. There's also a paper (NeurIPS WS 2025) showing that rule-based masking is as effective as LLM summarization

pi-recap: swap in summaries, and recover when needed

I wanted to erase tool call results even more aggressively. So pi-recap delegates summarization to an LLM. But wait, here's the important concept. While the main agent works, a shadow clone that shares the same model and the same context as the main agent writes a 1–3 line summary for each tool result. After a few turns, the original is quietly swapped out for the summary. If it's needed later, the original can be restored with recap_recover.

The key point is that the clone uses exactly the same prefix as the main agent. Since it hits the provider's prefix cache, the summarization input is billed at cached rates. Even more important, an LLM that knows nothing summarizing will miss a lot of meaning, but when it compresses its own content, it knows which information to keep and which information isn't needed.

In an actual coding session, tool result tokens in the main context went from 75.5K to 7.0K, a 90.7% reduction. When I analyzed 226 summaries from a long session, the original-recovery call was made 0 times out of 127 instances.

But a fair question arises here. What if the summaries were garbage, or they cut off something that was needed?

Looking at the agent's reasoning logs, there's not a single complaint about the swaps. Is that normal? My judgment is this: if it had been insufficient, it would have shown.

Agents are structured to decide their next action based on observations (tool results) (ReAct). When an observation diverges from expectations, behavioral changes like re-checking or retrying become immediately visible.
Models also know quite well what they don't know. Language Models (Mostly) Know What They Know (Kadavath 2022). The ability to not fabricate answers when the context lacks them has been a benchmark item since SQuAD 2.0. Pretty strong evidence, right?
When information is insufficient, it shows up in performance before it shows up in words. As Lost in the Middle demonstrated, information loss in long contexts leads directly to accuracy drops. The reason compression research like CoACT unconditionally preserves "information directly tied to the next action" is that removing it causes performance to drop immediately and visibly.

To be honest about one caveat, there's research showing that models don't always say what they really think (Reasoning Models Don't Always Say What They Think, 2025). So I judge by "behavioral signals" rather than "words": recovery calls, re-exploration, asking the same thing again. The fact that not a single such signal appeared across 127 swaps means that no insufficiency cases were observed.

And since the swaps aren't announced to the agent, it keeps running fine without knowing. Can I say I've implemented "quietly removing"...?