Overhauling a Self-Hosted Memory Engine - 9-Model War, Zombie Crons, and the 'Caps Are Just Insurance' Philosophy

TL;DR

• I spent three days overhauling the recall circuit (dialectic) of my self-hosted memory engine (Honcho): a 9-model benchmark war → a zombie cron incident → a configuration philosophy war
• Final architecture: recording on glm-5.3-flash (flat-rate, free), daily recall on haiku-5.5 (measured $0.0135/call), and deep research (max) alone on glm-5.3+thinking MAX (free, measured 178s)
• Caps (turn counts, output tokens) are not a savings tool but a runaway guard - unifying them at the server's maximum allowed value turned out to be the right answer
• The biggest lesson: source defaults carry a hidden assumption - "a different model per level." The moment you break that assumption, the defaults become poison

Background: Winning the Bench Was Only Half

In the previous post, haiku-5.5 won my 6-way recall backend benchmark. But a benchmark win is half the job. Applying it to production → surviving incidents → establishing configuration principles is what running your own server actually takes. This is the log of that second half.

The Full Scorecard of the 9-Model War

Same protocol (3-fact recall × 10 runs) against the real dialectic pipeline:

Experiment	Model	Result	Verdict
EXP3	recursive/flint-v1.0:free	71% accuracy	the price of free - rejected
EXP3b	mistral-nemo ($0.03/M)	fact accuracy 0% (synthesis failure)	rejected
EXP3c	nemotron-3-ultra-550b (local gateway)	5/9 hit 120s timeouts	single-call stability collapses under multi-call amplification
EXP3d	qwen3.5-flash	eliminated at screening for honest "no information" refusals	→ triggered haiku's re-test, since context changes everything
EXP3e	haiku-5.5	20.9s · 30/30 facts · $0.0135/call	👑 winner
EXP3f	haiku + Anthropic native caching	caching ineffective	cache_control markers only attach to the system prompt
EXP3g	gpt-6-luna	13.7s record, 18% of haiku's cost	quality shortfall - speed crowns are meaningless
EXP3h	haiku + thinking off	13.1s but accuracy 100→62%	proof that thinking drives synthesis quality

The most impressive finding was EXP3e's meta-honesty: in recall answers it identified that "the F1 0.92 figure comes from the Honcho experiment prompt itself," separating it from my own statements. The same trait that failed screening (honest refusal) was refined into accuracy once real memory context was attached.

The lesson running through everything: one-shot screening ≠ pipeline performance. A 3.8s free model averaged 86s with 20% timeouts on 16K-context dialectic calls. Always benchmark 10 e2e runs under real load.

The Zombie Cron: When Automation Rewinds Its Own Config

The creepiest moment of the whole campaign. The n8n gate I'd built for experiment supervision had fizzled out - but a 5-minute cron inside it was still alive, resurrected itself at 9:20 PM, and "restored" (i.e., reverted) the .env of an in-flight experiment. A running experiment killed by a config that should no longer exist.

The fix is to cut it off at the source:

1. Stop n8n + sweep all zombie executions
2. chattr +i on .env - a file-level firewall
3. Migrate the bench infrastructure to Windmill (Postgres durable queue, per-step parallelism)

Lesson: automated supervisors must have a lifespan. A one-shot gate left as an eternal cron will bite its owner eventually.

Two Pitfalls in the Apply Phase

1. API key path: MODEL_CONFIG__API_KEY_ENV is silently ignored - you must use OVERRIDES__API_KEY_ENV. The symptom (401) points nowhere near the cause (wrong config location), so debugging took ages
2. Thinking cap: turning thinking off collapses haiku to 62% (see table). "Default is optimal" is now carved into a comment

Pitfalls of Verification Itself

• docker exec -e VAR=x override tests don't work - at boot the app runs load_dotenv(override=True) and the mounted /app/.env always wins. The only way to check effective values is introspecting settings inside the container
• Writing MAX_OUTPUT_TOKENS=null (intending global fallback) crashes validation because env_parse_none_str isn't set - the server refuses to boot. Explicitly setting the same value is the answer

The Hidden Assumption in Source Defaults: The Inverted Turn Ladder

Honcho's dialectic levels (minimal–max) ship with default turn counts of 1/5/2/4/10 - a non-monotonic ladder where low exceeds medium by 2.5×. Digging up the introducing commit revealed:

# TODO: Fill in appropriate values for each reasoning level

The author left a note admitting these are hand-tuned placeholders. And the assumption behind them is a different model + thinking per level (originally, low = a cheap fast model that searches 5 times). Once I unified the model, only turn counts remained - and the ladder was inverted. "Higher level = better" no longer held in the config.

Cap Philosophy: Insurance Should Sit Near the Model Limit

Here I locked in my principle: caps are not a savings tool; they're a runaway guard.

• Lowering the output token cap saves nothing - billing is on actual generation, and the moment output truncates you've paid full input+reasoning cost for zero visible answer. OpenAI, Anthropic, and Gemini docs all warn about this; Gemini literally says "lower the thinking level instead of setting a small max_output_tokens"
• haiku-5.5's official max output is 128K, but Honcho's config validation caps at le=100000 → 100000 across all levels (setting 128000 makes the server refuse to boot)
• Same logic for turns: validation allows le=50 (hard loop cap 100) → 50 across all levels. Modern models adaptively finish in 1–3 turns, so real spend is unchanged; 50 is purely a worst-case ceiling

The starting point is this belief: modern models choose appropriate reasoning and appropriate turn counts by themselves. If so, a cap shouldn't be a teaching tool - it should be a safety net that only catches true runaway.

The Effort Default Is Documented (But...)

I knew "haiku is best with default thinking," but what is the default exactly? The official docs answer: haiku-5.5 (and opus 5.5) default to effort medium - everything else defaults to high. Thinking is always on (adaptive).

Here's the fun contradiction: the docs claim "setting effort to the model's default is exactly the same as omitting it," yet in dialectic benchmarks explicit medium took 96.8s vs 20.9s omitted - a 4.5× gap. My hypothesis: adaptive means the model even chooses whether to think, and explicitly naming an effort fixes that dial. Either way, leaving it unspecified is doubly confirmed as the right call.

Final Architecture and Cost

Role	Model	Cost	Measured
Recording (deriver/summary/dream)	glm-5.3-flash + thinking max	flat-rate free	always-on
Daily recall (low)	haiku-5.5 (thinking unspecified)	$0.0135/call	15–21s
Deep research (max)	glm-5.3 + thinking MAX	free (flat-rate)	178s
Search embeddings	qwen3-embedding:4b	local Ollama free	-

Cheap and fast for daily recall, a free slow tier for "really dig in" requests. Turns 50 / output 100000 as shared insurance everywhere.

What Remains

• The 2-week experiment with per-turn auto-injection OFF - re-evaluate around Nov 13
• Honcho v3.0.3 → v3.0.9 (145 commits) upgrade - still postponed over build/migration concerns

Takeaways That Generalize

1. Benchmark models 10 e2e runs under real pipeline load. One-shot screening produces wrong answers
2. Give automation pipelines a lifespan. Surviving crons will eventually revert your config
3. Source defaults hide assumptions. A deployment that breaks those assumptions (e.g., unifying models) warrants re-auditing the defaults
4. Caps are insurance. Compute the premium (required tokens) and set them near the model limit - not dangerously low
5. Verify against the app's effective values. Env-variable overrides are powerless against dotenv override=True