Benchmarking 6 LLM backends for a self-hosted AI memory pipeline: does fast, stable, nearly free exist?

TL;DR

• Swapped the LLM backend of my self-hosted AI memory pipeline (Honcho dialectic) six ways and ran the same query 10 times per backend
• Baseline (flat-rate zai plan, glm-5.3): 30.4s average, 100% fact accuracy
• Of five challengers, two won on speed, but only Claude haiku-5.5 also matched accuracy: 20.9s average, 30/30 facts, $0.013 per call
• Conclusion: "fast, stable, nearly free" did exist. It just required redefining "nearly free" as 1.3 cents per call
• All six experiments ran unattended on an n8n watcher-gate chain; the only human intervention was judgment

Background: what are we measuring

Honcho accumulates conversation representations, then on each query stuffs a 16K+ token memory context into an LLM to synthesize "what is this user working on". The problem: this call eats 20-30 seconds per turn. Injection is off for now, but I still wanted hard numbers on whether a cheaper, faster backend could replace it.

Key trap: this benchmark is not a one-shot prompt. It is a synthesis task over 16K of context, so short-prompt leaderboards do not transfer. A free model that answered a bare prompt in 3.8s averaged 86s inside the pipeline.

Method

• Same query, 10 runs: "What is Myeongchul currently worried about in his g2b parser project?"
• Scoring: 3 facts (F1 target 0.92 / hallucinated-documents bottleneck / current best 0.74-0.75) appearing in the answer
• Adoption bar: p50 under 25s + fact recall at least 80%
• Automation: a serial chain of five n8n watcher-gate workflows (5-min poll, gate, .env swap, 10-run measurement, restore, messenger report)

Results

Backend	Model	Latency (avg/p50)	Facts	Cost per 10 runs	Verdict
zai direct (incumbent)	glm-5.3 + low	30.4s	100%	flat rate	baseline
OpenRouter free	flint-v1.0:free	86.3s, 2 timeouts	71%	$0	reject
OpenRouter ultra-cheap paid	mistral-nemo	5.0s	0%	$0.003	reject
Local free gateway	nemotron-3-ultra-550b	97.2s, 5 timeouts	50%	$0	reject
Frontier fast	qwen3.5-flash	29.9s / 23.9s	66.7%	$0.068	reject
Frontier fast, direct	claude-haiku-5.5	20.9s / 20.05s	100% (30/30)	$0.135	winner
Frontier fast, auto-cache	gpt-6-luna (low)	13.7s / 10.8s	60%	$0.025	reject (quality)

Three takeaways

1. Speed-quality cliff: a 3B-class model (nemo) was 6x faster but recalled zero facts and answered English to Korean questions. Memory synthesis needs mid-tier-or-better intelligence
2. The real price of free is the queue: free backends mostly suffered latency blowups and timeouts; even the trusted top-tier candidates on my local free gateway all failed probes (latency spikes, 502s, hard refusals, thinking-leak)
3. Honest models fail cold screenings: haiku-5.5 truthfully said "I have no such information" to a context-free screening prompt and got cut. But production runs carry the 16K context. A direct finals run scored 30/30, and it even self-identified that some facts were verbatim prompt text and separated sources. The screening was a false negative

Four traps the unattended pipeline learned

1. Gate agents disobey "answer SKIP only" and append explanations. The report filter must use notContains, not notEquals
2. A completion flag created at start cannot distinguish running from done. Separate running lock files
3. A re-firing gate's "restore" step killed a running experiment mid-measurement (real incident). Restore only after checking every sibling lock
4. Never pick models from one-shot screening. Always 10 e2e runs under real pipeline load

Conclusion

haiku-5.5 was the only backend meeting both bars: 1.4x faster, equal accuracy, zero timeouts, no thinking leakage. But the cost structure differs: the flat-rate plan has zero marginal cost, while haiku-5.5 re-uploads the 16K context every call at about $0.013 each, roughly $40/month at 100 turns a day.

So the final call: keep the flat-rate backend as the dormant setting while injection is off, and preserve a one-line .env switch to haiku-5.5 for the moment speed matters. The value of a benchmark is not just who wins. It is knowing exactly what the switch costs.

Postscript: the caching follow-ups

OpenAI-family models (gpt-6-luna) get automatic prompt caching with just a provider pin: telemetry showed 40-76% of input tokens served from cache, pushing p50 to 10.8s and cost to $0.0025 per call. Fact recall collapsed to 60%, so still not adopted. The Anthropic path fared worse: the cache marker only lands on the system prompt, leaving the tool-loop resends (where most tokens live) uncached - zero savings without a marker-placement patch. Summary: haiku-5.5 owns quality, luna owns speed and cost, and fast+stable+nearly-free exists, just not with quality on top. I also tried switching haiku's thinking off: 20.9s became 13.1s, and accuracy collapsed from 100% to 62%. Those 7 seconds were the quality.