An Honest freellmapi Review - What Happens When You Put 412 Free Models on a Real Workload

TL;DR

freellmapi is a self-hosted unified LLM gateway: 34 free-tier providers and a 412-model catalog behind one API key. I bolted it onto a real workload - my memory engine's recall circuit - and ran it for three days.
Verdict first: highly recommended as an experiment/backup lane, not recommended for latency-sensitive production. Where a paid small model finishes in 21 seconds, the free pool took 44–166 seconds.
That said, the core promise - automated free-model routing - held up: the smartest strategy completed 5/5 runs with full fact accuracy. The problems were speed and language consistency.
This is a hands-on review backed by a 5-configuration benchmark, a 141-model pool expansion, and strategy sweeps. Zero marketing copy.

What It Is

freellmapi aggregates free LLM providers (AI Horde, NVIDIA, Cloudflare Workers AI, Groq, Google, OpenRouter's free pool, plus dozens of community gateways) and serves them behind a single OpenAI-compatible /v1 endpoint. Core features:

A virtual auto model - send "auto" (or omit the model) and the gateway picks. Strategy presets: priority / balanced / smartest / fastest / reliable / custom / decision
A min_context_window floor - '32k/128k/512k/1m' presets that exclude small-context models from the pool entirely
Operations features: profiles, sticky sessions, response cache, quota observations, rate-limit cooldowns
Runs on a single sqlite file; dashboard settings changes are hot-reloaded (no restart)

I wired it into the recall circuit (dialectic) of my self-hosted memory engine (Honcho) - a task that reads ~16K tokens of stored memory and must answer precisely. Slow or sloppy answers get caught instantly.

Three Days, Five Configurations

Same protocol (a 3-fact recall question × 5–10 runs):

Setup	Routed model	Avg latency	Fact accuracy	Language
Paid baseline (haiku-5.5)	-	20.9s	30/30	Korean always
smartest + 1M floor	nemotron-3-ultra-550b	103.9s	5/5	2/5 English
fastest + 1M floor	gemini-3.1-flash-lite	11–15s	0/5 ⚠	Korean
smartest + 128k floor	dots-3-note-preview	85.8s	4.5/5	Korean
↑ after +141-model expansion	dots/nemotron/sensenova, different each call	113.6s	5/5	1/5 English

Three things to read from this.

1. The fastest strategy must not be used for recall. Answers arrived in 11 seconds - as generic platitudes ("working on quality improvement") with none of the memory facts (F1 targets, per-epoch scores). A fast wrong answer is not the goal. This isn't the gateway's fault - strategy choice is the user's responsibility - but anyone picking by strategy name will fall into this exact trap.

2. Expanding the pool makes things worse. Dropping the floor from 1M to 128k grew the candidate list, and for a while routing settled reliably on dots-3-note-preview, a decent model. Then I added keys for four more providers (+141 models) and routing got MORE erratic - a different model every call, average latency up from 85.8s to 113.6s. The root cause is the free pool's inherent state churn and latency variance; adding candidates doesn't fix it.

3. Catalog rank is not reality. The catalog showed intelligence-rank-17 GLM 5.3 and Qwen3.8 Max (1M context) at the top - but they weren't actually routable, because no live upstream keys existed for their platforms. "In the catalog" and "usable" are different things. Adding the keys fixes it, but the docs never surface this gap.

What I Liked

The free-pool aggregation actually works. I auto-signed-up on 5 providers, added keys, and 141 models joined instantly (profiles auto-include new models). Completion rate on the smartest runs was perfect. For "infrastructure to experiment with many models for free," it is exactly sufficient.
Hot-reload is thorough. I edited routing strategy and context floor directly in sqlite, and every call re-read the DB - no restarts. The operator experience of flipping switches is excellent.
Observability at the DB level. requests, decision_requests, provider_quota_state, rate_limit_cooldown tables - I could trace "why was this model chosen?" Self-hosting privilege, well used.
Client compatibility. Accepts OpenAI/Anthropic/Gemini headers; one unified key. A virtual auto id for clients (like Honcho) that require a non-empty model field is a thoughtful detail.

What Fell Short

High latency variance. Same strategy, same query: 43s to 166s. It's the free pool's nature, but expect disappointment if you want "reasonably consistent."
Language roulette. Routing to a different model per call meant answers flip-flopped between Korean and English. Fatal for a recall engine. (You can force language via a system-prompt profile - I just hadn't turned it on.)
Docs trust the source code. The Cloudflare key must be a composite account_id:api_token - documented only in the code. The dashboard is an SPA with no /openapi.json. Bring a shovel.
Sticky sessions hurt benchmark reproducibility. Models get pinned per message hash; I couldn't find an off switch.

Who Should Use It

Strongly recommended: free-model sweep benchmarking infrastructure, backup/fallback lanes, and answering "can this task run on free models?" It's why my environment could churn through 141 dialectic candidates in a day.
Not recommended: latency-sensitive production (user-facing recall/chat). The paid small model's 21s / 100% / fixed-language package beats free - re-confirmed today.
Conditional: slow-but-fine deep work (overnight batch summarization) pairs well with free pool + smartest - but force language with a profile first.

Operator Tips (All Personally Stepped On)

A 1M context floor narrows the pool until usage concentrates on few providers - 128k is the practical sweet spot
Settings can be edited directly in the sqlite settings table (hot-reloaded)
If the client key is lost, recover it from settings.unified_api_key in the DB
Don't trust catalog rank - add the key and make one real auto call to see what actually routes

Verdict

9/10 as experiment infrastructure, 4/10 as production recall. The project honestly solves the problem it claims ("free models behind one key"), and the depth of operations features (quota observations, cooldowns, profiles) exceeds expectations. But the physics of free pools - latency variance, model churn, language - is something no gateway can solve. freellmapi hands that weight to the user honestly. I liked the honesty.

Environment: self-hosted freellmapi (docker), 412-model catalog, 2026-10-29~31. Workload: Honcho dialectic recall over 16K memory context. Baseline: haiku-5.5 via OpenRouter.

★ PINION ★