AI quality gate
Production LLMOps Gateway: Evaluation CI/CD and Latency Optimization
Sits in front of every AI call. A question it has already answered comes back from memory in milliseconds, an easy one goes to a cheap model and a hard one to a strong model, and every answer is recorded with what it cost. Before a change ships, a fixed set of questions is asked again, and the change is blocked if the answers got worse.
The problem it solves
The model bill grows with every request, tail latency is whatever the provider's queue is that day, and nobody can show that last week's prompt change did not make the answers worse. A semantic cache saves money only if it never serves the answer to a different question, and a quality bar is real only if a failing change cannot merge.
What was measured
- Measured
- Target
0 / 78
look-alike questions served a wrong cached answer at 0.96, 23 October 2026
measured79.8%
of list-price model spend avoided on a 250-question replay
measured264
hermetic tests: no network, no Redis, no key
measured$0.00
marginal cost per request on the free stack
measured
How it worksfor engineers and technical reviewers
In technical terms
A FastAPI gateway in front of every model call, with an OpenAI-compatible route any client can point at. Repeated questions come back from a Redis 8 vector cache without a model call, new ones are scored for complexity and sent to a light or a heavy chain of models with failover, and every step is a span of one trace with tokens and dollars, mirrored to Langfuse. A GitHub Actions workflow answers a golden set with the real pipeline, has DeepEval and Ragas judge the answers, and fails the pull request when the hallucination rate passes 5% or faithfulness drops below 0.95. Runs at zero cost on free models.
Architecture
A guard normalizes every request, neutralizes chat-template tokens and spoofed tags, flags injection phrasings, and applies input and per-tier output token ceilings. The semantic cache checks an exact key first, then embeds the question with bge-small on the CPU and asks Redis for the three nearest cached questions in the same namespace (prompt version, corpus fingerprint, embedding model), through an HNSW index with cosine distance. A neighbour above the 0.96 threshold is served only if a contrast guard finds no contradiction between the two questions on a number, a code or an opposite term; measured, that is what keeps the domestic meal limit from being served for the international one. A miss goes through hybrid retrieval (dense and BM25 fused by reciprocal rank) and a complexity router built from named signals, which costs no model call and explains itself in the trace. The chosen tier's chain answers, and a 429, a timeout, a withdrawn model, a refusal or an uncited answer hands the request to the next model, then to the other tier; a bad key stops it. Answers stream as Server-Sent Events, and every model call is priced twice: what it cost, and what the same tokens cost at Claude Haiku 5.5 or Claude Opus 5.5 list prices. Each step is a span mirrored to Langfuse as a typed observation, generations with usage and cost details. The quality gate runs a deterministic retrieval check on every pull request and, when the answers can have changed, DeepEval and Ragas as LLM-as-a-judge on 37 golden cases, then fails the build on the thresholds and on any metric that drops below the accepted baseline.
Stack
- Python 3.12
- FastAPI
- Pydantic v2
- Server-Sent Events
- Redis 8 vector search
- fastembed (bge-small)
- Langfuse
- DeepEval
- Ragas
- GitHub Actions
- OpenRouter
- Docker
- GitHub Pages
Proof points
- Threshold trade-off measured on 23 October 2026 with the real embedding model: two rephrasings of one question have a median similarity of 0.889, while two look-alike questions with different answers reach 0.961. At the 0.96 threshold the cache alone would serve 1 of 78 look-alikes the wrong answer; with the contrast guard, none, and the guard blocked none of the 78 true rephrasings.
- On a 250-question help-desk replay through the real pipeline, the cache answered 199 requests (172 exact repeats, 27 semantic matches) with no wrong answer, avoiding 79.8% of the model spend at list price. The same replay against Redis 8 gave the same hits, with lookups at 2.4 ms (exact) and 3.6 ms (semantic, embedding included) at the median.
- Retrieval check with no model, run on every pull request: the passage that holds the answer was in the top four for all 30 answerable golden questions, with a context precision of 0.975.
- The production image runs in a 512 MB container at 348 MiB with the embedding model baked in, connects to Redis 8, creates its vector index, and served a semantic hit in 15 ms end to end.
- 264 hermetic tests: no network, no Redis, no key. The real OpenAI and Anthropic SDK adapters run against a mock transport, the Langfuse SDK exports to memory, DeepEval's metrics run through the gateway's own judge, and the project page is held to its recordings by tests. One more test runs the cache against a real Redis 8.
Request pipeline
- 01
Guard
Unicode folded, template tokens and spoofed tags neutralized, injections flagged, token ceilings
- 02
Cache
Exact key, then Redis KNN above 0.96 cosine, refused when the questions contradict each other
- 03
Route
Named signals and retrieval spread pick the light or the heavy chain; no model call
- 04branch
Fallback
429, timeout, withdrawn model or uncited answer: next model, then the other tier
- 05
Trace
One span per step to Langfuse: tokens, actual cost, list-price cost, p50/p95/p99
- 06
Gate
DeepEval and Ragas on a golden set; a pull request fails above 5% hallucination