The Evidence Layer Costs Three Milliseconds

The rig

Three lanes on one machine, all answering the identical minimal chat completion. Lane one is a mock OpenAI-compatible upstream that answers in about half a millisecond: the floor. With the model taken out of the picture, every measured millisecond belongs to the proxy in front of it. Lane two is LiteLLM, the most widely deployed open-source LLM proxy, pointed at that mock. Lane three is the Scrutari gateway, pointed at the same mock.

Our lane ran the real release binary against a real Postgres with our production migrations: real TLS termination, a real API key minted by our own operator tool and verified with Argon2id, the production PHI scanner, quota reservation, catalog routing, per-tenant credential injection, and the signed audit row on every call. Three substitutions, all behind a compile-time development feature that release builds physically exclude: keys read from local files instead of Key Vault, a static provider credential, and the provider base URL steered at the mock. The hot path measured is the hot path customers hit.

The driver was oha, firing a fixed request rate for 30 to 45 seconds over 32 connections and recording every request's latency. p50 is the median; p99 is the worst one percent.

The first run was the bad news

At 50 requests per second we were fine but slow: a 33.8 ms median against LiteLLM's 9.9. At 150 we collapsed. The gateway served 66 requests a second and queued the rest into multi-second latencies while LiteLLM did not blink.

Isolating it took one extra pass: an authenticated call to a trivial endpoint showed the same latency floor, so the cost was not the AI pipeline. It was authentication. We verify every API key with Argon2id at memory-hard parameters, which burns roughly 20 to 50 ms of CPU per request. That is a deliberate security posture, and a comment in our own auth code predicted the consequence almost to the number: a hard cap of a few dozen authenticated requests per second per pod. The benchmark just made the comment real.

Removing the ceiling without giving up the property

The fix is a verified-credential cache, and the design matters more than the code. On a successful Argon2id verify, we cache a SHA-256 digest of the presented credential mapped to the verified key's id plus a fingerprint of the stored hash, for 30 seconds. On the next request with the same key, the digest matches and Argon2 is skipped.

Four properties make that safe. The cache key is derived from the credential itself, which carries over 130 bits of entropy, so forging an entry requires the very secret being proven. Only successful verifies are cached, so a wrong key pays full Argon2 on every attempt and brute-force cost is unchanged. Each entry binds to the exact key row and hash it was verified against, so a re-minted or rotated key falls back to a full verify. And revocation latency does not move: the key row itself, where revocation, expiry, and tenant state live, was already consulted through a 30-second cache on every request, and that path is untouched.

Same passes, cache in place

The ceiling is gone, the median dropped eight-fold, and at 300 requests per second the roles reversed: the Python proxy saturated while the gateway held three-millisecond medians with the signed audit record still written on every call.

A second machine tells the tail-latency truth. The harness reproduced on an Apple-silicon laptop from a clean clone (which shook out four hidden assumptions in our setup along the way, all now fixed in the repo), and on quiet hardware the p99 stays under five milliseconds at every rate, signing included. The container's fatter tails above were shared-CPU neighborhood noise, not the gateway.

One fairness note. LiteLLM is a fine tool doing a different job: it is a routing convenience layer, and in these runs it was not signing anything, scanning anything, or enforcing quotas. That is exactly the point of the comparison. The premise against an evidence-bearing gateway is that provable audit trails must cost something painful. Measured: the whole evidence layer, signing included, rides inside about three milliseconds of median overhead. The expensive part of our stack was never the evidence. It was password hashing, and that was fixable in an afternoon.

What we did not measure

Streaming responses, provider failover under fault injection, the MCP proxy, and response caching each deserve their own pass, and streaming is next since most real chat traffic streams. These numbers also come from a modest shared container; absolute figures will differ on your hardware, which is why the shape of the curves, not the digits, is the claim.

Reproduce it

The harness is a directory in our workspace: a key generator, a seed script, a mock upstream, and a matrix runner that prints the tables above. Everything runs on localhost with throwaway keys against a scratch database, and the development-only seams it relies on are compiled out of release builds. If you run a gateway, yours deserves the same afternoon: point a constant-rate load tool at it, find your ceiling before your customers do, and publish what you find.

And the evidence the three milliseconds buys is public and checkable: the anchor transparency feed publishes the signed roots, and the open-source verifier proves any exported pack against them on any machine, ours included, without talking to us.