Sample LLM Cost & Latency Audit
A worked example of the audit deliverable. The audited subject is my own open-source project, StackIntercept, so every finding and number below is independently verifiable from public source. This is a format-and-method sample, not a claim about any client's production spend.
What this document is
The paid LLM Cost & Latency Audit produces a short report: where calls and money go, which requests can safely avoid an upstream call, what to change first, and how to measure the result. This sample applies that exact structure to one repository so you can judge the depth and honesty of the work before engaging.
All measurements here come from a local script against a mock upstream. They are reproducible without API keys. No provider credentials, client data, or production traffic were used.
1. Scope and subject
- Subject: StackIntercept — a local OpenAI-compatible Rust proxy for repeated LLM calls.
- Question: Where does latency come from, and which requests can avoid an upstream call without changing answer correctness?
- Method: config review + local benchmark against a mock upstream with a fixed 50 ms delay.
- Environment: Windows, Rust 1.94.0, dev build, measured 2026-07-15.
2. Where calls and latency go
The dominant cost driver in an LLM gateway is repeated identical or near-identical calls that still reach the provider. The local benchmark isolates each path:
| Path | Median latency | Relative to cold miss | Upstream call |
|---|---|---|---|
| Cold miss, no cache | 53.6 ms | 1.00x | Yes |
| Exact cache hit | 2.3 ms | 0.04x | No |
| Streaming exact cache hit | 1.8 ms | 0.03x | No |
| Semantic cache hit | 50.1 ms | 0.93x | No |
| Routed fallback request | 53.6 ms | 1.00x | Yes (fallback provider) |
Read the last column first. An exact cache hit removes the upstream call entirely — that is the mechanism that reduces provider spend, not the raw latency number. The latency figures are meaningful as a measure of the local overhead each path adds.
3. What is safe to cache
Blind caching changes answers. The audit checks that the proxy excludes non-deterministic or stateful requests:
- Requests with
temperature > 0are not cached. - Tool / function-calling and structured-output requests are not cached.
cache_control: no_storeis respected.- Non-2xx upstream responses are never cached.
- Cache keys include provider and routing namespace, so a routed answer cannot be replayed as a passthrough answer.
These rules are documented in docs/design.md. A real audit verifies them against your traffic, not just against docs.
4. Findings and priorities
F1 — Exact caching is on by default. Default config is CacheMode::Exact with a 3600 s TTL and 20,000 entries. For workloads with repeated deterministic prompts, this is the first saving to measure. Priority: measure hit rate before changing anything.
F2 — Semantic caching is opt-in and carries most of its own overhead. A semantic hit measured 50.1 ms versus 2.3 ms for exact. It avoids the upstream call but adds local embedding work. Enable it only where phrasing varies but surrounding context is stable; the implementation scopes similarity to a context bucket rather than the whole store. Priority: second, after exact-cache hit rate is known.
F3 — Model routing is off by default (allow_model_rewrite: false) and is blocked for tools, structured output, multimodal, non-deterministic, and security/financial-style prompts. Cost savings from routing are real only where the cheaper model is correct for the task. Priority: evaluate per workload, never globally.
F4 — Reactive failover is on for 429/500/502/503/504 before any body bytes are sent. This protects availability; it is not a spend control. Priority: verify it does not mask an upstream quota problem by silently moving cost to a fallback provider.
F5 — Stream completion must gate cache insertion. An HTTP 200 on an SSE stream is not proof of a complete response. Caching a truncated stream replays a partial answer. This proxy tracks stream completion separately before inserting into the cache. Priority: correctness check in any proxy audit.
5. How to measure the result
Every upstream call avoided is a call not billed. Track the transparent response headers the proxy emits (x-stack-intercept: hit|miss and the routing headers) and compare:
hit rate = cache hits / total eligible requests
upstream calls avoided = cache hits
cost change = avoided calls x your provider's per-call price
latency change = compare hit-path vs miss-path medians
The report gives you this measurement harness so savings are attributed to a specific change, not guessed.
6. Limits, stated plainly
- These numbers come from a mock upstream on one machine. They are not production latency or cost figures.
- Savings depend entirely on your real request mix. A workload with no repeated deterministic calls saves little from exact caching.
- This project is not a full gateway: it has no built-in rate limiting, spend caps, health-based load balancing, or circuit breaker.
- I do not promise a fixed percentage reduction. I measure your traffic and report what the changes would actually do.
7. Deliverable format
A paid audit returns: (1) spend and call-driver analysis, (2) a safe-to-cache map for your request types, (3) prioritized findings like F1–F5 above, (4) a measurement harness, (5) a 30-minute walkthrough. If you hire me to implement, the audit fee is credited toward the project.
See the service and pricing: Contract Engineering Services. Source and reproduction steps: github.com/sidsri14/stack-intercept — run python benchmark.py after cargo build.