Sample LLM Cost & Latency Audit

A worked example of the audit deliverable. The audited subject is my own open-source project, StackIntercept, so every finding and number below is independently verifiable from public source. This is a format-and-method sample, not a claim about any client's production spend.

What this document is

The paid LLM Cost & Latency Audit produces a short report: where calls and money go, which requests can safely avoid an upstream call, what to change first, and how to measure the result. This sample applies that exact structure to one repository so you can judge the depth and honesty of the work before engaging.

All measurements here come from a local script against a mock upstream. They are reproducible without API keys. No provider credentials, client data, or production traffic were used.

1. Scope and subject

2. Where calls and latency go

The dominant cost driver in an LLM gateway is repeated identical or near-identical calls that still reach the provider. The local benchmark isolates each path:

PathMedian latencyRelative to cold missUpstream call
Cold miss, no cache53.6 ms1.00xYes
Exact cache hit2.3 ms0.04xNo
Streaming exact cache hit1.8 ms0.03xNo
Semantic cache hit50.1 ms0.93xNo
Routed fallback request53.6 ms1.00xYes (fallback provider)

Read the last column first. An exact cache hit removes the upstream call entirely — that is the mechanism that reduces provider spend, not the raw latency number. The latency figures are meaningful as a measure of the local overhead each path adds.

3. What is safe to cache

Blind caching changes answers. The audit checks that the proxy excludes non-deterministic or stateful requests:

These rules are documented in docs/design.md. A real audit verifies them against your traffic, not just against docs.

4. Findings and priorities

F1 — Exact caching is on by default. Default config is CacheMode::Exact with a 3600 s TTL and 20,000 entries. For workloads with repeated deterministic prompts, this is the first saving to measure. Priority: measure hit rate before changing anything.

F2 — Semantic caching is opt-in and carries most of its own overhead. A semantic hit measured 50.1 ms versus 2.3 ms for exact. It avoids the upstream call but adds local embedding work. Enable it only where phrasing varies but surrounding context is stable; the implementation scopes similarity to a context bucket rather than the whole store. Priority: second, after exact-cache hit rate is known.

F3 — Model routing is off by default (allow_model_rewrite: false) and is blocked for tools, structured output, multimodal, non-deterministic, and security/financial-style prompts. Cost savings from routing are real only where the cheaper model is correct for the task. Priority: evaluate per workload, never globally.

F4 — Reactive failover is on for 429/500/502/503/504 before any body bytes are sent. This protects availability; it is not a spend control. Priority: verify it does not mask an upstream quota problem by silently moving cost to a fallback provider.

F5 — Stream completion must gate cache insertion. An HTTP 200 on an SSE stream is not proof of a complete response. Caching a truncated stream replays a partial answer. This proxy tracks stream completion separately before inserting into the cache. Priority: correctness check in any proxy audit.

5. How to measure the result

Every upstream call avoided is a call not billed. Track the transparent response headers the proxy emits (x-stack-intercept: hit|miss and the routing headers) and compare:

hit rate = cache hits / total eligible requests
upstream calls avoided = cache hits
cost change = avoided calls x your provider's per-call price
latency change = compare hit-path vs miss-path medians

The report gives you this measurement harness so savings are attributed to a specific change, not guessed.

6. Limits, stated plainly

7. Deliverable format

A paid audit returns: (1) spend and call-driver analysis, (2) a safe-to-cache map for your request types, (3) prioritized findings like F1–F5 above, (4) a measurement harness, (5) a 30-minute walkthrough. If you hire me to implement, the audit fee is credited toward the project.

See the service and pricing: Contract Engineering Services. Source and reproduction steps: github.com/sidsri14/stack-intercept — run python benchmark.py after cargo build.