Know exactly which service spent your LLM budget — down to the token.
The moment more than one service starts calling an LLM provider, the bill stops making sense. Azure OpenAI and Vertex AI invoice you per deployment and per model — not per team, not per feature, and definitely not per microservice. You get one large number at the end of the month and no defensible way to split it.
On Kubernetes this gets worse rather than better. A dozen pods across several namespaces share the same provider credentials, so from the provider's side every request looks identical. The usual workarounds — one API key per service, or asking each team to self-report — either multiply your secret-management burden or produce numbers nobody trusts.
Unified Model Gateway (UMG) solves this at the only place where the information still exists: the egress path itself.
UMG is a pass-through proxy that sits between your services and your LLM providers. Calling services keep their existing SDK — the OpenAI/LangChain client, or Google's google-genai client — and change exactly one thing: the base_url. No request rewriting, no bespoke client library, no migration project.
Because every call now passes through one process, UMG can see and record what the provider bill can't: which service made the request, which logical model it asked for, which physical deployment served it, and how many tokens it burned doing so.
Every request carries an X-Caller-Service header, plus tenant headers for organization, account, and user. Missing headers default to unknown and raise a warning rather than blocking traffic — so instrumenting your fleet is incremental, and non-compliant callers show up on a dashboard instead of causing an outage.
Services request logical aliases like chat-fast or embed-doc-large, never a physical deployment name. UMG resolves the alias against a per-cluster model list, walks an ordered fallback chain, and injects the right provider credential. Swapping the backing model becomes a config change, not a fleet-wide code change.
UMG reads the usage block off the upstream response on a streaming-safe tee — accumulating SSE lines without buffering the full body, so metering never adds latency or breaks streaming. Azure's usage and Vertex's usageMetadata are both normalised to the same prompt/completion/total shape.
Token counters are emitted through OpenTelemetry to a collector, then scraped by Prometheus. Each measurement is labelled with the caller service, the alias, the physical deployment, the provider, the model, and the capacity class — which means the split you could never get from your provider bill is now a single query.
Want to know which service is responsible for this month's Azure spend? Sum umg_total_tokens by caller_service. Want to know whether a team's costs come from long prompts or long generations? Compare umg_prompt_tokens against umg_completion_tokens — they price differently, and the difference tells you whether to attack context size or output length.
Multiply the token counts by your negotiated per-model rates and you have per-service, per-model cost. Not an estimate — a measurement.
umg_total_tokens, umg_prompt_tokens, and umg_completion_tokens — each labelled by alias, deployment, provider, caller service, model, and capacity class. Cheap to aggregate, safe to keep at fleet scale.
umg_http_requests_total with matching labels, plus latency and time-to-first-byte histograms. Fallback invocations, retries, circuit-breaker skips, and mid-stream truncations are all counted per deployment — so a provider degrading is visible before it becomes an incident.
Organization, account, and user identifiers are deliberately kept out of Prometheus labels — unbounded cardinality is how monitoring stacks fall over. They ride instead on a dedicated structured JSON log line, correlated by request ID, so per-tenant chargeback stays available without endangering the metrics backend.
Putting every LLM call through one hop only works if that hop is more reliable than what it replaced. Each alias carries an ordered fallback chain, per-deployment retry budgets, and explicit timeouts. A per-replica circuit breaker takes sick deployments out of rotation and probes them back in.
The error taxonomy is deliberate: transient failures retry then fall over and count against the breaker, while caller mistakes surface verbatim without retries and without penalising the deployment. A bad request from one team never trips a breaker that would affect another.
UMG runs one instance per environment-and-cluster pair, deployed with the same shared Helm chart as any other internal service. Alias names stay identical across clusters while the fallback chains differ, so each cluster prefers its local cloud and every service's code stays portable between them.
It currently fronts Azure OpenAI, Azure AI Foundry, and Vertex AI deployments across development and production clusters in both Azure and GCP, serving traffic from multiple internal services.
We can help you get per-service attribution in place — or build the gateway layer with your team.
Triops Info