LLM Cost & Behavior Tracing
Every LLM call across the platform is traced — model, tokens, prompt, answer and USD cost — and surfaced in an admin dashboard, so AI spend and behavior are never a black box.
The problem
AI features quietly accumulate API costs and it is hard to answer "what did that answer cost, and what prompt produced it?" after the fact.
My role
Design + build
LLM Cost & Behavior Tracing
A tracing layer wraps every model call in Alex OS (Claude and OpenAI) and writes a structured record per call: model, token counts, prompt, response and computed USD cost. Traces are rendered in a dedicated LLM tab on the admin analytics page, making it possible to audit any AI feature end-to-end and spot cost regressions immediately.
Key decisions
Traces are written to an append-only JSONL file rather than a database table: no migration to run, no write load on the CMS database, and the whole history stays greppable. The wrapping happens at the client layer instead of inside each feature, so every new AI feature is traced from its first call whether or not its author thought about observability. USD cost is computed per call and stored in the trace itself, so spend reports read straight from the record instead of being recomputed later.
Outcome
Complete per-call audit trail of AI usage and spend across the whole platform.
- 100% of LLM calls traced
- Per-call USD cost attribution
Limitations
Traces store prompts and answers verbatim — that is exactly what makes them useful for auditing behavior, and it also means the trace log is itself sensitive and stays behind the admin.