LLM Cost & Behavior Tracing
Every LLM call across the platform is traced — model, tokens, prompt, answer and USD cost — and surfaced in an admin dashboard, so AI spend and behavior are never a black box.
- Category
- AI & RAG
- Tech stack
- FastAPI
- Python
- Claude API
- OpenAI API
- Metrics
- 100% of LLM calls traced
- Per-call USD cost attribution
- Role
- Design + build
Content
A tracing layer wraps every model call in Alex OS (Claude and OpenAI) and writes a structured record per call: model, token counts, prompt, response and computed USD cost. Traces are rendered in a dedicated LLM tab on the admin analytics page, making it possible to audit any AI feature end-to-end and spot cost regressions immediately.
Problem
AI features quietly accumulate API costs and it is hard to answer "what did that answer cost, and what prompt produced it?" after the fact.
Decisions
Traces are written to an append-only JSONL file rather than a database table: no migration to run, no write load on the CMS database, and the whole history stays greppable. The wrapping happens at the client layer instead of inside each feature, so every new AI feature is traced from its first call whether or not its author thought about observability. USD cost is computed per call and stored in the trace itself, so spend reports read straight from the record instead of being recomputed later.
Limitations
Traces store prompts and answers verbatim — that is exactly what makes them useful for auditing behavior, and it also means the trace log is itself sensitive and stays behind the admin.
Outcome
Complete per-call audit trail of AI usage and spend across the whole platform.
How this was built
Built with AI assistance (Claude Code). The architecture, the decisions and the trade-offs are mine; a lot of the implementation was written in pair with the model. I work this way deliberately — it's why one person can run a system this size — and the reasoning behind every call here is documented above.