Privacy choices

This site may send the owner a minimal page-visit notice with page, browser, device type, and resolution. If you accept analytics, it can also remember an anonymous visit/session ID and collect richer engagement signals.

Read the privacy notice
AlexPersonal Operating System
Alex OS Online
Back to Projects
LinkedIn
ProjectJuly 13, 2026

LLM Cost & Behavior Tracing

Every LLM call across the platform is traced — model, tokens, prompt, answer and USD cost — and surfaced in an admin dashboard, so AI spend and behavior are never a black box.

Category
ai_rag
Tech stack
FastAPI, Python, Claude API, OpenAI API
Metrics
100% of LLM calls traced, Per-call USD cost attribution
Role
Design + build

Content

A tracing layer wraps every model call in Alex OS (Claude and OpenAI) and writes a structured record per call: model, token counts, prompt, response and computed USD cost. Traces are rendered in a dedicated LLM tab on the admin analytics page, making it possible to audit any AI feature end-to-end and spot cost regressions immediately.

Problem

AI features quietly accumulate API costs and it is hard to answer "what did that answer cost, and what prompt produced it?" after the fact.

Outcome

Complete per-call audit trail of AI usage and spend across the whole platform.

aiobservabilitycostsystem

Explore with Alex OS

Every LLM call on the platform is logged with its model, token count, prompt, response, and USD cost — so AI spend is always auditable.

LLM Cost & Behavior Tracing

A tracing layer wraps every Claude and OpenAI call across the platform, writing a structured record per call — model, tokens, prompt, response, and computed USD cost — surfaced in a dedicated admin dashboard tab.

The Problem Worth Solving

Design Decision: Wrap at the Call Layer

Rather than relying on provider dashboards or aggregate billing exports, the tracing layer intercepts each model call in-process (FastAPI + Python), computes USD cost from token counts at write time, and stores a complete structured record immediately. Tradeoff: slight overhead per call, in exchange for full fidelity and no dependency on external tooling.

Verified Outcomes

100% of LLM calls across the platform are traced. Per-call USD cost attribution is available for every AI feature — Claude and OpenAI — making cost regressions and prompt behavior visible the moment they occur, not after a billing cycle.