How to Track LLM Costs in AI Agents

When an engineering team deploys a stateless chatbot, calculating costs is simple: multiply total prompt tokens by input pricing, total completion tokens by output pricing, and sum across users.

In autonomous AI agents, naive cost calculations completely break down.

Because agents plan, invoke tools, ingest large payloads, and self-correct across multi-turn loops, their token consumption is non-linear, multi-dimensional, and prone to explosive cost anomalies. A single bug in a retry loop or an oversized context accumulation can turn a $0.02 task into a $45.00 run in seconds.

This guide provides the complete engineering blueprint for tracking, attributing, and governing LLM costs in autonomous agent systems.

Why Agent Costs Explode: The Geometry of a Multi-Step Run

To understand why agent costs require per-span telemetry, consider how token context expands during execution:

Turn 1: [System Prompt (2,000) + User Goal (200)]                    = 2,200 tokens
Turn 2: [Turn 1 Context (2,200) + Tool Call 1 Result (4,500)]        = 6,700 tokens
Turn 3: [Turn 2 Context (6,700) + Tool Call 2 Result (8,000)]        = 14,700 tokens
Turn 4: [Turn 3 Context (14,700) + Self-Correction Retry (3,000)]    = 17,700 tokens
Turn 5: [Turn 4 Context (17,700) + Final Synthesis]                  = 18,200 tokens
─────────────────────────────────────────────────────────────────────────────
Total Tokens Processed across 5 turns: 59,500 prompt tokens!

Even though the final answer might only be 200 output tokens, the cumulative input tokens re-sent on every iteration create an exponential cost curve.

1. Per-Span Token Price Attribution Formula

Cost cannot be calculated as a global batch job at the end of the month. It must be computed deterministically at span ingestion time and attached to the trace.

The Granular Price Formula:

$$\text{Span Cost} = (T_{\text{prompt}} \times P_{\text{input}}) + (T_{\text{completion}} \times P_{\text{output}}) + (T_{\text{cache_read}} \times P_{\text{cache_read}}) + (T_{\text{cache_write}} \times P_{\text{cache_write}})$$

Where:

• $T_{\text{prompt}}$: Uncached input tokens
• $T_{\text{cache_read}}$: Tokens served from provider prefix cache (typically discounted by 75-90%)
• $T_{\text{completion}}$: Output generated tokens
• $P$: Model-specific rate cards per million tokens

2. Implementing Real-Time Span Cost Calculation in Python

Here is how Splyntra-compatible token cost calculation is implemented inside an OpenTelemetry Span Processor:

from dataclasses import dataclass
from typing import Dict

@dataclass
class ModelPricing:
    input_per_million: float
    output_per_million: float
    cache_read_per_million: float = 0.0
    cache_write_per_million: float = 0.0

# Master Model Pricing Table (Updated dynamically in Splyntra)
PRICING_TABLE: Dict[str, ModelPricing] = {
    "gpt-4o": ModelPricing(input_per_million=2.50, output_per_million=10.00, cache_read_per_million=1.25),
    "gpt-4o-mini": ModelPricing(input_per_million=0.15, output_per_million=0.60, cache_read_per_million=0.075),
    "claude-3-7-sonnet": ModelPricing(input_per_million=3.00, output_per_million=15.00, cache_read_per_million=0.30, cache_write_per_million=3.75),
    "claude-3-5-haiku": ModelPricing(input_per_million=0.80, output_per_million=4.00, cache_read_per_million=0.08),
}

def calculate_span_cost(
    model: str,
    prompt_tokens: int,
    completion_tokens: int,
    cached_tokens: int = 0
) -> float:
    """Calculates exact USD cost down to 6 decimal places."""
    pricing = PRICING_TABLE.get(model.lower())
    if not pricing:
        return 0.0  # Fallback for unknown/custom self-hosted models
        
    uncached_prompt = max(0, prompt_tokens - cached_tokens)
    
    cost = (
        (uncached_prompt / 1_000_000) * pricing.input_per_million
        + (cached_tokens / 1_000_000) * pricing.cache_read_per_million
        + (completion_tokens / 1_000_000) * pricing.output_per_million
    )
    return round(cost, 6)

3. Multi-Dimensional Cost Attribution

Once cost is calculated on every individual span, engineering teams can group and filter spend across critical operational dimensions:

┌─────────────────────────────────────────────────────────────┐
│             MULTI-DIMENSIONAL SPEND ATTRIBUTION             │
├─────────────────────────────────────────────────────────────┤
│ • By Workflow:  Support Agent ($4,210) | Research ($1,840)  │
│ • By Tenant:    Enterprise Customer A ($850) | Customer B   │
│ • By Step Name: Web Scraping Node ($1,200) | Summarizer     │
│ • By Model:     Claude 3.7 Sonnet ($5,100) | GPT-4o-mini    │
└─────────────────────────────────────────────────────────────┘

Essential Dimensions to Attach via OTel Baggage:

1. tenant.id / customer.id — For B2B SaaS usage-based customer billing and chargebacks.
2. agent.workflow_id — Comparing unit economics across different agent implementations.
3. agent.step_name — Pinpointing which node inside a graph consumes the largest share of token budget.

4. Implementing Hard FinOps Circuit Breakers

Alerting on high spend via Slack or email is reactive. By the time someone reads an alert on a Saturday morning, a looping agent could have burned thousands of dollars.

You must implement Runtime Spend Circuit Breakers:

class AgentRunBudgetManager:
    def __init__(self, max_run_cost_usd: float = 2.00, max_steps: int = 12):
        self.max_run_cost_usd = max_run_cost_usd
        self.max_steps = max_steps
        self.cumulative_cost = 0.0
        self.step_count = 0

    def record_step(self, span_cost: float):
        self.step_count += 1
        self.cumulative_cost += span_cost
        
        # Hard Ceiling 1: Cost limit exceeded
        if self.cumulative_cost > self.max_run_cost_usd:
            raise BudgetExceededException(
                f"Run aborted: Spend (${self.cumulative_cost:.4f}) exceeded budget cap of ${self.max_run_cost_usd:.2f}"
            )
            
        # Hard Ceiling 2: Step count limit exceeded (loop protection)
        if self.step_count > self.max_steps:
            raise LoopDetectedException(
                f"Run aborted: Execution steps ({self.step_count}) exceeded step limit of {self.max_steps}"
            )

5. FinOps Optimization Strategies for Agents

With granular span telemetry in place, optimization becomes an exact engineering process:

1. Context Compaction & Sliding Windows: Don't pass raw 10,000-token tool outputs into the next reasoning step. Summarize or extract JSON schemas before inserting into history.
2. Anthropic / OpenAI Prompt Caching: Place static system instructions and tool definitions at the very beginning of the prompt to maximize prefix cache hits ($90%$ savings).
3. Model Downshifting by Step: Use high-reasoning frontier models (claude-3-7-sonnet, gpt-4o) exclusively for planning, and downshift to lightweight models (gpt-4o-mini, claude-3-5-haiku) for structured extraction and formatting.
4. Tool Result Truncation: Enforce strict character limits on web scrapers and database queries before the results are injected into prompt context.

Next Steps & Related Technical Guides

• AI Agent Monitoring: Metrics You Should Track — Golden signals for agent health and FinOps.
• AI Agent Observability: What You Need to Monitor in Production — Pillar guide to production telemetry.
• How to Monitor AI Agents with OpenTelemetry — Instrumenting your agents with OTel SDKs.