AI Agent Monitoring: Metrics You Should Track

Traditional SRE relies on the Four Golden Signals: Latency, Traffic, Errors, and Saturation.

While these signals remain necessary for backend servers, they are completely inadequate for autonomous AI agents. An agent can have 100% HTTP 200 responses, sub-second API latencies, zero server errors, and yet:

• Be stuck in an infinite reasoning loop.
• Spend $40.00 on a single customer support query.
• Hallucinate tool parameters on every single turn.
• Violate security policies without throwing an unhandled exception.

To run agents reliably, you need a new operational telemetry framework. Here are the 12 Golden Metrics for AI Agent Observability in production.

The 12 Golden Signals of Agent Telemetry

┌────────────────────────────────────────────────────────────────────────┐
│                   THE 12 GOLDEN SIGNALS FOR AI AGENTS                  │
├─────────────────┬───────────────────┬──────────────────────────────────┤
│ Reliability     │ FinOps & Scale    │ Quality & Security               │
│ 1. Goal Compl.  │ 5. Cost / Task    │ 9. Trace Risk Score              │
│ 2. Step Count   │ 6. Context Satur. │ 10. Hallucination Rate           │
│ 3. Loop Entropy │ 7. Token Velocity │ 11. HITL Escalation Rate         │
│ 4. Tool Fail %  │ 8. Cache Savings  │ 12. State Drift Rate             │
└─────────────────┴───────────────────┴──────────────────────────────────┘

1. Goal Completion Rate (GCR)

The north-star metric for agent efficacy.

• Definition: The percentage of agent runs that successfully fulfill the user's explicit objective.
• Formula: $$\text{GCR} = \frac{\text{Successful Runs}}{\text{Total Completed Runs}} \times 100$$
• How to Track:
  • Deterministic evaluation (e.g., did the agent insert the expected row or return a verified order ID?).
  • Asynchronous LLM-as-a-judge scorers run against trace outputs in CI or batch sampling.

2. Step Count Distribution ($p_{50}, p_{90}, p_{99}$)

The pulse of agent reasoning efficiency.

• Definition: The number of reasoning, planning, and tool execution steps required to finish a workflow.
• Target Profile: A tight clustering around $p_{50}=3$, $p_{90}=6$, with a hard ceiling at $p_{99} \le 12$.
• Why It Matters: When a prompt change or model update causes $p_{95}$ step counts to jump from 5 to 18, your agent is struggling to converge on answers.

Step Count Distribution:
45% │ █████████ (3-4 steps: Optimal)
35% │ ███████   (5-7 steps: Acceptable)
15% │ ███       (8-12 steps: Degraded)
 5% │ █         (>15 steps: Potential Infinite Loop ⚠)

3. Loop Entropy & Repetition Coefficient

The early warning system for stuck agents.

• Definition: A mathematical measure of action diversity within an agent run.
• Formula: $$\text{Entropy} = -\sum_{i=1}^{K} P(\text{tool}_i) \log_2 P(\text{tool}_i)$$
• Interpretation: If an agent calls search_database 8 times in a row with identical parameters, entropy drops to near zero.
• Action: Alert and trigger automated run cancellation when an agent executes $\ge 3$ identical tool calls consecutively.

4. Tool Invocation Failure Rate (TIFR) & Retry Density

Measuring the stability of agent tool interactions.

• Definition: The ratio of failed tool executions (HTTP errors, invalid JSON schemas, timeouts) to total tool calls.
• Key Dimension: Track Schema Conformance Rate separately from Upstream API Availability. If schema conformance drops below 98%, your tool documentation in the system prompt is ambiguous.

5. Cost Per Completed Task (CPS)

The unit economic health metric.

• Definition: The total dollar spend across all LLM spans, embeddings, and tool calls divided by successful goal completions.
• Formula: $$\text{CPS} = \frac{\sum \text{Span Costs}}{\text{Successful Tasks}}$$
• Why It Matters: Tracking cost per million tokens is a vanity metric. A smaller model that takes 20 looping steps to solve a problem is often more expensive than a frontier model that solves it in 2 steps.

6. Context Window Saturation Ratio

Preventing context window degradation and latency spikes.

• Definition: The ratio of active tokens in context to the model's maximum context length. $$\text{Saturation} = \frac{\text{Current Tokens}}{\text{Model Context Limit}}$$
• Operational Threshold: When saturation exceeds 70%, agent reasoning accuracy degrades sharply (the "lost in the middle" phenomenon).

7. Token Velocity (Tokens / Second)

Measuring streaming throughput and generation velocity.

• Components:
  • Time-to-First-Token (TTFT): Measures prompt processing and queue delay.
  • Inter-Token Latency (ITL): Measures raw generation speed.
• Diagnostic Value: If TTFT spikes while ITL remains constant, your prompt context is growing too large or the provider is experiencing cold start latency.

8. Prompt Cache Hit Ratio

Tracking FinOps optimization efficiency.

• Definition: The percentage of prompt tokens served from prefix caches (e.g., Anthropic Prompt Caching or OpenAI Cached Tokens).
• Target: $>75%$ cache hit rate for system instructions, static tool schemas, and persistent persona definitions.

9. Dynamic Trace Risk Score Distribution

Real-time security telemetry.

• Definition: The statistical distribution of composite risk scores ($0.0 - 1.0$) calculated across all spans.
• Alert Condition: Any run with a risk score $>0.75$ triggers an automatic security event in Splyntra.

10. Hallucination & Retrieval Relevance Score

RAG and context grounding fidelity.

• Definition: The percentage of agent assertions that are grounded in retrieved context documents.
• Scoring: Calculated asynchronously using embedding semantic similarity and synthetic contradiction checks.

11. Human-in-the-Loop (HITL) Escalation Rate

Measuring agent autonomy boundaries.

• Definition: The percentage of runs that require human intervention or approval before proceeding.
• Trend Target: Should decline over time as agent policies and tool reliability mature.

12. State Drift Rate

Measuring deviation from planned trajectory.

• Definition: The divergence between the initial planned task breakdown and the actual executed path.
• Indicator: High drift indicates that initial planning prompts are failing to account for tool return structures.

Recommended Alerting Matrix for AI Agents

Next Steps & Related Technical Guides

• AI Agent Observability: What You Need to Monitor in Production — The 5 pillars of agent telemetry.
• How to Monitor AI Agents with OpenTelemetry — Instrumenting metrics and traces.
• How to Track LLM Costs in AI Agents — Granular cost attribution algorithms.