How to Monitor AI Agents with OpenTelemetry

Proprietary observability SDKs create vendor lock-in and fragment your telemetry across separate dashboards. Your backend microservices export traces using OpenTelemetry (OTel), but your AI agents are trapped in bespoke, incompatible logging formats.

The solution is standardizing on OpenTelemetry GenAI Semantic Conventions. By treating your agent workflows as standard OpenTelemetry distributed traces, you unify your AI agents, databases, message queues, and API gateways into one continuous observability graph.

This guide provides a production-ready tutorial on instrumenting autonomous AI agents with OpenTelemetry in Python and TypeScript.

The OpenTelemetry GenAI Semantic Conventions

The OpenTelemetry community has standardized attributes for generative AI and agentic systems. When instrumenting your agents, adhere to these standard attribute namespaces:

Architecture: How Agent Spans Flow to Splyntra

┌─────────────────────────────────────────────────────────────┐
│                       AI AGENT RUNTIME                      │
│                                                             │
│  [Agent Loop] ──► [LLM Span] ──► [Tool Span]                │
│         │                                                   │
│         ▼                                                   │
│  OpenTelemetry SDK (TracerProvider + BatchSpanProcessor)    │
└──────────────────────────────┬──────────────────────────────┘
                               │ OTLP / gRPC (port 4317)
                               │ or OTLP / HTTP (port 4318)
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                 SPLYNTRA INGESTION PIPELINE                 │
│                                                             │
│  1. PII & Secret Redactor                                   │
│  2. Token Price Calculator ($ / span)                       │
│  3. Trace Risk & Injection Scoring Engine                   │
│  4. Time-series & Graph Storage                             │
└─────────────────────────────────────────────────────────────┘

Implementation in Python

1. Install OpenTelemetry Dependencies

pip install opentelemetry-api \
            opentelemetry-sdk \
            opentelemetry-exporter-otlp-proto-http \
            opentelemetry-instrumentation

2. Configure the Global Tracer Provider & Exporter

Create an initialization module (telemetry.py):

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

def init_agent_telemetry(service_name: str = "support-agent-service"):
    # 1. Define service resources
    resource = Resource.create({
        "service.name": service_name,
        "service.version": "1.0.0",
        "deployment.environment": os.getenv("ENV", "production"),
    })

    # 2. Configure OTLP Exporter (pointing to Splyntra or any standard collector)
    otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "https://ingest.splyntra.com/v1/traces")
    api_key = os.getenv("SPLYNTRA_API_KEY", "")

    headers = {}
    if api_key:
        headers["x-splyntra-key"] = api_key

    exporter = OTLPSpanExporter(
        endpoint=otlp_endpoint,
        headers=headers,
    )

    # 3. Use BatchSpanProcessor for non-blocking asynchronous exports
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(
        BatchSpanProcessor(
            exporter,
            max_queue_size=2048,
            schedule_delay_millis=500,
            max_export_batch_size=512,
        )
    )

    trace.set_tracer_provider(provider)
    return trace.get_tracer("splyntra.agent.tracer")

3. Instrumenting the Multi-Step Agent Execution

Now instrument the reasoning loop, LLM completions, and tool executions:

import time
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from telemetry import init_agent_telemetry

tracer = init_agent_telemetry()

def execute_agent_task(user_query: str, user_id: str):
    # Root Workflow Span
    with tracer.start_as_current_span(
        "agent.run",
        attributes={
            "gen_ai.agent.name": "refund_assistant",
            "gen_ai.system": "splyntra",
            "enduser.id": user_id,
            "gen_ai.prompt": user_query,
        }
    ) as root_span:
        step = 0
        is_finished = False

        while not is_finished and step < 5:
            step += 1
            with tracer.start_as_current_span(
                f"agent.step",
                attributes={
                    "gen_ai.agent.step_number": step,
                }
            ) as step_span:
                # Step A: LLM Call Span
                with tracer.start_as_current_span(
                    "llm.generate",
                    attributes={
                        "gen_ai.system": "openai",
                        "gen_ai.request.model": "gpt-4o",
                        "gen_ai.request.temperature": 0.2,
                    }
                ) as llm_span:
                    # Simulated LLM generation
                    time.sleep(0.4)
                    llm_span.set_attribute("gen_ai.usage.prompt_tokens", 850 + (step * 200))
                    llm_span.set_attribute("gen_ai.usage.completion_tokens", 120)
                    llm_span.set_attribute("gen_ai.response.finish_reasons", ["tool_calls"])

                # Step B: Tool Call Span
                with tracer.start_as_current_span(
                    "tool.execute",
                    attributes={
                        "gen_ai.tool.name": "database_lookup",
                        "gen_ai.tool.call_id": f"call_{step}",
                        "gen_ai.tool.parameters": '{"order_id": "ORD-9912"}',
                    }
                ) as tool_span:
                    try:
                        # Simulated tool work
                        time.sleep(0.15)
                        tool_result = '{"status": "delivered", "amount": 149.00}'
                        tool_span.set_attribute("gen_ai.tool.result", tool_result)
                        tool_span.set_status(Status(StatusCode.OK))
                    except Exception as e:
                        tool_span.record_exception(e)
                        tool_span.set_status(Status(StatusCode.ERROR, str(e)))

                if step >= 2:
                    is_finished = True

        root_span.set_attribute("gen_ai.agent.total_steps", step)
        root_span.set_status(Status(StatusCode.OK))

Implementation in TypeScript / Node.js

1. Install Node.js OpenTelemetry Packages

npm install @opentelemetry/api \
            @opentelemetry/sdk-trace-node \
            @opentelemetry/exporter-trace-otlp-http \
            @opentelemetry/resources \
            @opentelemetry/semantic-conventions

2. Configure the NodeSDK Tracer

import { trace, SpanStatusCode } from "@opentelemetry/api";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";

export function initTracer(serviceName = "typescript-agent-service") {
  const provider = new NodeTracerProvider({
    resource: new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
      [SemanticResourceAttributes.SERVICE_VERSION]: "1.0.0",
    }),
  });

  const exporter = new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "https://ingest.splyntra.com/v1/traces",
    headers: {
      "x-splyntra-key": process.env.SPLYNTRA_API_KEY || "",
    },
  });

  provider.addSpanProcessor(new BatchSpanProcessor(exporter));
  provider.register();

  return trace.getTracer("splyntra-agent-ts");
}

3. Instrumenting an Agent in TypeScript

const tracer = initTracer();

export async function runAgentTask(taskId: string, inputPrompt: string) {
  return tracer.startActiveSpan(
    "agent.execution",
    {
      attributes: {
        "gen_ai.agent.name": "code_reviewer",
        "gen_ai.agent.task_id": taskId,
        "gen_ai.system": "splyntra",
      },
    },
    async (rootSpan) => {
      try {
        // LLM reasoning step
        await tracer.startActiveSpan("llm.reasoning", async (llmSpan) => {
          llmSpan.setAttribute("gen_ai.request.model", "claude-3-7-sonnet");
          llmSpan.setAttribute("gen_ai.usage.prompt_tokens", 2400);
          llmSpan.setAttribute("gen_ai.usage.completion_tokens", 450);
          llmSpan.end();
        });

        // Tool execution step
        await tracer.startActiveSpan("tool.git_diff", async (toolSpan) => {
          toolSpan.setAttribute("gen_ai.tool.name", "get_git_diff");
          toolSpan.setAttribute("gen_ai.tool.parameters", JSON.stringify({ pr: 402 }));
          toolSpan.setStatus({ code: SpanStatusCode.OK });
          toolSpan.end();
        });

        rootSpan.setStatus({ code: SpanStatusCode.OK });
      } catch (err: any) {
        rootSpan.recordException(err);
        rootSpan.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      } finally {
        rootSpan.end();
      }
    }
  );
}

Best Practices for Agent Telemetry in Production

1. Never Block Execution: Always wrap the trace exporter in a BatchSpanProcessor. Synchronous exporters add tens of milliseconds of HTTP overhead to every LLM turn.
2. Propagate Context Across Async Queues: If your agent delegates tasks via Celery, BullMQ, or Kafka, inject traceparent headers to keep the trace graph connected.
3. Attach Baggage for Multi-Tenant Isolation: Use OpenTelemetry Baggage to propagate tenant_id and organization_id down to all child spans automatically.
4. Scrub PII Before Serialization: Implement an in-memory OTel SpanProcessor to mask credit card numbers, email addresses, and API tokens before spans leave your application memory.

Next Steps & Related Technical Guides

• How to Trace AI Agents with OpenTelemetry — Advanced distributed tracing across multi-agent graphs.
• AI Agent Observability: What You Need to Monitor in Production — Pillar guide to production telemetry.
• How to Track LLM Costs in AI Agents — Calculating real-time token attribution and budget caps.