LLM Observability

Agentgateway can send LLM telemetry to specialized observability platforms for prompt logging, cost tracking, audit trail, and performance monitoring.

How it works

Agentgateway exports LLM telemetry via OpenTelemetry, which can be forwarded to LLM-specific observability platforms. These platforms provide the following.

• Prompt/response logging - Full request and response capture
• Token usage tracking - Monitor costs across models and users
• Latency analytics - Track response times and identify bottlenecks
• Evaluation - Score and evaluate LLM outputs
• Prompt management - Version and manage prompts

Configuration

Enable OpenTelemetry tracing with LLM-specific attributes.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
  tracing:
    otlpEndpoint: http://localhost:4317
    randomSampling: true

binds:
- port: 3000
  listeners:
  - routes:
    - backends:
      - ai:
          name: openai
          provider:
            openAI:
              model: gpt-4o-mini
      policies:
        backendAuth:
          key: "$OPENAI_API_KEY"

Agentgateway automatically includes these LLM-specific trace attributes: