Self-Hosted AI Observability: Why Every AI Agent Needs a Trace

Updated June 24, 2026

The Problem: AI Agents Are Black Boxes

When you’re building autonomous AI agents, the scariest question is: what did it actually do?

Without observability, every agent session is a black box:

• ✅ You see the input and the output
• ❌ You don’t know which tools were called
• ❌ You don’t know how many LLM calls were made
• ❌ You don’t know where latency was introduced
• ❌ You can’t measure the impact of prompt engineering changes

Real example: I deployed an AI agent to manage my infrastructure, and within the first week, observability revealed it was making 3x more LLM calls than necessary due to a retry loop bug. That’s a 67% efficiency loss — completely invisible without tracing.

What This Observability Stack Actually Monitors

Key Point: This observability stack doesn’t just monitor AI agents — it monitors everything on my HomeLab server:

• ✅ AI Agents: Traces every tool call, LLM request, and decision point
• ✅ Infrastructure: CPU, memory, disk, network metrics
• ✅ Serviços: Application performance, API response times, error rates
• ✅ Sites: Uptime, response time, error tracking
• ✅ AI Costs: Token usage, provider costs, budget tracking
• ✅ Custo: Real-time cost monitoring for AI operations

The Solution: End-to-End Observability Pipeline

Built a complete OpenTelemetry-based observability stack that traces every AI agent interaction — from tool call to LLM request to decision point — and aggregates it all in Grafana dashboards for real-time monitoring.

Architecture Diagram:

┌─────────────────┐
│   AI Agent       │
│   (Tool calls)   │
└────────┬─────────┘
         │ OTLP (HTTP)
         ▼
┌─────────────────┐
│ OpenTelemetry   │
│ Collector        │
│ (otlp_grpc)     │
└────────┬─────────┘
         │ Trace data
         ▼
┌─────────────────┐
│  Jaeger         │
│  (Tracing UI)   │
└─────────────────┘

         │ Metrics
         ▼
┌─────────────────┐
│ Prometheus      │
│ (Metrics)       │
└────────┬─────────┘
         │
         ▼
┌─────────────────┐
│ Loki            │
│ (Logs)          │
└────────┬─────────┘
         │
         ▼
┌─────────────────┐
│ Grafana         │
│ (Dashboards)    │
└─────────────────┘
        

The Observability Stack

1. OpenTelemetry Collector (OTLP Endpoint)

What it does: Receives tracing and metrics from AI agent sessions, normalizes data, and forwards to Jaeger, Prometheus, and Loki.

Configuration (docker-compose.yml):

otel-collector:
  image: otel/opentelemetry-collector:0.100.0
  command: ["--config=/etc/otel-collector-config.yaml"]
  volumes:
    - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
  ports:
    - "4318:4318"  # OTLP HTTP
    - "4317:4317"  # OTLP gRPC

otel-collector-config.yaml:
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      exporters: [loki]

2. Jaeger (Distributed Tracing)

What it does: Stores and visualizes trace data from every AI agent session, tool call, and LLM request.

Deployment (docker-compose.yml):

jaeger:
  image: jaegertracing/all-in-one:1.55
  ports:
    - "16686:16686"  # Web UI
    - "14250:14250"  # Jaeger gRPC
  environment:
    - COLLECTOR_OTLP_ENABLED=true

3. Prometheus (Metrics Collection)

What it does: Collects metrics from everything — AI agents, infrastructure, services, sites. Measures CPU, memory, disk, network, API response times, error rates, and more.

Key metrics:

Metric	Description	Labels
agent_tool_calls_total	Total tool calls by name	tool_name, status
agent_llm_calls_total	Total LLM calls	model, provider
agent_tool_latency_seconds	Tool call latency	tool_name, provider
agent_token_usage	Token consumption	model, prompt_tokens, completion_tokens
ai_cost_total	AI costs in USD	model, provider
host_cpu_usage_percent	Host CPU usage	host
host_memory_usage_percent	Host memory usage	host
api_response_time_ms	API response time	endpoint, status
service_error_rate	Service error rate	service, error_type

4. Loki (Log Aggregation)

What it does: Centralizes and indexes logs from all services and applications on the HomeLab server. Supports log queries, filtering, and correlation with traces and metrics.

Benefits:

• 🔍 Search logs across all 15+ services
• 📊 Correlate logs with traces and metrics
• ⚡ Fast query performance
• 💾 Long-term retention (30 days)
• 🎨 Rich log formatting and filtering

5. Grafana (Dashboards & Alerts)

What it does: Real-time dashboards visualizing agent behavior, infrastructure health, service performance, and costs — aggregating everything from Prometheus and Jaeger and Loki.

Real-Time Alerting: Prometheus → Alertmanager → n8n → Telegram

Complete Alert Pipeline:

Prometheus → Alertmanager (9093) → n8n Webhook → Telegram Bot → Me (and others)

7 Automated Alert Rules

Rule	Condition	Severity	What It Alerts On
High Error Rate	rate(agent_error_rate[5m]) > 0.1 for 2m	Warning	AI tools failing frequently
High LLM Call Count	sum(agent_llm_calls_total) > 100 for 5m	Warning	Excessive AI calls
Slow Tool Latency	histogram_quantile(0.95, rate(agent_tool_latency_seconds_bucket[5m])) > 2 for 5m	Warning	Slow tool execution
Token Usage Spike	increase(agent_token_usage[1h]) > 10000 for 1h	Info	Unusual token consumption
AI Cost Spike	increase(ai_cost_total[1h]) > 5 for 1h	Warning	Unusual AI spending
Critical LLM Failure	rate(agent_llm_calls_total{status="error"}[5m]) > 0.5 for 1m	Critical	LLM API failures
No Tool Calls	agent_tool_calls_total == 0 for 10m	Info	Agent stopped working

What We Learned in the First Week

Discovery 1: 3x LLM Call Optimization

Problem: AI agent was making redundant LLM calls due to a retry loop.

Observability revealed:

• 300 LLM calls in first week
• 200 were redundant (retry loop)
• Only 100 were needed for correct decisions

Result: Rewrote prompt engineering to include “don’t retry without new context” instruction.

Outcome: Reduced LLM calls by 67% within 2 days of deployment.

Discovery 2: Variable Latency (200ms to 8s)

Problem: Tool call latency varied wildly between 200ms and 8 seconds.

Observability revealed:

• Database tool: 200ms (cached)
• API tool: 2s (no cache)
• External service tool: 8s (no cache, high latency)

Result: Implemented caching and parallel tool execution.

Outcome: Average tool latency reduced from 2.5s to 0.8s.

Discovery 3: Prompt Engineering Impact Is Measurable

Problem: Hard to tell if prompt changes actually improve agent performance.

Observability revealed:

• Prompt version A: 15 tool calls per session, 50% error rate
• Prompt version B: 10 tool calls per session, 30% error rate
• Prompt version C: 8 tool calls per session, 20% error rate

Result: Prompt version C was 47% more efficient (8 vs. 15 tool calls).

AI Cost Monitoring

Real-Time Cost Tracking:

• Monitors token usage by model and provider
• Tracks cost in USD per operation
• Alerts on budget thresholds
• Cost trends over time (24h, 7d, 30d)

Infrastructure & Site Monitoring

Beyond AI Agents:

• CPU & Memory: Real-time host and container resource usage
• Network: Bandwidth usage, connection rates
• Sites: Uptime, response time, error rates for all hosted sites
• Serviços: Application performance, API health checks
• Dashboard 1: Agent Execution Overview
• Dashboard 2: Agent Tool Performance
• Dashboard 3: AI Token Usage & Cost
• Dashboard 4: Infrastructure Health
• Dashboard 5: Service Performance

Production Considerations

High Availability

• Jaeger with Postgres: Persistent trace storage
• OTEL Collector clustering: Multiple collectors with load balancing
• Prometheus federation: Multi-tenant metrics
• Loki with PostgreSQL: Persistent log storage
• Grafana clustering: Multi-instance dashboards

Data Retention

• Jaeger trace retention: Development: 7 days, Production: 30 days, Archive: 90 days (S3 storage)
• Prometheus retention: Development: 15 days, Production: 30 days
• Grafana dashboard cache: TTL: 5 minutes (fresh data), Auto-refresh: 30s for dashboards
• Loki log retention: Development: 7 days, Production: 30 days

The Takeaway: Observability Is Non-Negotiable

AI agents are complex systems with many moving parts. Without observability, you’re flying blind.

My observability stack:

• ✅ Traces for every tool call, LLM request, and decision point
• ✅ Metrics for latency, error rates, token usage, AI costs
• ✅ Logs for full application visibility
• ✅ Dashboards for real-time monitoring
• ✅ 7 alert rules for proactive incident response
• ✅ Integration with n8n for automated alerts → Telegram

Results:

67%LLM call optimization

3xlatency reduction

47%prompt engineering impact

Fasterincident detection

Built with:

OpenTelemetry, Jaeger, Prometheus, Loki, Grafana, OTLP, Prometheus Alertmanager, n8n, Docker

---