Observability¶

G6 includes a built-in observability stack powered by Prometheus and Grafana. The Recursive Architect engine exports real-time metrics on task execution, token usage, tool calls, self-healing, cost tracking, and constraint satisfaction — giving operators full visibility into what the AI is doing, how much it costs, and whether safety bounds are holding.

Architecture¶

┌──────────────────────┐
│  Recursive Architect │
│  (ArchitectMetrics)  │
│    port 9092         │
└──────────┬───────────┘
           │ scrape /metrics
           ▼
┌──────────────────────┐     ┌──────────────────────┐
│     Prometheus       │◄────│     PushGateway       │
│     port 9090        │     │     port 9091         │
└──────────┬───────────┘     └──────────▲────────────┘
           │ query                      │ push
           ▼                            │
┌──────────────────────┐     ┌──────────┴────────────┐
│      Grafana         │     │   AlertDispatcher     │
│     port 3000        │     │   (CI/CD alerts)      │
│  dashboards + alerts │     └───────────────────────┘
└──────────────────────┘

Pull model: Prometheus scrapes the ArchitectMetrics HTTP server every 5 seconds. Metrics are registered with a per-instance CollectorRegistry so multiple architect runs don't collide.

Push model: The CI/CD AlertDispatcher pushes deployment event counters to PushGateway for short-lived jobs (deploy scripts, health checks) that exit before Prometheus can scrape them.

Metrics Reference¶

Counters (monotonically increasing)¶

Gauges (current value)¶

Histograms¶

Pre-built Dashboard¶

The g6_architect.json dashboard is auto-provisioned into Grafana with 10 panels:

Agent Runtime Metrics¶

Separate from the architect engine, the agent runtime (agent_claude, agent_openai, agent_autogen, agent_smolagents, …) records per-backend reliability metrics in components/mvp/agent_runtime/metrics.py (MetricsCollector, a process-wide singleton via get_collector()).

Metric families¶

Latency percentiles are nearest-rank over a sliding window of the most recent 1024 samples per (agent, status) — bounded memory, percentiles over recent behaviour.

Exposure surfaces¶

• Prometheus text — the agent metrics server (mvp.agent_runtime.server) serves /metrics on AGENT_METRICS_PORT (default 9090), started automatically by LifecycleMixin.start(). Scraped by the agent_runtime job in infra/prometheus.yml.

  Port 9090 collides with Prometheus

  The default 9090 is also Prometheus's conventional port. On a single host (or if Prometheus publishes host port 9090) set AGENT_METRICS_PORT to a free port (e.g. 9093) and update the agent_runtime target in infra/prometheus.yml to match.

• REST JSON — GET /observability/agent-metrics returns the same data as JSON (bases/mvp/rest/routes/health.py, via MetricsCollector.snapshot()) for the GUI and any non-Prometheus consumer. It reflects the in-process collector — agent activity in that API process. For cross-process aggregation and time-series, use Grafana (scrape the runtime endpoint).

• GUI — Settings → Monitoring renders the live snapshot (latency p50/p95 bars, inference counts, token throughput, circuit-breaker badges), polling /observability/agent-metrics every 5 s. Set NEXT_PUBLIC_G6_GRAFANA_URL to surface an "Open in Grafana" link.

Pre-built dashboard — g6_agent_runtime.json¶

Auto-provisioned into Grafana (4 panels):

Alerts¶

Three deployment alerts are provisioned in Grafana via deploy-alerts.yml:

Configuration¶

Deployment modes¶

• Dev (docker-compose.yml): Prometheus on localhost:9090, Grafana on localhost:3000
• Prod (docker-compose.prod.yml): Prometheus internal only (no exposed port), Grafana on port 3000
• Blue-green (docker-compose.base.yml): Shared Prometheus + Grafana across blue/green slots

How it Works¶

The ArchitectMetrics class (components/mvp/recursive_architect/metrics.py) wraps prometheus_client counters, gauges, and histograms. Every recording method updates both:

1. In-memory counters — always available, used by the TUI dashboard
2. Prometheus collectors — only if prometheus_client is installed

If prometheus_client is not installed, the class silently falls back to in-memory counters only. The TUI and API continue to work — you just lose the Prometheus/Grafana pipeline.

FAQ¶

Do I need Prometheus and Grafana to run G6?

No. ArchitectMetrics falls back to in-memory counters when prometheus_client is not installed. The CLI/TUI displays metrics directly from these counters. Prometheus and Grafana add historical time-series storage and visual dashboards but are not required for operation.

Can I use a different monitoring backend?

Yes. Metrics are exposed as standard Prometheus exposition format on /metrics (port 9092). Any Prometheus-compatible scraper works — Datadog Agent, New Relic, Victoria Metrics, Thanos, or Grafana Alloy.

What is the difference between pushed and scraped metrics?

Scraped metrics (pull model) are used by ArchitectMetrics for long-running architect sessions. Prometheus fetches them every 5 seconds. Pushed metrics are used by AlertDispatcher for short-lived CI/CD jobs (deploy scripts, health checks) that may exit before Prometheus can scrape them. PushGateway acts as a buffer.

How do I add custom metrics?

1. Add a new Counter/Gauge/Histogram in ArchitectMetrics._init_prometheus()
2. Add a corresponding key to self._counters in __init__
3. Write a record_*() method that updates both the in-memory counter and the Prometheus collector
4. The metric will automatically appear in Prometheus and can be added to the Grafana dashboard

What dashboard panels come pre-built?

Ten panels covering subgoal progress, token usage and rate, tool calls, self-healing success rate, cost tracking, constraint satisfaction, HITL queue depth, branch execution duration percentiles, and crash count. See the Pre-built Dashboard section above.