Skip to content

Llm Router Harness

Component id: llm_router

Source contract: BlockContract for llm_router.

Overview

LLM routing layer via OpenRouter and Ollama with resource enforcement, model discovery, hardware-aware recommendations, and LiteLLM-based completion with custom callback hooks.

When to use:

  • Routing LLM calls through OpenRouter or local Ollama backends
  • Local inference with hardware-aware model selection
  • Resource-bounded LLM usage with token tracking

Good at

  • open-ended generation
  • structured-output prompts with explicit format
  • summarisation and rewriting

Not suited for

  • deterministic arithmetic
  • verifiable factual lookup without retrieval grounding
  • long-horizon planning without intermediate verification

Inputs and outputs

  • Required: prompt
  • Optional: model, system_prompt, max_tokens, temperature
  • Outputs: completion_text

Mitigates

Known Failure Modes

  • hallucination: Fluent text not grounded in any source.
  • timeout: Provider exceeded the call timeout.
  • rate_limit: Provider returned 429 or quota-exceeded.

State Surface

  • Persistence: session
  • Queryable via: scorecard
  • Retention: 90d

Extension Points

  • training_signal_inputs: ('agent_routing_outcomes', 'provider_errors')
  • validation_gate: unit_plus_codex_review
  • allowed_mutation_surfaces: ('routing_policy', 'provider_thresholds')

Verification

  • Method: external_evidence_or_human_review
  • Human review gate: False
  • Fallback block: none