Skip to content

Self-Optimisation

Five components implementing a cybernetic self-optimisation loop that continuously improves pipeline speed, cost, and quality through automated analysis, Pareto-optimal intervention selection, and CSF-gated auto-apply.

Overview

The self-optimisation cluster closes the feedback loop on G6 pipeline execution. Given a workflow trace (step timings, costs, quality scores), three specialist analyzers identify bottlenecks on their respective axes and propose interventions. opt_meta then computes a Pareto frontier across all three objectives, uses a multi-armed bandit (MAB) to select the most promising intervention plan, and optionally auto-applies it through a CSF safety gate. Outcomes are recorded and fed back into the MAB for continuous learning.

Evidence-first launch language

This cluster supports optimisation recommendations, CSF-gated trace mutation, and outcome learning. It should not be marketed as fully autonomous production self-improvement or guaranteed reliability gain until a customer workflow has before/after evidence showing the improvement. For launch, describe the value as "bounded optimisation with validation evidence and human review available."

opt_shared provides the shared type vocabulary: WorkflowTrace, WorkflowStep, Bottleneck, Intervention, OptimizationReport, and InterventionAction literals. All schemas use Pydantic frozen models with field validators for numeric bounds and step count caps.

Components

Component Description Gateway Tools
opt_speed Latency bottleneck detection and speed interventions optimization_report (speed axis)
opt_cost Cost bottleneck detection and cost-reduction interventions optimization_report (cost axis)
opt_quality Quality gap detection and quality improvement interventions optimization_report (quality axis)
opt_meta Pareto coordinator, MAB selection, CSF-gated auto-apply optimize_pipeline, optimization_stats, optimization_apply

Architecture

graph TD
    TRACE["WorkflowTrace"] --> SPEED["opt_speed<br/>analyze_and_recommend"]
    TRACE --> COST["opt_cost<br/>analyze_and_recommend"]
    TRACE --> QUALITY["opt_quality<br/>analyze_and_recommend"]

    SPEED --> META["opt_meta<br/>coordinate"]
    COST --> META
    QUALITY --> META

    META --> PARETO["Pareto Frontier"]
    PARETO --> MAB["MAB Selection"]
    MAB --> CSF{"CSF Gate"}
    CSF -->|pass| APPLY["Auto-Apply"]
    CSF -->|fail| BLOCK["Block + Record"]
    APPLY --> OUTCOME["Record Outcome"]
    BLOCK --> OUTCOME
    OUTCOME -->|feedback| MAB

Key Patterns

Cybernetic Loop. The system forms a bounded control loop: trace -> analyze -> select -> apply -> observe -> learn -> repeat. The MAB balances exploration of novel interventions with exploitation of previously successful ones, but production claims should be based on measured outcomes from the target workflow.

Tri-Axis Analysis. Speed, cost, and quality are analyzed independently by specialist blocks, then unified via Pareto optimality. Configurable weights ({"speed": 1/3, "cost": 1/3, "quality": 1/3}) allow trading off objectives.

CSF-Gated Auto-Apply. Interventions are never blindly applied. Each intervention maps to hazards via HAZARD_MAP, and SafetyVerifier.union_bound must pass before execution. csf_fail_closed=True blocks actions even if CSF itself is unavailable.

Security Hardening. 12-task hardening pass: schema validators (step caps, numeric bounds), NaN/inf filtering, state bounds (failed interventions cap, bandit decay), BFS iteration limits, unknown model warnings, and post-execution resource checks.