Solver Accuracy¶

solver_accuracy — solver quality history and improvement trail.

Cluster: Uncategorised | Type: component | MCP Tools: 9

Overview¶

Solver quality history tracker and improvement trail. Records per-run accuracy scores across four dimensions (completion rate, checkpoint pass rate, optimisation quality, composite score), supports optional user ratings, and provides trend analysis and historical querying via a SQLite-backed store.

MVP storage scope

solver_accuracy is designed for local MVP and pilot workflows. Its SQLite store is useful for tracking a single user's solver runs, trend direction, CSV exports, and regression gates on one machine or one small deployment. It is not a multi-tenant production analytics backend. For paid-customer scale, shared dashboards, long-term retention, or cross-user reporting, move the accuracy history into the service database or an analytics store with backups, tenancy boundaries, access controls, and retention policy.

When to use:

Tracking solver accuracy improvements over time across model and prompt changes
Providing a quality history feed to opt_meta for adaptive optimisation decisions
Gating releases on minimum composite accuracy thresholds

Example:

from mvp.solver_accuracy import SolverAccuracyBlock, SolverAccuracyInput

block = SolverAccuracyBlock(name="accuracy")
result = block.infer(SolverAccuracyInput(
    op="record",
    run_id="run_abc123",
    completion_rate=0.92,
    checkpoint_pass_rate=0.88,
    opt_quality_score=0.75,
))
# result.ok → True; result.value → SolverAccuracyOutput with composite_score

Works well with: solver, benchmark_runner, opt_meta

Public API¶

`SolverAccuracyGateError(ValueError)`¶

The LLM did not produce a usable, validated regression-gate verdict.

`GateVerdict`¶

Validated solver_accuracy regression-gate verdict for one window.

Field	Type	Default
`decision`	`str`	required
`escalated`	`bool`	False
`reasons`	`tuple[str, ...]`	()
`rationale`	`str`	''
`degraded`	`bool`	False
`raw_response`	`str`	''

`SolverAccuracyGatePlanner`¶

Runtime-first facade with deterministic fallback, one-way verdict clamp, and a

Constructor:

Parameter	Type	Default
`runtime`	`SolverAccuracyGateRuntime \\| None`	None

Methods:

`decide(stats: dict[str, Any]) -> GateVerdict`¶

`SolverAccuracyBlock(AIBlock[SolverAccuracyInput, SolverAccuracyOutput, None])`¶

AIBlock wrapper around SolverAccuracyStore exposing 4 ops via infer().

Field	Type	Default
`planner`	`Any`	field(default=None)

Methods:

`infer(data: SolverAccuracyInput) -> Result[SolverAccuracyOutput]`¶

Dispatch to the appropriate op handler.

`health() -> dict`¶

`AccuracyRecord(BaseModel)`¶

One row in the quality history: all sub-scores plus final.

Field	Type	Default
`run_id`	`str`	required
`timestamp`	`float`	required
`completion_rate`	`float`	Field(ge=0.0, le=1.0)
`checkpoint_pass_rate`	`float`	Field(ge=0.0, le=1.0)
`opt_quality_score`	`float`	Field(ge=0.0, le=1.0)
`composite_score`	`float`	Field(ge=0.0, le=1.0)
`user_rating`	`float \\| None`	Field(default=None, ge=0.0, le=1.0)
`final_score`	`float`	Field(ge=0.0, le=1.0)
`problem_type`	`str`	''
`complexity`	`str`	''
`strategy`	`str`	''
`config_flags`	`dict[str, bool]`	Field(default_factory=dict)
`failure_category`	`str`	''
`failure_description`	`str`	''
`partial_credit`	`float`	Field(default=0.0, ge=0.0, le=1.0)

`StepAccuracyRecord(BaseModel)`¶

Per-step accuracy record (Issue 17).

Field	Type	Default
`run_id`	`str`	required
`step_number`	`int`	required
`step_name`	`str`	required
`status`	`Literal['completed', 'failed', 'skipped', 'distilled']`	required
`tokens_used`	`int`	Field(default=0, ge=0)
`latency_ms`	`int`	Field(default=0, ge=0)
`quality_score`	`float`	Field(default=0.0, ge=0.0, le=1.0)

`FailureDetail(BaseModel)`¶

Structured failure categorization (Issue 22).

Field	Type	Default
`category`	`str`	''
`description`	`str`	''
`partial_credit`	`float`	Field(default=0.0, ge=0.0, le=1.0)

`SolverAccuracyInput(BaseModel)`¶

Field	Type	Default
`op`	`Literal['record', 'get_recent', 'get_trend', 'rate', 'check_threshold', 'record_steps', 'get_step_accuracy', 'get_by_type', 'export_csv', 'get_summary', 'describe']`	required
`run_id`	`str`	''
`completion_rate`	`float`	Field(default=0.0, ge=0.0, le=1.0)
`checkpoint_pass_rate`	`float`	Field(default=0.0, ge=0.0, le=1.0)
`opt_quality_score`	`float`	Field(default=0.0, ge=0.0, le=1.0)
`user_rating`	`float \\| None`	Field(default=None, ge=0.0, le=1.0)
`limit`	`int`	Field(default=20, ge=1, le=1000)
`days`	`int`	Field(default=7, ge=1, le=3650)
`threshold`	`float`	Field(default=0.5, ge=0.0, le=1.0)
`window`	`int`	Field(default=14, ge=1, le=3650)
`min_samples`	`int`	Field(default=10, ge=0, le=10000)
`max_ci_width`	`float`	Field(default=0.3, ge=0.0, le=1.0)
`deployment_scope`	`Literal['local_mvp', 'production_analytics']`	'local_mvp'
`problem_type`	`str`	''
`complexity`	`str`	''
`strategy`	`str`	''
`config_flags`	`dict[str, bool]`	Field(default_factory=dict)
`failure_detail`	`FailureDetail \\| None`	None
`step_results`	`list[StepAccuracyRecord]`	Field(default_factory=list)
`filter_problem_type`	`str`	''
`filter_complexity`	`str`	''

`SolverAccuracyOutput(BaseModel)`¶

Field	Type	Default
`op`	`str`	required
`record`	`AccuracyRecord \\| None`	None
`records`	`list[AccuracyRecord]`	Field(default_factory=list)
`trend`	`dict[str, float \\| int \\| None]`	Field(default_factory=dict)
`success`	`bool`	True
`error`	`str`	''
`threshold_met`	`bool`	True
`observed_score`	`float`	0.0
`decision`	`Literal['pass', 'REVIEW_REQUIRED']`	'pass'
`decision_reasons`	`list[str]`	Field(default_factory=list)
`step_records`	`list[StepAccuracyRecord]`	Field(default_factory=list)
`csv_data`	`str`	''
`summary`	`dict`	Field(default_factory=dict)
`confidence_interval`	`tuple[float, float] \\| None`	None
`sample_size_warning`	`str`	''
`degraded`	`bool`	False
`degradation_reason`	`str \\| None`	None
`completion_state`	`CompletionState`	'qualified-draft'
`agentic_evidence`	`dict \\| None`	None

`SolverAccuracyStore`¶

Thread-safe SQLite store for solver quality time-series.

Constructor:

Parameter	Type	Default
`db_path`	`str \\| None`	None

Methods:

`close() -> None`¶

`record(run_id: str, completion_rate: float, checkpoint_pass_rate: float, opt_quality_score: float, _timestamp: float | None = None, problem_type: str = '', complexity: str = '', strategy: str = '', config_flags: dict[str, bool] | None = None, failure_category: str = '', failure_description: str = '', partial_credit: float = 0.0) -> AccuracyRecord`¶

Compute composite_score and final_score, persist row, return AccuracyRecord.

`rate(run_id: str, user_rating: float) -> AccuracyRecord | None`¶

Load existing record, blend score, persist, return updated AccuracyRecord.

`get_recent(limit: int = 20) -> list[AccuracyRecord]`¶

`get_trend(days: int = 7) -> dict`¶

`record_steps(run_id: str, steps: list[dict]) -> int`¶

Save step-level accuracy records for a given run.

`get_step_accuracy(run_id: str) -> list[dict]`¶

Query step accuracy records for a given run.

`get_weakest_steps(days: int = 30) -> list[dict]`¶

Find which steps fail most often over the given period.

`get_by_type(problem_type: str = '', complexity: str = '', days: int = 30) -> list[AccuracyRecord]`¶

Query accuracy records filtered by problem type and/or complexity.

`export_csv(days: int = 30) -> str`¶

Export recent accuracy records as a CSV string.

`get_summary(days: int = 30) -> dict`¶

Aggregate statistics over the given period.

`total_runs() -> int`¶

Functions¶

`agentic_planner_enabled(default_enabled: bool = True) -> bool`¶

Decide whether the agentic solver_accuracy gate planner should run.

`validate_gate_verdict(verdict: GateVerdict) -> None`¶

Assert a FINAL gate verdict is canonical.

`deterministic_gate_verdict(n: int, mean: float | None, threshold: float, ci_width: float | None, min_samples: int, max_ci_width: float) -> GateVerdict`¶

Demoted-real regression-gate floor (the zero-LLM baseline) for pass vs review.

`apply_gate_floor(candidate: GateVerdict, baseline: GateVerdict) -> GateVerdict`¶

One-way verdict-severity clamp (output-verification-loop): stricter-of.

`grounded_gate_decision(stats: dict[str, Any], planner: 'SolverAccuracyGateRuntime | None' = None) -> tuple[GateVerdict, Any, bool, str]`¶

The ONE shared grounded gate chokepoint for EVERY regression-gate surface.

MCP Tools¶

Operation	Source
`ops`	solver_accuracy_mcp
`help`	solver_accuracy_mcp
`describe`	solver_accuracy_mcp
`check_threshold`	solver_accuracy_mcp
`get_trend`	solver_accuracy_mcp
`get_summary`	solver_accuracy_mcp
`list_patterns`	solver_accuracy_mcp
`explain_threshold_gate`	solver_accuracy_mcp
`list_strategies`	solver_accuracy_mcp

Solver Accuracy¶

Overview¶

Public API¶

SolverAccuracyGateError(ValueError)¶

GateVerdict¶

SolverAccuracyGatePlanner¶

decide(stats: dict[str, Any]) -> GateVerdict¶

SolverAccuracyBlock(AIBlock[SolverAccuracyInput, SolverAccuracyOutput, None])¶

infer(data: SolverAccuracyInput) -> Result[SolverAccuracyOutput]¶

health() -> dict¶

AccuracyRecord(BaseModel)¶

StepAccuracyRecord(BaseModel)¶

FailureDetail(BaseModel)¶

SolverAccuracyInput(BaseModel)¶

SolverAccuracyOutput(BaseModel)¶

SolverAccuracyStore¶

close() -> None¶

rate(run_id: str, user_rating: float) -> AccuracyRecord | None¶

get_recent(limit: int = 20) -> list[AccuracyRecord]¶

get_trend(days: int = 7) -> dict¶

record_steps(run_id: str, steps: list[dict]) -> int¶

get_step_accuracy(run_id: str) -> list[dict]¶

get_weakest_steps(days: int = 30) -> list[dict]¶

get_by_type(problem_type: str = '', complexity: str = '', days: int = 30) -> list[AccuracyRecord]¶

export_csv(days: int = 30) -> str¶

get_summary(days: int = 30) -> dict¶

total_runs() -> int¶

Functions¶

agentic_planner_enabled(default_enabled: bool = True) -> bool¶

validate_gate_verdict(verdict: GateVerdict) -> None¶

deterministic_gate_verdict(n: int, mean: float | None, threshold: float, ci_width: float | None, min_samples: int, max_ci_width: float) -> GateVerdict¶

apply_gate_floor(candidate: GateVerdict, baseline: GateVerdict) -> GateVerdict¶

grounded_gate_decision(stats: dict[str, Any], planner: 'SolverAccuracyGateRuntime | None' = None) -> tuple[GateVerdict, Any, bool, str]¶

MCP Tools¶

`SolverAccuracyGateError(ValueError)`¶

`GateVerdict`¶

`SolverAccuracyGatePlanner`¶

`decide(stats: dict[str, Any]) -> GateVerdict`¶

`SolverAccuracyBlock(AIBlock[SolverAccuracyInput, SolverAccuracyOutput, None])`¶

`infer(data: SolverAccuracyInput) -> Result[SolverAccuracyOutput]`¶

`health() -> dict`¶

`AccuracyRecord(BaseModel)`¶

`StepAccuracyRecord(BaseModel)`¶

`FailureDetail(BaseModel)`¶

`SolverAccuracyInput(BaseModel)`¶

`SolverAccuracyOutput(BaseModel)`¶

`SolverAccuracyStore`¶

`close() -> None`¶

`rate(run_id: str, user_rating: float) -> AccuracyRecord | None`¶

`get_recent(limit: int = 20) -> list[AccuracyRecord]`¶

`get_trend(days: int = 7) -> dict`¶

`record_steps(run_id: str, steps: list[dict]) -> int`¶

`get_step_accuracy(run_id: str) -> list[dict]`¶

`get_weakest_steps(days: int = 30) -> list[dict]`¶

`get_by_type(problem_type: str = '', complexity: str = '', days: int = 30) -> list[AccuracyRecord]`¶

`export_csv(days: int = 30) -> str`¶

`get_summary(days: int = 30) -> dict`¶

`total_runs() -> int`¶

`agentic_planner_enabled(default_enabled: bool = True) -> bool`¶

`validate_gate_verdict(verdict: GateVerdict) -> None`¶

`deterministic_gate_verdict(n: int, mean: float | None, threshold: float, ci_width: float | None, min_samples: int, max_ci_width: float) -> GateVerdict`¶

`apply_gate_floor(candidate: GateVerdict, baseline: GateVerdict) -> GateVerdict`¶

`grounded_gate_decision(stats: dict[str, Any], planner: 'SolverAccuracyGateRuntime | None' = None) -> tuple[GateVerdict, Any, bool, str]`¶