Skip to content

Benchmark Runner¶

Cluster: Uncategorised | Type: component | MCP Tools: None

Overview¶

Structured evaluation harness for G6 components and solver pipelines. Defines benchmark cases and suites, executes them with timing, and persists results as JSONL for historical comparison and CI gates.

This is a lightweight benchmark harness, not a complete evaluation platform. It is useful for deterministic component checks, launch evidence, CI regression gates, and historical JSONL records. For nuanced model evaluation, partial-credit grading, rubric scoring, or semantic comparisons, pair it with a dedicated evaluator such as align_evals or a custom executor that returns a structured verdict.

Scoring caveat

pass_rate counts every case the runner marks as passed. pass_rate_scored counts only cases with a real assertion target via expected_output / expected_output_set. Regression gates use pass_rate_scored, so asserted crashes count as scored failures. Cases without expected_output are not scored unless the executor returns a structured {"success": true|false} verdict for the case-level pass/fail result.

Adapter caveat

AIBlock report access is bounded by reports_root unless trusted_local_report_path=True is explicitly set for in-process local use. External transports should use named executor modes (value, echo, json_success) and must treat Python callable executors as local-only evidence. Block outputs include completion_state, warning_card, evidence, request_id, task_id, and run_id so degraded benchmark evidence is surfaced rather than treated as cosmetic success.

When to use:

Running automated quality benchmarks in CI pipelines
Comparing solver or component accuracy across commits
Generating reproducible JSONL benchmark reports

Example:

from mvp.benchmark_runner import BenchmarkRunner, BenchmarkSuite, BenchmarkCase

suite = BenchmarkSuite(name="solver_v2", description="Small solver regression suite")
suite.add_case(BenchmarkCase(
    name="math_simple",
    description="Exact arithmetic answer",
    input_data={"goal": "2+2"},
    expected_output="4",
))

runner = BenchmarkRunner(executor=lambda input_data: "4", accuracy_threshold=1.0)
result = runner.run(suite)

assert result.passed == 1
assert result.pass_rate_scored == 1.0
assert result.regression_gate_passed is True

Works well with: solver, solver_accuracy, align_evals

Public API¶

`BenchmarkRunnerInput(BaseModel)`¶

Field	Type	Default
`op`	`str`	required
`parameters`	`dict[str, Any]`	Field(default_factory=dict)

`BenchmarkRunnerOutput(BaseModel)`¶

Field	Type	Default
`op`	`str`	''
`result`	`dict[str, Any]`	Field(default_factory=dict)
`message`	`str`	''
`degraded`	`bool`	False
`degradation_reason`	`str \\| None`	None
`error_code`	`str \\| None`	None
`evidence`	`dict[str, Any]`	Field(default_factory=dict)
`completion_state`	`str`	'qualified-draft'
`warning_card`	`dict[str, Any]`	Field(default_factory=dict)
`request_id`	`str \\| None`	None
`task_id`	`str \\| None`	None
`run_id`	`str \\| None`	None

`BenchmarkRunnerBlock(AIBlock)`¶

AIBlock wrapper for running benchmark suites and retrieving results.

Methods:

`infer(input: BenchmarkRunnerInput) -> Result[BenchmarkRunnerOutput]`¶

`CaseResult`¶

Field	Type	Default
`case_name`	`str`	required
`passed`	`bool`	required
`duration_ms`	`float`	required
`output`	`Any`	required
`error`	`Optional[str]`	None
`expected_match`	`Optional[bool]`	None
`expected_output`	`Any`	None
`expected_output_set`	`bool`	False

`RunResult`¶

Field	Type	Default
`suite_name`	`str`	required
`case_results`	`List[CaseResult]`	field(default_factory=list)
`suite_metadata`	`Optional[Dict[str, Any]]`	None
`regression_gate_passed`	`Optional[bool]`	None
`regression_gate_reason`	`Optional[str]`	None
`next_human_action`	`Optional[str]`	None

Methods:

`total() -> int`¶

`passed() -> int`¶

`failed() -> int`¶

`scored() -> int`¶

Cases that carried an expected_output and were genuinely

`pass_rate() -> float`¶

`pass_rate_scored() -> float`¶

Pass rate computed only over cases with a real assertion.

`avg_duration_ms() -> float`¶

`BenchmarkRunner`¶

Run a BenchmarkSuite with a callable executor and collect results.

Constructor:

Parameter	Type	Default
`executor`	`Callable[[Dict[str, Any]], Any]`	required
`require_expected_output`	`bool`	True
`accuracy_threshold`	`Optional[float]`	None

Methods:

`run(suite: BenchmarkSuite) -> RunResult`¶

`BenchmarkCase`¶

Field	Type	Default
`name`	`str`	required
`description`	`str`	required
`input_data`	`Dict[str, Any]`	required
`expected_output`	`Optional[Any]`	None
`tags`	`List[str]`	field(default_factory=list)
`expected_output_set`	`Optional[bool]`	None

Methods:

`has_expected_output() -> bool`¶

Whether this case has an assertion target.

`BenchmarkSuite`¶

Field	Type	Default
`name`	`str`	required
`description`	`str`	required
`cases`	`List[BenchmarkCase]`	field(default_factory=list)
`suite_metadata`	`Optional[Dict[str, Any]]`	field(default_factory=lambda: {'purpose': '', 'risk_level': 'unspecified', 'assertion_strategy': '', 'reviewer_required': None})

Methods:

`add_case(case: BenchmarkCase) -> None`¶

`filter_by_tag(tag: str) -> 'BenchmarkSuite'`¶

Return a new suite with only cases that have the given tag.

Functions¶

`run_result_to_dict(run_result: RunResult) -> dict`¶

`save_jsonl(run_result: RunResult, path: str) -> None`¶

Append run result as a JSONL record to the given file path.

`load_jsonl(path: str) -> List[dict]`¶

Load all run result records from a JSONL file.