Job Validation Suite¶

Gold-case evaluation harness for G6 job agents.

Cluster: Job Agents | Type: component | MCP Tools: 9

Overview¶

Gold-case evaluation harness for G6 job agents. Defines reviewed V2 evaluation cases, runs them against job agent blocks, scores outputs across answer quality, abstention quality, provenance coverage, and action safety, and provides CI gate checking.

The suite provides deterministic, offline internal validation for job-agent release gates (BETA: re-grounded, pending independent re-review and user sign-off — NOT a production-readiness certification): all 39 job agents have static reviewed gold cases, each case includes explicit scoring criteria, grounding sources, utility rationale, review metadata, and safety/provenance/pilot coverage. Fast CI is deterministic and offline by default.

LLM judge caveat

The live LLM judge and OpenAI web-search grounding paths are opt-in paid API paths. They are routed through mvp.llm_router and require both G6_JOB_EVAL_LLM=1 and ALLOW_PAID_API=1. Unless those gates are enabled and the live checks are run in the target environment, results should be treated as deterministic internal validation, not external certification or clinical/legal/financial assurance.

When to use:

Running regression tests on job agent outputs against reviewed gold cases
Gating CI pipelines on job agent quality thresholds without network access
Running selected live job-block evaluations before release
Comparing job agent performance across model, prompt, or tool changes

Validation modes:

schema: fast default mode; validates reviewed V2 fixtures and does not execute live job blocks or make LLM calls.
live: executes selected job blocks. By default this runs G6_JOB_EVAL_JOBS=finance,lawyer; set G6_JOB_EVAL_ALL=1 to run all available jobs.
LLM judge: disabled by default; requires G6_JOB_EVAL_LLM=1 ALLOW_PAID_API=1.

Operator env flags surfaced by MCP:

G6_JOB_VALIDATION_SUITE_AGENTIC_RUNTIME: enables or disables the validation-verdict agentic runtime; the deterministic safety floor still applies.
G6_DISABLE_LLM: global kill switch that forces deterministic-only behavior.
G6_JOB_EVAL_LLM: opt-in flag for paid per-case LLM judging; it is separate from schema-mode fixture validation.

Verification commands:

python -m mvp.job_validation_suite.ci_validate_gold_cases
python -m pytest tests/mvp/job_validation_suite -m "not heavy" -q
python -m pytest tests/mvp/job_validation_suite -m live_eval -q

Example:

from mvp.job_validation_suite import CIRunner, EvalRunner, SuiteResult

schema_passed, schema_result = CIRunner(mode="schema").check_job("lawyer")

runner = EvalRunner(job_name="lawyer")
result: SuiteResult = runner.run()
# result.overall, result.passes_threshold(target_pct=80)

Works well with: benchmark_runner, align_evals, job_framework

Public API¶

`CIRunner`¶

Check one or more job agents against their target_pct thresholds.

Constructor:

Parameter	Type	Default
`mode`	`str`	'schema'

Methods:

`check_job(job_name: str) -> tuple[bool, SuiteResult]`¶

Run eval for job_name and check against its CI gate.

`check_all_available() -> dict[str, tuple[bool, SuiteResult | AvailabilityFailure]]`¶

Check all jobs that have gold case files.

`report() -> str`¶

Return a human-readable summary of all available job checks.

`EvalRunner`¶

Run gold-case evaluation for one job agent.

Constructor:

Parameter	Type	Default
`job_name`	`str`	required
`history_db`	`str \\| None`	None

Methods:

`case_count() -> int`¶

`run() -> SuiteResult`¶

Run all gold cases and return an aggregated SuiteResult.

`run_with_regression_check(threshold: float = 0.05) -> tuple[SuiteResult, dict]`¶

Run evaluation and check for regressions against history.