Gdpval Harness¶
GDPval empirical grounding harness for G6 job agents.
Cluster: Uncategorised | Type: component | MCP Tools: None
Overview¶
Empirical grounding harness that evaluates G6 job agents against the OpenAI GDPval benchmark dataset. Loads tasks from HuggingFace (openai/gdpval, 2.29 GB cached), maps occupations to job agent components, routes each task to the appropriate agent, scores deliverables against rubrics with optional LLM-as-judge, and emits JSONL run records. Includes calibration tools for computing per-job prior success probabilities from benchmark runs.
When to use:
- Evaluating job agent quality against a standardised occupational benchmark
- Computing calibrated priors for Bayesian confidence estimation
- Regression testing after job agent modifications
- Generating evidence for tier advancement decisions
Works well with: All job_* components, align_evals, self_training, adapt_bayesian
Public API¶
GdpvalHarnessInput(BaseModel)¶
| Field | Type | Default |
|---|---|---|
op | str | required |
parameters | dict[str, Any] | Field(default_factory=dict) |
GdpvalHarnessOutput(BaseModel)¶
| Field | Type | Default |
|---|---|---|
op | str | '' |
result | dict[str, Any] | Field(default_factory=dict) |
message | str | '' |
GdpvalHarnessBlock(AIBlock)¶
AIBlock wrapper for GDPval empirical grounding harness tasks and scoring.
Methods:
infer(input: GdpvalHarnessInput) -> Result[GdpvalHarnessOutput]¶
GDPvalLoader¶
Constructor:
| Parameter | Type | Default |
|---|---|---|
cache_dir | str \| Path \| None | None |
local_override | str \| Path \| None | None |
Methods:
iter_tasks(limit: int | None = None) -> Iterator[dict]¶
Yield GDPval task dicts.