Skip to content

Align Evals

align_evals — mvp.align_evals

Cluster: Safety & Alignment | Type: component | MCP Tools: 30

Overview

align_evals computes evaluation metrics (accuracy, precision, recall, F1, exact_match, MSE, MAE) in pure Python with macro-averaging, providing automated quality gates for pipeline outputs. The 27-operation MCP sub-package adds benchmark management, eval sets, progress tracking, baseline comparison, read-only storage health, and a list_patterns readout of the applied deterministic reliability patterns.

When to use:

  • Scoring model predictions against ground truth labels
  • Comparing pipeline outputs across experiments with stored benchmarks
  • Automated quality gates that pass or fail based on metric thresholds

Example:

from mvp.align_evals import AlignEvalsBlock, EvalInput

block = AlignEvalsBlock(name="eval")
result = block.infer(EvalInput(
    predictions=["cat", "dog", "cat"],
    ground_truth=["cat", "dog", "dog"],
    metrics=["accuracy", "f1"],
))
# result.ok → True; result.value → EvalOutput with scores={"accuracy": 0.667, "f1": ...}

Works well with: adapt_sklearn, adapt_pandas, grounding

Pilot-readiness caveats

align_evals is suitable for pilot workflows and first-user MCP quality gates, but treat it as a lightweight evaluation harness rather than a full experiment-tracking platform.

  • batch_evaluate evaluates multiple runs and reports per-run details. When all runs share the same sample count it reports a simple per-run average; when run sizes differ it automatically switches to a sample-weighted aggregate (so a small run cannot mask a large one) and flags this via the aggregation / weighted_aggregation_required fields in the details.
  • Stored result payloads expose raw JSON columns such as scores_json, details_json, and metadata_json. Decode these fields before presenting them directly to non-technical users.
  • add_metric records custom metric metadata for discovery, but it does not execute arbitrary user-supplied metric functions. Use the built-in metric registry for executable scoring.
  • Custom metrics registered with add_metric appear in list_metrics only; evaluate will still reject names outside the built-in METRIC_REGISTRY.
  • Regression loss metrics (mse, mae) are lower-is-better and pass when score <= threshold. Classification and exact-match metrics pass when score >= threshold.
  • Block-level errors are returned as failed Result objects. The MCP server wrapper converts validation and runtime failures into degraded response payloads, so production callers should handle both failure shapes.
  • completion_state reports whether the eval operation completed, degraded, or failed; it is not a replacement for the quality-gate passed verdict.
  • warning_card and evidence summarize non-sensitive operation evidence and limitations without including raw predictions or ground-truth sample values.
  • The health op is read-only and reports configured db path, storage mode, WAL status, and row counts. It does not read secrets or execute backend work.
  • MCP surfaces are intentionally split: skill/mcp/server.py exposes a stateless 5-tool Tier 1 surface, while align_evals_mcp/server.py exposes the full persistent Tier 2 surface.
  • NORTH_STAR weakest-link caveat: a complete and passing metric result can still be the wrong measure for the user's intent. Use the evaluation profile, threshold rationale, domain review, or a semantic evaluator when closed-form metrics do not capture the real task.

Public API

AlignEvalsBlock(AIBlock[EvalInput, EvalOutput, None])

Evaluation harness that computes classification and regression metrics.

Field Type Default
name str 'align_evals'
resource_bounds ResourceBounds \| None None
usage ResourceUsage field(default_factory=ResourceUsage)

Methods:

infer(data: EvalInput) -> Result[EvalOutput]

EvalInput(BaseModel)

Input to AlignEvalsBlock.

Field Type Default
predictions list[str \| int \| float] required
ground_truth list[str \| int \| float] required
metrics list[str] Field(default_factory=lambda: ['accuracy'])
threshold float 0.5
threshold_strategy str 'first'
evaluation_profile EvaluationProfile Field(default_factory=EvaluationProfile)

EvalOutput(BaseModel)

Field Type Default
scores dict[str, float] required
confidence_intervals dict[str, tuple[float, float]] Field(default_factory=dict)
confidence float 0.0
passed bool required
n_samples int required
details list[str] Field(default_factory=list)
next_steps list[str] Field(default_factory=list)
degraded bool False
degradation_reason str ''
completion_state CompletionState 'qualified-draft'
warning_card dict[str, Any] Field(default_factory=dict)
evidence list[dict[str, Any]] Field(default_factory=list)

Methods:

model_post_init(__context: Any) -> None

AlignEvalsMCPBlock(AIBlock[MCPEvalsInput, MCPEvalsOutput, dict])

Field Type Default
name str 'align_evals_mcp'
state dict \| None None
db_path str ':memory:'
resource_bounds ResourceBounds \| None None
usage ResourceUsage field(default_factory=ResourceUsage)

Methods:

infer(inp: MCPEvalsInput) -> Result[MCPEvalsOutput]

MCPEvalsInput(BaseModel)

Field Type Default
op EvalsMCPOp required
predictions list[Any] Field(default_factory=list)
ground_truth list[Any] Field(default_factory=list)
metrics list[str] Field(default_factory=list)
threshold float 0.5
result_id str \| None None
result_name str \| None None
benchmark_id str \| None None
benchmark_name str \| None None
eval_set_id str \| None None
eval_set_name str \| None None
metric_name str \| None None
metric_fn str \| None None
run_name str \| None None
score float \| None None
baseline_name str \| None None
data list[dict[str, Any]] Field(default_factory=list)
content str \| None None
query str \| None None
limit int 20
tags list[str] Field(default_factory=list)
metadata dict[str, Any] Field(default_factory=dict)

MCPEvalsOutput(BaseModel)

Field Type Default
op str required
success bool required
scores dict[str, float] Field(default_factory=dict)
passed bool False
n_samples int 0
details dict[str, Any] Field(default_factory=dict)
result_id str \| None None
result dict[str, Any] \| None None
results list[dict[str, Any]] Field(default_factory=list)
benchmark_id str \| None None
benchmark dict[str, Any] \| None None
benchmarks list[dict[str, Any]] Field(default_factory=list)
eval_set_id str \| None None
eval_set dict[str, Any] \| None None
eval_sets list[dict[str, Any]] Field(default_factory=list)
progress list[dict[str, Any]] Field(default_factory=list)
metrics list[dict[str, Any]] Field(default_factory=list)
report str ''
count int 0
stats dict[str, Any] Field(default_factory=dict)
message str ''
error str ''
degraded bool False
degradation_reason str ''
completion_state CompletionState 'qualified-draft'
warning_card dict[str, Any] Field(default_factory=dict)
evidence list[dict[str, Any]] Field(default_factory=list)

Methods:

model_post_init(__context: Any) -> None

MCP Tools

Operation Source
evaluate align_evals_mcp
batch_evaluate align_evals_mcp
compare align_evals_mcp
summarize align_evals_mcp
report align_evals_mcp
store_result align_evals_mcp
get_result align_evals_mcp
list_results align_evals_mcp
delete_result align_evals_mcp
store_benchmark align_evals_mcp
get_benchmark align_evals_mcp
list_benchmarks align_evals_mcp
create_eval_set align_evals_mcp
get_eval_set align_evals_mcp
list_eval_sets align_evals_mcp
track_progress align_evals_mcp
get_progress align_evals_mcp
compute_baseline align_evals_mcp
compare_to_baseline align_evals_mcp
add_metric align_evals_mcp
list_metrics align_evals_mcp
export_report align_evals_mcp
import_eval_data align_evals_mcp
stats align_evals_mcp
info align_evals_mcp
health align_evals_mcp
list_patterns align_evals_mcp
verified align_evals_mcp
qualified-draft align_evals_mcp
blocked-escalated align_evals_mcp