Research Pipeline¶

Build a six-stage research pipeline that searches the web, scrapes primary sources, builds a persistent retrieval index, grounds claims against an indexed corpus, synthesises findings with Claude, and proves logical consistency via propositional DPLL.

Illustrative end-to-end example

Every component and operation named here is real, but this is a worked illustration of how G6 composes — not a copy-paste script. Most steps run offline against built-in defaults; agent_claude needs a paid API key (set ALLOW_PAID_API=1 plus a provider key), and the formal_methods proof needs a solver backend (the built-in propositional solver, or Z3) available in your deployment.

GoalInput¶

{
  "goal": "Produce a grounded analysis of memory-safe systems programming with formally verified conclusions",
  "context": "Survey Rust, Ada/SPARK, and verified C subsets. Target audience: systems architects evaluating language adoption. All claims must be grounded against the ai_ml and engineering knowledge bases.",
  "constraints": [
    "Minimum 15 web sources scraped and indexed",
    "RAG retrieval must use BM25 ranking",
    "Every factual claim must have grounding confidence >= 0.70",
    "Final synthesis must be logically consistent (propositional DPLL SAT)"
  ],
  "resource_bounds": {
    "max_execution_seconds": 300,
    "max_tokens_per_hour": 200000
  },
  "checkpoints": [
    {
      "name": "grounding_confidence_gate",
      "predicate": "metric_above",
      "params": {"metric": "grounding_confidence", "threshold": 0.70},
      "description": "Log warning if grounding confidence falls below 0.70"
    }
  ],
  "subtasks": [
    {
      "goal": "Search for primary sources on memory-safe systems languages",
      "context": "Use ctx_search with DuckDuckGo engine. Cover Rust borrow checker, Ada/SPARK GNAT prover, CompCert verified C, and recent memory safety mandates.",
      "constraints": ["max_results: 20", "engine: duckduckgo"]
    },
    {
      "goal": "Scrape and extract structured content from top sources",
      "context": "Use ctx_scrapling to fetch full-text content. Apply CSS selector filtering where available, fall back to regex HTML stripping.",
      "constraints": ["Extract title, body text, and publication date", "Skip paywalled or empty responses"]
    },
    {
      "goal": "Build a persistent RAG index with BM25 retrieval",
      "context": "Use ctx_rag to ingest scraped documents. Chunk by sentence boundaries, index with TF-IDF, enable BM25 ranking via rank_bm25 for query-time retrieval.",
      "constraints": ["Chunk size: sentence-level", "BM25 top_k: 10"]
    },
    {
      "goal": "Ground all factual claims against the indexed knowledge bases",
      "context": "Use grounding component with domains ai_ml and engineering. Each claim from the synthesis must have a grounding score. Use ctx_colbert as a secondary retrieval path for low-confidence claims.",
      "constraints": ["Minimum grounding confidence: 0.70", "Flag ungrounded claims for manual review"]
    },
    {
      "goal": "Synthesise findings into a structured analysis",
      "context": "Use agent_claude to produce a coherent report. Input: top RAG chunks + grounding results. Output: structured analysis with sections per language, comparative table, and recommendation.",
      "constraints": ["Include comparative table", "Cite grounding sources inline"]
    },
    {
      "goal": "Prove logical consistency of conclusions",
      "context": "Use formal_methods with propositional DPLL to verify: (rust_memory_safe AND spark_formally_verified AND compcert_proven) -> adoption_recommendation_sound.",
      "constraints": ["Strategy: propositional", "Must return SAT"]
    }
  ]
}

Pipeline Diagram¶

graph TD
    A[ctx_search] -->|URLs + snippets| B[ctx_scrapling]
    B -->|full text| C[ctx_rag / BM25]
    C -->|top chunks| D[grounding]
    D -->|scored claims| E[agent_claude]
    C -->|low-confidence| F[ctx_colbert]
    F -->|re-ranked| D
    E -->|synthesis| G[formal_methods / DPLL]
    G -->|SAT proof| H((Verified Report))

What You Need¶

Tier: Researcher
Components: ctx_search, ctx_scrapling, ctx_rag, ctx_colbert, grounding, agent_claude, formal_methods

Step-by-Step¶

Step 1: Search for Primary Sources¶

{
  "component": "ctx_search",
  "operation": "search",
  "params": {
    "query": "memory safe systems programming Rust Ada SPARK CompCert",
    "engine": "duckduckgo",
    "max_results": 20
  }
}

Returns a list of WebResult objects with URL, title, and snippet. No API key required for DuckDuckGo.

Step 2: Scrape Full Content¶

{
  "component": "ctx_scrapling",
  "operation": "scrape",
  "params": {
    "url": "https://example.com/rust-memory-safety-paper",
    "extract_text": true
  }
}

Scrapling tries native Scrapling first, falls back to httpx + regex HTML stripping. CSS selectors are supported via regex approximation.

Step 3: Ingest into RAG Index¶

{
  "component": "ctx_rag",
  "operation": "add",
  "params": {
    "documents": ["...scraped full-text content..."]
  }
}

Then retrieve with BM25:

{
  "component": "ctx_rag",
  "operation": "retrieve",
  "params": {
    "query": "formal verification of memory safety properties",
    "top_k": 10,
    "method": "bm25"
  }
}

Persistent Index

The RAG index persists in the block's state dict across invocations. Add documents incrementally — they accumulate in the TF-IDF and BM25 indices.

Step 4: Ground Claims¶

{
  "component": "grounding",
  "operation": "ground",
  "params": {
    "query": "Rust's borrow checker eliminates use-after-free vulnerabilities at compile time",
    "domain": "engineering",
    "top_k": 5
  }
}

Returns a confidence score — the mean TF-IDF retrieval similarity between the query and the built-in seed facts (8–10 per domain). It measures retrieval quality, not whether the claim is true: a value below 0.70 just means the seed facts are textually far from the claim, which is a signal to route to ctx_colbert for secondary retrieval and re-grounding.

Step 5: Synthesise with Claude¶

{
  "component": "agent_claude",
  "operation": "infer",
  "params": {
    "messages": [
      {
        "role": "user",
        "content": "Based on the following grounded research context, produce a structured comparative analysis of memory-safe systems languages.\n\n[Retrieved chunks and grounding scores inserted here]"
      }
    ],
    "max_tokens": 4096
  }
}

agent_claude calls the Anthropic API — a paid step, so set ALLOW_PAID_API=1 and provide a key. The other five stages run without external API keys.

Step 6: Prove Logical Consistency¶

{
  "component": "formal_methods",
  "operation": "verify",
  "params": {
    "formula": "(rust_memory_safe AND spark_formally_verified AND compcert_proven) -> adoption_recommendation_sound",
    "solver": "propositional"
  }
}

(solver is one of z3, propositional, prolog, lean; the propositional solver parses the atoms directly from the formula.)

Proofs, Not Opinions — with one caveat

The propositional DPLL solver returns SAT or UNSAT — a deterministic, reproducible result for the formula as written, fundamentally different from an LLM "agreeing" that a conclusion follows. The caveat: it proves the formula you supplied, not that the formula faithfully captures your real argument. Read the formula yourself before relying on it (see Code Review for the full translation-faithfulness caveat).

What Happened¶

G6 orchestrated six components in a pipeline:

ctx_search retrieved 20 web results via DuckDuckGo
ctx_scrapling extracted full-text content from primary sources
ctx_rag built a persistent BM25 retrieval index over the scraped corpus
grounding scored every factual claim against the indexed knowledge bases
agent_claude synthesised a structured comparative analysis
formal_methods proved logical consistency of the final conclusions via DPLL

Each component's output feeds the next. The RAG index and grounding scores persist across sessions — follow-up queries reuse the same index at zero marginal cost.

Why G6 Over a Bare LLM¶

A capable LLM can summarise papers and answer research questions. G6 adds prebuilt pipeline templates with a persistent retrieval index, numeric grounding scores, and formal consistency proofs — all wired together so follow-up queries reuse cached indices instead of starting from scratch. One GoalInput JSON triggers the entire research workflow — no prompt engineering, no glue code, no manual orchestration.