Troubleshooting¶

Common issues and solutions when working with G6Solver.

cognee import raises AttributeError¶

Cause: The cognee package has a broken installation that raises AttributeError (related to starlette) on import, even though it is installed.

Solution: This is a known upstream issue. G6 components that depend on cognee use except Exception: guards (not except ImportError:) and fall back gracefully. No action needed.

# G6 pattern for cognee imports
try:
    import cognee
except Exception:  # Not ImportError -- cognee raises AttributeError
    cognee = None

A tool reports a missing capability or dependency¶

Cause: Components that need heavier libraries (ML, deep learning, document conversion, media) live in extension packages that aren't installed by default. Without the package, the component returns a clear diagnostic (e.g. [MARKITDOWN_ERROR] markitdown is not installed) rather than crashing.

Solution: Install the matching extension package for your tier from the setup page:

pip install g6-ml g6-context --index-url https://packages.g6solver.com/simple/

Each package lists what it provides and which optional binaries (ffmpeg, Z3, Lean, …) it expects. See the setup page for the full list.

FastMCP namespace conflict in pytest¶

Cause: MCP sub-packages use lazy import patterns that can trigger namespace warnings during test discovery.

Solution: This is expected behavior and does not affect functionality. Tests will pass normally. No action needed.

Docker build failures (self-host)¶

Applies only to self-hosted server deployments.

Cause: Missing environment variables or incorrect build context.

Solution: Ensure required environment variables are set:

# Required for Docker deployment
export POSTGRES_PASSWORD=your_password_here

# Build from workspace root (correct context)
docker build -f bases/mvp/mcp/Dockerfile -t g6-mcp .

LLM calls failing silently¶

Cause: Empty or invalid OPENROUTER_API_KEY.

Solution: In the desktop GUI, open Settings -> OpenRouter, paste a valid sk-or-... key, and click Save. This writes OPENROUTER_API_KEY to ~/.g6/.env; the key is not displayed after saving.

For terminal, MCP, REST, or service use, confirm OPENROUTER_API_KEY is set in your environment or ~/.g6/.env. In the GUI, Settings shows whether a key is configured (without displaying it), and g6 --diagnostics reports key and backend status.

If using Ollama locally, ensure the Ollama server is running:

ollama serve

Vision tools return [VISION_NO_KEY] or [COST_WARNING]¶

Cause: ctx_vision uses OpenRouter/litellm vision models for image description, OCR, screenshot analysis, comparison, and batch image processing. These calls can incur third-party API charges, so G6 requires both an API key and explicit cost consent.

Solution: Configure OPENROUTER_API_KEY, then give explicit cost consent — either pass allow_paid_api=True on the specific vision tool call (so consent is visible at the call site) or set ALLOW_PAID_API=1 for the runtime:

# in ~/.g6/.env
OPENROUTER_API_KEY=your_openrouter_key_here
# Optional global opt-in for this runtime only:
ALLOW_PAID_API=1

Self-Training: When to Continue, When to Stop¶

G6's self-training loop is iterative. Knowing when to persist and when to stop is the most important practical skill for getting value from the system.

Continue when the accuracy trajectory is clearly upward — each iteration is producing measurable lift (even if small). The self-training loop typically needs 3–10 cycles to show significant improvement at the T3 level. Early iterations often fix low-hanging fruit (scorer bugs, format mismatches); later iterations address deeper issues (tool gaps, reasoning depth). If you are seeing consistent positive deltas on each iteration, let the loop run.

Stop early when the trajectory is flat or negative after 2–3 iterations. A null result after multiple cycles is a signal, not a failure — it means one of the failure modes below applies. When stopping early:

1. Diagnose which failure mode is responsible (see below)
2. Either try a different approach (different tools, different task decomposition, different T-level) or
3. Document an empirical justification for why G6 tooling cannot improve performance on this task class, together with a hypothesis about what would be needed (e.g., model fine-tuning, a custom component, a different evaluation rubric)

The goal is never to force improvement where it does not exist — it is to understand why and act on that understanding.

G6 Failure Modes and Hyperparameter Selection¶

When G6 does not improve performance, one of the following failure modes is almost always responsible. Diagnosing the correct one determines your next action.

Wrong tool selection¶

If the wrong tools are selected for a task class, G6 may not only fail to improve performance — it may actively hurt it. The additional infrastructure (tool orchestration, prompt injection, output parsing) adds overhead without producing value, and incorrect tool application can steer the model away from correct answers.

Diagnosis: Compare per-task results between baseline (no tools) and G6. If G6 is getting tasks wrong that the baseline gets right, tool selection is the likely culprit.

Fix: Review which tools are being routed to which task categories. Reduce to 2–3 tools per category rather than 10. Remove tools that are not producing value. The self-training loop's failure analysis should identify this, but manual review of the tool-call traces is often faster.

Not enough harness engineering cycles¶

The T3 self-training loop typically needs 3–10 cycles to show significant improvement. Early cycles fix infrastructure issues (scorer bugs, format mismatches — often the highest-value, lowest-cost fixes). Later cycles address tool selection, prompt engineering, and custom producers. Stopping after 1–2 cycles may miss the real gains.

Diagnosis: If accuracy is trending upward but the delta is small, you are likely in the early-infrastructure-fix phase. Continue.

Fix: Run more iterations. Each iteration should target the single worst failure class. If the improvement curve has flattened after 5+ cycles, the remaining issues are likely not addressable through harness engineering alone.

Harness over-fitted to a small training set¶

If the harness has been optimised on too few examples, it may over-fit — performing well on the training set but failing to generalise to unseen tasks. This is the most common hyperparameter error.

Recommended minimum dataset sizes:

Diagnosis: If performance on the training split is high but performance on the held-out evaluation split is low or declining, the harness is over-fitted.

Fix: Increase the training split to at least 25 instances. Use 10–25% of the full dataset for training. Using more than 25% may produce stronger results on the specific benchmark but risks over-fitting to the dataset rather than the task class.

Task not amenable to tool learning¶

Some tasks require capabilities that cannot be provided by tool orchestration — they genuinely need model-level changes (fine-tuning, RLHF, or architectural modifications). SkillsBench is the canonical example: G6 scored 24% vs baseline 28% (n=25, p >> 0.05) because the benchmark evaluates skills that are already provided to the model, leaving no room for tool-based improvement.

Diagnosis: If the task requires capabilities the model fundamentally lacks (e.g., domain-specific knowledge not available through grounding, or perceptual skills like image understanding on a text-only model), tool learning will not help.

Fix: This is a genuine limitation. Document the finding and consider whether fine-tuning, a different base model, or a custom component would address the gap.

Task too simple or already saturated¶

If the base model already achieves near-ceiling performance on a task, G6 has limited room to add value. ARC-AGI v1/v2 is the canonical example: the baseline was already 94–96%, and G6 showed no significant lift. Tool-calling overhead may even cause slight regressions on tasks the model handles well natively.

Diagnosis: If baseline accuracy is above ~85–90%, the task may be saturated for the current model.

Fix: If the goal is cost reduction rather than accuracy improvement, consider using a smaller, less capable model with G6. The harness infrastructure can often compensate for the weaker model, achieving comparable accuracy at lower per-token cost. This is a well-supported use case — GPQA Diamond was validated on local qwen3.5:35b-a3b (a much smaller model than Claude Opus), and the G6 harness provided a +6.0pp lift (full set, n=197; not statistically significant).

Base model not capable enough to use G6 effectively¶

G6's tools require a model that can follow complex tool-calling instructions, interpret structured outputs, and reason about multi-step workflows. If the underlying LLM is not capable enough, it will misuse or ignore the tools.

Rule of thumb: Any current frontier model (Claude Opus/Sonnet, GPT-4+, Gemini Pro+) will give good coverage of G6's capabilities. Mid-tier models (Claude Haiku, GPT-3.5-class) work for simpler workflows but may struggle with complex multi-tool orchestration. Small local models (< 7B parameters) are generally not recommended for G6's full tool suite.

Task does not need T2/T3¶

Not every task benefits from the full self-training loop. Many tasks are well-served by T0 (fixed tool-calling) or T1 (one-time harness optimisation). Running the T3 loop on a task that only needs T0 wastes cycles and may introduce unnecessary complexity.

Diagnosis: If a single round of tool configuration (T1) produces good results, stop there. T2/T3 is for tasks where performance needs to compound over time or adapt to shifting requirements.

Fix: Match the T-level to the problem. See the T0–T3 framework for guidance on which level fits which problem type.

Needs a custom build¶

Some deployments require capabilities beyond G6's standard component library — frontier-level performance on a specific domain, real-world deployment constraints (latency, compliance, integration with proprietary systems), or task classes that need bespoke components.

Fix: Contact the G6 team at [email protected] to discuss a managed deployment or custom build. This is the appropriate path when standard self-training has been exhausted and the remaining gap requires engineering beyond harness optimisation.

Getting help¶

If your issue is not listed here:

1. Check the Configuration page for environment variable requirements
2. Review the Architecture pages for system design context
3. Run g6 --diagnostics to collect a diagnostic bundle, then contact support at [email protected] with the error message and reproduction steps