Skip to content

Data Processing

Structured data analysis, schema-enforced extraction, and background task queues -- the components that transform raw data into actionable inputs.

Overview

Data Processing is a compact cluster of three components that handle structured data workflows. adapt_pandas is the most mature, offering a two-tier architecture: Tier 1 provides eight core DataFrame operations via AdaptPandasBlock, while Tier 2 exposes 26 MCP tools through AdaptPandasMCPBlock with SQLite-backed persistence across five tables.

adapt_instructor provides LLM-powered structured extraction using schema definitions, ensuring that unstructured text is parsed into validated Pydantic models. adapt_rq manages background task queues for long-running data processing jobs that should not block the main agent loop.

The cluster's SQLite persistence (configured via PANDAS_DB_PATH, defaulting to ~/.pandas_mcp/pandas.db) means that DataFrames, analysis results, and job state survive process restarts -- important for iterative data exploration sessions.

Components

Component Description MCP Tools
adapt_pandas DataFrame operations, profiling, dataset management 26
adapt_instructor Schema-enforced LLM extraction --
adapt_rq Background task queue management --

Architecture

graph TD
    PANDAS[adapt_pandas] --> TIER1[Tier 1: AdaptPandasBlock<br/>8 core ops]
    PANDAS --> TIER2[Tier 2: pandas_mcp<br/>25 MCP tools]
    TIER2 --> STORE[PandasStore<br/>SQLite 5 tables]
    INST[adapt_instructor] --> LLM[llm_router]
    RQ[adapt_rq] --> PANDAS
    RQ --> INST

Key Patterns

Two-Tier MCP Architecture. adapt_pandas separates backward-compatible block operations (Tier 1) from the full MCP tool surface (Tier 2). Tier 1 handles load, describe, filter, transform, aggregate, join, pivot, and export. Tier 2 adds advanced analytics, profiling, dataset management, pipeline operations, and search.

SQLite Persistence. The PandasStore persists DataFrames and metadata to SQLite, keyed by dataset name. This enables multi-turn data exploration where an agent can load a CSV in one turn, filter it in another, and visualize it in a third -- all without re-ingestion.

Structured Extraction. adapt_instructor bridges unstructured and structured data by using LLM calls with Pydantic model schemas as output constraints. This is particularly useful for extracting tabular data from natural language descriptions or semi-structured documents.