Multimodal Product Search¶

Build a cross-modal product search that understands text, images, and audio queries, enabling retrieval across modalities via a shared embedding space. This recipe requires a purpose-built multimodal embedding model or external encoder for production semantic alignment; adapt_trm can help with deterministic feature comparison, persistence, clustering, and smoke checks over numeric vectors, but it is not the shared embedding model.

Illustrative example — G6 orchestrates around external models

This shows how G6's media components compose; every G6 component and operation named below is real, but the shared embedding model and the vector index are external (clearly marked) — G6 does not ship a trained cross-modal encoder or a vector database. Treat it as a worked illustration of G6's role (preprocessing, feature extraction, transcription, deterministic checks, orchestration), not a copy-paste script.

GoalInput¶

{
  "goal": "Build a cross-modal product search that retrieves results across text, image, and audio queries",
  "context": "E-commerce catalogue with 50k product SKUs. Each product has a title, description, up to 5 photos, and optional vendor audio clips. Customers search using typed queries, photo uploads (visual search), and voice messages. The system must return relevant products regardless of which modality the query arrives in.",
  "constraints": [
    "All three modalities (text, image, audio) must map to a shared embedding space using an external or separately trained embedding model",
    "Cross-modal retrieval recall@10 must exceed 0.80",
    "Image preprocessing must normalise resolution to 224x224 before feature extraction",
    "Audio transcription must handle accented speech with WER below 15%",
    "The embedding model must be validated separately from G6 orchestration"
  ],
  "resource_bounds": {
    "max_execution_seconds": 600,
    "max_tokens_per_hour": 250000
  },
  "subtasks": [
    {
      "goal": "Preprocess and normalise product photos",
      "context": "Use adapt_image to resize, crop, and normalise product photos to 224x224 RGB tensors. Apply background removal for clean feature extraction. Handle varying aspect ratios via centre-crop with padding.",
      "constraints": ["Output: 224x224 normalised tensors", "Preserve aspect ratio with centre-crop"]
    },
    {
      "goal": "Extract visual features from product images",
      "context": "Use ctx_vision to run OCR on product labels, detect objects in product photos, and extract feature vectors. OCR captures text printed on packaging; object detection identifies product category cues.",
      "constraints": ["OCR confidence threshold: 0.85", "Object detection: top 5 labels per image"]
    },
    {
      "goal": "Transcribe voice queries to text",
      "context": "Use adapt_voice (stt_transcribe) to convert customer voice messages to text. Handle background noise, accented speech, and multilingual queries. Output normalised transcripts ready for embedding.",
      "constraints": ["WER below 15%", "Support at least 5 languages", "Trim silence and normalise volume"]
    },
    {
      "goal": "Encode all modalities with a production embedding model",
      "context": "Use an external or separately trained multimodal encoder to map text queries, image feature vectors, and audio transcripts into a shared embedding space. Use adapt_trm only for deterministic checks over supplied numeric vectors, such as comparing candidate embeddings, clustering examples, or storing pilot feature sequences.",
      "constraints": ["Embedding dimension chosen by the external model", "Batch encoding for catalogue indexing", "Cosine similarity for retrieval scoring", "Do not rely on adapt_trm as the cross-modal encoder"]
    },
    {
      "goal": "Build shared embedding space for cross-modal retrieval",
      "context": "Use multimodal to fuse text, image, and audio embeddings into a shared vector space. Index the full product catalogue. At query time, encode the incoming query (any modality) and retrieve top-k nearest neighbours across all indexed products.",
      "constraints": ["Shared space must support all three modality pairs", "Top-k retrieval with k=10", "Recall@10 >= 0.80"]
    }
  ]
}

Pipeline Diagram¶

graph TD
    A[adapt_image<br/>resize] -->|224x224 images| B[ctx_vision<br/>OCR + object detection]
    B -->|visual features| D[external encoder<br/>shared embedding model]
    C[adapt_voice<br/>stt_transcribe] -->|transcripts| D
    T[Text Queries] -->|raw text| D
    D -->|shared embeddings| E[your vector index<br/>NN search]
    E -->|candidate vectors| G[adapt_trm<br/>local checks + storage]
    E --> F((Cross-Modal Search))

What You Need¶

Tier: Builder
Components: adapt_image, ctx_vision, adapt_voice, adapt_trm (plus your own external embedding model and vector index)

Step-by-Step¶

Step 1: Preprocess Product Photos¶

{
  "component": "adapt_image",
  "operation": "resize",
  "params": {
    "image_path": "/data/catalogue/photos/SKU-12345-01.jpg",
    "target_size": [224, 224],
    "keep_aspect": true
  }
}

Resizes each product photo to 224x224 for downstream feature extraction. adapt_image also exposes crop (centre-crop to preserve aspect ratio), adjust_color, and convert_format — chain them as needed. (There is no built-in background-removal op; use an external tool if you need it.)

Step 2: Extract Visual Features¶

{
  "component": "ctx_vision",
  "operation": "extract_text",
  "params": {
    "image_path": "/data/catalogue/photos/SKU-12345-01.jpg",
    "min_confidence": 0.85
  }
}

extract_text runs OCR; call detect_objects on the same image for category cues (ctx_vision also offers classify, describe, and answer_question). OCR captures text visible on product packaging (brand names, ingredient lists, model numbers). Object detection identifies category cues — a camera lens, a shoe sole, a circuit board — that help disambiguate visually similar products.

OCR + Detection Synergy

OCR alone misses visual context (a red shoe vs. a blue shoe). Object detection alone misses text (brand names, sizes). Combining both produces a richer feature vector that captures what the product is and what it says.

Step 3: Transcribe Voice Queries¶

{
  "component": "adapt_voice",
  "operation": "stt_transcribe",
  "params": {
    "audio_path": "/data/queries/voice/query-98765.wav",
    "language": "auto"
  }
}

Speech-to-text lives in adapt_voice (stt_transcribe), not adapt_audio (which handles format conversion, BPM/spectral analysis, and source separation). It converts raw audio to text transcripts; if you need to pre-clean the waveform first, adapt_audio's normalize_audio / source_separate ops can run before transcription.

Multilingual Fallback

Set language: "auto" to let the transcription model detect the spoken language. For catalogues serving multiple markets, this avoids forcing customers into a single language and improves recall for non-English voice queries.

Step 4: Encode with a Production Embedding Model¶

external_multimodal_encoder is not a G6 component — it is a placeholder for your own production model (CLIP-style, or per-modality projection heads). The JSON below shows the conceptual interface, not a G6 invoke_component call.

{
  "component": "external_multimodal_encoder",
  "operation": "encode",
  "params": {
    "inputs": [
      {"modality": "text", "content": "wireless noise-cancelling headphones"},
      {"modality": "image", "content": "...visual feature vector from ctx_vision..."},
      {"modality": "text", "content": "...transcript from adapt_audio..."}
    ],
    "embedding_dim": "model_default",
    "batch_mode": true
  }
}

A production multimodal encoder maps all three modalities into the same vector space. Per-modality projection heads or a CLIP-style architecture align the representations so that semantically similar items cluster together regardless of whether they arrived as text, image features, or audio transcripts.

adapt_trm is not the encoder

adapt_trm does not ship a trained transformer, CLIP-style model, or learned cross-modal projection heads. Use it after encoding when you need deterministic local checks, lightweight sequence embeddings over supplied numeric features, nearest-neighbor comparisons, clustering, or persistent pilot examples.

{
  "component": "your_vector_index",
  "operation": "search",
  "params": {
    "query_embedding": "...shared embedding vector from your external encoder...",
    "index": "product_catalogue",
    "top_k": 10,
    "similarity": "cosine"
  }
}

Nearest-neighbour search runs over your vector store (FAISS, pgvector, etc.) populated by Step 4's encoder — the G6 multimodal component is a simulation-only translation/model-routing block and does not do retrieval. For text-only retrieval, G6's ctx_rag (retrieve) gives you a built-in TF-IDF/BM25 surface. Index the full catalogue once (text descriptions + image features + audio metadata all encoded via Step 4). At query time, encode the incoming query — typed text, a photo upload, or a voice message — and retrieve the top 10 nearest neighbours by cosine similarity across the shared space.

Cross-Modal Retrieval in Practice

A customer photographs a pair of running shoes and the system returns matching products that were indexed from text descriptions alone. A voice query saying "red leather handbag with gold buckle" retrieves products whose photos match, even if the text description says "crimson purse with brass clasp". The shared embedding space bridges vocabulary and modality gaps.

What Happened¶

G6 orchestrated five components in a multimodal pipeline:

adapt_image resized raw product photos to 224x224 for feature extraction
ctx_vision extracted OCR text and object-detection labels from product images
adapt_voice transcribed voice queries into clean text transcripts (stt_transcribe)
your external embedding model encoded all modalities into a shared vector space
adapt_trm optionally compared, clustered, or stored supplied numeric vectors for deterministic local checks during pilot workflows
your vector index performed cross-modal nearest-neighbour retrieval over those embeddings — G6 orchestrates, validates, and persists around it

Each modality feeds through its own preprocessing path before converging at the embedding model. The shared embedding space means a single query in any modality searches the entire catalogue regardless of how each product was originally indexed. G6's value here is orchestration, validation, persistence, retrieval, and repeatable workflow execution around the model, not pretending that adapt_trm alone solves cross-modal representation learning.

Why G6 Over a Bare LLM¶

A capable LLM can describe images and transcribe audio. G6 adds persistent retrieval infrastructure — vector indices over large catalogues, cross-modal embedding spaces, and cosine similarity search with measurable recall thresholds. Prebuilt pipeline templates compose image preprocessing, feature extraction, audio transcription, and vector retrieval into a reusable workflow triggered by one GoalInput JSON — no re-ingestion on every query.