Skip to content

Multimodal

multimodal — mvp.multimodal

Cluster: Multimodal | Type: component | MCP Tools: None

Overview

Cross-modal translation and generation router that dispatches between registered text, image, audio, video, 3D mesh, and animation routes. The component provides the unified schema, route registry, capability catalogue, zero-cost local preview/summary routes, and optional premium/pro adapter routes for richer media backends such as adapt_image, adapt_voice, ComfyUI, OpenRouter, and Blender. Clean installs return useful local outputs; configured services improve quality when explicitly requested.

When to use:

  • Rendering a zero-cost text prompt card as PNG/SVG when no paid image backend is configured
  • Summarising image inputs locally by source, size, and dimensions where available
  • Rendering a lightweight local mesh/scene preview from metadata
  • Returning local WAV preview tones and audio metadata summaries when speech backends are not configured
  • Routing richer text-to-image, image-to-text, speech, and 3D-render tasks to configured premium/pro adapters when a backend is requested

Example:

from mvp.multimodal import MultimodalBlock, MediaInput

block = MultimodalBlock(name="multimodal")
result = block.infer(MediaInput(
    source_type="text",
    media_type="image",
    prompt="Workflow passed 12/13 checks; one deployment warning remains.",
))
# result.value.backend -> "local"; result.value.metadata["output_role"] -> "preview"

Production caveats:

  • When no backend is specified, route selection prefers the lowest currently estimated cost and skips unavailable routes. Use backend="openrouter", backend="comfyui", or backend="adapt_voice" when quality matters more than zero-cost local output.
  • Non-local route prices are deployment estimates, with environment-variable overrides such as G6_MULTIMODAL_COST_OPENROUTER_IMAGE; they are operational routing metadata, not invoice-grade billing records.
  • Forced backends are preflighted before execution and fail with setup guidance if keys, URLs, or optional packages are unavailable.
  • Optional premium/pro backends:
  • backend="openrouter": set OPENROUTER_API_KEY for model-generated images and vision.
  • backend="comfyui": set COMFYUI_URL for ComfyUI image generation.
  • backend="adapt_voice": install/configure voice engines or relevant API keys for TTS/STT.
  • backend="blender": install Blender and put blender on PATH, or set BLENDER_PATH, for real 3D rendering.
  • Local media routes set metadata.output_role (preview, audio_preview, or metadata_summary) so user interfaces can label them honestly.
  • Local preview routes are deterministic production previews, not photorealistic generation, speech synthesis, transcription, or physically accurate 3D rendering.
  • MultimodalBlock(router=TranslationRouter()) and TranslationBlock(router=...) support workflow-local route registries for test isolation and multi-tenant services.

Operational notes:

  • Premium/pro API backends may transmit prompts, images, or audio to third-party providers. Ask for user consent and use allow_paid_api=True where the downstream adapter requires it.
  • Outputs are returned as base64 strings; large images, audio, or renders can become memory-heavy in long agent chains. Persist large artifacts to files or object storage at the workflow boundary.
  • Blender rendering uses local subprocesses and session/temp files. In hosted deployments, run it in a constrained worker with explicit CPU, memory, timeout, and filesystem cleanup policy.
  • Route prices are routing estimates. Billing, quotas, and customer-facing cost reports should use provider invoices or the product billing ledger, not estimated_cost_usd alone.

Works well with: sensory_fusion, ctx_rag, integration

Public API

MultimodalModelInfo

Describes multimodal capabilities of a model.

Field Type Default
model_id str required
provider str ''
supports_vision bool False
supports_audio bool False
supports_image_gen bool False
supports_video bool False
supports_tts bool False
supports_stt bool False
max_image_size int 0
max_audio_duration_sec float 0.0
input_modalities list[str] field(default_factory=list)
output_modalities list[str] field(default_factory=list)

MultimodalBlock(AIBlock[MediaInput, MediaOutput, dict])

Unified facade for multimodal translation, model info, and routing.

Field Type Default
name str 'multimodal'
resource_bounds ResourceBounds \| None None
usage ResourceUsage field(default_factory=ResourceUsage)
auto_register_routes bool True
router TranslationRouter \| None None

Methods:

infer(data: MediaInput) -> Result[MediaOutput]

MediaInput(BaseModel)

Input for multimodal translation / generation.

Field Type Default
op str 'translate'
media_type MEDIA_TYPE_LITERAL 'text'
source_type MEDIA_TYPE_LITERAL 'text'
mime_type str ''
data_base64 str ''
data_url str ''
text str ''
prompt str ''
negative_prompt str ''
width int 0
height int 0
duration_seconds float 0.0
model str ''
output_format str ''
backend str ''
allow_paid_api bool False
metadata dict[str, Any] \| None None

MediaOutput(BaseModel)

Output from multimodal translation / generation.

Field Type Default
media_type str 'text'
mime_type str ''
data_base64 str ''
output_path str ''
text str ''
backend str ''
tokens_used int 0
cost_usd float 0.0
metadata dict[str, Any] \| None None
error str ''
degraded bool False
degradation_reason str \| None None
completion_state COMPLETION_STATE_LITERAL 'qualified-draft'
warning_card dict[str, Any] \| None None
evidence dict[str, Any] Field(default_factory=dict)
request_id str \| None None
task_id str \| None None
run_id str \| None None

TranslationRoute

Maps a source→target modality pair to a handler function.

Field Type Default
source_type str required
target_type str required
handler Callable[[MediaInput], Result[MediaOutput]] required
backend str ''
priority int 0
estimated_cost_usd float 0.0
quality int 50
local bool True
description str ''
pricing_env_var str ''
pricing_source str 'static'
availability_check Callable[[], tuple[bool, str]] \| None None
plan_tier str 'standard'

Methods:

current_cost_usd() -> float

Return current estimated cost, allowing deployment-time overrides.

availability() -> tuple[bool, str]

Return whether the route backend is currently invokable.

TranslationRouter

Registry of cross-modal translation routes.

Methods:

register(route: TranslationRoute) -> None

Register a translation route.

register_once(route: TranslationRoute) -> None

Register a route unless the same source/target/backend already exists.

find_route(source_type: str, target_type: str, backend: str = '') -> TranslationRoute | None

Find the best route for a source→target translation.

list_routes() -> list[dict[str, str]]

Return all registered routes.

TranslationBlock(AIBlock[MediaInput, MediaOutput, None])

Routes cross-modal translations to registered backend blocks.

Field Type Default
name str 'translation'
resource_bounds ResourceBounds \| None None
usage ResourceUsage field(default_factory=ResourceUsage)
auto_register_routes bool True
router TranslationRouter \| None None

Methods:

infer(data: MediaInput) -> Result[MediaOutput]

Functions

get_model_info(model_id: str) -> MultimodalModelInfo | None

Look up model capabilities by ID, tolerating served spellings.

get_model_info_or_unknown(model_id: str) -> MultimodalModelInfo

Return model info, falling back to an unknown-capabilities entry.

list_vision_models() -> list[str]

Return model IDs that support vision input.

list_image_gen_models() -> list[str]

Return model IDs that support image generation.

get_router() -> TranslationRouter

Return the global translation router.