Physical AI — Robotic Pick-and-Place¶

Plan and simulate a robotic pick-and-place pipeline for warehouse automation, from body schema definition through grasp simulation to trajectory generation.

Simulation-only capability

The Physical AI components in this build are simulation and planning tools. They do not include real robot, PLC, GPIO, camera, lidar, ROS, serial, Modbus, or hardware-safety drivers. tactile_fusion consumes caller-supplied arrays and uses heuristic contact/slip estimates; it does not read real tactile sensors, estimate validated friction coefficients, or provide safety-rated slip detection. Treat generated trajectories, G-code, tactile outputs, and device commands as software artifacts that require human review and a separate validated hardware integration layer before use on physical equipment.

GoalInput¶

{
  "goal": "Plan and simulate a robotic pick-and-place pipeline for warehouse automation",
  "context": "A 6-DOF robotic arm with a parallel-jaw gripper must pick irregular parcels from an inbound conveyor and place them into outbound totes. The workspace includes simulated RGB-D, wrist force/torque, and tactile sensor data. All grasp candidates must be simulated before reviewed trajectory artifacts are produced.",
  "constraints": [
    "Advisory physics risk score must stay below 0.35 before trajectory artifacts are reviewed",
    "Kinematic chain must be validated against joint limits before planning",
    "Force sensor readings must be fused with depth data within 10 ms latency budget",
    "Trajectory must avoid collision with conveyor frame and neighbouring parcels"
  ],
  "resource_bounds": {
    "max_execution_seconds": 240,
    "max_tokens_per_hour": 150000
  },
  "guardrails": [
    {
      "name": "physics_risk_gate",
      "predicate": "metric_above",
      "params": {"metric": "physics_risk", "threshold": 0.35},
      "message": "Halt review if advisory physics risk exceeds 0.35"
    }
  ],
  "breakpoints": [
    {
      "name": "pre_execution_review",
      "description": "Pause for human review before any trajectory artifact is handed to a separate validated hardware integration",
      "active": true
    }
  ],
  "subtasks": [
    {
      "goal": "Define the robot body schema and kinematic chain",
      "context": "Use embodiment to specify the 6-DOF arm: link lengths, joint types (revolute), joint limits, DH parameters, and end-effector (parallel-jaw gripper with 80 mm max aperture).",
      "constraints": ["All joints must have defined position and velocity limits", "End-effector aperture must be parameterised"]
    },
    {
      "goal": "Generate prototype 3D geometry for target objects",
      "context": "Use mesh3d to create software-only bounding meshes for parcels from pre-segmented point cloud samples. Treat these as conservative planning artifacts, not production geometry.",
      "constraints": ["Use indexed triangle mesh arrays", "Verify geometry with specialist tooling before real collision checking or manufacturing"]
    },
    {
      "goal": "Fuse RGB, depth, and force sensor data into a unified scene representation",
      "context": "Use sensory_fusion to combine the overhead RGB-D camera feed with the wrist force/torque sensor. Produce a registered point cloud with per-point force annotations.",
      "constraints": ["Sensor registration error must be below 2 mm", "Latency budget: 10 ms per fusion cycle"]
    },
    {
      "goal": "Process gripper contact patterns from tactile arrays",
      "context": "Use tactile_fusion to interpret a caller-supplied pressure distribution across simulated gripper tactile pads. Detect contact, compute a spatial slip heuristic, classify coarse surface texture, and estimate grip stability.",
      "constraints": ["Treat slip detection as a spatial heuristic, not temporal slip tracking", "Do not output friction coefficients or safety decisions from tactile_fusion alone"]
    },
    {
      "goal": "Estimate contact, breakage, and stability risk for candidate grasps",
      "context": "Use physics_prediction for deterministic planning estimates: contact-force checks, stack stability, breakage risk, and interpolated joint waypoints. Treat object meshes, gripper geometry, candidate poses, and friction estimates as reviewed inputs to a separate validated simulator when high-fidelity grasp scoring is required.",
      "constraints": ["Advisory risk score must stay below 0.35", "Review at least 5 candidate grasp poses per object in a dedicated simulator before hardware use"]
    },
    {
      "goal": "Generate collision-free trajectory commands for the best grasp",
      "context": "Use motor_control to gate a proposed joint-space trajectory artifact from the home position to the pre-grasp pose, then a Cartesian approach to the grasp pose, followed by a retract-and-place motion. Treat any planner output as simulated and review-only.",
      "constraints": ["Trajectory must respect joint velocity limits", "Collision margin: 20 mm clearance from obstacles"]
    },
    {
      "goal": "Orchestrate the full sense-plan-act loop",
      "context": "Use adapt_physical_ai to wire the simulated pipeline: perceive (sensory_fusion + tactile_fusion) -> plan (physics_prediction + motor_control) -> produce reviewed trajectory artifacts. Run the loop continuously for each parcel in the simulated conveyor scene.",
      "constraints": ["Cycle time target: under 4 seconds per pick-and-place", "Abort and re-plan if grasp probability drops below threshold mid-execution"]
    }
  ]
}

Pipeline Diagram¶

graph TD
    A[embodiment<br/>body schema + kinematics] --> G[adapt_physical_ai<br/>sense-plan-act orchestrator]
    B[mesh3d<br/>object 3D geometry] --> E[physics_prediction<br/>planning estimates]
    C[sensory_fusion<br/>RGB + depth + force] --> E
    D[tactile_fusion<br/>contact patterns] --> E
    E -->|grasp candidates + probabilities| F[motor_control<br/>action gating]
    A -->|joint limits + DH params| F
    F -->|reviewed trajectory artifacts| G
    G --> H((Simulated Pick-and-Place<br/>Artifacts))

What You Need¶

Tier: Builder
Components: adapt_physical_ai, motor_control, embodiment, physics_prediction, sensory_fusion, tactile_fusion, mesh3d
Optional dependencies for higher fidelity prototypes: trimesh for advanced mesh operations, open3d for point-cloud registration, cadquery for parametric CAD, and genesis-world for Genesis-backed physics. Without them, fallback simulation modes are intentionally limited. Even with them installed, this use case remains a reviewed simulation workflow rather than a production robotics or fabrication stack.

Step-by-Step¶

Step 1: Define the Robot Body Schema¶

{
  "component": "embodiment",
  "operation": "set_base_type",
  "params": {
    "name": "warehouse_arm_6dof",
    "joints": [
      {"name": "shoulder_pan",  "type": "revolute", "limits": [-3.14, 3.14], "max_velocity": 2.0},
      {"name": "shoulder_lift", "type": "revolute", "limits": [-1.57, 1.57], "max_velocity": 2.0},
      {"name": "elbow",         "type": "revolute", "limits": [-2.35, 2.35], "max_velocity": 3.0},
      {"name": "wrist_1",       "type": "revolute", "limits": [-3.14, 3.14], "max_velocity": 3.5},
      {"name": "wrist_2",       "type": "revolute", "limits": [-3.14, 3.14], "max_velocity": 3.5},
      {"name": "wrist_3",       "type": "revolute", "limits": [-3.14, 3.14], "max_velocity": 3.5}
    ],
    "end_effector": {
      "type": "parallel_jaw_gripper",
      "max_aperture_mm": 80,
      "max_force_n": 40
    }
  }
}

Returns the full kinematic chain with DH parameters, link lengths, and forward kinematics solver. This schema is referenced by motor_control for trajectory validation. (embodiment builds the body incrementally — set_base_type establishes the chassis, then register_actuator / register_sensor add each joint and sensor; the single declarative call above is condensed for illustration.)

Step 2: Generate Prototype Object Geometry¶

{
  "component": "mesh3d",
  "operation": "point_cloud_to_mesh",
  "params": {
    "points": "... pre-segmented RGB-D point samples for parcel ...",
    "method": "convex_hull"
  }
}

Produces a triangle mesh approximation for each parcel when trimesh or open3d is available. The convex hull approximation is conservative and may overestimate collision volume. Use it for simulation and planning review only; validate geometry with a dedicated CAD/robotics stack before any real collision checking, grasp execution, or manufacturing workflow.

Step 3: Fuse Sensor Data¶

{
  "component": "sensory_fusion",
  "operation": "fuse",
  "params": {
    "sensors": [
      {"type": "rgb_d", "source": "overhead_camera", "frame_id": "camera_link"},
      {"type": "force_torque", "source": "wrist_ft_sensor", "frame_id": "wrist_link"}
    ],
    "registration_method": "extrinsic_calibration",
    "max_latency_ms": 10
  }
}

Returns a registered point cloud with per-point force annotations. The extrinsic calibration uses the known camera-to-wrist transform to align coordinate frames.

Sensor Registration

Pre-compute the camera-to-base transform offline using a checkerboard calibration. At runtime, sensory_fusion applies the cached transform — no iterative alignment needed, keeping latency under 10 ms.

Step 4: Process Tactile Contact Patterns¶

{
  "component": "tactile_fusion",
  "operation": "detect_contact",
  "params": {
    "taxel_grid": "... 16x16 pressure matrix supplied by the caller ...",
    "contact_threshold": 0.5
  }
}

Returns contact status, center of pressure, contact area, and a spatial slip heuristic. Use separate estimate_grip_stability and classify_surface calls for grip and texture signals. These are advisory simulation features, not validated tactile sensing or friction estimation.

Tactile caveat

tactile_fusion can help prototype how a workflow might react to pressure-distribution patterns, but it does not perform hardware acquisition, temporal slip-window analysis, or friction-coefficient estimation. For real grasp control, use validated tactile hardware, calibrated datasets, a dedicated robotics simulator, and an independently reviewed safety controller.

Step 5: Estimate Contact and Stability Risk¶

{
  "component": "physics_prediction",
  "operation": "assess_risk",
  "params": {
    "objects": [
      {"name": "parcel", "mass": 1.8, "position": [0.0, 0.0, 0.12], "support_base": [0.32, 0.24]}
    ],
    "force_vector": [0, 0, 18.0],
    "penetration_depth": 0.002,
    "relative_velocity": 0.1,
    "material": "plastic",
    "object_class": "parcel",
    "action_type": "grasp",
    "breakage_force_n": 120.0
  }
}

Returns advisory stability, breakage, and contact-force estimates. For high-fidelity per-grasp success probabilities or contact wrenches, export the reviewed scene to a dedicated robotics simulator or validated hardware stack.

Step 6: Gate Trajectory Artifacts¶

{
  "component": "motor_control",
  "operation": "plan_action",
  "params": {
    "kinematic_chain": "... from embodiment body schema ...",
    "start_config": [0, -1.57, 0, 0, 0, 0],
    "grasp_pose": {"position": [0.45, 0.12, 0.08], "orientation": [0, 0, 0, 1]},
    "place_pose": {"position": [0.60, -0.30, 0.15], "orientation": [0, 0, 0, 1]},
    "planner": "rrt_star",
    "collision_objects": ["... conveyor frame mesh ...", "... neighbouring parcel meshes ..."],
    "collision_margin_mm": 20,
    "phases": ["approach", "grasp", "retract", "transit", "place", "release", "home"]
  }
}

Returns a reviewed trajectory artifact with waypoints for each phase. This is not physical execution: any real trajectory planner, robot controller, PLC, CNC controller, or ROS bridge must be supplied and validated separately.

Step 7: Orchestrate the Sense-Plan-Act Loop¶

{
  "component": "adapt_physical_ai",
  "operation": "simulate_trajectory",
  "params": {
    "body_schema": "warehouse_arm_6dof",
    "perception_pipeline": ["sensory_fusion", "tactile_fusion"],
    "planning_pipeline": ["physics_prediction", "motor_control"],
    "cycle_time_target_sec": 4.0,
    "grasp_abort_threshold": 0.85,
    "continuous": true,
    "max_retries_per_parcel": 2
  }
}

Abort and Re-Plan

In simulation, a tactile_fusion slip heuristic can mark the current plan for review, then adapt_physical_ai can produce a revised planning artifact via physics_prediction and motor_control. This does not interrupt physical motion or prove a real grasp is slipping.

Review artifact only

In this release, the abort/re-plan loop produces simulated planning artifacts. It does not interrupt or command a real robot. Any physical abort path must be implemented in a separate validated controller.

What Happened¶

G6 orchestrated seven components in a sense-plan-act loop:

embodiment defined the 6-DOF arm's kinematic chain, joint limits, and gripper parameters
mesh3d generated prototype convex hull meshes for each parcel from pre-segmented point samples
sensory_fusion combined camera and force/torque data into a registered scene representation
tactile_fusion processed caller-supplied contact patterns to detect contact, estimate grip stability, classify coarse surface texture, and compute a spatial slip heuristic
physics_prediction produced advisory contact-force, stability, breakage-risk, and waypoint estimates for reviewed planning
motor_control gated reviewed trajectory artifacts through all pick-and-place phases
adapt_physical_ai orchestrated the continuous loop, handling re-planning on grasp failure

Each component's output feeds the next stage. The loop runs continuously — one cycle per parcel — with simulated tactile feedback producing review signals for re-planning artifacts.

Why G6 Over a Bare LLM¶

This example demonstrates structured simulation, safety gates, schema checks, and deterministic component handoffs. It is not a claim that G6 ships production robotics control, real sensor drivers, or validated trajectory execution.

A capable LLM can reason about robotics concepts and generate control code. G6 adds structured numeric and schema-checked computation for simulation-style workflows: forward kinematics data structures, heuristic physics estimates, tactile signal processing, prototype mesh artifacts, and reviewed trajectory artifacts. Prebuilt pipeline templates compose perception, simulation, and motion-planning-style steps into a workflow triggered by one GoalInput JSON, but real robotics deployment still requires dedicated hardware drivers, validated planners, safety systems, and human review.