Symbolic 3D Grounding and Software-Pipelined VLA Task Planning for Multi-Arm Manipulation

Rethinking Visual Tokens in Closed-Loop Multi-Arm Manipulation

A common assumption in Vision-Language-Action (VLA) system design is that high-level task planners must continuously ingest multi-view RGB camera frames at every decision step. In multi-manipulator simulation and industrial workcells, however, streaming high-resolution base64 images on every turn introduces two compounding bottlenecks: it forces the physics simulator to run expensive GPU render-product pipelines (such as NVIDIA Isaac Sim Replicator viewports), and it appends thousands of visual tokens to a conversation history that grows quadratically with episode length.

In M9G's IsaacSim_GeminiRobotics testbed (Mode 1, three_robot_tower.py), three 7-DoF Franka Emika FR3 manipulators (FR3_1, FR3_2, FR3_3) surround a shared central stacking platform at [0.0, 0.0] while drawing from three outer source tables holding nine numbered blocks (B1..B9). Rather than passing overhead RGB frames (latest_rgb = None, --enable-cameras disabled), our closed-loop controller grounds the LLM strictly on 50 Hz 3D /tf transforms (WorkspaceState), a deterministic spatial blueprint synthesized by RuleBasedTaskGenerator, and a pre-execution kinematic and ownership shield (RuleBasedTaskVerifier) driving a 50 Hz damped least-squares (DLS) inverse kinematics controller over FastDDS unicast.

Decoupling Strategic Synthesis from Deterministic Step Execution

To isolate the impact of frontier reasoning versus step-execution latency, GeminiRoboticsNode._run_automated_comparison_workflow() executes a continuous three-stage hardware-in-the-loop protocol inside a single simulation process. Between stages, the node publishes Empty() to /reset_simulation, restoring all nine blocks to Tables 1–3 and homing FR3_1..3 across a 4.0-second mechanical settle window:

• Stage 1 — Legacy Homogeneous VLA Loop (gemini-robotics-er-2-preview): Uses gemini-robotics-er-2-preview for both Turn-0 strategy generation (thinking_budget=256) and every subsequent tool-dispatch turn (thinking_budget=512).
• Stage 2 — Pure Frontier Reasoning Loop (gemini-4-argon): Runs gemini-4-argon on Turn 0 (thinking_budget=256) and across all step turns (thinking_budget=512) to measure the scheduling gain of frontier reasoning and its raw step-latency cost.
• Stage 3 — Hybrid Brain-Executor Routing (1x gemini-4-argon + 10x gemini-3.5-flash-lite): Invokes gemini-4-argon strictly once on Turn 0 (thinking_budget=256) to synthesize the 10-turn software-pipelined schedule, then delegates Turns 1–10 to gemini-3.5-flash-lite with thinking_budget=0, escalating back to Argon only if a pick fault or RuleBasedTaskVerifier rejection occurs.

Empirical Telemetry Across the Three-Stage Evaluation

All three stages achieved 100% task completion (9/9 blocks stacked on the central tower) and a 100% pre-execution safety verifier pass rate, confirming that symbolic 3D /tf grounding suffices for collision-free multi-arm coordination. However, the three architectures differed sharply in schedule horizon, step latency, and API cost:

Quadratic Context Accumulation & Thinking-Budget Economics

Even with camera streams disabled, input tokens represent 96.4% to 97.8% of total token volume across all three stages. Each turn transmits a static base context B_base ≈ 2,800 tokens—comprising SYSTEM_PROMPT and ROBOT_TOOLS JSON schemas (~1,450 tokens) plus the Turn-0 RuleBasedTaskGenerator 3D blueprint and strategy (~1,350 tokens)—alongside a rolling increment ΔB_turn ≈ 250–380 tokens of multi-turn function_call and controller function_response records. Re-sending the rolling conversation history on every turn yields quadratic input growth with respect to horizon T:

Compressing the schedule from 13 turns (Stage 1) to 10 turns (Stages 2 and 3) eliminates the three largest context turns at the end of the episode (where each call exceeds 4,500 input tokens), saving 10,929 input tokens automatically. Furthermore, in Stage 1, gemini-robotics-er-2-preview consumed 1,086 internal thinking tokens billed at the $10.00/1M output rate ($0.01086, or 13.8% of Stage 1's entire cost), whereas in Stage 2, allowing thinking_budget=512 on routine step turns inflated mean step latency on Turns 1–2 to 13.31 seconds. By restricting reasoning to Turn 0 on gemini-4-argon ($2.00/1M in, $10.00/1M out) and executing Turns 1–10 on gemini-3.5-flash-lite ($0.30/1M in, $2.50/1M out, thinking_budget=0), Stage 3 decomposes trial cost as follows:

Inside the 10-Turn Software-Pipelined Schedule

In the shared 3-arm station, the central stacking zone at [0.0, 0.0] is governed by a spatial mutex (center_occupied_by): only one manipulator may execute a central place motion on any given turn, whereas outer-table pick and go_home motions execute in kinematically disjoint zones. Inspecting the raw JSONL trace reveals how gemini-4-argon exploits this spatial decoupling compared to gemini-robotics-er-2-preview:

Because only one arm can occupy [0.0, 0.0] per turn, overlapping one central place with one outer-table pick or go_home on every turn after Turn 2 is the shortest conflict-free schedule for a 3-arm shared-center station. Moreover, on Turns 9 and 10, the Argon schedule proactively overlaps go_home(FR3_1) and go_home(FR3_2) while Layers 8 and 9 are being placed, achieving complete workspace clearance in 20 verified tool calls (workload Gini coefficient 0.0339 across 7, 7, 6 actions).