Let the VLM plan, let the VLA act

“Put the red pen in the blue stationery box” initially sounds like a visual question. Once a gripper approaches the pen, the questions become physical: where should it grasp, which way should it turn, what happens after lifting, and what evidence will establish that the pen was placed correctly?

I explored this in a Franka Panda simulation built with PyBullet. Qwen2.5-VL interprets the scene, breaks down the task, and checks the result. π0 connects through openpi on the action side, generating movement from observations and subtasks. Grounding, primitive controllers, and protocol validation make the handoff between them observable and debuggable.

01Assign responsibilities through model outputs

The “large brain / small brain” analogy describes responsibilities in this system. VLMs can reason about space, and VLAs often inherit a vision-language backbone. Separating the layers gives semantic planning and action generation explicit interfaces.

The VLM handles semantics at task transitions. The VLA repeatedly receives fresh observations during execution. This relative difference in timescale does not establish a measured control frequency for this project; motor servo control remains with the underlying controller.

02Qwen2.5-VL turns scene understanding into a plan

Qwen2.5-VL ↗ processes images and text, with grounding and structured-output capabilities. Here it acts as a planner: identify the relevant objects, interpret the instruction, and decide which changes should happen in which order.

“Put all the pens away” must expand into individual objects such as red_pen and blue_pen. A plan that retains only the category pen leaves the executor without a definite target and makes omissions difficult to detect.

The /vlm/plan endpoint therefore returns symbolic primitives. A subtask can follow this chain:

locate_object -> move_to_pregrasp -> grasp -> lift
              -> move_to_target -> place -> release

JSON needs semantic checks as well

This is one step from a plan, rather than a complete executable task:

{
  "action": "move_to_target",
  "object": "red_pen",
  "target": "blue_stationery_box",
  "parameters": {
    "relation": "inside",
    "place_orientation": "align_with_container_long_axis"
  }
}

JSON repair handles minor syntax errors. Pydantic checks fields and types, and an action whitelist restricts which primitives can be called. Further checks establish whether objects exist, names are canonical, compound tasks include all their subtasks, and spatial relations match the instruction.

Replacing inside with left_of can leave the JSON perfectly valid. Parsing is therefore only the first check. Ambiguous semantics or missing objects should lead to clarification or replanning rather than an executor guessing what was intended.

03How π0 and OpenVLA connect understanding to action

A VLA produces outputs in an action space. It must handle the current view, robot state, gripper conventions, and continuity over time. A verbal instruction to “move toward the target” is insufficient for that interface.

π0 ↗ adds an action expert to a pretrained VLM and uses flow matching to generate continuous actions. The original OpenVLA ↗ predicts discretized action tokens that are decoded into continuous controls. Both are VLAs, but their action representations and inference processes require distinct adapters.

The main integration path here is π0 through an openpi ↗ remote policy. OpenVLA provides a reference for action modeling and further adaptation; the names do not describe interchangeable service implementations.

/vla/action packages the current RGB image, robot_state, and the primitive's subtask prompt into an observation, calls the remote policy, and translates its output into the client's action protocol.

RGB_t + robot_state_t + current_subtask
    -> observation adapter
    -> π0 remote policy
    -> action chunk
    -> action-space adapter
    -> controller

Action chunks still need fresh feedback

An action chunk describes a short future sequence. The client can execute a prefix, acquire a new observation, and query the policy again. Prefix length trades off latency, continuity, and feedback freshness; a changed scene should not be followed by unconditional playback of an old chunk.

The unified interface can express ee_delta: end-effector translation increments, rotation increments, and gripper control. This illustrates the adapted protocol. It does not mean every π0 or OpenVLA checkpoint natively produces these same seven quantities.

ee_delta = [dx, dy, dz, droll, dpitch, dyaw, gripper]

Joint versus end-effector actions, normalization statistics, coordinate frames, units, and gripper opening conventions must each match the executor. Equal vector dimensions do not establish equal physical meaning.

04Grounding needs the pen's axis as well as its position

red_pen in a VLM plan is a semantic target. Execution also needs its image region and a transformation into world coordinates.

/vlm/ground supplies a bounding box, center pixel, shape, and approximate image-plane orientation. The client uses RGB-D back-projection, transforms camera coordinates into world coordinates, and estimates an object axis through point-cloud PCA.

object label -> bbox / pixels -> depth back-projection
             -> camera coordinates -> world coordinates
             -> point-cloud PCA -> object axis

For elongated objects such as pens and rulers, a box center misses orientation. Grasping needs an appropriate gripper yaw; placement also depends on the container's long axis and free space. A bounding box's long edge is only a rough clue, affected by perspective, occlusion, and background depth. Point-cloud estimates likewise require valid depth and a suitable object region.

The detector path integrates YOLO-World and retains a color-and-shape fallback. Simplified PyBullet materials differ from real images, so an open-vocabulary detector may miss a simulated pen. The fallback helps integration, but its use should be logged so its output is not mistaken for detector performance.

05Two LoRA adaptations learn different mappings

LoRA ↗ adapts frozen base weights with a small set of low-rank parameters. VLM and VLA adaptation serve different objectives here, and their datasets need different units of supervision.

Supervised fine-tuning has been run on the VLM side. One training record lists a base of approximately 8.34B parameters and about 47.6M trainable LoRA parameters, roughly 0.57%. Training loss over one epoch fell from approximately 0.5158 to 0.0028. These values describe that training run; they neither replace held-out scene evaluation nor translate into a manipulation success rate.

The VLA work remains at trajectory collection, LoRA adaptation, and rollout evaluation. A primitive controller can generate demonstrations. At each timestep, images, robot state, the active subtask, and the executed action need to stay aligned. Retaining failures and retries helps explain how action errors accumulate.

I would split evaluation by scenes and episodes, keeping adjacent frames from the same trajectory out of both sides of a train/test split. Memorizing target positions and grasping successfully in a new layout are different outcomes.

06Four endpoints form a replayable loop

The FastAPI service exposes four endpoints. The PyBullet client captures observations, transforms geometry, and executes controls:

The client retains primitive-only and VLA-assisted paths. The former helps establish correct geometry and action conventions; the latter introduces the learned policy incrementally. An episode should connect its plan, grounding, actions, executed steps, and verification result, rather than retaining only a final successful screenshot.

07Completion is a claim about the world

After placement, /vlm/verify evaluates the final image, original instruction, and executed steps. The simulator can independently check geometric relations such as inside, left_of, and on_top_of, alongside long-axis alignment.

Three outcomes need separate treatment: the plan was complete, the controller executed the steps, and the resulting object relations satisfy the goal. A VLM can misjudge occlusion or spatial relations. Simulation geometry supplies an independent reference; real deployment needs perception and measurements in place of simulator ground truth.

Evaluation should distinguish plan validity, compound-task completeness, grounding quality, grasp and placement success, replanning, and collisions. Without a complete evaluation table, training loss and a few demonstrations cannot establish overall success rates.

08Continuing beyond single-view simulation

The current system mainly uses front RGB. A wrist-image key exists in the interface, but currently receives a copy of the front image. That does not provide a wrist viewpoint. Actual multi-view control requires camera calibration, synchronization, and inputs compatible with the policy's training setup.

What matters most to me is the information preserved at each handoff: a VLM target that can be grounded, a VLA action with a definite physical meaning, and an execution result that can inform the next plan. The layered system becomes useful when this loop keeps working and exposes a specific failure when it does not.