Dynamic object manipulation

PhysVAMPhysics-oriented Future Value Action Model
for Dynamic Object Manipulation

Anticipate task progress. Guide dynamic actions.

Future Values. Dynamic Actions. Concept, method, value predictions, demonstrations, and results.
58.83%

Simulation success

+27.00 pp over PUMA
64.17%

Real-world success

+27.50 pp over PUMA
4 horizons

Future task values

h = 4, 8, 16, 32 steps

Simulation: paper Table 1. Real world: macro-average over three scenarios, paper Table 8.

Abstract

Predicting future scene or object states does not necessarily reveal how robot–object interactions will advance or hinder a task. This challenge is particularly pronounced in dynamic robotic manipulation, where objects continue moving during perception and execution. Effective action generation therefore requires not only modeling object dynamics but also anticipating how future interactions will shape task progress. To this end, we propose PhysVAM, a Physics-oriented Future Value Action Model that represents anticipated interactions through their effects on future task progress. This value-based formulation provides task-directed foresight without reconstructing future visual states. Conditioned on current observations and historical object and robot motion, PhysVAM jointly learns action generation and multi-horizon future-value prediction. The value branch learns horizon-specific latent representations through future task-value prediction from successful and failed trajectories. These future-value representations are integrated into the action policy through gated cross-attention, guiding action generation toward interactions that facilitate task success. Extensive simulation and real-world experiments demonstrate superior task success, with absolute success-rate improvements of 27.0 and 27.5 percentage points over the strongest baselines in simulation and real-world tasks, respectively.

The idea

Guiding actions with future values

Guide action generation toward interactions that facilitate task success.

Moving objects can turn a promising grasp into a missed interaction. PhysVAM predicts how future robot–object interactions will advance or hinder task progress, using current observations and object and robot motion histories. The resulting multi-horizon value representations provide task-directed guidance for action generation, helping the policy choose interactions that move the task toward success.

Visual future modeling compared with multi-horizon future-value modeling; predicted values track interaction setbacks and progress.
Future-value representations guide actions toward interactions that advance task progress.

Method

Motion. Value. Action.

Historical motion provides context. Future values provide task-directed guidance.

PhysVAM architecture: observation and motion encoding, multi-horizon ValueDiT, and gated cross-attention into ActionDiT.
01

Encode motion

Object positions and estimated velocities condition ActionDiT. Recent robot actions provide temporal context to ValueDiT.

02

Predict progress

Horizon-specific queries learn future task-value representations at 4, 8, 16, and 32 steps.

03

Guide actions

Gated cross-attention injects latent value features into the action branch to generate a 32-step action chunk.

See the predictions

Value in Action

Follow predicted task progress as interactions unfold.

Simulation rollouts with predicted values at h = 4, 8, 16, and 32 steps.
Interaction-level evidence

Across 220 grasp attempts, the mean predicted value change at h = 4 is +0.254 for lifts and −0.185 for misses.

Evaluation

Results

Simulation, real-world results, and ablations from the paper.

Simulation

Original Table 1: full simulation comparison and transferability of value guidance, including all scenes, mean and standard deviation, success rates and completion times.

Real-world Results

Real-world success rate (%)
MethodConveyorTurntableRollingAverage
DynamicVLA40.022.525.029.17
PUMA47.532.530.036.67
PhysVAM72.565.055.064.17

Value Ablations

Original Table 2: value branch, value representations, multi-horizon prediction, and future progress prediction ablations.

Motion Ablations

Original Table 3: object-motion representations and robot-motion history ablations.

Real-world comparisons

Different motions. Timely actions.

Conveyor · Cube

Prompt: Transfer the moving cube into the basket.

DynamicVLA

✗

PUMA

✗

PhysVAM

✓

Conveyor · Orange

Prompt: Put the moving orange in the basket.

DynamicVLA

✗

PUMA

✗

PhysVAM

✓

Turntable · Cup

Prompt: Move the moving cup into the open-top box.

DynamicVLA

✗

PUMA

✗

PhysVAM

✓

Turntable · Bread

Prompt: Place the moving bread on the plate.

DynamicVLA

✗

PUMA

✗

PhysVAM

✓

Rolling · Bottle

Prompt: Pick up the rolling white bottle and place it in the basket.

DynamicVLA

✗

PUMA

✗

PhysVAM

✓

PhysVAM Demonstrations