Physical AgentLearning Manipulation in the Native Distribution Retains Reasoning and In-Context Learning

TL;DR

We post-trained an open 4B general-purpose agent into a physical agent, on four B200 GPUs and about 1k robot demonstrations. After training it does robot tasks on par with a VLA built from the same model and data, and it is still a general agent: it keeps reasoning, keeps answering questions about images, and learns new tasks from demonstrations placed in its context.

On LIBERO it succeeds in 79.2% of rollouts, level with the VLA-style baseline at 80.0%, and it does better when the instruction changes (23.3% against 15.0%). Its general capabilities stay largely intact: 56.8% on MMLU-Pro against 61.8% before training, and 53.7% on MMMU against 46.8%, where the baseline scores zero on both. The key result is in-context learning. Given one robot demonstration in its context and no parameter update, it succeeds more often: from 44.8% to 56.2% over 21 tasks, and from 17.1% to 31.4% on the hard ones, nearly double. The VLA-style baseline cannot do this. With the same demonstration, its success falls to zero. We never trained for it; the ability is inherited from the general-purpose agent.

1What a physical agent is, and how it works

Compared with a VLA and Code as Policies

bowl = detect("black bowl")plate = detect("plate")pick(bowl)place_on(bowl, plate)

Code as Policies: Liang et al., 2023. VLA examples: OpenVLA (Kim et al., 2024) and π0.5 (Black et al., 2025).

The agent loop

The instruction is given once. In each turn the agent receives two camera views, writes a short reasoning paragraph and one tool call, and gets back a tool response that only says the call ran. Whether a grasp held, or the task is done, it has to judge from the next views.

The agent receives

User, once at the start

pick up the black bowl between the plate and the ramekin and place it on the plate

Observation · turn 1 of 12

agent viewwrist view

←Play→

The agent writes

Agent reasoning

Agent tool call

Tool response, then the next views

Physical post-training

All parameters are updated, including the vision encoder, with next-token cross-entropy.

2Learning physical skills from a general-purpose agent

Physical Agent performs close to the Qwen4B-OFT baseline on the original tasks, and better under task and position perturbations.

Qwen4B-OFT is the baseline: the same Qwen3.5-4B and the same 1k demonstrations, with a continuous action head instead of tool calls.

On the original tasks the two are within one point. Under task perturbation, Physical Agent reaches 23.3% against 15.0%, with the gains in Goal and Long. Position perturbation remains hard for both.

3Still a general agent

Two models with the same manipulation score end up very different as general models.

Physical Agent updates every parameter on robot data, with no general-data co-training and no frozen modules, and still answers general questions: 56.8% on MMLU-Pro, down five points from the original model. The action-head model, which keeps its language head, produces no parseable correct answers.

4In-context learning

Language agents pick up a new task from an example in the prompt. We study whether a physical agent can do the same, and find that it can.

Show the agent one demonstration of a task it has not been trained on, and it succeeds more often. We never trained it to do this.

We test on LIBERO-PRO task perturbations, with no parameter updates. A task demo shows the evaluated task from another initial state; a distractor demo shows a different task in the same scene.

What goes into the first turn

No demo728 tokensWith a demo2,685 tokens

systemtool schema

in-context demonstrationno demonstration

tool call8 targets

tool call8 targets

tool call8 targets

user“put the wine bottle in the bowl”

→

agentreasoningtool call8 targets

A task demo helps and a distractor demo hurts. Across all 21 tasks a task demo raises success from 44.8% to 56.2%. A distractor demo lowers it to 36.2%. That 20-point gap between the two shows the agent is reading what the demonstration does, not just benefiting from more context. On the hard subset, a task demo nearly doubles success, from 17.1% to 31.4%.

The action-head model cannot act with the longer context at all. Qwen4B-OFT solves 31.4% of the tasks with no demonstration and essentially none with either kind. It was trained only on current observations; Physical Agent was trained to act on its own history, and a demonstration is just more history in the same format.

“put the wine bottle in the bowl”

no demo

0 / 5never grasps the bottle

task demo

3 / 5bottle in the bowl

“pick the alphabet soup and place it in the basket”

distractor demo

0 / 5reaches for the cream cheese

task demo

5 / 5soup in the basket

5Four GPUs, a 4B model

We fine-tune a 4B model on four B200 GPUs, in under 50 GPU-hours.

PEAR numbers are quoted from its project page; evaluation protocols differ.

6Reflection and recovery from execution errors

Every demonstration it learned from is a success. Nobody showed it a failed grasp, or what to do after one. In natural rollouts it still recovers from some of its own grasp failures.

Three natural rollouts

↻Replay

1Attempt

1. 1

   Attemptclose and test the lift

   Closes on the bottle and lifts.

2. 2

   Mismatchmodel reflection

   “The previous lift did not give me confirmed grasp evidence …”

3. 3

   Correctcorrective action

   “I open the fingers and lower back toward the target to re-establish a better alignment.”

4. 4

   Verifyobject follows the gripper

   The bottle lifts with the gripper.

15 of 80 rollouts had a failed grasp.

8 recovered

5 not recovered

2 unresolved

Reflection after the first failure, in 14 of them.

2 explicit recognition

8 expressed uncertainty

4 no recognition

7Limitations and future directions

This is a first version, trained on four GPUs. Here is where it stands today and where we are taking it next.

Work with us

Everything on this page was trained on four GPUs. The next step needs many more.

We want to take physical agents from supervised post-training to large-scale reinforcement learning, where the agent improves from its own experience instead of demonstrations. That takes far more compute than we have today: we are short on GPUs.

BibTeX

@misc{xu2026physicalagent,
  title        = {{Physical Agent}: Learning Manipulation in the Native Distribution
                  Retains Reasoning and In-Context Learning},
  author       = {Xu, Junxiao and Yan, Rui and Zhao, Ye},
  year         = {2026},
  howpublished = {\url{the address of this page}},
  note         = {Project website}
}