WorldOracle Tests Whether World Models Can Reliably Predict Real-World Executions

Key Takeaways

WorldOracle is a benchmark for action-conditioned world models (ACWMs) that scores each model’s predicted rollout against the paired real-world execution, with both starting from the same initial observations and following the same action sequence.

Seven representative models were evaluated; arm motion was predicted relatively well, but object positions and movements often deviated from reality, and performance dropped over complete operation sequences.

The platform opened to developers on October 16 and accepts both open-weight and closed-source models.

In one WorldOracle case, the robot’s grasp missed, yet the world model predicted a successful pick. Credit: [WorldOracle team / AgiBot]

In one case showcased by WorldOracle, a robot’s grasp missed, yet the world model predicted that the robot had successfully picked up the fruit.

Given the initial frame and action sequence of a real manipulation episode, the model generated a rollout that looked plausible on screen but differed from the actual execution. The success it depicted never happened in reality.

Researchers from Princeton University, the University of Texas at Austin, the University of Toronto, NVIDIA and AgiBot have jointly introduced WorldOracle, a benchmark specifically designed to evaluate action-conditioned world models (ACWMs) by comparing their predictions with real-world executions that start from the same initial observations and follow the same action sequences.

The team evaluated seven representative models — among them DreamDojo, Cosmos 3, IRASim and GE-Sim 2.0 — and identified two prominent problems. First, while the motion paths of robot arms are predicted relatively well, the positions and movements of manipulated objects can still deviate from reality. Second, models that perform well over short horizons can degrade significantly over complete operation sequences.

On October 16, WorldOracle, an evaluation platform for action-conditioned world models, officially opened to developers. The research team has collected real manipulation episodes in advance, recording both video and actions. Developers use the initial frames and action sequences provided in the evaluation set to generate predicted videos and submit them to the platform, which scores each submission against the corresponding real execution record. The results show whether a model has accurately predicted the consequences of the actions, helping developers pinpoint weaknesses and verify improvements.

Project page: https://worldoracle-benchmark.github.io/

Paper: https://arxiv.org/abs/2610.18826

How Do Seven World Models Perform?

Arm motion is right, but objects don’t always follow

The predicted motion of a robot arm reaching toward a piece of fruit and closing its gripper may closely match its actual motion, yet the model can still incorrectly predict whether the fruit is lifted and where it moves next. To assess the two separately, the research team compared the position and motion of the robot arm and of the manipulated object in each predicted video against the real-world recording, and scored them individually.

Spatial alignment scores for the robot arm and the manipulated object. Credit: [WorldOracle team / AgiBot]

The results show that all five models evaluated on the human-operated AgiBot G2 data scored higher on the arm than on the object. In other words, even when the arm in the generated video follows the given actions well, the position and movement of the object may still depart from reality. IRASim scored 78.8 on the robot arm but only 29.1 on the manipulated object, while Cosmos 3 scored 89.4 and 45.4, respectively. The remaining models showed gaps of varying size.

Accuracy declines as the rollout horizon grows

Time presents a second challenge. Completing a continuous task often requires multiple grasps, placements and adjustments, and the object state left by one interaction becomes the starting point for the next. Whether a model stays close to reality must therefore be tested over the full process.

WorldOracle evaluates both the first 30 seconds of each trajectory and the complete long sequence.

WorldOracle-SV scores of the same models under the two evaluation horizons. Credit: [WorldOracle team / AgiBot]

On the same set of AgiBot G2 teleoperation data, the WorldOracle-SV scores of all five models fell. DreamDojo dropped from 50.56 to 27.08, and Cosmos 3 from 53.99 to 26.60, nearly halving in both cases. GE-Sim 2.0 fell from 73.86 to 63.33; it posted the highest score of the five models at both horizons and the smallest decline over the full sequence.

The relative ranking also shifted. IRASim scored below Ctrl-World, DreamDojo and Cosmos 3 over the first 30 seconds, but above all three over the full sequence.

What Can WorldOracle Measure?

WorldOracle evaluation framework: real actions are recorded, the model predicts what follows, and the result is scored against the real execution. Credit: [WorldOracle team / AgiBot]

Accurately predicting a failure earns a high score

WorldOracle starts from pre-collected records of real manipulation. A model is given the initial frame and the action sequence and generates a predicted video, which is compared with the real execution of the same actions.

What models must predict includes missed grasps, drops and spills. If an operation fails in reality, accurately predicting that failure counts as a correct prediction.

Failed executions therefore serve as an important class of test samples: they reveal whether a model follows the given actions and their consequences, or defaults to a successful version of the task.

The benchmark covers two robot embodiments, AgiBot G2 and YAM, as well as first-person human-hand manipulation. Four tasks (fruit picking, towel hanging, water pouring and drawer pulling) involve interaction with rigid, deformable, fluid and articulated objects, respectively. Task coverage varies by embodiment, as defined in the evaluation set.

The…

News is gathered automatically from public robotics & AI feeds on a schedule.