MirroS Launches Code-as-World for Video Physics
MirroS releases Code-as-World-VL-4B and 9B models on GitHub, converting real videos into executable MuJoCo physics programs.
Stock photo for illustration only, not from the actual event
- MirroS has published its GitHub repository and Code-as-World-VL-4B and 9B checkpoints under the Apache 2.0 license.
- The system transforms input videos into Executable World Representations (EWR) running on the MuJoCo engine.
- An abductive search agent loop runs for up to 5 rounds to propose, execute, render, and verify candidate rollouts.
- QuantiPhy evaluation scores reach 55.4 for the 9B model and 58.6 for the 27B reasoning model.
The MirroS development team has released its GitHub repository along with two model checkpoints, Code-as-World-VL-4B and Code-as-World-VL-9B, under the Apache 2.0 license. Both models are fine-tuned versions derived from Qwen3.5-4B and Qwen3.5-9B, served as BF16 safetensors via vLLM behind an OpenAI-compatible /v1 endpoint using 16 sampled frames per video and a --max-model-len of 4608.
The MirroS technical report points out that current video models, 3D reconstruction techniques, and captions each capture only a portion of a scene without uncovering its underlying mechanisms. Code-as-World addresses this limitation by representing scenes as an Executable World Representation (EWR), structured as a triple p = (C, E, A).
In the published implementation, this triple compiles into a scene.json file executed within MuJoCo, utilizing two interchangeable engines: an animation engine focused on kinematic poses and a physics engine handling forces and contacts.
Stock photo for illustration only, not from the actual event
Recovering an EWR from a video input is treated as an inverse problem, tackled by the team through abductive search. An agent executes a propose, instantiate, execute, render, and verify loop for up to K = 5 rounds. For video inputs, SAM 3 provides instance masks and image-plane tracks, VGGT-Omega estimates depth and camera geometry, and SAM 3D generates per-object meshes. Candidate rollouts are then projected back into the input view and compared against selected key frames across RGB, depth, masks, and trajectories.
"The MirroS technical report argues that video models, 3D reconstruction, and captions each recover part of a scene but none recovers its mechanism."
MirroS Technical Report
The training process consists of two distinct phases. Phase 1 involves supervised fine-tuning on 73,335 image-space QA pairs drawn from RefCOCO, RefCLEF, and GOT-10K, covering extent, position, displacement, velocity, and acceleration. Phase 2 applies GRPO to world-space VQA derived from 1,585 text-driven and 988 video-driven executable worlds, rewarded on scale-normalized numerical accuracy along with unit and format terms. The training infrastructure utilized eight NVIDIA H100 GPUs.
Translating raw video inputs directly into executable physics simulations represents a major milestone in bridging computer vision and robotics. By extracting programmatic world mechanics rather than just visual pixels, AI systems can better understand gravity, forces, and physical interactions, paving the way for advanced embodied artificial intelligence and simulation tools.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment