Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Broader context Not relevant

   Robots Need More than VLA and World Models

   Elis Karcini, Faisal Mehrban, Quang-Nam Nguyen et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Robots Need More than VLA and World Models argues that robot learning needs ways to turn unstructured behavioral data into grounded supervision, beyond scaling policies.

   Abstract

   Generalist robot intelligence is often framed as a policy-scaling problem: collect more robot demonstrations, train larger Vision-Language-Action (VLA) models, and expect broader generalisation. In this position paper, we argue that this framing is incomplete. The central bottleneck is not only policy learning, but the absence of mechanisms that convert the world's abundant unstructured behavioural data into grounded robot supervision. Human motion, internet video, simulation rollouts, and interactive demonstrations contain rich information about tasks, goals, contacts, failures, and physical constraints, yet most of this information is not directly usable by robot policies because it lacks …

2. Close Broader context Cited by this paper Not relevant

   In-Context Robot Learning with VLM Agents

   Dong-Zhou Cheng, Taoran Yi, Ye Fang et al. · 2026 · cited 13 times · Not summarized yet

   In-Context Robot Learning with VLM Agents studies learning robot behavior from deployment-time demonstrations and feedback, while ZAP trains through simulation.

   Abstract

   Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initi…

3. Close Another approach to the problem Not relevant

   Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

   Hongtao Wu, Ya Jing, Chilam Cheang et al. · 2023 · Robotics · cited 537 times · AI summary available

   GR-1 uses large-scale video pretraining and predicts robot actions and future images for language-conditioned manipulation.

   Abstract

   Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a straightforward GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data…

4. Close Another approach to the problem Cited by this paper Not relevant

   Skill2Real: Agentic Skill Learning for Zero-Shot Sim-to-Real Robot Manipulation

   Xin-Chen He, Si-Yu Ma, Chang Yu et al. · 2026 · cited 2 times · Not summarized yet

   Skill2Real transfers agentically learned manipulation skills from simulation using a proposer-verifier-governor loop and no task-policy fine-tuning.

   Abstract

   Transferring robotic skills from simulation to reality requires task knowledge that remains usable across differences in perception, dynamics, and embodiment. We introduce Skill2Real, an agentic policy framework that learns executable skills through a shared application programming interface (API). A Proposer-Verifier-Governor (PVG) loop uses privileged simulation evidence to diagnose outcomes and validate updates, while keeping learned skills grounded in public observations and API semantics. The Cerebellum first acquires local manipulation skills; the Brain then learns task-level composition with the Cerebellum frozen. Both memories transfer to the real robot without task-policy fine-tunin…

5. Close Another approach to the problem Not relevant

   RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

   Yufei Wang, Zhou Xian, Feng Chen et al. · 2023 · Robotics · cited 331 times · AI summary available

   RoboGen generates diverse simulated tasks and training supervision to scale robotic skill learning with minimal human input.

   Abstract

   We present RoboGen, a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation. RoboGen leverages the latest advancements in foundation and generative models. Instead of directly using or adapting these models to produce policies or low-level actions, we advocate for a generative scheme, which uses these models to automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision. Our approach equips a robotic agent with a self-guided propose-generate-learn cycle: the agent first proposes interesting tasks and skills to develop, and then generates correspondi…

6. Close Another approach to the problem Not relevant

   A Few Words Go a Long Way: Language Guided Robot Policy Synthesis

   Daphne Chen, A. Jain, E. Goossen et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 4 times · Not summarized yet

   ARCHITECT synthesizes modular robot programs with language-guided coding agents and perception and control tools.

   Abstract

   While vision-language-action models have demonstrated impressive zero-shot manipulation capabilities, they remain fundamentally black box policies that are difficult to interpret, adapt, or correct when they inevitably fail. In this work, we propose ARCHITECT, a framework that treats robot policy acquisition as an interactive program synthesis task. ARCHITECT leverages the reasoning capabilities of LLM coding agents to synthesize modular robot programs that utilize a suite of perception and control tools. Unlike end-to-end models where distribution shift leads to unpredictable, cascading failures, our modular architecture allows users to isolate failures and localize feedback at the level of…

7. Close Same question Cited by this paper Not relevant

   MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation

   Abhay Deshpande, M. Guru, Rose Hendrix et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 26 times · Not summarized yet

   MolmoBot tests whether simulation-only training can produce zero-shot real-world manipulation, using large-scale synthetic data across robots and tasks.

   Abstract

   A prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation. We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data…

8. Close Another approach to the problem Cited by this paper Not relevant

   Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

   Jia-Min Zhou, Qihang Zhang, Gangwei Xu et al. · 2026 · cited 22 times · Not summarized yet

   Zero-WAM uses human videos as task specifications for a causal video-action model to execute unseen manipulation tasks.

   Abstract

   Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen…

Prior work it builds on

1. Close Same method, another problem Used in depth by this paper Not relevant

   Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation

   Ze-Xi Li, Ye-Hang Zhang, Wen-Qian Li et al. · 2026 · cited 6 times · Not summarized yet

   Robo-Harness K1, like ZAP, equips a vision-language agent with a tool interface for robot manipulation, using perception tools to inform generic motions.

   Abstract

   Foundation vision-language models (VLMs) understand objects, instructions, and spatial relations, yet translating this capability into robotic manipulation remains difficult. Vision-language-action (VLA) models require extensive demonstrations and may compromise pretrained understanding, while direct RGB-only VLM control is costly and strongly dependent on model capability. We introduce Robo-Harness K1, a robot-use agent (RUA) framework that exposes perception as tools. The agent queries calibrated depth, persistent visual anchors, spatial measurements, and grasp hypotheses, then selects generic motions from the returned evidence. This interface makes 3D geometry accessible without changing …

2. Close Same method, another problem Not relevant

   VLMs are Native Action Models: Unleashing the Manipulation Capabilities of VLMs with Explicit Action-Space Grounding

   Jing-Kun Yi, Shu-Jie Zhang, Zi-Rui Zhou et al. · 2026 · cited 1 time · Not summarized yet

   VLMs are Native Action Models exposes pretrained vision-language capabilities through explicit action-space grounding for manipulation.

   Abstract

   Pretrained vision-language models (VLMs) already exhibit strong visual, spatial and reasoning capabilities relevant to robotic manipulation. The key challenge in turning pretrained VLMs into generalist manipulation policies is to expose these capabilities through an appropriate action interface. Vision-language-action (VLA) models provide a unified trainable policy, but typically introduce learned action tokens, action heads, or action experts whose semantics must be learned from robot data pretraining. In contrast, modular foundation-model-based manipulation methods can exploit pretrained VLM capabilities for zero-shot manipulation, but rely on external planning and control scaffolds that a…

3. Close Another approach to the problem Not relevant

   OpenVLA: An Open-Source Vision-Language-Action Model

   Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al. · 2024 · Conference on Robot Learning · cited 4,235 times · Not summarized yet

   OpenVLA trains an open-source vision-language-action policy on internet vision-language data and robot demonstrations, then explores efficient task fine-tuning.

   Abstract

   Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on …

4. Close Shared data, tool or method Not relevant

   Evaluating Real-World Robot Manipulation Policies in Simulation

   Xuanlin Li, Kyle Hsu, Jia-Yuan Gu et al. · 2024 · Conference on Robot Learning · cited 642 times · Not summarized yet

   SIMPLER provides simulated environments and methods for evaluating real-world manipulation policies while addressing visual and control differences.

   Abstract

   The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot …

5. Close Another approach to the problem Not relevant

   PoCo: Policy Composition from and for Heterogeneous Robot Learning

   Lirui Wang, Jialiang Zhao, Yilun Du et al. · 2024 · Robotics: Science and Systems Conference · cited 76 times · Not summarized yet

   PoCo composes policies trained across heterogeneous tasks, modalities, and domains to generalize manipulation at scene and task levels.

   Abstract

   Training general robotic policies from heterogeneous data for different tasks is a significant challenge. Existing robotic datasets vary in different modalities such as color, depth, tactile, and proprioceptive information, and collected in different domains such as simulation, real robots, and human videos. Current methods usually collect and pool all data from one domain to train a single policy to handle such heterogeneity in tasks and domains, which is prohibitively expensive and difficult. In this work, we present a flexible approach, dubbed Policy Composition, to combine information across such diverse modalities and domains for learning scene-level and task-level generalized manipulat…