Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Proche Contexte plus large Pas pertinent

   Robots Need More than VLA and World Models

   Elis Karcini, Faisal Mehrban, Quang-Nam Nguyen et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 2 fois · Pas encore résumé

   Robots Need More than VLA and World Models soutient que les politiques robotiques doivent exploiter des données comportementales non structurées, dont les simulations, pour obtenir une supervision ancrée.

   Résumé scientifique

   Generalist robot intelligence is often framed as a policy-scaling problem: collect more robot demonstrations, train larger Vision-Language-Action (VLA) models, and expect broader generalisation. In this position paper, we argue that this framing is incomplete. The central bottleneck is not only policy learning, but the absence of mechanisms that convert the world's abundant unstructured behavioural data into grounded robot supervision. Human motion, internet video, simulation rollouts, and interactive demonstrations contain rich information about tasks, goals, contacts, failures, and physical constraints, yet most of this information is not directly usable by robot policies because it lacks …

2. Proche Contexte plus large Cité par cet article Pas pertinent

   In-Context Robot Learning with VLM Agents

   Dong-Zhou Cheng, Taoran Yi, Ye Fang et al. · 2026 · cité 13 fois · Pas encore résumé

   In-Context Robot Learning with VLM Agents étudie l’apprentissage robotique à partir du contexte et d’interactions au déploiement, tandis que ZAP apprend le contrôle sur des tâches simulées.

   Résumé scientifique

   Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initi…

3. Proche Autre approche du problème Pas pertinent

   Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

   Hongtao Wu, Ya Jing, Chilam Cheang et al. · 2023 · Robotics · cité 537 fois · Résumé IA disponible

   GR-1 préentraîne un modèle génératif sur des vidéos pour prédire actions et images futures en manipulation, alors que ZAP apprend le contrôle par ajustement supervisé et GRPO en simulation.

   Résumé scientifique

   Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a straightforward GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data…

4. Proche Autre approche du problème Cité par cet article Pas pertinent

   Skill2Real: Agentic Skill Learning for Zero-Shot Sim-to-Real Robot Manipulation

   Xin-Chen He, Si-Yu Ma, Chang Yu et al. · 2026 · cité 2 fois · Pas encore résumé

   Skill2Real transfère des compétences apprises en simulation via une boucle de vérification et des mémoires de compétences, tandis que ZAP entraîne une politique sans données robotiques réelles.

   Résumé scientifique

   Transferring robotic skills from simulation to reality requires task knowledge that remains usable across differences in perception, dynamics, and embodiment. We introduce Skill2Real, an agentic policy framework that learns executable skills through a shared application programming interface (API). A Proposer-Verifier-Governor (PVG) loop uses privileged simulation evidence to diagnose outcomes and validate updates, while keeping learned skills grounded in public observations and API semantics. The Cerebellum first acquires local manipulation skills; the Brain then learns task-level composition with the Cerebellum frozen. Both memories transfer to the real robot without task-policy fine-tunin…

5. Proche Autre approche du problème Pas pertinent

   RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

   Yufei Wang, Zhou Xian, Feng Chen et al. · 2023 · Robotics · cité 331 fois · Résumé IA disponible

   RoboGen génère automatiquement tâches, scènes et supervisions en simulation pour apprendre des compétences, tandis que ZAP ajuste un modèle vision-langage sur des tâches simulées.

   Résumé scientifique

   We present RoboGen, a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation. RoboGen leverages the latest advancements in foundation and generative models. Instead of directly using or adapting these models to produce policies or low-level actions, we advocate for a generative scheme, which uses these models to automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision. Our approach equips a robotic agent with a self-guided propose-generate-learn cycle: the agent first proposes interesting tasks and skills to develop, and then generates correspondi…

6. Proche Autre approche du problème Pas pertinent

   A Few Words Go a Long Way: Language Guided Robot Policy Synthesis

   Daphne Chen, A. Jain, E. Goossen et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 4 fois · Pas encore résumé

   ARCHITECT synthétise des programmes robotiques modulaires à partir d’instructions et d’outils, tandis que ZAP entraîne une politique de contrôle avec des tâches simulées.

   Résumé scientifique

   While vision-language-action models have demonstrated impressive zero-shot manipulation capabilities, they remain fundamentally black box policies that are difficult to interpret, adapt, or correct when they inevitably fail. In this work, we propose ARCHITECT, a framework that treats robot policy acquisition as an interactive program synthesis task. ARCHITECT leverages the reasoning capabilities of LLM coding agents to synthesize modular robot programs that utilize a suite of perception and control tools. Unlike end-to-end models where distribution shift leads to unpredictable, cascading failures, our modular architecture allows users to isolate failures and localize feedback at the level of…

7. Proche Même question Cité par cet article Pas pertinent

   MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation

   Abhay Deshpande, M. Guru, Rose Hendrix et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 26 fois · Pas encore résumé

   MolmoBot étudie le transfert zéro-shot de la simulation au robot réel sans ajustement sur tâches réelles, comme ZAP, mais avec des données synthétiques simulées à grande échelle.

   Résumé scientifique

   A prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation. We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data…

8. Proche Même question Cité par cet article Pas pertinent

   Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

   Jia-Min Zhou, Qihang Zhang, Gangwei Xu et al. · 2026 · cité 22 fois · Pas encore résumé

   Zero-WAM vise la généralisation à des tâches de manipulation inédites à partir de vidéos humaines, tandis que ZAP teste des tâches et stratégies absentes de son entraînement simulé.

   Résumé scientifique

   Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen…

Les travaux antérieurs sur lesquels il s’appuie

1. Proche Autre approche du problème Utilisé en profondeur par cet article Pas pertinent

   Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation

   Ze-Xi Li, Ye-Hang Zhang, Wen-Qian Li et al. · 2026 · cité 6 fois · Pas encore résumé

   Robo-Harness K1 donne à un agent VLM des outils de perception géométrique et des mouvements génériques, tandis que ZAP apprend le contrôle par entraînement simulé.

   Résumé scientifique

   Foundation vision-language models (VLMs) understand objects, instructions, and spatial relations, yet translating this capability into robotic manipulation remains difficult. Vision-language-action (VLA) models require extensive demonstrations and may compromise pretrained understanding, while direct RGB-only VLM control is costly and strongly dependent on model capability. We introduce Robo-Harness K1, a robot-use agent (RUA) framework that exposes perception as tools. The agent queries calibrated depth, persistent visual anchors, spatial measurements, and grasp hypotheses, then selects generic motions from the returned evidence. This interface makes 3D geometry accessible without changing …

2. Proche Même question Pas pertinent

   VLMs are Native Action Models: Unleashing the Manipulation Capabilities of VLMs with Explicit Action-Space Grounding

   Jing-Kun Yi, Shu-Jie Zhang, Zi-Rui Zhou et al. · 2026 · cité 1 fois · Pas encore résumé

   VLMs are Native Action Models examine comment convertir les capacités préentraînées d’un VLM en politique de manipulation, question également abordée par l’interface d’outils de ZAP.

   Résumé scientifique

   Pretrained vision-language models (VLMs) already exhibit strong visual, spatial and reasoning capabilities relevant to robotic manipulation. The key challenge in turning pretrained VLMs into generalist manipulation policies is to expose these capabilities through an appropriate action interface. Vision-language-action (VLA) models provide a unified trainable policy, but typically introduce learned action tokens, action heads, or action experts whose semantics must be learned from robot data pretraining. In contrast, modular foundation-model-based manipulation methods can exploit pretrained VLM capabilities for zero-shot manipulation, but rely on external planning and control scaffolds that a…

3. Proche Autre approche du problème Pas pertinent

   OpenVLA: An Open-Source Vision-Language-Action Model

   Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al. · 2024 · Conference on Robot Learning · cité 4 235 fois · Pas encore résumé

   OpenVLA ajuste un modèle vision-langage-action préentraîné sur des démonstrations robotiques, tandis que ZAP enseigne le contrôle sans données robotiques réelles.

   Résumé scientifique

   Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on …

4. Proche Données, outil ou méthode en commun Pas pertinent

   Evaluating Real-World Robot Manipulation Policies in Simulation

   Xuanlin Li, Kyle Hsu, Jia-Yuan Gu et al. · 2024 · Conference on Robot Learning · cité 642 fois · Pas encore résumé

   SIMPLER fournit des environnements simulés pour évaluer des politiques de manipulation sur robot réel, et ZAP évalue notamment sa politique sur des tâches réelles.

   Résumé scientifique

   The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot …

5. Proche Autre approche du problème Pas pertinent

   PoCo: Policy Composition from and for Heterogeneous Robot Learning

   Lirui Wang, Jialiang Zhao, Yilun Du et al. · 2024 · Robotics: Science and Systems Conference · cité 76 fois · Pas encore résumé

   PoCo compose des politiques apprises à partir de données hétérogènes entre tâches et domaines, tandis que ZAP entraîne une politique à partir de simulation sans données robotiques réelles.

   Résumé scientifique

   Training general robotic policies from heterogeneous data for different tasks is a significant challenge. Existing robotic datasets vary in different modalities such as color, depth, tactile, and proprioceptive information, and collected in different domains such as simulation, real robots, and human videos. Current methods usually collect and pool all data from one domain to train a single policy to handle such heterogeneity in tasks and domains, which is prohibitively expensive and difficult. In this work, we present a flexible approach, dubbed Policy Composition, to combine information across such diverse modalities and domains for learning scene-level and task-level generalized manipulat…