1. Bibliographic Information

1.1. Title

Self-Adapting Improvement Loops for Robotic Learning

1.2. Authors

Calvin Luo*1 , Zilai Zeng*1, Mingxi Jia1, Yilun $\mathbf { D } \mathbf { u } ^ { 2 }$ , Chen Sun1 1Brown University, Harvard University

1.3. Journal/Conference

The paper is an arXiv preprint, indicating it has not yet undergone formal peer review for a specific journal or conference. However, given the authors' affiliations with prestigious institutions like Brown University and Harvard University, and the topic's relevance to major machine learning and robotics conferences (like ICLR, NeurIPS, CoRL, which the authors cite for their previous work), it is likely intended for publication in a top-tier venue in the field of AI, robotics, or machine learning.

1.4. Publication Year

2025 (Published at 2025-06-07T04:34:37.000Z)

1.5. Abstract

This paper introduces the Self-Adapting Improvement Loop (SAIL), a novel framework designed to enable robotic agents to continuously improve their performance on novel tasks through self-collected online experience. The core idea is to iteratively update an in-domain video model (pretrained on expert demonstrations) using trajectories generated by the robot itself. This self-produced experience is collected via Inverse Probabilistic Adaptation (IPA), which combines the in-domain model with a powerful internet-scale pretrained video model to facilitate generalization to unseen tasks. The authors apply SAIL to a diverse set of MetaWorld tasks and two real-robot manipulation tasks, demonstrating continuous performance improvements over multiple iterations for tasks initially unseen during the in-domain model's training. A key finding is SAIL's surprising robustness to whether and how self-collected experience is filtered, and to the quality of the initial in-domain demonstrations. SAIL effectively leverages summarized internet-scale data and online experience to bootstrap a high-performance video model for solving novel robotic tasks through self-improvement.

1.6. Original Source Link

Official source: https://arxiv.org/abs/2506.06658 (preprint) PDF link: https://arxiv.org/pdf/2506.06658v1.pdf (preprint)

2. Executive Summary

2.1. Background & Motivation

The paper addresses a critical challenge in robotic learning: the generalization of visual planners to unseen tasks. While video generative models trained on expert demonstrations have proven effective as text-conditioned visual planners, their performance is often limited by the specific in-domain examples they were trained on. Leveraging web-scale video datasets (pre-collected offline data) has shown promise in improving generalization (e.g., Adapt2Act), but this still relies on static, offline data. The fundamental problem is that agents need to move beyond fixed datasets and continuously improve in an online manner from self-collected behaviors and feedback, especially in the "era of experience." This continuous improvement from interaction with the environment is crucial for agents to refine their performance on specific tasks of interest and adapt to novel scenarios.

2.2. Main Contributions / Findings

The paper's primary contributions are:

• Introduction of SAIL: The proposal of the Self-Adapting Improvement Loop (SAIL) framework, which enables an in-domain video model to iteratively update itself on self-produced trajectories. These trajectories are collected through adaptation with an internet-scale pretrained video model, leading to steady performance improvement for specified tasks.
• Demonstrated Continuous Improvement: Extensive evaluations on the MetaWorld task suite and two real-robot manipulation tasks show that SAIL consistently improves performance over multiple iterations for novel tasks previously unseen during the in-domain video model's initial training. This highlights the effectiveness of combining large-scale offline data with online self-acquired experience.
• Robustness to Filtering and Data Quality: The discovery that SAIL is remarkably robust to various filtering strategies for self-collected experience (even no filtering) and the quality of initial in-domain demonstrations (even suboptimal data). This suggests SAIL's practical applicability in real-world scenarios where expert data or careful filtering might be costly.
• Virtuous Loop of Adaptation and Self-Improvement: The work demonstrates that the combination of web-scale data (via IPA) and self-collected experience is critical for facilitating a virtuous loop of continuous improvement, showing that training on either independently fails to achieve strong iterative enhancement.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

• Video Generative Models: These are machine learning models capable of creating new video sequences. They learn patterns and dynamics from existing videos to generate plausible future frames or entire video clips, often conditioned on text prompts or other inputs. Recent advancements, particularly with diffusion models, have significantly improved their visual quality and physical fidelity.
• Text-Conditioned Visual Planners: In robotics, a visual planner uses a video generative model to synthesize a sequence of future visual frames (a "visual plan") conditioned on a text description of a desired task. This plan shows how the environment might evolve if the task is successfully executed. For example, a prompt like "push the red block" would generate a video showing the robot arm pushing the red block.
• Inverse Dynamics Model (IDM): An IDM is a machine learning model that, given two consecutive visual observations (frames) and potentially the current robot state, predicts the action (e.g., motor commands, end-effector pose changes) that would cause the robot to transition from the first observation to the second. It essentially "inverts" the robot's dynamics to find the control inputs needed to achieve a desired visual change.
• Diffusion Models: A class of generative models that learn to reverse a diffusion process. They start with random noise and gradually "denoise" it over several steps to produce a coherent image or video. Text-to-video diffusion models condition this denoising process on a text prompt to generate videos matching the description.
• Score Composition: A technique used in diffusion models to combine score functions from different models or conditions. A score function (or score) in a diffusion model estimates the gradient of the log-probability density of the data distribution, which guides the denoising process. By composing scores, one can blend the generative capabilities or conditional controls of multiple models.
• MetaWorld: A challenging benchmark for multi-task and meta reinforcement learning in a simulated robotic environment. It features a wide array of manipulation tasks (e.g., reach, push, pick-and-place), allowing for rigorous evaluation of generalization and learning efficiency. It provides ground-truth success evaluations, making it suitable for quantitative comparisons.
• Online Learning / Self-Collected Behaviors: In contrast to offline learning (where models are trained on a fixed dataset), online learning involves an agent continuously interacting with its environment, collecting new data (self-collected behaviors), and updating its model based on this fresh experience. This allows for continuous improvement and adaptation to changing or novel situations.

3.2. Previous Works

The paper builds upon a line of research combining video generation with robotic control.

• Video Generation for Decision Making (e.g., UniPi [14]): Prior work has established that video generative models can serve as dynamics models, reward functions, or pixel-based planners for decision-making tasks. UniPi [14] is particularly relevant as the proposed method SAIL bases its implementation on its framework.

  • UniPi Framework: UniPi utilizes a text-to-video diffusion model to synthesize a text-conditioned sequence of future frames (a visual plan). This plan is then translated into executable actions by a separately trained Inverse Dynamics Model (IDM). The IDM takes consecutive pairs of frames from the visual plan and predicts the action required to transition between them. This decouples the planning (visual frames) from the execution (actions).
  • Implications: The quality of the visual plan heavily influences downstream robotic performance and generalization. UniPi demonstrated how to achieve universal policies via text-guided video generation.

• Adapting Pretrained Video Models (e.g., Adapt2Act [5], Probabilistic Adaptation [27]):

  • Challenge: Video generative models trained only on in-domain videos often struggle with generalization to novel tasks due to a paucity of data scale.
  • Solution: Integrating knowledge from large-scale datasets (e.g., web-scale video data) with in-domain examples can improve generalization.
  • Probabilistic Adaptation (PA) [27]: This technique performs adaptation through score composition during the sampling stage of diffusion models, without finetuning the weights of large pretrained models. It effectively guides the generation process of a small in-domain model using a more general pretrained model.
  • Inverse Probabilistic Adaptation (IPA) [5]: Adapt2Act extends PA to IPA. IPA creates a powerful, generalizable, text-conditioned visual planner by combining a large-scale model pretrained on web-scale video data with a video model trained on a small set of in-domain demonstrations via score composition.
    • How it works (Intuition): The adapted video model draws upon large-scale motion priors and powerful zero-shot text conditioning capabilities from the web-pretrained video model to facilitate generalization. Simultaneously, it leverages the in-domain video model to generate visual plans that respect the environment-specific visual characteristics and dynamics of the robotic setting. The result is an adapted video model that can generate in-domain-appearing plans for novel, unseen tasks conditioned on natural language.

• Self-Improving Generative Models (e.g., for LLMs [28, 29, 30], VideoAgent [33]):

  • Concept: Agents can continuously improve by learning from self-produced cumulative experience. This has been explored for Large Language Models (LLMs) where they can act as reward functions or data synthesizers.
  • VideoAgent [33]: This work refines video generation through self-conditioning consistency and feedback from a VLM (Vision-Language Model), collecting successful plan rollouts for finetuning.
  • Differentiation from VideoAgent: SAIL bases its improvement loop on self-adaptation using internet-scale video priors to synthesize improved visual plans for tasks unseen during initial in-domain training. It also demonstrates robustness to suboptimal initial data and relaxed filtering requirements.

3.3. Technological Evolution

The field has evolved from relying solely on in-domain expert demonstrations for robotic learning to increasingly incorporating large-scale, diverse, pre-collected offline data (like web-scale videos) to enhance generalization capabilities. The next logical step, and where this paper fits, is to move beyond purely offline data to enable continuous online improvement through self-collected experience. This allows agents to adapt to novel tasks and refine their skills in a self-supervised or self-adaptive manner, addressing the limitations of fixed datasets and the high cost of expert data collection. SAIL represents this shift towards experience-driven learning for visual planners.

3.4. Differentiation Analysis

Compared to related work, SAIL's core innovations and differentiations are:

1. Iterative Self-Improvement Loop for Visual Planners: While Adapt2Act (and IPA) enabled generalizable visual planning using offline internet data, SAIL introduces an iterative loop where the in-domain video model continuously finetunes itself on self-collected online experience gathered using the IPA-adapted planner. This moves beyond static generalization to dynamic self-improvement.
2. Leveraging Internet Priors for Online Learning: SAIL uniquely combines the generalization power of internet-scale video models (through IPA) with online self-collected experience. This fusion allows for the generation of higher-quality visual plans for novel tasks from the outset, which then enables the collection of more successful trajectories for finetuning. This creates a virtuous cycle that is not present when using in-domain models alone for self-improvement.
3. Robustness to Data Quality and Filtering: A significant differentiator is SAIL's demonstrated robustness. It shows consistent improvement even without explicit filtering of self-collected trajectories (meaning even unsuccessful attempts can contribute to learning) and when initialized with suboptimal in-domain demonstration data. This makes SAIL more practical for real-world robotic deployments where perfect expert data or meticulous filtering are often unavailable or expensive.
4. Bootstrapping High-Performance Visual Planners: SAIL effectively bootstraps a high-performance video model for novel robotic tasks. It starts with a potentially limited in-domain model, enhances its generalization via IPA, and then continuously refines it through online interaction, ultimately achieving performance beyond what either component could do in isolation.

4. Methodology

The Self-Adapting Improvement Loop (SAIL) is a framework designed to enable a video generative model, initially trained on a set of in-domain demonstrations, to iteratively improve its visual planning performance for a specific task of interest in a self-adaptive manner. This involves integrating the in-domain model with a large-scale pretrained video model and continuously updating the in-domain model with self-collected experience.

4.1. Principles

The core idea behind SAIL is to create a virtuous loop where an in-domain video model learns from its own successful interactions with the environment. This self-improvement is facilitated by:

1. Strong Generalization Prior: Utilizing a large-scale internet-pretrained video model to provide robust motion priors and text-conditioned generalization capabilities, especially for novel tasks not seen during the in-domain model's initial training.
2. Domain-Specific Refinement: Employing an in-domain video model to ensure the generated visual plans are consistent with the environment's specific visual characteristics and dynamics.
3. Online Experience Collection: Generating trajectories by executing visual plans in the real or simulated environment.
4. Iterative Finetuning: Feeding these self-produced trajectories back into the in-domain video model for finetuning, thereby continuously improving its ability to generate performant visual plans for the task of interest.

4.2. Core Methodology In-depth (Layer by Layer)

SAIL's methodology can be broken down into three main components: Video Models as Visual Planners, Inverse Probabilistic Adaptation (IPA), and the Self-Adapting Improvement Loop itself.

4.2.1. Video Models as Visual Planners

SAIL bases its implementation on the UniPi framework [14], which conceptualizes video generative models as visual planners for decision-making.

Process:

1. Visual Plan Synthesis: A text-to-video model is used to synthesize a text-conditioned sequence of future frames. This sequence represents the desired task plan in visual form. For instance, given a prompt like "close the drawer," the model generates a video showing a robot closing the drawer.
2. Action Translation: To physically execute this visual plan, a separately trained Inverse Dynamics Model (IDM) is employed. The IDM translates consecutive pairs of visual frames from the plan into executable robotic actions. These actions are then performed directly in interaction with the environment.
   • The IDM is typically task-agnostic and trained on general in-domain interaction data. Its role is to bridge the gap between visual goals and robot movements.
   • The paper states the IDM takes as input the embeddings of two video frames, extracted using VC-1 [36], and outputs a prediction of the action that enables the transition between them.
   • The paper provides the conceptual formula for the IDM: $ f_{\mathrm{in}}(o_t, o_{t+1}) = a_t $ Where:
     • $f_{\mathrm{in}}$: Represents the inverse dynamics model.
     • $o_t$: Denotes the observation (e.g., video frame) at time $t$.
     • $o_{t+1}$: Denotes the observation (e.g., video frame) at time $t+1$.
     • $a_t$: Represents the action predicted by the IDM that transitions the robot from state $o_t$ to $o_{t+1}$.
3. Control Loop: The execution strategy (how frequently the plan is re-evaluated and actions are derived) can vary.
   • Open-loop control: Execute all actions from a single visual plan sequentially without re-planning. This is computationally cheap but prone to error accumulation if the environment deviates.
   • Closed-loop control: Execute only the first action, then re-plan based on the new observation. This is reliable but computationally expensive due to frequent re-planning.
   • Semi-open-loop control: A balance, executing a few actions before re-planning.

4.2.2. Inverse Probabilistic Adaptation (IPA)

IPA is a training-free approach that adapts generally pretrained text-to-video models for domain-specific video generation. It leverages score composition during the sampling procedure of diffusion models.

The TPA function in Algorithm 1 refers to Inverse Probabilistic Adaptation (IPA) as per the paper's description in Section 3.2.

Formula: The score predicted by an in-domain video model $\epsilon_{\theta}$ is composed with the score prediction of a web-scale pretrained model $\epsilon_{\mathrm{general}}$ during the sampling procedure, as depicted in the following function:

$$\widetilde{\epsilon}_{\mathrm{inv}} = \epsilon_{\mathrm{general}}(\tau_t, t) + \alpha \Bigl( \epsilon_{\mathrm{general}}(\tau_t, t \mid \mathrm{text}) + \gamma \epsilon_{\theta}(\tau_t, t \mid \mathrm{text}) - \epsilon_{\mathrm{general}}(\tau_t, t) \Bigr)$$

Where:

• $\widetilde{\epsilon}_{\mathrm{inv}}$: The adapted score function used for visual planning. This is the combined score that guides the diffusion sampling process to generate a video.

• $\epsilon_{\mathrm{general}}(\tau_t, t)$: The score predicted by the internet-scale pretrained video model (e.g., AnimateDiff) for a given noisy video latent $\tau_t$ at time step $t$. This represents the base denoising guidance from the general model without text conditioning.

• $\epsilon_{\mathrm{general}}(\tau_t, t \mid \mathrm{text})$: The score predicted by the internet-scale pretrained video model for a noisy video latent $\tau_t$ at time step $t$, conditioned on a text prompt. This incorporates the zero-shot text conditioning and general motion priors from the large model.

• $\epsilon_{\theta}(\tau_t, t \mid \mathrm{text})$: The score predicted by the in-domain video model (with parameters $\theta$) for a noisy video latent $\tau_t$ at time step $t$, conditioned on a text prompt. This brings in the environment-specific visual characteristics and dynamics learned from the in-domain data.

• $\alpha$: The guidance scale of text-conditioning. A higher $\alpha$ increases the influence of the text prompt on the generated video.

• $\gamma$: The prior strength. This parameter controls the influence of the in-domain video model's score during composition.

• $\tau_t$: The noisy video latent at a particular sampling time step $t$ in the diffusion process.

  Intuition: The formula essentially takes the base denoising guidance from the general model (first term), and then adds a scaled difference. This difference term first calculates the text-conditioned general model's score and the text-conditioned in-domain model's score. The in-domain model's score is weighted by $\gamma$. The final part subtracts the unconditioned general model's score. This way, the in-domain model guides the generation to be domain-specific, while the general model provides broad generalization and text-conditioning capabilities. This combined score is then used to iteratively refine the noisy latent $\tau_t$ into a coherent visual plan.

4.2.3. Self-Adapting Improvement Loop (SAIL)

SAIL is an iterative framework that combines offline data with online experience to create a visual planner that continuously improves for a particular task of interest.

Algorithm 1: Self-Adapting Improvement Loop (SAIL)

Input:

• Initial in-domain video model $\epsilon_{\theta}$
• Inverse dynamics model $f$
• Frozen internet-pretrained video model $\epsilon_{\mathrm{general}}$
• Number of iterations $K$
• Number of rollouts per iteration $N$
• Environment env
• Task prompt $g$
• In-domain initial training data $\mathcal{D}_{\mathrm{ini}}$

Output:

• Self-improved in-domain model $\hat{\epsilon}_{\theta}$

  1: $\hat{\epsilon}_{\theta} \gets \epsilon_{\theta}$ 2: $\mathcal{D} \gets \mathcal{D}_{\mathrm{ini}}$ or $\phi$ $\triangleright$ Initialize finetuning data with $\mathcal{D}_{\mathrm{ini}}$ or an empty set 3: for $i = 1 , . . . , K$ do 4: $\mathcal{D}_{\mathrm{self}} \gets \phi$ 5: $\widetilde{\epsilon}_{\mathrm{inv}} \longleftarrow \mathrm{TPA}(\widehat{\epsilon}_{\theta}, \epsilon_{\mathrm{general}}, g)$ 6: for $j = 1, . . . , N$ do 7: env.reset(g) 8: $\mathcal{D}_{\mathrm{self}} \gets \mathcal{D}_{\mathrm{self}} \cup \mathrm{Visual\_Planning\_Rollout}(\mathrm{env}, \widetilde{\epsilon}_{\mathrm{inv}}, f)$ $\triangleright$ Optional data filtering 9: end for 10: $\mathcal{D} \gets \mathcal{D} \cup \mathcal{D}_{\mathrm{self}}$ 11: Finetune in-domain model $\hat{\epsilon}_{\theta}$ with accumulated data $\mathcal{D}$ $\triangleright$ $f$ can be optionally finetuned 12:end for 13: return $\hat{\epsilon}_{\theta}$

Step-by-step Explanation of Algorithm 1:

1. Initialization of In-Domain Model (Line 1): The in-domain video model to be improved, initially pretrained as $\epsilon_{\theta}$, is set as the current adaptable model $\hat{\epsilon}_{\theta}$.

2. Initialization of Finetuning Data (Line 2): The dataset $\mathcal{D}$ used for finetuning is initialized. This can either be the original in-domain initial training data $\mathcal{D}_{\mathrm{ini}}$ or an empty set $\phi$. The choice depends on whether the past "expert" data should be continuously included in finetuning, or if the model should only adapt to self-collected experience.

3. Iteration Loop (Lines 3-12): The main SAIL process runs for $K$ iterations.

4. Initialize Self-Collected Data (Line 4): For each iteration $i$, a temporary dataset $\mathcal{D}_{\mathrm{self}}$ is initialized as empty. This will store the trajectories collected in the current iteration.

5. Adaptation with IPA (Line 5): The current in-domain model $\hat{\epsilon}_{\theta}$ is combined with the frozen internet-pretrained video model $\epsilon_{\mathrm{general}}$ using the Inverse Probabilistic Adaptation (TPA) function (explained above). This generates the adapted score function $\widetilde{\epsilon}_{\mathrm{inv}}$, which serves as the visual planner for this iteration. This step leverages the strong generalization capabilities of the internet-scale model and the domain understanding of the in-domain model to create effective plans for the task prompt $g$.

6. Rollout Collection Loop (Lines 6-9): The robot performs $N$ rollouts in the environment using the adapted visual planner.

7. Environment Reset (Line 7): For each rollout $j$, the environment env is reset for the given task prompt $g$.

8. Visual Planning Rollout (Line 8): The Visual_Planning_Rollout function is called:

   • It uses the adapted visual planner $\widetilde{\epsilon}_{\mathrm{inv}}$ to synthesize visual plans (sequences of frames) for the task prompt $g$.
   • The inverse dynamics model $f$ translates these visual plans into executable actions.
   • These actions are then executed in the environment env.
   • The resulting trajectory (sequence of observations and actions) is collected and added to $\mathcal{D}_{\mathrm{self}}$.
   • Optional Data Filtering: During this step, the collected trajectories can optionally be filtered (e.g., only successful trajectories are kept). The paper explores the impact of this filtering.

9. Accumulate Data (Line 10): After all $N$ rollouts for the current iteration are collected, the self-collected data $\mathcal{D}_{\mathrm{self}}$ is added to the accumulated finetuning dataset $\mathcal{D}$.

10. Finetune In-Domain Model (Line 11): The in-domain model $\hat{\epsilon}_{\theta}$ is then finetuned using the entire accumulated dataset $\mathcal{D}$. This step updates the model parameters based on both initial demonstrations (if $\mathcal{D}_{\mathrm{ini}}$ is included) and the newly self-collected experience. The inverse dynamics model $f$ can also be optionally finetuned, but the paper states it was kept frozen for fairness in their experiments.

11. Return Self-Improved Model (Line 13): After $K$ iterations, the final self-improved in-domain model $\hat{\epsilon}_{\theta}$ is returned.

    This loop ensures that as the robot interacts with the environment, its in-domain model continuously adapts and improves its ability to perform the task of interest, especially for novel tasks.

Figure 1 from the original paper helps visualize this framework:

该图像是示意图，展示了自适应改善环 (SAIL) 的框架。左侧展示了如何利用互联网规模的视频和领域内视频进行预训练，右侧描述了自适应改善循环的各个步骤，包括适应、视觉计划滚动和微调，同时链接两个预训练的视觉生成模型。公式 $f_{in}(o_t, o_{t+1}) = a_t$ 描述了逆动力学。

The left side shows the initial setup: an in-domain video model (trained on general demonstrations) and a general internet-pretrained video model. These two are composed using IPA to form an adapted visual planner. This planner then interacts with the environment, producing trajectories. The right side depicts the SAIL loop, where these self-produced trajectories are fed back to finetune the in-domain model, leading to continuous improvement.

5. Experimental Setup

The authors evaluate SAIL across two primary robotic settings: a simulated environment (MetaWorld-v2) and a real-world robot (Franka Emika Panda arm).

5.1. Datasets

5.1.1. Synthetic Environment: MetaWorld-v2

• Source: MetaWorld-v2 [34] is a benchmark for multi-task and meta reinforcement learning.
• Characteristics: Provides a wide selection of manipulation tasks and ground-truth success evaluations.
• In-domain Training Data: 25 demonstrations collected from 7 different MetaWorld tasks (marked with an asterisk in Table A1). These are used for initial training of the in-domain video model and inverse dynamics model.
• Evaluation Tasks: 6 MetaWorld tasks, of which 5 are novel tasks (not marked with an asterisk in Table A1), specifically chosen to assess generalization.
• Self-Collection: For each SAIL iteration, 30 trajectories are collected from the environment during visual planning for in-domain finetuning.

Concrete Example of Data Sample (Conceptual): A MetaWorld demonstration would consist of a video sequence showing a robot arm performing a task (e.g., closing a door, pushing a block) and the corresponding actions. For instance, a video of a robot arm successfully closing a door would be an in-domain demonstration.

The following table (Table A1 from the original paper) lists the tasks and associated text prompts used for evaluating SAIL:

Table A1: Task-Prompt Pairs. We include a comprehensive list of tasks and their text prompts for in-domain training and evaluation. " * > denotes tasks seen during initial training of the in-domain model. We also provide the prompts used to interface with the internet-pretrained text-to-video model during adaptation with IPA.

5.1.2. Real-World Environment: Franka Emika Panda Robot Arm

The real-world experiments demonstrate SAIL's practicality and robustness to real-world factors.

Task 1: Pushing Colored Cups

• Scene: A consistent setting of 3 differently colored cups (e.g., as shown in Figure 1).
• Objective: Robot arm accurately locates and pushes a specified colored cup forward, conditioned on natural language.
• In-domain Training Data:
  • Set of four colors: red, green, blue, pink.
  • 12 possible unique tasks formed from combinations of these seen colors (e.g., "push red cup" when red, green, blue are present).
  • 10 human-teleoperated demonstrations for each of the 12 tasks, totaling 120 training videos.
• Generalization Evaluation:
  • Two novel, unseen colors: orange, purple.
  • Evaluation is an average over 5 rollouts for every possible pair combination of a seen color with a novel color. For example, for "push orange cup," this would include scenarios where orange is present alongside red, green, blue, and pink.
  • This translates to 30 videos per novel color (e.g., 5 rollouts * 6 combinations for orange + seen colors).
• Self-Collection: In each SAIL iteration, previous self-collected data is combined with initial demonstrations for in-domain finetuning.

Concrete Example of Data Sample (Conceptual): A video of a Panda arm picking up a red cup, with the text prompt "push the red cup". For evaluation, the robot might be prompted "push the orange cup" where orange is a novel color.

Task 2: Opening Colored Drawers

• Scene: Two distinctly colored closed drawers.
• Objective: Robot arm selects and opens the drawer specified via a user-provided text prompt.
• In-domain Training Data:
  • Set of three colors: red, green, blue.
  • 24 possible drawer placement combinations for each ordered pair of seen colors (e.g., red and green drawers, red and blue, etc.). There are 6 such pairs.
  • This amounts to a total of 144 human-teleoperated demonstration training videos.
• Generalization Evaluation:
  • One novel, unseen color: yellow.
  • Performance is calculated as an average over 12 rollouts for every possible pairing of the novel color with a seen color (e.g., yellow and red, yellow and green, yellow and blue).
  • This totals 36 self-collected trajectories per iteration.
• Self-Collection: Similar to the cup pushing task, previous self-collected data is combined with initial demonstrations for in-domain finetuning.

Concrete Example of Data Sample (Conceptual): A video of a Panda arm opening a red drawer, with the text prompt "open the red drawer". For evaluation, the robot might be prompted "open the yellow drawer" where yellow is a novel color.

5.1.3. Dataset Choice Justification

The chosen datasets (MetaWorld-v2 and real-world Panda arm tasks) are effective for validating SAIL's performance:

• MetaWorld-v2 allows for thorough assessment of visual planning performance trends across many novel tasks and provides ground-truth success evaluations for quantitative comparisons.
• Real-world Panda arm experiments demonstrate the practicality and robustness of the approach against real-world confounding factors like lighting conditions, and validate its ability to generalize to unseen color combinations.

5.2. Evaluation Metrics

The primary evaluation metric used across all experiments is Success Rate.

5.2.1. Conceptual Definition

Success Rate quantifies the proportion of attempts (trajectories or rollouts) in which the robot successfully completes the specified task according to predefined criteria. It measures the effectiveness and reliability of the visual planner and inverse dynamics model in achieving the desired robotic behavior. A higher success rate indicates better performance and generalization.

5.2.2. Mathematical Formula

Let $N_{\text{total}}$ be the total number of evaluation attempts (rollouts or trajectories) for a given task, and $N_{\text{success}}$ be the number of those attempts that are deemed successful. The success rate (SR) is calculated as:

$$\mathrm{SR} = \frac{N_{\text{success}}}{N_{\text{total}}}$$

5.2.3. Symbol Explanation

• $\mathrm{SR}$: The Success Rate, expressed as a percentage or a fraction between 0 and 1.
• $N_{\text{success}}$: The count of successful task completions.
• $N_{\text{total}}$: The total number of attempts or rollouts for the task.

How Success is Judged:

• MetaWorld: Ground-truth success evaluations are provided by the simulation environment.
• Real-World (Panda Arm): Success is judged by human observers for evaluation. This human-evaluated success signal is also used for optional data filtering on the rollouts.

5.3. Baselines

The paper primarily compares SAIL against:

• In-Domain Only: This baseline represents the in-domain video model that is finetuned using self-collected experience without the adaptation step involving the internet-scale pretrained video model (i.e., without IPA). This highlights the importance of leveraging large-scale priors for effective self-improvement.

• SAIL (IPA): The proposed method, which utilizes Inverse Probabilistic Adaptation (TPA in Algorithm 1) to combine the in-domain model with the internet-scale model for planning, and then finetunes the in-domain model on self-collected experience.

• SAIL (PA): An additional baseline explored in Appendix D.1, where SAIL is implemented using Probabilistic Adaptation (PA) [27] instead of IPA. This helps to compare different score composition strategies for adaptation within the SAIL framework. PA uses the in-domain model as the main denoiser, guided by the general model, whereas IPA uses the general model as the main denoiser, guided by the in-domain model.

  These baselines are representative because they isolate the contribution of the internet-scale pretrained video model and the specific adaptation technique (IPA vs. PA) to the overall self-improvement process.

5.4. Implementation Details

5.4.1. Inverse Dynamics (IDM)

• Architecture: A small MLP network built on top of a pretrained pixel-based representation network.

• Representation Network: VC-1 [36] is used for extracting embeddings of video frames.

• Input: Embeddings of two video frames.

• Output:

  • MetaWorld: Predicts 4 dimensions (likely joint positions or end-effector deltas). Input/Output Dimension: 1536 / 4.
  • Panda Arm: Predicts 7 dimensions (likely end-effector position and orientation, or joint torques/velocities). Output Dimension: 7.

• Frame Skip:

  • MetaWorld: Consecutive frames (frameskip of 1).
  • Panda Arm: Frameskip of 16 (IDM predicts action between frames 16 steps apart).

• Parameter Count: Total 85.81M parameters, with 85.80M inherited from VC-1, and 10759 from the additional MLP.

• Finetuning: The IDM is not finetuned during SAIL iterations; it is kept frozen and reused for all tasks within the same environment to ensure fairness and highlight the visual plan quality.

• Training Hyperparameters: The following are the hyperparameters for Inverse Dynamics Model Training (Table A2 from the original paper):

  Hyperparameter	Value
  Input Dimension Output Dimension (MetaWorld)	1536
  	4
  Output Dimension (Panda)	7
  Training Epochs	20
  Learning Rate	1e-5
  Optimizer	AdamW

  Table A2: Hyperparameters of Inverse Dynamics Model Training. We list the relevant hyperparameters of training the inverse dynamics model.

5.4.2. In-Domain Model

• Architecture: Based on AVDC [3], a small-scale diffusion model that conditions on natural language and an initial pixel frame. An additional Cross-Attention layer is added to every level of the U-Net to improve text-conditioning.

• U-Net Instantiation:

  • MetaWorld: 3 ResNet blocks.
  • Panda Arm: 2 ResNet blocks.

• Parameter Count: The following are the In-Domain Model Components (Table A3 from the original paper):

  Component	# Parameters (Millions)
  U-Net (MetaWorld)	116.71
  U-Net (Panda Arm)	93.38
  Text Encoder (openai/clip-vit-base-patch32)	63.2

  Table A3: In-Domain Model Components. SAIL relies on a small in-domain text-to-video model, which we base our implementation off of prior work [3]. We list the size of the components of the model architecture used.

  • Total for MetaWorld: 179.91M parameters.
  • Total for Real-World: 156.58M parameters.

• Initial Training:

  • MetaWorld: 70K training steps.
  • Panda Arm: 88K steps.
  • Batch Size: 8.
  • Learning Rate: 2e-5.

• SAIL Finetuning:

  • MetaWorld: 10K steps per iteration, batch size 4, learning rate 1e-5.
  • Panda Arm Pushing: 8K steps per iteration, batch size 8, learning rate 2e-5.
  • Panda Arm Drawer Opening: 10K steps per iteration, batch size 8, learning rate 1e-5.

• Hardware: Single NVIDIA A6000 or RTX3090 GPU.

5.4.3. Internet-Domain Model

• Model: AnimateDiff [6] (approximately 2B parameters) is used as the frozen internet-pretrained video model for Inverse Probabilistic Adaptation.

• Image Conditioning: SparseCtrl [37] is used to enable image-conditioned video generation.

• Parameter Count: The following are the AnimateDiff Components (Table A4 from the original paper):

  Component	# Parameters (Millions)
  VAE (Encoder)	34.16
  VAE (Decoder)	49.49
  U-Net	1302.16
  Text Encoder	123.06
  ControlNet	496.73

  Table A4: AnimateDiff Components. SAIL relies on a internet-scale text-to-video model; in this work we use AnimateDiff. We thus list the size of components of the AnimateDiff checkpoint used. The checkpoint is used purely for inference, and is not modified or updated in any way. Note that the VAE Decoder is not utilized in our framework.

  • Total: 2.005B parameters.

• Usage: Used purely for inference; its weights are frozen and not updated.

5.4.4. Visual Planning Hyperparameters

• Future Frames: Predicts 8 future frames, conditioned on the current observation and task prompt.
• Sampling: DDIM [38] sampling for 25 steps.
• Text-Conditioning Guidance Scale ($\alpha$ in IPA formula):
  • MetaWorld: 2.5
  • Panda Arm Pushing: 7.0
• Prior Strength ($\gamma$ in IPA formula): 0.5

5.4.5. Choices of Control Loop

• Panda Arm Pushing and Drawer Opening: Open-loop control is employed (all 8 actions from a single visual plan executed sequentially). This prioritizes execution speed, as visual plans were deemed sufficiently accurate.
• MetaWorld Experiments: Semi-open-loop control is utilized (executing half of the plan, e.g., 4 actions, before re-planning). This balances performance and efficiency.

6. Results & Analysis

6.1. Core Results Analysis

The SAIL framework demonstrates continuous performance improvement across both simulated and real-world environments for novel tasks.

The following figure (Figure 2 from the original paper) showcases the average success rate of SAIL on MetaWorld and Panda Arm tasks:

该图像是图表，展示了SAIL在MetaWorld和真实机器人任务中的成功率。图表列出了四个任务的成功率随迭代次数的变化，其中SAIL表现出更为稳健的改进行为。通过对比SAIL与仅使用领域内数据的结果，显示SAIL能够在多个迭代中持续提升任务成功率。

• MetaWorld Results (Leftmost plot):
  • SAIL (IPA) shows a clear upward trend in average success rate over 3 iterations for 6 MetaWorld tasks (5 novel).
  • The initial success rate at Iteration 0 for SAIL (IPA) is higher than In-Domain Only, highlighting the immediate benefit of IPA's adaptation with large-scale offline data for novel task generalization.
  • In-Domain Only (without IPA) also shows some initial improvement but does not consistently hold over multiple iterations and does not achieve as high overall performance as SAIL (IPA). This underscores the critical role of IPA in facilitating sustained self-improvement.
• Panda Arm Results (Middle two plots for cup pushing, Rightmost plot for drawer opening):

  • For the novel tasks of pushing orange and purple cups, and opening a yellow drawer, SAIL (IPA) consistently improves performance over iterations.

  • In contrast, In-Domain Only either shows negligible improvements or, in the case of pushing the purple cup and opening the yellow drawer, experiences a monotonic decrease in performance. This demonstrates that finetuning on self-collected experience alone (without the IPA adaptation) can sometimes reinforce suboptimal behaviors if the initial planning capabilities for novel tasks are insufficient.

  • The improvement is averaged over 30-36 rollouts, across different combinations of novel and seen colors, indicating robust generalization.

    These results highlight that SAIL effectively leverages large-scale offline data (via IPA) alongside online self-collected experience to achieve self-improving performance on novel tasks, a capability not consistently matched by using the in-domain model alone.

The following figure (Figure 3 from the original paper) provides qualitative results on visual plans refinement:

该图像是一个示意图，展示了三个任务在迭代0（上）和迭代2（下）的视觉计划。图中机器人在进行"推动橙色杯子"、"打开黄色抽屉"和"关闭抽屉"的操作。尽管迭代0的视觉计划中物体模糊且未能完成指定任务，但经过两次SAIL迭代后，视觉计划得到了显著改善。

• Qualitative Improvement: Figure 3 illustrates the refinement of visual plans for real-robot manipulation and MetaWorld tasks from Iteration 0 to Iteration 2.
  • At Iteration 0, without prior experience on the specified novel tasks, IPA often synthesizes blurry objects and leads to incorrect task execution in the visual plan.
  • After two iterations of SAIL, the visual plans show significant improvement: they are clearer, objects are well-defined, and the plans depict successful task completion behaviors even with random initial object locations.
  • This qualitative improvement in visual plan quality directly translates to the robot arm's ability to execute the task successfully in the actual environment interaction. This confirms that SAIL not only boosts quantitative metrics but also generates more coherent and actionable visual foresight.

6.2. Data Presentation (Tables)

The following table (Table A5 from the original paper) provides a detailed breakdown of MetaWorld task performance:

Table A5: MetaWorld Task Performance. We provide a detailed list of task performance for the leftmost plot in Figure 2. We report the mean success rate across 6 tasks, aggregated over 3 seeds each. Settings with improving behaviors are highlighted with shaded backgrounds. Compared to indomain only baselines, SAIL (IPA) enables continuous improvement on average task performance across iterations, and achieves the best overall success rate on Iteration 2.

• Detailed MetaWorld Analysis (Table A5):
  • For Door-Close* (a seen task), both In-Domain Only and SAIL (IPA) achieve high performance and show some improvement, with SAIL (IPA) reaching slightly higher at Iteration 2.
  • For Drawer-Open (a novel task), SAIL (IPA) shows a significant jump from 27.8% at Iteration 0 to 43.3% at Iteration 1, demonstrating effective self-improvement, although it slightly dips at Iteration 2 (37.8%). In-Domain Only has much lower initial performance and slower, smaller gains.
  • Window-Close shows 0% for both, suggesting it might be a particularly challenging task or require more iterations/different initial conditions.
  • Window-Open (novel task) sees continuous improvement with SAIL (IPA) from 52.2% to 73.3%, whereas In-Domain Only's performance degrades by Iteration 2.
  • Button-Press shows very low performance for both, with SAIL (IPA) achieving minor gains.
  • The Average row clearly shows SAIL (IPA)'s continuous improvement from 24.4% to 34.4%, consistently outperforming In-Domain Only which only reaches 28.1%.

6.3. Ablation Studies / Parameter Analysis

6.3.1. SAIL without Experience Filtering

The following figure (Figure 4 from the original paper) shows ablation results on data filtering:

该图像是图表，展示了SAIL方法在MetaWorld和Panda Arm Pushing任务中的成功率随迭代次数的变化。图中分别比较了在有和没有过滤的情况下的表现，结果显示无论是否进行数据过滤，SAIL方法的表现均持续提升，验证了其在缺乏oracle过滤信号时的稳健性。

• Robustness to Filtering: The paper investigates the impact of filtering self-collected data (keeping only successful trajectories) versus not filtering (using all trajectories regardless of outcome).
  • MetaWorld (Figure 4a): Surprisingly, not filtering actually leads to slightly better performance than filtering for both In-Domain Only and SAIL. This suggests that even failed demonstrations can provide meaningful behavioral information for finetuning.
  • Panda Arm (Figure 4b): No filtering still facilitates continuous improvement over every iteration through SAIL. This is a significant finding, indicating that SAIL is robust in settings where manual curation of experience (filtering) is expensive or impractical.
  • Relabeling Strategy: For the Panda arm pushing tasks, a relabeling strategy was also tested, where unsuccessful trajectories were prepended with "not" in the text prompt. While this was preferable to no filtering for the In-Domain Only model, it did not substantially aid performance when large-scale text-to-video priors (i.e., IPA) were utilized, further emphasizing SAIL's inherent robustness.

6.3.2. SAIL with Suboptimal Data

The following figure (Figure 5 from the original paper) shows SAIL results with suboptimal in-domain data:

该图像是图表，展示了在不同迭代中，使用传统的在域训练与自适应改善循环（SAIL）方法的任务成功率变化。作为对比，三个子图分别代表仅在域数据、SAIL（IPA）与平均任务表现的成功率，显示在不同情况下，SAIL方法的表现持续改善，成功率 surpasses 在域基础线。

• Robustness to Initial Data Quality: To test SAIL's robustness, the in-domain model was initialized with suboptimal data (simulated trajectories where 70% of actions were random, resulting in low task success). No filtering was applied during SAIL iterations.

  • Even under these challenging conditions, SAIL (IPA) demonstrates continuously improving behavior for the 4 highlighted novel MetaWorld tasks (Drawer Close, Window Open, Button Press, and an unspecified fourth, likely from the average).
  • The average performance of SAIL (IPA) (rightmost plot in Figure 5) consistently increases across iterations, surpassing the In-Domain Only baseline.
  • In-Domain Only with suboptimal initial data and no filtering shows no significant improvements on average. This is because, without the IPA adaptation, the in-domain model struggles to collect sufficient successful online experience and may reinforce its suboptimal behavior through unfiltered finetuning.

• Explanation for Robustness: SAIL's robustness to suboptimal initialization is attributed to IPA's ability to overcome the suboptimality gap by providing strong generalization capabilities from the internet-scale model. This enables the collection of enough performant trajectories in early iterations to bootstrap further improvement.

  The following figure (Figure A1 from the original paper) shows full SAIL results with suboptimal in-domain data without experience filtering across 6 tasks:

  该图像是一个包含三个部分的图表，展示了在不同迭代下，基于原始领域数据、SAIL方法和平均任务表现的成功率。第一部分显示了仅使用原始领域数据时的表现，第二部分展示了SAIL方法应用后的评估，最后一部分汇总了平均成功率。整体结果显示SAIL方法在新任务上实现了显著的性能提升。

The following table (Table A6 from the original paper) details task performance with suboptimal initial data:

Table A6: Detailed Task Performance with Suboptimal Initial Data. We compare visual planning performance across iterations on in-domain only, SAIL (IPA) and additional SAIL (PA) setups. We report the mean success rate across 6 tasks, aggregated over 3 seeds each. Settings with improving behaviors are highlighted with shaded backgrounds.

• Comparison of IPA vs. PA (Table A6):
  • Probabilistic Adaptation (PA) also exhibits improving behaviors on several tasks and average task performance, similar to IPA.
  • 3 out of 6 unseen tasks continuously improve through SAIL (PA), whereas IPA enables improvements on 4 unseen tasks over iterations.
  • SAIL (IPA) generally achieves higher task performance on average (37.0% at Iteration 2 vs. 35.6% for PA) and the best overall success rate on the last iteration.
  • The authors conclude that IPA serves as a more robust adaptation technique, especially with suboptimal in-domain initialization, allowing more performant trajectories to be collected and subsequently facilitating improvements of the in-domain video model through SAIL.
• Challenges with Very Low Initial Success: For tasks like Drawer-Open and Window-Open under suboptimal data (where initial success rates are 0% or very low), it is difficult for performance to improve significantly. When filtering is not applied, the model may continue to reinforce suboptimal trajectories, hindering meaningful gains, similar to the In-Domain Only case. This highlights a boundary condition where sufficient initial successful experience is needed, even if minimal, for the self-improvement loop to kick in effectively.

6.4. Additional Plan Visualizations

The appendix provides additional visual plans and their execution results, further supporting the qualitative findings.

• SAIL with Experience Filtering (Figures A2-A9): These figures demonstrate successful execution for various tasks like Drawer Close, Window Close, Orange Cup Pushing, Purple Cup Pushing, and Yellow Drawer Opening with experience filtering applied. They show how visual plans become clearer and lead to successful robot actions after multiple SAIL iterations.
• Filtering-free SAIL (Figures A10-A12): These figures show that SAIL can still lead to successful execution even without experience filtering for tasks like Drawer Close, Orange Cup Pushing, and Window Close (even with suboptimal data), reinforcing the robustness claims.

7. Conclusion & Reflections

7.1. Conclusion Summary

This work introduces SAIL (Self-Adapting Improvement Loop), a novel framework for solving novel robotic tasks through visual planning. SAIL begins with an in-domain video model (pretrained on a small set of demonstrations) and leverages Inverse Probabilistic Adaptation (IPA) with a large-scale internet-pretrained video model to create a performant, generalizable visual planner. This planner then iteratively collects experience trajectories which are used to finetune and self-improve the in-domain video model. By effectively combining large-scale offline data with online self-acquired experience, SAIL demonstrates its ability to bootstrap a high-performance text-conditioned visual planner for desired tasks. The experimental results, across both MetaWorld simulations and real-world Panda arm tasks, show continuous performance improvements for novel tasks. Furthermore, SAIL proves remarkably robust to the absence of experience filtering and the quality of initial in-domain demonstration data.

7.2. Limitations & Future Work

The authors acknowledge the following limitations:

• Initial Success Rate Assumption: SAIL implicitly assumes that the initial in-domain model, when adapted with an internet-pretrained video model, achieves a reasonable success rate to collect online experience and initiate self-improvement. This assumption may not hold if the novel task is excessively challenging, potentially leading to a lack of sufficiently good trajectories to learn from.
• Trade-off in Internet-Pretrained Model Choice: The selection of the internet-pretrained video model involves a trade-off between video quality (and thus the strength of the motion prior) and computational cost. The authors chose AnimateDiff [6] for its balance of generation quality and efficiency. Future work could explore more recent video generative models for even better visual quality and potential improvements in downstream robotic performance.

7.3. Personal Insights & Critique

This paper presents a highly practical and impactful approach to robotic learning. The SAIL framework elegantly addresses the core challenge of generalization to novel tasks and continuous improvement by synergistically combining the strengths of large-scale internet data and online self-supervised learning.

Inspirations and Applications:

• Democratization of Robot Learning: The robustness to suboptimal initial data and lack of filtering is a crucial insight. This significantly lowers the barrier to entry for deploying robotic systems, as collecting vast amounts of perfect expert demonstrations is often the most expensive and time-consuming part of robot learning. SAIL suggests that robots could start with less-than-perfect human guidance and iteratively refine their skills.
• Lifelong Learning for Robotics: SAIL provides a strong foundation for lifelong learning in robotics. A robot could be deployed, continuously collect experience, and adapt to new variations or entirely new tasks without needing to be re-engineered or re-trained from scratch on a new offline dataset.
• Bridging Simulation-to-Reality Gap: The success on real-world Panda arm tasks demonstrates the potential for SAIL to be effective in physical environments, which is often a major hurdle for sim-to-real transfer. The visual planning approach might intrinsically offer some robustness to real-world complexities as the IDM acts on visually-derived goals.
• Human-Robot Interaction: The text-conditioned nature of the visual planner makes SAIL highly adaptable to human instructions, allowing users to specify novel tasks in natural language and have the robot incrementally learn to perform them.

Potential Issues or Areas for Improvement:

• "Cold Start" Problem: While SAIL is robust to suboptimal data, the initial success rate (Iteration 0) still needs to be "reasonable." For extremely novel or complex tasks where IPA itself cannot generate any successful plans, SAIL might struggle to initiate the improvement loop. Future work could explore mechanisms to "kickstart" learning in such scenarios, perhaps by incorporating human feedback during initial failures or using pre-exploration strategies.

• Efficiency of Finetuning: Finetuning a diffusion model (even a small one) for 10K steps per iteration, especially with growing accumulated data $\mathcal{D}$, can still be computationally intensive. Investigating more parameter-efficient finetuning (e.g., LoRA, adapters) or online learning algorithms specifically designed for diffusion models could further enhance scalability.

• Long-term Catastrophic Forgetting: As the model continuously updates on self-collected data, there's a potential risk of catastrophic forgetting of previously learned skills, especially if the task distribution shifts significantly or if initial expert data is no longer included in the finetuning process (if $\mathcal{D}_{\mathrm{ini}}$ is not continuously used). The paper mitigates this by including previous data, but a deeper analysis of long-term stability would be beneficial.

• Definition of "Success": While MetaWorld provides ground truth, real-world human evaluation for success can be subjective and time-consuming. Developing more robust, automated success metrics or reward functions for real-world scenarios would be crucial for wider adoption of filtering-free or relabeling strategies.

• Generalization to New Objects/Environments: The real-world experiments focus on novel colors within a fixed set of objects (cups, drawers). Testing SAIL's ability to generalize to entirely new object categories, different robot morphologies, or significantly altered environments would be the next frontier.

  Overall, SAIL represents a significant step towards enabling robots to learn and adapt autonomously in complex, dynamic environments, pushing the boundaries of experience-driven robot learning.