1. Bibliographic Information

1.1. Title

InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation

1.2. Authors

Shuai Yang$^{2,3\dagger*}$, Hao Li$^{1,3*}$, Yilun Chen$^3$, Bin Wang$^{2,3}$, Yang Tian$^3$, Tai Wang$^3$, Hanqing Wang$^3$, Feng Zhao$^1$, Yiyi Liao$^2$, Jiangmiao Pang$^3$

• Affiliations:
  1. University of Science and Technology of China
  2. Zhejiang University
  3. Shanghai Artificial Intelligence Laboratory

1.3. Journal/Conference

Published as a preprint on arXiv.

• Status: Preprint (Published at UTC: 2025-07-23).
• Context: While currently a preprint, the work involves major institutions like Shanghai AI Lab and presents significant benchmarking efforts, indicating high relevance in the embodied AI community.

1.4. Publication Year

2025

1.5. Abstract

This paper addresses the "catastrophic forgetting" problem in Vision-Language-Action (VLA) models, where fine-tuning on robotic tasks degrades the model's general multimodal reasoning capabilities. The authors introduce InstructVLA, an end-to-end model that couples a Vision-Language Model (VLM) backbone with a specialized action expert using a Mixture-of-Experts (MoE) adaptation. They propose a training paradigm called Vision-Language-Action Instruction Tuning (VLA-IT) and curate a corresponding 650K-sample dataset. The model outperforms baselines by substantial margins (e.g., 92% improvement over a fine-tuned OpenVLA on the new SimplerEnv-Instruct benchmark) while preserving general vision-language abilities.

1.6. Original Source Link

https://arxiv.org/abs/2507.17520

2. Executive Summary

2.1. Background & Motivation

• Core Problem: Robots need to understand complex world dynamics (Reasoning) and perform precise movements (Action). Current VLA models, typically initialized from large Vision-Language Models (VLMs), suffer from a trade-off: when trained on specific robotic manipulation data, they lose their broad, pre-trained "common sense" and reasoning abilities (catastrophic forgetting). Conversely, general VLMs cannot control robots directly.
• Importance: Real-world robots must handle unstructured commands (e.g., "I'm hungry, get me something healthy") rather than just rigid templates (e.g., "Pick up the apple"). Preserving the reasoning power of VLMs is crucial for this flexibility.
• Gap: Existing solutions either simply co-train on both data types (which often fails to deeply integrate reasoning into action) or embed reasoning as text chains (which constrains the model to structured patterns). There is a lack of a unified architecture that can dynamically switch between "thinking" and "acting."

2.2. Main Contributions / Findings

1. InstructVLA Architecture: A generalist VLA model that uses a Mixture-of-Experts (MoE) adapter to dynamically switch between linguistic reasoning and latent action generation, preserving the VLM's original capabilities.
2. VLA-IT Training Paradigm: A two-stage training recipe: (1) Action Pretraining to build a specialized action expert, and (2) Vision-Language-Action Instruction Tuning to align high-level reasoning with low-level control using a curated dataset.
3. VLA-IT Dataset: A new dataset of 650K human-robot interactions annotated with diverse tasks like scenario captioning, question answering, command rewriting, and context creation.
4. SimplerEnv-Instruct Benchmark: A rigorous test suite of 80 zero-shot tasks designed to evaluate instruction following and situated reasoning, where InstructVLA significantly outperforms state-of-the-art baselines (e.g., +30.5% over SpatialVLA on in-domain tasks, +92% over OpenVLA on generalization tasks).

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, one must grasp the following concepts:

• Vision-Language-Action (VLA) Models: AI models that take visual inputs (images/video) and text instructions, and output robot actions (e.g., gripper movement). They are often built on top of Vision-Language Models (VLMs) like GPT-4V or LLaVA, which process images and text but output only text.
• Catastrophic Forgetting: A phenomenon in neural networks where learning a new task (e.g., robot control) causes the network to drastically forget previously learned knowledge (e.g., general object recognition or dialogue).
• Instruction Tuning: A training method where a model is fine-tuned on a collection of tasks described via natural language instructions, improving its ability to follow new, unseen instructions.
• Mixture of Experts (MoE): A neural network architecture where different parts of the network ("experts") specialize in different sub-tasks. A "gating" mechanism decides which expert to activate for a given input. This allows the model to have specialized pathways for "reasoning" vs. "acting" without interference.
• LoRA (Low-Rank Adaptation): An efficient fine-tuning technique that freezes the main model weights and trains only small, low-rank matrices added to the layers. This drastically reduces computational cost.
• Flow Matching: A generative modeling technique (similar to Diffusion models) used here to generate continuous robot actions. It learns to transform a simple noise distribution into a complex distribution of valid robot actions.

3.2. Previous Works

The paper builds upon and contrasts with several key works:

• RT-1 / RT-2 (Robotics Transformers): Pioneering VLA models. RT-2 co-trains a VLM on web data and robot data, treating actions as text tokens.
• OpenVLA: A recent open-source VLA based on Llama 2, using action quantization (turning continuous actions into discrete bins). It often suffers from forgetting when fine-tuned.
• Octo: A generalist robot policy trained on massive datasets (Open X-Embodiment), using a diffusion-based head for action generation.
• Magma & ChatVLA: Models that try to preserve VLM capabilities by co-training on vision-language and manipulation data. InstructVLA argues these often neglect complex embodied reasoning.
• ECoT (Embodied Chain-of-Thought): Embeds reasoning steps (plans) into the training data. InstructVLA critiques this for relying on rigid structured reasoning that constrains general multimodal ability.

3.3. Differentiation Analysis

• Vs. Co-training (Magma): While Magma trains on mixed data, InstructVLA uses MoE to structurally separate the parameters responsible for language reasoning from those for action generation, preventing interference.
• Vs. Action-Only (OpenVLA): InstructVLA explicitly trains the model to "think" (reason about the scene) before "acting," whereas standard VLAs often map pixels directly to actions.
• Vs. Explicit Planners (SayCan): Instead of using a separate LLM for planning and a policy for acting, InstructVLA is end-to-end, doing both within a single VLM backbone.

4. Methodology

4.1. Principles

The core principle of InstructVLA is decoupling and realignment.

• Decoupling: High-level semantic reasoning (understanding "pick the healthy fruit") and low-level motor control (moving the arm precisely) require different internal representations. Forcing one set of weights to do both leads to conflict (forgetting).
• Realignment: The model should use its reasoning capabilities to guide the action generation. To achieve this, the authors use a single VLM backbone equipped with a Mixture-of-Experts (MoE) adapter to dynamically switch contexts, and a separate Action Expert module to handle the high-frequency physics of movement.

The following figure (Figure 2 from the original paper) illustrates this architecture, showing the VLM reasoning, MoE adaptation, and the Flow Matching action expert:

该图像是示意图，展示了InstructVLA模型的整体结构。该模型结合了视觉-语言理解与机器人操作，生成过程分为三个步骤：语言响应、潜在动作生成和动作解码。MoE适配允许模型在推理与动作预测之间交替，最终通过动作专家解码出执行动作。

4.2. Core Methodology In-depth (Layer by Layer)

4.2.1. Architecture: The Embodied VLM

The system consists of three main components:

1. VLM Backbone (Eagle2-2B): A pre-trained Vision-Language Model that processes the image and text instruction.
2. Latent Action Queries: A set of $N$ learnable tokens, denoted as $Q \in \mathbb { R } ^ { N \times D }$, where $D$ is the hidden dimension of the VLM.
   • Function: These queries are fed into the VLM after the text tokens. They "read" the VLM's internal state to extract intentions for movement.
   • Output: The VLM produces hidden states corresponding to these queries, denoted as $C \in \mathbb { R } ^ { N \times D }$. These represent the "plan" or "intent" in a latent space.

4.2.2. Mixture of Adaptation Experts (MoE)

To prevent the manipulation training from overwriting the VLM's language abilities, the authors implement an MoE mechanism using LoRA (Low-Rank Adaptation) adapters.

• Step 1: Experts Definition. The model uses multiple LoRA modules as "experts." Some specialize in language tasks, others in action intent tasks.

• Step 2: Gating Mechanism. A "scale head" (a small neural network classifier) predicts gating coefficients $\lambda_i$ based on the input context. This decides how much influence each expert should have.

• Step 3: Adaptive Blending Formula. The hidden state $\boldsymbol{h}$ is computed by combining the original weights $W_0$ and the weighted outputs of $K$ LoRA experts. The formula is:

  $$\boldsymbol { h } = W _ { 0 } \boldsymbol { x } + { \sum _ { i = 0 } ^ { K } } \boldsymbol { \bar { B } _ { i } } A _ { i } \boldsymbol { x } \cdot \boldsymbol { \alpha _ { i } } \cdot \lambda _ { i }$$

  • Symbol Explanation:

    • $\boldsymbol{h}$: The final output hidden state of the layer.
    • $W_0$: The frozen original weight matrix of the VLM backbone.
    • $\boldsymbol{x}$: The input vector to the layer.
    • $K$: The total number of experts.
    • $A_i \in \mathbb { R } ^ { r \times d }$ and $\boldsymbol { \bar { B } _ { i } } \in \mathbb { R } ^ { d \times r }$: The low-rank matrices for the $i$-th LoRA expert ($r$ is the rank, $d$ is the dimension).
    • $\boldsymbol { \alpha _ { i } }$: The LoRA scaling factor.
    • $\lambda_i$: The gating coefficient (weight) predicted by the scale head for expert $i$, determining its contribution.

  • Significance: This allows the model to essentially "switch gears." When processing text, $\lambda$ favors language experts; when processing action queries, $\lambda$ favors action experts.

4.2.3. Action Expert: Flow Matching Model

The "Action Expert" is a separate Transformer module that takes the latent plan $C$ from the VLM and generates the actual physical actions (e.g., joint angles or gripper position).

• Input Fusion: It combines:

  1. Image Features: From a DINOv2 encoder (enhanced with FiLM layers for conditioning).
  2. Latent Actions ($C$): The intent vectors from the VLM.
  3. Noisy Action Embeddings: For the generative process.
  4. Proprioception: Current robot state.

• Generation Mechanism (Flow Matching): Instead of predicting discrete tokens (like RT-2), it uses Flow Matching, a method to generate continuous trajectories. The training objective minimizes the difference between the predicted velocity field and the direction pointing towards the target action. The loss function is:

  $$\mathcal { L } _ { F M } = \mathbb { E } \left[ \| V _ { \theta } ( { \bf A } ^ { \tau } , q _ { t } ) - ( \epsilon - { \bf A } ) \| ^ { 2 } \right]$$

  • Symbol Explanation:
    • $\mathcal { L } _ { F M }$: The Flow Matching loss.
    • $V _ { \theta }$: The neural network (Action Expert) parameterized by $\theta$.
    • $\tau \in [0, 1)$: The time step in the flow generation process (from noise to data).
    • ${\bf A}$: The ground truth action chunk (sequence of actions).
    • $\epsilon$: Sampled Gaussian noise, $\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$.
    • ${\bf A}^\tau$: The interpolated action at time $\tau$, defined as ${\bf A}^\tau = \tau {\bf A} + (1-\tau)\epsilon$.
    • $q_t$: The conditioning context (image features, latent actions from VLM).
    • $(\epsilon - {\bf A})$: The target vector field direction (derivative of the interpolation path with respect to $\tau$). Note: The paper's notation $(\epsilon - {\bf A})$ implies the path goes from Action to Noise or vice versa depending on definition; standard flow matching paths are often $x_t = tx_1 + (1-t)x_0$. If $A^\tau$ is defined as above, the derivative w.r.t $\tau$ is $A-\epsilon$. The paper uses $(\epsilon - A)$, which is simply the negative direction. The core idea is training the network $V_\theta$ to predict the vector that transforms the distribution.

• Inference: During deployment, actions are generated by solving the Ordinary Differential Equation (ODE) using Euler integration:

  $$\mathbf { A } ^ { \tau + 1 / N } = \mathbf { A } ^ { \tau } + \frac { 1 } { N } V _ { \theta } ( \mathbf { A } ^ { \tau } , q _ { t } )$$

  • Starts from noise $\mathbf{A}^0 \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and takes $N$ steps (e.g., $N=10$) to reach the final action.

4.3. Two-Stage Training Recipe

The training is split to ensure stability and performance.

Stage 1: Action Pre-training

• Goal: Train the Action Expert to understand "Language Motion" (text descriptions of movement) and map them to physical actions.
• Data: Heterogeneous manipulation data.
• Loss: $\mathcal { L } = \mathcal { L } _ { L M } + \mathcal { L } _ { F M }$.
  • $\mathcal { L } _ { L M }$: Cross-entropy loss on language output (predicting motion descriptions).
  • $\mathcal { L } _ { F M }$: Flow matching loss for actions.
• Trainable Parameters: Only the Action Expert, the Action Queries $Q$, and a specific "Action LoRA" adapter on the VLM. The main VLM is frozen.

Stage 2: Vision-Language-Action Instruction Tuning (VLA-IT)

• Goal: Re-activate the VLM's reasoning and align it with the pre-trained Action Expert.
• Method:
  1. Freeze: The Action Expert is now frozen.
  2. Add: A new "Language LoRA" adapter and the MoE Scale Head.
  3. Train: Only the MoE module (adapters + gating) is trained.
• Data: A mix of:
  • VLA-IT Dataset: (Curated 650K samples with complex instructions).
  • Multimodal Datasets: (Standard VQA data to preserve general knowledge).
  • Manipulation Data: (To maintain control skills).

5. Experimental Setup

5.1. Datasets

The authors use a combination of standard and self-curated datasets.

1. Manipulation Data:

   • Google Fractal & Bridge Data V2: Large-scale robotic manipulation datasets containing trajectories of robot arms performing tasks (e.g., opening drawers, picking objects).
   • Language Motion: The authors add "language motion" annotations—textual descriptions of the end-effector's movement (e.g., "move right and open gripper")—to help bridge text and action.

2. VLA-IT Dataset (Novel Contribution):

   • Scale: 650,000 samples.

   • Creation: Annotated using GPT-4o. It takes 3 frames from an episode and generates 4 types of data:

     1. Scenario Captioning: Describing the scene.
     2. Question Answering: Commonsense questions about the scene.
     3. Command Rewriting: Paraphrasing simple commands into complex ones (e.g., "Open the drawer" $\to$ "I need to access the storage, please open it").
     4. Context Creation: Inferring user intent (e.g., "I'm thirsty" $\to$ robot should fetch a drink).

   • Figure 3 in the paper (see image below) shows examples of these annotations:

     该图像是示意图，展示了视语言行动指导调优的数据示例。左侧为原始数据集示例，展示厨房场景及任务指令；中间部分为对场景理解的相关问答；右侧则阐述指令理解和规划的过程，包括命令重写和上下文创建。

3. Multimodal Benchmarks: Standard VLM evaluation sets like MMMU, MME, TextVQA.

5.2. Evaluation Metrics

1. Success Rate (SR):

   • Definition: The percentage of evaluation episodes where the robot successfully completes the assigned task.
   • Formula: $\text{SR} = \frac{\text{Number of Successes}}{\text{Total Attempts}} \times 100\%$

2. Multimodal Metrics (e.g., MMMU, BLEU-4, CIDEr):

   • MMMU / MME: Composite scores evaluating reasoning across disciplines (Math, Physics, etc.) on images.
   • BLEU-4 / CIDEr: Standard NLP metrics measuring the overlap between generated text and ground truth text (used for evaluating the instruction response capability).

5.3. Baselines

• OpenVLA (7B): A leading open-source VLA model.
• SpatialVLA (3B): A VLA focused on spatial representations.
• RT-1-X / RT-2-X: Google's transformer-based robot policies.
• Magma (8B): A model co-trained on multimodal and manipulation data (a key comparison for the "co-training" strategy).
• GPT-4o + Expert: A pipeline baseline where GPT-4o processes the instruction and calls a low-level policy.

6. Results & Analysis

6.1. Core Results Analysis

6.1.1. Multimodal Understanding

The experiments show that InstructVLA (Generalist) preserves and even enhances the VLM's capabilities compared to baselines that suffer from catastrophic forgetting.

The following are the results from Table 1 of the original paper. Note that InstructVLA (Generalist) achieves scores comparable to or better than pure VLMs (like Eagle2), while OpenVLA collapses to 0.0 on many metrics.

6.1.2. Robotic Manipulation Performance

In the robotic tasks, InstructVLA demonstrates superior performance both in standard settings (SimplerEnv) and the new, complex instruction settings (SimplerEnv-Instruct).

The following are the results from Table 2 of the original paper:

Analysis:

• Expert vs. Generalist: The "Expert" model (Stage 1 only) performs well on standard tasks but fails on the complex "Situated Reasoning" tasks (SR: 10.4%). The "Generalist" model (Stage 2) drastically improves reasoning (SR: 48.8%), proving the value of the VLA-IT stage.
• Comparison: On the challenging SimplerEnv-Instruct, InstructVLA (Generalist) scores 46.0%, compared to 14.2% for the original OpenVLA and 23.9% for fine-tuned OpenVLA. It even beats the "System 2" approach (OpenVLA + GPT-4o rewriting) which scores 35.6%.

6.2. Ablation Studies

The authors analyze key components in Figure 6 (summarized here):

1. Language Motion: Adding language motion data during pre-training improves success rate by 10.5%. It helps ground visual cues to action primitives.

2. Latent Action Queries: Using 64 tokens is optimal. Too few limit diversity; too many reduce efficiency.

3. Action Expert Design: Removing the DINOv2 vision encoder from the expert causes a 50% drop. FiLM conditioning adds another 15.3% gain.

4. "Think" Strategy: Enabling the model to generate text reasoning ("Thinking") before acting improves performance by 36.1% on reasoning tasks compared to acting directly.

   The following figure (Figure 10 from the original paper) visualizes the impact of "Thinking" (reasoning enabled) on various tasks:

   该图像是图表，展示了30个情境推理任务在启用与未启用推理情况下的性能表现。图中任务分为三个类别：子任务、常识推理和工具使用常识。启用推理时，模型的整体性能普遍得到提升。

7. Conclusion & Reflections

7.1. Conclusion Summary

InstructVLA successfully bridges the gap between high-level multimodal reasoning and low-level robotic control. By employing a Mixture-of-Experts architecture and a Two-Stage Training pipeline (VLA-IT), it achieves state-of-the-art results. It mitigates catastrophic forgetting, allowing the robot to function as a generalist that can understand complex, nuanced human instructions while executing precise physical actions.

7.2. Limitations & Future Work

• Sensory Modalities: The current model relies only on a single image and text. It lacks depth perception or tactile feedback, which caused some failure cases (e.g., missing a grasp due to depth ambiguity).
• Real-to-Sim Gap: The authors noted failures due to visual discrepancies between simulation and reality (e.g., lack of reflections).
• Future Work:
  • Integrating depth and tactile sensors.
  • Using large-scale synthetic data (Digital Twins) to reduce reliance on real-world data collection.
  • Expanding to more diverse environments.

7.3. Personal Insights & Critique

• Innovation: The MoE approach to separate "Thinking" and "Acting" pathways is a highly logical solution to the catastrophic forgetting problem. It mirrors the human brain's separation of motor cortex (action) and prefrontal cortex (planning).
• Benchmark Value: The creation of SimplerEnv-Instruct is a significant contribution. Most robot benchmarks use simple templates ("pick A"). Testing on "situated reasoning" (e.g., "I need to clean, get a tool") pushes the field towards true utility.
• Potential Issue: The method relies heavily on GPT-4o for data generation. As noted in the paper's own analysis, GPT-4o can hallucinate or misunderstand physical scenes (e.g., confusing temporal order). The quality of the VLA-IT dataset is thus capped by the capability of the teacher VLM, which is a potential bottleneck for future scaling.
• Applicability: The framework is general enough to be applied to other domains requiring dual capability (reasoning + continuous output), such as autonomous driving (reasoning about traffic rules + steering control).