1. Bibliographic Information

1.1. Title

SpatialActor: Exploring Disentangled Spatial Representations for Robust Robotic Manipulation

The title clearly states the paper's focus: proposing a new model named SpatialActor for robotic manipulation. It highlights the core technical approach, which is creating "disentangled spatial representations," and the primary goal, which is to achieve "robustness."

1.2. Authors

Hao Shi¹, Bin Xie², Yingfei Liu², Yang Vue¹, Tiancai Wang², Haoqiang Fan², Xiangyu Zhang³, Gao Huang⁴

The authors are affiliated with several prominent institutions:

• ¹Tsinghua University (Department of Automation, BNRist): A top university in China, well-known for its engineering and computer science programs.

• ²Dexmal, ⁴StepFun: Companies that likely focus on robotics and AI applications.

• ³MEGVII Technology: A well-known Chinese AI company specializing in computer vision (Face++).

  The collaboration between a leading academic institution and several industrial research labs suggests that the work is grounded in both theoretical rigor and a strong drive for practical, real-world application.

1.3. Journal/Conference

The paper is available on arXiv, a preprint server. The "Published at (UTC)" date of "2025-11-12T18:59:08.000Z" indicates a placeholder for a future publication, suggesting it has been submitted to a top-tier conference or journal for review. Given the topic and quality, likely venues would be major robotics conferences like CoRL (Conference on Robot Learning), ICRA (IEEE International Conference on Robotics and Automation), or top AI/ML conferences like NeurIPS or CVPR.

1.4. Publication Year

The arXiv submission date is not explicitly mentioned, but the placeholder publication date is for 2025. The citations include papers from up to 2025, suggesting this is a very recent work.

1.5. Abstract

The abstract outlines the key problems in robotic manipulation. It criticizes existing point-based methods for losing semantic information and image-based methods for their entangled representations, which are sensitive to real-world depth sensor noise. The paper proposes SpatialActor, a framework that addresses these issues by disentangling semantics and geometry. Its core components are:

1. A Semantic-guided Geometric Module (SGM) that fuses geometric information from noisy depth maps with more robust geometric priors derived from RGB images via an expert model.

2. A Spatial Transformer (SPT) that uses low-level spatial cues for accurate 2D-to-3D mapping and feature interaction.

   The abstract highlights impressive results: state-of-the-art performance on the RLBench benchmark (87.4% success), significant robustness improvements (13.9% to 19.4%) under noisy conditions, and strong few-shot generalization.

1.6. Original Source Link

• Original Source Link: https://arxiv.org/abs/2511.09555

• PDF Link: https://arxiv.org/pdf/2511.09555v1.pdf

  The paper is currently available as a preprint on arXiv. This means it has not yet completed the peer-review process for an official publication venue.

2. Executive Summary

2.1. Background & Motivation

The core problem this paper addresses is the lack of robustness in robotic manipulation systems, especially when dealing with imperfect sensor data from the real world. For a robot to interact precisely with its environment (e.g., pick up an object, insert a peg), it needs a deep understanding of both what objects are (semantics) and where they are in 3D space (geometry).

Existing approaches have significant limitations:

• Point-based methods: These methods use 3D point clouds directly. While they explicitly represent geometry, the process of creating a point cloud from sensor data can be sparse, leading to a loss of fine-grained object details (semantics). They are also expensive to annotate, limiting the scale of pre-training.

• Image-based methods: These methods take RGB images and depth maps (RGB-D) as input. They often fuse these two sources of information early on into a single, entangled representation. The problem is that real-world depth sensors are notoriously noisy (due to reflective surfaces, poor lighting, etc.). This noise corrupts the geometric information, and because geometry and semantics are entangled, the noise also disrupts the model's semantic understanding, leading to a sharp drop in performance. As the paper notes, even minor noise causes a significant performance drop in a state-of-the-art model like RVT-2.

  The paper's innovative entry point is to disentangle semantics and geometry. Instead of mixing them in a shared feature space, SpatialActor processes them separately. This separation is designed to prevent noise from the depth channel from corrupting the clean semantic information extracted from the RGB channel. Furthermore, it proposes a novel way to construct a robust geometric representation by combining the best of both worlds: the fine-grained but noisy details from the raw depth map and a coarse but robust geometric "prior" generated from the clean RGB image using a powerful, pre-trained depth estimation model.

2.2. Main Contributions / Findings

The paper makes several key contributions:

1. A Disentangled Framework (SpatialActor): It proposes a novel architecture that explicitly decouples semantic and geometric representations to improve robustness against sensor noise. This is a fundamental departure from prior works that jointly model them.
2. Semantic-guided Geometric Module (SGM): This module intelligently fuses two complementary sources of geometric information:
   • Fine-grained but noisy geometry from the raw depth sensor.
   • Coarse but robust geometry estimated from the high-quality RGB image using a pre-trained "expert" depth model. A gating mechanism adaptively combines them, preserving details while filtering noise.
3. Spatial Transformer (SPT): This component enhances spatial reasoning by incorporating low-level spatial cues directly into the transformer architecture. It uses positional encodings derived from 3D coordinates to help the model understand the precise spatial relationships between different parts of the scene, which is critical for fine-grained manipulation.
4. State-of-the-Art Performance and Robustness: Through extensive experiments, the paper demonstrates that SpatialActor not only achieves top performance on standard benchmarks (RLBench) but is also significantly more robust to depth noise, spatial perturbations, and generalizes better to new tasks with limited data (few-shot learning). This validates the effectiveness of the disentangled design.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, one must be familiar with the following concepts:

• Robotic Manipulation: This field of robotics focuses on enabling robots to physically interact with and manipulate objects in their environment. This includes tasks like grasping, pushing, placing, and assembling objects.
• RGB-D Data: This refers to data captured by sensors that provide both a standard color image (RGB) and a per-pixel depth map ($D$), which indicates the distance of each pixel from the sensor. RGB-D cameras like the Intel RealSense are common in robotics.
• Point Cloud: A set of data points in a 3D coordinate system. Point clouds are often generated from RGB-D data or LiDAR sensors and are a direct way to represent the 3D geometry of a scene.
• Vision-Language Models (VLMs): These are models, like CLIP (Contrastive Language-Image Pre-training), that are pre-trained on vast amounts of image-text pairs from the internet. They learn a shared embedding space where a text description (e.g., "a red apple") and a corresponding image are mapped to nearby points. This allows the model to understand visual concepts described in natural language, which is useful for instruction-following robots.
• Transformers: A neural network architecture originally designed for natural language processing that relies on a mechanism called self-attention. It allows the model to weigh the importance of different parts of the input sequence when processing a specific part. In computer vision (Vision Transformer or ViT), an image is broken into patches, which are treated as a sequence of tokens. The transformer can then learn relationships between different parts of the image.
• Self-Attention: The core mechanism of a transformer. For a given token, self-attention calculates a score against every other token in the sequence. These scores are then used to create a weighted sum of all tokens, producing a new representation for the original token that is context-aware. The formula is: $ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $ where $Q$ (Query), $K$ (Key), and $V$ (Value) are linear projections of the input tokens, and $d_k$ is the dimension of the key vectors.
• Depth Estimation Models: These are deep learning models trained to predict a depth map from a single RGB image. Modern models like Depth Anything are trained on massive and diverse datasets, making them very robust at inferring geometric structure from 2D images, even in challenging conditions.
• Rotary Positional Encoding (RoPE): A technique for encoding the positional information of tokens in a transformer. Unlike absolute positional embeddings that are added to the tokens, RoPE rotates the token embeddings based on their position. This method has been shown to be very effective at capturing relative positional relationships.

3.2. Previous Works

The paper categorizes prior work into several groups, which it aims to improve upon:

• 2D-based Approaches: Models like R3M and Diffusion Policy learn manipulation policies directly from 2D images. While they benefit from powerful pre-trained 2D vision models, they lack an explicit understanding of 3D space, which makes them struggle with tasks requiring precise geometric reasoning or handling occlusions.
• Point Cloud-based Methods: Models like PolarNet and AnyGrasp use 3D point clouds as their primary input. They have an explicit geometric representation, but suffer from sparsity (missing details between points) and the high cost of acquiring and annotating large-scale 3D data.
• Voxel-based Representations: Models like PerAct discretize the 3D space into a grid of voxels. This provides a structured representation for reasoning, but it is computationally expensive, especially at high resolutions.
• Multi-view RGB-D Approaches: This is the most direct competitor category. Models like RVT (Robotic View Transformer) and its successor RVT-2 take RGB-D data from multiple camera views and feed them into a transformer architecture. They typically use a shared feature space where semantic and geometric information are entangled. While powerful, their performance is fragile and degrades significantly when the depth data is noisy, which is the central problem SpatialActor is designed to solve.

3.3. Technological Evolution

The field of robotic manipulation has evolved from relying on simple proprioceptive (joint-level) states to leveraging rich visual information.

1. Early Vision: Simple computer vision techniques were used to detect objects.

2. Deep Learning Rise: CNNs became dominant for processing 2D images to guide robots.

3. 3D Integration: The availability of cheap RGB-D sensors led to methods using point clouds and voxels to explicitly model the 3D world.

4. Transformer Era: Inspired by successes in NLP and vision, transformers (RVT, PerAct) were adapted for robotics. They can process information from multiple camera views and fuse it with language instructions, enabling more complex, multi-stage tasks.

5. Foundation Models: Recent work leverages large-scale pre-trained models (VLMs like CLIP, depth estimators like Depth Anything) as powerful priors for semantics and geometry.

   SpatialActor sits at this latest stage. It doesn't just use foundation models but proposes a more principled way to combine them by disentangling their outputs to build a representation that is both accurate and robust.

3.4. Differentiation Analysis

The core innovation of SpatialActor compared to its main competitors (like RVT-2) is the disentanglement of semantics and geometry.

• RVT-2 (Entangled): Feeds RGB and Depth data into a shared encoder. The features for semantics and geometry are mixed together. Pro: Simple and can learn joint patterns. Con: Noise in the depth channel can easily corrupt the entire representation, harming semantic understanding.
• SpatialActor (Disentangled): Processes RGB and Depth in separate streams.

  • The semantic stream uses the clean RGB image to get high-quality semantic features via a VLM.

  • The geometric stream is more complex: it doesn't just rely on the noisy raw depth. It creates a robust geometric representation by fusing the raw depth (for detail) with a clean geometric estimate generated from the RGB image (for robustness).

    This disentangled design acts as a firewall, preventing depth noise from contaminating the semantic features. The SGM module then carefully reconstructs a high-quality geometric representation, making the final model far more resilient to the challenges of real-world sensor data.

4. Methodology

4.1. Principles

The core principle of SpatialActor is "divide and conquer". Instead of treating the multi-modal input (RGB, depth, language, robot state) as a monolithic block of information, the framework separates it into distinct channels—semantics and geometry—to process them according to their strengths and weaknesses. The intuition is that RGB images provide high-quality semantic information but ambiguous geometry, while depth maps provide direct geometry but are often noisy. By handling them separately and fusing them intelligently, the system can be more robust and precise than methods that entangle them from the start.

4.2. Core Methodology In-depth (Layer by Layer)

The SpatialActor framework can be broken down into the following stages, as illustrated in Figure 2 of the paper.

The following figure (Figure 2 from the original paper) shows the overall framework of SpatialActor.

该图像是一个示意图，展示了SpatialActor框架中的多个模块。左侧部分包含处理噪声深度信息的几何编码器和处理RGB图像的语义编码器，通过多尺度门控融合（SGM）来提取和结合不同的信息。中间部分展示了空间转换模块（SPT），用于实现视图级和场景级的交互。右侧为执行模块，显示了机械臂的动作。这一框架旨在增强机器人在复杂环境中的操作能力和稳定性。

4.2.1. Inputs

The system takes a set of inputs for each decision step: $ X = { I ^ { v } , D ^ { v } } _ { v = 1 } ^ { V } , P , L $

• $I^v \in \mathbb{R}^{H \times W \times 3}$: The RGB image from camera view $v$.
• $D^v \in \mathbb{R}^{H \times W}$: The corresponding depth map from view $v$.
• $V$: The total number of camera views.
• $P \in \mathbb{R}^{d_p}$: The robot's proprioceptive state (e.g., joint angles, gripper position).
• $L$: The natural language instruction for the task (e.g., "put the red block on the blue block").

4.2.2. Disentangled Feature Extraction

The RGB and depth inputs are processed separately to extract semantic and geometric features.

1. Semantic Features: The RGB image $I^v$ and language instruction $L$ are passed through a pre-trained Vision-Language Model (VLM) like CLIP. This produces language-aligned semantic features $F_{\mathrm{sem}}^v$ for the image and text features $F_{\mathrm{text}}$.
2. Geometric Features (Raw): The raw, potentially noisy depth map $D^v$ is passed through a standard convolutional encoder (e.g., a ResNet) to produce fine-grained but noisy geometric features $F_{\mathrm{geo}}^v$.

4.2.3. Semantic-guided Geometric Module (SGM)

This is the first key innovation. The goal is to create a robust geometric representation that is less sensitive to noise. The SGM does this by fusing the noisy features from the raw depth map with a clean "expert prior" generated from the RGB image.

The following figure (Figure 3 from the original paper) details the SGM and SPT modules.

该图像是示意图，展示了语义引导几何模块（SGM）和空间变换器（SPT）的结构。SGM通过多尺度门融合方法对噪声深度信息进行处理，而SPT则利用空间位置编码模块进行视图和场景级别的交互。

1. Expert Geometric Prior Generation: The high-quality RGB image $I^v$ is fed into a frozen, large-scale pre-trained depth estimation model (like Depth Anything v2). This model has learned to infer geometric structure from semantics on a massive dataset, so it produces a robust but potentially coarse geometric representation, $\hat{F}_{\mathrm{geo}}^v$. $ \hat{F}{\mathrm{geo}}^v = \mathcal{E}{\mathrm{expert}}(I^v) \in \mathbb{R}^{H \times W \times C} $

   • $\mathcal{E}_{\mathrm{expert}}$: The frozen depth estimation expert model.
   • $\hat{F}_{\mathrm{geo}}^v$: The robust, coarse geometric features.

2. Raw Geometric Feature Extraction: Simultaneously, the raw depth map $D^v$ is processed by a trainable depth encoder to get fine-grained features. $ F_{\mathrm{geo}}^v = \mathcal{E}_{\mathrm{raw}}(D^v) \in \mathbb{R}^{H \times W \times C} $

   • $\mathcal{E}_{\mathrm{raw}}$: The trainable depth encoder (e.g., ResNet-50).
   • $F_{\mathrm{geo}}^v$: The fine-grained, noisy geometric features.

3. Adaptive Gated Fusion: The SGM then adaptively fuses these two complementary geometric representations. A gating mechanism decides pixel-wise whether to trust the fine-grained details from the raw depth or the robust prior from the expert. First, a gate $G^v$ is computed by concatenating both feature maps and passing them through a small Multi-Layer Perceptron (MLP). $ G^v = \sigma\left(\mathrm{MLP}\left(\mathrm{Concat}(\hat{F}{\mathrm{geo}}^v, F{\mathrm{geo}}^v)\right)\right) $

   • $\mathrm{Concat}(\cdot, \cdot)$: Concatenates the two feature tensors along the channel dimension.

   • $\mathrm{MLP}(\cdot)$: A multi-layer perceptron that learns to produce the gate values.

   • $\sigma(\cdot)$: The sigmoid activation function, which outputs values between 0 and 1 for the gate $G^v$.

     The final fused geometric feature map $F_{\mathrm{fuse-geo}}^v$ is a weighted combination controlled by the gate $G^v$. $ F_{\mathrm{fuse-geo}}^v = G^v \odot F_{\mathrm{geo}}^v + (1 - G^v) \odot \hat{F}_{\mathrm{geo}}^v $

   • $\odot$: Element-wise multiplication.

   • When a value in the gate $G^v$ is close to 1, the fusion favors the fine-grained feature from the raw depth $F_{\mathrm{geo}}^v$. When it's close to 0, it favors the robust expert prior $\hat{F}_{\mathrm{geo}}^v$. This allows the model to dynamically suppress noise.

     Finally, the fused geometric features $F_{\mathrm{fuse-geo}}^v$ are concatenated with the semantic features $F_{\mathrm{sem}}^v$ to form the combined spatial features $H^v$ for that view.

4.2.4. Spatial Transformer (SPT)

This is the second key innovation. Once we have the per-view features $H^v$, the SPT's job is to reason about their spatial relationships and fuse information across different views to form a coherent scene understanding.

1. Proprioception Fusion: The robot's own state $P$ is projected by an MLP and added to the visual features to make them aware of the robot's current configuration. $ \widetilde{H}^v = H^v + \mathsf{MLP}(P) $

2. Low-level Spatial Cue Injection (2D-to-3D Projection): To enable precise spatial reasoning, the model needs to know the 3D location corresponding to each feature token. For a pixel (x', y') in the image of view $v$, its 3D coordinate (x, y, z) in the robot's base frame is calculated using the camera's intrinsic and extrinsic parameters. $ [ \boldsymbol{x} , \boldsymbol{y} , \boldsymbol{z} , 1 ] ^ { \top } = E ^ { v } \left( d \cdot ( K ^ { v } ) ^ { - 1 } [ \boldsymbol { x } ^ { \prime } , \boldsymbol { y } ^ { \prime } , 1 ] ^ { \top } \parallel 1 \right) $

   • $K^v$: The camera intrinsic matrix for view $v$, which maps 3D camera coordinates to 2D image coordinates.
   • $E^v$: The camera extrinsic matrix for view $v$, which transforms coordinates from the camera frame to the robot's base frame.
   • d = D^v(x', y'): The depth value at pixel (x', y').
   • $\parallel$: Vector concatenation.

3. Rotary Positional Encoding (RoPE): The calculated 3D coordinates (x, y, z) are used to generate positional encodings. The paper uses RoPE.

   • First, a set of frequencies is defined: $ \omega _ { k } = \lambda ^ { - 2 k / d } , \quad k = 0 , 1 , \ldots , \frac { d } { 2 } - 1 , \quad d = D / 3 $ where $\lambda = 10000$ and $D$ is the feature dimension. This creates a spectrum of frequencies from low to high.
   • Sinusoidal embeddings are then computed for each coordinate axis $u \in \{x, y, z\}$: $ \cos_{\mathrm{pos}} = [\cos(\omega_k u)]{u \in {x, y, z}, k = 0, \ldots, d/2 - 1} $ $ \sin{\mathrm{pos}} = [\sin(\omega_k u)]_{u \in {x, y, z}, k = 0, \ldots, d/2 - 1} $
   • These embeddings are applied to the feature vectors $\widetilde{H}^v$ using rotation, which modifies the features based on their 3D position. This allows the subsequent attention layers to be inherently aware of relative spatial positions. $ T^v = \tilde{H}^v \odot \cos_{\mathrm{pos}} + \mathrm{rot}(\tilde{H}^v) \odot \sin_{\mathrm{pos}} $ where $\mathrm{rot}(\cdot)$ rotates pairs of feature dimensions, effectively mixing them based on the sine component of the positional encoding.

4. Hierarchical Attention: The SPT uses a two-level attention mechanism:

   • View-level Interaction: Self-attention is applied to the tokens $T^v$ within each view separately. This allows the model to refine features by consolidating context from within a single viewpoint.
   • Scene-level Interaction: The refined tokens from all views are concatenated together with the language features $F_{\mathrm{text}}$. Another round of self-attention is applied to this combined set of tokens. This crucial step fuses information across all views and modalities, creating a unified scene representation that understands the relationships between objects seen from different angles and connects them to the language command.

4.2.5. Action Prediction

The final refined tokens are used to predict the robot's action.

1. A lightweight decoder produces a per-view 2D heatmap. The target 3D translation (x, y, z) is determined by finding the pixel with the highest heatmap value (argmax) and lifting its 2D coordinates to 3D using the camera model.
2. An MLP takes the feature vectors around this peak location and regresses the end-effector rotation $\theta = (\theta_x, \theta_y, \theta_z)$ and the binary gripper state $g$ (open/close).
3. The final action is $A = (x, y, z, \theta_x, \theta_y, \theta_z, g)$.
4. The model is trained with a composite loss: cross-entropy loss for the 2D heatmaps, cross-entropy loss for discretized rotation angles, and a binary classification loss for the gripper state.

5. Experimental Setup

5.1. Datasets

• RLBench: A large-scale, challenging benchmark for robotic manipulation simulated in CoppeliaSim. It features a Franka Emika Panda arm. The paper uses 18 different tasks with a total of 249 variations (e.g., different object colors, positions). For each task, 100 expert demonstrations are used for training and 25 unseen episodes for testing. Observations include RGB-D images from four cameras (front, left/right shoulder, wrist).

  • Data Sample Example: An episode would consist of a sequence of observations (four RGB-D images, robot state) and the corresponding expert action (target end-effector pose and gripper state) needed to complete a task like "push the red button, then the green button".

• ColosseumBench: A benchmark designed specifically to evaluate the generalization and robustness of manipulation policies under environmental changes. The paper uses 20 tasks and evaluates performance under perturbations like changing the size of the manipulated object, the size of the container object, and the camera pose.

  These datasets were chosen because they are standard in the field and provide a comprehensive test of multi-task learning, precision, robustness, and generalization.

5.2. Evaluation Metrics

• Success Rate: This is the primary metric for evaluating task completion.

  1. Conceptual Definition: It measures the percentage of trials in which the robot successfully completes the given task according to the task-specific criteria. A higher success rate indicates a more effective and reliable policy.
  2. Mathematical Formula: $ \text{Success Rate} (%) = \frac{\text{Number of Successful Trials}}{\text{Total Number of Trials}} \times 100 $
  3. Symbol Explanation:
     • Number of Successful Trials: The count of test episodes where the policy achieved the task goal.
     • Total Number of Trials: The total number of test episodes evaluated.

• Average Rank: This metric evaluates the overall performance of a model across multiple tasks by ranking it against other models on each task and then averaging the ranks.

  1. Conceptual Definition: It provides a summary of how a model performs relative to its competitors across a suite of tasks. A lower average rank is better, indicating that the model consistently ranks high (e.g., 1st, 2nd) on many tasks.
  2. Mathematical Formula: $ \text{Average Rank} = \frac{1}{N_{\text{tasks}}} \sum_{i=1}^{N_{\text{tasks}}} \text{Rank}_i $
  3. Symbol Explanation:
     • $N_{\text{tasks}}$: The total number of tasks in the benchmark.
     • $\text{Rank}_i$: The rank of the model on task $i$ (e.g., 1 for the best, 2 for the second best, etc.).

5.3. Baselines

The paper compares SpatialActor against a wide range of state-of-the-art models, representing different approaches to robotic manipulation:

• PerAct: A voxel-based method using a Perceiver-style transformer. It is a strong baseline that discretizes 3D space.

• RVT and RVT-2: The most direct competitors. These are image-based methods using transformers to process multi-view RGB-D data. They represent the state-of-the-art in entangled representation learning, making them the perfect benchmark to demonstrate the benefits of SpatialActor's disentangled approach.

• Act3D, 3D Diffuser Actor: Other recent 3D representation learning methods.

• C2F-ARM-BC, HiveFormer, PolarNet: Earlier methods representing different architectural choices (coarse-to-fine, point cloud-based, etc.).

• R3M, MVP, VoxPoser: Baselines used in the ColosseumBench evaluation.

  This comprehensive set of baselines ensures that the performance gains of SpatialActor are measured against the best and most representative methods currently available.

6. Results & Analysis

6.1. Core Results Analysis

The main results on the RLBench benchmark are presented in Table 1.

The following are the results from Table 1 of the original paper:

• Overall Performance: SpatialActor achieves an average success rate of 87.4%, surpassing the previous state-of-the-art RVT-2 (81.4%) and 3D Diffuser Actor (81.3%) by a significant margin of ~6.0%. It also has the best (lowest) average rank of 2.3.
• High-Precision Tasks: The most striking results are on tasks that demand high spatial precision.
  • On Insert Peg, SpatialActor achieves an incredible 93.3% success rate, whereas RVT-2 only manages 40.0%. This is a +53.3% improvement, strongly suggesting that the robust and precise spatial representation from SGM and SPT is highly effective for tasks requiring fine alignment.
  • On Sort Shape, SpatialActor achieves 73.3% compared to RVT-2's 35.0%, another massive gain (+38.3%). This task requires both shape recognition (semantics) and precise placement (geometry), and SpatialActor excels.

6.2. Data Presentation (Tables)

Robustness Under Noisy Conditions (Table 2)

This experiment is central to the paper's thesis. By injecting Gaussian noise into the point clouds derived from depth maps, the authors simulate real-world sensor imperfections.

The following are the results from Table 2 of the original paper:

• RVT-2's performance degrades rapidly with noise: 81.4% (no noise) -> 72.5% (light) -> 68.4% (middle) -> 57.0% (heavy).
• SpatialActor demonstrates remarkable resilience: 87.4% (no noise) -> 86.4% (light) -> 85.3% (middle) -> 76.4% (heavy).
• The performance gap widens as noise increases. SpatialActor outperforms RVT-2 by +13.9% (light), +16.9% (middle), and +19.4% (heavy).
• On the Insert Peg task under heavy noise, RVT-2's performance drops to 0%, while SpatialActor maintains a respectable 61.3%. This is definitive proof that the disentangled representation and the SGM module are extremely effective at mitigating the effects of depth noise.

Few-Shot Generalization (Table 3)

This test evaluates how well a model can adapt to 19 new tasks after being trained on a different set of tasks, using only 10 demonstration videos for each new task.

The following are the results from Table 3 of the original paper:

SpatialActor achieves an average success rate of 79.2%, while RVT-2 only gets 46.9%. This is a massive +32.3% improvement. This suggests that the robust and well-structured representations learned by SpatialActor provide a much better foundation for transferring knowledge to new, unseen tasks, requiring less data to adapt.

6.3. Ablation Studies / Parameter Analysis

The ablation study in Table 5 is crucial for validating the contribution of each component of SpatialActor. The baseline is RVT-2, which is an entangled model.

The following are the results from Table 5 of the original paper:

1. Baseline (RVT-2): 81.4% success (no noise), 57.0% (heavy noise).

2. + Decouple: Simply disentangling the semantic and geometric streams (without SGM or SPT) already provides a huge boost. Performance rises to 85.1% (+3.7%) with no noise and 68.7% (+11.7%) with heavy noise. This confirms that preventing noise from the depth channel from corrupting semantic features is highly effective.

3. + SGM: Adding the Semantic-guided Geometric Module further improves performance to 86.4% (+1.3%) and 73.9% (+5.2%). The improvement is much larger under heavy noise, proving that the SGM's ability to fuse expert priors with raw depth is key to its noise robustness.

4. + SPT (Full SpatialActor): Finally, adding the Spatial Transformer brings performance to its peak at 87.4% (+1.0%) and 76.4% (+2.5%). The SPT contributes by enabling more precise spatial reasoning through its explicit modeling of low-level 3D positional cues.

   This step-by-step analysis clearly demonstrates that each proposed component contributes meaningfully to the model's final performance and robustness.

Real-World Evaluation

The real-world experiments (Table 6) confirm that the benefits observed in simulation translate to a physical robot. SpatialActor achieves an overall success rate of 63% across 8 tasks (15 variations), compared to 43% for RVT-2. This +20% absolute improvement in the real world is a strong validation of the method's practical utility. The qualitative results in Figures 5, 6, and 7 further show SpatialActor's superior stability and precision in grasping and placement compared to RVT-2.

The following figure (Figure 5 from the original paper) shows the model's generalization performance.

该图像是一个示意图，展示了在不同条件下评估SpatialActor的鲁棒性，包含默认设置、操控目标、接收对象、亮度和背景的变化。右侧柱状图显示了各条件下的性能评分，最高为80。

The following figure (Figure 6 from the original paper) shows a qualitative comparison on a real-world task.

该图像是一个示意图，展示了RVT-2和本研究提出的方法在进行物体抓取任务时的表现。上方是RVT-2的操作示例，其精确度不足，标注为'imprecise'。下方是我们的模型在相同任务中的表现，显示出更好的抓取成功率和精准度。

7. Conclusion & Reflections

7.1. Conclusion Summary

The paper successfully identifies a critical weakness in existing image-based robotic manipulation methods: their vulnerability to real-world sensor noise due to entangled semantic-geometric representations. The proposed SpatialActor framework provides an elegant and effective solution by disentangling these two information streams.

The core contributions—the Semantic-guided Geometric Module (SGM) for building a robust geometric representation and the Spatial Transformer (SPT) for precise spatial reasoning—are shown to be highly effective. SpatialActor not only sets a new state-of-the-art on the RLBench benchmark but, more importantly, demonstrates vastly superior robustness to noise, better generalization to new tasks, and successful deployment on a real robot. The work makes a strong case that disentangled spatial representations are a crucial step towards building more reliable and general-purpose robotic systems.

7.2. Limitations & Future Work

The authors candidly discuss failure cases in the supplementary material (Figure 8), which point to several limitations and directions for future work:

• Instruction Understanding Errors: The model sometimes misunderstands complex or ambiguous language instructions (e.g., opening the wrong drawer). This suggests that the current VLM, while powerful, could be improved. Integrating more advanced Large Language Models (LLMs) could enhance high-level reasoning and planning.
• Long-Horizon Task Failures: The model can fail in tasks requiring a long sequence of steps (e.g., stalling after placing a few cups). This points to a need for better long-term memory or planning capabilities, such as an episodic memory module.
• Semantic Ambiguity: The model can get confused when multiple similar-looking objects are present (e.g., stacking the wrong cup). This is a semantic grounding problem that could be addressed with more sophisticated attention mechanisms.
• Susceptibility to Distractors: In the real world, background clutter can sometimes distract the policy. This highlights the need for more robust attention filtering.

7.3. Personal Insights & Critique

This paper is an excellent piece of research with clear motivation, a well-designed methodology, and comprehensive experiments that strongly support its claims.

Strengths:

• Problem-Driven Innovation: The work is motivated by a real, practical problem (sensor noise) rather than just chasing incremental benchmark scores. The solution (disentanglement) is a direct and principled response to this problem.
• Clever Use of Foundation Models: The way SpatialActor uses the pre-trained depth expert is very clever. Instead of just using it as a black box, it fuses its robust but coarse output with the fine-grained but noisy raw data, getting the best of both worlds. This is a great example of how to creatively leverage existing powerful models.
• Rigorous Evaluation: The experimental design is top-notch. The combination of standard benchmarks, targeted noise/perturbation tests, few-shot generalization experiments, and real-world validation provides a convincing and multi-faceted case for the method's superiority.

Potential Areas for Improvement/Critique:

• Computational Cost: The model uses multiple large networks (a VLM, a depth expert, plus its own encoders and transformer). The computational overhead of this framework might be significant, potentially limiting its application in resource-constrained or real-time critical scenarios. The paper does not discuss inference speed or computational costs.

• Complexity of the System: While effective, the system has many moving parts. This could make it harder to debug, tune, and deploy compared to simpler end-to-end models.

• Scalability to More Complex Scenes: The experiments are conducted in tabletop scenarios. While diverse, it remains to be seen how the approach scales to more cluttered, dynamic, and unstructured environments like a home kitchen or a warehouse.

  Overall, SpatialActor presents a compelling architectural paradigm for robotic manipulation. The principle of disentanglement for robustness is a powerful idea that could inspire many future works in robotics and other fields where multi-modal fusion is critical. The paper is a significant step forward in building robotic systems that can operate reliably outside the pristine conditions of a lab.