In the field of embodied AI robot systems, a critical challenge lies in the scarcity of high-quality, annotated robotic manipulation data. Real-world data collection for embodied AI robots is often costly, time-consuming, and limited in scale, which severely restricts the training of deep models for complex tasks. Conversely, internet-scale video data offers a vast, diverse, and readily available resource rich in physical interaction cues. However, these videos typically lack explicit action and state annotations, making it difficult to extract meaningful representations that can guide embodied AI robot behavior. This discrepancy creates a significant bottleneck: how can we leverage abundant but unlabeled video data to enhance the learning capabilities of embodied AI robots? To address this, I propose a novel two-stage method based on video pre-training that enables efficient behavior learning for embodied AI robots without relying heavily on labeled robotic datasets.
The core idea is to first learn a generalizable latent state representation from unlabeled videos through unsupervised pre-training, and then use this representation within a multimodal Transformer architecture to generate adaptive actions for various embodied AI robot platforms. This approach allows embodied AI robots to benefit from the richness of video data, improving their task performance, generalization to unseen environments, and robustness. In this article, I will detail the methodology, present extensive experimental validation, and discuss the implications for advancing embodied AI robot research.
The first stage of the method focuses on extracting latent states from videos. The framework is built upon a variational autoencoder (VAE) architecture designed to disentangle static (environment) and dynamic (robot interaction) features from video frames without any supervision. Let a video frame at time $t$ be denoted as $\mathbf{x}_t \in \mathbb{R}^{H \times W \times 3}$. The process begins with a pre-trained visual encoder, such as MAE-Large, to extract initial visual features:
$$\mathbf{f}_t = \text{MAE}(\mathbf{x}_t) \in \mathbb{R}^{N \times D}$$
where $N=49$ is the number of image patches and $D=1024$ is the feature dimension. These features are then fed into a Transformer module, termed M-Former, which uses learnable query vectors $\mathbf{Q} \in \mathbb{R}^{K \times D}$ (with $K=57$) to produce compact representations. The output is explicitly split into static features $\mathbf{F}_s \in \mathbb{R}^{49 \times D}$ and dynamic features $\mathbf{F}_d \in \mathbb{R}^{8 \times D}$:
$$\mathbf{F}_s, \mathbf{F}_d = \text{M-Former}(\mathbf{x}_t, \mathbf{Q})$$
The dynamic features are further processed by a Residual Quantized VAE (RQ-VAE) to obtain a discrete, compact latent state $\tilde{\mathbf{F}}_d$. RQ-VAE performs multi-level residual quantization with a codebook size of 16 and depth of 8, refining the representation step by step. The quantization process for level $l$ can be summarized as:
$$\mathbf{r}^{(l)} = \mathbf{F}_d^{(l)} – \mathbf{e}_{k^{(l)}}, \quad k^{(l)} = \arg\min_{k} \|\mathbf{F}_d^{(l)} – \mathbf{e}_k\|^2$$
where $\mathbf{e}_k$ are codebook vectors, and $\mathbf{r}^{(l)}$ is the residual passed to the next level. The final quantized dynamic feature $\tilde{\mathbf{F}}_d$ is a concatenation of the selected codebook indices across all levels. To ensure proper disentanglement, a cross-frame consistency constraint is enforced. Given a source frame $\mathbf{O}_t$ and a target frame $\mathbf{O}_{t+k}$, their static and dynamic features are extracted. Then, reconstruction is performed by swapping static features: the decoder reconstructs images using the source’s dynamic features with the target’s static features, and vice versa. The decoder, a Vision Transformer (ViT), generates the reconstructed frame $\hat{\mathbf{O}}$ as:
$$\hat{\mathbf{O}} = \text{Decoder}(\tilde{\mathbf{F}}_d, \mathbf{F}_s, \mathbf{e}_{\text{rob}})$$
where $\mathbf{e}_{\text{rob}}$ is a learnable robot-type embedding (e.g., for Franka, Google WidowX) that conditions the model on the specific embodied AI robot platform, enhancing cross-platform adaptability. The training loss combines multiple objectives to ensure effective learning:
$$\mathcal{L}_{\text{total}} = \lambda_{\text{commit}}\mathcal{L}_{\text{commit}} + \lambda_{\text{recons}}\mathcal{L}_{\text{recons}} + \lambda_{\text{perceptual}}\mathcal{L}_{\text{perceptual}} + \lambda_{\text{static}}\mathcal{L}_{\text{static}}$$
Here, $\mathcal{L}_{\text{commit}}$ is the quantization loss, $\mathcal{L}_{\text{recons}}$ is the pixel-wise reconstruction loss, $\mathcal{L}_{\text{perceptual}}$ is the LPIPS-based perceptual loss, and $\mathcal{L}_{\text{static}}$ is the static feature matching loss. The weights are set as $\lambda_{\text{commit}}=0.5$, $\lambda_{\text{recons}}=1.0$, $\lambda_{\text{perceptual}}=0.5$, and $\lambda_{\text{static}}=0.5$. This framework enables the extraction of generic latent states that capture the essential dynamics of embodied AI robot interactions, which are then frozen and used in the second stage.
The second stage involves learning behavior policies using a multimodal Transformer. This model integrates three types of inputs: language instructions (encoded by a pre-trained T5 model), visual observations (from the MAE encoder), and the latent dynamic features from the first stage. Additionally, robot-type embeddings are included to tailor actions to specific embodied AI robot morphologies. The input sequence is constructed by concatenating these features along with action query tokens for each time step. The multimodal Transformer employs a block-wise attention mechanism: full attention within the condition sequence (language, vision, latent features, robot type), unidirectional attention from conditions to action queries, and causal masking among action queries to maintain temporal causality. The output action queries are processed by dynamic action heads—separate MLP networks for each robot type—to predict continuous actions. For an embodied AI robot with 6-DoF control, the action head outputs position delta $\Delta \mathbf{x} \in \mathbb{R}^3$, rotation delta $\Delta \boldsymbol{\theta} \in \mathbb{R}^3$, and gripper state $\Delta g \in \mathbb{R}$. The action loss is computed as:
$$\mathcal{L}_{\text{action}} = \mathcal{L}_{\text{SmoothL1}}(\Delta \mathbf{x}) + \mathcal{L}_{\text{SmoothL1}}(\Delta \boldsymbol{\theta}) + \mathcal{L}_{\text{BCE}}(\Delta g)$$
where $\mathcal{L}_{\text{SmoothL1}}$ is used for continuous values and $\mathcal{L}_{\text{BCE}}$ (binary cross-entropy) for the gripper. During inference, a time integration strategy from ACT is used to smooth actions: an action buffer stores recent predictions, and a weighted sum produces the final action, reducing jitter and improving stability for the embodied AI robot.
To validate the method, experiments were conducted in simulation environments. The training setup involved specific datasets and hyperparameters. For the latent state extraction, pre-training used a mix of video data, and for behavior learning, datasets like CALVIN and SIMPLER were employed. Key hyperparameters are summarized in the following tables.
| Parameter | Latent State Extraction | Behavior Policy Learning |
|---|---|---|
| Batch Size | 32 | 16 |
| Epochs | 20 | 20 |
| Learning Rate | 0.00015 | 0.0002 |
| Optimizer | AdamW | AdamW |
| Weight Decay | 0.0001 | 0.0001 |
| Warm-up Epochs | 1 | 1 |
| Component | Parameter | Value |
|---|---|---|
| M-Former | Static Num Queries | 49 |
| Dynamic Num Queries | 8 | |
| Hidden Size | 768 | |
| Number of Heads | 12 | |
| ViT Decoder | Patch Size | 16 |
| Number of Layers | 4 | |
| Hidden Size | 768 | |
| RQ Codebook | Number of Codes | 16 |
| Latent Dimension | 16 | |
| Quantization Depth | 8 |
| Component | Parameter | Value |
|---|---|---|
| GPT Backbone | Number of Layers | 12 |
| Hidden Size | 768 | |
| Number of Heads | 12 | |
| Action Prediction Head | Number of Layers | 2 |
| Hidden Size | 384 |
The evaluation was performed in two primary simulation environments: CALVIN and SIMPLER. For CALVIN, which features a Franka Panda embodied AI robot in desktop tasks, two settings were tested: D-D (train and test in the same environment D) and ABC-D (train in environments A, B, C and test in unseen environment D). The metric is the average task length in a chain of 5 sequential tasks, with a perfect score of 5.0. Baseline methods included MT-R3M (using R3M visual representations), UniPi (a video generation planning method), and Moto (an autoregressive pre-training approach). The performance improvement rate is calculated as: (Our method’s average length – Baseline’s average length) / 5.0 × 100%.
| Model | Task Completion Rate (1 to 5) | Average Length |
|---|---|---|
| MT-R3M | 0.486, 0.180, 0.102, 0.080, 0.022 | 0.870 |
| Moto | 0.600, 0.352, 0.232, 0.120, 0.024 | 1.328 |
| Our Method | 0.880, 0.688, 0.648, 0.420, 0.340 | 2.976 |
| Model | Task Completion Rate (1 to 5) | Average Length |
|---|---|---|
| MT-R3M | 0.529, 0.234, 0.105, 0.043, 0.018 | 0.929 |
| UniPi | 0.560, 0.160, 0.080, 0.080, 0.040 | 0.920 |
| Moto | 0.779, 0.555, 0.380, 0.256, 0.167 | 2.137 |
| Our Method | 0.880, 0.732, 0.368, 0.460, 0.290 | 2.730 |
The results show that our method achieves an average length of 2.976 in D-D and 2.730 in ABC-D, outperforming all baselines. Compared to Moto, the improvement is 32.96% in D-D, demonstrating the efficacy of the latent state extraction for embodied AI robot tasks. The cross-environment generalization in ABC-D also highlights the robustness of the learned representations.
In the SIMPLER environment, which simulates realistic physics and includes variations for robustness testing, the embodied AI robot platform was a Google robot arm. Tasks included grasping a can, moving near an object, and opening/closing a drawer. The model was trained on a mixture of RT-1 and Bridge-V2 datasets, with frame intervals adjusted to 1 Hz. The evaluation metric was the average success rate over 30 trials per task.
| Task | Grasp Can (Horizontal) | Grasp Can (Vertical) | Grasp Can (Upright) | Average Grasp | Move Near Object | Open Drawer | Close Drawer | Average Drawer | Overall Average |
|---|---|---|---|---|---|---|---|---|---|
| Moto | 0.520 | 0.170 | 0.560 | 0.417 | 0.495 | 0.046 | 0.120 | 0.083 | 0.332 |
| Our Method | 0.760 | 0.220 | 0.490 | 0.490 | 0.418 | 0.027 | 0.648 | 0.338 | 0.415 |
Our method achieves an overall average success rate of 0.415, compared to 0.332 for Moto, indicating a 2.8% absolute improvement. This shows that the latent states enable better adaptation for the embodied AI robot. To test robustness, environmental perturbations were introduced: changing desk texture, background, lighting, adding distractors, and varying camera viewpoint.
| Task | Perturbation Type | Moto Success Rate | Our Method Success Rate |
|---|---|---|---|
| Grasp Can | Desk Texture | 0.171 | 0.273 |
| Background Change | 0.360 | 0.260 | |
| Lighting Change | 0.416 | 0.153 | |
| Distractors | 0.298 | 0.344 | |
| Average for Grasp | 0.311 | 0.258 | |
| Move Near Object | Background Change | 0.317 | 0.467 |
| Lighting Change | 0.256 | 0.349 | |
| Distractors | 0.316 | 0.516 | |
| Camera View Change | 0.124 | 0.274 | |
| Average for Move | 0.253 | 0.402 | |
| Overall Average | 0.282 | 0.330 |
Under perturbations, our method maintains competitive performance, with an overall average of 0.330 versus 0.282 for Moto. The drop in grasping under lighting changes suggests a limitation in visual feature extraction under extreme conditions, which could be addressed in future work. Nonetheless, the embodied AI robot demonstrates robustness across most variations.
Ablation studies were conducted to analyze the contribution of each loss component in the latent state extraction stage. The experiment used the CALVIN D-D setting, and results are shown below.
| Model Variant | Task Completion Rate (1 to 5) | Average Length |
|---|---|---|
| No Reconstruction Loss | 0.688, 0.424, 0.288, 0.152, 0.064 | 1.616 |
| No Quantization Loss | 0.792, 0.544, 0.296, 0.224, 0.136 | 1.992 |
| No Perceptual Loss | 0.822, 0.564, 0.360, 0.242, 0.186 | 2.174 |
| No Static Loss | 0.792, 0.552, 0.424, 0.264, 0.152 | 2.184 |
| Full Loss Function | 0.880, 0.688, 0.648, 0.420, 0.340 | 2.976 |
Removing any loss leads to performance degradation, confirming that all components are essential. The reconstruction loss is most critical, followed by quantization, perceptual, and static losses. This validates the design of the disentanglement framework for embodied AI robot applications.
Furthermore, to assess the embodiment-agnostic nature of the learned latent states, a cross-embodiment experiment was performed in SIMPLER. The policy was trained on a mixed dataset of Google robot data (RT-1) and WidowX robot data (Bridge-V2), then tested on the Google embodied AI robot. The results were compared against training only on Google data.
| Training Data Configuration | Grasp Can Average | Move Near Object | Drawer Average | Overall Average |
|---|---|---|---|---|
| Single Dataset (Google only) | 0.433 | 0.389 | 0.278 | 0.367 |
| Mixed Dataset (Google + WidowX) | 0.490 | 0.418 | 0.338 | 0.415 |
The mixed dataset training yields an overall average of 0.415, higher than 0.367 for single-dataset training. This indicates that the latent states, conditioned on robot-type embeddings, capture generic manipulation patterns that transfer across different embodied AI robot platforms, enhancing the target robot’s performance.
In conclusion, the proposed method effectively leverages unlabeled video data to improve behavior learning for embodied AI robots. By disentangling static and dynamic features through unsupervised pre-training and integrating them via a multimodal Transformer, the approach addresses data scarcity and enhances generalization. Experimental results in CALVIN and SIMPLER simulations show significant improvements over baselines in task completion, cross-environment adaptation, and robustness. For embodied AI robot systems, this work provides a pathway to utilize abundant video resources, reducing reliance on costly annotated data. Future directions may include incorporating lighting-invariant features for better robustness and extending the framework to more complex embodied AI robot tasks in real-world settings.