The conceptual lineage of embodied intelligence can be traced to the seminal dichotomy presented by Alan Turing. While one path led to the abstract, disembodied intelligence prevalent in today’s AI, the other envisioned machines learning through physical interaction with the world via sensors—a vision that has crystallized into the modern paradigm of embodied AI. In essence, embodied intelligence refers to intelligent systems possessing a physical form that can engage in information and energy exchange with their environment. This “embodiment” is crucial for solving real-world problems, as interaction is fundamental to understanding and manipulation. The recent, explosive advancement of large-scale pre-trained models, or foundation models, has ignited a transformative fusion with embodied AI. These models, trained on internet-scale data encompassing text, images, and more, possess vast world knowledge, remarkable reasoning capabilities, and strong zero-shot generalization. They offer a potential solution to the long-standing challenges in robotics: data scarcity and poor generalization. By serving as the “cognitive brain” for embodied AI robot systems, foundation models can significantly enhance perception, understanding, high-level planning, and natural human-robot communication, paving the way for truly general-purpose robotic assistants. This article presents a comprehensive technical survey from the perspective of an active researcher in the field, detailing how foundation models are revolutionizing embodied intelligence across perception, control, system architecture, and data acquisition.
Perception and Scene Understanding
The primary interface between an embodied AI robot and its world is perception. Foundation models provide unprecedented capabilities for parsing multi-modal sensor data—images, text, depth—and forming a coherent, actionable understanding of the scene. This process can be decomposed into several key technical approaches.
1.1 Direct Multi-Modal Model Comprehension
Large Multi-Modal Models (LMMs) like GPT-4V are endowed with powerful cross-modal understanding, enabling them to directly interpret visual scenes in the context of language instructions. Frameworks such as ViLA utilize these models for closed-loop task planning. The model receives an initial observation and a goal, generates an action step, executes it, and then uses the new visual observation as feedback to replan. This creates a robust planning loop where the LMM’s reasoning is continuously grounded in real-time visual feedback. The process can be formalized as finding an action sequence \(A = (a_1, a_2, …, a_T)\) that maximizes the probability of achieving goal \(G\) given initial observation \(O_0\) and model parameters \(\theta\):
$$ A^* = \arg\max_A P(A | O_0, G; \theta) $$
where each subsequent action is conditioned on the history: \(a_t \sim P(\cdot | O_0, a_{1:t-1}, o_{1:t-1}, G; \theta)\).
1.2 Multi-Modal 3D Scene Modeling
While 2D image understanding is powerful, embodied AI robots operate in a 3D world. A significant research thrust focuses on constructing 3D scene representations imbued with semantic features from foundation models. Early works like PerAct use a Transformer-based architecture to process 3D voxelized scenes and language instructions, outputting discretized 6-DoF actions. The core innovation is the fusion of geometric and linguistic features within a unified architecture:
$$ \mathbf{F}_{scene} = \text{Transformer-Encoder}(\text{Voxelize}(O_{3D})) $$
$$ \mathbf{F}_{lang} = \text{CLIP-Text-Encoder}(G) $$
$$ A = \text{Decoder}(\text{Perceiver}(\mathbf{F}_{scene}, \mathbf{F}_{lang})) $$
More recent methods leverage advanced 3D representations like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) to create dense, queryable 3D language fields. For instance, LangSplat optimizes a 3DGS representation to store multi-scale CLIP features, enabling efficient open-vocabulary queries like “the blue mug on the left” directly in 3D space. The rendering of a semantic feature \(f\) at a 3D point \(\mathbf{x}\) from a Gaussian mixture model is:
$$ f(\mathbf{x}) = \sum_{i=1}^{N} \alpha_i \mathcal{N}(\mathbf{x} | \boldsymbol{\mu}_i, \boldsymbol{\Sigma}_i) \mathbf{c}_i $$
where \(\alpha_i\) is the opacity, \(\boldsymbol{\mu}_i\) and \(\boldsymbol{\Sigma}_i\) are the mean and covariance of the i-th Gaussian, and \(\mathbf{c}_i\) is its stored semantic feature vector (e.g., from CLIP).
1.3 Affordance and Constraint Extraction
Effective interaction requires understanding what actions are possible (affordances) and what constraints must be respected. Foundation models encode commonsense knowledge about object usage and physical constraints. Methods like VoxPoser elegantly demonstrate this by using an LLM to write code that queries a Visual Language Model (VLM) to synthesize 3D value maps for robot trajectories. The LLM, given a task, generates code that calls the VLM with specific prompts about the scene to produce an “affordance map” \(M_a(\mathbf{p})\) (attraction) and a “constraint map” \(M_c(\mathbf{p})\) (repulsion) over 3D positions \(\mathbf{p}\). The optimal trajectory \(\tau^*\) is then found by optimizing:
$$ \tau^* = \arg\max_{\tau} \left( \int_{\tau} M_a(\mathbf{p}) \, d\mathbf{p} – \lambda \int_{\tau} M_c(\mathbf{p}) \, d\mathbf{p} \right) $$
subject to kinematic and dynamic constraints. This zero-shot synthesis of actionable spatial constraints is a hallmark of how foundation models transfer internet-scale knowledge to embodied AI robot control.
1.4 Integrating Human Feedback
Natural language provides a flexible channel for humans to correct and guide embodied AI robots. Systems like OLAF and YAY Robot employ LLMs as “critics” to relabel sub-optimal robot actions based on human verbal corrections. If a human says “No, move the cup closer to the plate,” the LLM reinterprets the recent failed action segment and generates a corrected label for fine-tuning the policy. This creates a data-efficient learning loop: \((o_t, a_t, \text{“correction”}) \rightarrow (o_t, a_t^{corrected})\). Furthermore, visual prompting—where users click on keypoints in an image—addresses the imprecision of language for low-level geometry. Frameworks like MOKA use these clicks to ground VLM-based question-answering, generating precise motion plans like “move from keypoint A to keypoint B.”
Hierarchical Control and Planning
The integration of foundation models into the control stack of an embodied AI robot is typically hierarchical, leveraging their strengths in reasoning while relying on lower-frequency, specialized controllers for execution. We categorize this hierarchy into four levels, summarized in Table 1.
| Control Level | Core Function | Foundation Model Role | Representative Works |
|---|---|---|---|
| Demand-Level | Interpret high-level user intent and decompose it into achievable sub-goals. | Task decomposition, commonsense reasoning, feasibility checking. | SayCan, PaLM-E, EmbodiedGPT |
| Task-Level | Execute or sequence predefined skills (e.g., navigate, pick, place). | Skill selection, parameter generation (e.g., coordinates for “pick”), API calling. | CaP, OK-Robot, VIMA |
| Planning-Level | Generate detailed motion trajectories and action sequences for a specific task. | Spatio-temporal reasoning, value/affordance map synthesis, path planning. | VoxPoser, 3D-VLA, Text2Motion |
| Action-Level | Output low-level control commands (joint angles, end-effector poses). | Direct regression of action vectors from multi-modal inputs. | RT-1/2, Gato, RoboFlamingo |
Table 1: Hierarchy of control in embodied AI robots empowered by foundation models.
2.1 Demand-Level: Understanding and Decomposition
At this highest level, the model acts as an interface, translating ambiguous human requests into a structured plan. A seminal framework is SayCan, which combines the semantic knowledge of an LLM (the “Say”) with a learned value function representing the robot’s physical capabilities (the “Can”). For a set of primitive skills \(\mathcal{S}\), the probability of choosing skill \(s_i\) given instruction \(G\) and state \(o\) is:
$$ P(s_i | G, o) \propto \exp(\beta_{\text{say}} \cdot \text{LLM}(s_i, G) + \beta_{\text{can}} \cdot V_{\phi}(s_i, o)) $$
where \(V_{\phi}\) is the learned value function. This ensures plans are both semantically correct and physically feasible. Other works like EmbodiedGPT and LLM-Planner employ Chain-of-Thought (CoT) prompting to generate detailed, step-by-step reasoning before outputting a plan, significantly improving performance on long-horizon tasks.
2.2 Task-Level: Skill Orchestration
Here, the model’s output is a sequence of high-level skill calls or code. Code as Policies (CaP) demonstrates this by using few-shot prompting to make LLMs generate Python code that calls perception and control APIs. Given a few example command-code pairs, the LLM can generalize to new commands, performing spatial arithmetic and logic to, for example, “stack the green block on the blue one and move it near the door.” The model \(M\) generates code \(C\) as:
$$ C = M(\text{[Examples]} \oplus \text{“Command: “}G \oplus \text{“# Code:”}) $$
This code is then executed to control the robot. Similarly, OK-Robot integrates off-the-shelf models (OWL-ViT for detection, CLIP for search, LangSam for segmentation) in an open-loop pipeline orchestrated by logical rules to perform pick-and-place from natural language queries.
2.3 Planning-Level: Trajectory Synthesis
This level involves generating the continuous path or sequence of poses to achieve a sub-goal. As discussed, VoxPoser synthesizes trajectories from value maps. Another approach is embodied world models, like 3D-VLA, which are generative models trained on interaction data. Given a current state \(o_t\) and goal description \(G\), they can predict a future goal state \(o_{t+H}\) or a sequence of actions. The planning objective can be framed as:
$$ A^*_{t:t+H} = \arg\max_{A_{t:t+H}} \mathbb{E}[ \mathcal{R}(o_{t+H}) ] \quad \text{s.t.} \quad o_{t+k+1} \sim \mathcal{P}_{\theta}(o_{t+k+1} | o_{t+k}, a_{t+k}) $$
where \(\mathcal{P}_{\theta}\) is the learned world model dynamics and \(\mathcal{R}\) is a reward function aligned with the goal. These models enable “mental simulation” for planning.
2.4 Action-Level: Direct Control
Some end-to-end approaches bypass explicit planning and output low-level actions directly. The Robotics Transformer (RT) series is exemplary. RT-1 uses a FiLM-conditioned EfficientNet and TokenLearner to process images and text, feeding them into a Transformer decoder that outputs discretized action tokens (e.g., delta end-effector pose, gripper open/close). The action prediction for timestep \(t\) can be seen as:
$$ a_t = \text{Decoder-Transformer}( \text{TokenLearner}( \text{FiLM}(E_{img}(I_t), E_{txt}(G)) ) ) $$
where \(E_{img}\) and \(E_{txt}\) are encoders. RT-2 goes further by co-training on web-scale vision-language data and robot data, representing actions as text tokens within the model’s vocabulary (e.g., “pose 0.1 0.2 0.3”), enabling remarkable semantic generalization.
System Architectures
The architectural paradigms for building embodied AI robot systems with foundation models fall into two broad categories, each with distinct trade-offs, as summarized in Table 2.
| Architecture | Core Principle | Pros | Cons | Examples |
|---|---|---|---|---|
| End-to-End Transformer | Single, monolithic model trained to map multi-modal inputs directly to actions. | High performance, elegant, enables emergent capabilities from scale. | Data-hungry, computationally heavy, less interpretable, hard to debug. | RT-1/2, Gato, ACT (in ALOHA) |
| Modular with Frozen Models | Orchestrates multiple pre-trained, frozen specialist models (LLM, VLM, etc.) via prompting or code. | Leverages powerful off-the-shelf models, interpretable, flexible, data-efficient. | Latency from sequential calls, prompt engineering, integration complexity. | TidyBot, VoxPoser, Instruct2Act |
Table 2: Comparison of predominant system architectures for embodied AI.
The end-to-end approach, exemplified by RT-1/2 and Gato, seeks to create a unified model that ingests images, text, and past actions to predict the next action token. This paradigm benefits enormously from scaling laws: as model size and diverse data increase, generalization improves. The training objective is a standard sequence modeling loss:
$$ \mathcal{L} = -\sum_{t} \log P(a_t | I_{1:t}, G, a_{1:t-1}; \theta) $$
However, it requires massive, curated robot datasets and significant compute.
The modular approach treats foundation models as tools. For example, TidyBot uses CLIP to classify objects and an LLM to summarize user preferences into placement rules (e.g., “clothes go in the hamper”). Instruct2Act employs an LLM as a planner that generates Python code; this code calls APIs for SAM to segment an object and a motion planner to generate a trajectory. The system’s flexibility comes from the LLM’s ability to compose these tools dynamically based on the task. The decision process can be abstracted as:
$$ \text{Plan} = \text{LLM}(G, O, \text{API\_Descriptions}) $$
$$ \text{Execute}(\text{Plan}) \rightarrow \text{Call SAM}(\text{query}), \text{Call Controller}(\text{pose}), … $$
This design is highly interpretable and adapts quickly to new tools but suffers from cumulative latency and brittleness in prompt design.
Data Sources and Acquisition
The performance of any embodied AI robot system is fundamentally constrained by the quality and diversity of its training data. Foundation models both create new demands for data and enable novel methods for its acquisition. The primary sources are compared in Table 3.
| Data Source | Description | Key Techniques/Models Used | Challenges |
|---|---|---|---|
| Simulators | Programmatic generation of tasks and environments in virtual physics engines. | LLMs for task/asset generation (RoboGen), Reinforcement Learning, Domain Randomization (DrEureka). | Sim-to-real gap, fidelity of physics and rendering. |
| Imitation Learning (IL) | Recording human demonstrations via teleoperation or motion capture. | Action Chunking Transformers (ACT), Diffusion Policies, Behavior Cloning. | Costly to scale, human demonstrator bias, distributional shift. |
| Video Learning | Learning from vast online videos of human or animal activity. | Inverse Dynamics Models (VPT), Reward labeling with VLMs (RoboCLIP), Affordance learning (VRB). | No action labels, domain gap (human vs. robot embodiment), distractors. |
Table 3: Primary data sources for training embodied AI robot models.
3.1 Simulation
Simulators like Isaac Gym and benchmarks like BEHAVIOR-1K provide safe, parallelizable, and cost-effective data generation. The key innovation is using LLMs to automate this process. For instance, RoboGen employs a “propose-generate-learn” loop: an LLM proposes a task (e.g., “water a plant”), then generates the simulated environment and training supervision (e.g., via reinforcement learning or motion planning). DrEureka uses LLMs to automatically write reward functions and domain randomization parameters for Sim2Real transfer, optimizing for policy performance in reality. The objective is to find simulation parameters \(\phi\) and a reward \(R_{\psi}\) that maximize real-world policy return \(\eta\):
$$ \phi^*, \psi^* = \arg\max_{\phi, \psi} \eta_{\text{real}}(\pi_{\theta^*}) \quad \text{s.t.} \quad \theta^* = \arg\max_{\theta} \mathbb{E}_{\tau \sim p_{\phi}}[\sum R_{\psi}(\tau)] $$
where \(\pi_{\theta^*}\) is the policy trained in simulation with parameters \(\phi\) and reward \(R_{\psi}\).
3.2 Imitation Learning from Demonstrations
Directly mimicking expert demonstrations remains highly effective for learning complex skills. The ALOHA system popularized low-cost, bimanual teleoperation, collecting (\(o_t, a_t\)) pairs. Its ACT algorithm uses a Conditional Variational Autoencoder (CVAE) with a Transformer architecture to predict chunks of future actions, improving temporal consistency. The training maximizes the Evidence Lower Bound (ELBO):
$$ \mathcal{L}_{\text{CVAE}} = \mathbb{E}_{q(z|s,a)}[\log p(a|s,z)] – \beta D_{KL}(q(z|s,a) || p(z)) $$
where \(s\) is the state/observation, \(a\) is the action chunk, and \(z\) is a latent variable. Frameworks like UMI extend this by designing hardware and software interfaces for portable, in-the-wild data collection, enabling large-scale diverse dataset assembly.
3.3 Learning from Internet Videos
The ultimate scalable data source is the internet’s vast video repository. The challenge is the lack of action labels. Video PreTraining (VPT) tackles this by training an inverse dynamics model (IDM) on a small amount of labeled video to predict actions between frames, then using it to label a large unlabeled corpus. The IDM learns \(P(a_t | o_t, o_{t+1})\). RoboCLIP takes a different approach, using the similarity in a pre-trained VLM’s embedding space between a task description/video and an agent’s video to create a reward signal for reinforcement learning:
$$ R(o_{0:T}) = \text{cosine\_sim}( E_{\text{VLM}}(\text{task}), E_{\text{VLM}}(\text{video}(o_{0:T})) ) $$
This allows learning from a single video or text description without manual reward engineering.
Challenges and Future Directions
Despite rapid progress, significant challenges remain for robust, real-world deployment of foundation-model-powered embodied AI robots.
1. The Scarcity of Real-World Robot Data: While initiatives like RT-X aggregate datasets across labs, the scale is minuscule compared to internet data for LLMs. Collecting diverse, in-the-wild physical interaction data is expensive and slow. Future work must focus on cheaper, automated data collection frameworks and better algorithms for sample-efficient learning from limited real data, perhaps through more advanced simulation-to-real transfer and foundation model priors.
2. Inference Latency and Real-Time Control: The inference speed of large models (especially LMMs) is often incompatible with high-frequency robot control (e.g., 10-100 Hz). Current solutions involve using smaller models (RT-1) or relegating the large model to a slower, asynchronous planning loop. Research into model distillation, quantization, and efficient architectures for robotics is critical. The trade-off between \( \text{Performance} = f(\text{Model Size}, \text{Inference Time}) \) needs explicit optimization for the embodied domain.
3. Embodied Multi-Agent Collaboration: Solving complex real-world problems will require teams of embodied AI robots. Coordinating them with foundation models introduces challenges in shared world modeling, communication, and task decomposition. Preliminary work like Swarm-GPT for drone choreography and AutoRT for multi-robot data collection points the way. The core problem involves joint planning where the action space is the Cartesian product of individual agent actions: \( \mathcal{A} = \mathcal{A}_1 \times \mathcal{A}_2 \times … \times \mathcal{A}_N \), and the foundation model must reason about collaboration and conflict.
4. Safety, Reliability, and Verification: The black-box nature and potential for hallucination in foundation models pose serious safety risks in physical systems. Techniques for verifiable planning, uncertainty quantification (like KnowNo), and building guardrails that prevent physically impossible or dangerous actions are essential. This intersects with the need for models that have a deeper, more grounded understanding of physics and cause-effect relationships.
In conclusion, the integration of foundation models represents a paradigm shift for embodied AI. By providing a reservoir of commonsense knowledge, reasoning ability, and zero-shot generalization, they are transforming robots from single-task specialists towards adaptive, general-purpose agents. The fusion has progressed from high-level task specification to detailed motion synthesis, enabled by novel architectures and data strategies. However, the path to reliable, real-world deployment is fraught with technical hurdles centered on data, speed, coordination, and safety. Overcoming these challenges will require concerted efforts at the intersection of robotics, machine learning, and human-computer interaction, steadily advancing us toward the long-envisioned future of intelligent machines that seamlessly interact with and assist in our physical world.