A newly published online paper in ZTE Technology Journal examines one of the most consequential system-level problems in modern robotics: how to move embodied intelligence from cloud-based remote inference to closed-loop execution on the device itself. The paper, authored by Xu Mengwei and Xu Daliang of the School of Computer Science at Beijing University of Posts and Telecommunications, argues that embodied intelligence is entering a critical phase in which large models must operate under strict real-world constraints, including control frequencies of 10 to 100 Hz, limited power budgets, and unstable or intermittent network connectivity. The work is supported by the National Natural Science Foundation of China under grant 62522202 and was released online on August 28, 2026, with citation details recorded as Xu M W, Xu D L, ZTE Technology Journal, 2026, 32(4): XX-XX.
The paper opens with the Moravec paradox, a long-standing observation that abstract reasoning tasks often prove easier for computers than the perception, motion, and physical interaction skills that even young children can perform. Over the past decade, deep learning and large models have made remarkable progress in digital domains such as language, vision, and code, yet they still struggle to drive robots through open physical environments for grasping, navigation, collaboration, and maintenance. As artificial intelligence expands from screens, text, and cloud services into homes, factories, transportation, healthcare, and space facilities, embodied intelligence becomes a central challenge for the next decade. The effectiveness of embodied intelligence systems will determine whether large models can evolve from digital assistants into physical agents.
In this context, two major technical routes have emerged. The first is the vision-language-action model, or VLA, which uses a vision-language model as its backbone, encodes camera observations, language instructions, and historical states into a unified representation, and directly outputs robot actions. The second is the world-action model, or WAM, which uses future state prediction—such as images, latent representations, or other forms—as its core training objective and extends world-model thinking toward real robot action generation. Together, these models are pushing embodied intelligence from task-specific policies toward general foundation models, allowing robots to understand instructions, anticipate environmental changes, and generate continuous actions in more complex and open scenarios. However, the journey from laboratory demonstration to real deployment still faces a critical system efficiency challenge. Because robots must make continuous decisions while interacting with the physical world, network jitter directly degrades control performance, and disconnection can even cause task failure. As a result, embodied intelligence is shifting from cloud-based large-model inference to models that reside on the device and complete closed-loop control locally.
Yet stable robot manipulation typically requires control frequencies of 10 to 100 Hz. Billion-parameter VLA models and video-diffusion-based WAM models often cannot reach such inference efficiency on end-side computing platforms. The paper therefore discusses and forecasts efficient on-device embodied intelligence systems from both algorithm and system perspectives, emphasizing the co-design of algorithms, systems, and chips. The common optimization objectives are action success rate, closed-loop latency, and energy efficiency. The key strategies include few-step generation, action-aware compression, cache reuse, asynchronous chunk execution, and specialized runtimes.
-
Model Architectures and the Bottlenecks of On-Device Embodied Intelligence
On-device embodied intelligence is primarily driven by two classes of large models. One class is the VLA, which uses a vision-language model as its backbone and maps observations and instructions directly to actions. The other is the WAM, which supports planning and control by predicting future observations. Their inference architectures and computational characteristics differ, and the end-side bottlenecks they face are also different. Understanding these differences is essential for designing effective optimizations.
-
VLA Inference Flow and Bottlenecks
A VLA model is typically composed of a visual encoder, a vision-language backbone network, and an action head. During inference, the robot’s current visual observation is first converted by the encoder into visual tokens. These tokens, together with tokenized language instructions, are fed into the backbone network for a forward computation, often called the prefill stage, producing a context representation that fuses scene understanding and instruction intent. The action head then uses this representation as a condition to generate low-level control commands, usually in the form of action chunks that output several future steps at once and execute them in sequence.
In terms of action head implementation, early approaches discretized continuous actions and generated them autoregressively token by token. More recent high-performance models increasingly adopt diffusion or flow-matching action experts. These methods start from random noise and generate continuous action vectors through multi-step denoising iterations. Because they are effective at continuous control and modeling multimodal action distributions, they are becoming mainstream.
The end-side bottlenecks of VLA models are interrelated and jointly constrain system efficiency. First, the visual encoder and backbone network account for the main computational cost. A single image frame is often expanded into hundreds to thousands of visual tokens, far more than language and action tokens, leading to high attention computation and feed-forward network operations during the prefill stage. Because each memory access in this stage is accompanied by many floating-point operations, performance is mainly limited by compute rather than memory bandwidth. According to roofline model analysis, visual encoding and the backbone can be classified as compute-bound.
In contrast, the multi-step denoising process of the action expert repeatedly reads model weights and key-value caches at each step, while the per-step computation is relatively limited. As a result, its performance becomes memory-bandwidth-bound. Measurements on two end-side graphics processing units further show this effect. On Jetson Thor, the action expert latency share of the Pi-0.5 model is about 38 percent. On a consumer-grade RTX 5090, the action head latency share of GR00T N1.7 rises to 59.2 percent, but its model compute utilization is only 13.2 percent, while the backbone reaches 33.0 percent. The more fully the backbone is accelerated, the more prominent the relative cost of action generation becomes. Simply increasing peak compute cannot effectively remove this bottleneck.
This two-stage structure—a compute-bound backbone paired with a memory-bound action expert—directly causes a mismatch between control frequency and model inference latency. Unoptimized 3B to 7B parameter VLA models on edge platforms typically achieve only about 6 to 19 Hz. In measurements on Jetson Thor with the state-of-the-art open-source model Pi-0.5, the official reference implementation achieves an equivalent inference frequency of only 5.6 Hz. Even after system optimization, it reaches only 9.3 Hz, still far below the 30 Hz target frequency desired for dexterous manipulation.
Moreover, the tokens of interest in the prefill stage and the action generation stage are not the same. The former focuses on global scene understanding, while the latter focuses on local regions closely related to the current manipulation. Therefore, directly transferring visual token pruning methods designed for general vision-language models to VLA scenarios often causes key tokens required by actions to be mistakenly removed, harming task success rate. This phenomenon requires that compression strategies for on-device VLA models be action-aware.
-
WAM Inference Flow and Bottlenecks
World models have two typical uses in embodied scenarios. First, they can act as action generators that directly output actions while predicting the future. Second, they can act as rollable environment simulators that perform lookahead rollouts over candidate action sequences and select the best one. Regardless of the use, the computational cost is concentrated in future prediction generation. Future frames or latent states must be obtained through multi-step diffusion sampling, so a single prediction is already expensive. Long-horizon rolling generation for planning further amplifies this cost. On one hand, memory usage grows linearly with the number of rollout steps. On the other hand, prediction errors accumulate gradually. Together, these factors limit the usable planning depth. Generating frame by frame in pixel space is much more expensive than rolling in latent space.
Measurements of Cosmos3-Nano show that in end-to-end latency, the four-step denoising loop contained in a single action chunk generation accounts for about 78 percent on Jetson Thor. Unlike the VLA action expert, the WAM generation module already exhibits compute-bound characteristics at batch size 1. Its arithmetic intensity is about 926 FLOP per byte, far above the Jetson Thor performance bottleneck inflection point of about 450 FLOP per byte. Its model compute utilization reaches 65.8 percent. Therefore, its latency is mainly determined by the raw compute demand of multi-step denoising, and simply increasing memory bandwidth yields limited benefit.
The memory constraint is even more severe. The peak memory of a single inference for this model reaches about 30 GiB, which cannot run on a 32 GB consumer-grade RTX 5090 because of memory overflow. In addition, world models capable of real-time rolling still mostly rely on server-grade GPUs. How to compress them to end-side devices and complete rolling generation within the control cycle remains an insufficiently solved problem.
The following table summarizes the stage-level profiling and bottleneck localization for Pi-0.5, GR00T N1.7, and Cosmos3-Nano-Policy at batch size 1. These results are central to the paper’s argument that on-device embodied intelligence cannot be optimized by a single generic technique. Instead, the bottleneck must be identified per model and per stage.
Metric Pi-0.5 on Jetson Thor GR00T N1.7 on RTX 5090 Cosmos3-Nano on Jetson Thor End-to-end latency / ms 124.6 20.5 8121 Equivalent frequency / Hz 8.00 48.80 0.12 Backbone or conditioning stage / ms 76.8 (61.6%) 8.3 (40.8%) 1824 (22.5%) Action head or generation stage / ms 47.8 (38.4%) 12.1 (59.2%) 6298 (77.5%) Arithmetic intensity of the stage / FLOP per byte 50.8 135.8 926.2 Device performance bottleneck inflection point / FLOP per byte 450.4 139.4 448.9 Model compute utilization of the stage / % 7.6 13.2 65.8 Bottleneck classification Memory-bound Memory-bound Compute-bound These figures show that embodied intelligence is not a single uniform workload. The VLA action expert is memory-bound, while the WAM generation module is compute-bound. The former needs reduced memory access and launch overhead. The latter needs reduced denoising compute demand. This distinction shapes the algorithm and system optimizations discussed in the rest of the paper.
-
-
Efficient Algorithms for Embodied Intelligence
Efficient algorithms reduce inference cost without relying on specialized hardware. They include methods that directly modify the model itself and methods that reshape the inference execution process. Both are algorithm-level means. Because the diffusion action expert of VLA and the video diffusion transformer of WAM both use iterative diffusion as their computational core, the paper first discusses shared diffusion generation acceleration strategies, then separately examines VLA-specific and WAM-specific optimizations that exploit embodied structural features.
-
General Algorithm Optimizations
Few-step denoising generation is one of the most direct acceleration paths. Diffusion and flow-matching generation require multi-step iterative denoising, and the number of steps almost linearly determines inference latency. Consistency and flow-matching distillation methods compress a multi-step teacher model into a single-step or few-step student model, significantly improving inference speed while trying to preserve action generation quality. For example, OneDP uses a loss function that minimizes the score difference of the diffusion chain Kullback-Leibler divergence and distills a multi-step diffusion policy into a single-step generator, increasing action prediction frequency from 1.5 Hz to 62 Hz on an NVIDIA V100 GPU. Because actions are essentially time-series signals, FreqPolicy further imposes consistency constraints in the frequency domain to ensure the quality of single-step generated actions. Streaming Diffusion Policy maintains a variable-noise action buffer, fully denoises only the action currently to be executed, and keeps the noise state of subsequent actions, producing executable actions with less computation.
Reducing weight or activation precision is another acceleration path orthogonal to step reduction. Post-training quantization methods for video diffusion transformers, such as ViDiT-Q and PTQ4DiT, can compress weights and activations to 4 to 8 bits, effectively reducing memory access overhead. Quantization on the VLA side must be tightly coupled with control tasks and must use manipulation success rate as a core constraint.
Diffusion feature caching exploits the fact that intermediate features between adjacent denoising steps or adjacent generation blocks are often highly similar. Reusing these features can skip redundant computation without retraining the model. However, general diffusion caching methods are difficult to transfer directly to policies and world models because of architectural differences. Targeted designs have therefore been proposed. For example, block-wise adaptive caching, or BAC, designs a block-level adaptive caching mechanism for diffusion policies and points out that feature similarity between different time steps and network blocks changes non-uniformly.
Asynchronous action chunk execution addresses the reaction lag introduced at chunk boundaries. Although action chunking improves throughput, the inference latency required to generate the next chunk may prevent the robot from responding in time in dynamic environments. The basic idea of asynchronous execution is to generate the next chunk while executing the current chunk, thereby hiding inference latency inside the execution process. Because this method only involves chunk generation and execution scheduling, it can apply to any policy that outputs action chunks and therefore has good generality. A representative work, real-time chunking execution, formulates the problem as image inpainting. Actions already committed for execution are frozen as known quantities, while the remaining parts are treated as unknown quantities to be repaired. A guidance term ensures smooth transitions. This method requires no training and can adapt to any diffusion or flow-matching policy, tolerating inference latency exceeding 300 ms.
-
VLA-Specific Optimizations
VLA-specific optimizations share a common theme: explicitly exploiting structured features such as robot temporal continuity and action correlation. They can be grouped into four categories: efficient architectures, action-aware compression, action generation acceleration, and inference execution acceleration.
Efficient architectures reshape the model structure from multiple paths to address the high overhead of the backbone and encoder. To mitigate the quadratic complexity of self-attention, RoboMamba replaces the standard attention backbone with a state-space model of linear complexity. To reduce overall model size, TinyVLA and SmolVLA adopt sub-billion-parameter backbones and combine them with diffusion action experts for lightweight design. To address the problem that visual tokens dominate prefill computation, FastVLM starts from encoder design and directly reduces the number of visual tokens, significantly lowering first-token latency.
Action-aware compression differs from general model compression. These methods use manipulation success rate rather than reconstruction accuracy as the evaluation criterion and allocate the limited bit or token budget to the parts most critical for action generation. In quantization, saliency-aware quantized imitation learning, or SQIL, uses action saliency and a quantization-robust behavior cloning loss to concentrate precision budget on key states that strongly affect control performance. This allows a four-bit model to achieve about 2.5 times speedup on a Jetson GPU with nearly no performance loss. BitVLA further adopts native ternary weights, reducing model memory overhead to only 1.4 GB, equivalent to one-eleventh of the original OpenVLA model. In pruning, VLA-Pruner addresses the semantic-action gap caused by different attention distributions in the prefill stage and action decoding stage. It assigns two-level scores to each token based on both semantic importance and action relevance, preserving key tokens that actions truly depend on while removing redundancy.
Action generation acceleration directly shortens the decoding process of the action expert or autoregressive action head. One class of work aims to reduce iteration steps or enable parallel decoding. For example, CEED-VLA transforms Jacobi decoding through consistency distillation and early-exit strategies, achieving about 4.1 times acceleration for OpenVLA inference on an NVIDIA H100 GPU. Another class improves the representation and decoding form of actions. OpenVLA-OFT replaces the original autoregressive discrete generation with parallel decoding, action chunking, and continuous action representation, achieving about 26 times acceleration for OpenVLA inference on an NVIDIA A100 GPU and reaching a 25 Hz action control frequency on a real ALOHA robot. FAST compresses action token sequences through the discrete cosine transform, reducing generation cost at the representation level.
Inference execution acceleration reshapes decoding and serving execution without changing the model itself. Speculative action decoding alleviates the serial bottleneck of token-by-token generation through a draft-verify mechanism. Spec-VLA and FLASH cover autoregressive and diffusion action heads, respectively. FLASH uses the action expert to parallel-verify flow consistency and achieves about 3.04 times acceleration on a single RTX 4090D GPU. Key-value cache reuse exploits inter-frame redundancy to reduce computation. For example, VLA-Cache reuses the key-value representations of static visual tokens that change very little between adjacent frames and selectively recomputes task-relevant tokens, reducing per-step visual processing overhead and achieving up to about 1.7 times acceleration for OpenVLA inference on a single RTX 4090 GPU. ActionFlow, a cross-request pipeline method, further packages and schedules the memory-bound decoding stage and the compute-bound prefill stage across control steps, driving both compute and bandwidth resources toward saturation.
-
WAM-Specific Optimizations
The end-side efficiency of WAM ultimately depends on how to generate actions from the world model at low computational cost. The paper characterizes its action generation paradigm along two orthogonal dimensions. The first is the temporal coupling between prediction and action, which can be sequential, where future prediction is completed first and actions are derived afterward, or joint, where prediction and action are produced in the same generation process. The second is the maturity of the latent state on which actions depend, meaning whether actions rely on fully denoised prediction results or on intermediate representations that are not yet fully denoised. On this two-dimensional basis, the paper adds two orthogonal cost-reduction paths: using latent-space representations instead of explicit pixel generation, and execution-level optimization for long-horizon rollouts.
In the sequential paradigm, the execution flow follows prediction first, then action. Early implementations rely on complete future state prediction, which offers high fidelity but high computational cost. For example, Seer predicts future frames and then derives actions from an inverse dynamics model. DreamVLA instead predicts compact world knowledge representations such as dynamic regions, depth, and semantics rather than reconstructing full future images, reducing prediction cost.
In the joint paradigm, future states and actions are produced simultaneously through a shared denoising or forward process, allowing both to share physical dynamics priors. For example, PAD jointly denoises images and actions in a single diffusion process, while WorldVLA models actions and the world model in an autoregressive manner. The limitation of this paradigm is that actions must wait until the full denoising process is finished. Therefore, producing actions early from intermediate representations within a joint framework—combining information sharing and low latency—remains an open direction without a mature representative work. However, Unified World Models, or UWM, already set independent noise time steps for each modality, providing foundational support for generating actions early from intermediate representations.
Latent-space world modeling is a more fundamental cost-reduction approach: abandoning pixel-level generation and completing prediction in latent space. LAPA learns discrete latent actions from unlabeled videos and uses them for pretraining. FLARE aligns policy features with latent embeddings of future observations without generating complete future pixel representations. Both avoid explicit pixel-level prediction, effectively reducing the computational burden of world models at the representation and generation levels.
At the execution level, streaming rollouts and long-term memory suppress long-horizon error accumulation and manage the growing key-value cache through joint denoising, attention sinks, and historical memory compression. A representative work, Rolling Forcing, adopts three designs. First, it applies increasing noise to multiple frames within a window through joint denoising, updating the entire window at once rather than iteratively sampling frame by frame, thereby relaxing strict causal constraints between adjacent frames. Second, it uses an attention sink to retain the key-value state of the first frame as a global anchor over the long term and adjusts its rotary position encoding offset to keep relative positions bounded. Third, it applies few-step distillation on a significantly lengthened denoising window and conditions on self-generated history to mitigate exposure bias.
-
-
Efficient On-Device Embodied Inference Systems
Inference engines and runtimes for on-device embodied workloads have emerged rapidly in recent years, and the open-source ecosystem has begun to take shape. There are specialized frameworks for full-stack optimization of physical intelligence models, general foundations that evolved from on-device large language models and cover multiple heterogeneous backends, and end-to-end reference implementations maintained by the robot learning community. The paper systematically measures single-request latency on two classes of end-side GPUs, Jetson Thor and RTX 5090, for four representative embodied models: Pi-0, Pi-0.5, GR00T N1.7, and Cosmos3-Nano.
Model Device LeRobot / ms PhyAI / ms FlashRT / ms vla.cpp / ms realtime-vla / ms mllm / ms Pi-0 Jetson Thor 321.0 121.5 61.0 267.8 116.5 NA Pi-0 RTX 5090 127.5 31.5 22.2 61.6 32.7 NA Pi-0.5 Jetson Thor 179.5 107.7 86.9 309.1 123.8 NA Pi-0.5 RTX 5090 52.1 28.6 28.5 54.2 27.5 NA GR00T N1.7 Jetson Thor 133.5 95.4 65.8 164.6 NA NA GR00T N1.7 RTX 5090 47.0 20.6 19.3 32.7 NA NA Cosmos3-Nano Jetson Thor NA 8121.4 8027.5 NA NA NA Performance measurements are based on local tests of each project’s official repository as of August 2, 2026. NA indicates that the runtime does not support that model or that no usable data is available. The comparison shows that specialized runtimes are indeed faster in specific combinations. FlashRT on Jetson Thor achieves 86.9 ms for Pi-0.5 and 65.8 ms for GR00T N1.7. realtime-vla on RTX 5090 achieves 27.5 ms for Pi-0.5. These results are slightly better than PhyAI’s 107.7 ms, 95.4 ms, and 28.6 ms, respectively. However, their applicability is narrower. realtime-vla covers only the Pi series and DM0. Neither realtime-vla nor vla.cpp supports world-action models. Along with FlashRT, they do not provide cloud-edge collaborative inference. Although mllm has the broadest backend coverage, it does not yet include embodied models.
Among the six systems tested, PhyAI is the only framework that provides measured data for all four models, covers both VLA and WAM, and supports cloud-side scaling through data parallelism, tensor parallelism, and classifier-free guidance parallelism. Its latency is in the same tier as the fastest specialized runtimes. On RTX 5090 for Pi-0.5, it is only 0.1 ms slower than FlashRT and 1.1 ms slower than realtime-vla. Therefore, within the test scope, PhyAI achieves a good trade-off between coverage and latency. It uses one code path and one checkpoint to cover device, edge, and cloud deployment scenarios, avoiding repeated inference implementation for different environments while maintaining latency at the level of specialized runtimes.
The paper also compares the software capabilities of the six systems. The following table summarizes supported models, supported precision, hardware backends, and whether cloud-edge collaborative inference is supported.
System Supported models Supported precision Hardware backends Cloud-edge collaborative inference PhyAI VLA and WAM BF16, FP8, INT8, NVFP4, MXFP4, INT4 Jetson Thor, RTX 5090/4090, A40/A100, H20/H100 Yes (DP, TP, CFG parallelism) mllm General multimodal large language models W4A8, W8A8, W4A16 Arm CPU, Qualcomm Hexagon NPU, Ascend NPU, OpenCL, CUDA No LeRobot VLA and WAM FP32, BF16 Jetson Thor, RTX 5090/4090, A40/A100, H20/H100 Yes vla.cpp VLA W2A8 Jetson Orin Nano/Thor Yes realtime-vla VLA BF16 Jetson Thor, RTX 4090/5090 No FlashRT VLA and WAM BF16, FP8, NVFP4, INT8/INT4 Jetson Thor/Orin, RTX 5090/4090 No -
PhyAI
PhyAI is a high-performance inference framework for physical intelligence models, covering both VLA and WAM. Its most distinctive feature is that it uses a single code path across device-side embodied systems, edge servers, and the cloud. Model evaluation, cloud reinforcement learning replay, edge GPU serving, and robot body deployment share the same checkpoint and the same inference path, eliminating the need to write separate inference programs for each scenario. The framework supports low-bit inference optimizations such as NVFP4 and INT4. It also implements key techniques including diffusion transformer caching, operator fusion, and separate deployment of the vision-language backbone and diffusion action expert. The model adaptation layer is decoupled from the runtime, so a new architecture only requires a new adapter, while scheduling, memory management, operator selection, and parallel serving are provided uniformly by the runtime.
On Jetson Thor, PhyAI compresses the single-request latency of Pi-0.5 from 179.5 ms in the official reference implementation to 107.7 ms, about 9.3 Hz and a 1.67 times speedup, approaching the 10 Hz lower bound required for manipulation. On a consumer-grade RTX 5090, latency drops further to 28.6 ms, about 35 Hz, entering the real-time control frequency range required for dexterous manipulation.
-
mllm
mllm is a multimodal large-model inference engine for mobile devices. It is implemented in C and C++, does not rely on a heavy runtime, and supports multiple levels of weight and activation quantization, including W4A8, W8A8, and W4A16. Its outstanding advantage is broad backend coverage. In addition to CUDA and OpenCL, it supports Qualcomm Hexagon NPU, Ascend NPU, and Arm CPU, which are end-side heterogeneous compute resources not yet covered by the other frameworks. The llm.npu system further proposed on this engine resolves the NPU’s unfriendliness to dynamic shapes and activation outliers through prompt chunking, outlier extraction, and block-level out-of-order scheduling. It raises the prefill stage of billion-parameter models to more than 1,000 tokens per second. At present, mllm does not include VLA or WAM models, so its latency entries are recorded as NA. However, it can serve as a general foundation for carrying the VLA vision-language backbone and migrating it to the NPU side.
-
LeRobot
LeRobot is an end-to-end robot learning open-source framework maintained by HuggingFace. It targets real robot scenarios, is compatible with low-cost and edge hardware, and provides checkpoint loading, data processors, training code, reference implementation behavior, and a general asynchronous policy inference stack. It has integrated models that can run directly on the device, including SmolVLA, action chunking with transformer, diffusion transformer, and Pi-0. The framework covers the full model lifecycle and provides reference implementations rather than pursuing extreme operator-level optimization, so it mainly uses FP32 and BF16 reference implementations. In tests, its Pi-0.5 inference path achieves 179.5 ms on Jetson Thor, about 5.6 Hz, and 52.1 ms on RTX 5090, about 19.2 Hz.
-
vla.cpp
vla.cpp is the first unified C++ runtime at the ggml and llama.cpp level that natively supports flow matching and diffusion action experts. It covers multiple VLA architectures. The runtime supports key-value reuse of the vision-language prefix across solver steps, avoiding repeated computation of the prefix at every denoising step. It also uses ternary 1.58-bit W2A8 weight quantization to compress memory access and memory footprint, enabling operation on memory-constrained devices such as the 8 GB Orin Nano. It supports cloud-edge separated deployment through a zero message queue client-server form. In tests, its Pi-0.5 model achieves 309.1 ms on Jetson Thor, about 3.2 Hz, and 54.2 ms on RTX 5090, about 18.4 Hz. This indicates that its broad portability comes at a certain latency cost.
-
realtime-vla
realtime-vla decomposes the Pi-0.5 model into 24 general matrix multiplication operators and applies Triton kernel optimization level by level to achieve fully streaming inference. It currently covers models including Pi-0, Pi-0.5, and DM0. The framework does not introduce quantization and relies only on BF16 kernel tuning and streaming scheduling for acceleration, so its precision is fully aligned with the reference implementation. In tests, its Pi-0.5 model achieves 123.8 ms on Jetson Thor, about 8.1 Hz, and 27.5 ms on RTX 5090, about 36.4 Hz.
-
FlashRT
FlashRT is a real-time inference engine for small-batch, low-latency workloads such as VLA and WAM. It supports multiple low-precision options, including FP8, NVFP4, and INT8/INT4. Its hardware backends cover Jetson Thor, the RTX series, and Jetson AGX Orin. In addition to VLA, it also supports WAM models such as Cosmos3. In tests, its Pi-0.5 model achieves 86.9 ms on Jetson Thor, about 11.5 Hz, and 28.5 ms on RTX 5090, about 35.1 Hz.
-
-
Future Outlook: Algorithm-System-Chip Co-Design for Embodied Intelligence
Although the paper has surveyed a rich set of optimization methods at both algorithm and system levels, on-device embodied intelligence still has a clear gap before practical deployment. Even after optimization, current systems often struggle to stably reach the 10 Hz required for manipulation, let alone the 30 Hz or higher frequency desired for manipulation tasks. The measurements confirm this gap. The official reference implementation of Pi-0.5 on Jetson Thor achieves only 5.6 Hz, and even after optimization it reaches only 9.3 Hz. Cosmos3-Nano is even lower at 0.12 action chunks per second and cannot run on a consumer-grade RTX 5090 because of memory overflow.
The root cause differs between the two model classes. For VLA action generation under the single-sample batch processing that robots must use, memory access and launch overhead become the main bottlenecks. Model compute utilization on Jetson Thor is only 7.6 percent. On RTX 5090, the action head of GR00T N1.7 is only 13.2 percent. For WAM, the generation module already exhibits compute-bound characteristics at batch size 1, with utilization reaching 65.8 percent, and its latency is mainly determined by the raw compute demand of multi-step denoising. The former needs reduced memory access and launch overhead; the latter needs reduced denoising compute demand. The paper outlines potential research directions for algorithm-system-chip co-design.
-
Evaluation Layer: Building an End-to-End Performance Metric System for Embodied Intelligence
All optimization depends on correct measurement. Current evaluation of embodied large-model inference still mainly uses the latency and energy metrics of traditional large models and lacks an embodied-specific perspective. Such metrics neither capture the coupling between control frequency and action success rate nor reflect the real-time and energy constraints under perception-action closed-loop conditions. They therefore cannot effectively guide optimization for manipulation tasks. The paper argues that it is urgent to customize measurement methods and metric systems from the embodied perspective, jointly considering control frequency, action success rate, end-to-end latency, and closed-loop energy efficiency. Only when evaluation comes first can subsequent algorithm and system optimizations have clear objectives and comparable benchmarks.
-
Algorithm Layer: Few-Step Generation and Compression Targeting Action Quality
Algorithm-level optimization should also differentiate between the two bottlenecks. For the memory-bound VLA action expert, the main benefits come from compressing denoising steps to a single step and reducing memory access to weights and key-value caches through low-bit quantization. For the compute-bound WAM generation module, the focus should be on reducing or simplifying denoising steps and compressing latent-space dimensions, because the cost grows with raw compute demand.
More importantly, the optimization objective must change. Although diffusion caching and quantization are important for reducing memory footprint and improving speed, attempts to apply them to VLA and WAM remain limited, and most still use general objectives such as minimizing quantization error or reconstruction error. In embodied scenarios, however, minimum error does not equal optimal manipulation. A more reasonable approach is to use action success rate directly as the optimization target and evaluation criterion, making compression and acceleration accountable to actions rather than only to pixels or tensors.
-
System Layer: Specialized Runtimes and Multi-Model Collaboration
Specialized embodied inference runtimes such as vla.cpp and realtime-vla are still in their early stages. Most real robot deployments still use general large-model engines that were not designed for embodied scenarios and lack targeted optimization for control frequency constraints and perception-action closed loops. There remains a clear gap between the acceleration claimed by algorithms and their actual execution efficiency on real CPUs, GPUs, and NPUs. Algorithm-system cross-layer co-design is needed to close this gap.
On the other hand, end-side embodied devices often run more than just a control model. They may also host a general interaction model such as an OMNI-style multimodal model, yet existing work mostly considers single-model execution. How control models and interaction models can be cooperatively scheduled and executed in parallel on limited end-side resources is a key system problem that has not been sufficiently studied.
-
Chip Layer: Embodied Intelligence Accelerators That Unify VLM and DiT Support
Specialized chips for embodied workloads are almost absent. The core difficulty lies in heterogeneous execution requirements. In VLA, the vision-language backbone is compute-bound, while the diffusion action expert is memory-bound and requires multi-step iteration. Their execution modes differ significantly. How to make the same accelerator efficiently support both computing modes and natively support embodied and diffusion-specific operators such as adaptive normalization is a key architectural problem. Overall, work at this layer remains sparse and early, making it the most promising and most urgently needed direction in the full stack.
-
The paper concludes that on-device embodied intelligence is not merely a deployment choice but a system foundation for robots entering the real physical world. The contradiction between the high computational and memory-access overheads of multimodal perception, diffusion action generation, and world-model rolling, on one hand, and the stringent requirements of 10 to 100 Hz control, limited power, and unreliable networks, on the other, cannot be resolved by isolated model compression or hardware acceleration. It requires algorithm-system-chip co-design with action success rate, closed-loop latency, and energy efficiency as joint objectives. As embodied intelligence moves from research prototypes to real robots in homes, factories, transportation, healthcare, and space facilities, the ability to run embodied intelligence efficiently on the device will determine how quickly physical agents become practical, reliable, and widespread.