A new embedded interaction system demonstrates how a low-cost, domestically developed hardware and software stack can enable a humanoid robot to capture human motion through video, interpret body key points, convert those observations into joint-angle commands, and respond in real time. The work combines an HI3518E video acquisition platform running embedded Linux, the HiSilicon Media Process Platform, H.264 encoding, RTP/RTSP streaming, OpenPose-based pose extraction, inverse kinematics, and a FreeRTOS-based humanoid robot controller. The result is a complete human-to-humanoid robot interaction path that moves from camera input to robot motion without relying on a closed proprietary ecosystem.
1. A Humanoid Robot Interaction System Built for Practical Constraints
The project addresses a practical problem in humanoid robot development: how to make a humanoid robot react to human movement in a reliable, low-cost, and easy-to-use way. Instead of focusing only on gait control or balance, the system emphasizes dynamic human-machine interaction. The humanoid robot is expected to observe a person, extract body key points from video, translate those key points into joint-angle data, and then execute a corresponding motion.
This goal is important because a humanoid robot is intended to operate in environments built for people. Its human-like form can make interaction more natural, but it also creates technical demands. The humanoid robot must receive visual information, process it quickly, send control data without excessive delay, and drive multiple joints with sufficient coordination. The project therefore treats the humanoid robot not as an isolated machine but as part of a complete sensing, communication, and control chain.
The system also responds to a wider industrial concern. The work deliberately selects a domestic semiconductor platform and open-source software resources to reduce dependence on foreign proprietary components. HiSilicon is used as the core video-processing hardware, while Linux, OpenPose, FreeRTOS, ORTP, and MP4V2 provide an open and adaptable software foundation. For a humanoid robot platform intended for education, research, service, or low-cost deployment, this combination offers a useful balance between capability and accessibility.
The research identifies four major challenges. The first is adapting a domestic video-processing solution for real-time interaction. The second is maintaining video transmission quality and responsiveness under embedded hardware constraints. The third is extracting human body key points from a live camera stream rather than only from still images or offline video. The fourth is ensuring that the humanoid robot controller can receive and act on commands with real-time responsiveness.
2. Industry Context and the Growing Role of the Humanoid Robot
The global robotics sector has expanded rapidly because robots are increasingly used in manufacturing, healthcare, education, military support, and everyday service. Industrial robots are already common in factories, special-purpose robots serve in medical and defense settings, and service robots are entering homes and public spaces. Among these categories, the humanoid robot attracts particular attention because its appearance and movement pattern are closer to those of a person.
The humanoid robot has a long research history. Early experimental platforms in Japan explored bipedal walking, balance, and human-like motion. Later systems added vision, hearing, and more advanced joint control. International programs have produced well-known humanoid robot platforms that can walk, recognize objects, respond to gestures, and perform complex whole-body actions. These systems have demonstrated the potential of the humanoid robot, but they often rely on expensive hardware, specialized software, or tightly integrated proprietary designs.
In China, humanoid robot research began later but has accelerated through university and industry efforts. Institutions have developed bipedal platforms, soccer-playing humanoid robots, small educational humanoid robots, and larger service-oriented humanoid robot systems. Commercial humanoid robot products have also gained visibility in education, entertainment, and public service. The continuing challenge is not simply to build a humanoid robot that can move, but to build one that can interact with people using affordable, stable, and adaptable technology.
The project documentation cites the broader market context. In 2020, China contributed nearly 40 percent of the global industrial robot market and remained the world’s largest consumer of industrial robots for multiple years. The service robot sector has also grown. The reported global service robot sales and annual growth figures provide a useful reference for the scale of demand.
| Year | Global service robot sales value, hundred million USD | Annual growth rate |
|---|---|---|
| 2013 | 32.2 | 12.10% |
| 2014 | 35.1 | 9.00% |
| 2015 | 41.8 | 19.10% |
| 2016 | 54.0 | 29.20% |
| 2017 | 69.9 | 29.90% |
| 2018 | 92.5 | 32.30% |
| 2019 | 120.7 | 30.50% |
| 2020 | 156.9 | 30.00% |
Within this context, the humanoid robot is not only a research artifact. It is a potential interface between people and intelligent services. A humanoid robot that can see a person, understand basic body motion, and respond through physical movement can support education, remote presence, guided interaction, and research into embodied intelligence. The project’s contribution is to show how such interaction can be assembled from accessible modules rather than from a single closed platform.
3. System Architecture: From Camera to Humanoid Robot Motion
The system is organized into three connected modules: a video acquisition and transmission module, a human key-point processing module, and a humanoid robot control module. The video module is built around the HI3518E platform, an OV9712 image sensor, wireless or wired networking, and embedded Linux. The processing module runs on a PC and uses OpenPose together with inverse kinematics calculations. The control module is based on an STM32F103RCT6 microcontroller inside a humanoid robot and runs FreeRTOS for real-time task handling.
The data flow is direct. The camera captures video. The HI3518E platform processes and encodes the video into H.264. The stream is sent over RTP/RTSP through Ethernet or Wi-Fi. The PC receives the stream and extracts human body key points. The key points are converted into joint-angle values. The resulting commands are sent through Bluetooth or a serial link to the humanoid robot controller. The controller drives the servo motors so that the humanoid robot imitates or reacts to the observed motion.
| Module | Core hardware or software | Function in the humanoid robot interaction chain |
|---|---|---|
| Video acquisition and transmission | HI3518E, OV9712, embedded Linux, HiSilicon MPP, H.264, RTP/RTSP, MT7601U Wi-Fi | Captures video, processes image data, encodes H.264 streams, and transmits video to the PC over wired or wireless networks. |
| Human key-point processing | PC, OpenPose, OpenCV, Caffe, CUDA, inverse kinematics | Extracts body key points from the received video, builds a body-joint model, and converts key-point positions into joint-angle commands. |
| Humanoid robot control | STM32F103RCT6, FreeRTOS, HC-05 Bluetooth, 18 servo motors | Receives angle data, schedules control tasks in real time, and drives the humanoid robot joints to reproduce or respond to human motion. |
This modular design makes the system easier to debug and extend. Each stage can be tested independently before full integration. The video module can be verified with a player. The key-point stage can be verified with recorded or live streams. The humanoid robot control stage can be verified with serial commands. Once each stage is stable, the complete humanoid robot interaction loop can be assembled.
4. Hardware Foundation for Real-Time Video and Control
The hardware design centers on the HI3518E, an ARM9-based system-on-chip with integrated image-processing and video-encoding capabilities. The device is intended for low-power video applications and includes H.264 encoding, image signal processing, DDR memory, Ethernet MAC support, USB, UART, and camera interfaces. The project uses the HI3518EV200 variant, which supports a CPU frequency of up to 540 MHz and integrates a 64 Mb DDR memory. A 16 Mb SPI NOR Flash device stores boot and system images, while an SD card can provide additional user storage.
The camera front end uses the OV9712 image sensor. The sensor is a 1/4-inch CMOS device with one-megapixel resolution and WXGA output. It supports a maximum frame rate of 30 fps at 1280 x 768 and provides a DVP parallel data interface together with an I2C or SCCB control interface. Its internal processing includes black-level calibration, digital gain, and image formatting. These features make it suitable for a compact embedded video node while keeping the overall system cost under control.
| Component | Role | Reported specification |
|---|---|---|
| HI3518EV200 | Core video-processing system-on-chip | ARM9 core up to 540 MHz, integrated 64 Mb DDR, H.264 encoder, ISP, Ethernet MAC, USB, UART, camera interface |
| OV9712 | Image sensor for video acquisition | 1/4-inch CMOS, one-megapixel WXGA, up to 1280 x 768 at 30 fps, DVP output, I2C/SCCB control |
| MT7601U | USB Wi-Fi module | Single-chip Wi-Fi solution with a maximum rate of up to 150 Mbps |
| STM32F103RCT6 | Humanoid robot controller | Cortex-M3 core up to 72 MHz, 48 KB SRAM, 256 KB Flash, I2C/SPI/UART support |
| HC-05 | Bluetooth communication module | Default 9600 baud, full-duplex communication, 3.3 V to 5.0 V operating voltage |
| Servo motors | Humanoid robot joints | 18 servo motors used to simulate human body joints |
The board-level design includes power, clock, reset, storage, and peripheral interface sections. A 12 V DC supply is stepped down to 5 V, 3.3 V, 1.8 V, and 1.1 V rails for different parts of the system. The power section uses MP1494S, MP2162, and TPS7301 devices to provide the required voltage levels. The clock section uses a 24 MHz crystal for the main system clock and a 32.768 kHz crystal for the real-time clock. Reset is handled internally, with a reset signal exported to external flash devices.
Storage is provided by SPI NOR Flash for boot and system images, while DDR2 memory supports runtime operation. Peripheral interfaces include UART for debugging and external communication, USB 2.0 for the Wi-Fi module, Ethernet for wired networking, and a camera interface for the image sensor. These interfaces give the video node the flexibility to operate in both wired and wireless modes, which is important for a humanoid robot interaction system that may be deployed in different environments.
5. Embedded Software Stack and Driver Development
The software foundation is an embedded Linux system built for the ARM architecture. Development takes place in Ubuntu 16.04 LTS with a HiSilicon cross-compilation toolchain. The boot process follows the conventional embedded Linux sequence: a bootloader initializes the board and loads the kernel, the kernel initializes system resources and drivers, and the root file system provides user-space tools and libraries.
The bootloader is based on U-Boot and is adapted for the HI3518EV200 board. The first stage performs low-level initialization in assembly, including exception vectors, CPU mode setup, watchdog handling, clock configuration, and memory-controller initialization. The second stage is written in C and initializes peripherals, loads the kernel and root file system, and passes boot arguments to the kernel. The project documents board-specific changes to memory size, MMU configuration, SPI Flash access, eMMC controller initialization, and boot arguments.
The Linux 3.4 kernel is configured and compiled for the target platform. A default HiSilicon configuration file is used as a starting point, and menu-driven configuration is used to select the required drivers and file systems. The kernel image is built as a uImage and written to the correct flash partition. The documented partition layout reserves space for U-Boot, the kernel, and the root file system. The root file system is built with BusyBox and packaged as a JFFS2 image. Unnecessary directories and files are trimmed to fit the constrained embedded environment.
The HiSilicon Media Process Platform is then deployed. MPP provides a software framework that hides low-level hardware details and exposes APIs for video input, video processing, encoding, decoding, audio input, audio output, and video detection. The platform allows application developers to work at the media-processing level rather than manipulating registers directly. The project copies the required kernel modules and libraries into the root file system, configures the boot scripts to load them automatically, and verifies that the media platform initializes correctly after reboot.
| MPP module | Function |
|---|---|
| VI | Video input capture with cropping, scaling, and multi-channel output. |
| VPSS | Video processing subsystem for noise reduction, enhancement, sharpening, and multi-resolution output. |
| VENC | Video encoding module for H.264, MJPEG, JPEG, and related stream generation. |
| VDA | Video detection and analysis module. |
| VO | Video output module with configurable display output. |
| AI | Audio input capture and audio stream generation. |
Driver development focuses on the camera sensor and the USB Wi-Fi module. The camera sensor is connected through I2C for control and DVP for image data. The Linux I2C subsystem is used to register the sensor device and driver, match them through the bus, and invoke the probe function. The MPP sensor framework then provides callbacks for image signal processing, auto exposure, auto focus, and auto white balance. The sensor driver is compiled into the media component libraries and deployed to the target board.
The Wi-Fi module supports both access-point and station modes. In access-point mode, the board creates a wireless network that a PC or another device can join. In station mode, the board connects to an existing wireless network. The driver source is adapted for the target platform, compiled as a kernel module, and loaded at runtime. Configuration files set the network name, encryption mode, and password. The wireless interface is then brought up and assigned an IP address so that video can be streamed without a physical Ethernet cable.
6. Video Processing, H.264 Encoding, and Real-Time Transport
The video pipeline is implemented through the VI, VPSS, and VENC modules. The VI module receives timing signals from the image sensor, parses the input, and writes image data to memory. The VPSS module performs unified preprocessing and then produces multiple output channels at different resolutions. The VENC module encodes the processed image data into H.264 streams. This pipeline supports the project’s need to generate video at different resolutions for different purposes.
The VI module is configured for the OV9712 sensor with a 1280 x 720 capture region and a 30 fps frame rate. The image signal processor is initialized for auto exposure, auto white balance, and related functions. The video input device and channel are enabled only after their parameters are set. The VPSS module creates a group and multiple channels, binds the VI output to the VPSS input, and produces HD720, VGA, and QVGA outputs. The VENC module creates encoding channels, binds them to the VPSS channels, and starts receiving pictures for encoding.
| Stage | Main configuration and behavior |
|---|---|
| VI | Captures from the OV9712 sensor, sets a 1280 x 720 region, initializes ISP functions, and enables the input device and channel. |
| VPSS | Creates a processing group, binds to VI, and outputs HD720, VGA, and QVGA channels for flexible encoding or display. |
| VENC | Creates H.264 encoding channels, binds to VPSS outputs, and produces compressed streams for network transport or storage. |
H.264 is selected because it offers a high compression ratio, strong network compatibility, and broad player support. The encoded stream uses the familiar layered structure of the video coding layer and the network abstraction layer. The video coding layer handles compression and reconstruction, while the network abstraction layer packages the encoded data for transport. Sequence parameter sets and picture parameter sets carry essential decoding information and must be available before dependent frames can be decoded.
For real-time transmission, the system uses RTP over UDP together with RTSP for session control. RTP carries timestamped media packets and supports packetization of H.264 network abstraction units. RTSP provides the control dialogue between client and server, including options, description, setup, play, and teardown operations. The ORTP library is ported to the target platform to simplify RTP session creation, remote address configuration, payload type selection, and packet transmission. Large H.264 units are fragmented according to the maximum RTP payload size before transmission.
The RTSP server runs on the HI3518E board. It listens for client connections, processes client requests, and enters a sending state when a play request is received. The video stream is then packetized and sent to the client. The PC side can use a standard media player to receive and display the stream. This makes the video output accessible to the pose-processing stage without requiring custom player software.
The system also supports offline storage in MP4 format. The MP4V2 library is ported to the embedded platform, and the encoded H.264 frames are written into an MP4 container. Sequence parameter sets and picture parameter sets are added as required, and video samples are written frame by frame. This storage path is useful when the network is unstable or when recorded video must be reviewed later. It also extends the humanoid robot interaction system beyond live operation by allowing recorded human motion to be used for testing or demonstration.
7. Human Pose Extraction and Humanoid Robot Control
On the PC side, OpenPose is used to extract human body key points from the received video. OpenPose is an open-source pose-estimation project built on deep learning and implemented primarily in C++. It supports multi-person two-dimensional key-point detection and related pose-analysis tasks. The project environment includes OpenCV, Caffe, CUDA, and cuDNN so that the pose model can run efficiently on a suitable GPU.
The extracted key points include the positions of the shoulders, elbows, wrists, and other body joints. These positions are expressed in the camera coordinate frame. To control the humanoid robot, the system must convert these observed positions into joint angles. The project applies the Denavit-Hartenberg modeling method, commonly called the D-H method, to represent the kinematic chain of the human arm. The D-H parameters describe the link length, link twist, link offset, and joint angle for each joint.
| Link | Link length | Link twist | Link offset | Joint angle |
|---|---|---|---|---|
| 1 | 0 | pi divided by 2 | 0 | theta 1 |
| 2 | a2 | 0 | 0 | theta 2 |
| 3 | a3 | 0 | 0 | theta 3 |
Using the measured positions of the upper arm and forearm, the system constructs the transformation from the shoulder frame to the hand frame. The inverse kinematic solution then yields the shoulder rotation, shoulder abduction, and elbow angle. These values become the command data for the humanoid robot. The same principle can be extended to other joints, although the documented implementation focuses on the upper-body motion needed for interaction.
The humanoid robot controller is based on an STM32F103RCT6 microcontroller. The FreeRTOS real-time operating system is ported to the controller to provide task scheduling, inter-task communication, and predictable response. The porting process includes adapting the FreeRTOS configuration, replacing interrupt handlers, and ensuring that the kernel takes over the system tick and context switching. After the operating system is running, the application initializes the clock, interrupts, timers, serial ports, and Bluetooth module.
The control application creates a task that waits for incoming data from the PC. When angle data arrives through Bluetooth or a serial link, the task parses the command and forwards it to the servo motors. The humanoid robot uses 18 servo motors to represent body joints. Each servo receives a command frame with a header, product identifier, module identifier, instruction, parameters, and checksum. The controller must respect the timing and protocol requirements of the servo chain so that the humanoid robot motion remains stable.
The Bluetooth link uses the HC-05 module. The default communication speed is 9600 baud, and the module supports full-duplex communication. Before operation, the PC Bluetooth interface is paired with the humanoid robot module, and the correct serial port is selected in the pose-processing application. The command data is then transmitted wirelessly. A wired serial connection can also be used when a more stable link is required.
8. Validation, Observed Capabilities, and Limitations
The complete system is validated by running the video acquisition platform, the pose-processing application, and the humanoid robot controller together. The camera captures a person’s motion. The HI3518E platform encodes and transmits the video. The PC extracts key points and computes joint angles. The Bluetooth link carries the commands to the humanoid robot. The humanoid robot then moves its joints in response to the observed human motion.
The project reports that the video path can operate over both wired Ethernet and Wi-Fi. The wireless module can work in access-point mode or station mode. The video stream can be viewed in a standard player, and the encoded video can also be stored as an MP4 file. The humanoid robot controller can receive commands and drive the servo motors after FreeRTOS is ported and the control task is created. These results show that the interaction chain is functional from end to end.
The work also identifies clear limitations. The system is distributed across multiple modules, which makes integration less compact than a single-board design. Video latency is affected by network bandwidth and encoding or decoding load. The humanoid robot currently focuses on upper-body interaction because additional sensors such as gravity and ultrasonic sensors are not integrated. OpenPose can also produce unstable key points in some poses, which may cause temporary command errors. These limitations are important because they define the next stage of development.
Future work is expected to move in several directions. A more powerful HiSilicon system-on-chip with richer on-chip resources could support a more integrated design. Network optimization and video-data reduction could improve real-time performance. Additional sensors could enable walking, obstacle avoidance, and broader whole-body interaction. More systematic use of OpenPose and improved pose filtering could reduce recognition failures. Together, these steps would make the humanoid robot more capable in real environments.
9. Why the Work Matters for the Humanoid Robot Ecosystem
The significance of the project lies in its complete interaction path and its use of accessible technology. Many humanoid robot demonstrations focus on a single capability, such as walking or object recognition. This work connects video capture, embedded processing, network transmission, pose estimation, inverse kinematics, wireless command delivery, and real-time joint control into one operational flow. That makes it a useful reference for low-cost humanoid robot interaction research.
The project also contributes to the domestic hardware and software ecosystem. By using the HI3518E platform, the work demonstrates that a locally developed system-on-chip can support a practical video-interaction application. By relying on open-source software such as Linux, OpenPose, FreeRTOS, ORTP, and MP4V2, the design remains adaptable. Developers can modify the video pipeline, change the pose model, adjust the robot protocol, or add new sensors without rebuilding the entire stack from scratch.
For the humanoid robot field, the most important lesson is that interaction is a system-level problem. A humanoid robot cannot respond naturally if video capture is slow, if the network drops packets, if pose estimation is unstable, or if the controller cannot schedule tasks on time. The project shows how each layer must be designed with the others in mind. It also shows that a humanoid robot can be built around affordable modules while still delivering a meaningful interaction experience.
10. Key Takeaways
- The project demonstrates a complete human-to-humanoid robot interaction loop using video capture, H.264 streaming, OpenPose key-point extraction, inverse kinematics, and servo control.
- The HI3518E platform provides a low-cost embedded foundation for video acquisition, processing, encoding, and network transmission.
- Embedded Linux, U-Boot, the Linux kernel, BusyBox root file systems, and the HiSilicon Media Process Platform form a practical software stack for the video node.
- Camera and Wi-Fi driver development allow the humanoid robot interaction system to operate with both wired and wireless connectivity.
- RTP/RTSP streaming and MP4 storage give the system real-time viewing and offline review capabilities.
- OpenPose and D-H inverse kinematics convert human body key points into joint-angle commands for the humanoid robot.
- A FreeRTOS-based controller receives commands through Bluetooth or serial communication and drives 18 servo motors in the humanoid robot.
- The work identifies integration density, video latency, sensor coverage, and pose-estimation stability as the main areas for further improvement.
The project presents a practical path toward a low-cost, open, and adaptable humanoid robot interaction system. It does not claim to solve every problem in humanoid robot control, but it provides an end-to-end reference design that connects domestic video hardware, open-source vision software, real-time control, and physical robot motion. For research and education, that combination is valuable. For the wider humanoid robot ecosystem, it shows how accessible components can be organized into a meaningful interaction platform.