1. Introduction
1.1 Research Background and Significance
The rapid advancement of integrated circuits, 5G communication, and embedded technologies has created unprecedented opportunities for the development of humanoid robot systems. Human-computer interaction, particularly for humanoid robot platforms, has become increasingly important in specific working environments and for special service populations. The demand for high-reliability, low-cost, and user-friendly dynamically interactive humanoid robot systems has grown substantially in recent years. Under the increasingly severe international competition landscape, the localization of industrial chains has become a national development strategy in many countries. Open-source initiatives and system autonomy have emerged as primary objectives for enterprises worldwide.
Hisilicon, as a leading domestic chip manufacturer in China, has played a pivotal role in the country’s chip self-production process. The research and development of its products have effectively broken foreign technological blockades and monopolies, contributing significantly to the development of self-owned industries. This thesis focuses on human-computer interaction, employing the Hisilicon hardware platform based on ARM architecture, combined with Linux software operating systems, various hardware modules, and open-source software resources to achieve real-time response and imitation of human actions by a humanoid robot.
The humanoid robot represents one of the most sophisticated achievements in robotics research, integrating mechanics, electronics, computer science, artificial intelligence, and many other disciplines. Unlike traditional industrial robots confined to structured environments, humanoid robot systems are designed to operate in human-centric environments, making them inherently more versatile and adaptable. The dynamic interaction capabilities of a humanoid robot are essential for applications ranging from healthcare assistance to educational tutoring and entertainment.
1.2 Research Status
The concept of robots was formally introduced in the 1960s, and since then, developed countries have conducted extensive experiments and research in this field. The United States and Japan, as leading nations in robotics, have consistently pushed the boundaries of what humanoid robot systems can achieve. Japan’s Waseda University developed WABOT series humanoid robots with artificial vision, hearing, and mechanical devices in the 1970s. Honda’s ASIMO, introduced in 2000, represented a milestone in humanoid robot development, capable of recognizing moving objects, responding to gestures and postures, and performing path planning based on real-time information.
In the United States, Boston Dynamics’ Atlas series has demonstrated remarkable capabilities in dynamic locomotion and complex maneuvers. The latest Atlas can perform activities such as standing long jumps, backflips, and three consecutive jumps, showcasing advanced balance control and learning capabilities. Other significant contributions include France’s Aldebaran Robotics’ NAO robot, which has been widely adopted in education and research due to its powerful processing capabilities and comprehensive development resources.
China’s research on humanoid robot systems began in the mid-1980s, relatively late compared to Western countries. However, with strong support from national policies, Chinese universities and research institutions have achieved significant breakthroughs. The National University of Defense Technology developed China’s first humanoid robot “Forerunner” in 2000, standing 140cm tall and weighing 10kg with 17 degrees of freedom. Beijing Institute of Technology successfully developed the BHR series of robots, incorporating balance control, pressure sensing, and voice/visual interaction capabilities.
In the commercial sector, UBTECH Robotics has emerged as a major player in the humanoid robot market. Their Walker X robot, released in 2021, integrates six major AI technologies, including high-performance servo joints, multi-dimensional force sensing, multi-stereo vision, and comprehensive perception systems, enabling smooth and safe interactions in various real-world scenarios.
1.3 Technical Challenges
Several significant challenges must be addressed in the development of this humanoid robot interaction system. The first challenge involves adapting and developing a domestic solution suitable for audio-video transmission. Given the urgent need to develop IC industries, solutions based on self-developed chips are becoming increasingly important. This research employs Hisilicon HI3518E as the core processor for the video transmission system.
The second challenge is solving the real-time performance issues in video transmission while improving and optimizing video coding and decoding capabilities. The stable connection between video servers and clients, as well as efficient encoding/decoding performance, directly impacts the real-time nature of video transmission. The third challenge involves secondary development based on OpenPose for keypoint extraction from video data. While OpenPose demonstrates good performance in extracting joint point data from images and offline videos, its effectiveness for analyzing video streams from IP cameras requires further development.
The fourth challenge concerns the control of the humanoid robot itself. The internal system of the robot must undergo migration and development to ensure real-time interaction capabilities. This requires the implementation of appropriate real-time operating systems and control protocols for the humanoid robot platform.
2. System Requirements and Overall Design
2.1 System Requirement Analysis
Before the overall system development, it is essential to carefully analyze the actual requirements and practical issues facing the implementation. The goal of this research is to develop an independently controllable humanoid robot system capable of dynamic interaction. To achieve this, the system requirements are analyzed through decomposition, breaking down complex problems into manageable components.
The humanoid robot system must possess hearing, vision, and comprehensive controlled capabilities, along with good portability. For basic hardware selection, products from domestic manufacturers should be chosen to prevent foreign technology monopoly. This selection should also feature low cost, stable usage, and expandability. Available domestic hardware manufacturers include Hisilicon, MediaTek, and GigaDevice. For basic software, open-source operating systems suitable for cross-platform adaptation should be used, such as Linux, HarmonyOS, FreeRTOS, and RT-Thread. For upper-layer applications, either independent development or secondary development using open-source projects like OpenCV and OpenPose should be adopted.
2.2 Hardware Platform Design
Based on functional requirements, comprehensive consideration of development costs, implementability, and technical challenges, the entire system is divided into three modules: video transmission module, joint point data extraction module, and humanoid robot control module. The hardware characteristics of each module are summarized in Table 1.
| Module | Core Component | Key Specifications |
|---|---|---|
| Video Capture | Hisilicon HI3518E | ARM926@540MHz, 64Mb DDR, H.264 Encoder |
| Image Sensor | OV9712 | 1/4 inch, 1 megapixel, 1280×768@30fps |
| Wireless Module | MT7601U | 150Mbps, USB Interface |
| Data Processing | Dell G3 3590 | Intel Core i7, GTX 1660Ti, 16GB RAM |
| Robot Control | STM32F103RCT6 | Cortex-M3, 72MHz, 256KB Flash |
| Communication | HC-05 Bluetooth | 3.3V-5.0V, Default 9600 baud |
The HI3518E is a system-on-chip developed by Hisilicon, integrating an ARM9 processor core with a maximum clock speed of 540MHz. The chip employs a heterogeneous architecture, combining a CPU for instruction processing and a DSP for image computation operations. It incorporates 64Mb of DDR memory and an Ethernet MAC, while supporting external SPI Flash, SD card expansion, USB HOST interfaces, Camera Sensor interfaces (MIPI, LVDS, HISPI, and DVP data channels plus I2C control channels), and UART communication.
The humanoid robot platform utilized in this research is based on the STM32F103RCT6 microcontroller with Cortex-M3 core, boasting a maximum frequency of 72MHz. The system includes 48KB SRAM, 256KB FLASH, and various communication interfaces. The robot integrates 18 servo motors controlled by STM8S150K microprocessors. Communication follows a specific frame format as shown in Table 2.
| Parameter | Description |
|---|---|
| Header | Packet frame header, fixed at 0xff |
| Product ID | Digital servo motor product identifier, fixed at 0xb0 |
| Module ID | Identifier for the servo motor ID number |
| Instruction | Instruction name identifier |
| Para1~3 | Transmission parameters |
| Check Sum | Checksum bit |
2.3 Software Platform Design
The system development is primarily conducted in a Linux environment using Ubuntu 16.04 LTS. The video transmission system runs on a customized Linux kernel version 3.4.1, with the Hisilicon MPP media processing platform deployed for video handling. The humanoid robot platform runs the FreeRTOS kernel to enable real-time feedback capabilities. OpenPose is employed for secondary development of human body keypoint extraction.
The Linux operating system manages system resources, provides driver management for underlying hardware, and supplies call interfaces for upper-layer application development. The system comprises three main components: the bootloader, kernel, and root filesystem. The bootloader initializes basic hardware and loads the kernel image. The kernel maintains and serves the entire operating system, managing system scheduling, data storage, and peripheral device operations. The root filesystem provides the mounted file system after OS kernel startup, presenting kernel information to developers in a virtual mapping manner.
Hisilicon provides the MPP software platform to reduce development cycles. This platform encapsulates hardware-level media processing operations, offering API interfaces for audio-video capture, encoding/decoding, and custom output adjustments. OpenPose, developed by Carnegie Mellon University in 2017, uses convolutional neural networks (Caffe framework) for human keypoint recognition, demonstrating excellent robustness in practical engineering tests. FreeRTOS, a lightweight real-time operating system kernel featuring portability, customizability, and flexible scheduling policies, serves as the real-time control foundation for the humanoid robot.
2.4 System Framework
The overall system achieves real-time tracking and imitation control of human actions by the humanoid robot. The system architecture is illustrated in Figure. The HI3518E-based video capture system acquires video using the OV9712 sensor and transmits encoded streams via WIFI or Ethernet using RTSP protocol to a PC. On the PC side, OpenPose extracts human keypoint information, applies mathematical modeling based on inverse kinematics to compute angle data, and transmits commands to the humanoid robot via Bluetooth, enabling real-time interactive control of the humanoid robot.
3. Underlying Design of Video Transmission System
3.1 Basic Hardware System Design
The hardware design of the video transmission module encompasses several critical subsystems: power supply, clock generation, reset circuitry, storage, network interface, USB serial interface, and sensor interface. According to the chip datasheet and reference designs, the system employs MP1494S, MP2162, and TPS7301 voltage regulators to step down the 12V DC input. The power distribution structure is shown in Table 3.
| Component | Input Voltage | Output Voltage | Regulator IC |
|---|---|---|---|
| System Main Supply | 12V | 5V | MP1494S |
| Peripheral Supply | 5V | 3.3V | MP2162 |
| Core/DDR Supply | 3.3V | 1.8V/1.1V | TPS7301 |
For the MP1494S regulator, the output voltage can be determined using:
$$V_{out} = 0.807 \times \left(1 + \frac{R_1}{R_2}\right) \quad \text{(MP1494S)}$$
Similarly, for the MP2162 regulator:
$$V_{out} = 0.6 \times \left(1 + \frac{R_1}{R_2}\right) \quad \text{(MP2162)}$$
And for the TPS7301 regulator:
$$V_{out} = V_{ref} \times \left(1 + \frac{R_1}{R_2}\right) \quad \text{where} \; V_{ref} = 1.2246V \quad \text{(TPS7301)}$$
The clock system is built around the HI3518EV200 chip’s internal output, using feedback amplifier circuits with external crystals to form the clock generation module. A 24MHz crystal oscillator provides the main system clock, while a 32.768kHz crystal serves the internal RTC module. The reset system employs internal reset generation, where the chip’s POR signal is pulled low for a certain clock period after power-up to complete the reset operation.
For storage, the system uses a 16MB SPI NOR Flash (MX25L12835) connected to the FMC controller. This Flash stores boot code and system files. The chip’s built-in DDR2 memory handles runtime code execution. Peripheral interfaces include UART for debugging, USB 2.0 Host for WIFI module expansion, Ethernet PHY for wired network connectivity, and a DVP interface for the OV9712 image sensor.
3.2 Underlying Software System Design
Due to resource constraints in embedded systems, development follows a host-target approach. The cross-compilation environment is established on Ubuntu 16.04 with the arm-hisiv300-linux-uclibcgnueabi-gcc toolchain. The bootloader (U-Boot 2010.06) is customized for the HI3518EV200 platform, involving board-specific configuration, memory map adjustments, and driver modifications.
The Linux 3.4 kernel is configured and cross-compiled for the target platform. The kernel configuration process employs the menuconfig tool with the hi3518ev200_full_defconfig as the base configuration. The Flash partition layout is presented in Table 4.
| Partition | Size | Start Address | End Address |
|---|---|---|---|
| U-Boot | 1MB | 0x00000000 | 0x00100000 |
| Kernel | 3MB | 0x00100000 | 0x00400000 |
| Rootfs | 12MB | 0x00400000 | 0x01000000 |
The root filesystem is created using BusyBox, which is compiled with the cross toolchain and installed into a directory structure containing essential system directories (etc, dev, lib, tmp, var, mnt, home, proc). The JFFS2 filesystem image is generated using the mkfs.jffs2 tool. After successfully booting with the customized bootloader, kernel, and filesystem, the Hisilicon MPP platform is deployed by installing kernel modules (.ko files) and userspace libraries into the target system.
3.3 Device Driver Development
In embedded systems, device drivers serve as the crucial bridge between applications and hardware. Linux categorizes devices into three types: character devices, block devices, and network devices. The OV9712 camera sensor connects to the HI3518E via I2C for control and DVP for data. The I2C driver framework involves bus management, adapter abstraction, and device/driver matching. The sensor driver development follows Hisilicon’s ISP driver framework, registering callbacks for AE (Auto Exposure), AF (Auto Focus), and AWB (Auto White Balance) operations.
For the MT7601U USB WIFI module, driver porting addresses both AP (Access Point) and STA (Station) modes. The AP mode enables the video transmission system to act as a wireless hotspot, while STA mode allows it to connect to existing wireless networks. Configuration parameters for the access point include SSID, authentication mode, encryption type, and pre-shared key. After loading the driver with insmod and configuring the network interface, the WIFI module successfully provides wireless connectivity for video transmission.
4. Upper-Layer Application Development
4.1 Video Processing and Codec Design
The VI (Video Input) module of the HI3518E captures video data from the OV9712 sensor. The module supports various input timing standards including DC, BT601, BT656, and MIPI protocols. Initialization of the MPP system involves setting up the VB_CONF_S structure with appropriate video buffer pool parameters. The capture format is configured based on the sensor’s native resolution of 1280×720 at 30fps with BGGR pixel layout.
After capturing, the VI module can output to the VPSS (Video Process Sub-System) for further processing. The VPSS module performs operations including noise reduction, deinterlacing, scaling, and sharpening. For this system, VPSS is configured to output three channels with different resolutions: HD720 (1280×720), VGA (640×480), and QVGA (320×240), as described in Table 5.
| Channel | Resolution | Pixel Format | Application |
|---|---|---|---|
| Channel 0 | 1280×720 | YUV420 Semi-Planar | High-definition recording |
| Channel 1 | 640×480 | YUV420 Semi-Planar | Standard network transmission |
| Channel 2 | 320×240 | YUV420 Semi-Planar | Mobile preview |
The VENC (Video Encode) module receives images from VPSS channels and performs H.264 encoding. Each encoding channel operates independently with configurable bit rate control. The encoding process employs profile/level configuration, and the stream is output in NAL unit format. The relationship between VPSS channels and VENC channels involves a binding mechanism using the HI_MPI_SYS_Bind function.
4.2 Real-time Transmission Based on RTP/RTSP
For real-time video transmission, the H.264 elementary stream requires appropriate packetization. The H.264 standard employs a layered structure comprising the Video Coding Layer (VCL) and the Network Abstraction Layer (NAL). The VCL handles video content encoding, while the NAL formats data for network transport. Each NALU header contains a forbidden bit, importance indicator bits, and NALU type bits. Key NALU types are summarized in Table 6.
| nal_unit_type | NAL Type | Content |
|---|---|---|
| 1 | Non-IDR Slice | P-frames and non-IDR I-slices |
| 5 | IDR Slice | Instantaneous decoder refresh frame |
| 6 | SEI | Supplemental enhancement information |
| 7 | SPS | Sequence parameter set |
| 8 | PPS | Picture parameter set |
RTP (Real-time Transport Protocol) provides end-to-end delivery services for audio and video data. Each RTP packet contains a header with version, padding flag, extension flag, CSRC count, marker, payload type, sequence number, timestamp, and synchronization source identifier. The payload type must be configured for H.264, typically using dynamic payload type 96.
RTSP (Real Time Streaming Protocol) serves as an application-layer protocol for controlling streaming media sessions. It initiates and directs delivery of streaming multimedia from media servers to clients. The typical RTSP interaction sequence includes OPTIONS, DESCRIBE, SETUP, PLAY, and TEARDOWN methods. During the PLAY state, media data flows via RTP/UDP from server to client.
For UDP-based RTP transmission, large H.264 access units must be fragmented into smaller RTP packets. The fragmentation algorithm is described below. When the NAL unit size exceeds MAX_RTP_PKT_LENGTH (typically 1400 bytes), the data is fragmented using the FU-A (Fragmentation Unit A) mechanism. The algorithm for RTP packetization can be expressed as:
$$N = \left\lceil \frac{L – 1}{M} \right\rceil$$
where L represents the total NAL unit length, M is the maximum RTP payload size, and N is the number of fragments needed. Each fragment carries a 2-byte FU indicator and FU header containing the original NAL type and fragment start/end markers.
The ORTP library was ported to the target system to provide RTP functionality. Configuration involved adding H.264 payload type support through the rtp_profile_set_payload function. For Internet streaming, TCP-based transmission was also implemented for environments where UDP is blocked. The RTSP server implementation properly handles client requests through dedicated threads.
4.3 Offline Video Storage
For reliable offline storage, MP4 file format support was implemented using the MP4V2 library. MP4 organizes data in boxes (atoms) with a tree structure. Key boxes include ftyp (file type), moov (movie metadata), and mdat (media data). The MP4V2 library was cross-compiled for ARM and integrated into the video encoding application.
The storage implementation creates an MP4 file, adds an H.264 video track, configures the video profile level, and writes SPS/PPS parameter sets. Individual video frames are written using MP4WriteSample. This provides users with the capability to store video streams locally when network transmission is unavailable or when offline analysis of the humanoid robot interaction is required.
5. Human Keypoint Processing and Robot Control System Development
5.1 Human Keypoint Information Processing
OpenPose, the open-source project based on deep learning, enables real-time multi-person 2D pose estimation. Its deployment requires CUDA, cuDNN, Caffe, and OpenCV dependencies. After configuring these prerequisites, the OpenPose framework was built and integrated with the video transmission system. The system extracts keypoint information representing anatomical landmarks, which are then converted to angle data using inverse kinematics calculations.
For the humanoid robot control application, the relationship between camera coordinate space and robot actuator space must be established. The essential arm model is shown in. This modeling approach utilizes the D-H (Denavit-Hartenberg) parameter method to simplify the kinematic analysis of the humanoid robot.
According to the D-H convention, the transformation between consecutive joint coordinate frames is given by:
$$T_i^{i-1} = \begin{bmatrix} \cos\theta_i & -\sin\theta_i\cos\alpha_i & \sin\theta_i\sin\alpha_i & a_i\cos\theta_i \\ \sin\theta_i & \cos\theta_i\cos\alpha_i & -\cos\theta_i\sin\alpha_i & a_i\sin\theta_i \\ 0 & \sin\alpha_i & \cos\alpha_i & d_i \\ 0 & 0 & 0 & 1 \end{bmatrix}$$
The D-H parameters for the left arm are summarized in Table 7.
| Link | ai-1 | αi-1 | di | θi |
|---|---|---|---|---|
| 1 | 0 | π/2 | 0 | θ1 |
| 2 | a2 | 0 | 0 | θ2 |
| 3 | a3 | 0 | 0 | θ3 |
The overall transformation from the base frame to the end-effector frame is obtained by chain multiplication:
$$T_3^0 = T_1^0 \times T_2^1 \times T_3^2$$
Solving the inverse kinematics problem, the target joint angles for the humanoid robot can be derived from the known end-effector position obtained through OpenPose keypoint detection:
$$\theta_1 = \text{atan2}(p_y, p_x)$$
where px and py are the x and y coordinates of the wrist relative to the shoulder. Subsequent joint angles are calculated from the remaining position components, ultimately providing the servo motor commands for the humanoid robot.
5.2 Robot Control System Development
FreeRTOS, a lightweight real-time operating system kernel, was ported to the humanoid robot platform. The porting process involved downloading the kernel source, creating a project structure in Keil, adding appropriate port files for the ARM Cortex-M3 architecture, and configuring the FreeRTOSConfig.h header file for interrupt handling and system tick configuration.
After establishing the operating system foundation, application development focused on system initialization and task creation. The system clock, interrupts, timers, serial ports, and Bluetooth communication modules were initialized. A dedicated task was created to receive command data and transmit it to servo controllers. Communication parameters include:
| Parameter | Value | Description |
|---|---|---|
| Baud rate | 9600 | Default Bluetooth UART rate |
| Data bits | 8 | Standard UART data width |
| Stop bits | 1 | Standard UART stop width |
| Flow control | None | No hardware flow control |
| Operating voltage | 3.3V-5.0V | HC-05 module supply range |
The complete system realizes dynamic interaction between a human operator and the humanoid robot. Video captured by the OV9712 sensor is processed by the HI3518E and transmitted to the PC via RTSP. OpenPose extracts human keypoints from the video stream, the inverse kinematics module converts keypoint positions to joint angles, and the resulting commands are transmitted via Bluetooth to the humanoid robot. The FreeRTOS-based control system enables immediate response to incoming commands, achieving real-time humanoid robot interaction.
6. Conclusion and Future Work
This research successfully implemented a humanoid robot dynamic interaction system based on the domestic Hisilicon HI3518E processor. The complete system integrates a video transmission subsystem, human keypoint detection subsystem, and robot control subsystem, providing a full solution for real-time humanoid robot imitation of human actions. The main contributions include the design of a stable hardware platform, the porting of embedded Linux with device drivers, the development of real-time video streaming based on RTP/RTSP, the implementation of MP4 offline storage, and the successful deployment of OpenPose-based body tracking with inverse kinematics control for the humanoid robot.
The system demonstrates that the HI3518E-based solution offers a low-cost, reliable platform for humanoid robot interaction applications. The use of open-source components throughout the development process significantly reduced design cycles while maintaining system reliability and expandability. Furthermore, the complete reliance on domestic hardware and open-source software demonstrates the feasibility of indigenously developed interactive humanoid robot systems.
Future work could focus on several improvements. First, integrating all subsystems into a unified hardware platform using a more capable Hisilicon system-on-chip would significantly improve system integration and reduce latency. Second, enhancing network bandwidth and developing optimized encoding strategies would improve real-time video transmission quality. Third, incorporating additional sensors such as inertial measurement units and ultrasonic distance sensors into the humanoid robot would enable walking and obstacle avoidance capabilities, expanding the interaction range to full-body applications.
The current system design effectively addresses the core requirements of humanoid robot dynamic interaction within resource constraints while remaining extensible for future enhancements. The methodology presented here provides valuable insights for developing indigenous human-robot interaction solutions and contributes to the growing body of knowledge in humanoid robot research.