A master’s research project at Shantou University has produced a complete, domestically oriented humanoid robot interaction system built around the Hisi HI3518E embedded platform. The work, carried out by Kang Kai under the supervision of Li Xutao in the College of Engineering, focuses on a practical problem that sits at the intersection of embedded Linux, video streaming, computer vision, and real-time robot control: enabling a humanoid robot to capture human motion from video and respond or imitate it with low latency. The project describes a three-part architecture that combines an HI3518E-based video transmission system, an OpenPose-based human keypoint processing pipeline on a PC, and a FreeRTOS-controlled humanoid robot driven by 18 servos. The result is a low-cost, extensible, and largely self-reliant solution for dynamic interaction between a humanoid robot and a human operator.
The broader context is the rapid development of integrated circuits, 5G communications, embedded systems, and intelligent service machines. Within that landscape, the humanoid robot has become a particularly important category because its body plan is compatible with human environments, tools, and social expectations. A humanoid robot that can see, interpret, and respond to human movement has potential value in education, service, assistance, research, and specialized work settings. At the same time, international competition and supply-chain pressures have made domestic semiconductor and software ecosystems a strategic priority. Hisi, as a leading Chinese chip vendor, has played a significant role in this effort. The HI3518E, an ARM-based system-on-chip with integrated image processing and H.264 encoding capability, is positioned as a low-power, highly integrated core for video applications. Using it as the foundation for a humanoid robot interaction system therefore carries both technical and strategic significance.
- Why Humanoid Robot Interaction Matters in a Competitive Technology Era
The research begins from a clear premise: a humanoid robot should not be limited to preprogrammed gait cycles or remote-controlled gestures. It should be able to interact. Human-computer interaction is a central research topic in robotics, and as image processing, embedded computing, and sensor technology improve, machines can combine vision, hearing, touch, and motion to serve people more naturally. Compared with conventional control through a mouse, keyboard, or touchscreen, a humanoid robot that can see human movement and react in real time becomes more useful in environments where natural communication is required.
The project notes that global interest in service robots has risen sharply. From 2013 to 2020, worldwide service robot sales increased from 32.2 hundred million USD to 156.9 hundred million USD, with annual growth rates fluctuating between 9.00 percent and 32.30 percent. This growth reflects both commercial demand and technical progress. Industrial robots remain dominant in manufacturing, special robots serve medical and military roles, and service robots increasingly address daily human needs. The humanoid robot is a special case within service robotics because its form factor allows it to operate in spaces designed for people. That compatibility makes the humanoid robot a natural platform for dynamic interaction.
Year Global Service Robot Sales (100 million USD) Annual Growth Rate 2013 32.2 12.10% 2014 35.1 9.00% 2015 41.8 19.10% 2016 54.0 29.20% 2017 69.9 29.90% 2018 92.5 32.30% 2019 120.7 30.50% 2020 156.9 30.00% The project also emphasizes that China’s robot industry has grown quickly. Even under the influence of the pandemic, China contributed nearly 40 percent of the global industrial robot market in 2020 and remained the world’s largest consumer of robots for multiple years. However, international complications have affected integrated circuit and related supply chains, slowing growth in some robotics sectors and exposing core technology gaps. The ability to develop domestic solutions is therefore not only a commercial question but also a strategic one. A humanoid robot built on a domestic chip platform and open-source software components can reduce dependence on foreign monopolies while still delivering useful interaction capabilities.
- Global and Domestic Research Landscape for the Humanoid Robot
International research on the humanoid robot stretches back decades. Early work in Japan produced WABOT and later the WABIAN series, which explored bipedal walking, human-like motion, and increasingly sophisticated control architectures. Honda’s ASIMO became a landmark humanoid robot, capable of walking, responding to gestures, recognizing moving objects, and planning paths based on visual and audio information. Japan’s National Institute of Advanced Industrial Science and Technology developed the HRP series, including HRP-2, HRP-4C, and HRP-5P. These humanoid robot platforms explored whole-body motion planning, expressive behavior, environmental perception, and learning. HRP-5P, for example, used vision and learning methods to detect objects under imperfect lighting and was designed with many degrees of freedom to mimic human actions.
In the United States, Boston Dynamics and DARPA produced the Atlas series, a humanoid robot known for dynamic locomotion, jumping, and complex athletic movement. Atlas uses hydraulic actuation, onboard sensing, stereo cameras, and laser ranging to maintain balance and navigate difficult environments. Other nations have contributed platforms such as Johnnie, REEM-E, iCub, Witt, and the KHR series. France’s NAO humanoid robot became widely used in education and research because of its compact size, sensor suite, and open software environment.
China started later but has accelerated. The “Xianxingzhe” humanoid robot developed by the National University of Defense Technology in 2000 had 17 degrees of freedom and basic walking and speech functions. Beijing Institute of Technology produced the BHR series, including BHR-1, BHR-2, BHR-4, and BHR-5, which explored balance control, pressure sensing, speech, vision, and complex motion. Harbin Institute of Technology developed the HIT humanoid soccer robot, and Tsinghua University produced the small THBIP-II platform. In recent years, commercial companies such as UBTECH have advanced both small educational humanoid robot platforms and larger service humanoid robot platforms. The Alpha series and Walker series demonstrate how the humanoid robot is moving from laboratory demonstrations toward practical service and interaction scenarios.
Despite this progress, the project identifies four major challenges. First, a domestic video transmission solution suitable for embedded robotics is needed. Second, real-time video streaming must be stable and efficient, which places demands on encoding, network protocols, and client-server synchronization. Third, OpenPose-based keypoint extraction must be adapted from still images and offline video to live IP camera streams. Fourth, the humanoid robot itself must run a real-time control system that can receive processed motion commands and drive its servos without unacceptable delay. These challenges define the engineering scope of the work.
- System Requirements and Overall Architecture
The system requirements are driven by the goal of dynamic interaction. A humanoid robot used in a realistic environment should have vision, hearing, controlled motion, and portability. The design should avoid foreign technology lock-in where possible, use low-cost hardware, remain stable, and support future expansion. The project therefore selects domestic or widely available components and open-source software. The architecture is divided into three modules: a video transmission module based on the Hisi HI3518E, a human keypoint processing module running on a Dell G3 laptop, and a humanoid robot control module based on an STM32F103RCT6 microcontroller. Peripheral modules include a camera sensor, Wi-Fi, and Bluetooth.
Module Core Hardware or Software Primary Function Video transmission Hisi HI3518E, OV9712, MT7601U Wi-Fi Capture, process, encode, stream, and store video Human keypoint processing Dell G3 laptop, OpenPose, OpenCV, Caffe, CUDA Extract human joint information from video and convert it to angle data Humanoid robot control STM32F103RCT6, FreeRTOS, 18 servos, HC-05 Bluetooth Receive commands and drive the humanoid robot in real time The HI3518E is an ARM9-based system-on-chip with a maximum clock of 540 MHz. It integrates a CPU for instruction processing and a DSP for image operations, along with 64 Mb DDR memory and an Ethernet MAC. External options include 16 Mb SPI Flash, SD card expansion, Ethernet PHY, USB host for Wi-Fi, camera sensor interfaces such as MIPI, LVDS, HISPI, and DVP with I2C control, and UART ports. Its image subsystem includes a TDE drawing module, an IVE hardware acceleration module, a VGS video graphics subsystem, an ISP image signal processor, and an H.264 BP/MP/HP encoder. This combination makes the HI3518E well suited to a compact humanoid robot vision front end.
The PC used for keypoint processing is a Dell G3 3590 with an Intel Core i7 six-core twelve-thread processor, 16 GB RAM, a 1 TB solid-state drive, and an NVIDIA GeForce GTX 1660 Ti GPU. The humanoid robot is the Young PI platform from Handgun Technology (Shenzhen). It uses an STM32F103RCT6 with an ARM Cortex-M3 core at up to 72 MHz, 48 KB SRAM, 256 KB Flash, and I2C, SPI, and UART buses. It includes an HC-05 Bluetooth module and 18 digital servos. Each servo is controlled by an STM8S150K microcontroller and accepts commands through a serial protocol with a header, product ID, module ID, instruction, parameters, and checksum. The camera sensor is an OV9712, a 1/4-inch CMOS WXGA sensor with OmniPixel3-HS technology, capable of 1280 × 768 at 30 fps and communicating through I2C and DVP. The Wi-Fi module is an MT7601U single-chip device supporting up to 150 Mbps over USB.
The software stack includes Ubuntu 16.04 LTS as the development host, Linux 3.4.1 for the embedded target, the Hisi MPP media processing platform, OpenPose for human pose estimation, and FreeRTOS for real-time robot control. The overall data flow is straightforward: the HI3518E captures video, encodes it as H.264, and streams it over Wi-Fi or Ethernet using RTP/RTSP. The PC receives the stream, extracts human joint positions with OpenPose, converts those positions into joint angles using inverse kinematics, and sends commands over Bluetooth or serial to the humanoid robot. The robot’s STM32 main controller, running FreeRTOS, receives the commands and drives the 18 servos. The system therefore creates a closed loop from human motion to humanoid robot motion.
- Embedded Video Transmission Platform: Hardware Foundation
The hardware design of the video transmission module follows a minimal-system approach. The power module converts a 12 V DC input into the rails required by the board. An MP1494S synchronous step-down converter produces 5 V and 3.3 V rails. An MP2162 converter produces additional regulated voltage for sensitive components. A TPS7301 device generates 1.8 V for DDR and SDIO sections. The core logic requires 1.1 V. The design supplies 5 V to USB and key modules, 3.3 V to network, TF card, SPI Flash, sensor, and LED modules, 1.8 V to DDR and SDIO sections, and 1.1 V to the core module.
Voltage Rail Primary Loads Function 12 V DC input Board-level power input 5 V USB, key module Peripheral power and user input 3.3 V Network, TF card, SPI Flash, sensor, LED Main I/O and storage peripheral power 1.8 V DDR, SDIO Memory and SDIO interface power 1.1 V Core module Processor core power The clock system uses a 24 MHz crystal oscillator for the main system and a 32.768 kHz crystal for the real-time clock. The 24 MHz clock supports the CPU and peripheral timing, while the RTC clock maintains timekeeping. Reset is handled internally: after power is applied, the chip pulls the power-on reset signal low for a defined number of clock cycles and also drives an external reset output for the boot Flash device. Storage includes a 16 MB SPI NOR Flash, the MX25L12835, connected to the chip’s SPI Flash controller. The HI3518E also contains internal DDR2 memory, so the hardware design focuses mainly on power integrity for that section rather than on external DDR routing.
Peripheral interfaces include UART, USB 2.0, Ethernet, camera sensor, and USB Wi-Fi. The HI3518E provides three UART modules. UART0 is used for debugging, while UART1 and UART2 can support pan-tilt control, external devices, or alarm handling. The USB 2.0 host controller supports high-speed, full-speed, and low-speed modes, as well as control, bulk, isochronous, and interrupt transfers. It is used to connect the MT7601U Wi-Fi module. The Ethernet interface provides wired network communication for debugging and video streaming. The camera interface supports the OV9712 sensor through DVP data lines and I2C control. The OV9712 outputs 10-bit YUV data and uses signals such as XCLK, PCLK, HREF, VSYNC, SIOD, and SIOC. The MT7601U Wi-Fi module connects through USB differential lines and provides wireless networking in either access point or station mode.
- Software Bring-Up: Bootloader, Kernel, Root Filesystem, and MPP
Embedded development for the HI3518E uses a host-target model. Ubuntu 16.04 is installed in a virtual machine on the Windows host, and the Hisi cross-compilation toolchain is installed. The SDK package is extracted, and the script `cross.install.v300` installs the `arm-hisiv300-linux-uclibcgnueabi-gcc` toolchain. The environment variable is updated so that the toolchain is available from the command line. NFS and TFTP tools are configured to simplify file transfer between the host and the target board.
The bootloader is U-Boot 2010.06, adapted for the HI3518EV200 board. The porting process creates a board directory for the HI3518EV200, copies a reference configuration, and builds a `u-boot.bin` image. Initial attempts reveal that the board cannot boot correctly, so the startup code disables PMIC initialization, adjusts memory size and MMU settings, modifies memory address macros, initializes SPI Flash and eMMC controllers, updates relocation addresses, and corrects the boot arguments passed to the kernel. After these changes, the bootloader runs successfully and can be observed over a serial console.
The kernel is Linux 3.4.1. The configuration file `hi3518ev200_full_defconfig` is copied to `.config`, and `make menuconfig` is used to select required modules. The kernel image is built with `make uImage`. An initial boot attempt fails because the kernel partition does not match the U-Boot partition layout. The corrected partition table assigns 1 MB to U-Boot, 3 MB to the kernel, and 12 MB to the root filesystem. The kernel command line specifies 64 MB DDR, with 32 MB reserved for the operating system and another region for the media memory zone used by the MPP. The serial console runs at 115200 baud, and the root filesystem is stored in SPI Flash as JFFS2.
Partition Size Start Address End Address U-Boot 1 MB 0x00000000 0x00100000 Kernel 3 MB 0x00100000 0x00400000 Root filesystem 12 MB 0x00400000 0x01000000 The root filesystem is built with BusyBox. The configuration is customized for the target, with static or dynamic linking choices made according to resource constraints. The installation produces the standard `bin`, `sbin`, `usr`, and `linuxrc` entries. Additional directories such as `etc`, `dev`, `lib`, `tmp`, `var`, `mnt`, `home`, and `proc` are created. The JFFS2 image is generated with `mkfs.jffs2` and written to the correct Flash partition. After reboot, the kernel successfully mounts the root filesystem. The network interface is then enabled manually or through startup scripts.
The Hisi MPP platform is deployed after the base system is working. MPP provides a media processing framework that hides low-level driver details and exposes APIs for video input, processing, encoding, decoding, audio, and detection. The framework includes modules such as VI for video input, VPSS for video processing, VENC for video encoding, VDEC for decoding, VDA for detection, VO for output, and AI for audio input. For this project, the key modules are VI, VPSS, and VENC. The kernel modules and libraries from the MPP directory are copied into the root filesystem, and the `load3518e` script is added to the startup profile so that the media platform is initialized automatically. After reboot, MPP loads successfully and the video pipeline is ready for application development.
Device drivers are also developed for the camera sensor and USB Wi-Fi. The OV9712 sensor uses the I2C bus for control. The Linux I2C subsystem maintains separate driver and device lists and matches them through the bus core. The sensor driver registers callbacks for ISP, auto exposure, auto focus, and auto white balance. The implementation follows the MPP sensor framework and the sensor vendor’s specifications. The USB Wi-Fi driver for the MT7601U is ported for both access point and station modes. The driver source is configured for the HISI_3518E platform, compiled into a kernel module, and deployed with the appropriate configuration file. In access point mode, the module creates a wireless network with a specified SSID, authentication mode, encryption type, and password. In station mode, the `wpa_supplicant` tool is ported and configured. Both modes enable the video transmission system to operate over wireless links.
- Upper-Layer Video Pipeline: VI, VPSS, VENC, and H.264
Video is a sequence of images, and each image is composed of pixels. Pixels can be represented in RGB or YUV formats. RGB is common for display, while YUV is common for encoding and transmission. The HI3518E MPP provides a structured way to manage these video operations. The VI module handles video input. On the HI3518E, VI consists of one video input device and one physical channel, with support for multiple extension channels. The input device supports DC, BT601, BT656, and MIPI Rx timing. The physical channel can crop, scale, and rotate data before writing it to memory. VI can work in offline mode, storing data in DDR, or in online mode, sending data directly to VPSS.
The VI module is initialized after the MPP system variables are configured. The system variable structure defines the video buffer pool, including the number of buffers and their sizes. The sensor resolution is obtained, and three output resolutions are planned: 1280 × 720, 640 × 480, and 320 × 240. The ISP unit is initialized to handle auto exposure, auto white balance, auto focus, and wide dynamic range. The public ISP attributes specify a BGGR format, 30 fps, a capture origin of (0, 0), and a 1280 × 720 capture area. A separate thread runs the ISP process. The VI device and channel are then enabled with the appropriate attributes. If any step fails, the module is disabled and unloaded.
The VPSS module performs unified preprocessing and then outputs multiple channels at different resolutions. It uses groups and channels. A group is a virtualized hardware unit, and a channel performs scaling, frame rate control, cropping, and other operations. The HI3518EV200 supports multiple groups, and each group can have a physical channel and extension channels. In this project, one group and three channels are created. The group is configured for 1280 × 720 YUV420 with noise reduction enabled. The VI module is bound to the VPSS module. The three channels are configured for the target resolutions and then enabled. If a channel fails, the previously enabled channels are disabled in reverse order.
The VENC module performs video encoding. It supports multiple independent encoding channels and can add region-based overlays such as watermarks. The encoding channel receives semi-planar YUV 4:2:0 or 4:2:2 images, compares the input size with the channel size, and then encodes the image. The rate controller adjusts encoding parameters to control the output bitrate. In this project, H.264 is chosen because it offers a much higher compression ratio than MJPEG, better network compatibility, and better adaptability. Three encoding channels are created to match the three VPSS outputs. Each channel is configured with H.264 attributes, including image size, baseline profile, stream buffer, frame rate, and bitrate. The channels are created, started, and bound to the VPSS channels. A thread then handles the encoded stream. In a local storage test, the thread writes three H.264 elementary streams to files. These files can be played by VLC, confirming that the video pipeline is working.
H.264 is a digital video compression standard developed by the Joint Video Team. It achieves high compression by reducing temporal and spatial redundancy. A macroblock is the basic unit, typically 16 × 16 YUV data. Slices are composed of macroblocks, and frames are composed of slices. H.264 defines I-frames, P-frames, and B-frames. An I-frame is an intra-coded key frame and does not depend on other frames. A P-frame uses forward prediction from a previous frame. A B-frame uses bidirectional prediction from previous and future frames. A group of pictures, or GOP, is a set of frames that are strongly related. The first image in a sequence is an instantaneous decoder refresh image, which resets the decoder and allows a new sequence to begin.
The H.264 elementary stream is divided into a video coding layer and a network abstraction layer. The video coding layer handles compression and decompression. The network abstraction layer formats the data for transport. The raw bitstream is converted into a raw byte sequence payload, then an extended byte sequence payload, and finally a network abstraction layer unit. Multiple NAL units form the H.264 stream that is transmitted. This separation allows H.264 to adapt to different network environments. The NAL unit header contains a forbidden bit, a importance indicator, and a NAL unit type. Types include non-IDR slices, IDR slices, supplemental enhancement information, sequence parameter sets, picture parameter sets, delimiters, sequence end, stream end, and filler data. Sequence and picture parameter sets are especially important because they contain decoding parameters. They must appear before an I-frame for correct decoding. In live streaming, they are often repeated before each I-frame to support clients that join at different times.
NAL Unit Type Meaning 1 Non-IDR slice 5 IDR slice 6 Supplemental enhancement information 7 Sequence parameter set 8 Picture parameter set 9 Access unit delimiter 10 Sequence end 11 Stream end 12 Filler data - Real-Time Streaming with RTP/RTSP and Offline MP4 Storage
Internet video transmission can follow a download model or a streaming model. Downloading requires the entire file before playback, while streaming allows playback while data is still being received. Because the humanoid robot interaction system requires real-time response, the project uses streaming. RTP provides end-to-end real-time transport for audio and video. It includes synchronization source identifiers, payload types, sequence numbers, timestamps, and monitoring information. RTP is usually carried over UDP, and because it does not provide reliability by itself, the receiving side uses sequence numbers to reconstruct the stream. RTCP works alongside RTP to provide quality feedback. RTSP is an application-layer protocol for controlling streaming media. It supports operations such as pause and fast-forward, but it does not carry the media data itself. Instead, media data is usually carried by RTP over UDP or TCP. RTSP is similar to HTTP in syntax and message parameters, and it establishes a client-server session for media control.
RTP Field Function V RTP version P Padding flag X Extension flag CC CSRC count M Marker PT Payload type Sequence number Packet ordering and loss detection Timestamp Media timing and synchronization SSRC Synchronization source identifier CSRC Contributing source identifier The ORTP library is ported to implement RTP transmission. ORTP is an open-source RTP library written in C. The source is downloaded, the H.264 payload type is added, and the library is configured for the ARM target. The compiled shared libraries are copied to the target board. In the application, ORTP is initialized, a scheduler is created, a new session is created in send-only mode, and the session is configured for scheduling, blocking behavior, transport protocol, remote address, and payload type. Because H.264 NAL units can be larger than the maximum transmission unit, the code implements fragmentation. If the data length is within the maximum packet length, it is sent directly. Otherwise, the NAL unit is split into fragments, with the first fragment marked as the start and the last fragment marked as the end. The timestamp is increased as needed. After compilation, the executable is copied to the target board, and VLC can play the stream using an SDP file that describes the media session.
RTSP is implemented on top of the RTP transport. The server side on the target board initializes global variables, creates a listening thread, and waits for client connections. When a client connects, a message-handling thread processes requests and returns the appropriate responses. The VENC thread sends data only when the client is in the sending state after a PLAY request. The H.264 stream is divided according to the protocol and NAL unit requirements before transmission. The complete RTSP video transmission system starts by initializing the MPP system, configuring VI, VPSS, and VENC, creating a thread to process and send the video stream, creating an RTSP server thread, and creating a client message thread. The system supports both wired and wireless transmission. The video can be viewed in real time on a PC.
Offline storage is also implemented because network conditions may be poor and because recorded video may be needed for later review. The MP4 format is chosen because it is a widely supported container that can hold H.264 video and other data types. An MP4 file is composed of boxes organized in a tree structure. The file type box, or ftyp, is especially important and appears at the beginning of the file. The MP4V2 library is ported to simplify MP4 writing. The source is configured for the ARM target, compiled, and installed. The generated headers and libraries are added to the MPP project, and the shared libraries are copied to the target board. The VENC stream handling code is modified to package each frame into MP4. When the first frame is processed, an MP4 file is created and the time scale is set. Sequence parameter sets and picture parameter sets are added as video track information. Other frames are written as samples. The file is closed after the recording period. The resulting MP4 file can be played by VLC. This offline storage feature improves the system’s practicality and stability.
- Human Keypoint Extraction and Inverse Kinematics for Humanoid Robot Control
The PC-side processing pipeline uses OpenPose, an open-source project from Carnegie Mellon University. OpenPose is implemented in C++ and uses a convolutional neural network framework, Caffe, to perform 2D real-time multi-person keypoint detection, 3D single-person keypoint detection, and single-person tracking. The project deploys OpenPose with CMake, Visual Studio 2019, CUDA 11.1.1, cuDNN 8.1.0, and the required Caffe and OpenCV dependencies. The GPU is an NVIDIA GeForce GTX 1660 Ti, which accelerates inference. The system can process images, offline video, or live video streams. In this project, the live video stream comes from the HI3518E video transmission system acting as an IP camera.
OpenPose returns human joint positions, including the hands, elbows, shoulders, and other body keypoints. These positions are expressed in the camera coordinate system. To control the humanoid robot, the positions must be converted into joint angles. The project uses inverse kinematics based on the Denavit-Hartenberg, or D-H, parameter model. The D-H method simplifies the modeling of a robot’s kinematic chain by describing each link with four parameters: link length, link twist, link offset, and joint angle. The coordinate frame of each joint is defined so that the z-axis follows the joint axis, the x-axis follows the common normal or the normal to the plane of two intersecting axes, and the y-axis follows the right-hand rule. A transformation from one joint frame to the next requires four operations: rotation about the previous z-axis, translation along the previous z-axis, translation along the current x-axis, and rotation about the current x-axis. The resulting homogeneous transformation matrix relates the two frames.
Link Link Length Link Twist Link Offset Joint Angle 1 0 π/2 0 θ1 2 a2 0 0 θ2 3 a3 0 0 θ3 For the humanoid robot’s arm, the D-H model treats the shoulder rotation, shoulder abduction, and elbow as a three-link chain. The upper arm length and forearm length are represented by a2 and a3. The joint angles are θ1, θ2, and θ3. Using the transformation matrices, the position and orientation of the hand can be expressed relative to the shoulder. OpenPose provides the shoulder and hand positions in camera coordinates. By combining the measured hand pose with the transformation matrix, the system solves for the joint angles. These angles become the control data for the humanoid robot. The same principle can be applied to other limbs, allowing the humanoid robot to imitate human motion.
- FreeRTOS-Based Humanoid Robot Control System
The humanoid robot control system uses FreeRTOS, a small real-time operating system kernel. FreeRTOS provides task scheduling, inter-task communication through semaphores and message queues, and memory management. It is lightweight and suitable for microcontrollers with limited RAM and Flash. It is also open source, portable, and configurable. The project ports FreeRTOS to the STM32F103RCT6 used by the Young PI humanoid robot. The kernel source is obtained from the official website, and a Keil project is created. The core source files, portable layer files for the ARM Cortex-M3, and the configuration header `FreeRTOSConfig.h` are added to the project. Interrupt handling in `stm32f10x_it.c` and the startup file are modified so that FreeRTOS manages the relevant interrupts. After compilation, the firmware is flashed to the robot controller.
On top of FreeRTOS, the application initializes the system clock, interrupts, and timers. It also initializes the serial ports used for debugging and servo communication, and it initializes the Bluetooth module used to communicate with the PC. A task is created to wait for incoming data and forward it to the servos. The data can be sent in a transparent transmission mode, where the angle data from the human keypoints is passed directly to the servos, or it can be parsed and encoded according to the inverse kinematics results before being sent. The HC-05 Bluetooth module provides a full-duplex serial link. Its default baud rate is 9600, with 8 data bits and 1 stop bit. It operates in slave mode by default and can be configured through AT commands. The servo protocol uses a header byte of 0xff, a product ID of 0xb0, a module ID, an instruction, up to three parameters, and a checksum. Each of the 18 servos can be addressed individually.
Servo Command Field Description Header Fixed at 0xff Product ID Fixed at 0xb0 for the digital servo Module ID Identifies the servo Instruction Command name Parameter 1 to 3 Command parameters Check Sum Validation value When the PC and humanoid robot Bluetooth modules are paired, the correct serial port is selected on the PC, and the OpenPose code is modified to use that port. The humanoid robot then receives angle commands in real time. Because FreeRTOS provides deterministic task scheduling, the control loop can respond quickly. The humanoid robot can imitate upper-body motion, including shoulder rotation, shoulder abduction, and elbow movement. The project notes that the system focuses on upper-body interaction because the robot does not include gravity sensors, ultrasonic sensors, or other hardware needed for walking and obstacle avoidance. Even so, the combination of vision, keypoint extraction, inverse kinematics, and real-time control demonstrates a complete dynamic interaction loop.
- Full-System Operation and Observed Interaction
The complete system integrates three major parts: the video transmission system, the human keypoint processing system, and the humanoid robot control system. The camera captures video, the HI3518E processes and encodes it, and the video is streamed over Wi-Fi or Ethernet. The PC receives the stream, OpenPose extracts human joint positions, and the inverse kinematics module converts those positions into joint angles. The angle data is sent over Bluetooth or serial to the humanoid robot. The FreeRTOS-based controller receives the data and drives the servos. The result is a humanoid robot that can react to human motion in real time.
The operating procedure is simple. The video transmission system is started first. If wireless mode is used, the PC connects to the Wi-Fi network provided by the MT7601U module. The humanoid robot control system is then started. Finally, the OpenPose application is launched on the PC. The humanoid robot begins to receive commands and imitate the detected human motion. The project reports that the system functions as intended under suitable hardware and network conditions. The interaction effect is demonstrated in the project’s repository. The system shows that a low-cost humanoid robot can achieve dynamic visual interaction using domestic chip technology and open-source software.
- Challenges, Limitations, and Future Work
The project identifies several technical challenges. The first is building a domestic video transmission solution based on the Hisi HI3518E. The second is maintaining real-time video streaming and optimizing encoding and decoding. The third is adapting OpenPose from offline media to live IP camera streams. The fourth is porting a real-time operating system to the humanoid robot controller and ensuring timely servo response. Each of these challenges is addressed in the implemented system, but limitations remain.
The system is developed in a distributed manner, with separate modules for video transmission, keypoint processing, and robot control. This modularity simplifies development but reduces integration. Future work could use a more powerful Hisi system-on-chip with richer on-chip resources to create a more integrated humanoid robot platform. Video transmission still faces bandwidth and codec constraints, so future improvements should increase transmission bandwidth, use more reliable network equipment, and reduce redundant data in the encoded stream. The humanoid robot lacks gravity sensors, ultrasonic sensors, and other perception modules, so it cannot walk or avoid obstacles. The current interaction is therefore limited to the upper body. OpenPose also has limitations in keypoint detection under certain poses and conditions, which can cause some actions to fail. Future work could add gait planning and more sensors to enable full-body interaction and could further optimize OpenPose for robust performance.
- Conclusion: A Complete Low-Cost Humanoid Robot Interaction Path
The project delivers a complete humanoid robot dynamic interaction system based on the Hisi HI3518E. It includes a custom embedded Linux platform, a video capture and H.264 encoding pipeline, RTP/RTSP streaming, offline MP4 storage, OpenPose-based human keypoint extraction, inverse kinematics conversion, and FreeRTOS-based servo control. The system demonstrates that a humanoid robot can capture human motion, interpret it, and respond in real time using a combination of domestic hardware and open-source software. The work provides a practical reference for low-cost human-computer interaction solutions in the Hisi ecosystem and contributes to the broader development of autonomous, domestically supported humanoid robot technology.