In this thesis, I present a comprehensive study on the design and implementation of a humanoid robot control system driven by depth camera data. The primary goal is to enable intuitive human-robot interaction through body posture recognition and hand gesture recognition using a Kinect sensor, and then to control a humanoid robot to perform walking and other movements. The work encompasses the entire pipeline from data acquisition, image processing, feature extraction, and classification to kinematic modeling and gait planning for the humanoid robot.
The system architecture is built around three main components: the Kinect sensor as the perception device, a PC as the processing unit, and an Arduino-based humanoid robot as the actuator. The Kinect sensor captures RGB images, depth images, and skeleton joint data. The PC processes this data using OpenCV and the Kinect SDK to recognize static postures and hand gestures. The recognized results are mapped to specific motion commands, which are then transmitted via Bluetooth to the Arduino Mega 2560 controller. The humanoid robot, equipped with 17 digital servo motors, executes the corresponding motions, primarily forward walking and dance-like actions.
This research focuses on two main algorithmic contributions. First, a static posture recognition method based on the spatial angles formed by skeleton joints is proposed. By representing body segments as vectors in 3D space, the angles between these vectors allow robust classification of postures such as arms raised, arms extended, and single-arm movements. Second, a novel long-distance hand gesture recognition method is introduced. Unlike traditional approaches that rely solely on depth images, this method combines depth and color information to extract hand regions even when the user is far from the sensor. The hand skeleton is then computed, and its Hu moments are used as features for a Support Vector Machine (SVM) classifier. Experimental results demonstrate significant improvement in recognition accuracy compared to using hand contour Hu moments.
For the humanoid robot, I established a kinematic model using the Denavit-Hartenberg (D-H) method. The walking algorithm employs cubic spline interpolation to guarantee smooth ankle trajectories with second-order continuity. Inverse kinematics is used to compute the joint angles of the lower limbs. To prevent servo motor burnout, I also proposed a stability verification method that calculates the ankle torque during single-leg support. This allows the joint angle sequence to be validated before executing the walking motion.
The complete system was successfully integrated and tested. The humanoid robot was able to walk forward when the user raised both arms, and it performed specific dance motions when the user made certain hand gestures with the right arm extended. The system demonstrates the feasibility of using low-cost depth sensors for natural control of humanoid robots, opening up possibilities for applications in education, entertainment, and assistive robotics.
1. Introduction and Background
The field of robotics has evolved rapidly in recent decades, moving from industrial manipulators to autonomous and interactive systems. Humanoid robots, in particular, have attracted great attention because of their potential to work in human environments and perform tasks that require human-like locomotion and manipulation. However, controlling a humanoid robot remains challenging due to the complexity of bipedal walking, balance, and real-time interaction.
Traditional human-robot interaction methods often rely on remote controls, keyboards, or pre-programmed sequences. These methods are not intuitive and require training. To improve the user experience, natural user interfaces based on vision, voice, and gestures have been developed. Among these, depth cameras such as the Microsoft Kinect have revolutionized the field because they provide real-time depth and skeleton data at a low cost. Since the release of the Kinect in 2010, many researchers have explored its use for human tracking, gesture recognition, and robot control.
This thesis addresses the gap between depth-camera-based recognition and humanoid robot motion planning. I propose a complete system where the user controls a humanoid robot by performing specific postures and hand gestures. The main contributions are:
- A static posture recognition algorithm using skeleton vector angles, achieving high accuracy in real-time.
- A long-distance hand gesture recognition algorithm that combines color and depth images to extract hand skeletons, with Hu moment features and SVM classification.
- A gait planning algorithm for a humanoid robot using D-H kinematic modeling and cubic spline trajectory generation.
- A stability verification method to prevent servo overload during walking.
- Successful integration of the recognition and control modules, demonstrated with walking and dancing behaviors.
2. Kinect Sensor and Development Environment
The Kinect sensor used in this work is the original Kinect for Xbox 360 model. It consists of three main optics: an infrared projector, an RGB camera, and an infrared camera. The infrared projector emits a speckle pattern that is captured by the infrared camera to compute depth information. This process is based on the principle of light coding, where the distortion of the speckle pattern relative to a known reference encodes the distance to the sensor. The Kinect is capable of producing depth images at 30 frames per second with a resolution of 640×480, operating over a range of 0.8 to 4 meters.
The depth image contains 16-bit pixels, where the high 13 bits represent the distance from the sensor to the object, and the low 3 bits are used for user segmentation. When the skeleton tracking feature is enabled, the Kinect SDK classifies each pixel as belonging to a particular user or the background. It then applies a machine learning model trained on millions of depth images to identify 20 body joints, including the head, hands, elbows, knees, and feet. This skeleton tracking capability is the foundation for posture recognition.
On the software side, I used the Kinect for Windows SDK version 1.8, which provides APIs for accessing the color, depth, and skeleton streams. The image processing and computer vision tasks were implemented using OpenCV version 2.4.3. The development environment was Microsoft Visual Studio 2013 with C++ as the programming language. The Arduino IDE was used to program the Arduino Mega 2560 controller that drives the servos of the humanoid robot.
2.1 Depth Image Generation Principle
The infrared projector generates a speckle pattern that is invisible to the human eye. When this pattern hits objects at different distances, the shape and displacement of the speckles vary. By comparing the observed pattern with pre-captured reference patterns at different depths, the Kinect can determine the depth of each pixel. This method is known as light coding and allows accurate depth estimation without requiring an active sensor such as a laser scanner.
Mathematically, the depth image \(d(x,y)\) can be represented as a function of the observed speckle shift \(\Delta(x,y)\) relative to the reference plane at depth \(Z_0\):
$$
d(x,y) = \frac{Z_0}{1 + \frac{Z_0}{f B} \Delta(x,y)}
$$
where \(f\) is the focal length of the infrared camera and \(B\) is the baseline distance between the infrared projector and camera. This relationship enables the conversion of pixel-wise speckle displacement into metric depth values.
3. Posture Recognition
Posture recognition is divided into two categories: static and dynamic. Static recognition identifies a fixed pose at a given moment, while dynamic recognition analyzes sequences of movements. In this research, I focused on static posture recognition because it is sufficient for triggering discrete robot commands. The skeleton tracking data provides the 3D coordinates of 20 joints. Based on these coordinates, I define spatial vectors between adjacent joints to represent body segments.
Each joint is represented as a point \(p_i = (x_i, y_i, z_i)\), where \(i \in \{A, B, \ldots, T\}\) corresponds to the joints listed in the standard Kinect skeleton. For example, the vector representing the upper right arm is \(\overrightarrow{GH} = p_H – p_G\), where \(G\) is the right shoulder and \(H\) is the right elbow. Similarly, the right forearm is \(\overrightarrow{HI} = p_I – p_H\), with \(I\) being the right wrist.
To recognize a specific posture, I compute the angle between two relevant vectors. For a raised hand posture, the forearm vector should be nearly parallel to the torso vector \(\overrightarrow{EB}\), where \(E\) is the spine center and \(B\) is the neck. The angle \(\theta\) between two vectors \(\mathbf{u}\) and \(\mathbf{v}\) is given by:
$$
\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}
$$
$$
\theta = \arccos\left(\frac{u_x v_x + u_y v_y + u_z v_z}{\sqrt{u_x^2+u_y^2+u_z^2}\sqrt{v_x^2+v_y^2+v_z^2}}\right)
$$
For a horizontal arm posture, the forearm vector must be perpendicular to the torso vector and collinear with the upper arm vector. I allowed a tolerance of ±5 degrees for parallelism and ±5 degrees around 90 degrees for perpendicularity. The recognition algorithm compares the calculated angles with predefined thresholds and outputs the detected posture.
3.1 Experimental Verification
I conducted experiments with several users in front of the Kinect. The system was able to recognize four postures: both arms raised, left arm raised with right arm horizontal, both arms horizontal, and left arm horizontal. The recognition accuracy exceeded 90% when the skeleton was fully visible and no occlusions were present. The processing time was fast enough to support real-time interaction. Table 1 lists the postures and their recognition criteria.
| Posture | Vectors Used | Angle Condition |
|---|---|---|
| Both arms raised | Left forearm and torso, Right forearm and torso | Both angles < 5° |
| Left arm raised, right arm horizontal | Left forearm and torso; right forearm and torso; right forearm and right upper arm | Left < 5°; right perpendicular 85°–95°; right upper arm collinear 175°–185° |
| Both arms horizontal | Both forearms and corresponding upper arms | Collinearity ±5° and perpendicular to torso ±5° |
| Left arm horizontal | Left forearm and left upper arm; left upper arm and torso | Collinear ±5°; perpendicular ±5° |
The experimental setup is shown in the photos captured during the test. The user stands approximately 1.5 meters from the Kinect. A green skeleton overlay is drawn on the color image to visualize the tracked joints. The recognized posture is displayed on the screen, and the corresponding control command is sent to the robot.
4. Hand Gesture Recognition Based on Depth Information
Hand gestures provide a richer and more flexible interface than whole-body postures. However, the Kinect’s skeleton tracking only gives the position of the hand joint, not individual finger coordinates. When the user is close to the sensor, the depth image may contain enough resolution to distinguish fingers, but at distances beyond two meters, the hand region becomes too small. Therefore, I developed a method to extract a detailed hand region by combining depth and color images, and then used the hand skeleton’s Hu moments as features for classification.
4.1 Hand Region Extraction
The process begins with obtaining the depth and color frames from the Kinect. Since the color camera and depth camera are mounted at different positions, their images must be aligned. The Kinect SDK provides a mapping function that transforms depth coordinates to color coordinates. Using the hand joint location in the depth image, I crop a region of interest (ROI) from both the depth and color images around the hand.
For close range (within 2 meters), the depth image of the hand is relatively clear. I first apply a median filter to remove noise while preserving edges. The median filter output is given by:
$$
g(x,y) = \mathrm{med}\{f(x-i, y-j) : (i,j) \in S\}
$$
where \(S\) is a 3×3 or 5×5 neighborhood. This filter effectively removes speckle noise caused by the infrared sensor.
For long range (2 to 4 meters), the depth image of the hand is nearly unrecognizable because the number of pixels covering the hand is very small, and the depth variance is high. To overcome this, I propose the following procedure:
- Align the depth and color images.
- Crop the ROI around the hand center in both images.
- Segment the hand from the background in the depth image using a depth threshold based on the hand center depth. This gives a rough silhouette.
- Apply the same ROI to the color image and perform skin color detection or use the depth mask to isolate the hand pixels in color.
- Intersect the depth-derived hand mask and the color-derived hand mask to obtain a more reliable hand region.
- Since the depth values in the hand region are noisy, I perform an interpolation step: starting from the center pixel (which is known to be on the hand), I fill the region bounded by the detected edges with distance values estimated from nearby valid pixels.
- Then, apply Otsu thresholding to separate the hand from the background in this interpolated depth map.
Otsu’s method selects a threshold \(X\) that maximizes the between-class variance \(\sigma^2(x)\). The formulas are as follows:
$$
N = \sum_{i=0}^{255} n_i, \quad p_i = \frac{n_i}{N}
$$
$$
p_A = \sum_{i=0}^{x} p_i, \quad p_B = \sum_{i=x+1}^{255} p_i = 1 – p_A
$$
$$
w_A = \sum_{i=0}^{x} \frac{i p_i}{p_A}, \quad w_B = \sum_{i=x+1}^{255} \frac{i p_i}{p_B}
$$
$$
w_0 = \sum_{i=0}^{255} i p_i
$$
$$
\sigma^2(x) = p_A (w_A – w_0)^2 + p_B (w_B – w_0)^2
$$
The optimal threshold is chosen as \(X = \arg\max_{0 \le x \le 255} \sigma^2(x)\). This threshold separates the hand from the background with minimal classification error.
4.2 Hand Skeletonization
After obtaining a clean binary segmentation of the hand, I extract the hand skeleton using a parallel thinning algorithm. The Zhang-Suen (ZS) algorithm is a well-known fast parallel thinning method that iteratively deletes boundary pixels while preserving the connectivity and topology of the shape. The algorithm operates on a 3×3 neighborhood, as shown below:
P9 P2 P3 P4 P1 P8 P7 P6 P5
In each iteration, two sub-iterations are performed. In the first sub-iteration, a contour point \(P1\) is removed if all of the following conditions hold:
$$
2 \le B(P1) \le 6
$$
$$
A(P1) = 1
$$
$$
P2 \cdot P4 \cdot P6 = 0
$$
$$
P4 \cdot P6 \cdot P8 = 0
$$
In the second sub-iteration, the conditions are:
$$
2 \le B(P1) \le 6
$$
$$
A(P1) = 1
$$
$$
P2 \cdot P4 \cdot P8 = 0
$$
$$
P2 \cdot P6 \cdot P8 = 0
$$
Here, \(B(P1)\) is the number of non-zero neighbors among \(P2\) to \(P9\), and \(A(P1)\) is the number of 0-to-1 transitions in the ordered sequence \(P2, P3, \ldots, P9\). This algorithm produces a unit-width skeleton that preserves the main structure of the hand, including five fingers.
The skeleton image is then used to compute Hu moments, which are seven invariant moments derived from the second and third order central moments. These moments are invariant to translation, scale, and rotation, making them suitable for hand shape recognition. The definitions are as follows:
First, the \((p+q)\) order moment is:
$$
m_{pq} = \sum_{y} \sum_{x} x^p y^q f(x,y)
$$
The central moment is:
$$
\mu_{pq} = \sum_{y} \sum_{x} (x-\bar{x})^p (y-\bar{y})^q f(x,y)
$$
where \(\bar{x} = m_{10}/m_{00}\) and \(\bar{y} = m_{01}/m_{00}\). The normalized central moment is:
$$
\eta_{pq} = \frac{\mu_{pq}}{\mu_{00}^r}, \quad r = \frac{p+q}{2}+1
$$
The seven Hu moments are:
$$
\phi_1 = \eta_{20} + \eta_{02}
$$
$$
\phi_2 = (\eta_{20} – \eta_{02})^2 + 4\eta_{11}^2
$$
$$
\phi_3 = (\eta_{30} – 3\eta_{12})^2 + (3\eta_{21} – \eta_{03})^2
$$
$$
\phi_4 = (\eta_{30} + \eta_{12})^2 + (\eta_{21} + \eta_{03})^2
$$
$$
\phi_5 = (\eta_{30} – 3\eta_{12})(\eta_{30} + \eta_{12})[(\eta_{30} + \eta_{12})^2 – 3(\eta_{21} + \eta_{03})^2] + (3\eta_{21} – \eta_{03})(\eta_{21} + \eta_{03})[3(\eta_{30} + \eta_{12})^2 – (\eta_{21} + \eta_{03})^2]
$$
$$
\phi_6 = (\eta_{20} – \eta_{02})[(\eta_{30} + \eta_{12})^2 – (\eta_{21} + \eta_{03})^2] + 4\eta_{11}(\eta_{30} + \eta_{12})(\eta_{21} + \eta_{03})
$$
$$
\phi_7 = (3\eta_{21} – \eta_{03})(\eta_{30} + \eta_{12})[(\eta_{30} + \eta_{12})^2 – 3(\eta_{21} + \eta_{03})^2] + (3\eta_{12} – \eta_{30})(\eta_{21} + \eta_{03})[3(\eta_{30} + \eta_{12})^2 – (\eta_{21} + \eta_{03})^2]
$$
These seven values form the feature vector for each hand skeleton. The feature vector is fed into a Support Vector Machine (SVM) classifier. SVM is a supervised learning model that constructs a hyperplane in a high-dimensional space to separate different classes. For non-linear classification, I used the Gaussian radial basis function (RBF) kernel:
$$
K(\mathbf{x}, \mathbf{x}_i) = \exp\left(-\frac{\|\mathbf{x} – \mathbf{x}_i\|^2}{\delta^2}\right)
$$
where \(\delta\) is a kernel parameter set to 0.5 in my experiments.
4.3 Experimental Results for Hand Recognition
I conducted experiments to compare two feature representations: the Hu moments of the hand contour and the Hu moments of the hand skeleton. The experiments were performed at two ranges: within 2 meters (using the traditional depth-based hand extraction) and between 2 and 4 meters (using the proposed long-distance extraction method). Five hand shapes were selected for recognition: open palm, fist, pointing index finger, two fingers, and the “OK” sign. For each condition, I collected 40 training samples per hand shape and used the remaining samples for testing. The total number of test samples varied slightly due to invalid frames. The recognition accuracy is summarized in Table 2.
| Feature Type | Distance Range | Hand Shape 1 (%) | Hand Shape 2 (%) | Hand Shape 3 (%) | Hand Shape 4 (%) | Hand Shape 5 (%) | Overall (%) |
|---|---|---|---|---|---|---|---|
| Contour Hu | < 2m | 73.93 | 31.50 | 21.01 | 36.99 | 66.29 | 51.73 |
| Skeleton Hu | < 2m | 88.69 | 78.74 | 30.25 | 94.52 | 97.75 | 79.34 |
| Contour Hu | 2–4m | 100 | 78.95 | 69.39 | 24.14 | 66.36 | 68.98 |
| Skeleton Hu | 2–4m | 86.25 | 78.95 | 38.78 | 71.93 | 93.46 | 78.25 |
From the results, it is evident that using hand skeleton Hu moments significantly outperforms contour Hu moments in both distance ranges. The overall accuracy for skeleton features is about 79% within 2 meters and 78% in the 2–4 meter range, while contour features only achieve around 52% and 69%, respectively. The proposed long-distance extraction method does not degrade the recognition performance, proving its effectiveness. The lower accuracy for hand shape 3 (pointing index finger) in both cases suggests that the skeleton of this shape is more similar to the “two fingers” shape, causing confusion.
5. Humanoid Robot Platform and Motion Planning
The humanoid robot used in this thesis is a 17-degree-of-freedom biped robot controlled by an Arduino Mega 2560 microcontroller. It is equipped with LD-2015 digital servo motors that provide high torque and precise angle control. The robot is approximately 365 mm tall and weighs about 1.7 kg. The body frame is constructed from 2 mm thick aluminum alloy plates to balance weight and strength.
5.1 Kinematic Model
To analyze the walking motion, I established a kinematic model based on the Denavit-Hartenberg (D-H) convention. The robot’s lower limbs are modeled as a series of links and joints. Each joint has a coordinate system attached according to the D-H rules. The model parameters include the link length \(a_i\), link twist \(\alpha_i\), link offset \(d_i\), and joint angle \(\theta_i\). The homogeneous transformation matrix between consecutive frames is:
$$
T_i^{i-1} = \begin{bmatrix}
\cos\theta_i & -\sin\theta_i \cos\alpha_i & \sin\theta_i \sin\alpha_i & a_i \cos\theta_i \\
\sin\theta_i & \cos\theta_i \cos\alpha_i & -\cos\theta_i \sin\alpha_i & a_i \sin\theta_i \\
0 & \sin\alpha_i & \cos\alpha_i & d_i \\
0 & 0 & 0 & 1
\end{bmatrix}
$$
The world coordinate system is placed at the hip joint. For the swinging leg, the transformation from the hip to the ankle is composed of two rotations about the X-axis (pitch) and translations along the links. If \(\alpha_0\) is the hip pitch angle and \(\alpha_1\) is the knee pitch angle, the ankle position in the hip frame is:
$$
\begin{cases}
x = 0 \\
y = l_1 \sin(\alpha_0 – \alpha_1) + l_0 \sin\alpha_0 \\
z = -l_1 \cos(\alpha_0 – \alpha_1) + l_0 \cos\alpha_0
\end{cases}
$$
where \(l_0 = 50\) mm is the distance between the hip and knee, and \(l_1 = 40\) mm is the distance between the knee and ankle.
5.2 Gait Planning
For smooth walking, the ankle trajectory must have continuous first and second derivatives. I chose a cubic spline interpolation curve for the vertical foot trajectory \(z = f(y)\), where \(y\) is the forward direction. The trajectory is divided into two parts for the step. With a step length \(\lambda = 20\) mm and a maximum foot lift \(H = 5\) mm, the functions are:
$$
f(y) = -\frac{1}{100}y^3 – \frac{15}{100}y^2 – 64.3, \quad y \in [-10, 0]
$$
$$
f(y) = \frac{1}{100}y^3 – \frac{15}{100}y^2 – 64.3, \quad y \in [0, 10]
$$
These functions ensure continuity at the midpoint and provide a smooth lifting/lowering motion. Using the inverse kinematics equations derived from the position equations, I can compute the desired joint angles \(\alpha_0\) and \(\alpha_1\) for each point on the trajectory:
$$
\alpha_0 = \arctan\left(\frac{y}{-z}\right) + \arctan\left(\frac{l_1 \sin\alpha_1}{l_0 + l_1 \cos\alpha_1}\right)
$$
$$
\alpha_1 = \pi – \arccos\left[\frac{(l_0^2 + l_1^2) – (y^2 + z^2)}{2 l_0 l_1}\right]
$$
The complete walking cycle consists of ten states, as illustrated by the servo angle sequence in Table 3. The servos are labeled from A to J for the lower limbs. Each value corresponds to the target angle in degrees.
| Step | Action | Servo Sequence (A, B, C, D, E, F, G, H, I, J) |
|---|---|---|
| 1 | Initialization | A80, B100, C10, D170, E90, F90, G90, H90, I90, J90 |
| 2 | Lift left foot | H100 → I100 → G100 → E100 |
| 3 | Shift weight right and move left foot forward | A100 → H105 → E110 → A110 → C20 → G90 |
| 4 | Shift weight left and bring right foot forward | H80 → E100 → F100 → A100 → F110 |
| 5 | Recover center position | E90 → A80 → F90 → I90 → H90 |
Each transition is executed with a delay of 100 ms to allow the servos to reach their target positions. The sequence is repeated continuously to produce a stable forward walk.
5.3 Stability Verification
To prevent the servo motors from being overloaded during single-leg support, I computed the ankle torque. The robot’s body is simplified into three masses: the supporting leg, the torso, and the swinging leg. The torque \(M_o\) about the ankle pivot point \(O\) is:
$$
M_o = \sum_i \mathbf{F}_i \times \mathbf{L}_i
$$
For the simplified planar model, the torque components are:
$$
\begin{cases}
M_o(F_1) = -F_1 (L – 2L_1 \sin\theta) \\
M_o(F_2) = F_2 L_2 \sin\theta \\
M_o(F_3) = -F_3 (L – 2L_3 \sin\theta)
\end{cases}
$$
where \(F_1\), \(F_2\), \(F_3\) are the weights of the three parts, \(L\) is the distance from the ankle to the hip, \(L_1\), \(L_2\), \(L_3\) are the positions of the centers of mass, and \(\theta\) is the ankle roll angle. If the computed torque exceeds the servo’s rated torque, the ankle roll angle is reduced or the joint angles are adjusted until the torque is within the safe limit. This verification significantly reduced the risk of burning out the servos during experiments.
6. System Integration and Experimental Implementation
The complete system consists of two software modules: the PC-side recognition program and the Arduino-side motion control program. The PC program was written in C++ using the Kinect SDK and OpenCV. It performs the following steps:
- Initialize the Kinect sensor and open the color, depth, and skeleton streams.
- Display the color and depth images on the screen.
- Wait for a valid skeleton to be tracked.
- If the user’s posture is both arms raised, the program sends a command to the robot to start walking.
- If the user’s posture is right arm extended, the program triggers hand gesture recognition.
- The hand gesture is classified into one of the predefined shapes, and a corresponding command is sent to the robot.
- After the robot finishes the action, it sends back a completion signal, and the Kinect resumes skeleton tracking.
The Arduino program was developed in the Arduino IDE. It initializes all servo angles and then continuously monitors the serial port. When a valid command is received, it executes the corresponding motion sequence. The commands are encoded as simple characters or numbers:
| Command | Meaning | Robot Action |
|---|---|---|
| ‘W’ | Walk | Perform five-step forward walk |
| ‘1’ | Gesture 1 | Perform dance move 1 |
| ‘2’ | Gesture 2 | Perform dance move 2 |
| ‘S’ | Stop | Stop current motion and return to initial pose |
Wireless communication between the PC and the Arduino is achieved via Bluetooth modules. The PC uses a Bluetooth adapter, and the Arduino is connected to an HC-05-compatible module operating at the standard 2.4 GHz frequency. The pairing process is performed using BlueSoleil software on the PC, which creates a virtual COM port for serial data transmission.
The humanoid robot was tested in a laboratory environment. The user stood approximately 1.5 to 3 meters from the Kinect. When the user raised both arms, the robot began to walk forward in a straight line. When the user extended the right arm and made a specific hand gesture, the robot performed a corresponding dance motion. The robot’s walking speed was approximately 10 cm per second, which was sufficient for demonstration purposes. Figure 5 shows a sequence of the robot walking, and Figure 6 shows the robot performing a dance move.
7. Conclusion and Future Work
In this thesis, I presented a complete system for controlling a humanoid robot using depth-camera-driven human posture and hand gesture recognition. The main contributions are summarized as follows:
- I implemented a static posture recognition method based on skeleton vector angles, which achieved high accuracy and real-time performance. The method is robust to variations in user height and body size because it relies on angular information rather than absolute positions.
- I proposed a long-distance hand gesture recognition algorithm that combines color and depth information. The hand region is extracted using a depth-color intersection and interpolation technique, and the hand skeleton is computed using a robust thinning algorithm. Hu moments of the skeleton provide rotation- and scale-invariant features. An SVM classifier with an RBF kernel achieves about 78% accuracy in the 2–4 meter range, significantly outperforming contour-based features.
- I designed the gait planning algorithm for the humanoid robot using D-H kinematic modeling and cubic spline trajectories. The inverse kinematics solution allows precise control of the ankle position, and the stability verification method protected the servos from overload.
- I successfully integrated the recognition and control modules, enabling the humanoid robot to respond to user postures and gestures in real time. The system works reliably across a range of 0.8 to 4 meters, and the response time is acceptable for interactive applications.
Despite the successful implementation, there are several limitations and areas for future improvement:
- The posture recognition only handles static postures. Dynamic action recognition using temporal models such as Hidden Markov Models (HMM) or recurrent neural networks could greatly expand the vocabulary of commands.
- The hand gesture accuracy, especially for similar hand shapes, can be improved by using deep convolutional neural networks or by increasing the training dataset size. Also, depth-based finger tracking could provide more distinguishable features.
- The humanoid robot currently executes only pre-programmed motion sequences. Adding onboard sensors such as gyroscopes and accelerometers would enable closed-loop balance control and terrain adaptation.
- The system does not yet track the user’s position in space to make the robot follow the user or avoid obstacles. Integrating simultaneous localization and mapping (SLAM) with the Kinect’s point cloud data could lead to more autonomous behaviors.
- Finally, the communication latency and range could be improved by using Wi-Fi or higher-bandwidth wireless protocols instead of Bluetooth.
In conclusion, this research demonstrates a practical approach to using depth cameras for natural and intuitive control of humanoid robots. The methods and algorithms developed here provide a solid foundation for future work in human-robot interaction, assistive robotics, and intelligent automation.