1. Introduction
The advancement of artificial intelligence and robotics has driven the evolution of humanoid robots from simple mechanical structures to highly intelligent autonomous systems. Humanoid robots, with their human-like morphology and movement capabilities, are designed to operate in environments built for humans. This inherent adaptability makes them highly suitable for a wide range of applications, including disaster response, domestic assistance, industrial automation, and education. However, controlling a humanoid robot to perform complex, dynamic tasks remains a significant challenge due to their high-dimensional action spaces, complex kinematics, and the need for real-time decision-making. Traditional control methods, which rely heavily on precise mathematical modeling and rule-based logic, often struggle to provide the necessary flexibility and robustness in unstructured environments.
In recent years, deep reinforcement learning (DRL) has emerged as a powerful paradigm for developing intelligent control policies. By enabling agents to learn optimal behaviors through trial-and-error interactions with their environment, DRL has achieved remarkable success in various domains, including game playing, robotic manipulation, and autonomous navigation. The application of DRL to humanoid robot control has opened new avenues for tackling tasks that were previously intractable with classical methods. Among these tasks, basketball shooting stands out as a particularly challenging problem. It requires a seamless integration of perception, high-level decision-making, and precise low-level motor control under dynamic conditions.
The primary goal of the research presented in this thesis is to develop robust and efficient DRL-based methods for enabling a humanoid robot to learn basketball shooting skills in a simulation environment. I focus on overcoming two critical bottlenecks that hinder the practical application of DRL in this domain: low sample efficiency and slow convergence speeds, primarily caused by sparse reward signals and inefficient experience utilization.
To address these challenges, I propose two novel algorithmic frameworks. First, I introduce the PSR-SAC algorithm, an improvement over the Soft Actor-Critic (SAC) method. This approach integrates a Convolutional Block Attention Module (CBAM) into the visual representation learning pipeline to enhance the perception of crucial spatial and channel-wise features. Concurrently, I remodel the experience replay mechanism by creating separate global and success experience pools and implementing a priority-based sampling strategy that prioritizes high-value, recent, and successful experiences, thereby significantly boosting sample efficiency and accelerating convergence.
Second, to overcome the limitations of discrete action space modeling in dynamic scenarios, I propose the H-PPO algorithm, which operates in a hybrid action space. This allows the robot to simultaneously make discrete decisions about which action to perform and continuous decisions about the parameters of that action (e.g., shooting force or movement time). Furthermore, to mitigate the sparse reward problem, I integrate a YOLOv5 object detection model to provide a human-experience-based reward signal that guides the robot to align its orientation with the basket, a heuristic that dramatically reduces unproductive exploration and speeds up learning.
This thesis details the design, implementation, and experimental evaluation of these two methods within the Webots simulation environment, using the Robotis OP2 humanoid robot. My findings demonstrate significant improvements in training efficiency, convergence speed, and task success rates, providing a valuable contribution to the field of humanoid robot skill learning.
2. Related Work and Theoretical Foundations
2.1 Humanoid Robot Motion Control
The research on humanoid robot motion control can be broadly divided into two categories: classical control methods and learning-based methods. Classical methods, such as PID control, Zero Moment Point (ZMP) control, and Model Predictive Control (MPC), have been foundational in enabling stable bipedal locomotion and simple manipulation. These methods, however, depend on high-fidelity dynamic models and are sensitive to parameter uncertainties and environmental variations. They often require extensive manual tuning and lack the adaptability needed for tasks in unstructured settings.
Learning-based methods, especially DRL, have recently shown superior performance in generating complex and adaptive behaviors. DRL allows the robot to acquire skills directly from experience, without explicit environment models. This is particularly advantageous for tasks where modeling the interaction dynamics is difficult, such as multi-contact locomotion or object manipulation. I have observed that DRL-based policies can outperform classical controllers in terms of robustness and adaptability, enabling humanoid robots to perform tasks like navigating rough terrain, maintaining balance under disturbances, and executing complex whole-body movements.
2.2 Humanoid Robot Basketball Shooting
Early attempts at humanoid robot basketball shooting primarily used vision-based control systems combined with fuzzy logic or other classical algorithms. These approaches typically involve estimating the distance to the basket and applying a pre-computed or rule-based shooting force. While these methods can achieve high success rates in static, controlled environments, they are inherently limited. They are not generalizable to dynamic scenarios where the positions of the robot, ball, or basket change, as they rely on precisely calibrated parameters and do not incorporate a learning mechanism.
The shift towards DRL methods, as I propose in this thesis, represents a significant paradigm change. DRL-based methods allow for end-to-end learning from raw sensory input, such as images, directly to action commands. This eliminates the need for hand-crafted features and enables the policy to automatically adapt to changing environments. My research builds upon this concept, introducing algorithmic innovations to make the learning process for such a complex task more efficient and reliable.
2.3 Key Algorithmic Improvements in DRL
The advancement of DRL is closely tied to innovations in algorithm design. Two key areas of improvement relevant to my work are experience replay optimization and mixed action space modeling.
**Experience Replay**: Standard off-policy algorithms like DQN and SAC use uniform random sampling from a replay buffer. This approach is sample-inefficient, as it treats all transitions with equal importance. I have reviewed several improvements based on this concept. Prioritized Experience Replay (PER) uses TD-error to assign sampling probabilities, prioritizing transitions with larger prediction errors. Other methods, such as ER-MS, mix samples from a global buffer with the most recent episodes, while methods like PEC-SAC further emphasize the importance of recent experiences. My proposed PSR-SAC method synthesizes these ideas, combining prioritized sampling with a dedicated success experience pool and the mandatory inclusion of the most recent episode’s data. This combination ensures that the agent learns from the most informative, relevant, and current data, which is crucial for effectively mastering a multi-stage task like basketball shooting.
**Hybrid Action Spaces**: Many real-world tasks, including basketball shooting, inherently require both discrete decisions (e.g., which type of action to execute: move, turn, shoot) and continuous control (e.g., the force or duration of that action). Traditional methods either discretize the continuous parameters, losing precision, or ignore the hierarchical relationship between the two. Algorithms designed for parameterized action spaces, such as P-DQN, MP-DQN, and H-PPO, offer elegant solutions by having the policy network output both discrete and continuous components. My work applies and adapts the H-PPO algorithm to the domain of humanoid basketball shooting, which not only validates the algorithm’s applicability in a complex 3D environment but also contributes to understanding how to effectively model and solve such tasks.
3. PSR-SAC for Humanoid Robot Basketball Shooting
This chapter details the first major contribution of my research: the PSR-SAC algorithm for humanoid robot basketball shooting. The primary focus is on improving the sample efficiency and convergence speed of DRL algorithms. The overall framework is designed to enable a humanoid robot to learn the complete basketball shooting procedure by subdividing it into separate, trainable tasks.
3.1 Problem Definition and Task Decomposition
The complete mission of a humanoid robot in a basketball shooting competition involves multiple sequential sub-goals. The robot must first locate the ball, move to it, pick it up, and finally shoot it into the basket. Based on the FIRA HuroCup competition rules, a simulation environment was constructed using Webots. The entire process is not a single monolithic task; instead, I divided it into three distinct sub-tasks to simplify the learning process for the agent:
1. **Sub-task 1: Approaching the Ball**. The objective is for the robot to navigate to a position directly in front of the ball.
2. **Sub-task 2: Picking Up the Ball**. From the position in front of the ball, the robot must manipulate its arms to secure the ball.
3. **Sub-task 3: Shooting the Ball**. With the ball in hand, the robot must adjust its position and execute a shooting motion to throw the ball into the basket.
This decomposition creates a modular and scalable training pipeline. Each sub-task is formulated as an independent Markov Decision Process (MDP), and a distinct DRL agent is trained to master its respective objective. When performing the full task, the agents are invoked sequentially, creating a complete autonomous skill chain for the humanoid robot.
3.2 Markov Decision Process Modeling
For each sub-task, the interaction between the Robotis OP2 robot and its environment is modeled as an MDP, defined by the tuple $(S, A, P, R, \gamma)$.
**State Space (S)**: The state representation is crucial for the success of a vision-based DRL algorithm. The robot perceives its environment through its head-mounted camera, which captures RGB images. However, raw images are high-dimensional and contain redundant information. A multi-step preprocessing pipeline is used to create a compact and effective state representation:
– **Grayscale Conversion**: The 160×120 RGB image is converted to a single-channel grayscale image to reduce dimensionality while retaining essential structural information.
– **Resizing**: The grayscale image is resized to 84×84 pixels, a standard input size for DRL algorithms, balancing the need for detail with computational efficiency.
– **Frame Stacking**: A single image only provides partial observability. To incorporate temporal dynamics, such as the movement of the ball or the robot’s own velocity, I stack the last four consecutive grayscale frames. This creates a state with the shape 4×84×84, providing a richer representation of the environment.
– **Normalization**: Pixel values are normalized to the range [0, 1] to accelerate the learning process and improve numerical stability.
**Action Space (A)**: For the three sub-tasks in this chapter, a discrete action space is designed. The actions are implemented using the Webots API for the Robotis OP2, specifically the `Gait Manager` for locomotion and the `Motion Manager` for upper-body manipulation.
– **Sub-task 1 (Approach)** : The action space includes `forward`, `turn_left`, and `turn_right`. The parameters for these actions, implemented via the `setXAmplitude`, `setAAmplitude`, and `setYAmplitude` APIs, are preset as shown in Table 1.
**Table 1: Parameters for Locomotion Actions**
| Action Name | setXAmplitude | setAAmplitude | setYAmplitude |
| :— | :— | :— | :— |
| forward | 0.3 | 0 | 0.5 |
| backward | -0.3 | 0 | 0.5 |
| turn_left | 0.3 | 0.3 | 0.5 |
| turn_right | 0.3 | -0.3 | 0.5 |
– **Sub-task 2 (Picking)** : The action space includes `forward`, `backward`, `turn_left`, `turn_right`, `pick_up_ball`, and `secure_ball`.
– **Sub-task 3 (Shooting)** : The action space includes `forward`, `turn_left`, `turn_right`, and `shoot`.
**Reward Function (R)** : A sparse reward function is implemented. The robot receives a substantial positive reward for achieving the sub-task’s primary goal. A small negative reward is given if the episode fails or exceeds the maximum number of steps. The reward function is defined by Equation (3-1):
$$R(s, a) = \begin{cases} +100, & \text{if sub-task goal is achieved} \\ -10, & \text{if step limit exceeded} \\ -5, & \text{if the ball is knocked over} \\ 0, & \text{otherwise} \end{cases}$$
3.3 Visual Representation Learning with CBAM
The performance of a vision-based DRL agent is directly linked to the quality of the visual features extracted. I observed that a simple Convolutional Neural Network (CNN) might not be sufficient to focus on the most relevant objects, such as the ball and the basket, especially in the presence of background clutter. To address this, I enhanced the visual representation module by embedding the Convolutional Block Attention Module (CBAM) into the CNN architecture.
CBAM is a lightweight and effective attention module that sequentially infers attention maps along two independent dimensions: channel and spatial. It allows the network to emphasize relevant feature channels and focus on informative spatial regions. Given an intermediate feature map $F \in \mathbb{R}^{C \times H \times W}$, CBAM computes a channel attention map $M_c \in \mathbb{R}^{C \times 1 \times 1}$ and a spatial attention map $M_s \in \mathbb{R}^{1 \times H \times W}$. The process can be described by Equation (2-4):
$$F’ = M_c(F) \otimes F, \quad F” = M_s(F’) \otimes F’$$
The architecture of my proposed visual representation module is illustrated conceptually. The module begins with a series of convolutional layers to extract low-level and mid-level features. Crucially, after the initial feature extraction, the CBAM module is inserted. The channel attention module applies average and max pooling to the input feature map, aggregates these descriptors through a shared multi-layer perceptron, and generates a channel attention weight vector. This allows the network to selectively amplify important features. The spatial attention module, on the other hand, applies channel-wise pooling and a convolutional layer to generate a spatial attention weight map, enabling the network to focus on the location of important objects. The final output is a refined feature map that is more informative and task-relevant.
The CNN structure used in my experiments is detailed in Table 2, culminating in a 64×7×7 feature map before being processed by the attention module. This integration resulted in more focused and discriminative features, allowing the agent to make more accurate decisions.
**Table 2: CNN Feature Extraction Architecture**
| Layer | Input Channels | Output Channels | Kernel Size | Activation |
| :— | :— | :— | :— | :— |
| Conv 1 | 4 | 32 | 8, stride 4 | ReLU |
| Conv 2 | 32 | 64 | 4, stride 2 | ReLU |
| Conv 3 | 64 | 64 | 3, stride 1 | ReLU |
| CBAM | 64 | 64 | – | – |
3.4 PSR-SAC Algorithm
The core of my proposed method lies in the novel experience replay mechanism, which I term PSR (Priority Success Recent Experience Replay). The standard SAC algorithm draws samples uniformly from a single replay buffer. This fails to distinguish between important and unimportant transitions, leading to poor sample efficiency. To overcome this, PSR introduces a structured sampling strategy that leverages the learning signals more effectively.
The mechanics of the PSR strategy are as follows:
1. **Prioritized Sampling**: The method uses TD-error as the primary metric for assigning priority to each experience sample. This ensures that transitions with a larger discrepancy between the predicted and target Q-values are sampled more frequently, prompting the agent to learn more from its “mistakes”. A SumTree data structure efficiently manages these priorities.
2. **Success Experience Pool**: A separate buffer, dedicated solely to storing transitions from successful episodes, is created. This is based on the intuition that successful experiences are the most direct and valuable indicators of good behavior. By prioritizing sampling from this pool, the agent is consistently reinforced towards strategies that lead to task completion.
3. **Recent Experience Inclusion**: The most recent episode’s transitions are directly included in every training update’s mini-batch. This ensures that the agent explores and learns from the most current policy’s behavior, facilitating faster adaptation and improving stability.
The three components are combined to form a mini-batch. With a mini-batch size of 512, a proportion of the samples (e.g., 10%) is drawn from the success experience pool, while the rest is drawn from the global experience pool, both using prioritized sampling based on TD-error. The entire content of a temporary buffer holding the last episode’s transitions is also incorporated into the mini-batch. The complete PSR-SAC algorithm is detailed in Algorithm 1.
**Algorithm 1: PSR-SAC**
Input: Environment state $s$, Initial network parameters, Success sample ratio $\alpha$, Entropy coefficient $\beta$, Target network update frequency $K$, Start training time $T_0$, Target network update rate $\tau$
Output: Optimal policy network parameter $\theta^*$
1. Initialize Actor network $\mu_\theta$.
2. Initialize two Critic networks $Q_{\phi_1}, Q_{\phi_2}$.
3. Initialize capacity $N$ global experience buffer $D_{global}$ and success buffer $D_{success}$.
4. Initialize capacity $B$ temporary storage $D_{recent}$ for the most recent episode.
5. **for** $t = 1$ **to** $T$ **do**
6. Get action from policy: $a_t \sim \pi_\theta(\cdot|s_t)$.
7. Interact with the environment; store transition $(s_t, a_t, r_t, s_{t+1})$ in $D_{recent}$.
8. **if** $t \% B == 0$ **then**
9. Transfer data from $D_{recent}$ to $D_{global}$.
10. **if** episode success **then**
11. Transfer data from $D_{recent}$ to $D_{success}$.
12. **end if**
13. **end if**
14. **if** $t > T_0$ **then**
15. Initialize a temporary mini-batch $M_{temp}$ with data from $D_{recent}$.
16. Sample prioritized data from $D_{global}$ into $M_{temp}$.
17. Sample prioritized data from $D_{success}$ into $M_{temp}$.
18. Calculate Critic loss and update Critic networks.
19. Calculate Actor loss and update Actor network.
20. Update target Critic networks.
21. Clear $D_{recent}$.
22. **end if**
23. **end for**
3.5 Experimental Results
The experiments were conducted in the Webots simulation environment using the Robotis OP2 model. I compared my proposed CBAM-PSR-SAC algorithm against standard DRL baselines: DQN, PPO, and SAC. The algorithms were evaluated on all three sub-tasks, in both a fixed ball position and a random ball position scenario. Performance was measured using three metrics: average episode reward, convergence speed, and task success rate.
The training parameters for the PSR-SAC algorithm are presented in Table 3.
**Table 3: Training Parameters for PSR-SAC**
| Parameter | Value |
| :— | :— |
| Training Steps | 30000 |
| Global Experience Pool Size | 8192 |
| Success Experience Pool Size | 4096 |
| Discount Factor | 0.99 |
| Mini-batch Size | 512 |
| Start Training Time | 512 |
| Learning Rate | 0.0003 |
| Priority Importance Weight | 0.7 |
| Importance Sampling Correction Weight | 0.4 |
| Target Network Update Frequency | 50 |
| Success Sample Ratio | 0.1 |
The results consistently demonstrated that the CBAM-PSR-SAC algorithm outperformed the baseline methods. Table 4 and Table 5 display the convergence speed and success rate results, respectively.
**Table 4: Convergence Speed Comparison (Episodes to Convergence)**
| Algorithm | Fixed Ball Position | Random Ball Position |
| :— | :— | :— |
| | Sub-task 1 | Sub-task 2 | Sub-task 3 | Sub-task 1 | Sub-task 2 | Sub-task 3 |
| DQN | 600 | 670 | 520 | 1050 | 750 | 7800 |
| PPO | 780 | 1000 | 930 | 1330 | 1800 | 10000 |
| SAC | 400 | 120 | 300 | 1120 | 600 | 7000 |
| **CBAM-PSR-SAC** | **380** | **120** | **280** | **980** | **420** | **6500** |
Analysis of Table 4 shows that in the more challenging random ball position scenario, my method improved convergence speed by 12.5% for Sub-task 1, 30% for Sub-task 2, and 7.1% for Sub-task 3, compared to the standard SAC algorithm.
**Table 5: Task Success Rate Comparison (%)**
| Algorithm | Fixed Ball Position | Random Ball Position |
| :— | :— | :— |
| | Sub-task 1 | Sub-task 2 | Sub-task 3 | Sub-task 1 | Sub-task 2 | Sub-task 3 |
| DQN | 95.1 | 56.6 | 96.5 | 90.1 | 56.8 | 55.6 |
| PPO | 82.5 | 92.5 | 98.6 | 82.9 | 48.5 | 41.3 |
| SAC | 92.7 | 99.6 | 98.2 | 93.4 | 60.1 | 48.8 |
| **CBAM-PSR-SAC** | **99.6** | **99.9** | **99.3** | **96.8** | **68.4** | **58.2** |
The success rate results in Table 5 further confirm the superiority of my approach. In the random ball position scenario, CBAM-PSR-SAC improved the success rate by 3.4%, 8.3%, and 9.4% for Sub-tasks 1, 2, and 3, respectively, compared to SAC.
**Ablation Study on Attention Mechanism**: I performed an ablation study to isolate the contribution of the CBAM module. The CBAM-SAC variant was compared against other representation learning methods. As shown in Table 6, CBAM-SAC achieved the highest success rates across all sub-tasks, demonstrating the effectiveness of the attention module in enhancing perception.
**Ablation Study on Experience Replay**: Similarly, I conducted an ablation study on the experience replay mechanism, comparing PSR-SAC against other replay strategies like PER, ER-MS, and PEC. The results in Table 7 show that PSR-SAC led to superior success rates, particularly for the more complex Sub-task 3, validating the design choice of combining prioritized, successful, and recent experiences.
4. H-PPO for Humanoid Robot Basketball Shooting
This chapter presents my second major contribution, the H-PPO algorithm, which directly addresses the limitations of the discrete action space design used in the previous chapter, specifically for the shooting sub-task. I argue that the low success rate in dynamic scenarios is due to the discrete action space, which lacks the ability to modulate continuous parameters like the shooting force or the duration of movement. Furthermore, the sparse reward problem is tackled by integrating a human-experience-based reward shaping method.
4.1 Problem Definition and Hybrid Action Space
The core of this chapter is the transition from a purely discrete action space to a hybrid or parameterized action space for the shooting sub-task. The task is modeled as a Parameterized Markov Decision Process (PAMDP). In this framework, the action space is defined as a set of discrete actions $A_d = \{a_1, a_2, \dots, a_n\}$, where each discrete action $a$ is associated with a vector of continuous parameters $x^a \in \mathbb{R}^{m_a}$.
For the basketball shooting sub-task, the discrete actions and their corresponding continuous parameters are defined as follows:
1. **Move Forward**: A discrete action that moves the robot forward. It is associated with a continuous parameter representing the time of execution, which directly correlates with the distance traveled.
2. **Turn Left**: A discrete action to rotate left, with a continuous parameter for the rotation time.
3. **Turn Right**: A discrete action to rotate right, with a continuous parameter for the rotation time.
4. **Shoot**: A discrete action to perform the shooting motion, with a continuous parameter that controls the velocity of the ShoulderL joint, effectively controlling the shooting force.
The design is summarized in Table 8.
**Table 8: Hybrid Action Space Design for Shooting Sub-task**
| Discrete Action | Continuous Parameter | Range | Mapping Function |
| :— | :— | :— | :— |
| Move Forward | Movement Time | [0.2, 0.4] | $t = 0.3 + 0.1 \cdot p$ |
| Turn Left | Movement Time | [0.2, 0.4] | $t = 0.3 + 0.1 \cdot p$ |
| Turn Right | Movement Time | [0.2, 0.4] | $t = 0.3 + 0.1 \cdot p$ |
| Shoot | Shooting Force | [10, 15] | $v = 12.5 + 2.5 \cdot p$ |
In this design, the policy network outputs a scalar parameter $p \in [-1, 1]$, which is then mapped to the actual continuous parameter value. This allows the policy to make fine-grained adjustments to its execution, which is essential for adapting to varying distances to the basket. The linear mapping function (Equation 4-2) is used for this transformation.
$$y = (y_{max} – y_{min}) \cdot \frac{p – p_{min}}{p_{max} – p_{min}} + y_{min}$$
where $y$ is the actual parameter, $p$ is the network output in $[-1, 1]$, $y_{min}$ and $y_{max}$ are the minimum and maximum values of the target parameter range.
4.2 Network Architecture and H-PPO Algorithm
To learn in this hybrid action space, I adopted and customized the H-PPO algorithm. The network architecture is an Actor-Critic structure with a shared visual perception backbone. The Critic network is a state-value network, which estimates the value of a given state. The Actor network is split into two parallel branches:
1. **Discrete Actor Network**: This branch processes the extracted features and outputs a probability distribution over the discrete actions using a Softmax activation function.
2. **Continuous Actor Network**: This branch processes the extracted features and outputs the parameters (mean and standard deviation) of a Gaussian distribution for the continuous action parameter. This ensures maximum exploration and adaptability for continuous control.
During training, the discrete action is sampled from the Categorical distribution, and the continuous parameter is sampled from the Gaussian distribution. The H-PPO algorithm calculates a separate importance sampling ratio for the discrete and continuous policies. Both ratios are used to update the respective actor networks, but they share a common advantage function computed by the Critic network, as shown in Figure 4.10.
The loss functions for the H-PPO algorithm are given by Equations (4-6), (4-7), and (4-8). The discrete actor loss is a clipped policy gradient objective, similar to standard PPO, but applied to the discrete action probability.
$$L_{discrete}(\theta) = \mathbb{E}_t \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip} \left( r_t(\theta), 1-\epsilon, 1+\epsilon \right) \hat{A}_t \right) \right]$$
where $r_t(\theta)$ is the importance sampling ratio for the discrete action. The continuous actor loss is analogous but uses the log probability of the sampled continuous action.
4.3 Human-Experience-Based Reward Design
The sparse reward problem is a significant hurdle in training agents for complex tasks. In a basketball shooting task with only a sparse reward for scoring, an agent would need an enormous number of interactions to randomly stumble upon a successful shot. To overcome this, I propose a method to densify the reward signal by injecting human knowledge about the task.
The key human heuristic is: “For a successful shot, the robot should be facing the basket.” If the basket is not in the robot’s field of view, or is not near the center of the image, the chance of scoring is very low. Therefore, I designed a reward shaping function that guides the robot to align its orientation towards the basket.
To achieve this, I trained a YOLOv5n object detection model to detect the basket in the robot’s camera images. The training data consisted of 1700 images collected from the simulation environment, annotated with bounding boxes around the basket. The model achieved a high mAP@0.5 of 0.991, demonstrating its effectiveness and suitability for real-time inference.
During the DRL training process, when the robot takes an action, the current image is passed through the YOLOv5n model to detect the basket. The distance between the center of the detected bounding box and the center of the image is computed. This distance is then converted into a reward bonus. The formula for this reward shaping term is:
$$R_{human} = \beta \cdot e^{-\alpha \cdot |d_{center}|}$$
where $\beta$ is a scaling factor, $\alpha$ is a decay rate, and $|d_{center}|$ is the horizontal distance from the center of the image, in pixels, normalized by the image width. This reward signal provides the agent with immediate feedback, encouraging it to take actions that center the basket in its view, thus drastically reducing the exploration space and accelerating learning. The total reward is therefore the sum of the original sparse reward $R_{sparse}$ and this shaped reward: $R = R_{sparse} + \lambda \cdot R_{human}$.
4.4 Experimental Results
I evaluated the proposed YOLO-H-PPO algorithm against the discrete action space baselines (DQN, PPO, and the CBAM-PSR-SAC algorithm from Chapter 3) in the shooting sub-task. The training parameters are listed in Table 9.
**Table 9: Training Parameters for H-PPO**
| Parameter | Value |
| :— | :— |
| Training Steps | 50000 |
| Simulation Steps per Update | 512 |
| Discount Factor | 0.99 |
| Mini-batch Size | 512 |
| GAE Parameter | 0.95 |
| Epochs per Update | 4 |
| Clipping Coefficient | 0.1 |
| Entropy Coefficient | 0.01 |
| Value Function Loss Weight | 0.5 |
A comparison of the average episode reward during training is shown in Figure 4.13. The results highlight the strengths of my approach, especially in the random ball position scenario. The convergence speed and success rate comparisons are presented in Table 10 and Table 11.
**Table 10: Convergence Speed Comparison for Shooting Sub-task (Episodes)**
| Algorithm | Fixed Ball Position | Random Ball Position |
| :— | :— | :— |
| DQN | 520 | 7800 |
| PPO | 930 | 10000 |
| CBAM-PSR-SAC | 280 | 6500 |
| **YOLO-H-PPO** | **780** | **5000** |
In the random ball position scenario, YOLO-H-PPO converged in 5000 episodes, which is a significant improvement over all other methods. It is 23.2% faster than CBAM-PSR-SAC and 50% faster than PPO.
**Table 11: Success Rate Comparison for Shooting Sub-task (%)**
| Algorithm | Fixed Ball Position | Random Ball Position |
| :— | :— | :— |
| DQN | 96.5 | 55.6 |
| PPO | 98.6 | 41.3 |
| CBAM-PSR-SAC | 99.3 | 58.2 |
| **YOLO-H-PPO** | **99.1** | **75.3** |
The success rate in the random scenario is the most compelling evidence for the effectiveness of the hybrid action space. YOLO-H-PPO achieved a 75.3% success rate, which is a 17.1% absolute improvement over the CBAM-PSR-SAC algorithm.
A further advantage of the hybrid action space is the efficiency of its behavior. By being able to control the time of movement, the robot can make more precise and shorter adjustments to its position. This results in a lower average number of steps required to complete the task, as shown in Table 12.
**Table 12: Average Number of Steps to Complete Shooting Task**
| Algorithm | Fixed Ball Position | Random Ball Position |
| :— | :— | :— |
| DQN | 4.62 | 5.56 |
| PPO | 6.55 | 8.42 |
| CBAM-PSR-SAC | 3.95 | 6.27 |
| **YOLO-H-PPO** | **3.68** | **4.63** |
**Ablation Study on Hybrid Action Space**: I compared H-PPO directly with MP-DQN, another hybrid action space algorithm. The results in Table 13 show that H-PPO achieves a higher success rate, confirming its superior ability to optimize within this more complex action space.
**Ablation Study on Reward Design**: I tested the contribution of the reward reshaping component by comparing YOLO-H-PPO with a sparse-reward version of H-PPO. The results are shown in Table 14. The addition of the human experience reward resulted in a 17.3% improvement in convergence speed, demonstrating its importance in guiding efficient learning without compromising the final success rate.
5. Conclusion
In this research, I conducted an in-depth study on the application of deep reinforcement learning to solve the complex task of humanoid robot basketball shooting. The primary challenges of low sample efficiency, slow convergence, discrete action space limitations, and sparse rewards were addressed through two novel algorithmic frameworks. The experiments, conducted in a high-fidelity Webots simulation environment using the Robotis OP2 humanoid robot, validated the effectiveness of both proposed methods.
First, I developed the PSR-SAC algorithm, which significantly improved the sample efficiency and learning rate of the standard SAC algorithm. This was achieved through two key innovations: the integration of the CBAM attention module into the visual feature extraction pipeline, and a novel prioritized success recent experience replay mechanism. The CBAM module enables the robot to focus on task-critical features, such as the ball and basket, while the PSR replay mechanism ensures that the agent learns from the most valuable, current, and successful data. In the dynamic random ball position scenario, the PSR-SAC algorithm demonstrated substantial improvements in convergence speed by up to 30% and in task success rate by up to 9.4% compared to the standard SAC method, underscoring its contribution to enhancing the robot’s perception and decision-making capabilities.
Second, to overcome the inherent limitations of discrete action models in dynamic environments, I proposed the H-PPO algorithm. This method employs a hybrid action space that allows for simultaneous discrete action selection and continuous parameter control, which is crucial for fine-tuning actions like shooting force and movement distance. Additionally, I incorporated a human-experience-based reward shaping strategy, using a YOLOv5 object detection model to provide dense and informative feedback that guides the robot to orient itself towards the basket. For the shooting sub-task, this approach achieved a 75.3% success rate in the random ball position scenario, a relative improvement of 17.1% over the PSR-SAC method, while also converging 23.2% faster. This result demonstrates the benefits of richer action representations and well-shaped reward functions for tackling complex dynamic control tasks.
In conclusion, this thesis provides a new algorithmic framework for skill learning in humanoid robots, specifically highlighting the advantages of hybrid action space design and external knowledge integration. The research findings have significant implications for the development of more autonomous and adaptive humanoid robots capable of performing complex manipulation tasks in real-world settings. Future work could focus on transferring these simulation-trained policies to physical robots, exploring the incorporation of more sophisticated perception models, and extending the proposed framework to even more challenging whole-body control tasks.