My research journey began with a fundamental question: how can an embodied robot, particularly a humanoid platform, acquire complex manipulation skills in dynamic environments without extensive manual programming? Traditional robotic control methodologies, while effective in structured settings, often fail when confronted with the unpredictability of real-world scenarios. This challenge becomes particularly pronounced in tasks like basketball shooting, which demands precise motor control, real-time perception, and adaptive decision-making. The convergence of deep learning and reinforcement learning offers a compelling pathway forward, enabling embodied robot systems to learn from interaction rather than explicit instruction. Throughout my master’s research, I have focused on developing novel deep reinforcement learning (DRL) algorithms specifically tailored for humanoid robot basketball shooting, addressing critical bottlenecks in sample efficiency, convergence speed, and action space modeling.
The significance of this research extends beyond the basketball court. The ability of an embodied robot to learn complex motor skills through trial-and-error interaction mirrors the fundamental processes of human skill acquisition. As articulated in policy documents, humanoid robots represent a potentially disruptive technology that could reshape global industrial landscapes. The 2023 guidelines from China’s Ministry of Industry and Information Technology explicitly highlighted the transformative potential of these machines. By advancing DRL methodologies for embodied robot control, my work contributes to a broader vision of autonomous systems capable of operating seamlessly in human-centric environments.
Technical Foundations and Problem Analysis
Before delving into algorithmic innovations, I must establish the theoretical underpinnings of my approach. The intersection of deep learning and reinforcement learning provides the mathematical and computational framework through which an embodied robot can make sequential decisions in high-dimensional state spaces. Convolutional neural networks (CNNs) serve as powerful feature extractors, transforming raw visual input into compact representations amenable to policy optimization. The attention mechanism, inspired by human visual processing, allows the network to dynamically prioritize task-relevant information, a capability I found essential for basketball shooting where the ball and hoop represent critical elements in a cluttered visual field.
Reinforcement learning formalizes the learning problem as a Markov decision process (MDP), defined by the tuple $(S, A, P, R, \gamma)$, where $S$ represents the state space, $A$ denotes the action space, $P$ is the state transition probability, $R$ is the reward function, and $\gamma$ is the discount factor. The objective of the embodied robot agent is to learn an optimal policy $\pi^*$ that maximizes the expected cumulative reward, expressed mathematically as:
$$G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$
The state-action value function $Q^\pi(s,a)$ captures the expected return starting from state $s$, taking action $a$, and subsequently following policy $\pi$:
$$Q^\pi(s,a) = \mathbb{E}_\pi[G_t | S_t = s, A_t = a]$$
Among the DRL algorithms available, I selected Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO) as foundational approaches for my research. SAC, an off-policy algorithm grounded in maximum entropy reinforcement learning, optimizes a stochastic policy while simultaneously maximizing expected return and policy entropy:
$$J(\pi) = \sum_{t=0}^{T} \mathbb{E}_{(s_t, a_t) \sim \rho_\pi}[r(s_t, a_t) + \alpha \mathcal{H}(\pi(\cdot|s_t))]$$
where $\mathcal{H}(\pi(\cdot|s_t))$ denotes the entropy of the policy at state $s_t$, and $\alpha$ is the temperature parameter controlling the trade-off between exploration and exploitation. This entropy regularization encourages exploration, which proved valuable in the early stages of embodied robot training. Conversely, PPO, an on-policy algorithm, addresses the stability challenges of policy gradient methods by constraining policy updates through a clipped surrogate objective:
$$L^{CLIP}(\theta) = \mathbb{E}_t[\min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)]$$
where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ represents the probability ratio between new and old policies. This mechanism prevents excessively large policy updates, ensuring stable training in embodied robot applications.
Research Context and Motivation
The field of humanoid robot control has evolved significantly over recent decades, transitioning from classical control theory to learning-based approaches. Traditional methods, including PID control, zero moment point (ZMP) control, and model predictive control, rely on precise mathematical modeling and predefined control rules. While these approaches achieve high precision in controlled settings, they exhibit fundamental limitations in dynamic, unstructured environments. The inverse kinematics and dynamics computations required by these methods become computationally prohibitive as the degrees of freedom increase, and their sensitivity to model inaccuracies limits real-world applicability.
For the specific task of basketball shooting, early approaches heavily depended on visual serving and fuzzy logic control. One notable system employed fuzzy logic to estimate throwing force based on distance measurements from a vision system, achieving satisfactory performance in static scenarios but demonstrating poor adaptability when environmental conditions varied. Another approach utilized Petri nets for eye-hand-leg coordination during basketball throwing, enabling modular design and improved success rates in fixed scenarios. However, these systems shared a common weakness: they required extensive manual tuning and lacked the generalization capacity for dynamic environments.
The advent of DRL fundamentally changed this landscape. By enabling embodied robot agents to learn directly from environmental interaction, DRL bypasses the need for explicit modeling and manual rule design. An end-to-end approach using Deep Q-Networks (DQN) demonstrated that a humanoid robot could learn basketball shooting skills through trial-and-error, marking a significant departure from previous methods. Yet this approach struggled with sample efficiency and convergence speed, particularly in tasks requiring thousands of interactions to achieve satisfactory performance.
These observations motivated my investigation into algorithmic improvements that could accelerate learning while maintaining robustness. I identified two critical research directions: enhancing the experience replay mechanism for off-policy algorithms and developing effective action space modeling for hybrid discrete-continuous decision problems.
PSR-SAC Algorithm for Discrete Action Space
My first major contribution addresses the twin challenges of low sample efficiency and slow convergence in DRL-based embodied robot training. I proposed the PSR-SAC algorithm (Priority Success Recent Experience Replay-SAC), which integrates attention mechanisms with an innovative experience replay strategy. This approach decomposes the complex basketball shooting task into three sub-tasks: approaching the ball, retrieving the ball, and shooting the ball. Each sub-task is modeled as a separate MDP, allowing the embodied robot to learn specialized policies for distinct phases of the overall task.
The state space configuration deserves careful consideration. Using the humanoid robot’s head-mounted camera, I capture environmental information as raw RGB images at 160×120 resolution. These images undergo a preprocessing pipeline involving grayscale conversion, scaling to 84×84 pixels, stacking to capture temporal dynamics, and normalization. The stacking of four consecutive frames provides critical temporal information, enabling the embodied robot to perceive movement trajectories of the ball and hoop that would be invisible in single-frame observations. The state representation can be formalized as:
$$s_t = \phi(o_t, o_{t-1}, o_{t-2}, o_{t-3})$$
where $\phi$ represents the preprocessing function and $o_t$ denotes the raw observation at time $t$.
The action spaces for the three sub-tasks are designed based on the Robotis OP2’s control architecture. For the approach sub-task, I define three movement actions: forward, left-turn, and right-turn. The retrieval sub-task expands this set to four movements with additional arm manipulation actions. The shooting sub-task combines movement actions with a fixed-throw action. All movements are parameterized through the Gait Manager’s API, with specific parameters controlling amplitude and direction. For instance, the forward action uses the parameter settings $setXAmplitude=0.3$, $setAAmplitude=0$, and $setYAmplitude=0.5$, executing for 0.3 seconds.
Reward design employs a sparse reward structure to minimize bias. The embodied robot receives a positive reward of +100 upon completing the sub-task objective, a penalty of -10 for exceeding the maximum episode length, and -5 for falling over. All intermediate states yield zero reward. For the approach sub-task, success is defined as the Euclidean distance between robot and ball falling below 0.12 meters:
$$d = \sqrt{(x_{ball} – x_{robot})^2 + (y_{ball} – y_{robot})^2} < 0.12$$
In designing the visual representation learning module, I innovatively incorporated the Convolutional Block Attention Module (CBAM) into the CNN architecture. Standard CNN-based feature extraction often treats all spatial locations and channels equally, failing to emphasize task-critical features. CBAM addresses this limitation through sequential channel and spatial attention mechanisms. The channel attention module computes:
$$M_c(F) = \sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))$$
where $\sigma$ denotes the sigmoid function. The spatial attention module computes:
$$M_s(F) = \sigma(f^{7\times7}([AvgPool(F); MaxPool(F)]))$$
By embedding CBAM after the third convolutional layer, the network focuses on task-relevant objects such as the ball and hoop while suppressing background distractions. The architecture processes the 4×84×84 input through three convolutional layers, generating a 64×7×7 feature map before passing through CBAM. I deliberately omitted pooling layers to preserve positional information critical for precise embodied robot control. This design achieves an effective balance between representational power and computational efficiency, essential for real-time robot inference.
My most significant innovation lies in the experience replay mechanism. Traditional SAC employs uniform random sampling from a replay buffer, treating all experiences as equally valuable. I argue that this approach overlooks the inherent heterogeneity in sample importance. Inspired by human learning processes, where individuals derive greater benefit from studying successful attempts and recent experiences, I developed a priority-based sampling strategy. The method maintains two separate replay buffers: a global experience pool storing all interactions and a success experience pool exclusively storing episodes where the sub-task was successfully completed.
The sampling process combines three sources to construct the mini-batch. First, experiences from the most recent episode are always included, recognizing the particular relevance of recency in embodied robot skill acquisition. Second, priority sampling based on TD-error is applied to both global and success experience pools. The priority of each sample is computed as:
$$p_i = |\delta_i| + \epsilon$$
where $\epsilon$ prevents zero priority and $\delta_i$ represents the TD-error. The sampling probability is then:
$$P(i) = \frac{p_i^{\alpha}}{\sum_k p_k^{\alpha}}$$
Third, importance sampling weights correct for the bias introduced by prioritized sampling:
$$w_i = \left(\frac{1}{N} \cdot \frac{1}{P(i)}\right)^{\beta}$$
This comprehensive approach ensures the embodied robot learns from high-value samples while maintaining diversity in training data. The balanced sampling ratio is controlled by a parameter $\rho$, which determines the proportion of samples drawn from the success experience pool versus the global pool.
The complete PSR-SAC algorithm workflow proceeds as follows: the embodied robot interacts with the environment, generating experiences stored in the global buffer. Successful episodes are additionally stored in the success buffer. During network updates, experiences from the recent episode are directly included in the mini-batch, followed by prioritized sampling from both buffers according to the configured proportion. The sampled experiences update the Actor network, two Critic networks, and their targets. For the discrete action space, I adapt the Soft Q-target calculation to consider the complete action distribution:
$$Q_{target} = r + \gamma \left[V(s’) – \alpha \log \pi(a’|s’)\right]$$
where $V(s’)$ is evaluated by summing $Q(s’, a’)$ over all possible actions weighted by their probabilities. This formulation provides more stable value estimation compared to single-sample Monte Carlo approximation.
Extensive experiments validate the effectiveness of PSR-SAC across all three sub-tasks. In the fixed ball position scenario, PSR-SAC achieves convergence in 380 episodes for Sub-task 1, compared to 400 episodes for standard SAC, representing a 5% improvement. More significantly, in the random ball position scenario, PSR-SAC converges in 980 episodes versus 1120 for SAC in Sub-task 1 (12.5% improvement), 420 versus 600 for Sub-task 2 (30% improvement), and 6500 versus 7000 for Sub-task 3 (7.1% improvement).
| Algorithm | Sub-task 1 (Fixed) | Sub-task 1 (Random) | Sub-task 2 (Fixed) | Sub-task 2 (Random) | Sub-task 3 (Fixed) | Sub-task 3 (Random) |
|---|---|---|---|---|---|---|
| DQN | 600 | 1050 | 670 | 750 | 520 | 7800 |
| PPO | 780 | 1330 | 1000 | 1800 | 930 | 10000 |
| SAC | 400 | 1120 | 120 | 600 | 300 | 7000 |
| PSR-SAC | 380 | 980 | 120 | 420 | 280 | 6500 |
Testing of the trained models over 1000 episodes reveals consistent success rate improvements. In the random ball position scenario, PSR-SAC achieves 96.8% success in Sub-task 1 (versus 93.4% for SAC), 68.4% in Sub-task 2 (versus 60.1% for SAC), and 58.2% in Sub-task 3 (versus 48.8% for SAC). These improvements demonstrate that the combination of attention-enhanced perception and priority-based experience replay significantly enhances embodied robot learning efficiency and task performance.
The CBAM attention mechanism alone contributes measurable benefits. When comparing CBAM-SAC against standard SAC and other representation learning methods including DrQ-SAC, ResNet-SAC, and VAE-SAC, my approach consistently achieves superior convergence and success rates. Specifically, CBAM-SAC attains 96.3% success in Sub-task 1, 65.8% in Sub-task 2, and 56.2% in Sub-task 3 under random ball positions, outperforming all comparison methods. The ablation study confirms that attention mechanism enables the embodied robot to focus computational resources on task-relevant spatial regions, extracting more discriminative features for decision-making.
H-PPO Algorithm for Hybrid Action Space
Despite the successes of PSR-SAC, I identified a fundamental limitation in discrete action space modeling. In the shooting sub-task, the embodied robot operating in random ball position scenarios faced a challenging problem: how to determine the appropriate throwing force when the distance to the hoop varies. A fixed-throw action defined in discrete space cannot adapt to these variations. This realization motivated my second research contribution: the H-PPO algorithm with hybrid action space modeling.
Hybrid action spaces, also known as parameterized action spaces, combine discrete action selection with continuous parameter refinement. Formally, given a set of discrete actions $A_d = \{a_1, a_2, …, a_n\}$, each discrete action $a_i$ is associated with a continuous parameter vector $p_i \in \mathbb{R}^{d_i}$. A complete hybrid action is represented as the tuple $(a, p_a)$ where $a \in A_d$ and $p_a$ is the parameter vector for action $a$. The hybrid action space is defined as:
$$\mathcal{A} = \{(a, p_a) | a \in A_d, p_a \in \mathcal{P}_a\}$$
For the embodied robot basketball shooting task, I defined four discrete actions: forward movement, left turn, right turn, and shooting. The movement actions are parameterized by execution time, controlling the distance traveled. The shooting action is parameterized by throwing force, which directly influences the ball’s trajectory. This parameterization is illustrated through the velocity control mechanism, where the policy network outputs a normalized value in [-1, 1] that is subsequently transformed to actual velocity parameters:
$$v_{actual} = v_{min} + \frac{(v_{max} – v_{min})(v_{norm} + 1)}{2}$$
where $v_{min}=10$ and $v_{max}=15$ represent the feasible range determined through experimental analysis.
The H-PPO network architecture employs an Actor-Critic framework with separate discrete and continuous actor networks. The Critic network, operating as a state-value function $V(s)$, estimates the expected return from state $s$. This design choice avoids the complexity of Q-function approximation in hybrid action spaces. The discrete Actor network outputs a probability distribution over the discrete action set using Softmax activation:
$$\pi_d(a|s) = \text{Softmax}(f_{actor\_d}(s))$$
while the continuous Actor network outputs the mean and standard deviation of a Gaussian distribution for parameter sampling:
$$\pi_c(p|s) = \mathcal{N}(\mu(s), \sigma(s))$$
Both networks share the same visual feature extraction backbone but maintain independent output heads. This architecture enables simultaneous optimization of discrete action selection and continuous parameter refinement.
The H-PPO algorithm optimizes its objective through separate importance sampling ratios for discrete and continuous components:
$$\mathcal{L}^{CLIP}(\theta_d, \theta_c) = \mathbb{E}_t\left[\min\left(r_t^d r_t^c \hat{A}_t, \text{clip}\left(r_t^d, 1-\epsilon, 1+\epsilon\right)\text{clip}\left(r_t^c, 1-\epsilon, 1+\epsilon\right)\hat{A}_t\right)\right]$$
where $r_t^d$ and $r_t^c$ are the importance sampling ratios for discrete and continuous policies respectively, and $\hat{A}_t$ is the advantage estimate computed using Generalized Advantage Estimation (GAE):
$$\hat{A}_t = \sum_{l=0}^{T-t-1} (\gamma\lambda)^l \delta_{t+l}$$
with $\delta_t = r_t + \gamma V(s_{t+1}) – V(s_t)$.
A critical advancement in this research phase was the integration of human experience into reward function design. Traditional sparse rewards provide feedback only upon task completion, leading to inefficient exploration in large state spaces. To address this, I employed YOLOv5 object detection to provide real-time feedback on hoop position in the robot’s visual field. The intuitive human insight is that an embodied robot is more likely to score when the hoop is centered in its field of view. This knowledge is encoded into an auxiliary reward component:
$$r_{human} = \beta \cdot \exp\left(-\frac{d_{center}^2}{\tau}\right)$$
where $d_{center}$ represents the horizontal distance between the detected hoop center and image center, $\beta$ scales the reward magnitude, and $\tau$ controls the decay rate. The total reward combines the sparse task reward with the human experience reward:
$$r_{total} = w_1 \cdot r_{task} + w_2 \cdot r_{human}$$
To implement this design, I trained a YOLOv5n model, selecting the nano variant for its computational efficiency suitable for embedded embodied robot platforms. The training dataset consisted of 1700 images collected during robot basketball training sessions, annotated with hoop bounding boxes. The dataset was split into training (1360 images), validation (170 images), and test sets (170 images). After 400 epochs of training, the model achieved a mAP@0.5 of 0.991 and mAP@0.5:0.95 of 0.862 with a precision of 0.98 and recall of 0.99, demonstrating reliable hoop detection across various viewing angles.
| Metric | Value |
|---|---|
| mAP@0.5 | 0.991 |
| mAP@0.5:0.95 | 0.862 |
| Precision | 0.98 |
| Recall | 0.99 |
| Best Epoch | 137 |
The complete YOLO-H-PPO framework integrates these components seamlessly. During each interaction step, the embodied robot captures visual observations, processes them through the feature extraction backbone, and obtains both discrete action decisions and continuous action parameters from the policy networks. The selected hybrid action is executed in the simulation environment, and the YOLOv5 model simultaneously evaluates the hoop position to compute the human experience reward. This comprehensive reward signal, combining sparse task feedback with dense human guidance, accelerates learning by directing the embodied robot toward behaviors known to increase task success probability.
Experimental results demonstrate the substantial benefits of the hybrid action space design combined with human experience rewards. In the random ball position scenario, YOLO-H-PPO achieves convergence after 5000 training episodes, compared to 7800 for DQN, 10000 for PPO, and 6500 for CBAM-PSR-SAC. This represents a 23.2% improvement in convergence speed over my previous algorithm. The success rate improves correspondingly to 75.3%, representing a 17.1% improvement over CBAM-PSR-SAC (58.2%) and 34% over standard PPO (41.3%).
| Algorithm | Convergence Episodes (Fixed) | Convergence Episodes (Random) | Success Rate (Fixed) | Success Rate (Random) | Avg. Steps (Random) |
|---|---|---|---|---|---|
| DQN | 520 | 7800 | 96.5% | 55.6% | 5.56 |
| PPO | 930 | 10000 | 98.6% | 41.3% | 8.42 |
| CBAM-PSR-SAC | 280 | 6500 | 99.3% | 58.2% | 6.27 |
| YOLO-H-PPO | 780 | 5000 | 99.1% | 75.3% | 4.63 |
Additional analysis reveals the efficiency gains of hybrid action modeling. The YOLO-H-PPO algorithm accomplishes the shooting task in an average of 4.63 steps under random ball positions, compared to 6.27 steps for CBAM-PSR-SAC and 8.42 for PPO. This reduction stems from the adaptive movement duration and throwing force provided by continuous parameters, allowing the embodied robot to adjust its actions based on the current spatial configuration rather than executing fixed-parameter actions.
Temporal analysis of the training process reveals interesting dynamics. In the initial 1000 episodes, the embodied robot demonstrates erratic behavior, frequently misaligning its body orientation and selecting inappropriate throwing forces. By episode 10000, however, the robot exhibits smooth, purposeful actions—adjusting its angle to face the hoop, calibrating throwing force to the measured distance, and consistently executing successful shots. This progression illustrates the effectiveness of the combined algorithmic innovations in facilitating skill acquisition.
Ablation and Comparative Analysis
To isolate the contributions of each algorithmic component, I conducted systematic ablation studies. The human experience reward design was evaluated independently by comparing PPO with and without this component. Under sparse rewards, PPO required approximately 6200 episodes for convergence, while YOLO-enhanced PPO converged in 4000 episodes—a 35.5% improvement. For H-PPO, the human experience reward accelerated convergence from 8100 to 6700 episodes, representing a 17.3% improvement. Importantly, the reward design did not adversely affect success rates; both PPO and H-PPO showed minor improvements in task success with the enhanced reward signal.
The hybrid action space design was validated through comparison with state-of-the-art methods for parameterized actions. In the random ball position scenario, H-PPO achieved a success rate of 73.6%, substantially outperforming MP-DQN at 63.4% and discrete action space algorithms (DQN: 55.6%, PPO: 41.3%). Under fixed ball positions, all algorithms approached near-perfect performance, suggesting that the advantages of hybrid action modeling become particularly pronounced in dynamic scenarios requiring adaptive parameter selection.
| Algorithm | Action Space | Reward Design | Convergence (Episodes) | Success Rate (Random) |
|---|---|---|---|---|
| PPO | Discrete | Sparse | 10000 | 41.3% |
| YOLO-PPO | Discrete | Human Experience | 6200 | 43.2% |
| H-PPO | Hybrid | Sparse | 8100 | 73.6% |
| YOLO-H-PPO | Hybrid | Human Experience | 6700 | 75.3% |
| MP-DQN | Hybrid | Sparse | 8500 | 63.4% |
The parameter sensitivity analysis for the success experience sampling ratio $\rho$ in the PSR-SAC algorithm revealed interesting insight about the balance between exploration and exploitation. When $\rho$ was set to 0.9 or 0.5, the algorithm converged to local optima due to excessive prioritization of successful experiences. Setting $\rho$ to 0.1 achieved the optimal balance, utilizing successful experiences for guidance while maintaining sufficient exploration diversity through global experience sampling. When $\rho$ equaled 0, excluding successful experiences altogether slowed convergence.
Discussion and Future Directions
My research demonstrates that algorithmic innovations specifically designed for embodied robot learning can yield substantial improvements in both learning efficiency and task performance. The PSR-SAC algorithm addresses the fundamental challenge of sample efficiency through a multi-faceted approach combining attention-enhanced perception and intelligent experience selection. The H-PPO algorithm extends this work by enabling more expressive action modeling through hybrid discrete-continuous parameterization, coupled with human experience integration for accelerated learning.
Several limitations and future research directions deserve consideration. First, my experiments were conducted exclusively in simulation using the Webots platform. While simulation provides valuable controlled environments for algorithm development, transferring these policies to physical embodied robot platforms introduces challenges including the simulation-to-reality gap, sensor noise, and actuator dynamics. Future research should investigate domain randomization and system identification techniques to facilitate successful deployment on physical hardware.
Second, the low-level actions in my framework retain some degree of manual design through Gait Manager and Motion Manager control modules. A promising direction involves training the embodied robot to generate these primitive actions through hierarchical reinforcement learning, where higher-level policies provide guidance while lower-level policies learn motor primitives from first principles. This approach could enhance flexibility and reduce dependence on human-designed motion templates.
Third, the reward function designs currently rely on task-specific domain knowledge. For broader applicability, research into meta-learning and reward learning from demonstration could enable the embodied robot to infer reward functions autonomously, reducing the expert knowledge required for new tasks. Techniques such as inverse reinforcement learning and preference-based learning offer intriguing alternatives to manual reward engineering.
Fourth, the computational efficiency of the proposed algorithms requires optimization for real-time embedded hardware deployment. The integration of model-based planning with model-free learning could reduce sample requirements while maintaining policy performance. Additionally, exploring efficient network architectures such as depthwise separable convolutions or neural architecture search could further reduce computational overhead.
Finally, extending the proposed framework to more complex multi-skill tasks involving longer horizons and multiple sub-goals represents an exciting frontier. The ability of an embodied robot to compose learned skills in novel sequences could lead to highly adaptive behaviors in unstructured environments.
Conclusion
This thesis investigates deep reinforcement learning methodologies for humanoid robot basketball shooting, addressing the core challenges of sample efficiency, convergence speed, action space modeling, and reward design. Two complementary algorithms emerge from this research: PSR-SAC, which enhances off-policy learning through attention-based perception and priority-based experience replay, and H-PPO, which enables adaptive action selection through hybrid action space modeling and human experience integration.
The PSR-SAC algorithm contributes innovations in two key aspects. First, the integration of CBAM attention mechanism into the visual representation learning module enables the embodied robot to focus on task-critical features, improving perception efficiency. Second, the priority-based sampling strategy, combining global experience diverse sampling with success experience targeted reuse and recent episode prioritization, significantly enhances sample utilization. Experimental results demonstrate consistent improvements in convergence speed (7.1% to 30%) and task success rates (3.4% to 9.4%) across three basketball subtasks compared to standard SAC.
The H-PPO algorithm addresses the limitations of discrete action spaces in dynamic scenarios through three mechanisms. The hybrid action space modeling enables simultaneous selection of discrete actions and their continuous parameters, providing flexibility necessary for adaptive shooting. The YOLOv5-based human experience reward design introduces dense feedback signals that guide the embodied robot toward task-relevant behaviors. The integrated system achieves a 23.2% improvement in convergence speed and 17.1% improvement in success rate over the discrete action space baseline in random ball position scenarios.
Beyond the specific basketball application, my research contributes to the broader field of embodied robot learning by demonstrating effective strategies for addressing common DRL challenges. The hybrid action space design paradigm offers a generalizable framework applicable to various robotic manipulation tasks requiring both discrete mode selection and continuous parameter tuning. The human experience integration methodology illustrates how domain knowledge can be systematically incorporated into reinforcement learning frameworks to accelerate learning without constraining policy expressiveness.
The results of this thesis validate the feasibility of DRL-based approaches for complex humanoid robot skill acquisition in simulation environments. As algorithmic improvements continue to enhance sample efficiency and robustness, the transition from simulation to real-world embodied robot deployment becomes increasingly feasible, paving the way for autonomous systems capable of learning diverse manipulation skills in human-centered environments.