
1. Introduction
The field of robotics has witnessed a paradigm shift over the past decade, moving from traditional industrial manipulators toward autonomous systems capable of operating in unstructured, dynamic environments. Among various robotic platforms, the embodied robot—particularly the humanoid form factor—has attracted significant attention due to its anthropomorphic structure and potential for seamless human-robot interaction. Unlike wheeled or quadruped robots, an embodied robot with bipedal locomotion offers superior adaptability in environments designed for humans, such as stairs, narrow corridors, and uneven terrains. However, this structural advantage comes at the cost of tremendous control complexity, involving high-dimensional action spaces, strict dynamic balance constraints, and multimodal sensory integration.
Traditional approaches to humanoid embodied robot control have predominantly relied on model-based techniques, including zero moment point (ZMP) based walking, inverse kinematics (IK) solvers, and linear quadratic regulators (LQR). While these methods provide rigorous guarantees under well-defined conditions, they often fail to generalize to real-world scenarios where model parameters are uncertain and environmental dynamics are unpredictable. Moreover, manually designing control rules for complex tasks such as basketball shooting requires substantial expert knowledge and extensive tuning, which limits scalability and adaptability.
In recent years, deep reinforcement learning (DRL) has emerged as a powerful alternative for embodied robot control. By enabling an agent to interact with its environment and learn optimal policies through trial and error, DRL has demonstrated remarkable success in various domains, including locomotion, manipulation, and autonomous navigation. The key advantage of DRL lies in its ability to handle high-dimensional sensory inputs and discover sophisticated control strategies without explicit models. Nevertheless, applying DRL to real-world embodied robot tasks faces several critical challenges: sample inefficiency, slow convergence, sparse reward signals, and the difficulty of designing suitable action spaces for complex tasks.
This thesis focuses on the embodied robot basketball shooting task, which serves as an ideal testbed for evaluating intelligent control algorithms due to its requirements for precise perception, dynamic decision-making, and coordinated motor control. The research goal is to develop novel DRL-based methods that enable a humanoid embodied robot to learn effective basketball shooting skills in a simulated environment, while simultaneously addressing the fundamental limitations of existing DRL algorithms.
The main contributions of this work are summarized as follows:
(1) A practical framework for the embodied robot basketball shooting task is established in the Webots simulation platform. The entire task is decomposed into three sub-tasks: approaching the ball, grasping the ball, and shooting the ball. Each sub-task is formulated as a Markov Decision Process (MDP) with a discrete action space, and solved using a Soft Actor-Critic (SAC) based algorithm. This modular framework facilitates incremental learning and enables the embodied robot to handle the complete basketball shooting pipeline autonomously.
(2) A novel PSR-SAC algorithm is proposed to address the problems of low sample efficiency and slow convergence commonly encountered in DRL-based embodied robot training. The method integrates the Convolutional Block Attention Module (CBAM) into the visual representation learning module, enabling the embodied robot to focus on task-relevant features more effectively. Furthermore, a prioritized successful recent experience replay mechanism is introduced, which combines global experience with successful experiences and recent trajectory data to improve the utilization efficiency of training samples.
(3) To overcome the limitations of discrete action space modeling in dynamic scenarios, a hybrid action space design is proposed based on the H-PPO algorithm. This approach allows the embodied robot to simultaneously make discrete action decisions and continuous parameter selections, thereby enhancing its flexibility and adaptability in basketball shooting. Additionally, a human experience-based reward shaping method is developed by incorporating YOLOv5 object detection technology, which accelerates the convergence of the training process by providing denser reward signals.
2. Preliminaries and Related Work
2.1 Deep Learning Foundations
Deep learning, a subset of machine learning, leverages multilayer neural networks to automatically learn hierarchical representations from raw data. Unlike traditional feature engineering approaches, deep learning models can extract meaningful patterns directly from high-dimensional inputs such as images, audio, and text. For embodied robot perception, convolutional neural networks (CNNs) play an integral role in processing visual information. A standard CNN architecture comprises alternating convolutional layers and pooling layers, followed by fully connected layers. The convolutional operation computes the dot product between a learnable filter and local regions of the input feature map, as expressed by Equation (1):
$$y_{i,j} = \sum_{c=1}^{C} \sum_{m=1}^{k_h} \sum_{n=1}^{k_w} W_{c,m,n} \cdot x_{c,\, i+m-1,\, j+n-1} + b \tag{1}$$
where \(W\) denotes the convolution kernel weights, \(b\) is the bias term, \(C\) is the number of input channels, and \(k_h \times k_w\) defines the kernel size. The output feature map dimensions are calculated as:
$$O_h = \frac{H – k_h + 2p}{s} + 1, \quad O_w = \frac{W – k_w + 2p}{s} + 1 \tag{2}$$
where \(p\) is the padding size and \(s\) is the stride.
Although CNN features provide rich spatial information, not all features contribute equally to task completion. The attention mechanism addresses this by dynamically reweighting features, simulating the human selective attention process. The general attention operation is defined as:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^{\top}}{\sqrt{d_k}}\right) V \tag{3}$$
where \(Q\), \(K\), and \(V\) are the query, key, and value matrices respectively, and \(d_k\) is the dimension of the keys.
The Convolutional Block Attention Module (CBAM) extends this concept to CNN feature maps by sequentially applying channel attention and spatial attention. Given an intermediate feature map \(F \in \mathbb{R}^{C \times H \times W}\), the channel attention map is computed by:
$$M_c(F) = \sigma\left(MLP(\text{AvgPool}(F)) + MLP(\text{MaxPool}(F))\right) \tag{4}$$
and the spatial attention map is obtained by:
$$M_s(F) = \sigma\left(f^{7\times 7}([\text{AvgPool}(F); \text{MaxPool}(F)])\right) \tag{5}$$
where \(\sigma\) is the sigmoid function, \(f^{7\times 7}\) represents a convolution with \(7 \times 7\) kernel, and \([ ; ]\) denotes channel-wise concatenation. The refined feature map is obtained by successively applying these attention maps: \(F’ = M_c(F) \otimes F\) and \(F” = M_s(F’) \otimes F’\).
2.2 Reinforcement Learning Fundamentals
Reinforcement learning (RL) formulates sequential decision-making problems as Markov Decision Processes (MDPs), defined by the tuple \(\langle S, A, P, R, \gamma \rangle\). At each time step \(t\), the agent observes a state \(s_t \in S\), takes an action \(a_t \in A\), receives a reward \(r_t = R(s_t, a_t)\), and transitions to a new state \(s_{t+1\)} according to the transition probability \(P(s_{t+1} \mid s_t, a_t)\). The objective of the agent is to maximize the cumulative discounted reward:
$$G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k} \tag{6}$$
where \(\gamma \in [0, 1]\) is the discount factor. The state value function \(V^\pi(s)\) and the action value function \(Q^\pi(s, a)\) are defined as:
$$V^\pi(s) = \mathbb{E}_\pi \left[ G_t \mid s_t = s \right] \tag{7}$$
$$Q^\pi(s, a) = \mathbb{E}_\pi \left[ G_t \mid s_t = s, a_t = a \right] \tag{8}$$
The optimal policy \(\pi^*\) satisfies the Bellman optimality equations, providing the theoretical foundation for many RL algorithms.
The Soft Actor-Critic (SAC) algorithm is an off-policy maximum entropy RL algorithm that augments the standard objective with an entropy regularization term. The policy is trained to maximize both the expected return and the policy entropy:
$$J(\pi) = \mathbb{E}_{\pi} \left[ \sum_t \gamma^t \left( r(s_t, a_t) + \alpha \mathcal{H}(\pi(\cdot \mid s_t)) \right) \right] \tag{9}$$
where \(\alpha\) is the temperature parameter controlling the trade-off between exploration and exploitation. Critic networks are trained by minimizing the soft Bellman residual:
$$J_Q(\theta) = \mathbb{E}_{(s, a, r, s’) \sim D} \left[ \frac{1}{2} \left( Q_\theta(s, a) – \left( r + \gamma \mathbb{E}_{a’ \sim \pi_\phi} \left[ Q_{\bar{\theta}}(s’, a’) – \alpha \log \pi_\phi(a’ \mid s’) \right] \right) \right)^2 \right] \tag{10}$$
where \(\bar{\theta}\) denotes the target network parameters. The policy is updated by minimizing the Kullback-Leibler divergence:
$$J_\pi(\phi) = \mathbb{E}_{s \sim D} \left[ \mathbb{E}_{a \sim \pi_\phi} \left[ \alpha \log \pi_\phi(a \mid s) – Q_\theta(s, a) \right] \right] \tag{11}$$
The Proximal Policy Optimization (PPO) algorithm is an on-policy method that clips the objective function to ensure stable policy updates:
$$J_{\text{clip}}(\theta) = \mathbb{E}_t \left[ \min \left( \rho_t(\theta) \hat{A}_t, \text{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right] \tag{12}$$
where \(\rho_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}\) is the importance sampling ratio and \(\epsilon\) is the clipping hyperparameter.
2.3 Robot Basketball Shooting
Basketball shooting for embodied robot platforms has been explored through various methodological lenses. Early approaches relied on fuzzy logic control systems that estimated the distance between the robot and the hoop, then adjusted the throwing force according to predefined rules. Although these methods achieved reasonable performance in static settings, their generalization to dynamic environments remained limited. Subsequently, Petri net-based coordination architectures were developed to integrate eye-hand-leg coordination for basketball throwing, enhancing the multi-modal perception capabilities of the embodied robot. However, these systems exhibited rigidity and limited scalability, primarily due to their dependence on accurate environmental models and pre-defined control sequences.
More recently, cognitive-inspired approaches incorporating deep belief networks and particle swarm optimization have been proposed to simulate human decision processes in shooting tasks. While these methods demonstrated improved robustness compared to conventional fuzzy controllers, they still struggled with complex and rapidly changing scenarios. The advent of deep reinforcement learning has catalyzed a fundamental shift in embodied robot basketball shooting research. By enabling end-to-end learning directly from sensor data, DRL methods eliminate the need for explicit environment modeling and manual feature extraction, offering greater flexibility and adaptability in dynamic task settings.
3. PSR-SAC for Embodied Robot Basketball Shooting
3.1 Problem Statement and Method Overview
The proposed embodied robot basketball shooting framework addresses two fundamental challenges: first, the replacement of manually designed control rules with autonomous learning capabilities; second, the improvement of sample efficiency and convergence speed of DRL algorithms. The overall framework comprises four primary components: the Webots simulation environment with the Robotis OP2 humanoid embodied robot, a perception module, a decision module, and an execution module.
The simulation environment is constructed following the FIRA HuroCup basketball competition rules. The embodied robot starts at coordinates (90, 0) while the hoop is positioned at the origin. The ball is randomly placed on a pole within the curved region bounded by the two-point line. The robot’s mission is to approach the ball, retrieve it from the pole, and throw it into the hoop from an arbitrary position.
3.2 Sub-task Decomposition and MDP Formulation
Rather than treating basketball shooting as a monolithic task, I decompose it into three sequential sub-tasks, each with its own MDP formulation and decision-making model:
Sub-task 1: Approaching the ball. The embodied robot navigates from its starting position to a location immediately in front of the ball. The action space comprises three walking maneuvers: forward movement, left turn, and right turn. These actions are implemented through the Gait Manager module of the Robotis OP2, with walking amplitude parameters configured as shown in Table 1. The episode terminates when the Euclidean distance between the robot and the ball falls below 0.12 meters.
| Action | setXAmplitude | setAAmplitude | setYAmplitude |
|---|---|---|---|
| Forward | 0.3 | 0 | 0.5 |
| Backward | -0.3 | 0 | 0.5 |
| Left turn | 0.3 | 0.3 | 0.5 |
| Right turn | 0.3 | -0.3 | 0.5 |
Sub-task 2: Grasping the ball. In this phase, the embodied robot is positioned before the ball and must execute a combination of fine position adjustments and a grasping action. The action space includes forward movement, backward movement, left turn, right turn, and arm-based grasping actions. The grasping action is achieved through the Motion Manager, which defines the coordinated motion of the left arm and right arm.
Sub-task 3: Shooting the ball. After successfully grasping the ball, the embodied robot must adjust its position and orientation, then execute a throwing motion with appropriate force. The action space comprises forward movement, left turn, right turn, and a predefined shooting action. Success is defined as the ball’s trajectory intersecting the interior space of the basketball hoop.
For the state space, I employ a stacked multi-frame image representation. The raw visual input captured by the embodied robot‘s head camera is an RGB image with a resolution of \(160 \times 120\) pixels. This image undergoes preprocessing consisting of grayscale conversion, resizing to \(84 \times 84\) pixels, four-frame stacking, and normalization to the range [0, 1]. The resulting state representation is a \(4 \times 84 \times 84\) tensor that preserves both spatial and temporal information, enabling the embodied robot to perceive motion dynamics and directional cues essential for effective decision-making.
The reward function adopts a sparse reward design to clearly signal task completion:
$$R(s, a) = \begin{cases} 100 & \text{if sub-task completed} \\ -10 & \text{if episode length exceeds 30 steps} \\ -5 & \text{if ball is knocked down} \\ 0 & \text{otherwise} \end{cases} \tag{13}$$
3.3 Attention-based Visual Representation Learning
Effective visual perception is crucial for the embodied robot to locate objects of interest—namely, the ball and the hoop—and to make informed decisions. In this work, I enhance the standard CNN feature extractor with the CBAM module, strategically positioned after the convolutional layers to refine the high-level semantic features.
As shown in Table 2, the convolutional network consists of three layers without pooling operations, deliberately preserving spatial resolution to retain precise object positions. This design choice is motivated by the fact that positioning information is critical for basketball shooting, and pooling operations would introduce undesirable shift-invariance, potentially degrading the embodied robot‘s spatial awareness.
| Layer | Channels | Kernel Size | Stride | Padding | Output Size |
|---|---|---|---|---|---|
| Conv1 | 32 | 8 | 4 | 2 | 32 × 21 × 21 |
| Conv2 | 64 | 4 | 2 | 1 | 64 × 10 × 10 |
| Conv3 | 64 | 3 | 1 | 1 | 64 × 10 × 10 |
| CBAM | 64 | — | — | — | 64 × 10 × 10 |
The CBAM module applies both channel-wise and spatial-wise attention recalibration. The channel attention mechanism identifies which feature channels are most informative for the task, enhancing their responses and suppressing uninformative ones. Meanwhile, the spatial attention mechanism identifies which regions of the feature map warrant greater focus. For basketball shooting, these regions typically correspond to the ball and hoop locations. The combined attention mechanism enables the embodied robot to selectively focus on task-relevant information, improving both the accuracy and efficiency of visual representation extraction.
3.4 Prioritized Successful Recent Experience Replay
The decision-making module of the proposed framework employs a modified SAC algorithm with an enhanced experience replay mechanism. The standard SAC algorithm samples experiences uniformly from a single replay buffer, which may lead to inefficient use of important samples. To address this limitation, I propose the Priority Success Recent Experience Replay (PSR) mechanism, which consists of three key components:
(1) Global experience pool. A conventional experience replay buffer that stores all interaction experiences collected by the embodied robot during training. Each sample is associated with a priority value based on its TD-error. The priority of a sample is calculated as:
$$p_i = |\delta_i| + \epsilon \tag{14}$$
where \(\delta_i\) represents the TD-error of sample \(i\) and \(\epsilon\) is a small positive constant ensuring non-zero priority. The sampling probability is then proportional to the priority:
$$P(i) = \frac{p_i^{\alpha}}{\sum_j p_j^{\alpha}} \tag{15}$$
where \(\alpha\) controls the degree of prioritization. To compensate for the sampling bias introduced by non-uniform sampling, importance sampling weights are applied to the loss computation:
$$w_i = \left( \frac{1}{N} \cdot \frac{1}{P(i)} \right)^{\beta} \tag{16}$$
where \(\beta\) is a hyperparameter controlling the degree of bias correction.
(2) Success experience pool. A dedicated buffer that stores experiences exclusively from episodes where the embodied robot successfully completed the sub-task. This design is inspired by human cognitive learning, where successful demonstrations carry higher referential value. By explicitly separating and prioritizing successful experiences, the algorithm intensifies learning from the most informative state-action transitions, significantly improving the utilization of valuable samples and accelerating the learning process.
(3) Recent trajectory inclusion. In each training iteration, the most recent episode’s experiences are directly included in the mini-batch. This strategy, analogous to how basketball players refine their skills by focusing on their most recent attempts, ensures that the embodied robot continually learns from the latest experiences and adapts its policy in a timed manner. The final mini-batch composition is defined as:
$$\mathcal{B} = \mathcal{B}_{\text{recent}} \cup \mathcal{B}_{\text{global}} \cup \mathcal{B}_{\text{success}} \tag{17}$$
where \(\mathcal{B}_{\text{recent}}\) contains the most recent episode data, \(\mathcal{B}_{\text{global}}\) is sampled from the global experience pool using prioritized sampling, and \(\mathcal{B}_{\text{success}}\) is sampled from the success experience pool with the size ratio controlled by parameter \(k\). Empirical analysis demonstrates that a sampling ratio of \(k = 0.1\) yields the best balance between learning speed and stability. When the success ratio is too high, the algorithm tends to converge to sub-optimal local policies. Conversely, when the success ratio is zero, the benefits of success-guided learning are completely lost.
For the discrete action space, the SAC algorithm’s Soft Q-target computation is adapted to use the full action distribution rather than a single Monte Carlo sample:
$$y = r + \gamma \left( \sum_{a’} \pi_\phi(a’ \mid s’) Q_{\bar{\theta}}(s’, a’) + \alpha \mathcal{H}(\pi_\phi(\cdot \mid s’)) \right) \tag{18}$$
The critic loss function is formulated as:
$$J_Q(\theta) = \mathbb{E}_{(s, a, r, s’) \sim \mathcal{B}} \left[ \frac{1}{2} \left( Q_\theta(s, a) – y \right)^2 \right] \tag{19}$$
and the actor loss is:
$$J_\pi(\phi) = \mathbb{E}_{s \sim \mathcal{B}} \left[ \sum_a \pi_\phi(a \mid s) \left( \alpha \log \pi_\phi(a \mid s) – Q_\theta(s, a) \right) \right] \tag{20}$$
As summarized in Algorithm 1, the PSR-SAC algorithm alternates between experience collection and network parameter updates, progressively refining the policy of the embodied robot toward the optimal basketball shooting strategy.
| Algorithm 1: PSR-SAC | |
|---|---|
| Input | Current state \(s\), network parameters, success ratio \(k\), entropy coefficient \(\alpha\) |
| Output | Optimized network parameters |
| 1 | Initialize policy network, two critic networks, target networks, replay buffers |
| 2 | for \(t = 1, \ldots, T\) do |
| 3 | Select action \(a_t \sim \pi_\phi(\cdot \mid s_t)\) |
| 4 | Execute \(a_t\), observe \((s_{t+1}, r_t, \text{done})\) |
| 5 | Store \((s_t, a_t, r_t, s_{t+1})\) in global buffer; if done with success, also store in success buffer |
| 6 | if training condition met then |
| 7 | Compose mini-batch from recent, global, and success buffers |
| 8 | Compute critic and actor losses; update network parameters via Adam |
| 9 | Update target networks with soft update rate \(\tau\) |
| 10 | end for |
3.5 Experimental Setup and Results
Experiments were conducted on an RTX 3070 GPU with 32GB RAM, using Webots R2021b as the simulation platform and PyTorch 1.13.1 as the deep learning framework. The embodied robot was trained on three sub-tasks in both fixed-ball-position and random-ball-position scenarios. Table 3 summarizes the key hyperparameters used in training.
| Parameter | Value |
|---|---|
| Total training steps | 30000 |
| Global replay buffer capacity | 8192 |
| Success replay buffer capacity | 4096 |
| Discount factor \(\gamma\) | 0.99 |
| Mini-batch size | 512 |
| Learning rate | 0.0003 |
| Priority exponent \(\alpha\) | 0.7 |
| Importance sampling exponent \(\beta\) | 0.4 |
| Target network update frequency | 50 |
| Success sample ratio \(k\) | 0.1 |
Three evaluation metrics were employed: average episode reward, convergence speed (measured as the number of episodes required to reach a reward threshold), and task success rate (computed from 1000 test episodes). To ensure statistical robustness, each experiment was repeated three times with different random seeds and the results were averaged.
The overall comparison results are presented in Table 4. The proposed CBAM-PSR-SAC algorithm consistently achieves the fastest convergence across all sub-tasks and both scenarios. In the random ball position scenario, the improvements are particularly pronounced: the convergence speed of CBAM-PSR-SAC improved by 12.5%, 30%, and 7.1% over standard SAC for sub-tasks 1, 2, and 3 respectively.
| Algorithm | Fixed Position (Episodes) | Random Position (Episodes) | ||||
|---|---|---|---|---|---|---|
| Sub-task 1 | Sub-task 2 | Sub-task 3 | Sub-task 1 | Sub-task 2 | Sub-task 3 | |
| DQN | 600 | 670 | 520 | 1050 | 750 | 7800 |
| PPO | 780 | 1000 | 930 | 1330 | 1800 | 10000 |
| SAC | 400 | 120 | 300 | 1120 | 600 | 7000 |
| CBAM-PSR-SAC | 380 | 120 | 280 | 980 | 420 | 6500 |
The success rates after training, evaluated over 1000 test episodes, are reported in Table 5. In the random ball position scenario, CBAM-PSR-SAC surpasses SAC by 3.4%, 8.3%, and 9.4% for the three sub-tasks respectively, demonstrating the effectiveness of the proposed attention and experience replay improvements.
| Algorithm | Fixed Position (%) | Random Position (%) | ||||
|---|---|---|---|---|---|---|
| Sub-task 1 | Sub-task 2 | Sub-task 3 | Sub-task 1 | Sub-task 2 | Sub-task 3 | |
| DQN | 95.1 | 56.6 | 96.5 | 90.1 | 56.8 | 55.6 |
| PPO | 82.5 | 92.5 | 98.6 | 82.9 | 48.5 | 41.3 |
| SAC | 92.7 | 99.6 | 98.2 | 93.4 | 60.1 | 48.8 |
| CBAM-PSR-SAC | 99.6 | 99.9 | 99.3 | 96.8 | 68.4 | 58.2 |
To isolate the contributions of each component in the proposed method, I conducted ablation studies on the attention module and the experience replay mechanism. Table 6 shows the results of comparing CBAM-SAC with other representation learning approaches including DrQ-SAC, ResNet-SAC, and VAE-SAC. The CBAM-SAC algorithm achieves the highest success rates in all three sub-tasks, particularly for sub-task 3 where it reaches 56.2% compared to 32.5% for ResNet-SAC and 38.4% for VAE-SAC.
| Algorithm | Sub-task 1 (%) | Sub-task 2 (%) | Sub-task 3 (%) |
|---|---|---|---|
| SAC | 93.1 | 60.1 | 48.8 |
| DrQ-SAC | 91.2 | 47.5 | 42.1 |
| ResNet-SAC | 92.5 | 49.8 | 32.5 |
| VAE-SAC | 49.4 | 36.2 | 38.4 |
| CBAM-SAC | 96.3 | 65.8 | 56.2 |
Similarly, Table 7 evaluates the experience replay variants. The PSR-SAC algorithm outperforms PER-SAC, ER-MS-SAC, and PEC-SAC across all sub-tasks, achieving success rates of 96.4%, 67.5%, and 57.9% for sub-tasks 1, 2, and 3 respectively.
| Algorithm | Sub-task 1 (%) | Sub-task 2 (%) | Sub-task 3 (%) |
|---|---|---|---|
| SAC | 93.4 | 60.1 | 48.8 |
| PER-SAC | 95.8 | 66.2 | 52.8 |
| ER-MS-SAC | 94.2 | 53.6 | 49.6 |
| PEC-SAC | 93.3 | 45.2 | 46.5 |
| PSR-SAC | 96.4 | 67.5 | 57.9 |
The parameter sensitivity analysis for the success sample ratio \(k\) revealed an important trend: when \(k = 0.9\) or 0.5, the algorithm tends to converge to local optima due to over-reliance on successful experiences; when \(k = 0\), the convergence speed decreases due to the absence of success-guided learning. The optimal performance was achieved at \(k = 0.1\), which balances the contribution of successful experiences with sufficient exploration from the global pool.
4. H-PPO for Embodied Robot Basketball Shooting
4.1 Limitations of Discrete Action Spaces
While the discrete action space approach described in the previous chapter enables the embodied robot to learn basic basketball shooting skills, it suffers from inherent limitations when applied to dynamic task scenarios. Specifically, when the ball is placed at random positions, the embodied robot must adjust its shooting force according to the varying distance to the hoop. However, in a fixed discrete action space, the shooting force is predetermined and cannot be modulated in response to environmental changes. This inflexibility leads to low shooting accuracy, especially in random-position scenarios where the fixed force setting may be either too strong or too weak for the current configuration.
The experimental results presented in the previous chapter corroborate this observation: while CBAM-PSR-SAC achieves 99.3% shooting success in the fixed-position scenario, the success rate drops significantly to 58.2% in the random-position scenario. This substantial performance gap highlights the critical need for a more expressive action modeling approach.
4.2 Hybrid Action Space Framework
To overcome the limitations of discrete action spaces, I propose a hybrid action space (also known as parameterized action space) framework for the shooting sub-task. In this framework, the embodied robot‘s decision-making process is divided into two interconnected levels. The discrete level selects the type of action to execute from a finite set—forward movement, left turn, right turn, or shooting. The continuous level determines the associated continuous parameter values for the selected action, such as walking duration for movement actions or throwing force for the shooting action.
Formally, a hybrid action is represented as a tuple \((a_d, a_c)\) where \(a_d \in \mathcal{A}_d\) is the discrete action index and \(a_c \in \mathcal{A}_c \subset \mathbb{R}^m\) is the continuous parameter vector. The complete hybrid action space is defined as:
$$\mathcal{A} = \{ (a_d, a_c) \mid a_d \in \mathcal{A}_d, \ a_c \in \mathcal{A}_c \} \tag{21}$$
As illustrated in Table 8, the discrete actions for the shooting sub-task are encoded as integers from 0 to 3, with associated continuous parameter ranges.
| Discrete Action | Index | Continuous Parameter | Meaning | Range |
|---|---|---|---|---|
| Forward | 0 | \(t_{\text{move}}\) | Movement time | [0.2, 0.4] s |
| Left turn | 1 | \(t_{\text{move}}\) | Movement time | [0.2, 0.4] s |
| Right turn | 2 | \(t_{\text{move}}\) | Movement time | [0.2, 0.4] s |
| Shoot | 3 | \(v_{\text{throw}}\) | Throwing velocity | [10, 15] |
Since the continuous parameters have heterogeneous ranges, I apply a linear transformation to normalize them to [-1, 1]. The policy network outputs values in this normalized range, which are then transformed back to the actual parameter values via:
$$v_{\text{actual}} = \frac{(v_{\text{norm}} + 1) \cdot (v_{\max} – v_{\min})}{2} + v_{\min} \tag{22}$$
For example, if the continuous actor outputs \(v_{\text{norm}} = 0.2\) for the shooting action, the corresponding throwing velocity would be \(v_{\text{actual}} = \frac{(0.2 + 1) \cdot (15 – 10)}{2} + 10 = 13\).
4.3 H-PPO Network Architecture
The proposed H-PPO algorithm employs an Actor-Critic architecture as shown in Table 9. The Critic network is a state-value network, which is preferred over Q-functions in hybrid action space settings due to its ability to avoid the complexity of incorporating both discrete and continuous actions into the value estimation. The Actor network consists of two parallel sub-networks: a discrete actor that outputs a categorical probability distribution over discrete actions, and a continuous actor that outputs the mean and standard deviation of a Gaussian distribution for continuous parameters.
| Component | Structure | Activation | Output |
|---|---|---|---|
| Feature Extractor | 3 × Conv layers | ReLU | Feature vector |
| Critic | 3 × FC layers | ReLU / Linear | State value \(V(s)\) |
| Discrete Actor | 3 × FC layers + Softmax | ReLU / Softmax | Action probabilities |
| Continuous Actor | 3 × FC layers | ReLU / Tanh | Mean and std of Gaussian |
During execution, the discrete actor samples an action \(a_d \sim \pi_\phi(\cdot \mid s)\) from the categorical distribution, while the continuous actor samples continuous parameters \(a_c \sim \mathcal{N}(\mu_\theta(s), \sigma_\theta(s))\) from the Gaussian distribution. The final hybrid action \((a_d, a_c)\) is then executed by the embodied robot.
4.4 Human Experience-based Reward Design
Sparse reward signals present a major challenge in DRL-based embodied robot training. In the basketball shooting task, the embodied robot receives positive feedback only when the ball successfully enters the hoop—an event with extremely low probability in the early stages of training. This extreme scarcity of rewards leads to slow or even impossible convergence. To address this issue, I propose a human experience-based reward redesign method that leverages YOLOv5 object detection to provide intermediate guidance.
The core insight is derived from human basketball experience: the probability of a successful shot is significantly higher when the shooting direction is aligned with the hoop. Therefore, the embodied robot should be encouraged to orient itself such that the hoop appears near the center of its visual field. When the hoop appears on the left side of the image, the robot should be rewarded for turning left; conversely, when the hoop appears on the right, it should turn right.
To implement this guidance, I first trained a YOLOv5n object detection model on a custom dataset of 1700 images collected during the shooting sub-task training. The dataset was split into training (1360), validation (170), and test (170) subsets. The trained model achieved a mAP@0.5 of 0.991 and a mAP@0.5:0.95 of 0.862, demonstrating high detection accuracy. During RL training, the detection model runs in real-time on each camera frame to locate the hoop. The human experience reward is then computed as:
$$r_{\text{human}} = \gamma_{\text{decay}} \cdot e^{-\frac{(x_{\text{hoop}} – x_{\text{center}})^2}{\sigma^2}} \tag{23}$$
where \(x_{\text{hoop}}\) is the horizontal coordinate of the hoop in the image, \(x_{\text{center}}\) is the image center coordinate, and \(\gamma_{\text{decay}}\) is a scaling factor. The total reward is formulated as a weighted combination:
$$r_{\text{total}} = w_1 \cdot r_{\text{task}} + w_2 \cdot r_{\text{human}} \tag{24}$$
where \(r_{\text{task}}\) is the original task reward and \(w_1, w_2\) are balancing weights.
4.5 H-PPO Algorithm Details
The H-PPO algorithm extends the standard PPO framework to hybrid action spaces. The optimization process uses the generalized advantage estimation (GAE) function:
$$\hat{A}_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l} \tag{25}$$
where \(\delta_t = r_t + \gamma V(s_{t+1}) – V(s_t)\) is the TD residual and \(\lambda\) is the GAE parameter. The Critic network is updated by minimizing the value loss:
$$\mathcal{L}_{\text{critic}} = \mathbb{E}_t \left[ \left( V_\psi(s_t) – V_t^{\text{target}} \right)^2 \right] \tag{26}$$
The discrete actor is trained using the clipped PPO objective:
$$\mathcal{L}_{\text{discrete}} = \mathbb{E}_t \left[ \min \left( \rho_t^d \hat{A}_t, \operatorname{clip}(\rho_t^d, 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right] \tag{27}$$
where the importance sampling ratio for the discrete part is:
$$\rho_t^d = \frac{\pi_{\theta_d}(a_d \mid s_t)}{\pi_{\theta_d^{\text{old}}}(a_d \mid s_t)} \tag{28}$$
Similarly, the continuous actor is trained with:
$$\mathcal{L}_{\text{continuous}} = \mathbb{E}_t \left[ \min \left( \rho_t^c \hat{A}_t, \operatorname{clip}(\rho_t^c, 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right] \tag{29}$$
where the importance sampling ratio for the continuous part is:
$$\rho_t^c = \frac{\pi_{\theta_c}(a_c \mid s_t)}{\pi_{\theta_c^{\text{old}}}(a_c \mid s_t)} \tag{30}$$
The overall actor loss combines both components, and both share the same advantage estimates. Algorithm 2 outlines the complete H-PPO training procedure for the embodied robot basketball shooting task.
| Algorithm 2: H-PPO for Embodied Robot Basketball Shooting | |
|---|---|
| Input | Network parameters, hyperparameters |
| Output | Optimized network parameters |
| 1 | Initialize critic, discrete actor, continuous actor, replay buffer |
| 2 | for \(t = 1, \ldots, T\) do |
| 3 | for \(i = 1, \ldots, N\) do |
| 4 | Observe state \(s_t\), select \(a_d \sim \pi_{\theta_d}\), \(a_c \sim \pi_{\theta_c}\) |
| 5 | Execute hybrid action \((a_d, a_c)\), observe \((s_{t+1}, r_t)\) |
| 6 | Store experience in buffer |
| 7 | end for |
| 8 | for \(j = 1, \ldots, M\) do |
| 9 | Sample mini-batches and compute losses for critic, discrete actor, continuous actor |
| 10 | Update all network parameters via Adam optimizer |
| 11 | end for |
| 12 | end for |
4.6 Experimental Results and Analysis
For the H-PPO experiments, I trained the embodied robot on the shooting sub-task with both fixed and random ball positions. Table 10 reports the training hyperparameters used in these experiments.
| Parameter | Value |
|---|---|
| Total training steps | 50000 |
| Simulation steps per update | 512 |
| Learning rate annealing | Enabled |
| Discount factor \(\gamma\) | 0.99 |
| Mini-batch size | 512 |
| GAE parameter \(\lambda\) | 0.95 |
| Epochs per update | 4 |
| Advantage normalization | Enabled |
| Clipping coefficient \(\epsilon\) | 0.1 |
| Entropy coefficient | 0.01 |
| Value loss coefficient | 0.5 |
| Gradient clipping max norm | 0.5 |
Table 11 compares the convergence behavior of YOLO-H-PPO with discrete-space baselines. In the random position scenario, YOLO-H-PPO converges in 5000 episodes, representing a 23.2% improvement over CBAM-PSR-SAC (6500 episodes), 35.9% improvement over DQN (7800 episodes), and 50% improvement over PPO (10000 episodes).
| Algorithm | Fixed Position (Episodes) | Random Position (Episodes) |
|---|---|---|
| DQN | 520 | 7800 |
| PPO | 930 | 10000 |
| CBAM-PSR-SAC | 280 | 6500 |
| YOLO-H-PPO | 780 | 5000 |
Table 12 reports the task success rates over 1000 test episodes. YOLO-H-PPO achieves 99.1% success in the fixed position scenario and 75.3% in the random position scenario, outperforming CBAM-PSR-SAC by 17.1% in the random scenario. This demonstrates the significant advantage of hybrid action spaces in enabling precise continuous control over the shooting force.
| Algorithm | Fixed Position (%) | Random Position (%) |
|---|---|---|
| DQN | 96.5 | 55.6 |
| PPO | 98.6 | 41.3 |
| CBAM-PSR-SAC | 99.3 | 58.2 |
| YOLO-H-PPO | 99.1 | 75.3 |
Additionally, I measured the average number of steps per episode required to complete the shooting task, as presented in Table 13. YOLO-H-PPO achieves the task in 3.68 steps in the fixed scenario and 4.63 steps in the random scenario, fewer than both CBAM-PSR-SAC (3.95 and 6.27) and PPO (6.55 and 8.42). This improvement in execution efficiency can be attributed to the continuous parameter control, which allows the embodied robot to precisely adjust its movements and shooting force, thereby avoiding unnecessary repositioning steps.
| Algorithm | Fixed Position (Steps) | Random Position (Steps) |
|---|---|---|
| DQN | 4.62 | 5.56 |
| PPO | 6.55 | 8.42 |
| CBAM-PSR-SAC | 3.95 | 6.27 |
| YOLO-H-PPO | 3.68 | 4.63 |
To further assess the contribution of the hybrid action space design, I compared H-PPO with MP-DQN, another hybrid action space algorithm, as well as discrete-space baselines under sparse reward conditions. Table 14 presents the success rate comparisons. In the random position scenario, H-PPO achieves 73.6% success, surpassing MP-DQN by 10.2% and PPO by 32.3%.
| Algorithm | Fixed Position (%) | Random Position (%) |
|---|---|---|
| DQN | 96.5 | 55.6 |
| MP-DQN | 95.8 | 63.4 |
| PPO | 98.6 | 41.3 |
| CBAM-PSR-SAC | 99.3 | 58.2 |
| H-PPO | 99.5 | 73.6 |
The impact of the human experience-based reward design was validated through ablation studies. As shown in Table 15, the integration of human experience rewards significantly accelerates convergence without compromising task success. In the discrete action space, PPO with human experience rewards converges 35.5% faster than its sparse-reward counterpart. In the hybrid action space, H-PPO with human experience rewards achieves a 17.3% improvement in convergence speed. The success rates remain comparable, with slight improvements observed when using the human experience rewards.
| Algorithm | Sparse Reward Success (%) | Human Experience Success (%) | Convergence Speedup (%) |
|---|---|---|---|
| PPO | 41.3 | 43.2 | 35.5 |
| H-PPO | 73.6 | 75.3 | 17.3 |
Qualitative analysis of the training process revealed distinct behavioral differences between the initial and final models. During the early stages of training, the embodied robot fails to orient itself toward the hoop and struggles to modulate the throwing force properly. After 10000 training episodes, however, the embodied robot demonstrates precise body alignment and adaptive force control, successfully directing the ball into the hoop even under varying ball positions. This behavioral progression affirms the effectiveness of the proposed H-PPO method in enabling the embodied robot to acquire versatile basketball shooting capabilities in dynamic task environments.
5. Conclusion and Future Directions
This thesis has investigated the embodied robot basketball shooting problem through the lens of deep reinforcement learning, addressing the critical challenges of sample efficiency, convergence speed, action space modeling, and sparse reward handling. The research contributions can be summarized along three dimensions:
First, I developed a comprehensive simulation framework for embodied robot basketball shooting in the Webots environment, decomposing the overall task into three sequential sub-tasks and formulating each within the MDP framework. This modular architecture enables systematic training of the embodied robot for complex multi-stage tasks, offering a practical template for other embodied robot manipulation scenarios.
Second, the proposed PSR-SAC algorithm combines CBAM-based visual representation learning with a prioritized successful recent experience replay mechanism. By enhancing the embodied robot‘s perceptual awareness through attention-driven feature extraction and improving sample utilization through a multi-pool prioritization strategy, the method achieves significant improvements in both convergence speed and task success rate compared to conventional SAC algorithms. The experimental results across three sub-tasks validate the effectiveness of each component in the proposed approach.
Third, the H-PPO algorithm with hybrid action space modeling overcomes the inherent limitations of discrete action spaces in dynamic task scenarios. By allowing the embodied robot to choose both discrete actions and their associated continuous parameters, the method provides more flexible and precise control over the shooting operation. The human experience-based reward design, implemented via YOLOv5 object detection, addresses the sparse reward problem by providing dense intermediate feedback that guides the embodied robot toward learning effective shooting strategies. This approach demonstrates substantial improvements in shooting accuracy and training efficiency, particularly in scenarios with random ball positions.
While this thesis has achieved promising results in the simulated environment, several directions warrant further investigation. First, the transfer of the trained policies from simulation to real-world embodied robot platforms requires careful consideration of the sim-to-real gap, including differences in dynamics, sensor noise, and actuator delays. Domain randomization and system identification techniques could potentially mitigate these discrepancies. Second, the exploration of more sophisticated automated reward learning methods, such as inverse reinforcement learning or preference-based RL, could further reduce the reliance on hand-crafted reward components. Third, extending the hybrid action space framework to other embodied robot manipulation tasks, such as object sorting or tool usage, would demonstrate the generality of the proposed methodology. Finally, incorporating recent advances in Transformer-based policy architectures and multi-agent reinforcement learning could enable more complex embodied robot behaviors involving collaborative task execution.
