In the field of autonomous humanoid robot motion, the ability to efficiently recognize targets and execute tasks in dynamic environments is both significant and challenging. Among various complex tasks, the gate-passing task for a humanoid robot requires precise target identification, stable policy generation, and rapid adaptation to environmental changes. Traditional control methods, such as predefined joint trajectories or model-based path planning, often fail in dynamic scenarios where the gate position varies randomly. To address these limitations, I propose a deep reinforcement learning based approach for humanoid robot gate passing, implemented and validated in the Webots simulation platform with the Robotis OP2 humanoid robot as the testbed. Through extensive comparative experiments with mainstream deep reinforcement learning algorithms, I verify the effectiveness and adaptability of the proposed method under both fixed and random gate position settings.
The research focuses on two major contributions: a GAN-SAC algorithm that integrates Generative Adversarial Networks (GAN) into the Soft Actor-Critic (SAC) framework to improve policy generation stability and exploration efficiency, and a hybrid quality evaluation and priority-based sample selection mechanism (HQE-PSS) that enhances the experience replay process for more efficient use of high-quality samples. Both methods are systematically evaluated using metrics such as average episode reward, task success rate, and trajectory smoothness. The experimental results demonstrate that the proposed approaches outperform DQN, PPO, and SAC in convergence speed, policy stability, and task success rate for the humanoid robot gate-passing task.
1. Problem Analysis and Motivation
In the gate-passing task, the humanoid robot must navigate through a narrow gate structure while maintaining balance and avoiding collisions. The gate may be located at a fixed position in one scenario, or randomly placed within a specified region in another scenario. The dynamic nature of the environment imposes high requirements on the robot’s perception, decision-making, and motion control. Traditional reinforcement learning algorithms often suffer from slow convergence and unstable policy updates, especially when the reward signal is sparse or the environment is non-stationary. Additionally, the experience replay buffer used in many deep reinforcement learning algorithms stores samples uniformly, which leads to low utilization of important transitions and inefficient learning.
To overcome these issues, I first introduce a GAN-based augmentation mechanism into the SAC algorithm. The generator creates virtual environment samples, thereby enriching the training distribution and enabling the humanoid robot to explore a wider range of behaviors. The discriminator provides an adversarial reward signal that guides the policy network toward more robust and diverse actions. This GAN-SAC framework effectively mitigates the exploration-exploitation trade-off and accelerates convergence.
Second, I design HQE-PSS, a novel experience replay management mechanism. Instead of storing all samples equally, HQE-PSS evaluates each sample using dynamic quality scoring based on reward statistics and Z-score normalization. High-quality samples are stored in a separate region and are sampled with higher priority during training. Low-quality samples, such as those leading to falls or invalid actions, are periodically cleaned to free buffer space. This mechanism ensures that the humanoid robot learns primarily from informative transitions while still retaining a small proportion of low-quality samples to maintain exploration.
2. Problem Formulation: Markov Decision Process
I formulate the gate-passing task as a Markov Decision Process (MDP) defined by the tuple \( \langle S, A, P, R, \gamma \rangle \). At each time step \( t \), the humanoid robot observes a state \( s_t \in S \), selects an action \( a_t \in A \), and receives a reward \( r_t = R(s_t, a_t) \). The state transition function \( P(s_{t+1} | s_t, a_t) \) describes the dynamics of the environment, and \( \gamma \in [0,1] \) is the discount factor.
State Space: The state is constructed from visual observations captured by the onboard camera of the humanoid robot. The raw RGB image with dimensions \( 160 \times 120 \times 3 \) is preprocessed by resizing to \( 84 \times 84 \), converting to grayscale, and stacking four consecutive frames. This yields a state representation of dimension \( 4 \times 84 \times 84 \), which captures both spatial and temporal information necessary for decision-making.
Action Space: To reduce the complexity of high-dimensional joint control, I adopt a discrete action space consisting of three primitive actions: crawl forward, turn left, and turn right. These actions are carefully designed based on the kinematic capabilities of the Robotis OP2 humanoid robot. The crawling action involves a sequence of postural transitions, including standing, squatting, kneeling, leaning forward, and quadrupedal support. Each transition is realized by specifying target angles for the 20 servo motors. Table 1 summarizes the key servo angles for different postures during the crawling sequence.
| Servo ID | Standing | Squatting | Kneeling | Leaning Forward | Four-point Support |
|---|---|---|---|---|---|
| ShoulderR | 1.55 | 0.82 | -0.60 | 0.00 | 0.00 |
| ShoulderL | 0.08 | 0.12 | 1.67 | 1.46 | 1.45 |
| ArmUpperR | 0.05 | 0.05 | 0.05 | 0.31 | 0.30 |
| ArmUpperL | 0.90 | 0.90 | 0.90 | 0.71 | 0.71 |
| ArmLowerR | -0.26 | 1.16 | 1.16 | 0.80 | 0.79 |
| ArmLowerL | 1.10 | -0.22 | -0.22 | 0.19 | 0.19 |
| PelvYR | 0.50 | 0.50 | 0.50 | 0.50 | 0.49 |
| PelvYL | 0.50 | 0.50 | 0.50 | 0.50 | 0.48 |
| PelvR | 0.51 | 0.51 | 0.51 | 0.51 | 0.51 |
| PelvL | 0.49 | 0.49 | 0.49 | 0.49 | 0.49 |
| LegUpperR | -0.44 | 0.07 | 0.07 | 0.03 | 1.15 |
| LegUpperL | 1.43 | 0.88 | 0.88 | 0.96 | 1.17 |
| LegLowerR | 1.80 | 0.71 | 0.71 | 1.07 | 1.81 |
| LegLowerL | -1.44 | 0.28 | 0.28 | -0.78 | -1.43 |
| AnkleR | 1.28 | 1.28 | 1.28 | 0.78 | 1.21 |
| AnkleL | -0.30 | -0.31 | -0.31 | 0.21 | -0.23 |
| FootR | 0.51 | 0.51 | 0.51 | 0.51 | 0.52 |
| FootL | 0.49 | 0.49 | 0.49 | 0.49 | 0.49 |
| Neck | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 |
| Head | 0.90 | 0.90 | 0.90 | 0.90 | 0.61 |
Reward Function: The reward design is crucial for guiding the humanoid robot toward the goal. I employ a discrete reward function that provides immediate feedback for different events. Table 2 lists the reward values and their corresponding triggering conditions.
| Reward Item | Mathematical Expression | Trigger Condition |
|---|---|---|
| Fall penalty | \( r_{fall} = -100 \) | Inertial measurement unit detects imbalance |
| Timeout penalty | \( r_{timeout} = -50 \) | Number of decision steps in an episode exceeds 20 |
| Turning penalty | \( r_{turn} = -10 \) | Executing a left or right turn action |
| Success reward | \( r_{crawl} = 100 \) | Crawling action and successfully passing through the gate |
| Failure penalty | \( r_{fail} = -3 \) | Crawling action but not passing through the gate |
3. GAN-SAC Algorithm for Gate-Passing
Traditional SAC maximizes the expected return while also maximizing the entropy of the policy. The objective is given by:
\[
J(\pi) = \mathbb{E}_{s_t, a_t \sim \pi} \left[ r_t + \gamma \mathbb{E}_{s_{t+1}}[V(s_{t+1})] – \alpha \log \pi(a_t | s_t) \right]
\]
where \( \alpha \) is the temperature parameter balancing exploration and exploitation. The Q-function update follows:
\[
Q(s_t, a_t) = r_t + \gamma \mathbb{E}_{s_{t+1}}[V(s_{t+1})]
\]
Although SAC is effective for continuous control, its training data diversity is limited in sparse-reward dynamic environments. To address this, I integrate GAN into the SAC framework.
The proposed GAN-SAC architecture consists of four main components: the actor network (policy), two critic networks (Q1 and Q2), a generator, and a discriminator. The generator takes random noise \( z \sim N(0,1) \) as input and produces fake actions \( G(z) \). The discriminator distinguishes between real actions sampled from the current policy and fake actions from the generator. The adversarial reward \( r_{gan} \) is defined as:
\[
r_{gan} = D(a)
\]
The policy loss is modified to incorporate this adversarial reward:
\[
L_{policy} = – \mathbb{E} \left[ \min(Q_1, Q_2) \right] – \alpha \cdot r_{gan}
\]
where \( \alpha \) is set to 0.1 in my experiments. The generator and discriminator are trained alternately using the following losses:
\[
L_D = – \mathbb{E} \left[ \log D(a_{real}) \right] – \mathbb{E} \left[ \log (1 – D(a_{fake})) \right]
\]
\[
L_G = – \mathbb{E} \left[ \log D(G(z)) \right]
\]
The complete GAN-SAC training procedure is summarized in Algorithm 1.
| Input: State space \( S \) |
| Output: Action \( A \) |
| 1. Initialize Q networks \( Q_1, Q_2 \), policy network, target networks \( Q_1′, Q_2′ \). |
| 2. Initialize generator G and discriminator D. |
| 3. Initialize experience replay buffer with capacity N. |
| 4. Initialize Adam optimizer and MSE loss function. |
| 5. for episode = 1 to M do |
| 6. Sample a batch of data \( (s, a, r, s’, done) \) from the buffer. |
| 7. Compute target Q value: \( target\_Q = r + \gamma (1 – done) \min(Q_1′(s’), Q_2′(s’)) \). |
| 8. Compute current Q predictions: \( current\_Q_1 = Q_1(s,a), current\_Q_2 = Q_2(s,a) \). |
| 9. Compute Q loss: \( L_Q = MSE(current\_Q_1, target\_Q) + MSE(current\_Q_2, target\_Q) \). |
| 10. Update \( Q_1 \) and \( Q_2 \) via backpropagation. |
| 11. Sample action \( a \sim \pi(s) \), compute policy entropy. |
| 12. Get GAN reward: \( r_{gan} = D(a) \). |
| 13. Update policy network \( \pi \) via backpropagation. |
| 14. Generate fake actions: \( fake\_actions = G(z) \). |
| 15. Get real actions: \( real\_actions = \pi(s).detach() \). |
| 16. Compute discriminator loss: \( -[\log(D(real)) + \log(1 – D(fake))] \). |
| 17. Compute generator loss: \( -\log(D(fake)) \). |
| 18. Update D and G alternately. |
| 19. Soft update target networks: \( Q_i’ \leftarrow \tau Q_i + (1 – \tau) Q_i’ \). |
| 20. end for |
4. HQE-PSS Experience Replay Mechanism
The HQE-PSS mechanism aims to improve the utilization efficiency of experience samples in deep reinforcement learning. Traditional experience replay stores all transitions uniformly and samples uniformly, which leads to the dominance of low-quality samples. HQE-PSS introduces two key modules: Dynamic Quality Evaluation (HQE) and Priority-based Sample Selection (PSS).
4.1 Dynamic Quality Evaluation (HQE)
HQE evaluates each incoming transition based on its reward statistics relative to the current buffer. First, I compute the mean \( \bar{r} \) and standard deviation \( \sigma \) of all rewards in the buffer:
\[
\bar{r} = \frac{1}{N} \sum_{i=1}^{N} r_i, \quad \sigma = \sqrt{ \frac{1}{N} \sum_{i=1}^{N} (r_i – \bar{r})^2 }
\]
where \( N \) is the number of samples currently stored. Then, I assign a priority score to each sample using the exponential form of the Z-score:
\[
P(s,a) = \exp\left( \frac{r – \bar{r}}{\sigma} \right)
\]
A dynamic threshold \( T_{high} \) is defined as:
\[
T_{high} = \lambda \cdot \bar{r} + (1 – \lambda) \cdot \max(r)
\]
where \( \lambda \) is a hyperparameter controlling the threshold. Samples with \( P(s,a) \geq T_{high} \) are stored in the high-quality region; otherwise, they are placed in the low-quality region.
4.2 Priority-based Sample Selection (PSS)
During training, PSS samples 80% of the batch from the high-quality region and 20% from the low-quality region. This ensures that valuable experiences are replayed more frequently while retaining a small degree of exploration via low-quality samples. The sampling weights are adjusted as:
\[
w_i = \left( \frac{P_i}{\max(P)} \right)^{-\beta}
\]
where \( \beta \) controls the importance correction. This mechanism is summarized in Algorithm 2.
| Storage: For a new transition \( (s, a, r, s’, done) \): |
| 1. Compute \( \bar{r} \) and \( \sigma \) from the buffer. |
| 2. Compute priority \( P(s,a) = \exp((r – \bar{r})/\sigma) \). |
| 3. Set threshold \( T_{high} = \lambda \bar{r} + (1-\lambda) \max(r) \). |
| 4. If \( P(s,a) \geq T_{high} \), store in high-quality region; else store in low-quality region. |
| 5. If buffer is full, apply low-quality cleanup (remove 80% of samples below the 20th percentile). |
| Sampling: When sampling a batch of size B: |
| 1. Sample \( 0.8B \) transitions from the high-quality region randomly. |
| 2. Sample \( 0.2B \) transitions from the low-quality region randomly. |
| 3. Compute importance weights \( w_i \) as above. |
| 4. Return the batch and weights for training. |
5. Experimental Setup
All experiments are conducted in the Webots simulation platform (version R2021a) using the Robotis OP2 humanoid robot model. The robot has 20 degrees of freedom and is equipped with an inertial measurement unit, a camera, and joint encoders. The simulation environment mimics a FIRA obstacle course with a gate structure. The gate dimensions are 30 cm in height and 40 cm in width, while the robot is 46 cm tall. The overall scenario is 300 cm long and 200 cm wide.
Two task configurations are considered: fixed gate position and random gate position. In the fixed configuration, the gate remains at the same location for all episodes. In the random configuration, the gate’s x-coordinate and z-coordinate are sampled uniformly from \( U(1.03, 1.71) \) and \( U(1.50, 3.40) \), respectively. The humanoid robot starts at position \( (1.5, 1.1) \) in both cases.
Hardware and software environments are listed in Table 3.
| Component | Configuration |
|---|---|
| CPU | AMD Ryzen 9 5950x 16-Core Processor |
| Memory | 32 GB |
| GPU | NVIDIA GeForce RTX 3070 |
| Operating System | Windows 10 |
| Simulation Platform | Webots R2021a |
| Programming Language | Python 3.8 |
| Deep Learning Framework | PyTorch 1.13.1 |
| CUDA Version | 11.7 |
For all algorithms, I set the training episodes to 1500. The learning rates are 0.0003 for GAN-SAC and SAC, 0.0001 for PPO, and 0.001 for DQN. The discount factor is 0.99 for all algorithms. The experience replay buffer capacity is 5000 when applicable. For HQE-PSS, the high-quality sample ratio is 80%, and the low-quality cleanup threshold is 20%.
6. Results and Discussion
6.1 Comparison of GAN-SAC with Baseline Algorithms
I evaluate GAN-SAC against DQN, PPO, and SAC in both fixed and random gate scenarios. Table 4 reports the final task success rate and the number of episodes required to converge. The success rate is defined as the percentage of episodes in which the humanoid robot successfully passes through the gate without falling.
| Algorithm | Fixed Scenario Success Rate (Convergence Episodes) | Random Scenario Success Rate (Convergence Episodes) |
|---|---|---|
| GAN-SAC | 100% (220) | 92% (610) |
| SAC | 100% (420) | 89% (600) |
| DQN | 99% (420) | 90% (620) |
| PPO | 80% (oscillation) | 72% (400) |
Table 5 shows the average episode reward and convergence episodes for the training process.
| Algorithm | Fixed Scenario Reward (Convergence Episodes) | Random Scenario Reward (Convergence Episodes) |
|---|---|---|
| GAN-SAC | 100 (190) | 91 (400) |
| SAC | 100 (380) | 90 (430) |
| DQN | 100 (400) | 90 (500) |
| PPO | 75 (390) | 55 (600) |
In the fixed gate scenario, GAN-SAC converges to a 100% success rate in only 220 episodes, while SAC requires 420 episodes and DQN requires 420 episodes. PPO does not converge stably and achieves only 80% success with significant fluctuations. In the random gate scenario, GAN-SAC achieves 92% success at 610 episodes, which is higher than SAC (89%) and DQN (90%). PPO performs poorly with only 72% success. The reward curves indicate that GAN-SAC has a faster and more stable improvement in early training, thanks to the GAN-generated diverse samples that reduce the cost of exploration.
6.2 Evaluation of HQE-PSS
I further integrate HQE-PSS into each baseline algorithm and compare the results. Table 6 reports the success rates and convergence episodes for the original algorithms versus their HQE-PSS enhanced versions.
| Algorithm | Fixed Scenario Success Rate (Convergence) | Random Scenario Success Rate (Convergence) |
|---|---|---|
| GAN-SAC | 100% (220) | 92% (610) |
| GAN-SAC + HQE-PSS | 100% (180) | 93% (300) |
| SAC | 100% (420) | 89% (600) |
| SAC + HQE-PSS | 100% (300) | 90% (600) |
| DQN | 99% (420) | 90% (620) |
| DQN + HQE-PSS | 100% (280) | 91% (600) |
| PPO | 80% (oscillation) | 72% (400) |
| PPO + HQE-PSS | 98% (200) | 88% (400) |
The reward-based comparison is presented in Table 7.
| Algorithm | Fixed Scenario Reward (Convergence) | Random Scenario Reward (Convergence) |
|---|---|---|
| GAN-SAC | 100 (190) | 91 (400) |
| GAN-SAC + HQE-PSS | 100 (160) | 100 (220) |
| SAC | 100 (380) | 90 (430) |
| SAC + HQE-PSS | 100 (220) | 95 (400) |
| DQN | 100 (400) | 90 (500) |
| DQN + HQE-PSS | 100 (300) | 90 (500) |
| PPO | 75 (390) | 55 (600) |
| PPO + HQE-PSS | 100 (200) | 80 (330) |
The results clearly demonstrate that HQE-PSS significantly improves the training efficiency of all algorithms. For the humanoid robot gate-passing task, HQE-PSS reduces the number of episodes required to reach the maximum success rate in the fixed scenario: GAN-SAC from 220 to 180, SAC from 420 to 300, DQN from 420 to 280, and PPO from non-convergence to 200. In the random scenario, GAN-SAC with HQE-PSS converges at 300 episodes instead of 610, and the success rate increases from 92% to 93%. The reward curves show faster growth and less fluctuation, indicating that the priority sampling mechanism effectively focuses the learning on high-reward transitions.
6.3 Key Findings
The experimental results support the following conclusions:
- GAN-SAC enhances exploration and stability: By generating synthetic samples through GAN, the humanoid robot is exposed to a richer training distribution, which reduces the number of environmental interactions needed to discover successful gate-passing strategies. The adversarial reward from the discriminator further regularizes the policy, preventing premature convergence to suboptimal behaviors.
- HQE-PSS improves sample efficiency: The dynamic quality evaluation ensures that the experience buffer retains more informative transitions. The 80/20 sampling strategy balances exploitation and exploration, leading to faster convergence and higher final success rates across all tested algorithms.
- Adaptability to dynamic gate positions: In the random gate scenario, the proposed methods maintain robust performance, demonstrating their ability to generalize to changing environmental conditions. This is essential for real-world applications where the gate location is not known in advance.
- Trajectory smoothness: The robots trained with GAN-SAC and HQE-PSS exhibit fewer oscillations and more direct paths toward the gate, as observed from the simulation trajectories. This is indirectly reflected in the higher average episode rewards and lower variance.
7. Conclusion
In this thesis, I have proposed and validated a deep reinforcement learning based approach for humanoid robot gate-passing. The GAN-SAC algorithm integrates generative adversarial networks into the soft actor-critic framework to improve policy generation quality and exploration efficiency. The HQE-PSS mechanism enhances the experience replay process by prioritizing high-quality samples and periodically cleaning low-quality ones. Extensive experiments in Webots with the Robotis OP2 humanoid robot demonstrate that the proposed methods outperform DQN, PPO, and SAC in terms of task success rate, convergence speed, and reward stability. The combination of GAN-SAC and HQE-PSS achieves the best overall performance, with a 100% success rate in the fixed gate scenario and 93% in the random gate scenario. These results suggest that the proposed techniques can be generalized to other complex robot control tasks, such as mobile robot navigation, multi-robot collaboration, and dynamic obstacle avoidance. Future work will focus on transferring the policy to a physical Robotis OP2 platform and exploring adaptive hyperparameter tuning for the HQE-PSS mechanism.