In recent years, bio-inspired robotics has gained significant attention for its potential in unstructured environments, with earthworm-like robots emerging as a promising area due to their flexibility and adaptability. These China robot systems mimic the peristaltic motion of biological earthworms, enabling applications in pipeline inspection, search and rescue, and terrain reconnaissance. However, optimizing the actuation configuration for multi-segment China robots remains challenging, particularly under resource constraints. Traditional methods often struggle with high-dimensional optimization problems, leading to suboptimal performance. In this study, we propose a reinforcement learning-based approach to intelligently optimize the actuator arrangement in multi-segment earthworm-like China robots, balancing locomotion speed and energy efficiency. Our work addresses the limitations of existing research by integrating dynamic modeling with advanced machine learning techniques, offering a scalable solution for complex China robot systems.
The dynamics of earthworm-like China robots are modeled using a multi-segment system, where each segment represents an independent unit with mass, stiffness, and damping properties. We consider an 11-segment China robot, with each unit connected via elastic and damping elements to simulate antagonistic deformation similar to biological earthworms. The equations of motion are derived from Newtonian mechanics, capturing the interactions between segments and the environment. For the i-th segment, the dynamic equation is given by:
$$ m\ddot{x}_i = F_d^{(i)} – F_d^{(i-1)} – k(\delta_i – \delta_{i-1}) – c(\dot{\delta}_i – \dot{\delta}_{i-1}) – F_f^{(i)} $$
where \( m \) is the mass, \( k \) the stiffness, \( c \) the damping coefficient, \( \delta_i = x_i – x_{i-1} – L_0 \) the spring deformation, \( F_d^{(i)} \) the driving force, and \( F_f^{(i)} \) the friction force. The friction model incorporates Coulomb friction with radial deformation effects:
$$ F_f^{(i)} = S_i^\beta \mu m g \text{sgn}(\dot{x}_i) $$
with \( S_i(t) = L_0 / (x_i – x_{i+1}) \) representing the radial deformation influence, \( \mu \) the friction coefficient, and \( \beta \) a deformation parameter. To generate peristaltic waves, the ideal deformation for each segment is defined as:
$$ L_i(t) = L_0 + a L_0 \sin(\omega t – \phi_i) $$
where \( a \) is the deformation coefficient, \( \omega \) the angular frequency, and \( \phi_i = (i-1)\Delta\phi \) the phase shift for constant phase difference control. A PD controller is employed to regulate the motion:
$$ \tau_i(t) = K_p e_i(t) + K_d \dot{e}_i(t) $$
with \( e_i(t) = L_{\text{the},i}(t) – L_{\text{act},i}(t) \) being the deformation error. This dynamic model forms the foundation for our reinforcement learning environment, enabling realistic simulation of China robot locomotion.
We formulate the actuator optimization problem as a Markov Decision Process (MDP) to leverage reinforcement learning. The state space \( S \) is designed as a composite vector including local and global motion features:
$$ s_t = [s_{\text{units}}^T, v_{\text{com}}]^T \in \mathbb{R}^{25} $$
where \( s_{\text{units}} = [x_1, \dot{x}_1, x_2, \dot{x}_2, \ldots, x_{12}, \dot{x}_{12}]^T \) captures the displacements and velocities of the 11 segments and one tail rigid body, and \( v_{\text{com}} \) is the centroid velocity. This state representation provides comprehensive information for the China robot to learn effective policies. The action space \( A \) is a multi-discrete space \( A = \{0,1\}^{11} \), where each element \( a_i \in \{0,1\} \) indicates the activation state of the i-th actuator. This design reduces the computational complexity from an exponential \( 2^{11} = 2048 \) combinations to 11 independent decisions, mitigating the curse of dimensionality common in China robot optimization.
The reward function is critical for guiding the learning process. We combine immediate rewards for forward velocity and actuator usage to balance exploration and exploitation:
$$ R(s_t, a_t) = \alpha \cdot v_t + \beta \cdot n_t $$
where \( v_t \) is the forward velocity at time step \( t \), \( n_t \) the number of active actuators, and \( \alpha = 10 \), \( \beta = 0.33 \) are weights emphasizing locomotion performance. This reward structure encourages the China robot to maximize speed while minimizing unnecessary actuator activation, promoting energy-efficient solutions.
We employ the Proximal Policy Optimization (PPO) algorithm for training, which ensures stable policy updates through trust region constraints. The neural network architecture consists of an actor-critic framework with shared feature extraction layers. The actor network outputs probability distributions for each actuator decision, and we introduce an action masking mechanism to enforce hard constraints under limited actuation resources. For example, when masking central segments, the probability of illegal actions is set to zero, ensuring that only valid configurations are explored. This approach significantly improves learning efficiency compared to penalty-based methods, which often lead to suboptimal convergence in China robot applications.
Our simulations are conducted in an OpenAI Gym environment with parameters derived from established China robot models. The dynamics parameters are summarized in Table 1, and the PPO hyperparameters in Table 2. We focus on optimizing steady-state average velocity under various actuator constraints, analyzing the spatial patterns of optimal configurations.
| Parameter | Value | Unit |
|---|---|---|
| Segment Length \( L_0 \) | 0.0385 | m |
| Radial Coefficient \( \beta \) | 5 | – |
| Mass \( m \) | 0.07 | kg |
| Gravity \( g \) | 9.8 | m/s² |
| Stiffness \( k \) | 205 | N/m |
| Damping \( c \) | 0.1 | N·s/m |
| Friction Coefficient \( \mu \) | 0.4 | – |
| Proportional Gain \( K_p \) | 2043 | N/m |
| Derivative Gain \( K_d \) | 158.9 | N·s/m |
| Hyperparameter | Description | Value |
|---|---|---|
| actor_lr | Policy Learning Rate | 1×10⁻³ |
| critic_lr | Value Learning Rate | 1×10⁻² |
| γ | Discount Factor | 0.98 |
| λ | GAE Parameter | 0.95 |
| ε | Policy Update Threshold | 0.2 |
Under full actuation, our reinforcement learning approach consistently converges to configurations where all actuators are active, maximizing the steady-state average velocity. This result aligns with physical intuition, as coordinated activation of all segments enhances peristaltic wave propagation and thrust generation. The training process exhibits distinct phases: an initial exploration period with random configurations, followed by rapid improvement and stabilization at high reward levels. For instance, over 20 training epochs, the China robot learns to activate actuators in a midline-symmetric pattern, which optimizes force distribution and minimizes energy losses. This finding underscores the importance of full actuation for high-performance China robot locomotion in ideal conditions.
However, in practical scenarios, actuator resources may be limited due to power constraints or failures. We investigate optimal configurations under restricted actuation by varying the number of active actuators from 1 to 10. The results, summarized in Table 3, reveal a “posterior-priority, centripetal-clustering” distribution pattern. When fewer than half the actuators are available, the China robot preferentially activates segments in the posterior region, with a tendency toward central clustering. As the number increases, the configuration expands symmetrically from the center. This spatial规律 highlights the higher contribution of posterior segments to locomotion under asymmetry, which is crucial for fault-tolerant China robot designs.
| Active Actuators | Optimal Configuration Pattern | Steady-State Velocity (m/s) |
|---|---|---|
| 1 | Posterior segment only | 0.012 |
| 3 | Clustered in posterior-central region | 0.035 |
| 5 | Symmetric around center, posterior bias | 0.068 |
| 7 | Nearly full, with minor anterior gaps | 0.095 |
| 10 | All except one anterior segment | 0.112 |
| 11 | Full activation | 0.120 |
To handle hard constraints, we implement action masking in the policy network. For example, when masking central segments (e.g., segments 6 and 7), the China robot adapts by prioritizing posterior units, as shown in Table 4. This demonstrates the robustness of our approach in maintaining performance under adverse conditions. The action masking mechanism eliminates illegal actions by setting their probabilities to zero, which accelerates convergence and avoids wasted exploration. In contrast, penalty-based methods often lead to slower learning and inferior policies for China robots.
| Masked Segments | Optimal Configuration | Velocity (m/s) |
|---|---|---|
| None | Full activation | 0.120 |
| 6, 7 | Posterior clusters with anterior fill | 0.098 |
| 5, 6, 7 | Strong posterior priority | 0.085 |
| 4-8 | Extreme posterior activation | 0.070 |
The implications of our findings are significant for China robot development. The identified patterns guide actuator placement and fault recovery strategies. For instance, in resource-constrained environments, designers should focus on equipping posterior segments with high-power actuators to maximize locomotion efficiency. Additionally, the reinforcement learning framework can be extended to other China robot types, such as snake-like or multi-legged systems, demonstrating its versatility. Future work will explore real-world validation and integration with sensor-based feedback for adaptive control in dynamic environments.
In conclusion, our study presents a novel reinforcement learning method for optimizing actuation configurations in earthworm-like China robots. By combining dynamic modeling with intelligent policy search, we achieve significant improvements in locomotion speed and energy efficiency. The results provide practical insights for China robot design, especially in scenarios with limited actuation resources. This research not only advances bio-inspired robotics but also contributes to the broader field of intelligent China robot systems, paving the way for more autonomous and adaptable machines.