In the realm of space exploration and operations, teleoperation of robotic systems is a critical technology for performing tasks in hazardous or inaccessible environments. A persistent and significant challenge in space teleoperation is the substantial communication time delay between the ground station (operator) and the remote space robot. This delay, often on the order of several seconds, can destabilize direct continuous control loops and severely degrade operational performance, forcing operators into a inefficient “move-and-wait” strategy. Furthermore, limited communication bandwidth restricts the flow of sensory feedback, such as real-time video, impairing the operator’s situational awareness and decision-making. To address these issues, this work presents a comprehensive teleoperation simulation system that leverages 3D predictive graphics and virtual reality (VR) technology. The core idea is to allow an operator to interact in real-time with a virtual model of a dexterous robotic hand within a simulated environment. This local simulation provides immediate visual feedback and command validation, effectively decoupling the operator from the deleterious effects of the long communication delay. Validated command sequences are then transmitted across a simulated delayed channel to the remote physical dexterous robotic hand, which replicates the motions after the delay. This paper details the system architecture, the modeling of the virtual environment—including geometric, kinematic, and dynamic aspects—with a particular focus on the motion mapping from a human operator’s hand to the dexterous robotic hand. A mathematical model for simulating space communication delays is also presented, and the overall system is validated through simulation experiments.
The proposed system is designed to create a natural and efficient human-robot interaction interface for teleoperation. The fundamental architecture comprises four primary components: the Human Operator, the Master System (local site), the Communication Channel, and the Slave System (remote site). The human operator wears a data glove and uses standard input devices (keyboard, mouse) to control the virtual scene. The master system hosts the virtual reality simulation environment, which includes a precise 3D model of the dexterous robotic hand and its workspace. This environment performs real-time kinematics, dynamics, collision detection, and graphics rendering. The operator’s commands, captured via the data glove, are instantly applied to the virtual dexterous robotic hand, allowing for continuous and responsive control. The communication channel is modeled to introduce a substantial, variable time delay characteristic of space links. Finally, the slave system consists of the controller and the actual physical dexterous robotic hand located in the remote space environment. It receives the delayed command stream and executes the movements previously performed by the virtual model. This predictive display paradigm ensures operational stability and transparency for the operator, as the control loop is closed locally around the virtual model without delay.
The software architecture of the master system is modular and integrates several key functionalities. The core modules include: Human Motion Capture (via data glove), Motion Mapping (from human hand to dexterous robotic hand), Collision Detection, Virtual Force Computation, Collision Response, 3D Graphics Rendering, a Pseudo-Random Number Generator (for delay variation), and the Communication Delay Simulator. These modules interact within a framework built upon the Vega Prime graphics engine and specific libraries for virtual hand development and collision detection. The data flow begins with the data glove capturing the operator’s hand posture. This data is processed by the motion mapping module to generate corresponding joint angle targets for the virtual dexterous robotic hand. The kinematics module updates the hand’s pose, and the collision detection module (using the V-Clip library) checks for interactions with virtual objects. If contact occurs, virtual forces are computed based on a spring-damper contact model, and appropriate visual or haptic feedback can be generated. The graphics engine continuously renders the scene. Simultaneously, the validated joint trajectory data is sent to a buffer that applies a simulated transmission delay before being forwarded (in a simulation context) to the slave robot controller. This structure effectively separates the real-time interaction loop from the delayed physical execution loop.
Constructing a realistic and responsive virtual environment is paramount for effective predictive display. This involves several layers of modeling: geometric, kinematic, dynamic, and perceptual.
Geometric Modeling
The 3D models for the dexterous robotic hand and its operational environment are created using modeling tools like Multigen-Creator. The dexterous robotic hand is assembled from basic primitives (cubes, cylinders, spheres) into components representing links and joints. A hierarchical tree structure is employed to define the kinematic chain. Each link is a child node of its proximal joint, ensuring that transformations (rotations) applied to a parent joint are inherited by all downstream child links. This hierarchy is crucial for efficient forward kinematics computation during runtime. The entire assembly is exported as a scene graph compatible with the Vega Prime engine. The operational environment, which may include tools, panels, or other objects for the dexterous robotic hand to manipulate, is modeled similarly and integrated into the same virtual scene.
Kinematic Modeling
Accurate kinematic modeling is essential for controlling the dexterous robotic hand. The specific hand considered in this work is a three-fingered dexterous robotic hand, with each finger having three degrees of freedom (DOF): one abduction/adduction joint at the base and two flexion/extension joints. This structure is representative of many research-oriented dexterous robotic hands. The Denavit-Hartenberg (D-H) convention is used to systematically define the coordinate frames and parameters for each finger. For a single finger, the simplified structure and coordinate frames are shown schematically. The D-H parameters for the three-link finger are summarized in Table 1.
| Link i | Joint Angle $\theta_i$ | Twist Angle $\alpha_i$ | Link Length $a_i$ | Link Offset $d_i$ |
|---|---|---|---|---|
| 1 | $\theta_1$ | $0^\circ$ | $l_1$ | 0 |
| 2 | $\theta_2$ | $-90^\circ$ | $l_2$ | 0 |
| 3 | $\theta_3$ | $0^\circ$ | $l_3$ | 0 |
The transformation matrix between consecutive links, $^{i-1}\mathbf{A}_i$, is given by the standard D-H formula:
$$^{i-1}\mathbf{A}_i = \begin{bmatrix}
\cos\theta_i & -\sin\theta_i \cos\alpha_i & \sin\theta_i \sin\alpha_i & a_i \cos\theta_i \\
\sin\theta_i & \cos\theta_i \cos\alpha_i & -\cos\theta_i \sin\alpha_i & a_i \sin\theta_i \\
0 & \sin\alpha_i & \cos\alpha_i & d_i \\
0 & 0 & 0 & 1
\end{bmatrix}$$
For the finger, the overall transformation from the finger base frame {0} to the fingertip frame {3} is:
$$^{0}\mathbf{T}_{3} = ^{0}\mathbf{A}_{1} \cdot ^{1}\mathbf{A}_{2} \cdot ^{2}\mathbf{A}_{3}$$
The position of the fingertip, $\mathbf{P}_{tip} = [p_x, p_y, p_z, 1]^T$, in the base frame is obtained by multiplying this transformation matrix with the fingertip’s coordinates in its own frame (typically $[0, 0, 0, 1]^T$). The resulting forward kinematics equations are:
$$\begin{aligned}
p_x &= l_1 \cos\theta_1 + l_2 \cos\theta_1 \cos\theta_2 + l_3 \cos\theta_1 \cos(\theta_2 + \theta_3) \\
p_y &= l_1 \sin\theta_1 + l_2 \sin\theta_1 \cos\theta_2 + l_3 \sin\theta_1 \cos(\theta_2 + \theta_3) \\
p_z &= -l_2 \sin\theta_2 – l_3 \sin(\theta_2 + \theta_3)
\end{aligned}$$
These equations allow us to compute the fingertip position given the joint angles ($\theta_1, \theta_2, \theta_3$).
Inverse Kinematics
For motion mapping and control, we often need the inverse: determining the joint angles required to place the fingertip at a desired position $(p_x, p_y, p_z)$. Solving the system of equations above yields the inverse kinematic solution. For this specific 3-DOF finger geometry, a closed-form solution exists. By manipulating the equations, we can solve for $\theta_2$ and $\theta_3$ from the $p_z$ equation and the distance in the XY-plane, and then solve for $\theta_1$ using the $p_x$ and $p_y$ relations. This inverse solution is computed in real-time within the motion mapping module to determine the target joint angles for the dexterous robotic hand based on desired fingertip positions.
Motion Mapping from Human Hand to Dexterous Robotic Hand
A critical component for intuitive control is mapping the natural movements of the human operator’s hand to the potentially different kinematics of the dexterous robotic hand. We employ a fingertip position-based mapping strategy, which is well-suited for dexterous, precision manipulation tasks. The human hand is equipped with an 18-sensor CyberGlove data glove, which provides raw sensor values corresponding to the flexion/extension and abduction/adduction of each finger joint.
The mapping process involves three main steps:
- Obtain Human Fingertip Positions: The human hand is modeled as a kinematic chain with four joints per finger (including the metacarpophalangeal joint). Using a similar D-H parameterization and the joint angles estimated from glove data, the forward kinematics of the human finger is computed to find the position of each human fingertip relative to a fixed palm coordinate system.
- Map Multiple Human Fingers to Fewer Robotic Fingers: Our dexterous robotic hand has three fingers, while the human hand has five. An effective mapping is required. We use the following mapping scheme: the human thumb maps directly to one robotic finger. The human index and middle fingers are combined (averaged) to control a second robotic finger. The human ring and little fingers are combined to control the third robotic finger. Since the human and robotic fingers have different lengths, a scaling factor (weight) is applied independently in the X, Y, and Z directions. The weight for a mapped finger pair is based on the ratio of their characteristic lengths:
$$k_x = k_y = k_z = \frac{L^R_i}{L^H_i}$$
where $L^R_i$ is the effective length of the robotic finger and $L^H_i$ is the effective length of the corresponding human finger(s). The target position for the dexterous robotic hand’s fingertip in the palm frame is then calculated as a weighted combination of the mapped human fingertip positions. - Inverse Kinematics for the Dexterous Robotic Hand: The computed target fingertip position for each finger of the dexterous robotic hand (in the robotic hand’s palm frame) is fed into the inverse kinematics solver described earlier. This yields the set of joint angles ($\theta_1, \theta_2, \theta_3$) for each finger that will achieve the desired position. These joint angles are the commanded values sent to the virtual (and subsequently, the real) dexterous robotic hand.
This mapping algorithm ensures that the operator’s hand movements are intuitively translated into commands for the dexterous robotic hand, facilitating complex manipulation.
Collision Detection and Stable Grasping
To simulate realistic physical interaction, the system incorporates collision detection and response. We use the V-Clip collision detection library, which is fast and robust for convex objects. It operates on the polygonal mesh representations of the virtual dexterous robotic hand and the objects in the environment. V-Clip returns detailed contact information, including contact points, penetration depth, and separation distance.
When a collision is detected between the dexterous robotic hand and a virtual object, a virtual contact force is computed to provide feedback and prevent unrealistic inter-penetration. A simple spring-damper model is often used:
$$\mathbf{F}_{virtual} = -k_p \mathbf{d} – k_d \mathbf{v}$$
where $\mathbf{d}$ is the penetration depth vector, $\mathbf{v}$ is the relative velocity at the contact point, and $k_p$ and $k_d$ are stiffness and damping coefficients, respectively. This force can be used for haptic feedback if a force-feedback device is available, or it can be used visually to indicate contact.
A fundamental goal in manipulation is achieving a stable grasp. In the virtual environment, we assess grasp stability by checking the force closure condition. For a multi-fingered dexterous robotic hand, a grasp is considered force-closed if the contact forces can generate any arbitrary wrench (force and torque) on the object to resist external disturbances. A simpler quasi-static equilibrium check is often performed: the vector sum of all contact forces and any external forces (like gravity) acting on the object should be zero, and the net moment about the object’s center of mass should be zero.
$$\sum \mathbf{F}_i + \mathbf{F}_{ext} = 0$$
$$\sum (\mathbf{r}_i \times \mathbf{F}_i) + \mathbf{M}_{ext} = 0$$
where $\mathbf{F}_i$ is the contact force from finger $i$, $\mathbf{r}_i$ is the position vector from the object’s center of mass to the contact point, and $\mathbf{F}_{ext}$ and $\mathbf{M}_{ext}$ are external force and moment. The virtual contact forces computed from the penetration depth are used to evaluate these conditions. If equilibrium is satisfied within a threshold, the grasp is deemed stable, and the system can lock the current hand posture or proceed with a lifting operation. This provides the operator with a crucial indication of successful task execution in the predictive simulation.
Modeling and Simulating Space Communication Delay
The communication delay between ground and space is a combination of a large, fixed propagation delay (due to the speed of light over vast distances) and a smaller, variable delay (due to processing, routing, and network congestion). To faithfully test our teleoperation system, we model the total uplink delay $T$ as:
$$T = T_m + T_i$$
where $T_m$ is a constant delay simulating the propagation time (e.g., set to 3 seconds for Earth-Moon or Earth-geostationary orbit scenarios), and $T_i$ is a variable delay component simulating network jitter.
The variable delay $T_i$ is generated using a pseudo-random number generator (PRNG). To produce high-quality random sequences with a long period and good statistical properties, we implement a combined generator. It uses a parabolic map as the base generator and a linear congruential generator (LCG) to shuffle the output sequence. The algorithm is as follows:
- Generate an initial sequence of $L$ numbers $\{t_1, t_2, …, t_L\}$ using the parabolic map: $x_{n+1} = 4x_n(1-x_n)$.
- Use an LCG to produce a random integer index $j$ uniformly distributed between 1 and $L$.
- Output $r_n = t_j$ as the next random number.
- Generate a new number $y$ from the parabolic map and replace $t_j$ with $y$.
- Repeat steps 2-4 to produce the sequence $\{r_n\}$.
This combined generator provides a sequence that is sufficiently random for simulation purposes. The generated $T_i$ values are scaled to an appropriate range (e.g., 0 to 500 ms) and added to $T_m$. The command data packets from the master system are time-stamped and placed into a first-in-first-out (FIFO) buffer. They are released to the slave system controller only after the total simulated delay $T$ has elapsed since their generation time. This effectively recreates the challenging conditions of space teleoperation within our laboratory setup.
Simulation Experiments and Validation
To validate the effectiveness of the proposed teleoperation simulation system based on the dexterous robotic hand, a series of simulation experiments were conducted. The primary objective was to demonstrate that an operator could successfully perform a complex manipulation task—such as grasping and moving a virtual object—using the predictive display, despite the presence of a simulated multi-second communication delay.
The experimental setup consisted of the master simulation station running the Vega Prime-based virtual environment, with the operator wearing the CyberGlove data glove. The virtual scene contained a detailed model of the three-fingered dexterous robotic hand and a simple geometric object (e.g., a cube or a tool). The communication delay module was activated with $T_m = 3.0$ seconds and a variable $T_i$ with a mean of 0.2 seconds. The operator’s task was to control the virtual dexterous robotic hand to approach, grasp, and lift the object.
Due to the immediate response of the virtual dexterous robotic hand, the operator could perform continuous, fluid motions. The collision detection and virtual force feedback provided clear indications of contact. The system successfully calculated the joint angles for the dexterous robotic hand in real-time via the motion mapping algorithm. The grasp stability was evaluated visually and through the equilibrium checks. Once a stable virtual grasp was achieved, the corresponding joint trajectory was recorded and sent through the delay buffer.
In a separate simulation process acting as the slave system, the delayed joint angle commands were received and applied to an identical kinematic model of the dexterous robotic hand. The results showed that the slave dexterous robotic hand accurately replicated the entire manipulation sequence performed earlier by the operator on the virtual model, albeit after a 3+ second delay. The operator, watching only the predictive display, was able to complete the task efficiently without being hindered by the latency. This confirms that the predictive simulation effectively masks the long time delay from the operator’s control loop, enabling stable and performant teleoperation of the dexterous robotic hand.
A quantitative analysis was also performed to measure task completion time and success rate with and without the predictive simulation under simulated delay conditions. As expected, direct teleoperation (where the operator commands are sent directly to the delayed slave) resulted in frequent failures, overshoot, and significantly longer completion times due to the instability induced by delay. In contrast, the predictive simulation system maintained a high success rate and consistent task completion time, independent of the introduced communication delay, validating the core concept.
Conclusion
This work has presented a comprehensive framework for a space teleoperation simulation system centered around the control of a dexterous robotic hand. By employing 3D predictive graphics and virtual reality, the system successfully addresses the critical issue of long communication time delays inherent in space applications. The operator interacts in real-time with a high-fidelity virtual model of the dexterous robotic hand, receiving instant visual and simulated haptic feedback. This local interaction loop is devoid of destabilizing delays. The validated command sequences are then transmitted across a simulated delayed channel to the remote physical dexterous robotic hand. Key contributions include the detailed kinematic modeling of the dexterous robotic hand, the development of an intuitive fingertip-based motion mapping algorithm from a human operator to the dexterous robotic hand, the integration of robust collision detection and virtual force computation for realistic interaction, and the implementation of a configurable delay model that mimics space communication characteristics. Simulation experiments have demonstrated the system’s efficacy in allowing an operator to perform dexterous manipulation tasks smoothly and successfully, despite the presence of multi-second delays. This approach significantly enhances operational efficiency and safety for future space missions involving complex robotic manipulation. Future work will focus on integrating more advanced dynamics, enhancing the fidelity of tactile and force feedback, and testing the system with a physical prototype of the dexterous robotic hand in a hardware-in-the-loop simulation.