The human hand, as a primary natural interface for human-environment interaction, possesses remarkable dexterity and complexity. Achieving efficient and accurate hand motion and gesture recognition is paramount for enabling seamless human-machine-environment integration. Addressing the significant inter-individual variability in hand morphology and kinematics, this article presents a novel method for constructing a personalized, parameterized full-pose model of a dexterous robotic hand finger unit. This model is built upon magnetic-inertial sensor fusion technology, providing crucial technical support for precise perception and symbiotic interaction within human-cyber-physical systems.
Conventional techniques for creating digital twin models of the hand, such as laser 3D scanning, vision-based systems, and medical imaging, often impose stringent environmental requirements, rely on expensive equipment, involve cumbersome procedures, and may even pose risks (e.g., X-rays). Furthermore, these methods frequently lack a holistic consideration of global and key nodal information, such as precise fingertip tracking, leading to limited accuracy in full-pose (position and orientation) recognition. The proposed method leverages the biological constraints of finger anatomy to establish a finger unit kinematic model. By employing structured magnetic field marking and fusing data from inertial and magnetic sensors, a system for inverting the complete posture of a finger unit is developed, ultimately enabling the construction of a personalized digital twin.
1. Spatial Full-Pose Inversion of a Finger Unit
1.1 Full-Pose Inversion via Magnetic-Inertial Information Fusion
A single finger unit (excluding the thumb) comprises the Distal Interphalangeal (DIP), Proximal Interphalangeal (PIP), and Metacarpophalangeal (MCP) joints. The DIP and PIP joints are typically modeled as 1-degree-of-freedom (DOF) hinges allowing flexion/extension. The MCP joint offers 2 DOF: flexion/extension and abduction/adduction (within approximately ±15°). Despite the hand’s complex motion capabilities, under the physiological constraints of the finger’s skeletal structure, the posture of each phalanx is uniquely determined for a given fingertip position. Therefore, the full-pose reconstruction of a finger unit involves determining the fingertip position, the orientation of each phalangeal segment, and their lengths.
Biological tissues are largely non-magnetic. Therefore, the spatial magnetic field generated by a small permanent magnet (PM) attached to the fingertip is virtually unaffected by the hand’s tissue. This property allows for the establishment of a structured magnetic field. A triaxial magnetic sensor unit (MU) fixed at the metacarpal base (proximal to the MCP joint) measures the magnetic flux density B, which is intrinsically linked to the spatial position of the fingertip. A six-axis Inertial Measurement Unit (IMU) is placed on each phalangeal segment to measure acceleration $a_{ik}$ and angular velocity $\omega_{ik}$, which relate to the segment’s orientation. By fusing this magnetic and inertial data, both the fingertip position and the orientations of the phalangeal segments are obtained. Combined with a kinematic model of the finger, this enables full-pose inversion.
The algorithm for inverting the full-pose of the k-th finger unit, under biomechanical constraints, is as follows. The full-pose is defined by the set of phalangeal lengths $\{l_{ik}\}$, orientations $\{\Phi_{ik}\}$, and the fingertip position vector $\mathbf{r}_k$. The raw IMU data ($a_{ik}$, $\omega_{ik}$) is filtered using a Kalman filter to estimate the orientation $\Phi_{ik}$ of each segment. A kinematic model of the finger, incorporating anatomical constraints, is then used. The orientation of the fingertip segment determines the direction of the magnetic dipole moment vector $\mathbf{m}_k$ of the attached PM, assuming it is fixed perpendicular to the fingertip’s palmar plane. Using the magnetic flux density B measured by the MU and the known dipole moment direction $\mathbf{m}_k$, the position $\mathbf{r}_k$ of the PM (and thus the fingertip) is solved via a magnetic inverse problem, typically using a nonlinear optimization algorithm like Levenberg-Marquardt. Finally, with $\mathbf{r}_k$ and $\{\Phi_{ik}\}$ known, the kinematic model is solved inversely to obtain the personalized phalangeal lengths $\{l_{ik}\}$, completing the digital twin model.
1.2 Kinematic Modeling Under Phalangeal Biomechanical Constraints
For high-fidelity digital twinning of the hand, an accurate kinematic model is essential. Based on hand physiology, a spatial open-chain model for a single finger (excluding the thumb) is established, as shown in the conceptual framework. This model avoids the need for extensive per-user calibration. A global coordinate system $\{G\}$ is defined at the location of the MU on the metacarpal. The model accounts for the abduction/adduction angle $\theta_{0k}$ at the MCP joint.
Let the unit magnetic dipole moment vector be $\mathbf{m}_k = (e_k, f_k, g_k)^T$, constrained to be perpendicular to the fingertip segment’s plane. The forward kinematics relating joint angles to fingertip orientation and position are given by:
$$
\begin{aligned}
e_k &= \sin(\varphi_k) \cos(\theta_{0k}) \\
f_k &= \sin(\varphi_k) \sin(\theta_{0k}) \\
g_k &= \cos(\varphi_k)
\end{aligned}
$$
and
$$
\begin{aligned}
x_k &= l_{0k}\cos(\theta_{0k}) + [l_{1k}\cos(\varphi_{0k}) + l_{2k}\cos(\varphi_{0k} – \varphi_{1k})]\cos(\theta_{0k}) \\
y_k &= l_{0k}\sin(\theta_{0k}) + [l_{1k}\cos(\varphi_{0k}) + l_{2k}\cos(\varphi_{0k} – \varphi_{1k})]\sin(\theta_{0k}) \\
z_k &= l_{1k}\sin(\varphi_{0k}) + l_{2k}\sin(\varphi_{0k} – \varphi_{1k})
\end{aligned}
$$
where $\varphi_{0k}, \varphi_{1k}, \varphi_{2k}$ are the flexion/extension angles for the proximal, middle, and distal phalanges, respectively, $l_{0k}, l_{1k}, l_{2k}$ are the phalangeal lengths, and $\varphi_k = \varphi_{0k} – \varphi_{1k} – \varphi_{2k}$. Equation (1) shows that the fingertip orientation $\mathbf{m}_k$ can be derived directly from IMU-measured joint angles. Equation (2) shows the fingertip position $\mathbf{r}_k$ depends on both joint angles and lengths. Instead of manual measurement of $\{l_{ik}\}$, the PM’s position solved from the magnetic inverse problem provides $\mathbf{r}_k$, which is then used to compute $\{l_{ik}\}$ from the known $\{\Phi_{ik}\}$ and the kinematic equations.
1.3 Solving the Magnetic Inverse Problem
The position $\mathbf{r}_k$ of the fingertip PM is estimated by solving a magnetic dipole localization problem. When the distance between the PM and the MU exceeds roughly five times the PM’s largest dimension, the PM can be accurately modeled as a point magnetic dipole. The magnetic flux density $\mathbf{B}$ at the sensor location is given by the dipole equation:
$$
\mathbf{B}(\mathbf{r}, \mathbf{M}) = \frac{\mu_0}{4\pi} \left( \frac{3(\mathbf{M} \cdot \mathbf{r})\mathbf{r}}{r^5} – \frac{\mathbf{M}}{r^3} \right)
$$
where $\mathbf{M} = m \mathbf{m}$ is the magnetic dipole moment (A·m²), $m$ is its strength (obtained via calibration), $\mathbf{m}$ is its unit direction vector (from IMU data), $\mu_0$ is the vacuum permeability, and $\mathbf{r}$ is the vector from the dipole to the sensor. The components of $\mathbf{B}$ are:
$$
\begin{aligned}
B_{cx} &= \frac{\mu_0 m}{4\pi} \left( \frac{3(x_k e_k + y_k f_k + z_k g_k)x_k}{r_k^5} – \frac{e_k}{r_k^3} \right) \\
B_{cy} &= \frac{\mu_0 m}{4\pi} \left( \frac{3(x_k e_k + y_k f_k + z_k g_k)y_k}{r_k^5} – \frac{f_k}{r_k^3} \right) \\
B_{cz} &= \frac{\mu_0 m}{4\pi} \left( \frac{3(x_k e_k + y_k f_k + z_k g_k)z_k}{r_k^5} – \frac{g_k}{r_k^3} \right)
\end{aligned}
$$
Given the sensor measurements $\mathbf{B}_s = (B_{sx}, B_{sy}, B_{sz})$ and the known dipole moment direction $\mathbf{m}$, the position $\mathbf{r}_k = (x_k, y_k, z_k)$ is found by minimizing the nonlinear least-squares objective function:
$$
f(\mathbf{r}, \mathbf{M}) = \sum_{i \in \{x,y,z\}} (B_{si} – B_{ci})^2
$$
The Levenberg-Marquardt algorithm is employed for this optimization due to its robustness in solving such nonlinear, potentially ill-posed problems. The solved $\mathbf{r}_k$, combined with $\{\Phi_{ik}\}$, is then used in the kinematic equations to derive $\{l_{ik}\}$.
2. Magnetic-Inertial Fusion System for Finger Unit Pose Inversion
A prototype system was developed to implement the proposed method. The key components include:
- Permanent Magnet (PM): A cylindrical magnet (6 mm diameter × 10 mm height) attached to the fingertip, with a calibrated dipole moment $m = 0.3123$ A·m.
- Inertial Measurement Units (IMUs): MPU6050 sensors placed on each phalangeal segment to measure acceleration and angular velocity.
- Magnetic Sensor Unit (MU): An HMC5983 triaxial magnetometer fixed at the metacarpal base.
- Data Acquisition & Processing: An I2C multiplexer (TCA9548) manages multiple sensors, an STM32 microcontroller performs initial IMU data fusion (orientation estimation), and a host computer runs the main inversion algorithms and graphical user interface (GUI) developed in LabVIEW and MATLAB.
The system setup involves aligning the IMUs’ sensitive axes with the presumed flexion/extension axes of each finger segment, fixing the MU on the dorsum of the hand over the metacarpal, and attaching the PM to the fingertip with its dipole moment aligned perpendicular to the palmar surface of the distal phalanx.
3. Experimental Validation for Personalized Finger Unit Modeling
Experiments were conducted to validate the feasibility and accuracy of the proposed method and system, focusing on single-finger tracking.
3.1 Continuous Fingertip Trajectory Tracking on a Cylindrical Surface
To evaluate continuous path accuracy, the index finger of five right-handed volunteers performed a constrained motion: wrapping around a fixed cylindrical surface with the palm stationary. This defines a unique, repeatable fingertip trajectory. The system’s reconstructed trajectory was compared against ground truth from a high-precision commercial electromagnetic tracker (Polhemus Liberty).
The results demonstrated a close match between the trajectories estimated by our magnetic-inertial fusion system and the reference tracker for all volunteers, confirming the method’s validity for dynamic motion tracking. Individual differences in achievable fingertip positions were clearly captured. The average computation time per sample point was 30 ms, indicating good real-time performance.
3.2 Static Point Positioning Accuracy Validation
A quantitative assessment of static positioning precision was performed. Volunteers positioned their index fingertip to three predefined points (A, B, C) on a plane while keeping the hand stationary. The distance from the fingertip to the system’s origin was calculated and analyzed.
For each target point, 100 samples were taken. The average positioning error and standard deviation across five volunteers are summarized in Table 1. The results show that the average positioning error across all trials and subjects ranged between 1.43 mm and 2.81 mm, with standard deviations between 0.09 mm and 0.46 mm. This confirms the high accuracy and repeatability of the proposed full-pose inversion method. Notably, the error tended to decrease as the finger flexed more (from point A to C), consistent with observations from the continuous trajectory test.
| Target Point | Volunteer 1 Error ± Std (mm) | Volunteer 2 Error ± Std (mm) | Volunteer 3 Error ± Std (mm) | Volunteer 4 Error ± Std (mm) | Volunteer 5 Error ± Std (mm) |
|---|---|---|---|---|---|
| A | −2.37 ± 0.19 | 2.42 ± 0.10 | −2.74 ± 0.46 | −1.94 ± 0.41 | 2.81 ± 0.19 |
| B | −2.12 ± 0.18 | 2.39 ± 0.09 | 2.60 ± 0.20 | 2.56 ± 0.21 | −1.69 ± 0.12 |
| C | −1.53 ± 0.13 | 1.43 ± 0.13 | 2.30 ± 0.22 | 1.75 ± 0.17 | −1.46 ± 0.12 |
A comparison with related works on hand motion tracking, as shown in Table 2, highlights the advantages of our approach. While some methods rely purely on inertial sensing or require manual anthropometric measurements, our magnetic-inertial fusion method achieves superior fingertip positioning accuracy (1.43–2.81 mm) without needing personalized bone-length calibration for each user, offering a compact and robust solution.
| Reference | Sensing System | Permanent Magnet | Reported Position Error |
|---|---|---|---|
| [17] (Fang et al.) | Multiple IMUs | No | 7.8 – 12.3 mm |
| [18] (Yang et al.) | IMU & MU | Yes | 8.0 – 9.8 mm |
| [19] (Salchow et al.) | IMU & MU | Yes | 19.7 ± 2.2 mm |
| [20] | Multiple IMUs | No | 5.0 – 20.0 mm |
| This Work | IMU & MU | Yes | 1.43 – 2.81 mm |
4. Conclusion
This work presents a novel, high-precision method for full-pose inversion of a dexterous robotic hand finger unit, based on magnetic-inertial sensor fusion. The key contributions are:
- Personalized Kinematic Modeling: A spatial open-chain kinematic model for fingers was established based on biomechanical constraints. The use of a passive magnetic marker eliminates the need for tedious manual measurement of individual finger segment lengths.
- Sensor Fusion for Dimensionality Reduction: The fusion of inertial (orientation) and magnetic (position) data effectively reduces the high-dimensional full-pose inversion problem. By incorporating the kinematic model and the known dipole moment direction, high-accuracy fingertip positioning and segment orientation are simultaneously achieved.
- Experimental Validation: The developed prototype system was validated through experiments. Continuous path tracking showed strong agreement with a commercial tracker, and static point tests demonstrated a positioning accuracy in the range of [1.43, 2.81] mm, outperforming several existing methods.
The proposed method provides a robust and accurate technical pathway for creating personalized digital twin models of the hand. This capability is fundamental for advancing human-machine-environment perception and collaboration, a core direction in the development of next-generation Human-Cyber-Physical Systems (HCPS) and for enhancing the control and teleoperation of dexterous robotic hands. Future work will focus on extending the method to full-hand tracking with multiple fingers and integrating it into real-time teleoperation frameworks for complex manipulation tasks.