My research is driven by a fundamental challenge in robotics: enabling machines to interact with and understand their physical environment autonomously, especially when their primary sense—vision—fails. In cluttered industrial settings, murky underwater depths, or the stark void of space, visual sensors become unreliable or useless. In these scenarios, the sense of touch, or haptic perception, becomes the critical channel for gathering information. For a robot to grasp and manipulate an unknown object successfully, it must first construct a mental model of it. This work addresses the core problem of how to guide a multi-fingered dexterous robotic hand to autonomously and efficiently collect the geometric information of a completely unknown object using only tactile feedback, mimicking the intuitive strategies humans employ.
The problem is non-trivial. Without prior knowledge of the object’s shape, size, or orientation, planning a sequence of exploratory motions is complex. The strategy must not only be effective in gathering informative data but also inherently safe, avoiding unintended collisions that could damage the object or the dexterous robotic hand itself. While previous work has often assumed some prior knowledge (e.g., rough dimensions, known pose) or focused on simpler end-effectors, my goal was to develop a strategy for a fully autonomous, high-degree-of-freedom hand starting from near-zero information, save for an initial object location.
I began by studying the experts: humans. How do we explore an object we cannot see? Observations and studies in psychology have categorized human haptic exploration into distinct “Exploratory Procedures” (EPs)—stereotyped hand movements aimed at extracting specific object properties like texture, hardness, or global shape. Inspired by this, I conducted structured behavioral observation experiments. Participants, blindfolded and under specific constraints to mirror robotic limitations (using only fingertips, no sliding), explored a variety of unknown objects. The key was not just to catalog the EPs, but to understand the higher-order strategy: the sequence, the approach angles, and the decision-making logic behind the movements.
Three critical observations emerged from these experiments and formed the cornerstone of my robotic strategy:
- Humans begin with an initial, often random, palpation phase to get a coarse estimate of the object’s bulk and posture. This rough mental model then guides a more systematic exploration to refine shape and dimensions.
- Two primary EPs dominate early exploration: a Fingertip Contact EP (FCEP) for top-down assessment and stability, and a Pinch EP (PEP) for enveloping and exploring the sides.
- The object’s estimated dimensions and orientation directly dictate the strategy. Humans tend to orient their pinch along the object’s shorter axis (for stability) and move their exploration along the longer axis (for coverage).
Translating these biological insights into algorithmic rules for a dexterous robotic hand led to the proposed anthropomorphic exploration strategy. The entire process is logically divided into two sequential stages: Top Exploration and Side Exploration.
Stage 1: Top Exploration – Building the Initial Mental Model
The objective of this stage is to obtain a preliminary, low-resolution estimate of the object’s key parameters. The robot initiates the process by positioning its hand above the estimated object location. It then executes a series of FCEPs. The dexterous robotic hand approaches the object along a vertical direction, with its four fingers (index to pinky) extended. It moves downward in small steps $\Delta s$ until contact is registered by the tactile sensors on the fingertips. By detecting when contact is lost after subsequent movements, the algorithm identifies edge points of the object’s top surface.
From this sparse set of tactile contact points $P_t = \{ (\mathbf{g}_i, \mathbf{n}_{g_i}) \}, i=1,2,…,k$, several estimates are computed:
- Height (H): The maximum z-coordinate among all contact points: $H = \max\{z_1, z_2, …, z_k\}$.
- Top Surface Approximation: All contact points are projected onto the x-y plane. A Minimum Bounding Rectangle (MBR) is fitted to these projected points. This yields estimates for:
- Length ($A$) and Width ($B$) of the top face.
- Orientation angle ($\delta$) of the primary axis relative to the world frame.
At this point, the unknown object is approximated by a 3D bounding box defined by $(A, B, H, \delta)$. This “mental model,” while crude, provides the essential geometric priors needed to plan the next, more detailed phase of exploration.
Stage 2: Side Exploration – A Category-Driven Tactical Plan
With the bounding box parameters $(A, B, H, \delta)$ from the top exploration, the robot now shifts to a more comprehensive investigation of the object’s sides using the Pinch EP (PEP). The core idea is to classify the object based on its relative dimensions and the physical constraints of the dexterous robotic hand (like finger length $l$ and maximum grasp span $d_{max}$). For each category, a tailored strategy prescribes the approach direction $\eta$, the grasp axis $\mathbf{d}$, and the exploration sweep direction $\rho$.
To formalize the strategy, let’s define key motion primitives:
- Primitive 1 (Execute PEP): The dexterous robotic hand moves along an approach vector $\eta$, executes a pinch grasp along direction $\mathbf{d}$ with a step size $\Delta s$.
- Primitive 2 (Reposition): The hand releases, moves its arm a step $\Delta\xi$ along an exploration vector $\rho$, and re-initiates Primitive 1 at the new location.
The approach vector is defined as $\eta = [\cos\alpha, \cos\beta, \cos\gamma]$, and $\theta$ is its angle relative to the vertical z-axis. The choice of $\mathbf{d}$ is crucial and follows the human tendency: for an object with primary axes $\mathbf{a}$ (length $A$) and $\mathbf{b}$ (width $B$), the pinch is preferentially aligned with the shorter axis for a secure grip. Formally:
$$
\mathbf{d} =
\begin{cases}
[\pm\sin\delta, \pm\cos\delta, 0] \text{ or } [\pm\cos\delta, \mp\sin\delta, 0] & \text{if } A < d_{max} \\
[\pm\sin\delta, \pm\cos\delta, 0] & \text{if } A \geq d_{max} \text{ and } B < d_{max}
\end{cases}
$$
where $\delta$ is the orientation of the MBR’s long axis.
The classification system uses ratios comparing object dimensions to the dexterous robotic hand‘s capabilities, such as $c_1 = H/l$ and $c_2 = A/l$, and a scaling constant $R = 2/3$. The complete categorization and corresponding tactical plans are summarized in the table below.
| Category | Discriminant Inequalities | Side Exploration Strategy |
|---|---|---|
| 1: Compact Objects (e.g., Cube, Short Cylinder) |
$B \leq A < R \cdot B$ $H < R \cdot A$ $c_1 \leq 1$ |
Approach vertically ($\eta=[0,0,-1]$). For each valid $\mathbf{d}$, execute PEP $\lceil H/\Delta s \rceil$ times. Object is fully explored from the top. |
| 2: Tall, Thin Objects (e.g., Tall Prism, Bottle) |
$H > R \cdot A$ $B \leq A < R \cdot B$ $c_1 > 1$ |
Approach horizontally along the short sides ($\eta_1, \eta_2$). Sweep vertically ($\rho=[0,0,1]$). Execute Primitive 1 to the edge, then Primitive 2, repeating $\lceil c_1 \rceil$ times. Remaining top section uses Category 1 strategy. |
| 3: Flat, Elongated Objects (e.g., Book, Long Block) |
$A > R \cdot B$ $B > H$ $c_2 > 1, c_1 \leq 1$ |
Approach vertically ($\eta=[0,0,-1]$). Sweep horizontally along the long axis ($\rho=[\cos\delta, \sin\delta, 0]$). For each $\mathbf{d}$, execute PEP $\lceil H/\Delta s \rceil$ times, then reposition (Primitive 2) $\lceil c_2 \rceil$ times. |
| 4: Large Volumetric Objects (e.g., Large Box, Thick Book) |
$H > R \cdot A$ $A > R \cdot B$ $c_1 > 1, c_2 > 1$ |
Approach horizontally from the long sides ($\eta_1, \eta_2$). Sweep vertically ($\rho=[0,0,1]$). Execute PEP for a finger-length depth, reposition, repeat $\lceil c_1 \rceil$ times. The remaining central column is explored using the Category 3 strategy. |
If an object does not fit any category (e.g., extremely large or oddly proportioned), the strategy halts to prevent unsafe interaction, as the object is deemed unsuitable for exploration or grasp with the given dexterous robotic hand.
Implementation and Experimental Validation
To validate this strategy, I implemented it on a full humanoid robotic platform. The core actuator is a five-fingered dexterous robotic hand, the HIT/DLR II hand. Each finger has three independent degrees of freedom and is equipped with a flexible tactile sensor array at the fingertip, providing precise contact point location $\mathbf{g}_i$ and surface normal $\mathbf{n}_{g_i}$ upon contact. The hand was mounted on a 7-DOF anthropomorphic arm. For these experiments, the robot’s vision system was intentionally disabled, relying solely on proprioception and tactile feedback.
The exploration primitives (FCEP and PEP) were implemented as impedance-controlled motion trajectories. The hand approaches a target region, and the fingers close in a compliant manner until contact is detected. This ensures gentle interaction and adaptability to curved surfaces. The step size $\Delta s$ was set to 24mm, balancing point cloud density with exploration time. A diverse set of 16 objects, ranging from simple geometric primitives (spheres, cones, prisms) to everyday items (bottles, toy vehicles, calculators), was used for testing.
Results and Discussion
The experiments demonstrated the effectiveness and robustness of the proposed anthropomorphic strategy. The robot successfully and autonomously explored all test objects without any prior model, avoiding collisions and systematically covering the surface. The process consistently began with a top exploration phase, yielding the initial bounding box. The system then correctly classified each object and executed the corresponding side exploration strategy.
For a Category 1 object like a plastic bottle, the dexterous robotic hand performed vertical PEPs around its perimeter. For a Category 3 object like a toy train, the strategy involved a vertical approach followed by a horizontal sweep along its length. The resulting tactile point clouds were uniformly distributed and comprehensively captured the object’s main geometric features, providing excellent data for subsequent recognition or reconstruction algorithms. On average, each full exploration required about 15 steps, proving the strategy’s efficiency.
The key advantages of this strategy are threefold. First, it is truly autonomous and reactive, requiring only an initial object location. Second, it is inherently safe; the top-down initial exploration and the category-based plans naturally avoid catastrophic collisions. Third, it is biologically inspired, leveraging the efficiency of human haptic search heuristics. The classification scheme elegantly translates high-level human intuition (“this is tall and thin, so I’ll feel up and down the sides”) into actionable robotic commands.
This work lays a foundation for fully touch-based robotic interaction in vision-denied environments. The point clouds generated are of high quality, suitable for shape matching against a database or for direct model fitting using, for example, superquadric surfaces. Future work will integrate this exploration strategy with real-time recognition and reconstruction modules, closing the loop from initial unknown contact to a confident, actionable model. Furthermore, extending the strategy to include dynamic tactile primitives like controlled sliding could enrich the feature extraction capability, pushing the dexterous robotic hand closer to human-like tactile intelligence. By continuing to learn from human perception and translating those principles into robust algorithms, we advance the frontier of autonomous robotic manipulation in the complex, unstructured world.