A research team has reported a new multimodal pre-weld perception method that combines stereo dense geometric reconstruction, open-vocabulary visual candidate generation, and large language model (LLM) reasoning to produce robot-executable tool center point (TCP) pose sequences for mobile welding robots operating in open, unstructured manufacturing environments. The work, published in the Transactions of the China Welding Institution (ISSN 0253-360X), targets one of the most persistent bottlenecks in embodied intelligence for welding: the inability of conventional two-dimensional and three-dimensional vision pipelines to reliably identify a workpiece, localize a groove, and plan a path when workpiece pose is arbitrary, metallic surfaces are highly reflective, textures are weak, and edge structures are similar to one another.
The study, authored by WAN Yingcai, DING Wanqin, ZHONG Bin, ZHAO Yiyang, and CHEN Huabin, with affiliations spanning the School of Materials Science and Engineering and the School of Mechanical Engineering at Shanghai Jiao Tong University and China Yangtze Power Co., Ltd., positions embodied intelligence as a systems-level problem rather than a single perception task. Rather than treating groove detection as an isolated edge-detection exercise, the proposed framework unifies binocular and RGB observation, semantic candidate generation, measurable three-dimensional geometric evidence, welding knowledge reasoning, and robot-executable pose output inside a single coherent modeling pipeline.
1. Why Open Manufacturing Scenarios Break Conventional Weld Seam Perception
Welding remains a foundational process underpinning high-end equipment manufacturing and the construction of advanced manufacturing systems. It is the principal technical means for achieving structural connection and integral forming in major equipment, and it is widely deployed across energy equipment, aerospace, shipbuilding and marine engineering, and rail transit. Structural integrity, service reliability, and manufacturing self-sufficiency all depend directly on weld quality.
As high-end equipment moves toward larger scale, greater complexity, higher performance, and more extreme service conditions, welding manufacturing places progressively stricter demands on process stability, quality consistency, process perceptibility, and equipment intelligence. However, welding is characteristically non-visible, strongly time-varying, nonlinear, and governed by coupled multi-factor interactions. Scene reconstruction, dynamic process perception, process knowledge judgment, and intelligent control therefore remain significant bottlenecks.
As manufacturing objects shift from standardized, small-size components toward large-scale, site-based, multi-variety, and low-volume components, welding robots are migrating from repeated teaching and execution inside fixed workstations toward mobile welding systems designed for open manufacturing environments. Correspondingly, the core competence of a robotic welding system expands from reproducing a single trajectory to a comprehensive set of capabilities: perceiving the working environment, recognizing the target to be welded, locating the weld seam, planning a path, and generating the pose of the wire-tip tool center point.
Vision perception is the foundational element of intelligence in robotic welding. Prior research has explored laser structured light, monocular and binocular vision, RGB-D point clouds, molten pool vision, and multi-sensor fusion. Laser vision and structured light methods can achieve high weld localization accuracy under regular fixtures, known joint types, and stable imaging conditions, while three-dimensional point cloud methods have been widely applied to weld feature extraction, scene reconstruction, and robot trajectory planning. Such approaches have delivered a degree of teaching-free automatic programming and reduced the workload of manual teaching.
Yet existing two-dimensional and three-dimensional vision planning methods still exhibit clear limitations in open manufacturing scenarios. On one hand, random workpiece pose, complex working backgrounds, strong reflection on metal surfaces, weak groove texture, and abundant similar-edge interference cause methods that depend on local geometric features or a single edge detector to suffer target misrecognition, groove mislocalization, and unstable path generation. On the other hand, the welding of complex components frequently involves multi-layer multi-pass procedures, spatial curve seams, and continuously varying torch attitudes, so local weld feature extraction alone cannot satisfy the requirements of autonomous operation for global semantic understanding, joint relationship judgment, and executable pose planning.
Pre-weld perception in open manufacturing scenarios is therefore no longer a single weld edge detection problem. It is a continuous perception-and-decision process encompassing workpiece recognition, joint relationship understanding, groove geometric verification, start and end point determination, region of interest extraction, and TCP pose generation. How to achieve robust recognition of welding targets, valid region of interest determination, and executable TCP pose generation under complex backgrounds, weak textures, and strong interference is a key problem that must be solved for mobile welding robots to move from fixed-station teaching and execution toward autonomous operation in open scenarios. This is precisely the problem that embodied intelligence approaches must address.
2. The Limits of Existing Sensors and the Role Boundaries of Large Language Models
Most existing weld recognition and groove detection methods rely on explicitly known sensor mounting positions, preset joint types, stable scanning directions, or regular fixture conditions. When the welding object sits in an unstructured environment, two-dimensional semantic segmentation readily misclassifies plate edges, fixture edges, reflective strips, and background gaps as welding regions. Conversely, when only depth or point cloud geometric rules are used, the result is vulnerable to weak texture, occlusion, depth holes, and local noise, leading to missed detection of real grooves, misjudgment of groove root position, or discontinuous centerlines.
In recent years, deep stereo matching methods have substantially improved disparity estimation through cost volume construction, correlation modeling, and recurrent updating. Methods such as RAFT-Stereo, CREStereo, and FoundationStereo demonstrate that learning-based binocular matching can recover denser depth structure in complex scenes. However, general-purpose binocular models are primarily oriented toward natural scene benchmarks, and under the strong reflections of metal surfaces, narrow-scale groove structures, and industrial-grade geometric constraints found in welding scenes, task-specific geometric evidence is still required for constraint and rechecking.
Open-vocabulary detection, prompt-based segmentation, and visual foundation models have opened new technical routes for candidate generation in complex scenes. Segment Anything Model and Grounding DINO exhibit strong open candidate recall and cross-category generalization, and can generate potential workpiece, groove, and suspected weld seam regions in response to prompts. Meanwhile, large language models and vision-language models are increasingly applied to robot task understanding, scene reasoning, and action planning, making it possible for welding robots to transition from passive recognition to task-driven perception.
Nevertheless, foundation model output cannot be used directly as the final execution result of a welding robot. A welding object must satisfy explicit physical and process constraints, including the supporting relationship between adjacent workpieces, local groove morphology, weld centerline continuity, reasonableness of start and end points, torch attitude constraints, and TCP reachability. Open visual models and large language models are therefore better suited as candidate generation, task understanding, and knowledge reasoning units; they cannot replace three-dimensional geometric verification and the robot execution interface. This explicit role boundary is a central design decision of the reported method, and it is what allows embodied intelligence to remain physically grounded.
3. Defining the Pre-Weld Perception Task as a One-Shot Mapping
Pre-weld perception is the critical front-end stage through which a mobile welding robot achieves autonomous operation in open scenarios. It completes environment understanding, target recognition, groove localization, and path preparation before arc ignition and formal welding. The goal is not simply to obtain a two-dimensional weld seam mask, but to convert RGB and binocular visual observations into a pre-weld region of interest, two-dimensional and three-dimensional weld centerlines, start and stop points, a local groove geometric model, and a TCP pose sequence — all of which carry world-coordinate meaning for the robot and can be used for welding path planning.
Welding path planning for this task faces complex open environments characterized by unknown workpiece pose, strong reflection on metal surfaces, similar background edges, and weak groove texture. The core task is to establish a continuous perception-and-decision chain running from visual observation, through geometric understanding, to the robot path interface.
Formally, the mapping from observation to executable output can be described as follows. For a given pre-weld observation instant, the input comprises the RGB image, the rectified left and right binocular or infrared images, the camera intrinsic parameters, the calibration transform from the camera coordinate frame to the robot base coordinate frame, and a natural-language welding task prompt. The output comprises the verified pre-weld region of interest, the two-dimensional and three-dimensional centerlines, the welding start and end points, the local groove point cloud, the welding geometric path, and the wire-tip TCP pose sequence.
This definition reflects the one-shot character of the method. The system takes a single binocular and RGB observation together with a task prompt as its entry point, and after internal semantic, geometric, knowledge reasoning, and feedback rechecking stages, directly outputs a TCP trajectory that can be executed by the robot. No iterative re-teaching or repeated scanning is required.
4. A Semantic-Geometric-Knowledge Framework for Embodied Intelligence
The overall framework builds a semantic-geometric-knowledge driven pre-weld perception architecture for mobile welding robots in open environments. It accepts the RGB image, left and right binocular images, camera calibration parameters, robot coordinate frame information, and the welding task prompt as inputs, and establishes a complete methodological mainline comprising system perception, binocular geometric reconstruction, coarse weld candidate extraction, fine extraction and structural optimization, LLM reasoning and verification, and TCP pose generation and execution.
Within this architecture, the binocular geometric branch recovers dense depth information using a stereo matching network built on hybrid cost volumes and gated recurrent unit based updating, and further extracts point clouds, normal variation, curvature response, groove depth, and three-dimensional voxel features. The semantic branch uses an open visual model to generate candidate regions corresponding to workpieces, grooves, and suspected weld seams, and extracts semantic labels, segmentation confidence, morphological features, and task-relevance information from those candidate regions.
On this basis, the system first applies joint constraints from semantic information and three-dimensional geometric evidence to achieve coarse extraction of the weld region and obtain a continuous candidate band. It then combines local geometric consistency, groove structural features, and spatial continuity constraints to complete fine extraction of the weld region and centerline optimization, producing a weld trajectory that satisfies the requirements of robot path planning. The LLM subsequently performs reasoning, verification, and rechecking on the weld type, groove region reasonableness, start and end point positions, and trajectory continuity under structured candidate descriptions and welding process knowledge constraints, and outputs an interpretable pre-weld region of interest together with a trajectory description. Finally, the system converts the optimized two-dimensional and three-dimensional weld centerlines into a robot TCP pose sequence, completing path smoothing, torch attitude planning, and reachability verification so that the robot system can be called directly and executed automatically. The result is an integrated pre-weld processing flow running from visual perception, through geometric understanding and knowledge reasoning, to TCP pose generation.
5. Hybrid Cost Volumes and Recurrent Updating for Dense Binocular Reconstruction
Dense binocular geometric reconstruction is the foundation of the three-dimensional geometric feature branch. Its principal role is to provide measurable spatial geometric information for verifying groove candidate regions, reconstructing the three-dimensional weld centerline, and generating the TCP pose. To address the difficulty of binocular matching caused by strong reflection on metal surfaces, weak texture, and local occlusion in welding scenes, the network takes rectified left and right binocular images or infrared binocular images as input and proceeds through shared input adaptation, shared feature encoding, hybrid cost volume construction, lightweight cost aggregation, and context-guided GRU residual updating to obtain a final disparity map, which is then converted into a depth map and a three-dimensional point cloud using camera calibration parameters.
Given rectified left and right images, a shared input adaptation unit first maps both images into a unified feature space to reduce matching bias caused by brightness differences between the left and right images, infrared response differences, and local reflection. The left and right branches then share a common feature encoder that extracts multi-scale features at quarter, eighth, and sixteenth resolution levels.
A hybrid cost volume is constructed on the quarter-scale features. A correlation cost volume explicitly measures the matching similarity of left and right features under different disparity hypotheses, while a concatenation cost volume preserves the local structure and appearance information of the left and right images. The two are combined into a hybrid cost volume. The correlation component is used to explicitly measure matching similarity under varying disparity hypotheses, and the concatenation component is used to retain local structure and appearance information. Together they improve disparity discrimination in weak-texture and reflective regions. The hybrid cost volume then undergoes separable spatial-disparity convolution and lightweight disparity attention for cost aggregation.
The aggregated cost volume first predicts an initial disparity. Because the initial disparity may still contain local errors at occlusion boundaries, in narrow and deep grooves, and in weak-texture regions, a context-guided GRU residual update module is further employed for iterative correction. After the maximum number of residual updates is reached, a final disparity is obtained, and depth is computed from the camera focal length and the binocular baseline, with a small positive constant included to prevent division by zero.
Finally, using the camera intrinsic parameters and hand-eye calibration results, valid depth pixels are back-projected into the robot base coordinate frame to form a pre-weld scene point cloud. The stereo reconstruction result therefore outputs not only a depth map, but also supplies the geometric features used downstream for groove identification: normal variation, depth valleys, curvature response, base metal regions on both sides of the groove, and three-dimensional voxels. The stereo matching network is thus the core geometric reconstruction module in the pre-weld perception chain, converting two-dimensional visual observations into measurable three-dimensional geometric representations and providing the foundation for subsequent groove verification, centerline reconstruction, and TCP pose generation.
6. Extracting Groove-Related Geometric Evidence
After the depth map and point cloud are obtained, the system extracts geometric evidence related to the physical structure of the groove. A groove and its centerline are not arbitrary two-dimensional texture features; they are three-dimensional spatial structures jointly constrained by the surface relationship of adjacent workpieces, local geometric depressions, normal discontinuities, and torch reachability. The system therefore estimates surface normals from local point cloud neighborhoods and characterizes directional discontinuity through the difference between the current normal and the neighborhood average normal.
Multi-scale depth differencing is then used to enhance V-shaped groove weld regions and local depressions. A larger depth-difference response typically corresponds to the groove root depth estimate or a recessed region. The system further combines local second-order geometric variation to construct a curvature response, and applies a plane fitting residual to determine whether a candidate region is attached to a valid workpiece surface.
The supporting relationship of the base metal regions on both sides of the groove is used to judge whether reasonable adjacent workpiece surfaces exist on either side of a candidate region. Specifically, local point clouds are sampled on both sides of a candidate centerline and plane fitting is performed. If both fitted planes have a sufficient inlier ratio and the angle between their normals satisfies the joint geometric constraint range, the candidate is considered to have high groove structural rationality. The geometric branch can therefore supply the LLM with quantifiable physical evidence: whether groove depth information is reliable, whether base metal support exists on both sides of the groove, and whether the candidate is a genuine groove region or an interfering structure such as a base metal edge or a reflective strip in the groove.
7. Open-Vocabulary Semantic Candidates and Structured Representation
The semantic branch corresponds to the semantic feature module in the overall framework and is used to generate open candidate regions from the RGB image and to extract semantic elements including workpieces, grooves, suspected weld seams, base metal edges, and background interference. The open visual model does not directly output the final weld seam. Instead it generates a candidate set using a recall-first strategy.
For each candidate, the system extracts the category or semantic label, segmentation confidence, welded workpiece region area, aspect ratio, boundary morphology, relative relationship with the workpiece region, and degree of match with the task prompt. Combined with the effective depth ratio, base metal support on both sides of the groove, normal stability, and depth valley response output by the geometric branch, a unified structured candidate descriptor can be constructed.
The structured candidate description encodes semantic relevance, geometric consistency, effective depth ratio, workpiece normal plane, normal variation, groove root depth estimate response, and the base metal normal plane relationship on the groove flanks. This structured candidate expression means that the LLM no longer reads unconstrained image regions directly. Instead, it performs reasoning on evidence that is measurable, interpretable, and recheckable — a design choice that is essential for trustworthy embodied intelligence in industrial settings.
8. LLM-Guided Multi-Round Reasoning and Feedback Rechecking
The LLM-centered pre-weld perception multi-round reasoning and feedback rechecking workflow operates as follows. The semantic branch supplies candidate elements such as workpieces, grooves, suspected weld seams, and similar interference regions. The geometric branch supplies physical evidence such as groove depth maps, point clouds, normal variation, groove root depth values, and base metal support on both sides of the groove. Together they form a structured candidate description, which is fed into the LLM together with welding knowledge and process constraints.
The LLM does not directly output pixel-level weld seams. Under the constraint of measurable evidence, it performs task understanding, candidate role interpretation, relevance scoring, weldability explanation, and region of interest hypothesis generation. After the first reasoning round produces the top candidates, the system further checks geometric consistency, process executability, centerline continuity, reasonableness of start and end points, and TCP generability. If problems such as insufficient depth, missing base metal support on both sides of the groove, suspected base metal edge or reflection, or discontinuous centerline are present, abnormal evidence is fed back and reasoning is repeated. Through this closed loop of candidate interpretation, evidence checking, feedback updating, and candidate re-judgment, the system obtains a verified pre-weld region of interest, which is then used for centerline extraction, local groove point cloud reconstruction, and TCP pose sequence generation.
The LLM reasoning module receives the natural-language task prompt, the structured candidate descriptor, the spatial relationships among candidates, and welding process knowledge, and outputs candidate roles, task relevance scores, weldability explanations, and region of interest hypotheses. The welding knowledge includes joint type, groove form, start and end point direction, torch travel angle, torch inclination angle, wire extension length, and reachability constraints.
The responsibility of the LLM is not to give millimeter-level weld boundaries directly. Rather, under the constraints of structured semantic-geometric evidence, it determines whether a candidate belongs to the groove to be welded, to a base metal edge, to a reflective interference, or to a groove-like gap interference, and explains whether it has welding process significance. This is a carefully bounded role, and the researchers emphasize that such bounding is what makes embodied intelligence dependable in a physically constrained task.
To improve the stability of reasoning results, a multi-round feedback rechecking mechanism is introduced. After the first reasoning round produces the top candidates, the system checks their depth confidence, base metal support on both sides of the groove, centerline continuity, reasonableness of start and end points, and TCP generability. If a candidate exhibits problems such as insufficient depth, missing base metal support, suspected base metal edge or reflection, discontinuous centerline, or insufficient reachability, feedback information is generated and returned to the reasoning module. This forms a closed loop of candidate judgment, geometric and process checking, feedback updating, and candidate re-judgment.
This mechanism enables the system to actively exclude structures that are visually salient but physically invalid in open scenes, such as reflective strips, single-sided workpiece edges, fixture edges, and groove-like gap interference. The final candidate selection can be expressed as a joint optimization over semantics, geometry, LLM reasoning, process constraints, and TCP generability.
9. From Region of Interest to Two-Dimensional and Three-Dimensional Centerlines
When a candidate passes LLM reasoning and geometric and process rechecking, the system obtains the final pre-weld region of interest. Refinement and skeleton extraction are then performed within the region of interest to obtain a two-dimensional centerline.
Combining the depth map and coordinate transformation, the two-dimensional centerline is back-projected into a three-dimensional centerline in the robot base coordinate frame. The start point, end point, and several intermediate guide points are selected on the three-dimensional centerline, and the local groove tangential and normal directions as well as the workpiece support plane are estimated in the local neighborhood of each point. The three-dimensional centerline is resampled at a preset step size, and the local tangent is estimated from adjacent path points.
Let the local normal estimated from the local groove point cloud be the torch pointing direction, and let the cross product of that normal with the tangent define the lateral basis vector. The local torch coordinate frame can then be written accordingly. Taking into account the wire extension length, the TCP calibration offset, and the multi-pass welding offset, the TCP pose at the k-th path point for the m-th pass can be expressed, and the final TCP pose sequence follows from this formulation.
The system ultimately outputs the pre-weld region of interest, the two-dimensional and three-dimensional weld centerlines, the start and end points, the local groove point cloud, and the TCP pose sequence that can be used for robot path planning. Together, the overall framework, the stereo matching structure, and the LLM-guided reasoning workflow are unified into a single methodological flow that supports the complete pre-weld perception method running from binocular image input to executable TCP trajectory output.
10. Experimental Platform for Open-Scenario Validation
To verify the applicability of the proposed multimodal pre-weld perception method in open manufacturing scenarios, a mobile welding robot experimental platform was constructed. The platform consists of a magnetically attached mobile chassis, a six-degree-of-freedom robotic arm, a welding torch with an automatic wire feeding mechanism, a binocular vision sensor, a line laser sensor, a welding power source, a robot control system, and a perception and planning computing workstation.
The binocular vision sensor is mounted near the welding torch in an eye-in-hand configuration, and is used to synchronously acquire RGB images, left and right binocular images, and infrared images before arc ignition. The line laser sensor is used for local weld seam or groove profile measurement and provides a reference for geometric verification. The visual perception results are transformed into the robot base coordinate frame after camera-to-robot hand-eye calibration, and are further used for pre-weld region of interest determination, three-dimensional centerline reconstruction, local groove point cloud extraction, and TCP pose sequence generation.
| Module | Model or Composition | Key Parameters | Function |
|---|---|---|---|
| Mobile chassis | KGM32 series SCR-32 magnetically attached mobile platform | Dimensions 780 mm × 550 mm × 420 mm; mass approximately 130 kg; maximum vertical load 200 kg; four-wheel differential drive; permanent magnetic adhesion | Provides mobility, adhesion, and platform support on open steel structures |
| Robotic arm | Self-developed six-degree-of-freedom robotic arm | Maximum payload 10 kg; working radius 1.2 m; repeat positioning accuracy 0.01 mm | Executes torch TCP trajectory tracking and execution |
| Welding execution unit | TIG torch with automatic wire feeding mechanism | Torch mounted at the end of the robotic arm; wire diameter 1.2 mm | Completes welding end-effector action and wire feeding |
| Welding power source | Fronius TPS 440i | Used in coordination with the TIG torch and automatic wire feeding mechanism | Welding power unit |
| Binocular vision sensor | Orbbec Gemini 435Le | Baseline 95 mm; effective depth range 0.31–10 m; maximum image resolution 1280 × 800 | Acquires RGB images and left and right binocular or infrared images |
| Perception and planning workstation | High-performance GPU workstation | NVIDIA GeForce RTX 5090 GPU; AMD Ryzen 9 9950X CPU; 64 GB RAM; 4 TB SSD | LLM model inference |
11. Evaluation Metrics Spanning the Pre-Weld Perception Chain
The evaluation process focuses on three key stages of the pre-weld perception chain. First, for the binocular depth estimation results, mean absolute error, root mean square error, and absolute relative error are used to assess whether the three-dimensional reconstruction can provide a reliable depth basis for groove geometric evidence extraction. Second, for the LLM-guided candidate reasoning and region of interest determination results, region of interest level intersection over union, Dice coefficient, centerline error, endpoint error, and candidate judgment confidence are used to assess whether the fusion of semantic information, geometric evidence, and welding process knowledge can yield a compact, continuous, and positionally accurate pre-weld region of interest. Third, for the TCP pose generation results, point-wise three-dimensional error, individual coordinate component error, and effective overlap segment length are computed on the overlapping trajectory segment after common start-point alignment, in order to evaluate the spatial consistency and trajectory stability of the perception results after conversion into robot path preparation data.
| Evaluation Object | Main Metrics | Physical Meaning | Relationship to the Pre-Weld Chain |
|---|---|---|---|
| Binocular depth | MAE, RMSE, AbsRel | Measures three-dimensional geometric error and depth usability | Affects normals, depth valleys, point clouds, and groove models |
| LLM candidate reasoning | ROI IoU, Dice, centerline error, endpoint error, reasoning confidence | Measures candidate interpretation, pre-weld region of interest, and centerline stability | Affects the pre-weld region of interest, three-dimensional guide points, and TCP input |
| TCP trajectory | Three-dimensional error, component-wise error, overlap segment length | Measures spatial consistency of the path preparation result | Connects perception results to the robot control interface |
12. Binocular Depth Estimation Results
To verify the effectiveness of the binocular dense geometric reconstruction module in metal welding scenes, depth estimation evaluation experiments were carried out in a close-range steel plate welding scene. Test data were acquired with an Orbbec Gemini 435Le binocular camera and comprised 1 020 binocular image frames. Considering that factors such as metal surface reflection, edge occlusion, infrared illumination variation, and structured-light holes may introduce local reference errors, the structured-light depth output by the RGB-D sensor was used as a sensor reference value rather than as a strictly true-depth ground truth. The evaluation focused on comparing the consistency between different methods and the sensor reference depth, stability under changes in infrared imaging conditions, and the ability of the resulting depth to support subsequent groove geometric analysis.
The proposed method achieved the lowest depth estimation error on the welding scene images, with mean absolute error, root mean square error, and absolute relative error of 0.97 mm, 3.62 mm, and 0.002 4 respectively. The results are better overall than those of IGEV-Stereo, Selective-RAFT, and Selective-Stereo. Compared with IGEV-Stereo, the mean absolute error, root mean square error, and absolute relative error were reduced by 71.47 percent, 52.37 percent, and 61.90 percent respectively. Compared with Selective-RAFT, the reductions were 75.13 percent, 45.15 percent, and 67.57 percent. Compared with Selective-Stereo, the reductions were 78.44 percent, 76.03 percent, and 72.41 percent.
| Method | MAE (mm) ↓ | RMSE (mm) ↓ | AbsRel ↓ |
|---|---|---|---|
| Proposed method | 0.97 | 3.62 | 0.002 4 |
| IGEV-Stereo | 3.40 | 7.60 | 0.006 3 |
| Selective-RAFT | 3.90 | 6.60 | 0.007 4 |
| Selective-Stereo | 4.50 | 15.10 | 0.008 7 |
Visual comparison further indicates that the hybrid cost volume construction and GRU iterative updating mechanism can effectively improve disparity estimation stability under conditions of metal reflection, weak texture, and locally complex structure, thereby providing more reliable depth information support for subsequent point cloud generation, normal estimation, and candidate groove verification.
An absolute depth measurement comparison was also carried out. Under conditions of metal reflection, abrupt change in the workpiece thickness direction, and local occlusion, the raw structured-light depth exhibited a certain measurement deviation, with a representative measured value of 13.547 9 cm. In comparison, the value obtained by the proposed method was 13.821 0 cm, which is closer to the scale reference value. Combined with the error statistics, the proposed binocular depth estimation module demonstrates good absolute depth consistency and scene robustness in close-range welding scenes, and can provide a reliable spatial geometric foundation for subsequent point cloud reconstruction, groove geometric verification, three-dimensional centerline recovery, and TCP pose generation.
13. LLM-Guided Pre-Weld Candidate Reasoning Results
To verify the effect of LLM-guided pre-weld candidate interpretation and result rechecking, a locally deployed Gemma3 model was used as the welding task reasoning backend. Its ability in candidate understanding, pre-weld region of interest determination, and centerline generation in open welding scenes was evaluated while keeping the semantic candidate generation, geometric descriptor construction, and post-processing pipeline consistent.
Reasoning results were obtained for a flat plate butt welding scene with a V-shaped groove. The region to be welded presented interference factors including reflection on both sides of the groove, an aluminum alloy profile frame around the test plate, and local texture. When relying solely on two-dimensional semantic segmentation, large-area welded test plate regions, prominent edges, or groove-like gap structures were readily misjudged as weld seam regions. The semantic branch therefore first generates candidate regions for workpieces, grooves, and suspected weld seams, while the geometric branch extracts groove root and local structural abrupt-change information based on the depth map, normal vectors, curvature response, fitting residual, and groove root depth, providing the LLM with quantifiable candidate evidence.
After coarse candidate extraction, centerline optimization, and LLM rechecking, the system constrains a large-scale semantic candidate into a continuous, slender pre-weld region of interest consistent with the welding task, and outputs the start point, midpoint, and end point, providing a basis for subsequent three-dimensional guide point extraction and TCP pose sequence generation. In this process, the LLM does not directly undertake pixel-level weld seam segmentation. Instead, based on the structured candidate description, semantic segmentation results, three-dimensional geometric evidence, and welding task prompt, it performs candidate role interpretation, task relevance evaluation, groove structural rationality analysis, and region of interest hypothesis generation. The system then rechecks geometric consistency, centerline continuity, base metal support relationships on both sides of the groove, reasonableness of start and end points, and TCP generability. For candidate regions with insufficient depth evidence, suspected base metal edge or reflective interference, or discontinuous centerlines, feedback information is used to update candidate evidence and re-judge, realizing iterative reasoning across candidate interpretation, evidence checking, feedback correction, and result confirmation.
| Reasoning Model | Deployment | Test Groups | Success / Failure | ROI IoU ↑ | Dice ↑ | Centerline Error (pixel) ↓ | Endpoint Error (pixel) ↓ | Reasoning Confidence ↑ |
|---|---|---|---|---|---|---|---|---|
| Gemma3 | Local deployment | 13 | 13 / 0 | 0.874 3 | 0.968 5 | 1.59 | 14.58 | 0.96 |
In 13 groups of open welding scenes, the system completed candidate interpretation and pre-weld region of interest rechecking in every case, indicating that the structured candidate expression and geometric rechecking mechanism help improve the stability of LLM reasoning results. The region of interest level intersection over union, refined weld band Dice coefficient, and candidate judgment confidence were 0.874 3, 0.968 5, and 0.96 respectively, while the centerline error and endpoint error were 1.59 pixels and 14.58 pixels. These results indicate that the proposed method can obtain relatively accurate and continuous pre-weld region of interest and centerline results, and can satisfy the input requirements for subsequent three-dimensional centerline back-projection and TCP pose generation.
14. TCP Generation and Welding Path Planning Consistency
After the pre-weld region of interest, three-dimensional centerline, and local groove model were obtained, the ability of the TCP pose generation module to convert perception results into robot path preparation data was further verified. In the robot base coordinate frame, the TCP trajectory generated by pre-weld perception was compared with a line laser reference trajectory. Considering that the start and end positions and sampling ranges of the two trajectories differ, a common start-point alignment approach was adopted and trajectory consistency analysis was carried out within the overlapping path segment.
After common start-point alignment, the TCP-generated trajectory and the reference trajectory showed good consistency in spatial distribution. Within the overlap segment, the two trajectories extended essentially along the same physical weld direction, indicating that the proposed method can effectively recover weld spatial geometry and generate a continuous robot path. The point-wise three-dimensional error and individual coordinate component error distributions over the overlapping path segment show that the trajectory error is generally stable, with most errors concentrated within 2 mm. The scaled three-dimensional error had a mean value of 0.93 mm and a 95 percent quantile value of 1.68 mm, with only minor fluctuations at local positions, indicating good local continuity and spatial stability of the generated trajectory.
| Evaluation Item | Result | Description |
|---|---|---|
| Line laser travel trajectory | 1 011 points, 88.92 s | Reference motion path in the robot base coordinate frame |
| TCP-generated trajectory | 958 points, 76.95 s | Generated from pre-weld perception and local groove geometry |
| Common start-point overlap length | 341.61 mm | Used for fair comparison after removing non-overlapping start and end segments |
| Mean / median / maximum three-dimensional error | 1.100 mm / 1.143 mm / 1.994 mm | Reflects TCP trajectory consistency on the same physical path segment |
| Scaled error mean / 95 percent quantile | 0.93 mm / 1.68 mm | Corresponds to the error trend and high-quantile error in the overlap segment |
| X / Y / Z mean absolute error | 0.124 mm / 0.933 mm / 0.608 mm | Errors mainly originate in the Y and Z directions |
The line laser reference trajectory contained 1 011 trajectory points, while the TCP-generated trajectory contained 958 trajectory points, with an overlapping path length of 341.61 mm. The mean, median, and maximum three-dimensional errors on the overlap segment were 1.100 mm, 1.143 mm, and 1.994 mm respectively. The mean absolute errors in the X, Y, and Z directions were 0.124 mm, 0.933 mm, and 0.608 mm, with errors mainly distributed in the Y direction and the smallest error in the X direction.
Considered together, the generated TCP trajectory shows high consistency with the reference trajectory, verifying the effective connection among binocular geometric reconstruction, LLM candidate reasoning, and TCP pose generation. The results indicate that the proposed method can achieve stable conversion from pre-weld perception results to robot-executable path preparation data, and can meet the millimeter-level accuracy requirements of pre-weld path generation for mobile welding robots. For embodied intelligence in welding, this closes the loop between perception and physical action.
15. Principal Conclusions
- The research proposes an LLM-enhanced multimodal fusion pre-weld perception method for mobile welding robots using binocular vision. The method fuses RGB semantic information, binocular three-dimensional geometric information, and welding process knowledge to construct a pre-weld perception flow running from candidate generation, through structured reasoning and geometric rechecking, to TCP pose generation, achieving continuous conversion from a pre-weld region of interest to robot path preparation data in open scenarios.
- The research establishes a geometric evidence expression method oriented toward groove identification and TCP pose generation. Through descriptors including normal variation, groove depth, curvature response, local plane support, and base metal support on both sides of the groove, the method can effectively characterize local groove spatial structure and suppress interference from workpiece reflection, shadow, base metal edges, and groove-like gap structures, providing a geometric basis for weld centerline reconstruction and torch attitude construction.
- On 1 020 frames of depth evaluation data, the proposed method achieved a mean absolute error and root mean square error of 0.97 mm and 3.62 mm respectively. Across 13 open welding scenes, region of interest level intersection over union, refined weld band Dice coefficient, centerline error, and endpoint error were 0.874 3, 0.968 5, 1.59 pixels, and 14.58 pixels respectively. The generated TCP trajectory showed an average three-dimensional deviation of 1.100 mm on the common start-point overlap segment. The results verify the effectiveness of the proposed method in pre-weld region of interest determination, three-dimensional centerline recovery, and robot-executable welding path planning.
16. Significance for the Broader Field of Embodied Intelligence
The contribution of this work extends beyond a single welding application. The research addresses a general problem in embodied intelligence: how to combine the open-ended generalization of foundation models with the hard physical constraints that govern real industrial actuation. By explicitly defining the engineering role boundary of the LLM — as a task-level reasoning and decision-support unit operating over measurable, interpretable, and recheckable semantic-geometric-process evidence, rather than as a direct source of pixel-level weld regions or robot actions — the framework offers a template for deploying embodied intelligence safely and verifiably in high-stakes manufacturing contexts.
The semantic-geometric structured candidate representation is central to this approach. By jointly encoding open-vocabulary visual candidates with valid depth ratio, groove root depth estimation, groove neighborhood planar constraints, and bilateral base metal support relationships, the method improves false-positive suppression in complex open welding scenarios. This kind of evidence fusion is precisely what allows a language model to reason meaningfully about a physical task without overstepping into domains where its outputs cannot be trusted.
The binocular dense geometric reconstruction method, built on hybrid cost volumes and GRU-based iterative updating, supplies the three-dimensional evidence required for groove verification, including depth maps, point clouds, normal discontinuities, curvature responses, depth valleys, and neighborhood planar constraints. The LLM-driven candidate interpretation and feedback rechecking mechanism then performs candidate self-checking, evidence updating, region of interest confirmation, and TCP pose sequence generation under the guidance of welding knowledge and process constraints.
Together, these components form a pre-weld integrated perception framework running from images to TCP pose sequences, unifying binocular and RGB observation, semantic candidates, three-dimensional geometric evidence, welding knowledge reasoning, and robot-executable pose output within a coherent modeling pipeline. The reported accuracy figures — a mean absolute depth error of 0.97 mm, a region of interest intersection over union of 0.874 3, a weld region Dice coefficient of 0.968 5, and an average TCP trajectory three-dimensional deviation of 1.100 mm — indicate that the approach delivers the pre-weld perception accuracy and robotic execution adaptability required in complex open welding scenarios.
For the ongoing development of embodied intelligence in manufacturing, the work suggests a practical division of labor: foundation models generate and interpret candidates, geometric reconstruction validates physical plausibility, process knowledge enforces manufacturability, and the robot interface receives only verified, reachable, and continuous pose sequences. Each stage remains auditable, and each stage contributes a specific and bounded form of intelligence to the overall system. The researchers indicate that the framework provides a perception and decision foundation for autonomous robotic welding of complex components, and that the underlying principle of grounding open-ended model reasoning in measurable physical evidence is likely to transfer to other embodied intelligence domains where safety, precision, and traceability are non-negotiable.
17. Publication Details
- Title: LLM-enhanced stereo vision perception and TCP pose generation for embodied AI-enabled robotic welding
- Authors: WAN Yingcai, DING Wanqin, ZHONG Bin, ZHAO Yiyang, CHEN Huabin
- Affiliations: School of Materials Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China; School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China; China Yangtze Power Co., Ltd., Yichang 443000, China
- Journal: Transactions of the China Welding Institution
- ISSN: 0253-360X
- CN: 23-1178/TG
- Article Classification Codes: TG409; TP391.41; TP242
- Document Type: A
- Online First Publication Date: 2026-09-14
The reported work advances embodied intelligence for welding by demonstrating that a single binocular and RGB observation, combined with a natural-language task prompt, can be transformed into a robot-executable TCP pose sequence with millimeter-level spatial consistency. As mobile welding robots continue their migration from fixed workstations into open, site-based, and highly variable manufacturing environments, methods that unify semantic understanding, dense three-dimensional reconstruction, process knowledge reasoning, and verified execution interfaces will be essential. The findings reported here indicate that such unification is achievable, and that grounding large-model reasoning in measurable geometric and process evidence is a viable route toward dependable embodied intelligence in industrial robotic welding.