A research team has introduced a multimodal pre-weld perception method that couples dense binocular three-dimensional reconstruction, open-vocabulary visual candidates, task-specific geometric evidence, and large language model reasoning into a single pipeline that converts a pair of stereo images and a natural-language welding instruction into an executable tool center point pose sequence. The work positions itself explicitly within the emerging field of embodied intelligence for manufacturing, arguing that the next generation of mobile welding robots must move beyond fixed-station teaching and repetitive trajectory playback toward autonomous operation in open, unstructured fabrication environments.
The method, developed by researchers at Shanghai Jiao Tong University in collaboration with China Yangtze Power Co., Ltd., addresses a persistent bottleneck in robotic welding: the unreliable identification of workpieces, groove structures, and weld paths when workpiece poses are random, metallic surfaces produce strong specular reflections, groove textures are weak, and background edges closely resemble real seams. Traditional approaches that depend on structured laser light, single-modality edge detection, or rigid fixturing assumptions tend to produce false target identification, mislocated grooves, and unstable path generation under such conditions. The proposed framework treats pre-weld perception not as an isolated edge-detection problem, but as a continuous perception-and-decision chain spanning workpiece recognition, joint relationship understanding, groove geometry verification, start and end point determination, region of interest extraction, and tool center point pose generation.
Central to the contribution is a carefully bounded engineering role for the large language model. Rather than emitting pixel-level weld masks or direct robot motion commands, the language model operates on measurable, interpretable, and recheckable semantic, geometric, and process evidence, performing task-level reasoning and decision support. This design choice is presented as a prerequisite for trustworthy embodied intelligence in industrial settings, where every output must satisfy explicit physical and process constraints, including adjacent workpiece support relationships, local groove morphology, weld centerline continuity, start and end point rationality, torch orientation constraints, and tool center point reachability.

- Background: Why Open Manufacturing Scenarios Resist Conventional Weld Sensing
Welding underpins the structural integrity of energy equipment, aerospace systems, ship and marine engineering, and rail transportation. As高端 equipment moves toward larger scale, greater complexity, higher performance, and more extreme service conditions, welding manufacturing faces rising demands for process stability, quality consistency, process perceptibility, and equipment intelligence. Yet welding remains intrinsically difficult to observe, strongly time-varying, nonlinear, and governed by coupled multi-factor interactions. Scene reconstruction, dynamic process sensing, process knowledge judgment, and intelligent control continue to limit high-quality welding of complex components.
As fabrication targets shift from standardized, small-size components toward large, field-assembled, and high-mix low-volume structures, welding robots are migrating from repetitive teaching execution inside fixed workstations toward mobile welding systems operating in open manufacturing environments. The core competence of the robot therefore expands from single-trajectory reproduction to a composite capability set: workspace perception, target recognition, seam localization, path planning, and tool center point pose generation.
Vision sensing is the foundation of this capability expansion. Prior research has explored laser structured light, monocular and binocular vision, RGB-D point clouds, molten pool imaging, and multi-sensor fusion, and such methods have achieved a degree of teaching-free automatic programming. However, in open manufacturing scenarios, two limitations persist. First, random workpiece poses, complex backgrounds, strong metallic reflection, weak groove texture, and abundant similar-edge interference cause target misidentification and groove mislocalization. Second, complex components frequently involve multi-layer multi-pass welding, spatial curve seams, and continuously varying torch orientation, which cannot be satisfied by local seam feature extraction alone. What is required is global semantic understanding, joint relationship reasoning, and executable pose planning — precisely the capabilities that embodied intelligence seeks to unify.
- Defining the Pre-Weld Perception Task as a One-Shot Mapping
The researchers formalize pre-weld perception as a mapping from an observation input to a structured output. The input comprises an RGB image, rectified left and right stereo or infrared images, camera intrinsics, the calibration transform from the camera frame to the robot base frame, and a natural-language welding task prompt. The output comprises a verified pre-weld region of interest, two-dimensional and three-dimensional centerlines, start and end points, a local groove point cloud, a welding geometric path, and the tool center point pose sequence.
This formulation captures the one-shot character of the method. The system accepts a single stereo or RGB observation together with a task prompt, then internally performs semantic interpretation, geometric reconstruction, knowledge reasoning, and feedback rechecking, and finally emits a directly executable tool center point trajectory. No iterative re-observation or manual re-teaching is required between the image input and the robot path output, a property the authors associate with practical embodied intelligence deployment on mobile platforms.
The overall architecture follows a semantic–geometric–knowledge driven structure. The geometric branch recovers dense depth through a stereo matching network built on hybrid cost volume construction and gated recurrent unit based iterative updating, then extracts point clouds, normal variation, curvature response, groove depth, and three-dimensional voxel features. The semantic branch applies an open-vocabulary visual model to generate candidate regions for workpieces, grooves, and suspected weld seams, together with semantic labels, segmentation confidence, morphological descriptors, and task relevance information. Semantic information and three-dimensional geometric evidence are then jointly constrained to achieve coarse seam region extraction, producing a continuous candidate band. Local geometric consistency, groove structural characteristics, and spatial continuity constraints subsequently refine the seam region and optimize the centerline. The language model then reasons over groove type, groove region plausibility, start and end point placement, and trajectory continuity under structured candidate descriptions and welding process knowledge, producing an interpretable pre-weld region of interest and trajectory description. Finally, the optimized centerlines are converted into a tool center point pose sequence with path smoothing, torch orientation planning, and reachability verification.
- Dense Stereo Reconstruction as the Geometric Backbone
Dense binocular geometric reconstruction supplies the measurable spatial information required to verify groove candidates, reconstruct three-dimensional weld centerlines, and generate tool center point poses. To counteract strong metallic reflection, weak texture, and local occlusion that degrade stereo matching in welding scenes, the network accepts rectified left and right stereo or infrared image pairs and processes them through shared input adaptation, shared feature encoding, hybrid cost volume construction, lightweight cost aggregation, and context-guided gated recurrent unit residual updating.
A shared input adaptation unit first maps both views into a unified feature space, reducing matching bias caused by brightness differences, infrared response differences, and local reflections. Left and right branches then share a single feature encoder that extracts multi-scale features at one-quarter, one-eighth, and one-sixteenth resolution. At the one-quarter scale, a hybrid cost volume is constructed by concatenating a correlation volume, which explicitly measures matching similarity between left and right features across disparity hypotheses, with a concatenation volume, which preserves local structure and appearance information. Together the two components improve disparity discrimination in weakly textured and reflective regions.
The hybrid cost volume is aggregated through separable spatial–disparity convolutions and a lightweight disparity attention mechanism. An initial disparity estimate is produced first; because initial disparities may retain local errors at occlusion boundaries, in deep narrow grooves, and in weakly textured regions, a context-guided gated recurrent unit residual update module iteratively refines the estimate by looking up local cost features from the cost volume according to the current disparity and combining them with contextual features. After the maximum number of iterations, the final disparity is converted to depth using focal length, stereo baseline, and a small positive constant that prevents division by zero. Valid depth pixels are then back-projected into the robot base frame using camera intrinsics and hand-eye calibration to form a pre-weld scene point cloud.
The stereo module thereby functions as the core geometric reconstruction component of the pre-weld perception chain, converting two-dimensional visual observations into a measurable three-dimensional geometric representation and providing the basis for subsequent groove verification, centerline reconstruction, and tool center point pose generation.
- Task-Specific Geometric Evidence for Groove Verification
Once depth maps and point clouds are available, the system extracts geometric evidence tied to the physical structure of a groove. A weld groove is not an arbitrary two-dimensional texture feature; it is a three-dimensional spatial structure jointly constrained by the relationship between adjacent workpiece surfaces, local geometric depression, normal discontinuity, and torch reachability. Surface normals are estimated from local point cloud neighborhoods, and directional discontinuity is characterized by the difference between the current normal and the neighborhood mean normal. A multi-scale depth difference operator, formed from the difference between Gaussian-smoothed depth maps at two scales, enhances V-shaped groove weld regions and local depressions, with larger responses generally corresponding to groove bottom or recessed areas. A curvature response derived from local second-order geometric variation is also computed, and planar fitting residuals determine whether a candidate region lies on a valid workpiece surface.
The support relationship of base metal on both sides of the groove provides an additional physical check. Local point clouds are sampled on either side of a candidate centerline and fitted with planes. If both fitted planes contain a sufficient inlier ratio and their normal angle falls within the geometric constraint range for the intended joint type, the candidate is judged to have high groove structural plausibility. This allows the geometric branch to deliver quantifiable physical evidence to the language model: whether groove depth information is reliable, whether base metal support exists on both sides, and whether the candidate is a genuine groove region or an interference structure such as a base metal edge or a reflective band.
- Open-Vocabulary Semantic Candidates and Structured Descriptors
The semantic branch generates open candidate regions from RGB imagery, extracting semantic elements including workpieces, grooves, suspected weld seams, base metal edges, and background interference. Open-vocabulary detection, prompt-based segmentation, and visual foundation models provide strong candidate recall and cross-category generalization, allowing the system to propose potential workpiece, groove, and suspected seam regions according to prompts. Crucially, the visual foundation model does not output the final seam. It operates under a recall-first strategy that produces a candidate set, which is then filtered and interpreted downstream.
For each candidate, the system extracts a class or semantic label, segmentation confidence, welded workpiece region area, aspect ratio, boundary morphology, relative relationship to the workpiece region, and degree of match with the task prompt. Combined with the geometric branch outputs — valid depth ratio, base metal support on both groove sides, normal stability, and depth valley response — a unified structured candidate descriptor is constructed. The descriptor encodes semantic relevance, geometric consistency, valid depth proportion, workpiece normal plane, normal variation, groove bottom depth valley response, and bilateral base metal normal plane relationships.
This structured representation is the mechanism that keeps the language model grounded. Rather than reading unconstrained image regions, the reasoning module receives measurable, explainable, and recheckable evidence. The result is a candidate interpretation process that can be audited, an essential property when the downstream output drives physical robot motion. Within the embodied intelligence framing adopted by the authors, this grounding step is what separates useful industrial reasoning from unconstrained generative output.
- Large Language Model Reasoning with Closed-Loop Feedback Rechecking
The reasoning stage receives a natural-language task prompt, the structured candidate descriptors, spatial relationships among candidates, and welding process knowledge. It outputs a candidate role, a task relevance score, a weldability explanation, and a region of interest hypothesis. The welding knowledge base covers joint type, groove form, start and end direction, torch travel angle, work angle, wire extension length, and reachability constraints. The language model is not expected to produce millimetre-level seam boundaries; its responsibility is to judge, under the constraint of structured semantic and geometric evidence, whether a candidate corresponds to a groove to be welded, a base metal edge, a reflection artifact, or a groove-like gap interference, and to explain whether the candidate carries genuine welding process significance.
To stabilize reasoning outputs, the framework introduces a multi-round feedback rechecking mechanism. After a first round produces top-ranked candidates, the system checks depth confidence, bilateral base metal support, centerline continuity, start and end point rationality, and tool center point generability. If a candidate suffers from insufficient depth, missing base metal support, suspected base metal edge or reflection, centerline discontinuity, or inadequate reachability, feedback information is generated and returned to the reasoning module. This establishes a closed loop of candidate judgment, geometric and process inspection, feedback update, and candidate re-judgment.
The mechanism enables the system to actively exclude structures that are visually salient but physically invalid, including reflective bands, single-sided workpiece edges, fixture edges, and groove-like gap interference. Final candidate selection is expressed as a joint optimization over semantic score, geometric score, language model reasoning score, process score, and tool center point generability score, with weighting coefficients balancing the contributions of each evidence channel. This layered verification architecture is presented as a practical route toward dependable embodied intelligence in open fabrication settings, where a single unverified perception error can propagate directly into a defective weld.
- From Verified ROI to Weld Centerline and Tool Center Point Pose Sequences
Once a candidate passes language model reasoning and geometric and process rechecking, it becomes the final pre-weld region of interest. Refinement and skeleton extraction inside that region yield a two-dimensional centerline, which is then back-projected through the depth map and coordinate transforms into a three-dimensional centerline expressed in the robot base frame. Start point, end point, and several intermediate guide points are selected along the three-dimensional centerline, and the local groove tangent and normal together with the workpiece support plane are estimated in the neighborhood of each point.
The three-dimensional centerline is resampled at a preset step size, and local tangents are estimated from adjacent path points. With the torch pointing normal derived from the local groove point cloud and a transverse base vector formed by the cross product of the normal and tangent, a local torch coordinate frame is constructed at each path point. Accounting for wire extension length, tool center point calibration offset, and multi-pass weld offset, the tool center point pose for a given pass at a given path point is computed by displacing the path point along the transverse and normal directions and applying the calibrated offset.
The system finally outputs a package comprising the pre-weld region of interest, two-dimensional and three-dimensional weld centerlines, start and end points, the local groove point cloud, and the tool center point pose sequence. This output constitutes the executable interface between perception and robot control, and it is the point at which the perception stack of an embodied intelligence system becomes directly actionable on the shop floor.
- Experimental Platform and Evaluation Protocol
Validation was carried out on a mobile welding robot platform assembled for open manufacturing conditions. The platform integrates a magnetically adsorbed mobile chassis, a six-degree-of-freedom manipulator, a welding torch with automatic wire feeder, a binocular vision sensor, a line laser sensor, a welding power source, a robot control system, and a perception and planning workstation. The binocular sensor is mounted in an eye-in-hand configuration near the torch and synchronously acquires RGB imagery, left and right stereo imagery, and infrared imagery before arc ignition. The line laser sensor measures local seam or groove profiles and provides a reference for geometric verification. Perception results are transformed into the robot base frame after camera-to-robot hand-eye calibration and are then used for region of interest determination, three-dimensional centerline reconstruction, local groove point cloud extraction, and tool center point pose sequence generation.
Table 1. Main hardware configuration of the mobile welding robot experimental platform. Module Model or Composition Key Parameters Role Mobile chassis KGM32 series SCR-32 magnetically adsorbed mobile platform Dimensions 780 mm × 550 mm × 420 mm; mass approximately 130 kg; maximum vertical load 200 kg; four-wheel differential drive; permanent magnetic adsorption Provides mobility, adsorption, and platform support on open steel structures Manipulator Self-developed six-degree-of-freedom arm Maximum payload 10 kg; working radius 1.2 m; repeatability 0.01 mm Executes torch tool center point trajectory tracking Welding execution unit TIG torch with automatic wire feeder Torch mounted at the end of the manipulator; wire diameter 1.2 mm Performs welding end-effector motion and wire feeding Welding power source Fronius TPS 440i Operated with the TIG torch and automatic wire feeder Welding power supply unit Binocular vision sensor Orbbec Gemini 435Le Stereo baseline 95 mm; effective depth range 0.31 to 10 m; image resolution up to 1,280 × 800 Acquires RGB imagery and left and right stereo or infrared imagery Perception and planning workstation High-performance GPU workstation NVIDIA GeForce RTX 5090 GPU; AMD Ryzen 9 9950X CPU; 64 GB RAM; 4 TB SSD Large language model inference and geometric processing Evaluation spanned three critical stages of the pre-weld perception chain. For binocular depth estimation, mean absolute error, root mean square error, and absolute relative error were used to test whether the three-dimensional reconstruction provides a reliable depth foundation for groove geometric evidence extraction. For language-model-guided candidate reasoning and region of interest determination, region-of-interest-level intersection over union, Dice coefficient, centerline error, endpoint error, and candidate judgment confidence were used to test whether the fusion of semantics, geometric evidence, and welding process knowledge yields a compact, continuous, and accurately positioned pre-weld region of interest. For tool center point pose generation, pointwise three-dimensional error, per-axis component error, and valid overlap length were computed on the overlapping trajectory segment after common start point alignment, in order to evaluate the spatial consistency and trajectory stability of the conversion from perception results to robot path preparation data.
Table 2. Evaluation metrics for pre-weld perception and tool center point generation. Evaluation Target Primary Metrics Physical Meaning Relationship to the Pre-Weld Chain Binocular depth Mean absolute error, root mean square error, absolute relative error Measures three-dimensional geometric error and depth usability Affects normals, valley detection, point clouds, and groove modeling Language-model candidate reasoning Region-of-interest intersection over union, Dice, centerline error, endpoint error, reasoning confidence Measures candidate interpretation, pre-weld region of interest, and centerline stability Affects the pre-weld region of interest, three-dimensional guide points, and tool center point input Tool center point trajectory Three-dimensional error, component-wise error, overlap length Measures spatial consistency of path preparation results Connects perception outputs to the robot control interface - Depth Estimation Accuracy and Stability in Reflective Welding Scenes
Depth estimation was evaluated on close-range steel plate welding scenes. Test data were captured with the Orbbec Gemini 435Le binocular camera and comprised 1,020 stereo image frames. Because metallic reflection, edge occlusion, infrared illumination variation, and structured-light holes can introduce local reference error, the structured-light depth output of the RGB-D sensor was used as a sensor reference value rather than as strict ground truth. The evaluation therefore focused on consistency between methods, stability under varying infrared imaging conditions, and the ability of the resulting depth to support subsequent groove geometric analysis.
The proposed method achieved the lowest depth estimation error on the welding scene imagery, with mean absolute error of 0.97 mm, root mean square error of 3.62 mm, and absolute relative error of 0.002 4, outperforming IGEV-Stereo, Selective-RAFT, and Selective-Stereo. Relative to IGEV-Stereo, the three error metrics decreased by 71.47%, 52.37%, and 61.90% respectively. Relative to Selective-RAFT, the reductions were 75.13%, 45.15%, and 67.57%. Relative to Selective-Stereo, the reductions reached 78.44%, 76.03%, and 72.41%. Visualized comparisons confirm that the hybrid cost volume construction combined with gated recurrent unit iterative updating improves disparity estimation stability under metallic reflection, weak texture, and locally complex structures, thereby supplying more reliable depth information for point cloud generation, normal estimation, and candidate groove verification.
Table 3. Error comparison of learning-based stereo depth estimation methods. Method Mean Absolute Error (mm), lower is better Root Mean Square Error (mm), lower is better Absolute Relative Error, lower is better Proposed method 0.97 3.62 0.002 4 IGEV-Stereo 3.40 7.60 0.006 3 Selective-RAFT 3.90 6.60 0.007 4 Selective-Stereo 4.50 15.10 0.008 7 Absolute depth measurement was also compared. Under conditions of metallic reflection, abrupt workpiece thickness transitions, and local occlusion, the raw structured-light depth exhibited a measurable deviation with a representative value of 13.547 9 cm, whereas the proposed method produced 13.821 0 cm, which is closer to the ruler reference value shown in the scene. Combined with the statistical error results, the binocular depth module demonstrates good absolute depth consistency and scene robustness at close range, providing a dependable spatial geometric foundation for point cloud reconstruction, groove geometric verification, three-dimensional centerline recovery, and tool center point pose generation.
- Language-Model-Guided Candidate Interpretation and ROI Determination
To evaluate language-model-guided candidate interpretation and result rechecking, a locally deployed Gemma3 model served as the welding task reasoning backend. Candidate generation, geometric descriptor construction, and post-processing were held constant so that the evaluation could isolate candidate understanding, pre-weld region of interest determination, and centerline generation capability in open welding scenes.
In the assessed flat-plate V-groove butt welding scenario, the weld region was surrounded by reflective surfaces, an aluminum alloy profile frame around the test plate, and local texture interference. Two-dimensional semantic segmentation alone tended to misclassify large workpiece areas, prominent edges, or groove-like gap structures as weld regions. The semantic branch therefore first generated workpiece, groove, and suspected seam candidates, while the geometric branch extracted groove root and local structural transition information from depth maps, normal vectors, curvature responses, fitting residuals, and groove bottom depth, supplying quantifiable candidate evidence to the language model.
After coarse candidate extraction, centerline optimization, and language model rechecking, the system constrained a large semantic candidate down to a continuous, slender pre-weld region of interest consistent with the welding task, and emitted start, middle, and end points to support subsequent three-dimensional guide point extraction and tool center point pose sequence generation. Throughout this process the language model did not perform pixel-level seam segmentation. It performed role interpretation, task relevance evaluation, groove structural plausibility analysis, and region of interest hypothesis generation based on structured candidate descriptions, semantic segmentation results, three-dimensional geometric evidence, and the welding task prompt. The system then rechecked geometric consistency, centerline continuity, bilateral base metal support, start and end point rationality, and tool center point generability. For candidates with insufficient depth evidence, suspected base metal edge or reflection interference, or discontinuous centerlines, feedback information updated candidate evidence and triggered re-judgment, yielding iterative reasoning across candidate interpretation, evidence inspection, feedback correction, and result confirmation.
Table 4. Pre-weld candidate reasoning results using the locally deployed language model. Reasoning Model Deployment Test Groups Success / Failure ROI IoU (higher is better) Dice (higher is better) Centerline Error (pixels, lower is better) Endpoint Error (pixels, lower is better) Reasoning Confidence (higher is better) Gemma3 Local deployment 13 13 / 0 0.874 3 0.968 5 1.59 14.58 0.96 Across 13 open welding scenarios the system completed candidate interpretation and pre-weld region of interest rechecking in every case, indicating that structured candidate representation and geometric rechecking contribute to stable language model reasoning. Region-of-interest-level intersection over union reached 0.874 3, refined weld band Dice reached 0.968 5, and candidate judgment confidence reached 0.96, while centerline error and endpoint error measured 1.59 pixels and 14.58 pixels respectively. These figures indicate that the method produces accurate, continuous pre-weld regions of interest and centerline results, satisfying the input requirements of subsequent three-dimensional centerline back-projection and tool center point pose generation. For embodied intelligence architectures intended for field deployment, this combination of accuracy and interpretability is significant, because it allows an operator or supervisory system to trace each decision back to specific geometric and semantic evidence.
- Tool Center Point Trajectory Consistency with Reference Paths
With the pre-weld region of interest, three-dimensional centerline, and local groove model established, the next validation step examined the ability of the tool center point pose generation module to convert perception results into robot path preparation data. In the robot base frame, the trajectory generated from pre-weld perception was compared with a line laser reference trajectory. Because the start and end positions and sampling ranges of the two trajectories differ, a common start point alignment was applied and trajectory consistency analysis was restricted to the overlapping path segment.
After alignment, the generated tool center point trajectory and the reference trajectory showed good spatial consistency, with both extending along the same physical weld direction within the overlap segment, indicating that the method effectively recovers weld spatial geometry and generates a continuous robot path. Pointwise three-dimensional error and per-axis component error distributions across the overlapping segment were stable overall, with most errors concentrated within 2 mm. The scaled three-dimensional error mean was 0.93 mm and the 95th percentile was 1.68 mm, with only minor local fluctuations, demonstrating good local continuity and spatial stability of the generated trajectory.
Table 5. Consistency evaluation of generated tool center point trajectories. Evaluation Item Result Explanation Line laser travel trajectory 1,011 points, 88.92 s Reference motion path in the robot base frame Tool center point generated trajectory 958 points, 76.95 s Generated from pre-weld perception and local groove geometry Common-start overlapping segment length 341.61 mm Non-overlapping start and end portions removed for fair comparison Mean / median / maximum three-dimensional error 1.100 mm / 1.143 mm / 1.994 mm Reflects tool center point trajectory consistency along the same physical path segment Scaled error mean / 95th percentile 0.93 mm / 1.68 mm Corresponds to overlap segment error trend and high-percentile error X / Y / Z mean absolute error 0.124 mm / 0.933 mm / 0.608 mm Errors originate mainly in the Y and Z directions The quantitative agreement between the generated trajectory and the line laser reference confirms effective coupling between binocular geometric reconstruction, language model candidate reasoning, and tool center point pose generation. The method converts pre-weld perception results into robot-executable path preparation data with millimetre-level accuracy, meeting the pre-weld path generation requirements of mobile welding robots operating in open environments. Because the trajectory is derived from a verified region of interest rather than from raw edge detection, the resulting embodied intelligence pipeline retains traceability from the robot command back to the originating image observation and geometric evidence.
- Principal Innovations
The work advances four distinct contributions to embodied intelligence for robotic welding.
- An integrated pre-weld perception framework is proposed that maps images directly to tool center point pose sequences, unifying binocular and RGB observation, semantic candidates, three-dimensional geometric evidence, welding knowledge reasoning, and robot-executable pose output within a single coherent modeling pipeline.
- A binocular dense geometric reconstruction method based on hybrid cost volumes and gated recurrent unit iterative updating is developed, delivering three-dimensional evidence for groove verification including depth maps, point clouds, normal discontinuities, curvature responses, depth valleys, and neighborhood planar constraints.
- A semantic–geometric structured candidate representation method is established, jointly encoding open-vocabulary visual candidates with valid depth ratio, groove bottom depth estimation, groove neighborhood planar constraints, and bilateral base metal support relationships, thereby improving false-positive suppression in complex open welding scenarios.
- A language-model-driven candidate interpretation and feedback rechecking mechanism is constructed, performing candidate self-checking, evidence updating, region of interest confirmation, and tool center point pose sequence generation under the guidance of welding knowledge and process constraints.
- Conclusions and Outlook for Embodied Welding Intelligence
The reported results support three conclusions. First, the language-model-enhanced binocular multimodal fusion method for mobile welding robots integrates RGB semantic information, binocular three-dimensional geometric information, and welding process knowledge into a pre-weld perception flow that spans candidate generation, structured reasoning, geometric rechecking, and tool center point pose generation, achieving continuous conversion from pre-weld region of interest to robot path preparation data in open scenarios.
Second, the geometric evidence representation established for groove recognition and tool center point pose generation, comprising normal variation, groove depth, curvature response, local planar support, and bilateral base metal support descriptors, effectively characterizes local groove spatial structure and suppresses interference from workpiece reflection, shadow, base metal edges, and groove-like gap structures, providing a geometric basis for weld centerline reconstruction and torch orientation construction.
Third, on 1,020 frames of depth evaluation data the method achieved mean absolute error of 0.97 mm and root mean square error of 3.62 mm. Across 13 open welding scenarios, region-of-interest-level intersection over union, refined weld band Dice, centerline error, and endpoint error were 0.874 3, 0.968 5, 1.59 pixels, and 14.58 pixels respectively. Generated tool center point trajectories exhibited an average three-dimensional deviation of 1.100 mm on the common-start overlapping segment. These results validate the effectiveness of the method in pre-weld region of interest determination, three-dimensional centerline recovery, and robot-executable welding path planning.
For the broader field of embodied intelligence in manufacturing, the significance of the work lies in its explicit division of labor. Foundation vision models provide recall-oriented candidate generation, geometric reconstruction provides measurable physical evidence, a language model provides task-level reasoning and explanation, and process constraints provide the final acceptance criteria. No single component is trusted to produce the robot command on its own. As mobile welding robots continue their migration from fixed workstations into unstructured field environments, such layered and auditable architectures are likely to become the standard route by which embodied intelligence systems earn the confidence required for deployment on critical load-bearing structures in energy, marine, rail, and aerospace fabrication.
