Production + Research Project
Autonomous Lawn-Mower Robot Perception
A safety-critical LiDAR and multimodal perception stack for autonomous loading/unloading, slope traversal, grass-obstacle detection, and embedded deployment.
Overview
What is this project about?
A consumer mowing robot operates unsupervised around people and pets, on a compute budget of a microcontroller. Every stage of its mission — unloading itself, crossing to the lawn, mowing, returning — has its own way of going wrong, and the whole thing has to be safe by default.
Four modules on one shared sensor set. Ramp detection so the robot can load and unload itself; 3D grass-obstacle detection, first geometric and then camera–LiDAR fused; a 2D BEV safety detector small enough to run on the on-board MCU; and a dual-attention LiDAR–vision fusion study for the next generation.
The safety detector runs at roughly 110 fps on an STM32H7, so the safety net is always on rather than budget-dependent — and obstacle detection moved from geometry-only to fused perception without changing the sensor set.
My role. Perception algorithm developer across all four modules; the dual-attention fusion work was a research study that informed the next version rather than shipping.
System Architecture
Four perception modules, one safety stack
From a transport vehicle to the lawn and back, the robot leans on a layered perception stack. Switch between the four modules below to see what each one does, the tools behind it, the algorithm pipeline, and the on-vehicle result.
View as tablesame content, no interaction needed
| Stage | Input | Logic | Output | Why / role |
|---|---|---|---|---|
| Solid-State LiDAR + Camera Input | Point clouds + RGB frames | Calibrate and time-sync | Aligned 2D / 3D streams | One sensing base for all stages |
| Ramp Detection Stage A | LiDAR point cloud | Fit ramp plane + four edges | Drive-on geometry | Robot boards the transporter |
| Embedded BEV Safety Stage C | Projected BEV grid | MCU-side 2D safety detector | Real-time hazard flags | Always-on emergency stop |
| Grass Obstacle Detection Stage B | Point cloud + image | Geometry first, then 2D–3D fusion | 3D obstacles in grass | Avoids what mowers miss |
| Dual-Attention Fusion Stage D | LiDAR + camera features | Dual cross / self attention | Stronger fused detector | Research feeds production |
| Safe Autonomous Mowing Outcome | All four stage outputs | Arbitrate perception for control | Safe motion commands | Closes the mission loop |
Module A · PCL + Eigen + OpenCV
Up / down ramp detection
Detect a drive-on ramp and its four boundary lines from a solid-state LiDAR point cloud, so the robot can autonomously and stably load and unload itself onto a transport vehicle.
Raw point cloud
- Solid-state LiDAR stream, vehicle-front field of view
Crop, downsample, denoise
- ROI crop: x∈[0,20] · y∈[−5,5] · z∈[−2,2] m
- VoxelGrid downsample, leaf = 0.05 m
- StatisticalOutlierRemoval denoising
Normals + candidate tilted planes
- KdTree radius search r = 0.3 m → per-point normals
- Tilt constraint: normal–Z angle θ∈[5°,35°]
- RegionGrowing (normal diff <5°, point–plane <0.05 m) → N candidate clusters
RANSAC-like plane refinement
- ×100 iters: weighted 3-point sampling → plane; validity θ∈[5°,20°]; count inliers (<0.05 m); keep max set
- SVD least-squares on inliers → precise plane {n, d}
Multi-rule scoring → best ramp
- Weighted score Score = Σ wᵢ·sᵢ → highest candidate is the target ramp (Score < 0.4 → “not detected”)
| Criterion | Weight | Rule |
|---|---|---|
| Slope plausibility | 0.30 | θ∈[5°,20°] scores full |
| Projected area | 0.20 | larger is better (cap 15 ㎡) |
| Position / heading | 0.20 | front, 2–15 m |
| Planarity (RMSE) | 0.15 | smaller residual is better |
| Shape compactness | 0.10 | reasonable aspect ratio |
| Ground connection | 0.05 | low end joins the ground |
Local frame + boundary points
- Frame: n = Z-axis, downhill = X-axis; project inliers → 2D (u, v)
- Longitudinal 20 strips → 2 long edges; lateral 10 strips → 2 short edges → 4 boundary sets
Outlier removal
- 1D spacing-jump: gap > 3× mean → drop isolated points
- Sliding-window median (win = 5): deviation > 0.1 m → drop → 4 clean edge sets
Four boundary-line fitting
- RANSAC (30 iters) 2-point line + inliers
- TLS total least squares via SVD → precise line; endpoints by projection → 4 directed segments
Post-processing + RampInfo
- Pairwise intersection → 4 corners; validate (opposite edges ∥ <5°, width [1.5,5] m, length [2,20] m); EMA smoothing α = 0.3
- Output RampInfo { slope_angle, width, length, corners[4], boundary_lines[4], plane, confidence }
Module B · two versions
Grass obstacle detection
Detect 3D obstacles on grass — first with a pure point-cloud geometric pipeline (V1), then upgraded with camera–LiDAR semantic fusion for class-aware, more robust detection (V2).
Geometry-only pipeline — PCL
Raw point cloud
- Solid-state LiDAR
Filter chain
- PassThrough crop → VoxelGrid downsample → StatisticalOutlierRemoval
Ground segmentation
- PMF morphological filter → RANSAC plane (normal∠Z < 15°); ground points discarded
Euclidean clustering
- KD-Tree neighbor search r = 0.4 m · min 20 · max 5000 pts
PCA bounding box + rules
- PCA → OBB; constraints h∈[0.1,2] · w∈[0.1,3] · l∈[0.1,5] m · h/w < 5
3D obstacles
- Position / size / distance = √(cx² + cy²)
Camera–LiDAR semantic fusion — TensorRT
Point cloud
- Solid-state LiDAR
Image
- RGB camera
2D bounding boxes
- Per-frame 2D BBox list
Semantic mask
- Ground / obstacle pixel mask
Extrinsic point ↔ pixel
- Camera–LiDAR extrinsics inject semantics into each 3D point
Mask-based segmentation
- mask = ground → discard; obstacle → keep (replaces RANSAC)
2D-guided clustering
- Cluster points inside each BBox; associate 2D detection ↔ 3D cluster
Geometric filter
- PCA → OBB with the same V1 rule constraints
Triple-score fusion
- 0.4 × geometry + 0.4 × 2D IoU + 0.2 × semantic consistency
Class-aware 3D obstacles
- 3D obstacles with category label and confidence
Module C · STM32H7 MCU
Embedded 2D BEV safety detection
A lightweight 2D BEV obstacle detector deployed on an STM32H7 microcontroller — static memory plus integer optimization deliver 110 fps real-time detection that passes functional-safety testing.
Raw point cloud
- Solid-state LiDAR stream
Point-cloud preprocessing
- ROI crop (5 m × 5 m); ground removal z < −0.1 m
- Invalid filter (range = 0 / NaN); height clamp 0.05–2.5 m
BEV projection
- Cell index idx = (x − origin) / reso
- Per-cell point count; height diff max_z − min_z
Obstacle decision
- Density threshold count > 3; height diff > 0.1 m
- Connected components (8-neighbor merge)
BEV obstacle list
- Danger level + nearest-obstacle BEV coordinate
Module D · pre-research
Dual-attention LiDAR–vision fusion
A pre-research study (toward a company paper KPI) on dual-attention fusion that correlates scene geometry and texture features for stronger LiDAR–vision 3D detection.
LiDAR features
- 3D scene-geometry encoding
Image features
- 2D appearance / texture encoding
Fast correspondence
- Pre-compute point ↔ pixel index so fusion stays cheap at runtime
Explicit correspondence
- Geometric point ↔ pixel matching
Query-style attention
- Learned cross-modal attention weights
Geometry × texture interaction
- Two attention streams couple structure and appearance into a shared representation
Robust 3D detection
- Stronger on small objects, sparse LiDAR, and degraded images
