Research + Deployment Project
3D Scene Flow: Auto-Labeling & Production Deployment
An unsupervised point- and occupancy-level 3D scene-flow auto-labeling system, two deployable flow networks, an ultra-light production head, and full ONNX → TensorRT / Horizon J6E deployment.
Overview
What is this project about?
Per-point 3D motion cannot be annotated by hand at fleet scale, and the networks that estimate it are far too heavy for an automotive ECU. Both halves have to be solved, or neither is useful.
An unsupervised label engine that derives dense scene flow from consecutive LiDAR sweeps using geometric, cycle and smoothness constraints — so labels scale with mileage rather than with annotation budget — and an ultra-light head distilled from those labels, exported once to ONNX and compiled for two different silicon tiers.
The auto-labels raised the accuracy of established flow estimators when used as their training signal, which is the evidence that the labels are right; the distilled head runs on-vehicle on both a high-compute and a cost-optimised target from a single exported graph.
My role. Designed and built the auto-labeling system, the distilled production head, and the ONNX → TensorRT / Horizon deployment path.
Overview
Give every point — and every occupancy cell — a motion vector, with no human labels
3D scene flow is the dense per-point 3D motion field between two LiDAR sweeps — the geometric backbone for dynamic-object reasoning, velocity estimation and occupancy-flow prediction. Hand-labeling it is effectively impossible. This project built an unsupervised auto-labeling system that assigns a motion vector to every point and every occupancy (occ) cell, used those labels to sharpen existing flow estimators, distilled the result into an ultra-light production head, and shipped the whole stack through ONNX → TensorRT (NVIDIA Orin) and the Horizon J6E toolchain.
View as tablesame content, no interaction needed
| Stage | Input | Logic | Output | Why / role |
|---|---|---|---|---|
| Consecutive LiDAR Sweeps Input | Raw LiDAR Pt, Pt+1 | Compensate ego-motion | Aligned sweep pairs | Raw signal, zero labels |
| Unsupervised Auto-Labeler Core | Ego-motion-compensated sweep pairs | Solve for flow under geometric, cycle and smoothness constraints | Dense 3D motion labels | Supervision now scales with mileage instead of with annotation budget — which is the only way per-point motion labels exist at fleet scale at all. My role. Designed and built the label engine. |
| Lifts Existing Estimators Evidence train-only | Auto-labels used to retrain published baselines | Retrain established methods on the generated labels | Consistent accuracy gains | An unsupervised labeler cannot be checked against ground truth it does not have. Improving someone else's published model is the strongest available proof that the labels are real. |
| Ultra-Light Head Productize | Auto-labels as supervision | Distill into embedded-size net | Tiny flow predictor | On-vehicle compute budget |
| Single ONNX Graph Export | Trained / quantized head | Export portable .onnx | Graph to both runtimes | Train ↔ deploy stay aligned |
| TensorRT · Orin Target A | ONNX graph | Graph + precision optimize | Orin runtime engine | Fast NVIDIA inference |
| Horizon J6E Target B | ONNX graph | Convert + quantize via SDK | J6E runtime engine | Cheap embedded inference |
| On-Vehicle Occupancy Flow Output | Optimized runtime engine | Preprocess → infer → parse | Dense scene-flow field | Production motion estimation |
An unsupervised 3D scene-flow & occ-flow auto-labeler
The heart of the project: a self-supervised system that takes raw LiDAR sweeps and produces a dense 3D motion label for every point and every occupancy cell — no manual annotation in the loop. These auto-labels become the training signal for every downstream flow model.
Consecutive LiDAR sweeps
- Paired point clouds Pt, Pt+1 + ego-pose; no flow ground truth required
Dense motion field
- Predict a 3D vector per point and per occ cell; ego-motion compensated so only true object motion remains
Consistency objectives
- Nearest-neighbour / cycle / smoothness constraints replace human labels with geometry
Point- & occ-flow labels
- A reusable label bank that trains and stress-tests every downstream estimator
Auto-labels lift existing flow estimators
The acid test for a labeling system is whether its labels make other models better. Feeding our auto-labels into established 3D scene-flow estimators improved their prediction accuracy substantially — direct evidence that the generated supervision is both correct and useful.
Baseline estimators
Original supervision
Trained on their native, limited flow signals
Weaker on fast, distant and sparse objects
+ our auto-labels
Substantially sharper
Same architectures, richer dense supervision
Consistent accuracy gains across methods
An ultra-light 3D scene-flow head — two design routes
For mass production the flow predictor must be tiny and embedded-friendly. We explored two routes for the production head and compared them head-to-head: a graph-convolution + mathematical-constraint design, and a pure point-cloud deep-learning design fit directly to supervision.
Route A · Graph-conv + math
Geometry-guided
Graph convolution over local neighbourhoods
Explicit rigidity / smoothness constraints
Robust & interpretable on structured motion
Route B · Pure deep-learning
Data-driven
Point-cloud network regresses flow end-to-end
Fit directly to the auto-generated labels
Simplest graph to export and quantize
ONNX → TensorRT / Horizon J6E, with occ-flow inference
The trained head was exported to ONNX, optimized with TensorRT for NVIDIA Orin, and converted with the Horizon SDK toolchain for J6E. Against existing production options our solution held up well, and the same export produces the live occupancy-flow inference below.
Trained model → ONNX
- Floating-point or quantized graph exported to a portable .onnx
TensorRT on Orin
- Graph & precision optimization for the Orin runtime
SDK on J6E
- Convert & quantize through the Horizon J6E toolchain
On-vehicle inference loop
- Point-cloud preprocess → cache init → normalize → inference → output parsing → perf stats
Visualization
Auto-Flow, running
The end-to-end result — dense 3D scene flow auto-labeled and predicted on real driving sequences. Plays automatically and loops.