POC Project
End-to-End Driving: 11V + LiDAR Fusion
A sparse-centric end-to-end autonomous-driving stack fusing 11 cameras (7 pinhole + 4 fisheye) with LiDAR. I owned the BEV-fusion CUDA operator and the AI-planner training.
Overview
What is this project about?
Fusing 11 cameras and a LiDAR through a dense BEV grid means paying for every cell of a mostly-empty scene, on every frame. A sparse-centric design avoids that, but only if the aggregation step is fast enough to be worth it.
A fused CUDA operator that performs deformable aggregation across all 11 camera views and the LiDAR voxels in a single kernel instead of a chain of gathers, and an AI planner trained to emit motion prediction and planning in parallel from one shared query decoder.
The stack stays sparse from sensor to trajectory — nothing is ever rasterised into a dense grid — and planning shares the decoder with prediction rather than consuming its output through a second interface.
My role. I owned the fused BEV-fusion CUDA operator and the AI-planner training. The surrounding sparse perception stack was the collaboration's shared work.
View as tablesame content, no interaction needed
| Stage | Input | Logic | Output | Why / role |
|---|---|---|---|---|
| 11V + LiDAR Input | Surround images + LiDAR sweep | Multi-modal sensing | Raw camera and LiDAR data | Covers near + surround geometry |
| Sparse Encoders Encode | Images and LiDAR sweep | ViT/ResNet-FPN + voxel/pillar | Multi-scale sparse features | Cuts dense BEV overhead |
| 3D Queries Query | Instance anchors | Center + face keypoints | Projected sampling points | SparseDrive-style carrier |
| Fused CUDA Op Owned | 11V × 4-scale projections | Sample × weight × reduce | Aligned instance features | Unifies fusion in one kernel |
| Sparse BEV Fusion Fuse | Image + LiDAR features | Project, sample, aggregate | Fused sparse BEV | Stabilizes downstream tasks |
| Sparse Perception Perceive | Fused sparse BEV | Shared transformer decoding | Objects, tracks, map | Scene state for planning |
| Query Decoder Decode | Temporal, map, vision, LiDAR | L-layer query refinement | Shared planning queries | Connects motion and ego |
| AI Planner Owned | Shared decoder queries | Motion + planning together | Agent and ego futures | Bidirectional, game-aware planning |
| Turn-around Trajectory Output | Perception + planner outputs | Select executable future path | Final ego trajectory | Sensors to planning, end-to-end |
System logic
Fusion operator
