POC Project

One-Stage End-to-End Driving — 8V Pure Vision

From pixels to planning in a single network: 8-camera surround vision → unified BEV → three perception heads → self-supervised future-BEV prediction → a Diffusion-Flow AI planner, all trained end-to-end with no hand-designed 3D-label interface.

Timeline
2025.12–2026.04
Context
Bosch (XC-CN)
Role
World Models Algorithm Engineer
Stage
POC

Overview

What is this project about?

Problem

A modular driving stack hands 3D boxes from perception to planning. That interface quantises away everything the planner was not told to expect, and it blocks gradients — so nothing downstream can tell perception what it actually needed.

What I built

One network from pixels to trajectory. Eight surround cameras are lifted into a single shared BEV feature; 3D detection, online HD mapping and 3D occupancy all read that same feature; a self-supervised next-frame BEV forecast, scored by a frozen generative prior, makes the representation carry dynamics; and everything is tokenised into a Diffusion-Flow planner that denoises the ego trajectory and its neighbours' futures together.

Result

Perception, prediction and planning are optimised jointly: the planner's loss reaches back through the tokeniser and the BEV into the backbone, so perception is trained for what planning needs rather than for a detection benchmark.

My role. I integrated the static-perception, dynamic-perception and AI-planner components into the single one-model POC, and ran the daily train / eval / visualisation loop. Wording is kept high-level for enterprise confidentiality.

research e2e world-model perception
8V surround · pure vision Unified BEV feature 3 perception heads Self-supervised future BEV Diffusion-Flow planner
8VSurround cameras · multi-frame
1 BEVOne shared feature drives every head
3 heads3D detection · HD map · occupancy
One stagePerception + prediction + planning, joint

Logic map

Pixels to a planned trajectory — one differentiable pass

The centre column is what runs in the car. The right lane exists only while training. The pink arrow is the point of the whole project: the planner's loss reaches all the way back into the backbone.

Stages Sensing & perception Representation & forecast Planner interface Planning & output Generative prior
Path Runs at inference Train-time only Frozen / pretrained
Links Data flow (inference) Conditioning / structural Train-time only Gradient (end-to-end)
View as tablesame content, no interaction needed
StageInputLogicOutputWhy / role
8V Surround Cameras Input Eight surround cameras, several consecutive frames Sample a synchronised multi-view, multi-frame clip Raw multi-view image streams Cameras are the only sensor already on every trim level, so a vision-only stack is the one that can actually ship at volume.
Vision Backbone Backbone Per-camera image streams Cross-view attention across cameras, temporal fusion across frames Geometry-aligned, motion-aware view features Fusing across views and time before the lift means overlap regions and motion are resolved once, not re-derived by every downstream head.
Unified BEV Feature Lift Aligned multi-view features Lift perspective features into one bird's-eye feature volume One BEV feature, shared by every downstream branch One shared hub instead of a per-task neck: the heads cannot drift apart, and adding a head costs a decoder rather than a second pyramid. Trade-off. A shared trunk means head losses compete for capacity — the joint loss weighting becomes the thing you actually tune. My role. Owned the integration that made a single BEV serve static perception, dynamic perception and the planner in one model.
3D Detection Perception Shared BEV feature Regress 3D boxes and per-object motion Dynamic agents with velocity Reading boxes off the shared BEV keeps detection in the same frame as the map and the planner, so no re-projection step can lose accuracy.
Online HD Map Perception Shared BEV feature Decode lane elements and their topology Vectorised local map Building the map online removes the dependency on a pre-surveyed HD map, which is the single biggest cost blocker for wide deployment.
3D Occupancy Perception Shared BEV feature Predict a dense, class-labelled occupancy volume Occupied and free space Boxes cannot describe a fallen ladder or an open tailgate; a dense volume catches the long-tail geometry that boxes silently drop. Trade-off. Dense volumes are the most expensive head in the stack — resolution is traded directly against latency.
Future BEV t+1 Auxiliary train-only Current BEV feature Forecast the next-frame BEV; the real next frame is the label Predicted next-frame BEV (training only) Forcing the BEV to predict its own future makes it encode scene dynamics, not just a snapshot — and it costs no extra annotation. Trade-off. It is free supervision but not free compute: it is dropped at inference, so the car never pays for it.
Generative Critic Prior frozen Predicted next-frame BEV Score realism against a pretrained generative prior Realism signal for the forecast loss A pure regression loss lets forecasts blur into the mean; a frozen generative prior keeps them on the manifold of scenes that actually occur. Trade-off. Frozen means cheap and stable, but the prior can only judge what its pretraining covered.
Scene Tokens Interface Detection, map, occupancy and the BEV feature Encode heterogeneous outputs into one token sequence Conditioning tokens Tokens replace the hand-designed 3D interface between perception and planning — the piece that normally quantises away information and blocks gradients.
Diffusion-Flow Planner Plan Scene tokens Denoise a trajectory out of noise, conditioned on the tokens A distribution over plans, not a single guess A generative planner represents genuinely ambiguous situations (yield or go) as multiple modes, where a regression planner averages them into an unsafe middle. Trade-off. Multi-step denoising costs more than a single forward pass — the step count is the knob between plan diversity and latency. My role. Integrated the planner with the perception stack and ran the daily train / eval / visualisation loop for the joint model.
Ego + Agent Futures Output Denoised plan Emit the ego trajectory together with neighbouring-agent states Scene-consistent future trajectories Planning the ego and its neighbours in one shot keeps the plan internally consistent — the ego cannot assume a future the other agents contradict.

The architecture, end to end

One-stage 8V end-to-end driving architecture
One shared BEV feeds three perception heads and a future-BEV forecast; everything is tokenized into a Diffusion-Flow planner that denoises straight into the ego trajectory — no hand-designed 3D hand-off in the loop.

On real test drives

Surround perception (dynamic objects, online map, occupancy) and the generated ego trajectory — produced together by a single end-to-end network.

Why one stage

Lossy 3D interfacesOne shared BEV, no manual hand-off
Errors compound across modulesJoint end-to-end optimisation
Boxes miss long-tail shapesDense 3D occupancy head
A snapshot can't plan aheadPredict next-frame BEV
Forecasts drift off-manifoldFrozen generative critic
Hand-tuned cost functionsDiffusion-Flow generative planner
My role. Integrated the static-perception, dynamic-perception, and AI-planner components into the single one-model POC, and ran the daily train / eval / visualization loop. Wording is high-level to protect enterprise confidentiality.
Confidentiality note. Bosch (XC-CN) POC. The architecture is presented at a conceptual, portfolio level; customer data, calibration, training corpora, and quantitative results are intentionally omitted or sanitized.