Research Project

Controllable Surround-View Driving Generation

A controllable multi-view world model for driving: 3D layout + map + multi-granularity control signals injected into a diffusion process to generate geometrically-consistent 4V / 7V / 11V images and video — for data augmentation and open-loop simulation.

Timeline: 2023.05–2024
Context: PhiGent Robotics
Role: Generative Driving Algorithm Engineer
Stage: Pre-research

Overview

What is this project about?

Problem

Training data for surround perception is fixed once it is recorded — you cannot re-shoot the same intersection at night, in rain, with a cyclist added. Generated data can, but only if the generator is controllable and the views agree with each other geometrically.

What I built

A conditional surround generator: 3D boxes and map elements are projected per view into spatial conditions, while text, reference frames, lane structure and camera calibration are encoded into condition tokens and injected into a UNet diffusion backbone. The model evolved from OpenSora 1.0 plus SD 3.5 into an in-house model fused with MagicDrive.

Result

Conditioning every camera on the same projected 3D structure is what keeps 4, 7 or 11 views mutually consistent, so the output is usable both as perception training data and as open-loop simulation rather than as decorative video.

My role. Built the generator: the condition design, the encoders and injection path, and the training and evolution of the in-house model.

research world-model generative e2e

4V / 7V / 11V surround 3D layout + HD map Text · refs · lanes · rig tokens UNet diffusion backbone OpenSora → MagicDrive fusion

4V·7V·11VCamera configurations supported

6+Control signals per generation

2 usesAugmentation · open-loop simulation

V2RGB, depth, ego-pose control

Logic map

Noise to controllable surround worlds

Everything on the left is a condition; the diffusion backbone in the middle is the only thing that generates. Geometry stays aligned across 4, 7 or 11 views because every view is conditioned on the same projected 3D structure.

Stages Start latent Structural conditions Token conditions & V2 controls Diffusion backbone Output & downstream use Model lineage

Links Data flow (inference) Conditioning / structural

View as tablesame content, no interaction needed

Noise to controllable surround worlds — every node with its input, logic and output.
Stage	Input	Logic	Output	Why / role
Gaussian Noise Start	Pure random latents	Seed every camera view	Noisy surround latents	Enables diffusion generation
3D Boxes + Map Layout	Boxes and HD map	Project into camera planes	View-space scene structure	Anchors object-road geometry
Layout / Map Encoders Encode	Projected boxes and map	E_Layout + E_Map compress	Spatial condition features	Keeps cameras aligned
Condition Tokens Control	Text, refs, lanes, calibration	Encode multi-granularity controls	Condition token set	Steers appearance and rig
UNet Diffusion Backbone	Noise + all conditions	Inject conditions each step	Refined surround latents	Unifies structure and style
Iterative Guidance Denoise	Conditioned UNet updates	Repeated denoising rounds	Aligned multi-view latents	Holds cross-camera consistency
Surround RGB Decode	Geometry-aligned latents	Decoder restores pixels	Multi-view images / video	Generates controllable worlds
Data Augmentation Use	Generated surround scenes	Vary style, hold layout	Long-tail training data	Cuts collection burden
Open-Loop Simulation Use	Generated videos	Replay requested conditions	Synthetic driving clips	Tests controlled scenarios
OpenSora → MagicDrive Evolve	OpenSora 1.0 + SD 3.5	Fuse with MagicDrive ideas	Specialized in-house model	Improves driving fidelity
Depth + Pose Control V2	In-house model line	Add depth and ego control	RGB, depth, pose rollouts	Boosts trajectory controllability

Conditioned diffusion pipeline

7V controllable image generation pipeline

3D boxes and maps become spatial conditions; text, reference frames, lanes, and camera calibration become tokens. The UNet denoises from pure noise into aligned multi-view latents, then decodes pixels.

Scene replacement for augmentation

7V scene/style replacement: geometry held, appearance varied.

11V replacement uses the same map-conditioned consistency logic.

120 degree FOV single-view scene replacement

1V 120° FOV variant: controlled single-view regeneration.

Long-tail cone and lane-line synthesis without field collection.

Surround video generation

4V fisheye daylight rollout, generated as a temporally coherent clip.

Controllable 7V surround video after in-house driving pretraining.

V2: depth and ego control

v2 adds pixel-depth output as a second generated modality.

Ego-trajectory controllability improved for pose-guided rollouts.

Unbalanced real dataGenerate rare scenes on demand

Cross-camera driftProject boxes and maps per view

Style-only controlFuse layout, map, text, rig tokens

RGB-only worldsExtend to depth and ego-pose control

OmniNWM. After my departure, former colleagues led the follow-up OmniNWM direction: github.com/Ma-Zhuang/OmniNWM.

My role. Built the controllable driving generation pipeline at a high level: structured conditions, diffusion integration, and sanitized visualization for augmentation and open-loop simulation.

Confidentiality note. PhiGent Robotics research. Only sanitized generation results and high-level pipeline descriptions are shown; dataset details and internal evaluation metrics are omitted. The OmniNWM follow-up was led by former colleagues after my departure and is credited below.