Research Platform
Vector Traffic Generation & Sensor-Level Closed-Loop Simulation
Two halves of a controllable driving simulator: a structure-aware temporal vector world model that generates traffic as latents, and a sensor-level pipeline that reconstructs, populates and re-renders photorealistic surround video.
Overview
What is this project about?
A simulator has to be controllable and photorealistic at the same time. Vector simulators let you author behaviour but render nothing a perception model believes; video generators look real but cannot be steered scenario by scenario.
Two levels that meet in the middle. A structure-aware temporal autoencoder (STAR-AE) compresses variable numbers of agents and lanes into a fixed latent, and a conditional latent diffusion model (STRIDENet) rolls that latent forward into history-consistent future traffic. Beneath it, a sensor-level loop fuses Gaussian-Splatting reconstruction of the real background with the generated traffic through a mask-guided DiT video editor built on MagicDrive-V2.
Behaviour is authored as vectors and rendered as photoreal 7-camera surround video in the same loop, so a scenario can be changed at the level a person thinks about it — and the editor only regenerates the masked foreground instead of the whole frame.
My role. I designed and trained the vector level — the STAR-AE latent structure and the STRIDENet conditioning — and integrated it with the sensor-level pipeline. Reconstruction and the video editor are built on shared platform components.
This is one track inside the Generative Autonomous-Driving Simulation Platform: that project supplies the surround world model and the distilled sampler, this one supplies the traffic behaviour and the closed-loop re-rendering.
View as tablesame content, no interaction needed
| Stage | Input | Logic | Output | Why / role |
|---|---|---|---|---|
| Driving Scene Input | Recorded drives with map and sensor data | Split the same scene into a vector level and a sensor level | One scene into two parallel representations | Splitting once at the source is what allows behaviour and appearance to be controlled independently and recombined later. |
| STAR-AE Encode | A variable number of agents and lane elements over time | Assign elements to slots, then factorise attention over space and time | Per-slot latents | Diffusion needs a fixed-size, continuous space. Traffic is neither — this is the component that makes the rest of the vector level possible. Trade-off. Slots impose a ceiling on scene density; too few drops agents, too many wastes capacity on empty slots. My role. Designed and trained the latent structure. |
| Gaussian Splatting Reconstruct | Real frames plus auto-labels | Optimise Gaussians, then render at an arbitrary pose | Background renders | Reconstruction supplies the one thing generation is worst at: a background that is verifiably the real world, from a camera pose that was never actually driven. |
| Structured Latent Latent | Encoder posteriors | Reparameterise to a continuous latent | Continuous latent | A smooth latent is what makes interpolation and sampling meaningful — without it, diffusion over discrete traffic has nothing to move through. |
| Photoreal Background Background | Gaussian-Splatting renders | Render the static world at the simulated pose | Conditioning frames | Perception models are unusually sensitive to background realism; a synthetic background is the fastest way to make a simulator useless for closed-loop evaluation. |
| STRIDENet Generate | A noised latent plus the observed history | Denoise, conditioning on history through adaptive layer norm | A clean future latent | Conditioning on history rather than on a single frame is what keeps generated traffic continuous with what already happened, instead of teleporting. My role. Designed the conditioning scheme and trained the generator. |
| Future Traffic Output | Decoded latent | Decode back to agents and lanes | The what-happens layer | Traffic stays editable as vectors right up to the moment it is rendered, which is the only level at which a scenario can actually be authored. |
| Mask-guided DiT Editor Compose | Generated traffic and the reconstructed background | Freeze the background, synthesise the foreground, resolve the seam | Edited latents | Regenerating a whole frame throws away a background that was already correct; masking spends the model's capacity only where something actually changed. Trade-off. Masked editing depends on the mask being right — a bad boundary shows up as a visible seam rather than a soft error. |
| 7V Surround Video Output | The composed latent | Decode to surround frames | Photorealistic surround video | This is the closing of the loop: the simulator's output is the same thing the perception stack consumes in the car. |
Temporal vector AE in motion
The VAE encodes sparse, variable scenes into a fixed latent and reconstructs them — agents and lanes stay temporally coherent.
Architecture, both halves
Sensor-level closed loop
Mask-guided DiT — edit, don't regenerate
Four semantic masks partition every frame so the model only computes what must change.
| Mask | Region | Action |
|---|---|---|
| M_keep | Known background | Frozen — skip all compute |
| M_ctx | Reference background | Cached — provide K/V only |
| M_edge | Fg/bg boundary | Active — repair the seam |
| M_gen | Foreground | Active — generate by condition |
