Research Platform

Generative Autonomous-Driving Simulation Platform

A Cosmos-Transfer2.5 world model turned into a production simulation platform: sensor-level 7V generation, real-map scenarios, a WorldSim↔WorldModel bridge, 4-step distillation of 7V surround video, and one denoiser that serves three jobs.

Timeline: 2025.03–Present
Context: Bosch (XC-CN)
Role: World Models Algorithm Engineer
Stage: Ongoing

Overview

What is this project about?

Problem

Real driving logs cannot cover the rare interactions that matter most, and collecting them is slow, expensive and unsafe. Generated video could fill the gap — but a 35-step surround-video diffusion model is far too slow to sit inside a closed simulation loop.

What I built

A 7-camera surround world model on Cosmos-Transfer2.5, driven by real-map layouts (Ingolstadt OSM → BEV layout → 7V video) and connected to our closed-loop simulator over a semantic bridge, so a scenario can be authored, generated and edited rather than waited for.

Result

Four-step distillation (bridge consistency plus distribution matching) cut the sampler from 35 steps to 4 for up to ~13.9× faster rollouts, and a single denoiser now serves layout generation, Gaussian-Splatting repair and harmonization instead of three separately trained models.

My role. I own the distillation track end to end — schedule design, the alternating student/critic training loop, and the progressive 35→4→2→1 cascade — and the OneModel unification that collapsed three task-specific models into one. The platform itself is a team effort; scenario authoring and simulator integration are shared work.

This platform is the parent effort. Vector Traffic Generation & Sensor-Level Closed-Loop Simulation is the track inside it that decides what the traffic does and how the result is re-rendered.

research world-model generative e2e

Cosmos-Transfer2.5 base 7V surround · sensor-level OSM → layout → video WorldSim ↔ model gRPC First 7V 4-step distillation Flux → 7V style transfer All-in-one OneModel

7VSurround-view, multi-view consistent

4-stepDistilled from a 35-step teacher

~13.9×Peak measured inference speedup

1 modelLayout gen · GS-fix · harmonizer

01 · Cosmos-Transfer2.5 on internal driving data

A 7V generative world model, built on Cosmos-Transfer2.5 and validated on internal data.

02 · Real maps → 7V video (Ingolstadt OSM)

Download a real city map (OSM), convert it to the model's layout, and generate surround video — including a snow variant.

Control layout from OSM.

Generated 7V sensor video.

Same scene, snow weather.

03 · Flux → single-frame 7V: weather, season & style transfer

Lift a single-image Flux generator into a 7V surround model — one synchronized frame across every camera — then drive it with text to restyle the whole rig into new weather, seasons, and looks while the underlying geometry stays put.

Single-camera FluxExtended to a 7V surround rig, one consistent frame

One fixed lookText-driven weather / season / style swaps

Per-view driftCross-camera geometry held while appearance changes

Surround style transfer. A single synchronized 7V frame restyled across weather, season, and visual style — rain, snow, dusk and more — from one shared, text-controllable generator, with cross-camera consistency preserved.

04 · WorldSim ↔ World Model gRPC bridge

WorldSim serializes scene data (calibration, ego trajectory, dynamic obstacles, static map) over OSI semantics to the World Model server, which validates, caches, converts to Parquet, and asynchronously triggers Cosmos rendering — a semantic + async bridge between simulation and generation.

05 · First 4-step distillation of 7V surround video

Logic map · distillation

35 steps → 4 — compress the teacher's path, keep the picture sharp

Everything except the student is scaffolding: the teacher is frozen, the critic and both losses exist only during training, and only the student ships. Two losses because a big step can go wrong in two different ways — off the path, or off the data distribution.

Stages Frozen teacher Step schedule Student · path loss Critic · distribution loss Training loop Shipped result

Path Runs at inference Train-time only Frozen / pretrained

Links Conditioning / structural Train-time only

View as tablesame content, no interaction needed

35 steps → 4 — compress the teacher's path, keep the picture sharp — every node with its input, logic and output.
Stage	Input	Logic	Output	Why / role
35-step Teacher Teacher frozen	Real frames and their conditions	Run the slow, exact denoising trajectory	Reference path plus the student's initial weights	Freezing the teacher gives the student a target that cannot drift; if both moved, there would be nothing anchoring the compression to reality.
4-step Student Student	Teacher weights as initialisation	Take four large denoising steps, with classifier-free guidance folded into a single forward	Few-step samples	Baking guidance into the weights removes the second forward pass per step — the speedup compounds with the step reduction rather than fighting it. My role. Designed and trained the student; owned the distillation track end to end.
Sparsify Timesteps Schedule train-only	The teacher's noise-level schedule	Index-mapped sub-sampling of the teacher's timesteps	Four student steps, each covering a teacher span	Where the four steps land matters more than how many there are — a bad schedule wastes a step on a range where almost nothing changes.
Dynamic Critic Critic train-only	The student's current samples	Learn a score for the student's evolving output distribution	A direction back toward the real distribution	A fixed critic goes stale the moment the student improves; retraining it alongside keeps the supervision meaningful for the whole run. Trade-off. A trainable critic doubles the moving parts — the price is the alternating loop below.
Bridge Consistency Loss · path train-only	A teacher span and the matching student step	Match the average velocity across the bridged span	Path gradient	Matching average velocity over a span, rather than the endpoint, is what lets one student step legitimately replace many teacher steps.
Distribution Matching Loss · look train-only	Teacher and critic scores	Push the student's output distribution toward the real one	Distribution gradient	Path correctness alone yields blurry frames; this is the loss that buys back the sharpness few-step sampling normally loses.
Alternating Optimisation Loop train-only	Student and critic	Update the critic and the student in alternation, never in one graph	A stable training loop	Two adversarial-style networks in one backward graph is where 7-view video distillation runs out of memory; alternating keeps the graph small enough to fit. Trade-off. Alternating costs wall-clock steps, but it is the difference between training and not training at all at this resolution.
Progressive Compression Cascade train-only	The model from the previous level	Halve the step count, initialise from the level above	The next-level student	Jumping straight to one step diverges; halving keeps every level close enough to its teacher that the optimisation stays well-behaved.
4-step 7V Model Result	Path and distribution gradients	Four forward passes, seven views kept mutually consistent	Surround video at interactive speed	Speed is not a vanity metric here: below a certain rollout time the world model simply cannot sit inside the simulation loop at all. My role. Measured the speedup and integrated the distilled sampler into the platform.

Mode	Spec	Original	Distilled	Speedup
Single clip (first)	29 / 203 frames	796.76 s	106.32 s	7.49×
Single clip (steady)	29 / 203 frames	744.16 s	53.68 s	13.86×
Auto-regressive (long)	57 / 399 frames	1546.87 s	165.87 s	9.33×

Distilled 4-step model — surround generation at a fraction of the cost.

Editable platform — rare interaction scenarios

Edit the layout on a real clip, then regenerate with the 4-step model to produce hard-to-collect interaction data.

Raw captured clip.

Edited control layout.

4-step regenerated result.

06 · All-in-one OneModel — three modes, one denoiser

Logic map · OneModel

Three jobs, one denoiser

Layout generation, Gaussian-Splatting repair and harmonization look like three different problems. They differ only in what the sampler starts from and what it is conditioned on — so they can share one backbone.

Stages Shared backbone Mode 1 · layout generation Mode 2 · GS repair Mode 3 · harmonization Shared control path

Links Data flow (inference) Conditioning / structural

View as tablesame content, no interaction needed

Three jobs, one denoiser — every node with its input, logic and output.
Stage	Input	Logic	Output	Why / role
Shared Backbone Backbone	Conditions and a start latent	One 7-view video diffusion model	The same weights, driven three ways	Three task-specific models meant three trainings, three checkpoints and three sets of drift. One backbone means an improvement anywhere lands everywhere. Trade-off. A shared backbone cannot be tuned to death for any single task — worth it while all three tasks are still moving. My role. Designed and owned the OneModel unification.
Layout Generation Mode 1	Layout video + camera/time/7V	Generate from pure noise	Driving-scene video	Controllable surround generation
GS Fix Mode 2	Degraded GS video + optional layout	Denoise from degraded start	Clean video	Fills holes / artifacts
Harmonizer Mode 3	Disharmonized video + mask	ControlNet harmonization	Harmonized video	Natural fg/bg lighting
Pure Noise Start	Gaussian noise	Standard diffusion start	Latent to adapter	Maximum freedom
Degraded GS Latent Start	Low-quality GS render	Noise = the degradation itself	Latent to adapter	Keeps the real structure
Disharmonized Video Start	Inserted-asset video	Disharmony as the condition	Latent to adapter	Targets only the mismatch
Unified Condition Adapter Control	Layout / GS / disharmony / mask	Encode + inject into denoiser	Conditioned features	Three tasks, one control path
Shared Denoiser Denoise	Conditioned latent	Predict clean, consistent 7V	Denoised latent	One model holds all consistency
7V Video Output Output	Decoded latent	Layout · GS-fix · harmonized	High-quality surround video	One model serves three tasks

Data Training-set construction

GS-fix needs paired GT Same trajectory: degraded GS render → input, original real video → GT (~35W clips)

Harmonizer needs disharmony pairs SAM3 vehicle masks → match by object ID/IoU → build (disharmonized → harmonized) pairs

Vis The two new modes

GS-fix. Bottom: input · top: single-frame baseline · middle: our video result · then GT.

Harmonizer. Third column — our Cosmos-based result blends inserted assets most naturally.

My role. Built the platform end-to-end: the 7V world model on Cosmos-Transfer2.5, the OSM→layout→7V pipeline, the WorldSim↔model gRPC bridge, the 4-step distillation of 7-camera surround video (rCM + DMD2) — to our knowledge the first in our setting — the editable data platform, and the all-in-one OneModel. Numbers are the author's own measurements; details are sanitized.

Confidentiality note. Bosch (XC-CN) ongoing platform. Architecture, method, and the acceleration results the author measured are shown at a portfolio level; customer data, calibration, and product details are intentionally omitted or sanitized.