Driving Scene Synthesis on Free-form Trajectories with Generative Prior

Zeyu Yang^1, Zijie Pan^1, Yuankun Yang^1*, Xiatian Zhu², Li Zhang¹

¹Fudan University, ²University of Surrey

^*Equally contributed.

Abstract

Driving scene synthesis along free-form trajectories is essential for driving simulations to enable closed-loop evaluation of end-to-end driving policies. While existing methods excel at novel view synthesis on recorded trajectories, they face challenges with novel trajectories due to limited views of driving videos and the vastness of driving environments. To tackle this challenge, we propose a novel free-form driving view synthesis approach, dubbed DriveX, by leveraging video generative prior to optimize a 3D model across a variety of trajectories. Concretely, we crafted an inverse problem that enables a video diffusion model to be utilized as a prior for many-trajectory optimization of a parametric 3D model (e.g., Gaussian splatting). To seamlessly use the generative prior, we iteratively conduct this process during optimization. Our resulting model can produce high-fidelity virtual driving environments outside the recorded trajectory, enabling free-form trajectory driving simulation. Beyond real driving scenes, DriveX can also be utilized to simulate virtual driving worlds from AI-generated videos.

Method Overview

We propose leveraging video generative prior with rich spatiotemporal knowledge for generalizable driving scene synthesis from single-trajectory recorded videos.
We introduce DriveX, a novel driving scene synthesis framework for free-form trajectories that innovatively constructs an inverse problem to enable the use of a video diffusion model as a prior.
Extensive experiments demonstrate that DriveX significantly outperforms existing state-of-the-art alternatives in driving scene synthesis using single-trajectory recorded videos. Additionally, we showcase the superiority of our method in rendering novel trajectories of virtual driving worlds from AI-generated videos, even when faced with inherent content inconsistencies. This approach enables a more economically scalable large-scale simulation without the need to collect videos for each scene.

Driving Scene Synthesis on Free-form Trajectories with Generative Prior

Zeyu Yang^1, Zijie Pan^1, Yuankun Yang^1*, Xiatian Zhu², Li Zhang¹

¹Fudan University, ²University of Surrey

^*Equally contributed.

Abstract

Method Overview

Driving scene synthesis on novel trajectory

Comparisons on Waymo along novel trajectory

Comparison with DriveDreamer4D

Driving scene synthesis along novel trajectory from AI-generated videos

Driving Scene Synthesis on Free-form Trajectories with Generative Prior

Zeyu Yang1*, Zijie Pan1*, Yuankun Yang1*, Xiatian Zhu2, Li Zhang1

1 Fudan University, 2 University of Surrey

*Equally contributed.

Abstract

Method Overview

Driving scene synthesis on novel trajectory

Comparisons on Waymo along novel trajectory

Comparison with DriveDreamer4D

Driving scene synthesis along novel trajectory from AI-generated videos

Zeyu Yang^1, Zijie Pan^1, Yuankun Yang^1*, Xiatian Zhu², Li Zhang¹

¹Fudan University, ²University of Surrey

^*Equally contributed.