S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generation

Yurui Chen, Junge Zhang, Ziyang Xie, Wenye Li, Feihu Zhang, Jiachen Lu, Li Zhang

School of Data Science, Fudan University

Reconstruction Method of Dynamic scenes

We use a cascade sampling strategy to reconstruct the background and moving objects together. For each sample point on the ray casting from o with direction d, we use an object network whose weights are shared through all sampling stages to decode volume rendering attributes when the point is in the corresponding object bounding box while background/proposal network to decode the point not in any box. Notice that if the sampling point is not in the final stage, the network only outputs density and the gradient will not pass to the object network. Rendering weights computed by the proposal stage will be only supervised by the rendering weights in the next stage by proposal Loss. Each network is composed of a multi-scale hash grid and an MLP decoder. For better anti-aliasing rendering, we follow Zip-NeRF and aggregate the hash grid features in the cone for the background points before passing them to the MLP while only using down-weighting features for the object points.

Simulation Pipeline

The whole pipeline of our simulation system. We construct our background representation and foreground bank. Given any pose of the camera and object, we render for these two branches separately. Then we judge the occlusion relationship between the foreground and background by depth, process the edge and illumination of the inserted object, render the shadow, and finally obtain the simulation data with labels.

Object Insertion Refinement

We present two different approaches to refine synthetic data: (a) A naive neural network-based method that leverages networks to inpaint object edges and adjust the lighting corresponding to the object mask. (b) An optimization-based refinement approach that utilizes a differentiable PBR pipeline to apply SDS loss on the composed image, optimizing the background ambient lighting to achieve realistic relighting and shadow.

S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generation

Yurui Chen, Junge Zhang, Ziyang Xie, Wenye Li, Feihu Zhang, Jiachen Lu, Li Zhang

School of Data Science, Fudan University

Reconstruction Method of Dynamic scenes

Rendered RGB and Depth

Dynamic Scenes Reconstruction

Replay

New Track

New Track

Novel View

Simulation Pipeline

Object Insertion Refinement

Simulation: Car Insertion from Other Dataset with Refinement

Generated Objects