Reconstructing street scenes that stay physically realistic in driving simulations

Building high-fidelity 4D street scenes for autonomous driving simulation requires both photorealistic novel views and accurate motion modeling—but existing methods fail at both simultaneously. The authors discovered the problem: a mathematical ambiguity that forces the model to choose between learning spatial details or temporal dynamics. They propose Orthogonal Projected Gradient, which protects spatial structure first, then constrains temporal updates to avoid corrupting it, plus a smoothness constraint that enforces physically plausible motion. Tests show the method maintains visual quality while modeling time-varying effects needed for closed-loop driving training.