Fast Spatial Reasoning of Implicit 3D maps through Explicit Near-Far Sampling Range Prediction

IROS 2024
*Equal contribution
1MultiplEYE 2Brown University 3LG Electronics 4Seoul National University

(a) Hierarchical Sampling

(b) TSDF Sampling

TSDF-Sampling reliably renders neural surface fields with only 18 samples per ray for a 20x20x5 $\text{m}^3$ scene

Abstract

Multi-view neural surface reconstruction has exhibited impressive results. However, a notable limitation is the prohibitively slow inference time when compared to traditional techniques, primarily attributed to the dense sampling, required to maintain the rendering quality.

We present TSDF-Sampling — a novel approach that substantially reduces the number of samplings by incorporating the Truncated Signed Distance Field (TSDF) of the scene. While prior works have proposed importance sampling, their dependence on initial uniform samples over the entire space makes them unable to avoid performance degradation when trying to use less number of samples. In contrast, our method leverages the TSDF volume generated only by the trained views, and it proves to provide a reasonable bound on the sampling from upcoming novel views. As a result, we achieve high rendering quality by fully exploiting the continuous neural SDF estimation within the bounds given by the TSDF volume.

Notably, our method is the first approach that can be robustly plug-and-play into a diverse array of neural surface field models, as long as they use the volume rendering technique. Our empirical results show an 11-fold increase in inference speed without compromising performance.

Method

TSDF-Sampling integrates TSDF volume generated only by the train views, and the TSDF volume can provide sampling bound prior for upcoming novel views. Note that the model agnostic TSDF-Sampling can use any existing neural surface fields as a baseline. Below visualizes the coarse samples, and fine samples will be upsampled from the coarse samples.

Method - Hierarchical Sampling.

(a) Hierarchical Sampling

Method - TSDF Sampling.

(b) TSDF Sampling

Ablation

In the synthetic Garage dataset, TSDF-Sampling enables high-fidelty renderings of color, depth, and normal, with as few as 4 samples per ray.

Results

In the widely used Replica dataset, the model-agnostic TSDF-Sampling can render more complete and quality results with a limited number of neural network queries.

Ours Hierarchical Sampling
Ours Hierarchical Sampling

The PSNR drastically increases from 23 to 28, and the depth error decreases by more than 50%, when using the TSDF Sampling approach.

Ours Hierarchical Sampling
Ours Hierarchical Sampling

BibTeX

@article{
  title     = {Coming Soon},
  journal   = {},
  year      = {},
}