Eric Sun

NeRF-UAV: Inexpensive 3-D Reconstruction and View Synthesis Using RGB Camera Equipped Drones and Neural Radiance Fields

Started June 30, 2021

Abstract or project description

Digitizing and reconstructing objects has always been a goal of computer vision, especially as the Metaverse's influence continues to grow and the transition to digital life begins. However, classical methods of 3-D synthesis require expensive and impractical setups as well as controlled settings, which are inapplicable beyond laboratory settings. This paper proposes a novel deep learning application for aerial reconstruction through optimized Neural Radiance Fields for Unmanned Ariel Vehicles (NeRF-UAV). We then combine classic NeRF with recent developments a Pixel-shuffle Down-sampling (PD) techniques to overcome the low megapixel image resolution for affordable drones and compare our results to traditional photogrammetry and demonstrate superior recovery of both fine color and geometric detail. We also numerically show NeRF-UAV synthesizes more accurate views as compared against generic implementations of NeRF by conducting SSIM and PSNR tests. Finally, we propose an easily upscaleable framework to adapt this technology to swarms. Applications of this technology include the digitization of large objects, special effects, 3-D asset generation, and education.

The goal of this paper is to outline a methodology to utilize the flexibility of affordable over-the-self commercial drones to accomplish 3-D reconstruction and view synthesis. We leverage recent deep learning techniques in both image up-scaling and denoising as well as rendering to optimize our processes. Ultimately, NeRF-UAV is a framework for flight control and image processing that accomplishes state-of-the-art results and is adapted specifically for affordable, low-megapixel drones.

NeRF-UAV utilizes uses object-recognizing networks to create a uniform capture of a stationary object before digitizing it. NeRF, though accurate in controlled settings, fails to handle noise well. We posit that low-megapixel images with high background noise, like those retrieved from DJI Tello drones utilized in this study, create excessive artifacts when rendering by prematurely terminating rays during training. Our application of denoising techniques prevents the formation of artifacts and we adapt classic NeRF to low-megapixel images by combining both frameworks together to create superior results, which can be seen in the results section both qualitatively and quantitatively.

We also show better volume synthesis compared to traditional methods, which creates artifacts and missing meshes in areas without adequate coverage in the sample photos. We hypothesize that NeRF sampling allows for occupancy to be determined from more camera angles besides extrapolating from feature points observable from multiple angles, the methodology COLMAP, a conventional algorithm, employs.

We foresee the primary application for this technology is within the digitization of a wide arrange of objects. Since drones are not contained by capture size, any object from as small as a bottle to as large as a person or a building can be captured using the same algorithms, without significant modification.

In the near future, this same process can also benefit VFX rendering and 3-D asset generation. As mentioned before, current methodologies are either too size-constraining or expensive for public use. The ability for affordable drones to create convincing models allows for both quick and easy generation of 3-D assets. Currently, markets like the Video Game asset industry and VFX industry can also adapt this technique for mark-less captures of models, as well as bullet-time renderings.