IEEE VIS 2024 Content: Opening the Black Box of 3D Reconstruction Error Analysis with VECTOR

Opening the Black Box of 3D Reconstruction Error Analysis with VECTOR

Racquel Fygenson - Northeastern University, Boston, United States

Kazi Jawad - Weta FX, Auckland, New Zealand

Zongzhan Li - Art Center, Pasadena, United States

Francois Ayoub - California Institute of Technology, Pasadena, United States

Robert G Deen - California Institute of Technology, Pasadena, United States

Scott Davidoff - California Institute of Technology, Pasadena, United States

Dominik Moritz - Carnegie Mellon University, Pittsburgh, United States

Mauricio Hess-Flores - NASA-JPL, Pasadena, United States

Room: Bayshore VI

2024-10-17T15:18:00Z GMT-0600 Change your timezone on the schedule page
2024-10-17T15:18:00Z
Exemplar figure, described by caption below
We present VECTOR, software that visualizes 3D reconstruction error for easier comprehension and more informed input modification. VECTOR consists of image views that superimpose residual error vectors on top of input images and 3-dimensional camera views that show spatially how multiple images are calibrated by a reconstruction algorithm to render a 3D output.
Fast forward
Keywords

Computer vision, stereo image processing, optimization, error analysis, uncertainty, SLAM, SfM, robotics

Abstract

Reconstruction of 3D scenes from 2D images is a technical challenge that impacts domains from Earth and planetary sciences and space exploration to augmented and virtual reality. Typically, reconstruction algorithms first identify common features across images and then minimize reconstruction errors after estimating the shape of the terrain. This bundle adjustment (BA) step optimizes around a single, simplifying scalar value that obfuscates many possible causes of reconstruction errors (e.g., initial estimate of the position and orientation of the camera, lighting conditions, ease of feature detection in the terrain). Reconstruction errors can lead to inaccurate scientific inferences or endanger a spacecraft exploring a remote environment. To address this challenge, we present VECTOR, a visual analysis tool that improves error inspection for stereo reconstruction BA. VECTOR provides analysts with previously unavailable visibility into feature locations, camera pose, and computed 3D points. VECTOR was developed in partnership with the Perseverance Mars Rover and Ingenuity Mars Helicopter terrain reconstruction team at the NASA Jet Propulsion Laboratory. We report on how this tool was used to debug and improve terrain reconstruction for the Mars 2020 mission.