VolumeDeform: Real-time Volumetric Non-rigid Reconstruction

Veröffentlicht 2016


1University of Erlangen-Nuremberg     2Max Planck Institute for Informatics     3Stanford University

We present a novel approach for the reconstruction of dynamic geometric shapes using a single hand-held consumer-grade RGB-D sensor at real-time rates. Our method builds up the scene model from scratch during the scanning process, thus it does not require a pre-defined shape template to start with. Geometry and motion are parameterized in a unified manner by a volumetric representation that encodes a distance field of the surface geometry as well as the non-rigid space deformation. Motion tracking is based on a set of extracted sparse color features in combination with a dense depth constraint. This enables accurate tracking and drastically reduces drift inherent to standard model-to-depth alignment. We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints. The problem is tackled in real-time at the camera's capture rate using a data-parallel flip-flop optimization strategy. Our results demonstrate robust tracking even for fast motion and scenes that lack geometric features.

Dataset

We provide a dataset containing RGB-D data of a variety of objects, for the purpose of real-time non-rigid reconstruction. The RGB-D data contains sequences taken from a PrimeSense sensor (color and depth images). Additionally, we provide meshes extracted for several frame (live reconstruction and canonical model).
Please refer to this publication when using the dataset.

Format

For each scene, we provide a zip file containing a sequence of RGB-D camera frames (X_data.zip). Each sequence contains:

  • Color frames (frame-XXXXXX.color.png): RGB, 24-bit, PNG
  • Depth frames (frame-XXXXXX.depth.png): depth (mm), 16-bit, PNG (invalid depth is set to 0)

Camera Calibration: The color and depth camera intrinsics for each sequence are provided in colorIntrinsics.txt and depthIntrinsics.txt. Note that these are the default values provided and we did not perform any calibration.

Meshes: The extracted meshes are contained in X_canonical.zip (frame-XXXXXX.canonical.ply) and X_reconstruction.zip (frame-XXXXXX.mesh.ply) for every 100th frame and the last frame. The transformations from worldspace to cameraspace for these frames are also provided in X_reconstruction.zip (frame-XXXXXX.world-to-camera.txt).

License

The data has been released under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License.

Advent Calendar

adventcalendar_data.zip (212 MB)

adventcalendar_canonical.zip
adventcalendar_reconstruction.zip

Boxing

boxing_data.zip (181 MB)

boxing_canonical.zip
boxing_reconstruction.zip

Hoodie

hoodie_data.zip (280 MB)

hoodie_canonical.zip
hoodie_reconstruction.zip

Minion

minion_data.zip (385 MB)

minion_canonical.zip
minion_reconstruction.zip

Shirt

shirt_data.zip (348 MB)

shirt_canonical.zip
shirt_reconstruction.zip

Sunflower

sunflower_data.zip (493 MB)

sunflower_canonical.zip
sunflower_reconstruction.zip

Umbrella

umbrella_data.zip (343 MB)

umbrella_canonical.zip
umbrella_reconstruction.zip

Upper Body

upperbody_data.zip (679 MB)

upperbody_canonical.zip
upperbody_reconstruction.zip

Bibtex

@inbook{innmann2016volume,
author = "Innmann, Matthias and Zollh{"o}fer, Michael and Nie{ss}ner, Matthias and Theobalt, Christian 
          and Stamminger, Marc",
editor = "Leibe, Bastian and Matas, Jiri and Sebe, Nicu and Welling, Max",
title = "VolumeDeform: Real-Time Volumetric Non-rigid Reconstruction",
bookTitle = "Computer Vision -- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands,
             October 11-14, 2016, Proceedings, Part VIII",
year = "2016",
publisher = "Springer International Publishing",
address = "Cham",
pages = "362--379",
isbn = "978-3-319-46484-8",
doi = "10.1007/978-3-319-46484-8_22",
url = "http://dx.doi.org/10.1007/978-3-319-46484-8_22"
}

Quelle

Proceedings of the European Conference on Computer Vision (ECCV)
2016; S. 362-379; ISBN: 978-3-319-46483-1;
14th European Conference on Computer Vision (08. - 16.10.2016)
Amsterdam

Herausgeber

  • Springer International Publishing