Project Goals

I aim to create a method of generating real-time simplified scene reconstruction and displaying it via head mounted display. My method will involve attaching a camera to an Oculus DK2 and using computer vision techniques to track correspondences, calculate camera essential matrices, and generate a dense 3D reconstruction point cloud. I will then take the point cloud and generate simplified geometry for the environment (fitted planes) and display it on the head mounted display.

Intermediate Goals

  • Feature tracking in single-view video
  • Point cloud generation from feature triangluation
  • If needed, dense depth reconstruction
  • Geometry reconstruction from point cloud
  • Camera pass through to rift DK2
  • Geometry rendering on rift DK2
  • Displaying composited image on rift DK2

    • Motivation

    One of the essential parts of augmented reality is understanding the world around the user. A first step toward that is understanding the basic layout of the environment; finding the ground plane, wall planes and other large features which will allow virtual elements to be displayed and interact appropriately.

    Relevant Prior Work

    Real-Time Consensus-Based Scene Reconstruction using Commodity Graphics Hardware, R. Yang 2002
    Detailed Real-Time Urban 3D Reconstruction From Video, Pollefeys 2007
    Live Dense Reconstruction with a Single Moving Camera, Newcombe-Davison 2010
    Rahul Raguram’s research

    Project Demos

    A user will put on the Oculus DK2 with camera attachment, and after an initialization step, a simplified geometric view of their current envirionment will be displayed and will track as they move their head. I hope to have 3 modes: camera view; a view of what the camera is currently seeing, geometry view; a view of only the reconstructed scene, and overlay view; the camera view with the geometry composited over it.

    Note to Self: Methodology (Thoughts on Algorithms)

    Point Cloud -> Geometry via RANSAC, RECON or Hough Transform ??? Look for plane with largest amount of support
    Visual SLAM: Simultaneous localisation and mapping
    OpenCV Library
    PCL: Point Cloud Library and bundle adjustment
    Support plane and segmentation

    Kishore recommends placing optical pattern around room, using IR cameras with reflective balls or IR leds like array in EVE room to assist in tracking, since its already a solved problem.