TeleWindow

2018->ongoing/Tool/Research

Collaborators: Michael Naimark (Principal Investigator), Cameron Ballard, Barak Chamo, Ivy Huang, Grace Huang, Mateo Juvera Molina, Ada Zhao , and Bruce Luo

Early single camera prototype

TeleWindow is an ongoing telepresence research project being conducted under the guidance of Michael Naimark. The aim is to explore how a two-way, one to one communication, "telepresence window", system can be developed and how this system can aid communication in the digital age. Utilizing these different ingredients, we can prototype a device that can offer features such as two-way eye contact, stereoscopic 3D, and presence. All done in realtime.

Currently, we are using four Intel RealSense cameras in order to create a volumetric pointcloud representation of a person, and then we correct perspective based on the head/eye positions of the user. The stereoscopic eye-tracked volumetric display was acquired through SeeFront.

My general involvement in the project included/includes refinement of the system through prototyping and debugging. The specific components I worked on were integrating the initial off-axis perspective rendering algorithm, used the SeeFront API to integrate I/O between the SeeFront display and the software environment, and am also writing a synchronized multi-cam record/playback system for RealSense pointclouds.

Four camera coverage demonstrated through multicolor splat.

Diminishing occlusions through multi-cam coverage.

Rough real life environment capture. Monoscopic view for demonstration.

Demonstration of realtime movement and capture in the rendered virtual environment.