Magic Leap has been teasing us with the promise of seamlessly mixing the "real world" with computer graphics, aka CG. With half a billion dollars from Google, and possibly $1 billion more on the way, it seems possible that there's truth behind the hype.
While a few lucky people have seen demos, it wasn't until a few days ago that the general public got a glimpse of what they've been cooking up in this video. With "Shot directly though Magic Leap technology" tagged at the bottom, the video should provide some real clues about their prototype's capabilities.
Let's go frame by frame and see what we can extract from this sneak peek.
It isn't possible to project the color black; instead, it's necessary to darken the world around you. If you've ever tried to set up a projector in broad daylight, you know this intuitively -- it works better in a dark room. In the video, the background appears darkened. The Hololens utilizes sunglasses to darken the environment relative to the CG elements, but some users have still complained that graphics sometimes appear ghostly. This demo appears to show Magic Leap struggling with the same issue.
When a CG object passes behind a real object, you expect the real object to hide the CG object. This is called occlusion. In the video, the robot is occluded by the desk, and it looks incredibly good. Hard lines like the edge of the desk can be difficult to capture with laser scanners.
In our preview of the Hololens, we saw that models used for occlusion were often "soft" around the edges. Furthermore, thin objects like the legs of the desk can be easily lost. Either Magic Leap has really nailed real-time reconstruction or they have pre-modeled this environment.
Despite being really good, the occlusion isn't perfect. Just before the robot fades to black at 0:13, the right side of the leg shows a small strip of black, where the occlusion extends beyond where it should. This may seems like a small defect, but the human visual system can be aware of such defects without understanding them.
When the human eye focuses on an object nearby, objects in the distance become blurry, just like a camera does. The video goes out of its way to point out that with Magic Leap, the CG can be de-focused. With typical stereoscopic setups, like those found in a 3D movie or on the Hololens, objects may appear to exist in front or behind the screen, but for CG objects to appear sharp your eye must focus on the screen on which they are projected. It would appear that Magic Leap is giving us a glimpse of some sort of light field display.
To overlay CG elements, it is necessary to know where the user's head is relative to the environment (we experimented with this at the Hololens Academy). Lastly, their tracking, appears to need improving. During the solar system demo, there are visible pops and shakes. Again, this is a place where the visual system is very sensitive; you expect everything in your environment to move with the same relative motion. I'm a little surprised to see these defects in such a controlled demo, but they may just not be concerned with that area yet.