Extend Your Phone Screen with the HoloLens

By Jesse Vander Does / 01 Mar 2017

Topics: hololens | mixed reality | design | blog | lab

The HoloLens has three main inputs: voice, gaze and gesture. These were thoughtfully chosen by Microsoft designers to leverage what the HoloLens does well and to form a solid foundation for consistent interactions between applications.

Still, sometimes I feel like these are blunt instruments.

If you've used the Vive, you've gotten a taste of what a six-degrees-of-freedom (6DoF) controller can do for 3D interactions.

Two things pop out to me about the Vive controllers: 1. They make user interface (UI) discoverable and 2. They allow intuitive placement for objects in 3D space.

Our latest proof of concept uses your phone as a 6DoF controller to place objects in 3D and edit them. One end of the phone is used for creating new objects. The user slides the objects off of the phone screen into 3D space. Fine adjustments can be made to the placement of the object by positioning and rotating the phone and by using the screen to slide the cube off of the screen at particular position.

The other end of the phone has a different tool used for changing the color of the cube you've created. When the color dropper is positioned against a cube, a color wheel is displayed, using the HoloLens to extend the phone's display. When the user drags up and down on the phone screen, the color wheel spins, and the cube’s colors change.

Using a controller breaks one of the design rules specified by Microsoft, mainly that interactions should be a comfortable 1.5 meters from the user. This recommendation is designed to reduce eye strain, but also helps ensure that the items the user is interacting with stay within the frame. Nonetheless, there will most definitely be use cases that benefit from controllers.

Moreover, there are a lot of fun things we could do with a 6DoF controller that also has a touchscreen. Here are just a few ideas:

  • Draw a curved line on your phone, then extrude it by swiping your phone through space
  • Pull 3D objects back into your phone GhostBusters style
  • Render a slice plane on your phone when it intersects with a 3D object

This is, of course, just a tool, and it needs a community to make it work. So what would you do with this? Tweet us @AfterNowLab with your ideas for using your phone as a HoloLens controller.

UPDATE: UploadVR expended on this article: https://uploadvr.com/watch-smartphone-turn-6dof-controller-hololens/

Jesse Vander Does

Culver City, CA

Jesse worked for 5 years as a VFX artist and software developer. He now is a developer at HTML Fusion building web apps, bullet time photo rigs and holographic user interfaces.