Real-world locations bloom with mixed reality: Meet Frances the Fish.

By Melissa Kaplan / 02 Jan 2018

Topics: internet of things | mixed reality | hololens | blog

With virtual reality rapidly being explored and released in theme  parks and arcades, it's a great era for thrill-seeking. And there are  more advanced thrills on the horizon as vendors awaken to mixed and  augmented reality's potential to alter this dimension in exciting ways.

We're starting our field work in this realm with a mixed reality  experience for an animated salmon named Frances. Frances the Fish began  as a short film following a character-filled aquatic journey from the  river to the ocean and back again, sharing info about climate change  with the audience in the process. Here's the story:

Frances The Fish Teaser from Jonathan Skurnik on Vimeo.


It's clear this tale can be readily transformed into a  multi-dimensional, interactive experience. We began by turning the  AfterNow lab into a coral wonderland comfortable enough for an  exploratory salmon.

Here's Frances in action:

One of the more exciting things to notice in this prototype is the  way objects occlude the fish as it weaves in and out of the space -- a  significant detail for successfully completing the illusion. Frances  will fall out of sight when swimming behind desks and under tables. The  fish will also hide behind a range of plant life which the user can air  tap onto the seafloor.

Frances hiding.

This is just our initial build; the app can run (or rather, swim)  anywhere you go. But bring this to a theme park or aquarium and you open  up to a flood of ways to heighten the experience. Reading the  environment, the holographic fish can be made to react to hot spots and  convey information, either as a linear story or one that the user can  stitch together depending on how they navigate their visit.

As with any app on the Hololens, the user can enjoy themselves in  this mixed reality world with Frances without disrupting the reality of  other aquarium patrons -- though you do have the option to affect your  environment as part of an experience. Locations that offer wifi-enabled  devices can go a step further with a holographic UX that controls the  Internet of Things -- a service we're quickly building with applications  like Hololens Hue,  which changes the color and brightness of a Philips Hue light bulb.  Frances can potentially light up a real patch of flora when swimming by,  trigger live sounds, or set off other multimedia.

Theme parks and other real-world settings are already making creative  use of their spaces for unique installations that can only happen  there. The Void in New York makes its immediate environment immersion-friendly to  create an entertaining, hyper-realistic experience. The blend of real  and holographic can also be a powerful educational tool; the Jet  Propulsion Laboratory is using mixed reality and data from the Curiosity rovers to allow its visitors a chance to walk on Mars.

As we develop this app and continue working on more location-specific  mixed reality experiences, a combination of storytelling and UX that  incorporates the space will be key.

Banner image from

Questions about mixed reality or the Hololens? Get in touch.

Melissa Kaplan

Culver City