Consumer virtual reality (VR) and augmented reality (AR) are just around the corner, the biggest technology companies are getting involved and thousands of developers and filmmakers are producing experimental content.
How many headsets will be sold in 2016? Nobody really knows how fast or how big the market will grow. Although there are some predictions by BI Intelligence ($2.8B VR hardware market by 2020) and Digi-Capital (AR $120B and VR $30B by 2020).
The size of the market aside, headset displays are a part of our digital future. For example the team behind Oculus VR have spent years researching and developing the Oculus Rift, the head-mounted gaming display that is believed to create, along with appropriate software, a seismic shift in many professions and practical training in dangerous or sensitive fields.
The two key questions are;
- How to take advantage of this new technology for better training outcomes and
- How to make sure the content you build for training today will be adaptable to the mixed reality devices coming soon.
Most AR and VR content can already be experienced on flat screens as mobiles but without the factor of immersion. Your learners may not yet have AR or VR headsets, but you can field test scenarios on mobile or desktop before they do.
Let’s start with a clear distinction between VR and AR. Immersive virtual reality is a perception of being physically present in a non-physical world. It is created by surrounding the user of theVR system with an environment, sound and other stimuli that provide a total engrossing environment of immersion, imagination and interaction to increase engagement.
Augmented reality mixes the real and digital worlds, typically overlaying computer generated information, images and 3d content into a seamless experience with the real world.
Advances in both VR and AR are leading to hand movements and voice as natural input mechanisms.
Hardware
How does AR work?
Augmented reality is also known as Mixed Reality, combining the real world with computer generated imagery.
The simplest form of AR is to have information displayed in your field of view that is useful to your context. Google Glass is a good example of this, the information displayed is always a floating screen infront of you separate from the environment. Sony’s SmartEyeglass video demonstrates this use of AR well
This type of AR is great for hands free information as you need it, it’s similar to carrying a dashboard in your field of view wherever you go.
Augmented reality becomes a lot more interesting when it can seem to live in the environment you are looking at.
Markers
The most prevalent form of AR uses markers that a camera can recognise then uses that marker as a calibration point to align and overlay a computer generated image or 3D object.
Initially markers were blocky QR codes but now using image recognition any unique picture or photo can be the trigger to display an AR scene. This technique has been used for many years and is quite mature. We recommend trying out apps from Layar and Aurasma to create your own experimental AR experiences on your smartphone.
Here are some fun examples
Here is an example from LearnBrite’s game studio
Markerless Tracking
Things start to get a lot more exciting when the device can understand the environment around it. Instead of needing to look at a marker or trigger picture to keep an experience active, a user can instead look around in any direction. This is called markerless tracking. All the big companies have technology for markerless tracking, Facebook acquired 13th Lab, Apple acquired Metaio, Amazon has Flow technology, Microsoft has xBox Kinect technology built into its Hololens, Google has Project Tango and Sony has their Eyetoy technology. The only company noticeably missing the techonology is Samsung however they have a partnership agreement with Oculus.
13th Lab (Acquired by Oculus/Facebook 2014)
Metaio (Acquired by Apple 2015)
Its great to look at these through the “window” of a smartphone or tablet, but it’s amazing when you can see augmented reality through a headset or glasses, seamlessly integrating the virtual vision with your own.
AR Headsets and Glasses
Today a good collection of headsets are either in development or released with varying degrees of capabilities and prices.
- Microsoft Hololens Developer Edition Q1 2016 $3000 https://www.youtube.com/watch?v=SKpKlh1-en0
- Daqri https://www.youtube.com/watch?v=qIU456jtkmw
- Meta 1 $667 https://www.youtube.com/watch?v=b7I7JuQXttw
- Avegant Glyph https://www.youtube.com/watch?v=3cZ7QT3CdyU
- Google Glass was $1500 https://www.youtube.com/watch?v=P4QJGg_lMNI
- Google Magic Leap https://www.youtube.com/watch?v=kPMHcanq0xM
- Optinvent $799 Android SDK http://optinvent.com/
- Epson Moverio http://www.epson.com/cgi-bin/Store/jsp/Landing/moverio-bt-200-smart-glasses.do
- Osterhoutgroup $2750 http://www.osterhoutgroup.com/products-r7-glasses
- ReconJet $499 http://www.reconinstruments.com/products/jet/
- Vuzix M100 $1200 http://www.vuzix.com/consumer/products_m100/
- Cast AR $400 http://castar.com/
- Sony SmartEyeglass $899 http://developer.sonymobile.com/products/smarteyeglass/
VR Headsets and Glasses
Oculus Rift and Oculus Touch
Google Cardboard anyone can make one and can cost just a few dollars!
Especially Google Expeditions for Education
- Samsung Gear VR now $99 https://www.youtube.com/watch?v=-gnvQS2xhRg
- HTC Vive https://www.youtube.com/watch?v=8UnodUYnrG8
- Razor OSVR https://www.youtube.com/watch?v=dxeQVBwyh8Y
- Sony Morpheus https://www.youtube.com/watch?v=0USSvBSCcc8
- Viewmaster VR by Mattel for $30 https://www.youtube.com/watch?v=Oi_SyAVC5gI
Content Creation
There are two main content types computer generated 3D environments used for AR and VR or 360 degree panoramic photos/videos used for VR only.
360 Degree Video and Panoramas for VR
360 degree photos and videos benefit from capturing real world experiences at low cost for immersive playback. The quality of the experiences can be quite high, just like in traditional film, producing great results requires great talent which usually amounts to high cost. Even typical video recordings lack the flexibility for altering training experiences without expensive re-recording. And once experienced they hold little value for re-use.
Still, they are undeniably valuable and now even more accessible than ever given the number of camera rigs available to capture and YouTube’s brilliant move to support 360 degree videos in its video player, making these experiences available to millions of users on flat screens ready for immersive VR. Using Google Cardboard or Oculus Rift you can call up one of the many 360 videos on YouTube and jump right into the action. You will always be fixed to the position of the camera but you can look in any direction you choose. This brings about a whole new philosophy in film direction where the user needs to be guided to areas of interest within the panoramic scene or risk missing out on key events in the linear experience that happened out of view or behind them.
At LearnBrite we have been experimenting using the world’s leading branching scenario editor ChatMapper to build choose your own adventure 360 degree video experiences. Combining the immersiveness of 360 video with interactive storytelling.
A trick has been available to use YouTube for very simple branching videos for many years using annotations with links to Video A or Video B. This is a great place to experiment in a very simplified fashion. Take a look at this 360 video with annotation links at the top, choose Landing for example to jumps to the aircraft Landing section of video https://www.youtube.com/watch?v=HEEIzZ7UjRg
Alternatives to YouTube for playing 360 videos that support Oculus Rift are KRPano and EleVR
We have found that 360 videos are best kept to a fixed viewpoint and built to be looped so a learner can continue to look around and identify the piece of learning content, then make a decision to proceed further. We have also identified that even though it is possible to record stereoscopic(3D) 360 degree videos the stitching (production) and delivery quality adds little value to the end result for immersion. Stereoscopic recordings add depth to the immersive experience enabling the learner to judge the distance of items in the scene.
Another downside of 360 video is that they are recorded from a fixed point of view, you can look around in any direction but you cannot move your head left or right, up or down, or forward and backward. The discrepancy between head movement and the fixed camera position can create nausea. Although we believe there are some smart improvements that can be made to 360 immersive players that can emulate head movement and therefore reduce nausea. We look forward to further enhancements in this area for both pre-recorded and live streaming 360 video experiences.
Computer Generated 3D Scenes for AR and VR
Using computer generated 3D makes it easier to produce stereoscopic immersion adding the illusion of depth to experiences.
3D content is built using popular 3D modelling software, CAD or by scanning real world environments.
Once you have built your 3D content, you then need to choose how to deploy it to users.
Unity 3D is a very flexible game authoring tool. Its output can be compiled to work on many different devices. And importantly with many VR/AR devices including the Oculus and Hololens. The platform has a great ecosystem and is a hot acquisition target for those now relying on it for their developer ecosystems.
Unity plugin does not work in Chrome and is a huge blocker. It’s not a secret that users of online content take the path of least resistance, be they readers, gamers, or learners. Having to download an VR/game experience instead of being able to play it in a browser, especially if there is no perceived value in downloading the game (unlike very well reviewed games, or something that looks like YOU HAVE TO EXPERIENCE IT), it is akin to a brick wall for uptake.
WebGL output from unity does not perform acceptably across devices. It forces the publishing dimensions to be fixed or fullscreen meaning it will not work for responsive design. The webplayer is not designed from the ground up to use the features of HTML and WebGL, it is virtualized experience which brings with it some overheads but more importantly creates a black box distancing itself from web integration.
Getting through the sys-admin IT gatekeepers to install plugins in enterprise, government and education will likely always be a barrier to adoption.
Although WebGL is now available on more than 80% of browsers. Existing WebGL game development tools are still not mature enough for commercial use. And the number of games-people with WebGL expertise is small. This presents a significant opportunity. Unity is not a tool for instructional designers, content creators or web authors. A layer needs to exist allowing use of powerful HTML technologies without the need to be a games programmer.
The LearnBrite platform is built to allow easy authoring by content creators and optimized Responsive delivery using HTML and WebGL on all devices. It abstracts the need to be a games developer and focusses the content creator on the engagement outcomes.
We happily develop using Unity where a project can benefit from it. However a path to delivery in native HTML is always preferred if it removes the barrier to access.
Future
There are still many hurdles to overcome with tracking lag, calibration, stability, occlusion of objects and nausea. Building AR and VR for consumer and enterprise use is a big challenge, but with the commitment of some of the largest technology companies in the world with the increased computing power of computers and mobile devices its future is certain.
