Extreme Sensor Accuracy Benefits Virtual Reality, Retail, and Navigation
What minimizes lag that leads to VR “motion sickness,” explains why your store coupon app requires use of your smartphone’s accelerometer, and keeps fitness trackers and cars on track even when GPS fails?
Good Virtual Reality (VR) is an immersive experience, a simulated world with a hint of boundaries. Excellent VR is closer to the real thing. VR technology has to have a very high-density display with enough pixels to make sure that VR can emulate real-life details. Spatial stereo audio is also part of that immersive experience. That is, audio should sound like it’s emanating from the same place as the associated visual. Audio reception in a perfect VR experience would include the Doppler Effect and other physical vagaries of sound. Lastly, VR immersion should include the ability for the user to intuitively interact with the system, as you might in real life. However, VR has not yet reached perfection in any of the above areas; with a high level of visual detail, experientially accurate sound, or the ability to interact naturally in a virtual world. Alas, VR is still in the early days, causing “VR sickness” by making many users nauseated; a sickness that’s mainly due to a time lag of more than 20 ms, as the differences in sensory inputs conflict with each other.[i] The latest VR products have a lag delay of 6 to 10 ms, however, enabling lengthier and more enjoyable VR experiences.
VR will show nothing in the user’s actual surroundings, whereas Augmented Reality (AR) supplements the real-world view, much like a heads-up display with an overlay of information superimposed onto the user’s view of actual surroundings. For developers, there’s a “spectrum of immersion” in Virtual Reality (VR), depending upon the technology that’s put into play. VR can be as simple as sliding a smartphone into a Google cardboard device that looks a bit like a View Master (a vintage stereoscopic viewing toy), or VR can be quite immersive, with a 360° headset, in-sync spatial audio, and controllers and sensors for both hands and feet. People in the industry are increasingly using the term “XR” to refer to “AR/VR.” A more detailed look at the XR spectrum starts with VR and extends to AR at the other end of the spectrum, with Mixed Reality (MR) existing as a less confusing name for AR.
Micro Electro-Mechanical Systems (MEMS), a semiconductor technology used for creating tiny sensors such as accelerometers, gyroscopes, magnetometers, and more, is in wide use in the XR market. A significant player in the MEMS/sensor industry is InvenSense, now a part of TDK, with a sizeable $368 million (USD) share in the 2016 MEMS market. David Almoslino, Sr. Director Corporate Marketing at TDK InvenSense, has an excellent handle on the XR industry since sensors play a critical role in the outcome of the VR experience. Sensors do much more than sensing at InvenSense, however, and are found in a majority of XR headsets, controllers, and related peripherals. Sensors work in concert to gather and synthesize data in what’s known as sensor fusion. As Almoslino states, “HTC Vive has incorporated InvenSense technology for one-to-one tracking for how the head is moving. At the same time, HTC controllers have our tracking ability with InvenSense Inertial Measurement Units in each controller. All these sensors track and communicate so that when you are physically moving in a game, the Inertial Measurement Unit (IMU) recognizes the inputs and keeps them all together so that you can truly be immersed in a game.” Gaming is just one use for XR, however.
Augmented Reality (AR) is similar to VR but has the additional design burden of a heads-up display and potentially more sensors that feed data directly to the viewer. Many design challenges are shared. Improving the level of visual detail in XR to perfectly emulate reality may require a display that nears the resolution of the human eye, requiring a high density of pixels (≥2160 x 1080) and a frame per second (fps) rate of at least 60 fps. Field of View (FoV) should be at least 110°. High-performance computing is required to render data with a high pixel density and frame rate without adding lag, as the data processing burden is enormous. It is crucial that data from motion sensors (also known as IMUs) in the VR headset and hand controllers line up with the corresponding visual display on the headset. If not, lag ensues.
High-performance computing aside, much of the work in lowering lag resides in sensors. InvenSense is known for very accurate sensors. Real-world sensing translates to analog input that requires filtering, digitization, and additional processing, for which these sophisticated sensors have integrated microprocessors to process and format data before sending it to the main CPU.
Lars Johnsson, InvenSense’s Sr. Director of Product Marketing, explains how InvenSense IMUs reduce lag and ease the developer experience. “The sensors have integrated filtering with adjustable parameters that include bandwidth and noise. When taking the signal from analog to digital, there’s something called a Digital Motion Processor (DMP) that performs post-processing for sensor fusion, which we offer at certain data and sampling rates. Sensing followed by rapid conversion and post-processing happens locally in our sensor so that when it reaches the rendering engine, it is preprocessed. For VR, all developers have to do is say, ‘If the user looks 1° to the left and 10° up, here is what he should see,’ and the correct spot just gets presented to the screen.” In other words, calculating vectors for relational placement of the display in concert with the physical placement of the headset is done for you.
The sum of the parts of XR add up to a very complicated but exquisitely coordinated high-performance sensing and compute platform. Minutiae do not burden developers when using smart sensors that include practical algorithms. Algorithms will vary for sensors in different locations. As Nicolas Sauvage, InvenSense’s Sr. Director of Ecosystem, points out, “Sensors in the headset are less likely to experience the kind of speed and acceleration that hand controllers present. The performance of an IMU in the hand controllers has different performance tradeoffs than an XR headset.” InvenSense is tuned in to the finer details of VR design. Sauvage goes on to explain, “We fine-tune the performance of our chips to take advantage of these different performance trade-offs. Since your head with the headset will never be as fast as your hands, we can smartly adjust trade-offs in the acceleration of the head. Latency for motion sickness is significant here, but may not be as important elsewhere.”
Sensors Combat Motion Sickness
Sensor accuracy plays a very large part in avoiding motion sickness due to lag. There has to be a perfect alignment between where the user is looking and where the VR rendering engine thinks the user is looking. Add the rapid movement of two separate hand controllers and the action that’s integrated into the picture within the VR game, and you have a recipe for disaster without good sensors. Johnsson goes on to say, “With respect to having very low noise and very high-temperature stability, as the electronics quickly warm up, you don’t want signals to drift as they react to a temperature change. We compensate for these types of things, as they affect accuracy and can create lag.” InvenSense sensors are in the Oculus Rift, Microsoft HoloLens, HTC Vive, and numerous other XR products.
Augmented Reality requires an informative overlay onto a display, whether projected inside a Head-Mounted Display (HMD) visor or in a heads-up display in a car. A well-known example of AR/MR is Pokémon Go, which is played on a smartphone. In the game, Pokémon characters are superimposed on a smartphone screen as captured by the camera in various GPS locations. Other uses for AR include training and as a productivity enhancer. One lesser-known benefit of VR is that users are somewhat forced to focus on the content that’s strapped to their head. Unlike with a TV, VR would make it more difficult for users to look at their smartphones during advertising. Training employees with VR as a medium ensures that they cannot do something else while in the training session, for instance. Boeing found that AR, as tested in a manufacturing setting against a control group, increased productivity by 25%. According to the Harvard Business Review, AR improved productivity significantly in a warehouse. “At GE Healthcare a warehouse worker receiving a new picklist order through AR completed the task 46% faster than when using the standard process, which relies on a paper list and item searches on a workstation (view GE’s AR assisted productivity video here). Additional cases from GE and several other firms show an average productivity improvement of 32%.”
A three-axis accelerometer measures movement in three dimensions. Adding other sensors adds additional axes for acuity with more data. In the industry, it’s common to refer to a pressure sensor as an additional axis, for instance, because the sensor measures height in the air based on air pressure, not motion. Fusing the inputs gives more accurate data. A nine-axis sensor would include three degrees of freedom each from an accelerometer, a gyroscope, and magnetometer. Software algorithms complement sensor fusion. One prominent example in navigational mapping uses GPS as well as six-, seven-, or nine-axis IMUs that continuously measure orientation changes and speed. These IMUs keep travelers on track when GPS fades. For example, navigating a tunnel with a highly accurate IMU will accurately track a car’s progress without GPS, since error accumulates on a minuscule level when you have high levels of accuracy. The InvenSense Positioning Library (IPL) algorithms can implement tracking to complement navigation when GPS goes missing in an urban canyon. Other use cases include wearables that may incorporate power-hungry GPS only intermittently, preserving battery power while keeping true to course. Have you ever wondered why a smartphone application for in-store coupons would need permissions for accessing your gyroscope/accelerometer? Retail use cases include smartphone applications that can accurately track and monitor a person’s travel inside a store using a highly accurate six-axis IMU. Tracking can easily include how long a person stays in a particular location. With a standard store layout and accurate position tracking triggered by a single Bluetooth beacon as users enter the store and open their coupon apps, data can reveal information on shoppers. A shopper might get a pop-up offer on their smartphone app for a discount on Snuggies after walking away from a Snuggies display where they lingered a little too long. This accurate tracking is done without expensive video cameras. It’s easy to see that extremely accurate sensors are affecting more than VR.
Where is VR Headed?
The global VR Head-Mounted Display market is projected to increase to around 90 million units per year by 2021. VR has some challenges with fragmentation in platforms for developers. VR content is challenging across a fragmented landscape of various platforms with varying numbers of controllers and no one unifying standard, at least not one that’s been widely adopted yet, similar to how USB solved the problem for connectors. VR systems can come with up to two controllers, affecting the application of gameplay with each choice. Pricing puts the best VR systems out of reach for much of the existing gaming market. A lack of good content comes with the territory for a VR market fragmented by many platforms, making it that much more difficult for developers to create content that sells in volume across many platforms. These challenges are being solved, as many see XR as a wondrous experience and a productivity boon for a manufacturing sector with rising job openings and falling hiring rates.
The highly accurate sensors used in XR translate well to several other segments. As for InvenSense, the TDK purchase was a good thing. Almoslino’s perspective is seasoned by years of experience in sensors, where InvenSense excels. “InvenSense has had sensor success in the consumer products area and TDK in industrial, and together they complement each other. The automotive sector is going to be our next big growth scene.”
AR is already making significant headway into increasing productivity of workers in warehouse “pick and pack” activities. Transportation industries that include trains, busses, and automobiles will benefit from previously unaffordable, augmented heads-up displays (HUDs) that a decade ago were only available in sectors big budgets and critical importance, such as in military cockpits. It is without doubt that XR will have a significant impact on economies world-wide by increasing productivity and decreasing accidents.
Lynnette Reese is Editor-in-Chief, Embedded Intel Solutions and Embedded Systems Engineering, and has been working in various roles as an electrical engineer for over two decades. She is interested in open source software and hardware, the maker movement, and in increasing the number of women working in STEM so she has a greater chance of talking about something other than football at the water cooler.
[i] Mason, Betsy. “Virtual Reality Has a Motion Sickness Problem.” Science News, 8 Mar. 2017, www.sciencenews.org/article/virtual-reality-has-motion-sickness-problem.