Digital and Physical Worlds Collide

Sensor-based augmented reality and contextual awareness are already moving beyond the “cool” factor to drive new, high-utility applications in a wide range of applications.

Augmented reality (AR) uses sensor data and embedded vision and graphics processing to synchronize digital information with the physical world in real time. The resulting applications enhance users’ experience of their environment, and enable capabilities that will change the way we interface with and use devices. While AR’s killer app is yet to burst on the scene, we’re already seeing innovative new applications for consumers, healthcare, transportation, industry and defense – and it’s still early yet. But AR presents significant challenges as well as opportunities for developers. We talked to Stephane Gervais-Ducouret, global marketing for sensors at Freescale and Dirk Groten, CTO at Layar to give you an inside look at these exciting new technologies.

EECatalog: Augmented reality has an unquestionable cool factor in consumer mobile applications. What are some of the most interesting new applications for AR and contextual awareness in specialized vertical markets, such as medical, transportation, or defense applications?


Stephane Gervais-Ducouret, Freescale: Defense, and more specifically military aerospace, has successfully used AR. It is now indispensible: the need for higher speed, increasing flow of data and the necessity for vital decisions at the right time has pushed the technology to add virtual layers for heads-up displays (HUDs). Even if the biggest buzz is currently around consumer, with applications for smartphones and tablets, the main market that will benefit from AR technology first will be automotive and industrial. This new technology offers more information to the driver, such as driving directions, traffic updates or even an update of yournext meeting, without distracting the driver while driving on the road. With phones being prohibited while driving in some countries, the nascent AR technology will be adopted with specific content displayed on the screen in synchronization with the context (more content updates during a traffic jam, but minimum content at higher speeds). AR can also enable social networking and conducting office tasks in a car. Industrial markets have applications ranging from learning tools (how to assemble or fix an engine), to support operations (displaying information for each machine without the need of a dedicated screen) or even product support for business-to-business (recognizing the product and superimposing virtual layers to show how to use a product).


Dirk Groten, Layar: The cool factor in consumer applications is very short-lived. Once you’ve seen it and used it for a few minutes, the coolness edges off and most people will be left thinking “now what?” At Layar, we believe AR will become useful, not as a goal but as a means. And we focus on the consumer market. For example, by making print interactive – not because it’s cool, but because it’s so easy. You look at a page of a magazine that has an interesting film review and think, I’d like to see the trailer of that film. Just point your phone at the page and see the movie trailer appear right there. It’s time the industry takes AR out of the “cool” and into the “utility.”

We don’t focus on the verticals mentioned above as Layar so I believe other companies are more suited to answer the specific question. However, the Dutch Defense has been investigating use of AR (using Layar) in case of big calamities during popular events with a large public involved (e.g., open-air concerts, festivals, sports events or exhibitions). They use AR to show the emergency services all relevant locations, including live information about vehicles and team locations (ambulances, fire trucks, crowd control troops, etc.) and at the same created a public version that shows the event information and – in case of emergency – just the emergency exits and arrows for people to follow in case of evacuation.

EECatalog: What are some of the hardware and software challenges for developers of AR systems?

Gervais-Ducouret, Freescale: Adding virtual layers using markers or location can already be done by using SDKs from companies like Layar. However, developing immersive virtual reality experiences with intuitive access to the information is still challenging. Another challenge is to make sure that the application which has been developed will run smoothly and efficiently on affordable and low-power devices. Hence, usage of processors should be minimized to the benefit of hardware acceleration such as GPUs. Indeed general purpose GPUs are the best hardware solution to address the performance requirements in image recognition, embedded vision and graphic interfaces. OpenGL, OpenCV and OpenCL are some efficient tools; however, there is no framework to encompass these tools to interface them easily with Java and user interface sensors, for instance.

Using information from sensors to gain contextual information is still in its infancy for developing meaningful applications since it is still not easy to get access from aggregated sensor data.

Groten, Layar: The biggest challenge is to make it so easy to use that everyone (consumer) immediately understands it. That can only happen if the response is near instantaneous so users don’t have to wait to see what will happen. For vision-based AR (recognizing and tracking what’s in the camera view), this requires high processing power and good graphical capabilities. Only this year are we seeing devices (smartphones and tablets) on the market that start to meet the optimal requirements.

For geo-based AR (knowing where the user is and what direction he’s looking at), the problem is still the precision: GPS position and compass direction are still quite inaccurate to convey a real AR experience and probably only vision-aided AR, where buildings and other parts of the public space are recognized and tracked, will give the real experience.

In both cases you need a network connection to send video information to be recognized from a large database (as you can’t possibly cache the entire world locally in the phone) or pre-fetch local information. That network (wireless) connection is always going to be a bottleneck in getting a smooth experience.

EECatalog: How are developers addressing security and privacy issues related to AR?

Gervais-Ducouret, Freescale: If we cannot address effectively the privacy and security issues, AR adoption will suffer drastically. Indeed, the developers must provide an easy way for the user to control the information he is sharing. A common interface should become a standard for the industry to avoid confusion and to make sure that privacy is regarded with the same importance by the device manufacturer and developer. Security is crucial for automotive and industrial markets. For instance, the driver of the vehicle should be recognized before displaying information which can be disturbing for some people and necessary for others. But I do not believe that we should put in place a specific driving license to handle AR while driving.

Groten, Layar: I don’t think AR has any other security or privacy issues than any other application. There is some unchartered (legal) territory in AR, for example about who owns the space on which augmented reality is added. For example, would it be acceptable if people start adding “I love Pepsi” virtual stickers onto every Coca-Cola can or ad? (This can be done today with an application like Stiktu, that lets people add AR to any image just using the mobile phone.) We strongly believe that public space belongs to the public and AR gives people the possibility to reclaim the space (albeit virtually) the way they want it.

But if there’s a social component to an AR application, the same privacy and security concerns apply as to any social application (Facebook) and if there’s a location-based component to an AR application, the same privacy concerns apply as to any location-based application (Foursquare).

EECatalog: How will AR applications and sensor technologies drive new user interfaces?

Gervais-Ducouret, Freescale: AR technology success will depend on how developers embrace sensors for context awareness and user interfaces. This is already the case in some applications since sensors such as accelerometers, magnetometers and gyroscopes are used to match the position of the mobile device with its environment to virtually add adequate information. With the number of recognized objects increasing, accurate pointing using sensor fusion is becoming necessary. However, much more should be done to provide the user a long-term adoption of AR applications; for instance, the localization must be ready prior to any request from the user, hence, only the combination of inertial sensors with WiFi or GPS can provide the low power of continuous position tracking. Another improvement is pattern recognition of accelerometer data, which can provide information about what the user is intending, thus, starting the right application and preparing the data (avoiding the user to having to look at menus). Accelerometers and gyroscopes should also be used to improve camera tracking with recognized objects and efficiently add virtual information in a seamless way. Conducting gesture recognition is an example.

Groten, Layar: That’s an interesting question. Because right now, AR is mostly used on smartphones and that is an entire different user interface than if it were to be used with AR glasses. With smartphones, I think we will see more and more the “point to view” paradigm, where users can just point their phone at something to view more content. With glasses the challenge is of course the user control. Voice-control like Google showed with the Google Glass project is the most obvious option, but in general voice control still is in its infancy (even though Google Voice and Apple’s Siri show some progress) and only covers a small part of the richness of current touch-based user interfaces.

EECatalog: What advances need to occur in today’s available technologies for AR to reach critical mass?

Gervais-Ducouret, Freescale: AR’s key value is the intuitive display of information that matches our environment. Hence, AR needs to select the appropriate information and display it with virtual layers using our location and our context. One of the first technology improvements is to address the localization, and especially in-door positioning, by mastering the aggregation of multiple technologies: GPS, cell ID, WiFi positioning and sensor fusion. Secondly, context awareness technology should greatly involve various sensors to guess what the user wishes to do by understanding who, what and when. This is surely one of the major fields of innovation. Thirdly, applications and SDKs should heavily use hardware-accelerated software running on GPUs to allow fast image processing and great graphic display for immersive virtual reality experiences. Lastly, the user interface must become fully intuitive by using sensor fusion for gesture recognition, navigation and accurate pointing to “add sensing to our vision.”

Groten, Layar: Not much: AR needs to have an obvious utility to the user. It needs to enrich his life. Otherwise it’ll remain a gimmick, a cool game that you play for an hour or a one-time “wow” experience. It doesn’t have to look fancy or be immersive; it just needs to make sense. For example, with Stiktu, anyone can annotate any real object or image and share it with friends. Anyone else pointing their phone at the same object will see the annotation and be able to annotate it as well, creating a conversation on top of things. Or with Layar, any magazine, newspaper or billboard creator can add a digital layer on top of their print creating interaction with the reader.



Cheryl Berglund Coupé is editor of Her articles have appeared in EE Times, Electronic Business, Microsoft Embedded Review and Windows Developer’s Journal and she has developed presentations for the Embedded Systems Conference and ICSPAT. She has held a variety of production, technical marketing and writing positions within technology companies and agencies in the Northwest.

Share and Enjoy:
  • Digg
  • Sphinn
  • Facebook
  • Mixx
  • Google
  • TwitThis
Extension Media websites place cookies on your device to give you the best user experience. By using our websites, you agree to placement of these cookies and to our Privacy Policy. Please click here to accept.