Technology

Augmented Reality – basic overview about AR

Since we are working or have been working on several Augmented Reality projects, we decided to share some of our experiences with this technology. Augmented Reality is the view of a real-world environment, either directly or through a camera, which is augmented by digital information. In the current market, it is usually narrowed down to a mobile device or VR headset camera with additional content rendered on top. This article is going to cover different types of Augmented Reality, some of the different toolkits and the benefits and drawbacks of working with them on different platforms.

Most common types of AR

Image marker tracking: The most common method of AR uses 2D printed images as tracking targets. The AR software analyzes tracking points of the image by looking at sharp corners with high contrast, however not at color. Therefore tracking images must be asymmetrical and ideally feature many high contrast edges and corners. When it then finds those same tracking points via the device camera, any kind of graphics, animation or 3D model can then be applied on top of it. Most development tools also allow for multiple image targets at the same time, which allows for letting them interact with each other in the application. This AR method is best used to complement printed products like trading cards, booklets, books or business cards. Some development tools provide an option for extended tracking, which allows for creation of additional tracking anchors in the environment, so the original target doesn’t get lost even if the image marker is out of view.

3D object tracking: Some of the professional AR SDKs (software development kits) also offer 3D recognition of real world objects. It allows for tracking of a real world object and apply skins, graphic overlays or animations on top of it. It can be useful for product displays where only the base model is available at a store or exposition, but the app complements it by overlaying all the different variations and colors or showcase all the different features via animation. Tracking and stability work pretty well even on older devices (Galaxy S6, iPhone 6), but there are still things which need to be considered. Most importantly, the target object must be accurate to the real object or else the recognition might fail completely. All features of the 3D model target must also be located correctly on the real object, however small features or variations on the real object can be missing on the 3D model without causing problems. It is also important, that the object to track isn’t too simple. Simple objects will still track, but will be less accurate, meaning the overlay will float a bit or will not be correctly positioned. The main drawback to 3D object tracking is its prize. There are only a few SDKs out there, which support it and all of them require an expensive license per year and per application for this feature (few to several thousand $ per year per app).

Vuforias Model Targets demo

SLAM (simultaneous localization and mapping): The newest hit in Augmented Reality is SLAM based tracking. It started as extended tracking with the support of tracking images, but now works completely without any tracking targets. While not very reliable at the early stages before 2017, the almost simultaneous releases of new AR features in 2017/2018 from several companies in combination with the newest smart devices allow for very robust environment surface detection. Another great thing about SLAM AR is, that many SDKs to develop apps with it are offered for free. This technology can be used for a wide variety of applications like furnishing apps, home planers, games or presentation tools.

However, SLAM needs more than a camera to work. As with traditional marker tracking, the camera creates anchors around the environment on high contrast corner points. In addition it also uses the devices gyroscope, accelerometer and magnetometer to determine changes in rotation, acceleration and position. Only the newest devices are capable of taking advantage of all the sensors, which is why SLAM works either poorly or not at all on anything but the last and newest generation of phones and tablets. Google tried to took it even a step further with Project Tango, which needed special mobile devices which came with a fisheye-lensed camera in addition with infrared lights to create highly accurate environment tracking without the need for external hardware. However the project was recently shut down due to low demand and replaced with Googles recent AR technology for regular Android smart phones, AR Core.

Mixed Reality is the combination of virtual and Augmented Reality. This can be achieved by either using the camera of smartphone VR (Cardboard or GearVR) or the HTC Vive, or by use of the Microsoft Hololens, however the latter is still quite expensive and have only recently become more available. While there are mixed reality and/or smart glass solutions in development by other companies, none of them seem to be available yet. This means other than for development, using VR with regular phone / device cameras is the only viable option for smaller projects at the moment. This has some serious drawbacks. Most importantly the device camera only offers low resolution because of the tracking software and only has a small field of view, meaning you are only overlaying your digital content over a window in front of you, rather than being fully immersed in the virtual view. On top of that, running two different device SDKs is performance heavy for smartphones, which causes the noticeable camera delay to be even more prominent due to lower frames per second. Affordable mixed reality will definitely be market ready at some time in the near future, but in our opinion it is only viable for very special applications or professional enterprise solutions in its current state.

Location based AR: Most commonly known from the Pokemon GO game, location based AR overlays content based on the devices location provided by GPS. Apart from games, tourist landmarks or mapping information could also be displayed on the device. It usually comes paired together with gyroscopic tracking, since location tracking alone would mean static content, which makes the whole point of using AR obsolete. Since the location service is part of the native platform SDK, no additional AR SDKs are necessarily required, but there are SDKs which have  solutions that are ready to implement immediately.

Gyroscopic AR: Can be used for very simple apps on low-end devices, where neither positional nor target tracking are required. Otherwise not recommended, especially since SLAM does the same with adding a lot more to it. As with location based AR, it also requires no additional SDKs.

There are also other types of AR like facial tracking or shape detection, which won’t be covered in this post, as we have not worked with them yet.

Android vs iOS

Most SDKs allow for testing in the editor on PC or Mac, which accelerates development and testing by a lot, but the main platforms for AR currently are Android and iOS. Windows phones are supported by some SDKs, but their market share is still quite low compared to the other two which translates to a much smaller audience. In addition both Google and Apple have their own platform exclusive AR tools, where Microsoft focuses on Hololens in the AR market.

From all the testing we have done on a variety of devices, both platforms seem to do equally well when it comes to tracking detection and stability. The differences lie mostly on the range of supported devices, ease of development and costs.

Differences for Android:

• AR Core is free but officially only supports a small number of devices, without any supported tablets (Other SDKs support Android tablets)
• Free deployment to specific devices via side-loading (installing .apk files directly via USB).
• Cheaper deployment on the Google Play Store
• Easier to build, especially on Windows
• Larger audience, as Android is the most used mobile platform
• A very large amount of devices, which makes it more difficult to balance performance and test

Differences for iOS:

• ARKit is also free and supports all of the newest devices, as well as some medium ranged ones, also supports iPads
• More expensive deployment in iTunes store
• Side-loading only lasts for 7 days
• Private deployment to specific client devices require enterprise account (currently 300$/year)
• iOS Updates often cause difficulties for third party tools (Unity3D, Vuforia, etc.), which can result in development delays

Augmented Reality SDKs comparison

Currently, there are a lot of different AR SDKs out there, many specialize on a specific type or AR, some are free, some are expensive. We will cover only the most common ones for the AR types mentioned above.

Easy AR and AR Toolkit – both of them are free and offer basic image marker tracking and they also offer Unity3D integration. However we can’t recommend using them for anything professional, as both detection quality and stability are far inferior to professional SDKs. They are easy to use, so you can definitely play around for private projects or use them for small professional projects, but only if tracking quality is not a priority.

Apple ARKit – Apple acquired the Augmented Reality company Metaio in 2015 which, at that time, was the main competitor to Vuforia. This lead to the discontinuation of the Metaio AR SDK and the creation of ARKit. ARKit is offered for free and works on semi-modern to modern iOS devices, both phones and tablets. It offers only SLAM AR tracking, with the ability to detect horizontal and since the recent update also vertical planes in the environment. This allows for a variety of creative applications by placing any kind of content on floors, ceilings or walls. Originally only offered by Googles ARCore, ARKit now also offers an estimate of the real world light direction, which can be used to fit the lighting of augmented objects more closely to the real environment. The SDK comes for native development, as well as a Unity3D and Unreal integration.

Google AR Core – After project Tango has been terminated, Google announced AR Core as an answer to Apples ARKit. As ARKit only works for iOS devices, AR Core only works for Android devices. The main difference is, that AR Core supports only few and very modern phones, without official tablet support yet. Apart from that, it works pretty much the same, SLAM tracking, detection of horizontal and vertical planes, light source estimation and native, Unity3D as well as Unreal SDK.

Vuforia – The most known and popular AR SDK out there and also one of the, if not the, best SDK for Image marker and 3d target tracking. Unlike ARKit or ARCore, Vuforia offers a variety of AR types, with most of them having some form of license restrictions. Vuforia offers development licenses for free, which will add a watermark to the application. The only exception to this is Ground Plane, which is Vuforias free SLAM tracking. Cloud services are also available for 99$ per month. It is also directly implemented into the Unity3D Editor, which makes it easier if you are just starting with AR development.

1. Ground Plane – SLAM with horizontal and vertical plane detection – no license required – no watermark
2. Image Marker tracking – multiple image targets can be tracked simultaneously – image targets can be flat, on cubes or cylinders – 500$ per app – depending on the project, an enterprise license may be required (prize depends on the project, but in the range of several thousand $ per year per app)
3. 3D scan tracking – smaller object can be scanned with Vuforias object scanner app – 500$ per app – depending on the project, an enterprise license may be required
4. 3D model target – convert 3D objects into tracking targets – enterprise license required – additional VIP subscription account required to create model targets (10 are free)

Vuforia supports Android, iOS, UWP and Unity3D Editor
See Vuforia for more information

Wikitude – Supports a large variety of platforms and development frameworks. It also offers free development and has multiple prizing tiers.

1. Geo / location based – included in SDK Pro (2490€/year/app)
2. 2D image recognition – multiple images can be tracked simultaneously – also included in SDK Pro
3. SLAM – similar to ARKit and ARCore but not free – requires SDK Pro 3D (2990€/year/app) – main difference to the competing free SLAM SDKs are cross-platform support, digital eyewear support and that in general a lot more Android and iOS devices are supported, not just modern ones.
4. 3D object tracking – also included in SDK Pro 3D (2990/year/app)

Wikitude also offers Cloud recognition for 4490€/year/app. The final tier is the enterprise tier which is required for smart glass deployment and enterprise apps. Prize needs to be inquired.

Supported platforms: Android, iOS, tablets, smart glasses
Supported development frameworks: Unity3D, Xamarin, Titanium, Cordova
See Wikitude for more information.

Kudan – currently the main competitor to Vuforia, Kudan also offers a variety of AR types and comes with different license tiers, which, unlike other SDKs, are not limited to included features. On top of offering free development, they also offer a free AR Indie license, which allows you to publish applications with a watermark. Their production license costs 1k £/ year / app but usually covers everything for small to medium sized businesses. An Enterprise license is required, if your or your clients revenue capacity lies above 1 million £ / year. An official Unity3D plugin is also available.

1. 2D image marker tracking – multiple images can be tracked simultaneously
2. 3D target tracking – small and large objects are trackable
3. SLAM – markerless tracking. similar to competition

Supported platforms: Android, iOS
See Kudan for more information.

This concludes our basic AR overview. While this information is up to date now, the state of Augmented Reality will probably change very quickly as it is constantly receiving new advancements in both software and hardware. A lot of the things we have written about were just released up to a few months ago.

We hope to see you soon on our next post, which will continue with Jumpy Zergi production log #2.