Augmented Reality on Android

Augmented Reality on Android

What is Augmented Reality (AR)? Virtual 2D and 3D content superimposed onto a live image

Real World View Live camera view Virtual Content Information about points of interest Good for rough location of items outside our field of view GPS and Compassbased Augmented Reality Experience Limited to outdoor experiences

Real World View Live camera view Virtual Content Vision-based AR recognizes objects in field of view and aligns graphics tightly to target object Vision-based Augmented Reality Experience Enables a more immersive and interactive experience

Pull Video Frame from Camera, Scan for Known Objects or Features

Compare to Database of Known Images

Position and Orientation Determined X Y Z

Graphics are Rendered

SCANS COMPARES 30+ POSITIONS times per second RENDERS

OYXGEN HYDROGEN HYDROGEN

OYXGEN OYXGEN HYDROGEN HYDROGEN

Applications Today Gaming & Play 3D games in real world environments vs. virtual worlds Media / Advertising Making print, outdoor, TV media, and product packaging come alive Instructional Visualizing instructions e.g. furniture assembly In the future Turn Left on Main St. Text Translate words to multiple languages Visual Search Retrieving related information from web for the object in view Navigation & Discovery Visualizing points of interest

AR is gaining momentum According to Visiongain, in 2012, 25% of mobile apps will feature augmented reality. The firm forecasts the mobile AR industry will see $3 billion in global revenue by 2016, up from $87 million (in 2011) and $21 million in 2010. Consumer engagement is the main reason that (augmented reality) technology has been so successful. QR codes have proven to be effective engagement tools, which has lead many retailers, such as Macy s, to incorporate them into their marketing campaigns 16

Brands are using AR to engage consumers 17

AR deepens consumer interactions with brands Point of Advertising campaign has added AR element across media channels Point of Sale product comes alive on the shelves Point of Use added AR dimension when the product comes home 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 18

Vuforia Qualcomm s AR Platform 19 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform

Vuforia Platform Traction 21,000+ Registered Developers from 130 Countries 400+ Models of Smartphones and Tablets Supported 400+ Apps in Android Market and ios App Store

Accolades, Testimonials, and Honors RECOGNITION FOR OUR AWARD WINNING AR TECHNOLOGY Qualcomm s AR SDK has made it extremely easy for us to prototype, design, and develop our ideas and concepts. Morgan Jaffit, Co-Founder, Defiant Development your AR library is by far the best AR Library I ve ever used. It s amazing, works incredibly fast, and really easy to work with. Awesome job QCAR team!! Nicholas Rudolfsky, ios Developer, Smule When it comes to mobile Augmented Reality technology, Qualcomm is the top dog. Pocket-lint The opportunities to do amazing things which we quite literally couldn t live without are unparalleled with augmented reality. This is Qualcomm s secret weapon, and it may assure the company thrives, rather than simply survives the coming conflict. Rob Enderle, Analyst, TG Daily 2011 Future Mobile Entertainment Award for Mobile Augmented Reality

100s of articles in mainstream technology and business press

Near Field Usage Environments Table or Floor Wall Retail Shelf

Augmentable Objects Representations of real world elements that can be detected and tracked Image Target Frame Markers Simple 3D Objects Game boards Product packaging Posters Signs Greeting cards Business cards Books / magazine pages Game pieces Control cards Boxes Bottles Cans

Enables multiple cards to simultaneously trigger AR experiences Frame Markers

Allows users to press buttons by touching certain areas on an image or object Virtual Buttons

Features and Performance Vision Ordinary Images Simple 3D Images Marker Virtual Buttons Performance Simultaneous Object Recognition Optimized for Mobile Flexibility Unity Eclipse Xcode 3rd Party 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 27

Android Development Options Download Qualcomm Vuforia SDKs http://developer.qualcomm.com/ar Eclipse Unity 3 Advantages Free Lowlevel APIs provide enhanced flexibility and performance Fully integrated game engine reduces development time and cost Single app supports both ios and Android Availability Now Now

29 VuforiaAR Applications Structure

30 Example

Target Management

Creating Image Targets

Targets are Trackable DataSets

Camera start

Camera stop

Pixel Format Conversion The pixel format converter converts between the camera format (e.g. YUV12) to a format suitable for OpenGL ES rendering (e.g. RGB565) and for tracking (e.g. luminance)

Tracker The tracker detects and track real world objects in camera video frames. Different algorithms take care of detecting new targets or markers, and evaluating virtual buttons. The tracker can load multiple datasets, but only one can be active at a time The results are stored in a state object that is used by the video background renderer and can be accessed from application code.

State Object Structure Dataset Targets Teapot on target A 3D Assets Target A Target B

The State Object Member Description Camera Frame Current image Type of Data RGB YUV GRAYSCALE Trackables Events List of active trackable objects Planar Image Frame Marker Multi Target List of events Virtual Button1 Virtual Button 2, etc. Name 4x4 Matrix Name Button Press

Video Background Renderer The video background renderer renders the camera image stored in the state object. The performance of the background video rendering is optimized for specific devices.

Application Code Query the state object for newly detected targets, markers or updated states of these elements Update the application logic with the new input data Render the augmented graphics overlay 10/18/2012

Application States oncreate() loadtrackerdata() deinitapplication() loadtexture() InitApplication() onresume() Init Tracker() initapplicationar() initapplication() StartCamera() initrenddering() updaterrendering() onpause() StopCamera() ondestroy() destroytrackerdata() deinittracker() Native Java Native C/C++

Creating an Android AR App Java Native C/C++ Initialize SDK Start Trackers Handle Tracker Updates Close SDK myapp extends Activity {... void oncreate() { QCAR.init(..); }... void onresume() {... setupopenglviews(); mynativecamerastartup(); mynativetrackerstartup(); }... GLRenderer implements GLSurfaceView.renderer {... ondrawframe(..) { mynativerenderer() } } void ondestroy() { QCAR.deinit(); } } mynativecamerastartup () {... QCAR::CameraDevice.init(..); QCAR::CameraDevice.start(..); } mynativetrackerstartup {... } QCAR::Tracker.setActive (QCAR::PLANAR_IMAGE); mynativerenderer {... // Render video background // Update app logic using State Object // Render app assets }

QCAR::ImageTracker

Frame Markers

QCAR::MarkerTracker

Virtual Buttons

Buttons coordinates <?xml version="1.0" encoding="utf-8"?> <QCARConfig xmlns:xsi="http://www.w3.org/2001/xmlschemainstance" xsi:nonamespaceschemalocation="qcar_config.xsd"> <Tracking> <ImageTargetsize="247 173" name="wood"> <VirtualButton name="red" rectangle="-108.68-53.52-75.75-65.87" enabled="true" /> <VirtualButton name="blue" rectangle="-45.28-53.52-12.35-65.87" enabled="true" /> <VirtualButton name="yellow" rectangle="14.82-53.52 47.75-65.87" enabled="true" /> <VirtualButton name="green" rectangle="76.57-53.52 109.50-65.87" enabled="true" /> </ImageTarget> </Tracking> </QCARConfig>

Simple 3D Objects Enables products to trigger AR experiences, and interactive content to reflect geometry of product packaging Boxes Bottles Cans

Creating Simple 3D targets

Multi Image Targets

Swappable Datasets

Swappable Datasets contd.. The script exposes a list of datasets (that are part of the project) to be loaded at scene startup. A single script can be used across different scenes Mainly used to load/unload and activate/deactivate datasets at runtime. Can ONLY be used at scene startup but doesn t do anything when its properties are changed later on in the application lifecycle.

Dataset Script

Background Texture Access

Background Texture Access contd.. Video background access allows you to use the camera video as a texture through the use of shaders Two shadersare used vertex shader---converts video into grayscale and inverts the black and white fragment shader---listens for a touch on the screen ad then distort the video around that touch.

Occlusion Management

Occlusion Management contd.. Illustrates how shaders(four in this case) can be used to make objects appear semi-transparent. Pointing your camera at the FlakesBox target allows you see the teapot object inside the partially transparent looking box. Four shaders, one per object, are rendered in correct order (the video background, the checkerboard-box, the teapot inside and the occlusion box).

Image overlay Images to be replaced Change code in teapot.h

Cube Structure -1.00f, -1.00f, 1.00f, // front 1.00f, -1.00f, 1.00f, 1.00f, 1.00f, 1.00f, -1.00f, 1.00f, 1.00f, -1.00f, -1.00f, -1.00f, // back 1.00f, -1.00f, -1.00f, 1.00f, 1.00f, -1.00f, -1.00f, 1.00f, -1.00f, -1.00f, -1.00f, -1.00f, // left -1.00f, -1.00f, 1.00f, -1.00f, 1.00f, 1.00f, -1.00f, 1.00f, -1.00f, 1.00f, -1.00f, -1.00f, // right 1.00f, -1.00f, 1.00f, 1.00f, 1.00f, 1.00f, 1.00f, 1.00f, -1.00f, -1.00f, 1.00f, 1.00f, // top 1.00f, 1.00f, 1.00f, 1.00f, 1.00f, -1.00f, -1.00f, 1.00f, -1.00f, -1.00f, -1.00f, 1.00f, // bottom 1.00f, -1.00f, 1.00f, 1.00f, -1.00f, -1.00f, -1.00f, -1.00f, -1.00f Set of four vertices defining a plane One vertex/corner of a plane defined by x,yand z x Left Back z y Top Right Bottom Front

Image overlay contd.. z y Step 1: Replace values of teapot.hwith cube.h(find cube.hin dominos app ) Step2:Remove all coordinates expect Top. Step3:Set the zero vale of z coordinate of Top. Step4:Replace the images in assets folder with required Image. Top x Left Back Top Right Bottom Top Front 10/18/2012 61

Video Playback

Video Playback Play video on an image or texture on compatible devices Gracefully fall back to an alternate videoviewing experience on incompatible devices (android video play triggered on target)

Initialize SDK Start Trackers Handle Tracker Updates Close SDK URL Example Java ondrawframe(..) { public static Handler mainactivityhandler; public void displaymessage(string text) { Message msg = new Message(); msg.obj = text; mainactivityhandler.sendmessage(msg); protected } } void onresume() { super.onresume(); ImageTargetsRenderer.mainActivityHa ndler = new Handler() { @Override public void handlemessage(message msg) { Intent intent = new Intent(Intent.ACTION_VIEW); intent.setdata(uri.parse("http://ww w.qualcomm.com/partials/video/34133 ")); startactivity(intent); } Native C/C++ renderframe(){ for(int tidx = 0; tidx < state.getnumactivetrackables(); tidx++) { jstring js = env>newstringutf(trackable>getname ()); jclass javaclass = env->getobjectclass(obj); jmethodid method = env->getmethodid(javaclass, displaymessage", "(Ljava/lang/String;)V"); env->callobjectmethod(obj, method, js); }

Unity3d I. Installing the Extension into your unity project II. Compiling a simple AR app, and III. Mastering some of the more advanced AR topics. 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 65

Installation on Windows I. Download installer EXE-file, from the download page. II. Run installer III. Select a location for the package installation convenient to your development environment. The extension-only package will also be copied to the Standard Packages folder of your Unity installation 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 66

Compiling a Simple Project I. Create Project II. AR Assets and Prefabs to Scene III. 3D Objects to Scene and Attach to Trackables 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 67

Create Project Step 1 Step 2 Step 3 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 68

I. Step 1: Click On File-> New Project II. Step 2: Select Project path and name III. Step 3: Select Project package name 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 69

Structure inside Unity 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 70

Editor Contains the scripts required to dynamically interact with Trackabledata in the Unity editor. Plugins Contains Java and native binaries that integrate the QCAR SDK with the Unity Android application. Qualcomm Augmented Reality Contains the prefabs and scripts required to bring augmented reality to your Unity application. Streaming Assets Contains the swappable target datasets downloaded from the online Target Management System. 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 71

AR Assets and Prefabs to Scene ARCameraprefab:TheARCamerais responsible for rendering the camera image in the background and manipulating scene objects to react to tracking data ImageTargetprefab :This prefab represents a single Image Target Trackable object 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 72

Delete Main Camera 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 73

Adding Prefabs I. Delete main camera from hierarchy II. Add ARCamera and ImageTarget in hierachy from project window 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 74

ARCamera prefab 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 75

ImageTarget prefab 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 76

3D Objects to Scene and Attach to Trackables 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 77

I. Cube object (GameObject> Create Other > Cube) II. Lighting (GameObject> Create Other > Directional Light) 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 78

Directional Light 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 79

Animation 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 80

I. Select Object from hierarchy window II. In menu bar select( Windows-> Animation) 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 81

Animation Window 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 82

Step 4 Step 1 Step 2 Step 3 Step 5 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 83

I. Step 1:Press the recording button and save animation. II. Step 2:Press it for creating break points in animation. III. Step 3:Drage time line according to required length. IV. Step 4:Allows required type of movement (circular/hor/veretc) V. Step 5:It give options about to play(once/loop etc) 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 84

Android Deployment Process Platform Bundle Identifier and API Level Player Settings 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 85

Android Deployment Process Unity provides a number of settings when building for Android devices you will need to select from the menu (File > Build Settings > Player Settings ) to see the current settings. Also, choose your platform now Android or ios. Click onresolution and Presentationto select the required Default Orientation. Note: the Vuforia AR Extension now supports Auto Rotation. Click on Icon to set your application icon. Click onother Settings. Set theminimum API LeveltoAndroid 2.2 'Froyo' (API level 8)or higher. SetBundle Identifierto a valid name (e.g. com.mycompany.firstarapp). Next save your scene (File > Save Scene). Then open the build menu (File > Build Settings ). Make sure that your scene is part ofscenes in Build. If this is not the case either useadd Currentto add the currently active scene or drag and drop your saved AR scene from the project view into the Window. 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 86

Scripting Scripts Script, Data Set and Image Target ImageTargetB ehaviour 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 87

Virtual Button Case Statements

Loading Default Object w.r.ttrackable

Physics Engine Physics Properties 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 90

Physics Engine contd.. I. Rigidbodies: These are physically simulated objects.userigidbodiesfor things that the player can push around, eg. crates or loose objects, or move Rigidbodiesaround directly by adding forces to it by scripting. II. Character Controllers: use to make a humanoid character III. Collider: GameObjectthat has a Collider but not a Rigidbody. Static Colliders are used for level geometry which always stays at the same place and never moves around Box Collider: primitive shape of a cube Sphere Collider: primitive shape of a sphere Capsule Collider: primitive shape of a capsule Mesh Collider : creates a collider from the object's mesh, cannot collide with another Mesh Collider Wheel Collider : specifically for creating cars or other moving vehicles 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 91

VirtualButtonsExample Scripts 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 92

VirtualButtons Example contd.. Buttons 3D Object 93 Qualcomm Vuforia Vuforia Augmented Reality Platform 10/18/2012

Creating and deleting Virtual Buttons at run-time I. create a new Virtual Button for a given Image Target at run-time by calling the CreateVirtualButtonmember function on the corresponding instance of your ImageTargetBehaviour. II. destroy a Virtual Button by callingdestroyvirtualbutton, again defined in ImageTargetBehaviour 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 94

SoccerballExample Scripts Physics Engine 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 95

10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 96

Thank you! Regional Ecosystem Strategy Brigitte Alexander 1/17/12