Augmented Reality on Android

Transcription

1 Augmented Reality on Android

2 What is Augmented Reality (AR)? Virtual 2D and 3D content superimposed onto a live image

3 Real World View Live camera view Virtual Content Information about points of interest Good for rough location of items outside our field of view GPS and Compassbased Augmented Reality Experience Limited to outdoor experiences

4 Real World View Live camera view Virtual Content Vision-based AR recognizes objects in field of view and aligns graphics tightly to target object Vision-based Augmented Reality Experience Enables a more immersive and interactive experience

5 Pull Video Frame from Camera, Scan for Known Objects or Features

6 Compare to Database of Known Images

7 Position and Orientation Determined X Y Z

8 Graphics are Rendered

9 SCANS COMPARES 30+ POSITIONS times per second RENDERS

10 OYXGEN HYDROGEN HYDROGEN

11 OYXGEN OYXGEN HYDROGEN HYDROGEN

12

13

14

15 Applications Today Gaming & Play 3D games in real world environments vs. virtual worlds Media / Advertising Making print, outdoor, TV media, and product packaging come alive Instructional Visualizing instructions e.g. furniture assembly In the future Turn Left on Main St. Text Translate words to multiple languages Visual Search Retrieving related information from web for the object in view Navigation & Discovery Visualizing points of interest

16 AR is gaining momentum According to Visiongain, in 2012, 25% of mobile apps will feature augmented reality. The firm forecasts the mobile AR industry will see $3 billion in global revenue by 2016, up from $87 million (in 2011) and $21 million in Consumer engagement is the main reason that (augmented reality) technology has been so successful. QR codes have proven to be effective engagement tools, which has lead many retailers, such as Macy s, to incorporate them into their marketing campaigns 16

17 Brands are using AR to engage consumers 17

18 AR deepens consumer interactions with brands Point of Advertising campaign has added AR element across media channels Point of Sale product comes alive on the shelves Point of Use added AR dimension when the product comes home 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 18

19 Vuforia Qualcomm s AR Platform 19 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform

20 Vuforia Platform Traction 21,000+ Registered Developers from 130 Countries 400+ Models of Smartphones and Tablets Supported 400+ Apps in Android Market and ios App Store

21 Accolades, Testimonials, and Honors RECOGNITION FOR OUR AWARD WINNING AR TECHNOLOGY Qualcomm s AR SDK has made it extremely easy for us to prototype, design, and develop our ideas and concepts. Morgan Jaffit, Co-Founder, Defiant Development your AR library is by far the best AR Library I ve ever used. It s amazing, works incredibly fast, and really easy to work with. Awesome job QCAR team!! Nicholas Rudolfsky, ios Developer, Smule When it comes to mobile Augmented Reality technology, Qualcomm is the top dog. Pocket-lint The opportunities to do amazing things which we quite literally couldn t live without are unparalleled with augmented reality. This is Qualcomm s secret weapon, and it may assure the company thrives, rather than simply survives the coming conflict. Rob Enderle, Analyst, TG Daily 2011 Future Mobile Entertainment Award for Mobile Augmented Reality

22 100s of articles in mainstream technology and business press

23 Near Field Usage Environments Table or Floor Wall Retail Shelf

24 Augmentable Objects Representations of real world elements that can be detected and tracked Image Target Frame Markers Simple 3D Objects Game boards Product packaging Posters Signs Greeting cards Business cards Books / magazine pages Game pieces Control cards Boxes Bottles Cans

25 Enables multiple cards to simultaneously trigger AR experiences Frame Markers

26 Allows users to press buttons by touching certain areas on an image or object Virtual Buttons

27 Features and Performance Vision Ordinary Images Simple 3D Images Marker Virtual Buttons Performance Simultaneous Object Recognition Optimized for Mobile Flexibility Unity Eclipse Xcode 3rd Party 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 27

28 Android Development Options Download Qualcomm Vuforia SDKs Eclipse Unity 3 Advantages Free Lowlevel APIs provide enhanced flexibility and performance Fully integrated game engine reduces development time and cost Single app supports both ios and Android Availability Now Now

29 29 VuforiaAR Applications Structure

30 30 Example

31 Target Management

32 Creating Image Targets

33 Targets are Trackable DataSets

34 Camera start

35 Camera stop

36 Pixel Format Conversion The pixel format converter converts between the camera format (e.g. YUV12) to a format suitable for OpenGL ES rendering (e.g. RGB565) and for tracking (e.g. luminance)

37 Tracker The tracker detects and track real world objects in camera video frames. Different algorithms take care of detecting new targets or markers, and evaluating virtual buttons. The tracker can load multiple datasets, but only one can be active at a time The results are stored in a state object that is used by the video background renderer and can be accessed from application code.

38 State Object Structure Dataset Targets Teapot on target A 3D Assets Target A Target B

39 The State Object Member Description Camera Frame Current image Type of Data RGB YUV GRAYSCALE Trackables Events List of active trackable objects Planar Image Frame Marker Multi Target List of events Virtual Button1 Virtual Button 2, etc. Name 4x4 Matrix Name Button Press

40 Video Background Renderer The video background renderer renders the camera image stored in the state object. The performance of the background video rendering is optimized for specific devices.

41 Application Code Query the state object for newly detected targets, markers or updated states of these elements Update the application logic with the new input data Render the augmented graphics overlay 10/18/2012

42 Application States oncreate() loadtrackerdata() deinitapplication() loadtexture() InitApplication() onresume() Init Tracker() initapplicationar() initapplication() StartCamera() initrenddering() updaterrendering() onpause() StopCamera() ondestroy() destroytrackerdata() deinittracker() Native Java Native C/C++

43 Creating an Android AR App Java Native C/C++ Initialize SDK Start Trackers Handle Tracker Updates Close SDK myapp extends Activity {... void oncreate() { QCAR.init(..); }... void onresume() {... setupopenglviews(); mynativecamerastartup(); mynativetrackerstartup(); }... GLRenderer implements GLSurfaceView.renderer {... ondrawframe(..) { mynativerenderer() } } void ondestroy() { QCAR.deinit(); } } mynativecamerastartup () {... QCAR::CameraDevice.init(..); QCAR::CameraDevice.start(..); } mynativetrackerstartup {... } QCAR::Tracker.setActive (QCAR::PLANAR_IMAGE); mynativerenderer {... // Render video background // Update app logic using State Object // Render app assets }

44 QCAR::ImageTracker

45 Frame Markers

46 QCAR::MarkerTracker

47 Virtual Buttons

48 Buttons coordinates <?xml version="1.0" encoding="utf-8"?> <QCARConfig xmlns:xsi=" xsi:nonamespaceschemalocation="qcar_config.xsd"> <Tracking> <ImageTargetsize=" " name="wood"> <VirtualButton name="red" rectangle=" " enabled="true" /> <VirtualButton name="blue" rectangle=" " enabled="true" /> <VirtualButton name="yellow" rectangle=" " enabled="true" /> <VirtualButton name="green" rectangle=" " enabled="true" /> </ImageTarget> </Tracking> </QCARConfig>

49 Simple 3D Objects Enables products to trigger AR experiences, and interactive content to reflect geometry of product packaging Boxes Bottles Cans

50 Creating Simple 3D targets

51 Multi Image Targets

52 Swappable Datasets

53 Swappable Datasets contd.. The script exposes a list of datasets (that are part of the project) to be loaded at scene startup. A single script can be used across different scenes Mainly used to load/unload and activate/deactivate datasets at runtime. Can ONLY be used at scene startup but doesn t do anything when its properties are changed later on in the application lifecycle.

54 Dataset Script

55 Background Texture Access

56 Background Texture Access contd.. Video background access allows you to use the camera video as a texture through the use of shaders Two shadersare used vertex shader---converts video into grayscale and inverts the black and white fragment shader---listens for a touch on the screen ad then distort the video around that touch.

57 Occlusion Management

58 Occlusion Management contd.. Illustrates how shaders(four in this case) can be used to make objects appear semi-transparent. Pointing your camera at the FlakesBox target allows you see the teapot object inside the partially transparent looking box. Four shaders, one per object, are rendered in correct order (the video background, the checkerboard-box, the teapot inside and the occlusion box).

59 Image overlay Images to be replaced Change code in teapot.h

60 Cube Structure -1.00f, -1.00f, 1.00f, // front 1.00f, -1.00f, 1.00f, 1.00f, 1.00f, 1.00f, -1.00f, 1.00f, 1.00f, -1.00f, -1.00f, -1.00f, // back 1.00f, -1.00f, -1.00f, 1.00f, 1.00f, -1.00f, -1.00f, 1.00f, -1.00f, -1.00f, -1.00f, -1.00f, // left -1.00f, -1.00f, 1.00f, -1.00f, 1.00f, 1.00f, -1.00f, 1.00f, -1.00f, 1.00f, -1.00f, -1.00f, // right 1.00f, -1.00f, 1.00f, 1.00f, 1.00f, 1.00f, 1.00f, 1.00f, -1.00f, -1.00f, 1.00f, 1.00f, // top 1.00f, 1.00f, 1.00f, 1.00f, 1.00f, -1.00f, -1.00f, 1.00f, -1.00f, -1.00f, -1.00f, 1.00f, // bottom 1.00f, -1.00f, 1.00f, 1.00f, -1.00f, -1.00f, -1.00f, -1.00f, -1.00f Set of four vertices defining a plane One vertex/corner of a plane defined by x,yand z x Left Back z y Top Right Bottom Front

61 Image overlay contd.. z y Step 1: Replace values of teapot.hwith cube.h(find cube.hin dominos app ) Step2:Remove all coordinates expect Top. Step3:Set the zero vale of z coordinate of Top. Step4:Replace the images in assets folder with required Image. Top x Left Back Top Right Bottom Top Front 10/18/

62 Video Playback

63 Video Playback Play video on an image or texture on compatible devices Gracefully fall back to an alternate videoviewing experience on incompatible devices (android video play triggered on target)

64 Initialize SDK Start Trackers Handle Tracker Updates Close SDK URL Example Java ondrawframe(..) { public static Handler mainactivityhandler; public void displaymessage(string text) { Message msg = new Message(); msg.obj = text; mainactivityhandler.sendmessage(msg); protected } } void onresume() { super.onresume(); ImageTargetsRenderer.mainActivityHa ndler = new Handler() public void handlemessage(message msg) { Intent intent = new Intent(Intent.ACTION_VIEW); intent.setdata(uri.parse(" w.qualcomm.com/partials/video/34133 ")); startactivity(intent); } Native C/C++ renderframe(){ for(int tidx = 0; tidx < state.getnumactivetrackables(); tidx++) { jstring js = env>newstringutf(trackable>getname ()); jclass javaclass = env->getobjectclass(obj); jmethodid method = env->getmethodid(javaclass, displaymessage", "(Ljava/lang/String;)V"); env->callobjectmethod(obj, method, js); }

65 Unity3d I. Installing the Extension into your unity project II. Compiling a simple AR app, and III. Mastering some of the more advanced AR topics. 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 65

66 Installation on Windows I. Download installer EXE-file, from the download page. II. Run installer III. Select a location for the package installation convenient to your development environment. The extension-only package will also be copied to the Standard Packages folder of your Unity installation 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 66

67 Compiling a Simple Project I. Create Project II. AR Assets and Prefabs to Scene III. 3D Objects to Scene and Attach to Trackables 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 67

68 Create Project Step 1 Step 2 Step 3 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 68

69 I. Step 1: Click On File-> New Project II. Step 2: Select Project path and name III. Step 3: Select Project package name 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 69

70 Structure inside Unity 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 70

71 Editor Contains the scripts required to dynamically interact with Trackabledata in the Unity editor. Plugins Contains Java and native binaries that integrate the QCAR SDK with the Unity Android application. Qualcomm Augmented Reality Contains the prefabs and scripts required to bring augmented reality to your Unity application. Streaming Assets Contains the swappable target datasets downloaded from the online Target Management System. 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 71

72 AR Assets and Prefabs to Scene ARCameraprefab:TheARCamerais responsible for rendering the camera image in the background and manipulating scene objects to react to tracking data ImageTargetprefab :This prefab represents a single Image Target Trackable object 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 72

73 Delete Main Camera 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 73

74 Adding Prefabs I. Delete main camera from hierarchy II. Add ARCamera and ImageTarget in hierachy from project window 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 74

75 ARCamera prefab 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 75

76 ImageTarget prefab 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 76

77 3D Objects to Scene and Attach to Trackables 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 77

78 I. Cube object (GameObject> Create Other > Cube) II. Lighting (GameObject> Create Other > Directional Light) 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 78

79 Directional Light 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 79

80 Animation 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 80

81 I. Select Object from hierarchy window II. In menu bar select( Windows-> Animation) 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 81

82 Animation Window 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 82

83 Step 4 Step 1 Step 2 Step 3 Step 5 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 83

84 I. Step 1:Press the recording button and save animation. II. Step 2:Press it for creating break points in animation. III. Step 3:Drage time line according to required length. IV. Step 4:Allows required type of movement (circular/hor/veretc) V. Step 5:It give options about to play(once/loop etc) 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 84

85 Android Deployment Process Platform Bundle Identifier and API Level Player Settings 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 85

86 Android Deployment Process Unity provides a number of settings when building for Android devices you will need to select from the menu (File > Build Settings > Player Settings ) to see the current settings. Also, choose your platform now Android or ios. Click onresolution and Presentationto select the required Default Orientation. Note: the Vuforia AR Extension now supports Auto Rotation. Click on Icon to set your application icon. Click onother Settings. Set theminimum API LeveltoAndroid 2.2 'Froyo' (API level 8)or higher. SetBundle Identifierto a valid name (e.g. com.mycompany.firstarapp). Next save your scene (File > Save Scene). Then open the build menu (File > Build Settings ). Make sure that your scene is part ofscenes in Build. If this is not the case either useadd Currentto add the currently active scene or drag and drop your saved AR scene from the project view into the Window. 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 86

87 Scripting Scripts Script, Data Set and Image Target ImageTargetB ehaviour 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 87

88 Virtual Button Case Statements

89 Loading Default Object w.r.ttrackable

90 Physics Engine Physics Properties 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 90

91 Physics Engine contd.. I. Rigidbodies: These are physically simulated objects.userigidbodiesfor things that the player can push around, eg. crates or loose objects, or move Rigidbodiesaround directly by adding forces to it by scripting. II. Character Controllers: use to make a humanoid character III. Collider: GameObjectthat has a Collider but not a Rigidbody. Static Colliders are used for level geometry which always stays at the same place and never moves around Box Collider: primitive shape of a cube Sphere Collider: primitive shape of a sphere Capsule Collider: primitive shape of a capsule Mesh Collider : creates a collider from the object's mesh, cannot collide with another Mesh Collider Wheel Collider : specifically for creating cars or other moving vehicles 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 91

92 VirtualButtonsExample Scripts 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 92

93 VirtualButtons Example contd.. Buttons 3D Object 93 Qualcomm Vuforia Vuforia Augmented Reality Platform 10/18/2012

94 Creating and deleting Virtual Buttons at run-time I. create a new Virtual Button for a given Image Target at run-time by calling the CreateVirtualButtonmember function on the corresponding instance of your ImageTargetBehaviour. II. destroy a Virtual Button by callingdestroyvirtualbutton, again defined in ImageTargetBehaviour 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 94

95 SoccerballExample Scripts Physics Engine 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 95

96 10/18/2012 Qualcomm Vuforia Vuforia Augmented Reality Platform 96

97 Thank you! Regional Ecosystem Strategy Brigitte Alexander 1/17/12