DAQRI ARToolKit 6/Open Source Open-source AR platform for the future Phil Oxford Brookes University 2017-01
ARToolKit 6: Future AR platform Tools Frameworks Tracking and localisation Tangible user interaction Highfidelity AR Community IOT Multi-user virtual environments
Tracking and localisation Outdoor model-based localisation Spherical panorama-based localisation Visual-inertial SLAM (e.g. OKVIS)
High-fidelity AR Radiometric scanning methods using consumer RGBD cameras
Problem statement: The identity, emotional state, bodily position and non-hand actions are extremely rich sources of information for AR systems, but are routinely ignored. We need to apply active sensing to collect information from the user, and make it easier to integrate active sensing into AR applications, to fuse data from different sources, and to process time-based data for informative structure. Outcome: AR systems will be better user experiences with lower barriers to use.
Tangible user interaction Object and environment mapping - Kinect, RealSense integration to allow use of physical proxies / totems Integrate support for haptic data into virtual environment Library support for force-feedback hardware Integration of face tracking, gaze tracking, hand and gesture tracking
Problem statement: When we build interactive AR applications that are deployed in a fixed location, we face the challenge of sufficiently bringing computer-mediated content into the fixed space around the user. This is an outwards-in transferral of information. In contrast, in deploying mobile AR applications we face the opposite problem, that of guiding the user to the content.
BIDBOT Big Internet DataBase Of Things Virtual environments (i.e. AR content ) need to be attached to specific real-world things. Currently, any given system only knows the appearance of a tiny number of things, and its reach is therefore also tiny. We need to enable client applications to learn the appearance of any object in a global database of things Outcome: we enable the fine-grained tracking of a large (and expandable) set of things both in online and offline modes, and consequent sharing of state across time and space.
Barcodes as BIDBOT enablers? Machinereadable but conforming to the signpost principle
ARToolKit trackables 2D texture Square pictorial markers Square barcode markers Rectangular, flat Must contain surface texture including lines, edges Allows for existing imagery to become trackable Limited ability to track at distance Large number of textures able to be recognised Square Black or white borders (but not both) Variable border width Mono or colour interior Iconic Meaningful to the user Works at much greater range of distances As for square pictorial, but choice of barcode pattern Error detection/correction allows for very robust matching Fastest Different sizes can be mixed Very large number of markers possible
Template matching (pictorial) Patterns must be known in advance. Binarisation Interior region extraction Unwarping/pattern extraction Pattern/orientation matching Can vary: Pattern image Pattern resolution Border size vs. matrix (2D barcode) there is no step 4 Markers need not be known in advance. Improved robustness to mismatching when using EDC Not meaningful to humans Binarisation Interior region extraction Orientation extraction/code extraction/edc (error detection and correction) Can vary: Barcode size EDC type Border size
augmented.info augmented.info augmented.info A 2D-barcode well-suited for industrial AR The augmented.info marker Inner space for branding/ logo, cutout etc. Code space Continuous black border Quiet zone i.e. white outer border Online mode with example HTTP content retrieval Offline mode with pre-loaded content Camera Touch input Camera Touch input ARToolKit ARToolKit augmented.info augmented.info code registry 0x0214 0x0215=http://shapes.com/cone 0x0216 0x Content server e.g. http://shapes.com ID code e.g. 0x0215 augmented.info query engine Content fetcher e.g. via http Marker pose augmented.info ID code e.g. 0x0215 Pre-assigned code store 0x0214 0x0215= 0x0216 0x Content store Marker pose /cone=.model Rendering/interaction Display Rendering/interaction Display
IoT Support for ambient sensing from Zigbee (à la HomeKit) sensor networks, Bluetooth-LE beacons and sensors
Problem statement AR experiences are cartoon-ish. The visual presentation of the virtual environment is never matched to the optical properties of the environment in which it is being embedded. We need to sense those optical properties so that we can make virtual content with correct lighting, shadowing, and surface texture. Non-visual dimensions of AR experiences are also neglected and are likely to be significant drivers of future AR experiences and interaction. Outcome: enhancement of the aesthetic properties of AR experiences, and increased user engagement, acceptance, and satisfaction.
High-fidelity AR Radiometric scanning methods using consumer RGBD cameras HDR spherical panoramas for specular lighting Environment mapping (Kinect/Realsense) for dynamic object occlusion in video see-through AR Positional audio integration into virtual environment Dynamic stereo vision and rendering
High-fidelity AR
A word on open source
Better, cheaper, faster; pick any three? CHEAP Oh so late Hasty rubbish You re dreaming Costs all the money FAST mate But there is a way GOOD conditions apply
Open-source benefits: for the users Copyleft & commercially viable license Sourcecode understandable verifiable/secure modifiable Projects x1000s, Users x100 000
Open-source benefits: for enterprise and institutions Contributions from other commerciallymotivated developers Side benefits for recruiting/retention, marketing Incentivised development CHEAP AWE- SOME FAST Choose problems that benefit from the long lever Parallelise projects GOOD Thousands of eyeballs beats hurried in-house code review Fixes can go live immediately Use selects desired feature set
How Worklist Works User actions System actions Job creator Designer Developer Hifi Developer Worklist App Status Open source R&D Creates job If a Designer creates a job it will automatically go into Bidding Sets job to Bidding Suggestion Bidding Adds bid to job Accepts the bid Status set to In Progress In Progress Completes the work Set job to QA Ready QA Ready Pathway for open collaboration e.g. with publicly funded research, teaching Test/approves the work Sets job to Code Review Creates pull request on GitHub Code Review Incentivised development for nonmonetised part of business Leaves note on pull req tested, ready Reviews code Gets GitHub notification Merges with master Sets job to Merged Merged Sets job to Done Done Gets paid Gets paid Gets paid Enters payout queue
Upcoming DAQRI open source developments Q1 2017 Q2 2017 ARToolKit 6 public binary release Planar surface tracker on ios and Android Windows 10 UWP platform support augmented.info JSARToolKit texture tracking support
Participate 1. use and share the publicly-released ARToolKit 2. join ARToolKit forum/mailing list 3. follow @artoolkit_ on twitter 4. tell us about your project and we can promote it! 5. contribute source back to us via github