Augmented Reality Tactile Map with Hand Gesture Recognition Ryosuke Ichikari 1, Tenshi Yanagimachi 2 and Takeshi Kurata 1 1: National Institute of Advanced Industrial Science and Technology (AIST), Japan 2: Toyohashi University of Technology, Japan 13 July 2016, ICCHP 2016@ University of Linz, Austria 1
Walking Assists for the visuallyimpaired people Assisting technologies when they are walking White canes & guide dogs Assisting technologies with computer Remote Sensing by RGB(-D) cameras or LRF Voice navigation system based on localization technology such as GNSS Pre-Journey learning and re-viewing before/after walking Tactile Map Tactile trajectory 2
Tactile Map Maps with tactile presentation for the visually-impaired people UV offset printing:printing transparent plastic on the color maps PIAF(Picture in a flash):using heat sensitive papers which can swell black-printed area (lines/images) Casio s 2.5D printer:all-in-on solution for color and detailed tactile expression. UV Offset printing PIAF Casio s 2.5D printer 3
Problems about tactile map Difficulties for constructing mental image of the contents with haptic sense Needs experience and training Limitation of the amount of contents Production issues Cost: UV offset printing is expensive Needs know-how to design contents Tactile Map Automated Creation System (TMACS) [Watanabe et. al, ICCHP2014 ] 4
5 Proposed system:utilizing AR for tactile map Augmented Reality; AR Seamless integration of real world and virtual world Visual augmentation is main stream, but AR concept can be applied for audio sense AR Tactile Map(Proposed System): Audio-visual AR [Higa et. al 2007] Provides audio-visual feedbacks in response to interactive operations Enables dynamic and multimodal contents Can be utilized for learning tactile map
Characteristics of AR tactile map Augmentation for the tactile map by audio representation (for completely blind people) Enlarging/enhancing display for people with week sight Providing POR(Point of reference) information Intuitive interaction by actually touching tactile maps Configuration with off-the-shelf and varsatile products Hands-free arrangement Showing enlarged map with voice 6
Providing POR(Point of reference) information Target contents of the AR tactile map: POI(Point of Interest)and POR(Point of Reference) POR:Specific point location where it is easily recognized for confirming routes such as characteristic shape and material of ground (steps, stairs, sloop, door), sound/noise, and scent/odor) [Okuno et. al, CSUN2015] [Denoncin,CSUN2014] [Theberge, CSUN2015] Integration with POR/POI data base Voice navigation system with POR/POI data base [Okuno et. al, CSUN2015] 7
System configuration System configuration of the prototype system Tablet PC For Visual-Audio feedbacks RGB-D camera:depthsense DS325 (ToF, Range:15 cm ~100 cm ) For finger gesture recognition and robust extraction of hands area, measurement of the distance between the map and finger PC for recognition Arm for fixing tablet Tablet mount parts 8
Related Works ATMaps (Audio-Tactile Maps) using IVEO s Interactive tactile graphics. (ICCHP2016 workshop) Image to Sound Converter [Kuroda] Image is converted to sound by changing key and difference of L/R sound Transparent tactile sheet can be overlaid Display size and UI are limited by touch display Characteristics Kiko-AR by of cybernet AR Tactile systems Map: - Using Mobile versatile AR system products based on character recognition and audio presentation - Various sizes and types of tactile maps can be used Collaborating with a printing company to produce printed material - Intuitive interface for the system are focused with tactile presentation and AR augmentation. Need to shoot with smartphone s camera 9
Recognition of gesture Recognizing Tapping, Double Tapping, Flat-hand gesture gestures as interaction commands Algorithm for detecting Tapping Extracting hand area by RGB-D camera with background differencing of depth image and distance threshold Detecting state transition of distances within time window (1->2->1) Distance between map and finger Size of time window 2 1 1Finger on the map 2Finger in the air for tapping time Flat-hand gesture: Tracking flat-hand position when user touches maps by gradients and distances 10
11 Automatic identification and tracking of tactile map ORB Feature point detector/local feature descriptor is used for identifying tactile map with RGB image Estimating homography matrix between rectified image templates and input image
Mapping party fo Gathering acctual sites for making map (Mapping Party) Previewing by printing as tactile map Walking assistance by voice navigation
Virtual Mapping Party with AR Tactile Map Omni-directional image and 3D-Sound Recording Mapping Party Anytime, Anywhere Pre-learning of the walking route with the AR Tactile Map Requests for Mapping Utilized at voice navigation
Workshop on Virtual Mapping Party Held at Miraikan, Tokyo, Japan March 18 th -19 th, 2016. 1h x 6 times (total 42 participants) Collecting POR/POI by experiencing omnidirectional images(360 deg.) and 3D sounds with HMD(Oculus, GearVR. Etc.), PC and headphone Confirming and requesting POR/POI by AR tactile Map The number of the registered POR/POI: 598
Usage of the AR tactile map for virtual mapping party Allowing the visually-impaired people to join the mapping party by gesture Flat-hand: Confirming the existence of POR/POI where user touches Tap: Confirming POR/POI with Text-to-Sound when user taps the specific point Double Tap: Requesting the POR/POI for the specific points of the map
Summery of a questionnaire for the AR tactile map Accuracy of the gesture recognition has a room for improvement Requesting function is good because the it can reflect visually-impaired people's demand. Scale size of the map was not fitted. Considering to accuracy of the gesture recognition, the scale size of tactile maps and the density of the distribution of the contents need to be carefully considered. It seems to be interesting if the voice recognition can be combined with gesture interface. A participant with visual impairment like annotations printed in Braille, but another participant with visual impairment does not like them because they disturbed perception of the shapes of roads and buildings.
On going works Implementing another configuration of the AR tactile map with HP s Sprout. Sprout is a PC which has A projector A touch pad depth and color cameras Sprout enables touch recognition through the map Projection onto map Sheet tracking 3D scanning 17
AR tactile map with Sprout Tapping: Checking registered POR/POI Dragging: Searching Double Tapping: Requesting POR/POI Projecting POR/POI points on the map No other device or PC is required 18
19 Comparison with the proposed system Gesture recognitions Information augmentation System configuration & cost Limitation: The proposed system hand gesture recognition by RGB-D camera AR tactile map with Sprout touch detection by touch pad audio + enlarging image audio + enlarging Image + projection onto map PC($500~) + RGB-D camera($200) can be replaced with a tablet PC with RGB-D camera no visual augmentation onto map Sprout($1500) Size of map and UI for the map Advantages mobility, flexibility of UI and size of map, moderate price visual augmentation, accurate gesture recognition
Conclusions AR tactile map: Audio-visual augmentation for tactile map by utilizing hand gesture interface with RGB-D camera. Virtual Mapping Party with the AR tactile Map AR tactile map by HP s Sprout Future works Increasing accuracy of the gesture recognition Covering individual difference of gesture Noise and unclear situation (sunlight, close to paper) Other configurations of AR tactile maps (Sprout etc.) Acknowledgement This research is conducted as a part of the project sponsored by JST RISTEX. 20
Thank you! Ryosuke Ichikari, Ph.D. Postdoctoral Researcher at AIST Contact: r.ichikari@aist.go.jp 21