Ubiquitous Sensoy Intelligence Péte Koondi, Péte Szemes and Hideki Hashimoto Integated Intelligent Systems Japanese-Hungaian Joint Laboatoy Univesity of Tokyo, Komaba 4-6-1, Meguo-ku, Tokyo, JAPAN 153-8505 Budapest Univesity of Technology and Economics, H-1521, Budapest, Po. Box. 91. HUNGARY, Tel: +36 1 463 2184, fax: +36 1 463 3163, e-mail: koondi@elekto.get.bme.hu Abstact: The pape pesents an Intelligent Space (oom, coido o steet), which has ubiquitous sensoy intelligence including vaious sensos, such as cameas, micophones, haptic devices (fo physical contact) and actuatos with ubiquitous computing backgound. The Actuatos ae mainly used to povide infomation and physical suppot to the inhabitants, with the help of speakes, sceens, pointing devices, switches o obots and slave devices inside the space. The vaious devices of ubiquitous sensoy intelligence coopeate with each othe autonomously, and the whole space has high intelligence based on ubiquitous computing, which is used manly fo welfae suppot. Intelligent Space can guide and potect a blind peson in a cowded envionment with the help of a mobile agent. In emegence case such as fie, the Intelligent Space can guide the cowd to avoid accidents and tagedy. 1 Intoduction A conceptual figue of the Intelligent Space with ubiquitous sensoy intelligence is shown in Fig. 1. The Ubiquitous Sensoy Intelligence is ealised by Distibuted Intelligent Netwoked Devices [1], obots, which ae physical agents of the Intelligent Space, and Human. In the Intelligent Space, DINDs monito the space, and achieved data ae shaed though the netwok. Since obots in the Intelligent Space ae equipped with wieless netwok devices, DINDs and obots oganize a netwok. The Intelligent Space based on Ubiquitous Sensoy Intelligence supply infomation to the Human beings and it can help them physically by using obot agents. Conventionally, thee is a tend to incease the intelligence of a obot (agent) opeating in a limited aea. The Ubiquitous Sensoy Intelligence concept is the opposite of this tend. The suounding space has sensos and intelligence instead of the obot (agent). A obot without any senso o own intelligence can opeate in an Intelligent Space. The diffeence of the conventional and Intelligent Space concept is shown in Fig 1. Thee is an intelligent space, which can sense and tack the path of moving objects (human beings) in a limited aea. Thee ae some mobile obots contolled by the intelligent space, which can guide blind
Coopeation Netwok DIND DIND Senso DIND & Senso Actuato Senso & & DIND Actuato Actuato DIND Senso & Actuato Senso & Actuato Suppot AGENT HUMAN AGENT (ROBOT) (ROBOT) SPACE Infomative/electical Sevice Monitoing Physical Sevice Conventional concept Ubiquitous Sensoy Intelligence concept Fig. 1. Compaison of conventional and Ubiquitous Sensoy Intelligence concept pesons in this limited aea. The Intelligent Space ties to identify the behaviou of moving objects (human beings) and ties to pedict thei movement in the nea futue. Using this knowledge, the intelligent space can help avoiding the fixed objects and moving ones (human beings) in the Intelligent Space A mobile obot with extended functions is intoduced as a mobile haptic inteface, which is assisted by the Intelligent Space. The mobile haptic inteface can guide and potect a blind peson in a cowded envionment with the help of the Intelligent Space. The Intelligent Space leans the obstacle avoidance method (walking habit) of dynamic objects (human beings) by tacing thei movements and helps to the blind peson to avoid the collision. The blind peson communicates (sends and eceives commands) by a tactile senso. The pototype of the mobile haptic inteface and simulations of some basic types of obstacle avoidance method (walking habit) ae pesented. Some othe Intelligent Space pojects can be found in the Intenet [2,3,4,5]. The stuctue of this pape is as follows. Section 2 summaizes the basic components (DIND, Vitual Room, Ubiquitous Human Machine Inteface) of the existing Intelligent Space. Section 3 gives an impession of what kind of sevice this Intelligent Space can offe. The last section concludes the pape. 2 Basic Elements of Ubiquitous Sensoy Intelligence Thee inteesting elements of the cuent Intelligent Space with Ubiquitous Sensoy Intelligence Distibuted Intelligent Netwok Device Vitual Room Ubiquitous Human Machine Inteface ae selected and biefly descibed hee.
2.1 Distibuted Intelligent Netwok Device We can use as a definition: A space becomes intelligent, when Distibuted Intelligent Netwok Devices (DINDs) ae installed in it [1]. IND is vey fundamental element of the Intelligent Space. It consists of thee basic elements. The elements ae senso (camea with micophone), pocesso (compute) and communication device (LAN). DIND uses these elements to achieve thee functions. Fist, the senso monitos the dynamic envionment, which contains people and obots. Second, the pocesso deals with sensed data and makes decisions Thid, the DIND communicates with othe DINDs o obots though the netwok. Fig. 2 shows the basic stuctue of human decision and DIND. In actual system, six Sony EVI D30 pan-tilt CCD camea and geneal bt848 based image captue boad is adopted as a senso [6]. Fo the pocesso, industial standad Pentium III 500MHz PC is used and geneal 100baseT LAN cad is used as a netwok device. Robots ae able to use esouces of DINDs as thei own pats. Howeve, obots with thei own sensos may be consideed mobile DINDs. DIND (Distibuted Intelligent Netwok Device) Pocesso Tansfom infomation Refe database Make database Communication Communicate with obots o othe DINDs Senso Look at space Fig. 2 Fundamental stuctues of human decision and DIND 2.2 Vitual Room The aim of Vitual Room (VR) eseach poject is to eceate an envionment of a physical expeimental space fo studying diffeent motion contol and vision algoithms fo a given obot befoe eal wold implementation. The oom cuently contains the following objects: Passive objects: desks, chais Active objects: Thee obot agents Sensos: CCD cameas Actuatos: Lage Sceen
2.2.1 Simulation Famewok The poject is developed in C++, and gaphical implementation of the objects is achieved using Coin/Open Invento libay. Invento's foundation is supplied by OpenGL and UNIX, Invento epesents an object-oiented application policy built on top of OpenGL, poviding a pogamming model and use inteface fo OpenGL pogams. The cuent development opeating system is a Suse Linux 8.1, and Coin3D vesion is 1.0.4 2.2.2 Simulation examples The pesent state of the Vitual Room includes gaphical epesentations of the objects mentioned above. The gaphical envionment allows a walk though the vitual space and it is also possible to visualize the vitual image of each camea with this configuation. The images of a vitual camea and a eal camea ae compaed in Fig. 3. Both ooms (vitual and eal) have 6 pan-tilt cameas []. Vitual Room Real Room Fig. 3 Vitual and eal Room fom the 1 st CCD camea s point of view 2.3 Ubiquitous Human Machine Inteface Thee ae thee mobile obots in the cuent Intelligent Space. The most inteesting one is a special mobile human-machine inteface [7]. Thee ae thee basic communication channels, which the people use in daily convesation: audio, visual, and haptic. All thee communication-channels ae implemented on the UHMI. The design concept is shown in Fig. 4. The human use, who is in the Intelligent Space has an aim in his/he mind, which he/she want to ealize. Then the use makes diffeent type of commands to each his/he aim. Some commands ae associated with cetain pats of the human body. UHMI has special devices to make connections with cetain pat of the human body. A video Camea and a TV Sceen ae mounted on the UHMI fo visual communication. Speake and micophone ealize the audio communication. Haptic Inteface is mounted on the
obot to ealize a physical connection. The UHMI can be seen on Fig. 4. This UHMI is able to move to the use o it can guide him/he. A vey special application is the guidance of blind people. A blind peson can communicate (send and eceive commands) with a tactile senso. Intelligent Space Use`s Mind OBJECT Use`s Body EYE Inteface TV Sceen Monito with Speake Mobile Robot Platfom Pan-Tilt CCD Camea Micophon e Pocedues Commands #1 Commands #2 EAR VOICE ARM + HAND Speake Micophone Haptic Inteface Motivation: Pesonal Communication And Guiding Haptic Inteface Fig. 4 Ubiquitous Human Machine Inteface: concept and pictue 3 What Can Be Done In Intelligent Space? 3.1 3D Positioning of Human To suppot humans in the space, the Intelligent Space tacks them. Recognition of a human is done in two steps [8]. Fist, the aea o shape of a human is sepaated fom the backgound (Fig. 5). Second featues of the human as head, hands, feet, eyes etc. ae located (Fig. 5). Taking the images of thee pais of cameas, the 3D position of the human beings can be calculated (Fig. 6). The scanned aeas of camea pais ae ovelapped. To calculate 3D fom seveal camea views point coespondences ae needed. To establish these coespondences diectly fom the shape of the human is difficult. Instead of it, the head and hands of the human beings ae found fist and thei centes is used fo matching. A second motivation to futhe analyse the shape is that adaptive backgound sepaation in complex scenes detects ecently displaced objects. The above algoithms ae implemented in thee diffeent softwae modules (Camea Seve, 3D Reconstuction Module, Calibation Client) of the Intelligent Space. The eo of the estimated position of an object changes with the distance fom and pose of the camea. The eo is influenced by seveal factos; the pefomance of each camea, the method of image pocessing, etc. Kalman filte is applied to smooth the measued data.
Fig. 5 Sepaation of Human beings fom the backgound Fig. 6 Localization of Human beings by two pais of cameas 3.2 Map Building by Looking at People Mobile obots need maps of thei envionment fo navigation, localization and task specification. Mobile obots can navigate obustly without a pecise geometical model if some othe way of localization is given and a topological map is supplied. The suggested appoach is to look at the movements of people in the oom. In indoo envionments people and obots conside simila things as obstacles. This method has the additional advantage that it detects obstacles that most sensos fail to notice. Examples ae tapdoos, yellow lines on the floo o signs saying "Dange - Don't Ente". Positions of moving pesons wee obtained with about 20 Hz. Only positions with a vetical height between 1.65 and 2.00 metes and only blobs with at least 0.6 times the size of a head wee taken into account. In Fig. 7, the diffeent stages of the algoithm ae shown. Filteing out all suspected non-head positions educe some hand blobs eveywhee. Hee positions wee filteed out as head and hands ae much lowe than that of walking people. The Fig. 7(b) shows the topological map of the oom. The topological map
coectly avoids all static obstacles as well as the dynamic ones that wee not in the wold model. (a) Position of heads (b) Blued pixels (c) Topological map of the oom Fig. 7 Pocedue of geneating topological map 3.3 Mobile Robot Localization Neithe ange sensos to detect obstacle aound the obots, no position-sensos to estimate position and oientation of the obots ae needed in the Intelligent Space. The obots only equie passive colou bas and a communication device to communicate with DINDs. Due to communication between obots and DINDs, obots ae able to use sensos of DINDs feely. It is a kind of esouce shaing and this featue leads to low cost obots, since expensive sensos, such as gyoscope senso, lase ange finde, etc ae not needed. Only simple PLC boad, motos and wieless LAN (IEEE 802.11b) ae installed in the obots. In Fig. 8, the a obot that has space to load a buden on the top is shown. Cc R,t Xw2 Xw1 X Xw4 Xw3 θw Cw Fig. 8. Mobile Robots with colou bas and its ecognition and tansfomation Unlike the case of detecting a human, to localize a mobile obot in the Intelligent Space, only one DIND is equied. Because the height (z-axis) of the tagets is known, only one camea is needed fo a pecise 3D constuction. Two pocedues ae compaed in Fig. 9. Fom the acquied image, seaching colou egion and backgound sepaation is pefomed simultaneously. In this system, since the colou of taget is yellow, the egions of yellow ae seached. Backgound sepaation module compaes cuent image and backgound image, and selects only the egion whee the movement happened. Adaptable backgound is selected
to make the backgound sepaation module wok obustly [8]. Outputs fom these two modules ae compaed in the clusteing pocess. Only common egions ae conveyed to calculation of moment module, and 2D positions of the egions ae calculated. Since a DIND has a database of height of colou ba, one DIND is able to estimate the position of a colou taget. The 2D positions of colou tagets in image coodination ae conveted to 3D position in camea coodination. Accoding to Fig. 8, the following elation can be witten to tansfom coodinates fom the camea s coodinate system to the wold coodinate system, whee x is position, R is otational matix and t is tanslation matix. x = Rx + t (1) w c x + x l y + x = y = y l 2 2 2 2 1 4 2 1 4 2 ( ) cos θ, ( ) sinθ x + x l y + y l x = + y = + 2 2 2 2 2 3 2 2 3 2 ( ) cos θ, ( ) sinθ x + x l y + x = + y = y l 2 2 2 2 1 2 1 1 2 1 ( ) sin θ, ( ) cosθ x + x l y + y l x = y = + 2 2 2 2 1 y y 1 2 1 y4 y3 θ = tan ( ) = tan ( ) x x x x 3 4 1 3 4 1 ( ) sin θ, ( ) cosθ 1 2 4 3 π θ = + = + 2 2 1 y y π 4 1 1 y3 y2 tan ( ) tan ( ) x4 x1 x3 x2 These matixes ae found fom the calibation pocess of a DIND, based on Tsai s algoithm. Afte taget detection pocedue, the 2D Euclidean distances between detected tagets ae checked to get id of eos. If the distance is too long o too shot, the detected tagets ae neglected. Even afte this, the esult may stil contain ecognition eos. Accoding to the database of the DIND, each colou- code, placed below the yellow taget, is checked. If a colou-code, which is not shown in the database, is detected, it is emoved fom the list. Fom this pocess the DIND is able to ecognize which obot it is too. The geometical elations, stoed in the database, ae compaed between the colou-codes and it geneates the pose of the obot. The pose of the obot is calculated fom the pais of colou-codes, with the same time stamp. Pose of a obot is estimated fom geometical elation, if at least one pai is ecognized.
DIND 1 DIND 2 Image Acquisition Colo Model Refeing Clusteing Backgound Sepaation Calc. of Moments DIND Image Acquisition Colo Model Refeing Clusteing Backgound Sepaation Calc. of Moments 3D Conveting Data Base Matching of Clustes 3D Reconstuction Colo Code Reading Refeing Database Geneation of Robot Location (a) Pocedue fo human beings o obstacles (b) Pocedue fo mobile obots Figue 9. Compaison of positioning of a human beings and mobile obot 3.4 Active Resouce Supplement fo Mobile Robot Because DIND always watches mobile obots in the space, it can suppot them accoding to thei cuent situation. DIND can suppots diffeent mobile obots with diffeent esouces simultaneously. Thus, even a toy, if it has basic elements such as communication device, actuato, and ough contolle, can be utilized as a obot in the Intelligent Space. An expeiment is caied out to show an example of what DIND can do fo the mobile obot [9]. In this expeiment, self-localization behaviou of a mobile obot was inactivated intensively. Since it has no way to measue neithe position no speed, it is no moe than a toy, which has a adio eceive. A human opeato indicated desied taget points to DIND and it tanslated this infomation to the obot. The obot just followed the ode fom DIND whethe the output was coect o not. DIND continuously tied to fix position eos of the obot and it finally eached the goal. Fig. 10 shows how the obot gets to the goal. Accoding to the position, domain DIND fo the obot is decided. The whole aea of the laboatoy was divided into five aeas and each aea is watched by DIND. When a mobile obot taveses one aea to anothe, DIND sends infomation of the obot to the next DIND, which ae the domain DIND of the aea whee the obot will appea. 3.5 Human Motion Compession The scenaio, shown in Fig. 11, is that the Intelligent Space ecognizes what human is doing in it and sepaates the eal video data into index images and actions with time stamp. It will cause enomous data compession and help to make database. This function is expected to be utilized in suveillance camea system in shops o banks.
12.5m DIND-5 DIND-1 DIND-2 DIND-2 goal DIND-1 DIND-4 DIND-3 DIND-5 6m DIND-3 Fig. 10 Tajectoy of Mobile Robot DIND-4 stat : DIND Real Data Index Image Action and Condition with Time Stamp Fig. 11 Intelligent Video Recoding 3.6 Intepete between Human and Robot A human inteface is a necessay pat of an intelligent system, which inteacts with a human opeato. Howeve, since the human inteface depends on the system, a human opeato should lean human intefaces of each system. In the Intelligent Space, a human inteface module is located in the Intelligent Space, not in each intelligent system. Thus, development effots of systems ae lessened and a human opeato can contol many systems with only a single human inteface. Fig. 12 shows the concept. The Intelligent Space woks as an intepete between a human opeato and systems including obots.
Fig. 12 Intepete Function of Intelligent Space 3.7 Leaning walking habits [10] A simple example to illustate the impotance of this knowledge. Let's assume that a Japanese and an Ameican peson ae walking towads each othe. Recognising this situation, they ty to avoid each othe. Using thei geneal ule, the Japanese peson keeps left and the Ameican keeps ight and they ae again in font of each othe. In the end they may even collide (Fig. 13). If the Intelligent Space can lean the walking habit of a human being, it can send a pope command to the obot in such situation. Wall Ameican Japanase Desied diections of humans Diection of Fig. 13. Example fo the poblem of two diffeent default avoidance styles 3.7.1 Basic walking styles The main guiding ule of an aicaft caying dangeous mateial is to keep as fa fom the mountains as possible (see Fig. 14. a). Remaining in secet while an inspecto is seeking fo a ciminal leads to the opposite behavio, namely, get as close to the object as possible (see Fig. 14.b). A simple combination of these can descibe the main ule of a taffic system: keep close to ight o left side
a.) Keep as fa as possible b) Keep as close as possible Fig. 14. Obstacle avoidance styles 3.7.2 Mathematical desciption of walking habits [10] The Intelligent Space can detect objects aound the obot (Fig. 15). The scanned aea is divided into n scanned lines that ae pointed into diections of e v z (unique vectos, whee z = 1KZ ). The diffeent walking habits (obstacle avoidance methods) can be descibed by the epulsive (o attactive) foces as a function of obstacle distance. The walking path is defined by the sum of epulsive foces. The magnitude of epulsive foces, F, in the diffeent diections can be descibed by a potential field (Fig. 16). The magnitudes of epulsive foces ae usually invesely popotional to the distance but they can be descibed by any non-linea potential field. To achieve keep as fa as possible walking style, a symmetical potential field is necessay (Fig. 17.a). In the case of a keep close to left side style (Fig. 17.b), the epulsive foce must be bigge at the ight side of the moving object than that of the left side. An attactive foce might be geneated at the left side to get close to the obstacle, if the moving object is fa fom it. The Intelligent Space ties to lean the walking habits (obstacle avoidance methods) of the moving objects. obstacle obstacle Z-1 Z 1 2 3 4 1 2 3 4 1 Z f pot (.) f pot (.) f pot (.) f pot (.) f pot (.) f pot (.) obstacle Fig. 15 Geneating tajectoy
F y Potential field F = f ( x, y) x Robot Potential function w z ( x z ) Obstacle diection Fig. 16. Potential field of epulse foces a. b. Potential field of keep as fa as possible style Potential field of keep left style Fig. 17. Potential fields of two diffeent walking habits 4. Conclusion This pape pesented an impession of Ubiquitous Sensoy Intelligence in Intelligent Space poject. With the pogess of technology, some of peviously unbelievable systems appeaed fist in science fiction liteatue have actually become a pat of eveyday life like fo example space ockets and obots. In ou laboatoy we ae developing such a system, which is motivated by a high intelligence compute named HAL fom the movie Space Odyssey 2001. We believe, obots unde the contol of Intelligent Space have many good featues. Human opeatos can communicate with the obots with the help of Intelligent Space. We would like to emphasize that Intelligent Space is an intelligent system to suppot obots as well as human beings. The Intelligent Space as a technology, might change ou living envionment. Thee ae two diections, which must be enhanced. One is enlaging the aea, which is able to apply the Intelligent Space concept. Cuently the Intelligent Space is ealized only in an expeimental envionment. An Intelligent ailway station could the next taget. The othe is industialization of all possible technologies used in the Intelligent Space. It enables wide and commecial applications.
5 Ackonwledgement The authos wish to thank the JSPS Fellowship pogam, National Science Reseach Fund (OTKA T034654) and Contol Reseach Goup of Hungaian Academy of Science fo thei financial suppot. and the suppot stemming fom the coopeation between the Budapest Univesity of Technology and Economics and Politehnica Univesity of Timisoaa in the famewok of the Hungaian- Romanian Integovenmental S & T Coopeation Pogamme. 6 Refeences [1] J.-H. Lee and H. Hashimoto, Intelligent Space - Its concept and contents, Advanced Robotics Jounal, Vol. 16, No. 4, 2002. [2] Hashimoto Laboatoy at The Univesity of Tokyo http://dfs.iis.u-tokyo.ac.jp/~leejooho/ispace/ [3] MIT Poject Oxygen MIT Laboatoy fo Compute Science MIT Atificial Intelligence Laboatoy http://oxygen.lcs.mit.edu/e21.html [4] Field Robotics Cente, Canegie Mellon Univesity Pittsbugh, PA 15213 USA http://www.fc.i.cmu.edu/pojects/spaceobotics/publications/intellspacerob ot.pdf [5] Institut of Neuoinfomatics Univesität/ETH Züich Wintethuestasse 190 CH-8057 Züich http://www.ini.unizh.ch/~expo/2_0_0_0.html [6] T. Akiyama, J.-H. Lee, and H. Hashimoto, Evaluation of CCD Camea Aangement fo Positioning System in Intelligent Space, Intenational Symposium on Atificial Life and Robotics, 2001. [7] Pete T. Szemes, Joo-Ho Lee, Hideki Hashimoto, and Pete Koondi, Guiding and Communication Assistant fo Disabled in Intelligent Uban Envionment Poceeding of IEEE/ASME Intenational Confeence on Advanced Intelligent Mechatonics, July 20-24, 2003, Intenational Confeence Cente, Pot Island, Kobe Japan p.598-603, 2003. [8] Kazuyuki Moioka, Joo-Ho Lee, Hideki Hashimoto, Human Centeed Robotics in Intelligent Space 2002 IEEE Intenational Confeence on Robotics & Automation (ICRA'02), pp.2010-2015, May, 2002 [9] Guido Appenzelle, Joo-Ho Lee, Hideki Hashimoto, Building Topological Maps by Looking at People: An Example of Coopeation between Intelligent Spaces and Robots, Intl. Conf. on IROS, 1997 [10] S. Mizik, P. Baanyi, P. Koondi, and M. Sugiyama Vitual Taining of Vecto Function based Guiding Styles Tans. on AUTOMATIC CONTROL ISSN 1224/600X vol. 46(60) No.1 pp. 81-86. 2001.