Mixed Reality technology applied research on railway sector Yong-Soo Song, Train Control Communication Lab, Korea Railroad Research Institute Uiwang si, Korea e-mail: adair@krri.re.kr Jong-Hyun Back, Train Control Communication Lab, Korea Railroad Research Institute Uiwang si, Korea e-mail: jhbaek@krri.re.kr ABSTRACT In the middle of that the expectation are rising along with the chance to the application of the mixed reality technology and the potential possibilities, there is a constant attempt to link this technology with education and practical use. By applying such efforts and skills into the railway technology, we can train people with real images in a 3-dimension space and let them focus on the railway education. Moreover, we can provide education systems for railway maintenance which can be conducted in a safe and field-like way. By using the mixed reality technology that the reality and virtual reality environments are linked each other, we are to implement the system that we can provide, with more realistically and comfortably, graphictized and redundant passage information around the transfer locations using a personal mobile communication device. 1. INTRODUCTION Mixed Reality(MR) or Augmented Reality (AR) technique is the next generation of information processing technology for maximizing usability and usefulness of information by merging of real environment objects and virtual produced information to make users interact in real time[1]. Mixed Reality technology has been recognized as an important technology that can convey multimedia content information more accurate and more efficient by making the real world information eliminated or simplified. Now current mixed reality technology is being studied and applied in various fields. Traditional application areas of Mixed Reality technology that is in the assembling processing, investigation, medical fields and among the technology, MR technology applications in the industrial sector of automotive and robotic has already reached a mature stage of Yong-Kyu Kim, Train Control Communication Lab, Korea Railroad Research Institute Uiwang si, Korea e-mail: ygkim1@krri.re.kr technology for making profits in the industry[2]. In addition to these technological developments, various media are being developed, being grown up application opportunities of new technologies and potential possibility[3], we think this technology integrates into railway system and it will be developed for keeping up with current trends as a means of transportation, it needs the study of Mixed Reality about own railway system to take a step forward. In Korea, in the Mixed Reality technology expert groups such as SigARt (Special interest groupon Augmented Reality technology; Catholic University, Gwang ju Institute of Science and Technology, Korea University, National University, SAIT, So gang University, Se jong University, Han yang University, Hong ik University, Independence co, KAIST, KIST etc.) and in the industrial and academic field of IT, they introduced R & BD research methodology of integrated strategy fourth generation and they are trying to discover the opportunity of MR Killer Application and to deduct a interaction conception of a new paradigm and they also have interested in Mixed Reality technology Research and Development which is an important Enabling Technology to lead the next generation IT Industry greatly,[4] ISMAR(International Symposium on Mixed & Augmented Reality) conference will be held in October of this year in Korea. Chapter 2 of this paper, we will discuss the directivity and applicability about the study on traffic informative system using mixed reality and trains maintenance training system, and the last Chapter 3 will conclude on the basis of research directions from Chapter 2. ISBN: 978-1-61804-074-9 212
2. MIXED REALITY : MR A. Mixed Reality Mixed Reality(MR) is a technology for providing more improved immersion and realism to the users by giving them integration of real-worlds and virtual worlds seamless in the real time, it is also called Augmented Reality(AR).[5] According to Reality-Virtuality Continuum of Milgram[6], like figure 1, Mixed Reality is located in the middle of the real world and virtual world and it composes actual image with virtual computer graphics. It is gaining popularity because it has the advantage that can beyond the limitations in the virtual reality feeling and can keep feeling of the real world, even take the merit of virtual world. Figure 1. Reality-virtuality continuum of Milgram B. A providing system of traffic pass information using Mixed Reality It refers traffic pass informative system using Mixed Reality, is related to the system that can check the traffic pass information around the transfer location more realistic and more Comfortable by using Mixed Reality which means combining reality environments with virtual environments through the personal mobile communication devices like cell phone in detail. We know that current traffic pass information system is generally providing only limited and restricted functions such as arrival time of the transportation area, the minimum time it takes to the destination It is generally not easy for us to get to the destination that we want to go in case of a road new to us or complex areas because we must be secured the information about neighborhood and character of geographic information system. Even if there are many technologies and systems that provide the function to find our current location and destination by integrating GPS technology into mobile communication devices(cell phones, navigation) in order to relief such discomfort, such functions using two-or three-dimensional graphics map dissimilar to reality or are used images that was stored in advance. Due to this reason, it can reduce the reality and be limited the information. The purpose of this study is to solve these problems and to provide traffic pass information system used augmented Reality to enable users to check by means of display overlapped virtual images with information graphics about transportation of transfer area and traffic pass around the transfer place from the real images gained by the camera of personal communication devices mixed reality with technology overlapped visual object in the real ISBN: 978-1-61804-074-9 213
world that users look through their eyes by using personal mobile communication equipment with on-board camera. C. Technology for traffic pass informative system using mixed reality It consists of transfer information of transfer places to stop the transportation vehicles, MR server to provide a virtual images dealt information about each means of transportation with letters or graphics, and personal mobile communication devices displaying by means of overlapping realistic image of transfer place obtained through the camera with virtual images received from the server in order to implement traffic pass informative system using mixed reality. MR server consists of public object DB for storing and managing the public objet, personal object DB for storing and managing the personal objet transferred from personal mobile communication devices. Personal mobile communications device consists of camera for gaining images, GPS receiver for receiving the current location information, a part of communication for virtual images from MR server, a part of operation control for controlling by means of overlapping realistic image of transfer place obtained through the camera with virtual images received from the MR server, a part of display for showing images overlapped realistic image with virtual images, a part of key input for entering the key of users, and a part of power supply for supplying power. MR server receives transfer information about transportation that stops at transfer places such as bus stop, subway station, airport and information about each means of transportation from the information servers of transfer information and each means of transportation. And, transfer information of vehicle includes data of congestion level such as route of transportation and latency, the abovementioned information about each means of transportation includes spacing of each vehicle seat, change of indoor temperature and air flow, traffic convenience data like amount of fine dust, and carbon data like carbon Mileage about each vehicle. Also, AR server is stored as objet that includes transfer place information and virtual images about each means of vehicle in the measuring device like traffic light, station CCTV, traffic CCTV, temperature and carbon dioxide meter in measuring devices, such as temperature and carbon dioxide meter installed in the transfer place such as bus stop, subway station, airport by collecting information around the transfer places and then it is stored and managed by converting into virtual image treated with a letter or graphics. Through personal mobile communication devices with on-board cameras, we try to utilize and to check the extended information such as transfer place such as bus stop, subway station, airport, route of vehicle to transport passenger, transfer place information like intervals and the latency, information about ease of transportation, congestion level in the bus stop, subway station, airport, boarding delay time, air flow in the transfer place or transportation and air comfort by means of recognizing many kinds of objects that is the virtual image using transfer place information through letters or graphics for using transportation such as bus or subway. In this case, objet is a Augmented Reality service combining a wealth of information in real time, showing them on the actual screen view of personal mobile communication camera( for example, smart phones). It can be gained useful information about existing images displaying on the screen, it also can be used anywhere, anytime immediately by making and storing and managing new objet. ISBN: 978-1-61804-074-9 214
D. Education System for railway maintenance using Mixed Reality It refers to education system for railway maintenance using mixed reality, it means education system for railway maintenance using Mixed Reality that realistic images of the train will be displayed in the education room that is available the three-dimensional space and information depends on the movement of trainees will be extracted then virtual images of trains overlap with real images to induct the trainees be immersed in their needed education for maintenance of trains in detail. Generally, as trains is a mean of public transportation running on the track like subway or train, they have advantage over transporting a lot of things and people and it also have merits that it can move on schedule. These trains are required maintenance by the maintenance worker with having expertise. On the other hand, for training these workers of train maintenance, we is training the trainees by using existing real bogie and train for their education. But in case of using the real train, trainees can be hurt not only by using parts and component incorrectly even if they know all safety regulations, but also they have trouble with learning about the complex structure of train efficiently. In addition, there is also the problem that reduces level of education immersion due to have difficult in training structure installed in the narrow space of the train. This education system is intended to solve these problems, it is required to implement system for proceeding the education by providing the trainees education system for railway maintenance using Mixed Reality to increase their level of immersion by means of giving them useful information needed for train maintenance with feeling of reality through the mixed reality grafting real environment and virtual environment. Stereo camera (left) Figure 2. System for railway maintenance using Mixed Reality E. Technologies that meet the needs of education System for railway maintenance using Mixed Reality The system of train maintenance comprises camera for shooting real images of trains, indicators for displaying the 3-D real images installed in the training room, human body movement detectors for extraction three-dimensional coordinate information depends on the trainees movement in above-mentioned room, data storage means for storing 3- D virtual images consisting of letters or computer graphics to educate train workers through the 3-D coordinate, operation control means for handling to display by overlapping the train real images through the abovementioned indicators with above-mentioned virtual 3-D images through the 3-D coordinate information of human body movement detectors. Stereo camera USB communicat Mai n PC Bluetooth communication ISBN: 978-1-61804-074-9 215
At first when the trainees enter in the train room, the movement of them is detected by vision sensor that is a mean of detector of human body movement and then this gives 3-D coordinate information the trainees movement and it also gained the 3-D angle information by infrared sensors installed in the room or body sensor that trainees put on. The camera for shooting real images of trains using the stereo camera for obtaining 3-D images, shoot the external and internal of train in each four places. On the other hand, the projectors are installed at least in four places in the room, it displays 3-D real image of train. The human body movement detectors for extraction three-dimensional coordinate information depends on the trainees movement in training room, using vision sensor to obtain 3-D coordinate information in order to determine the position of trainee and to know the correlation with body. In this case, it is obtained 3-D angle information of trainees from body sensor that they wear or the infrared sensors installed in the room to extract information about energy such as the force rotation of trainees and power dynamics. The 3-D angle information from body sensor or the infrared sensors can blend three-dimensional coordinate information and extract the trainees action pattern information, virtual images with the letter or computer graphics according to Three-dimensional coordinates by means of extraction above-mentioned action pattern information for appropriate education of train can be indicated by indicator, projector. At this time, sensor is always used by many kinds of sensors like input sensor or Gyro sensor, and of course it can be chosen. Especially, above-mentioned the body sensor send the extracted 3-D angle information to operation control means with cable or wireless, in this case it is required to implement interface and this interface is known as a technology that is a communication using bluetooth module. On the other hand, data storage means store and manage extracted information of trainees action patterns by blending trainees 3-D angle information and 3-D coordinate information with train information DB and virtual images dealt information about each means of transportation with letters or graphics for their train training. The 3-D virtual images of the train training means for storing data is not an above-mentioned indicator but a HUD device that trainees wears, it is utilized by display on the education. And operation control means can use a variety of computer such as personal computers, laptop computers, industrial computers. It is required to implement more speakers to output educational sound of comment in the insides of education, and it is required to add voice information in the above-mentioned data storage means and then it will give them a voice guide about train components or example of manipulation according to their needs. Owe to such 3D training, trainees can be focused on their education and they can also get the safety training. Both of these factors are merits of this education system. 3. CONCLUSION Mixed Reality technology must be a technology that influences in development of all areas of IT technology in the present, and it may affect many changes and development of all IT technologies in the future. Now Europe plays a leading role in excavation the Mixed Reality technology and the application of the field more active than USA, especially in the industry such as motor car, construction, robot, broadcast and entertainment, the level of mixed reality is considerably high in Europe. Also, mixed reality can be applied to our variety of technologies on cell phone industry that is showing strong in Korea. Recently a ISBN: 978-1-61804-074-9 216
lot of smart phones are flowed in the market, and then some attempt to apply to mixed reality is progressing in the various fields. In this paper, the direction of the proposed research is a small part of the other studies. However, if we apply this Mixed Reality technology developing in many ways to the rail technology, railroad will be in public with unlimited development and more upgraded quality. REFERENCES [1]. R. T. Azuma A survey of augmented reality. In Presence : Teleoperators and Virtual Environments, 6(4)pp.355-385, 1997. [2]. Sang-Goog Lee, Industrial Augmented Reality, SAIT(Samsung Advanced Institute of Technology)Technical Report Oct-2005, pp 2-23 [3]. Jang sang hyeon, Gye bo gyeong (Korea Education & Research Information Service), Educational Application of the content of Augmented Reality [4]. Sang-Goog Lee, Mixed Reality technology research and development trends And prospects [5]. Milgram, p., & Keshino, F.(1994). A taxonomy of mixed reality visual display. IEICE Transactions on Information and System, E77-D,12,1321-1329. [6]. http://www.liteye.com/ ISBN: 978-1-61804-074-9 217