Mixed Reality Approach and the Applications using Projection Head Mounted Display

Transcription

1 Mixed Reality Approach and the Applications using Projection Head Mounted Display Ryugo KIJIMA, Takeo OJIKA Faculty of Engineering, Gifu University 1-1 Yanagido, GifuCity, Gifu Japan phone: , fax: Abstract The aim of this study is to explore the feasible technology for the multiple display environment that is inserted in the real world. In this paper, the wearable projector with an infrared camera and a light source is used in combination with the retro-reflective screens. The visible image and infrared light are projected to the screen and reflected back to the user's eye and the infrared camera. The screen location and the use's finger tip position are calculated using the image processing to enable the interaction. Several example applications or demonstrations are constructed to show the usage of this configuration. The screens function as both the visible screen and the high gain infrared marker, or supprting equipment of input at the same time. Owing to the high reflection gain of retro-reflective screens, a small, light weight projector can be used. Also the high contrast between the screen and the other environmental object in the captured infrared image decreases the difficulty of the image processing. KeyWords: Augumented Reality, Real World Oriented Computing, Ubiquitous Computing, Mixed Reality, Projection Head Mounted Display, Image Processing 1. Introduction 1.1 Sensing for Mixed Reality The sensing issue is one of the essential problem for the Mixed Reality in terms of the feasibility of this technology. The magnetic sensors such as Polhemus FASTRAK or Ascension Bird has been widely used, to measure the location of the user s head for the rendering and the location of the hand or input devices for the interaction. They are generally used within the laboratory environment and difficult to apply in the larger area of activity in our daily life. There have been many efforts to develop the alternative sensing devices, such as ultrasonic sensor, sensor fusion using gyroscope, electric compass, etc., and vision based sensing. However, they seem to be far from maturity and there is a room for the further investigation. The authors approach is to combine the display and sensing in one using the projection head mounted display (PHMD)[12] or the wearable projector, the retroreflective screen(s), and image processing. V02-1

2 1.2 Inserting Virtual Information to Object The Mixed Reality often means the superimposing the generated image over the whole user s vision using optical or video see-through head mounted display. While this type of approach can provide the freedom and flexibility, the displayed information seems to be related strongly to the object or the location in most of the cases. Therefore, the authors are focusing to give the additional graphical information to the specific objects[6] or the locations. The user wears a light weight wearable projector (PHMD) and an infrared camera with the light sources. The PHMD projects images only on the screens. At the same time, the infrared rays are irradiate to capture the image of the screen. Using the image processing, the computer calculates the location of the screen relative to the user s head as well as the fingertip position on the screen for the interaction. While this approach requires to attach the screen in advance where the information will be displayed, the screen with high reflection gain decreases the difficulty of sensing using image processing. 2. Proposed Configuration 2.1 Projection Head Mounted Display (PHMD) The Projection Head Mounted Display (PHMD) is relatively recent technology. The principle of the PHMD was proposed with the first prototype by the first author in 1995 [3][4]. This prototype is used for an application to layout and design the furniture of the room, based on the idea of moving the object among the multiple spaces and enable the seamless connection of the spaces. This idea is similar to that of Pick-and-Drop [8]. In a desktop type virtual environments on the CRT, a shape modeling application was running, and the user also exists in the surrounding virtual environment with PHMD. In the surrounding VE, the user can grab and change the location of furniture using a flying mouse (both for 2D and 3D). If a furniture s shape is not his taste, the user grabs it in the surrounding VE, and push it into the CRT monitor. Then the furniture appeared in the shape modeling application on the CRT and the user modifies the shape. At last the user grabs it with the same flying mouse, and draws it back to the surrounding VE. Afterward, the retro-reflective material was introduced to enable the stereoscopy for the projection display [5] in 1996 and soon the PHMD is used in combination with the retro-reflector [6][7]. The fundamental principle of the PHMD is the optical conjugation between the user s eye and the optical center of the projection. Due to this conjugation, the image is projected from the user s eye position and seen from the same position. This arrangement enables the user to watch the image without distortion when the screen has angle to the line of sight or even the screen is bent, because the viewing transformation is the inverse of the projection transformation and the distortion derived from the screen s shape is canceled (Fig.1). The author has developed a series of PHMD as shown in Fig. 2 - Fig. 6. One of the essential issue to make the small projector was the power-weight and power-heat ratio of the light source. A halogen lamp of five watts used for Rev.1 causes the overheat problem of the LCD panel even with the heat protection using a hot mirror and a heat absorber. The light source is divided from the projection head and they are connected by optical fibers in Rev.2. In Rev.3, the current prototype, hi-luminance white LED block is attached to the projection head. The projection head for the each eye was composed of an LCD panel (Sony LCX009AKB, 0.7 inches, 800x225 pixel), a projection lens (18 mm diameter) and the light source. The LED block generated the luminance of approximately 250 millicd/mm2 and was enough for the use within the room, but not for outdoor usage. The weight of this PHMD is about 150 grams. V02-2

3 LCD Projection Lens Eye Half Mirrot Screen (Left) Fig.1 Pribciple of PHMD (Middle), Fig. 2. Confirming the Optical Design (PHMD Rev. 0. Kijima, Oct 1998) (Right), Fig. 3. Head Mounted Configuration (PHMD Rev. 1. Kijima & Haza, Mar. 1999) Light Source Half Mirror Mirror Projection Head (Left) Fig. 4. Light Source is Divided for Heat Proection (PHMD Rev. 2. Haza & Kijima, Jul. 1999), (Middle) Fig.5. Current Prototype in Stereoscopic Configuration (Rev. 3. Haza, Miwa & Kijima, Dec. 1999), (Right) Fig.6 Projection Head The PHMD became more practical by the introduction of the retro-reflective material as the screen. This material reflects the incoming light back to the direction of the light source. The image is projected from the user s eye position to the reflector and goes back to the eye position. Almost all the energy from the projector goes back to the eye and there is only minimal loss. Therefore, small, light weight, dark projector can be applied as the PHMD. Another merit is the capability of stereoscopy. The image projected from one eye goes back to the same eye and does not reach to another eye. Then each eye of the user can see the different image from the corresponding projector. The retro-reflective screen is the cloth covered with small glass beads. This is the cheap material that is widely used for the road signs. Many displays can be attached easily in low cost. Also this material can be folded and stored in a pocket like as handkerchief Vision Based Measurement using Infrared Camera To achieve the relative location of the user s head to the screen, an infrared NTSC CCD camera and the infrared LCDs is attached to the PHMD, and the vision based measurement was applied. When the original shape was known, the screen s location relative to the user s head can be calculated from the image of the screen. The screen served as both the visual screen for the user and the anchor of the vision based measurement at the same time. Generally speaking, the method to control the environment, other than the image analysis algorithm, is important for the actual application. Since the calculation power of the wearable configuration is limited, the clever way to achieve the clear image of the necessary target is required. V02-3

4 Fig. 7. Differential Image Capture: (Top) Original, (Bottom) Binary Image Infrared Light is, (Left) ON, (Right) OFF Screen Verteces of Convexed Polygon Approximation Screen Edge Region 2 Region 3 Region 1 Diagoal Line Region 4 Resulting Vertex Initial Vertex Fitted Line Fig. 8. Detection of Screen Edge and Screen Verteces: (Top Left) Original Image, (Top Right) Approximation to Convex Polygon, (Bottom Left) Shrunken Quadrangle and Division, (Bottom Right) Line Fitting by Mean Suared Error Method The first device is that the LED and camera were optically conjugated with each other. Due to the high gain of retro-reflector, the small, low power LED can serve enough as the bright light source to achieve the clear image of the screen. The next idea was the differential image calculation. The infrared light was turned on and off synchronized to the frame and the difference of images was calculated. Thus the background noise was decreased largely and the high contrast image of the screen was achieved. The following is the brief description of the image processing. At first, the edge detection was performed from the captured image and the longest one was regarded as the outline of the screen. After compensating the distortion of the camera, 12 lines at every 15 degrees was circumscribed to this edge. The shape of screen was approximated as a convex polygon with 24 vertices. The number of vertices was reduced to the quadrangle, referring to the distance between the neighboring vertices and the angle of neighboring edges. Thus the first approximation of the quadrangle was achieved. Next, the image was divided into four regions using by the diagonal lines of this quadrangle. Using the minimum squared error method, four lines were fitted to edges in each V02-4

5 region. At last the screen s location relative to the user s head was calculated. This calculation is robust when a part of the screen is hidden by the user s hand, because this is based on the edge information, not the vertices' information. In addition, when the user places a finger on the screen, it is seen as the dark area in the bright screen. The second order moment of finger image is estimated and fingertip position is calculated. The user can interact with the displayed object with their fingertip. For example, the user can draw the graphics on the screen with their fingertip. The screen represents the drawing plane in the three dimensional spaces and the user can move it with the other hand, then the spatial object can be drawn. 3. Applications and Demos 3.1 Distributed Display Fig. 9 shows the demonstrations of the omni-display or distributed displays environment in our daily life. Top Left figure shows the virtual window demonstration. A video camera was placed outside the laboratory building, and displayed on the retro-reflective screen on the wall. According to the user s location, the motion parallax was realized by the simple image based rendering. Fig.9 (Top Right) shows the price tag application. A small tag of reflector was attached to the product in the shop, and the current price is projected on it. Fig.9 (Bottom Left) shows the calendar application. The calendar itself was the reflector without any print. The date was displayed on it. The user can draw his plan by using the fingertip operation on the calendar and call it back on by the fingertip. Fig.9 (Bottom Right) shows the internal state application. A reflector was attached to the back of the user s hand. According to the heart beat and blood pressure, the graphics of the blood vessel was animated. Projected Window Virtual Price Tag Projected Real Fig. 9. Distributed Display Demos: (Top Left) Projected Virtual Window on Wall Showing Scene Outside, (Top Right) Virtual Price Tag Attached To Product, (Bottom Left) Projected Calender with Schedule Board Functionality, (Bottom Right) Visualized Brood Vessel V02-5

6 3.2 Screen Location Measurement Fig. 10 is the demonstration application to show the active use of the screen location measurement. The sliced image of human brain using the MRI data was displayed on the screen. As the user held the real screen and moved it, the slicing plane in the virtual world was controlled by the location of the real screen. Thus the user can see various slices of the brain. Also the user can draw the graphical memorandum on the sliced image by their fingertip. Fig. 10. Application: Interactively Slicing MRI Brain Image with Hand-Held Screen 3.3 Medical Apprications Fig. 11 shows the appearance of the medical applications. The left figure shows the user is touching the mockup of human s knee with the projected internal structure. This mockup was made of the stiff bones and soft organic structures with skin. The user can touch with the hand feeling to confirm the position of bones, and can see the internal structure on the real mockup. Also, the user can select the projected image such as the whole structure, bones and tendons, blood vessel and major nerves, etc. The middle figure shows the appearance of the endoscope simulator. The user can insert the toy endoscope to the mockup of the knee, watching the projected endoscope immersing the internal structures. On the other monitor, the simulated image from the virtual endoscope was displayed. Probe Stomach Echo Image Stomach Fig.11. Medical Simulator (Left) Touching themodel of Knee with Internal Structure Displayed (Middle) Endoscope Simulator (Right) Ultrasonic Echo Image Projected on Stomach V02-6

7 The Polhemus sensors were used to measure the location of endoscope. The right figure shows the display of the ultrasonic echo image projected on the stomach. The infrared camera on the user s head measured the location of the ultrasonic probe by detecting four visual markers attached to it. The user can watch the echo image where the echo image was taken. 4. Summary A novel configuration of Mixed Reality using wearable projector with an infrared camera was described, focusing to attach the graphical information to the object and location. A small, light weight wearable projector was constructed and used with a retroreflective cloth screen. The infrared camera was attached to the PHMD, and the movement, location of both of the user's head and fingertip by the image processing, are used for the interaction. Several applications were constructed to show the example usage. Since this configuration is autonomous in terms of sensing, it can be used in larger space of our daily life, not limited to the space of a laboratory. References 1. Wisneski, C., Ishii, H., Dahley, A., Gorbet. M., Brave., S., Ullmer, B., Yarin, P., Ambient Displays: Turning Architectual Space into a Interface between People and Digital Information, Procs of the 1st Int. Workshop on Cooperative Buildings, pp , Springer, Pedersen, E. and Sokoler, T., AROMA: Abstract Representation of Presense Supporting Mutual Awareness. Procs. of CHI 97, pp , ACM Press, Kijima, R. and Hirose, M., A Compound Virtual Environment Using the Projective Head Mounted Display Procs of ICAT/VRST '95, pp , ACM-SIGCHI, Kijima, R., Ojika, R., Transition between VIrtual Environment and Workstation Environment with Projective Head-Mounted-Display, Procs. of IEEE Virtual Reality Annual International Symposium 1997, pp , IEEE, Ishikawa, J., 3D B-VISION, Journal of three dimensional image, 10-4, pp.11-14, (In Japanese) 6. Inami, M., Kawakami, N., Sekiguchi, D, Yanagida, Y., Maeda T., and Tachi, S., Visuo-Haptic Display Using Head-Mounted Projector, Procs of IEEE Virtual Reality 2000 Conference, pp , IEEE, Hua, H., Gao, C., Biocca, F., Rolland, J., P., An Ultra-liht and Compact Design and Implementation of Head Mounted Projective Displays, Procs. of IEEE Virtual Reality 2001 Conference, pp , IEEE, Rekimoto, J. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments. Procs of User Interface Software Technology (UIST 97), pp.31-39, ACM, Ullmer, B. and Ishii, H., "The metadesk: Models and Prototypes for Tangible User Interfaces," Procs of User Interface Software Technology (UIST '97), pp , ACM, Rekimoto. J., Nagao. K., "The world through the computer: Computer augumented interaction with real world environments", Procs of UIST 95, pp.29-35, ACM Fitzmaurice, G. W., Zhai, S., Chignell, M. H., "Virtual rea;lity for palmtop computers", ACM Transaction on Information Systems, 11-3, pp , ACM, Kijima, R., "Wearable Interface Device", Procs of Human Computer Interaction 2001, Vol.1, pp , Kijima. R., Yamada, E., Ojika., T., "A development of Reflex HMD - HMD with time delay compensation capability-", Procs of International Symposium on Mixed Reality 2001 (ISMR2001), pp.40-47, VRSJ, V02-7