The Influence of the Noise on Localizaton by Image Matching

Transcription

1 The Influence of the Noise on Localizaton by Image Matching Hiroshi ITO *1 Mayuko KITAZUME *1 Shuji KAWASAKI *3 Masakazu HIGUCHI *4 Atsushi Koike *5 Hitomi MURAKAMI *5 Abstract In recent years, location information of people has been attracting a lot of attention as an element for construction of ubiquitous environment. Especially, high precision of that location information is very important in case of emergency. In order to realize such a high precision, technical challenges will be evaluation of the precision of GPS information of mobile phones and how to improve it using mobile phone camera. In this paper, we examine the performance of a developed matching method using correlation between motion picture database and still picture of mobile phone camera. In addition, the robustness of the matching method against unexpected noises due to blurring of mobile phone camera images is evaluated theoretically as well as experimentally. Keywords GPS, Mobile Phone, Camera, Image Pattern Matching, Gauss Fitting. I I. INTRODUCTION N recent years, location information has become an indispensable element of our life. GPS location information is applied to a variety of purposes such as navigation to a destination or the identification of a particular place. To further improve the usefulness of such location information, it is essential to improve the precision of the current GPS systems. For example, in emergency situations such as when a person has fallen, more accurate location information is required. Highly accurate localization can also be said to be one of the present challenges for the construction of a ubiquitous environment. Therefore, as tool for acquiring location information, we recognize that the mobile phone; that anybody can carry for 24 hours a day, is of great importance. It is mandated by the Japanese government that all present mobile phones are equipped with the GPS function and many people now possess mobile phones with GPS 1). A general dedicated GPS has a very good accuracy with an error of about 1m under open sky, but in an environment where Manuscript submitted June 13, HIROSHI ITO is with the Graduate School of Science and Tecnology, Seikei University, Japan (iwtppc06@yahoo.co.jp). MAYUKO KITAZUME, SHUJI KAWASAKI, MASAKAZU HIGUCHI, ATSUSHI KOIKE, HITOMI MURAKAMI (hi-murakami@st.seikei.ac.jp) are with Depertment of Computer and Information Science, the Faculty of Science and Tecnology, Seikei University, Japan. there are buildings and other structures around, the location information accuracy cannot be obtained at all 2)-4). On the other hand, for the mobile phone GPS, even in places where the dedicated GPS cannot get location information, through the use of error correction by the mobile phone base station, location information can be obtained. The mobile phone GPS is less precise than the dedicated GPS but "the probability of obtaining location information" can be said to be high 3)-4). Therefore, the authors propose a method where an image taken by the mobile phone camera is matched with video database of the surroundings buildings, then the position of the photographer is determined taking the correlation between database and the image 5)-7). As a database, the video of representative buildings is taken, and then an image of a building within the same area is taken by the mobile phone camera. By so doing, the image taken by the mobile phone camera will be one of the frames of the continuous video in the database. The similarity between this frame and the mobile phone camera image will be high compared to other frames in the video sequence, so the position of the mobile phone image can be detected in this way. (Hereinafter, this method is referred to as the correction function method.) In fact, localization was performed using this technique, the mobile phone GPS error in buildings which within the area 100m radius narrowed down to 1/193, i.e., 163m2, for at least 78% of the locations. This result suggests that the use of the correlation function method for GPS localization has the possibility of improving the position detection accuracy even further. In this experiment, the database and mobile phone camera images were taken under almost the conditions. The characteristic evaluation of localization precision with respects to changes in sunlight, shift in photographed location, etc. was not performed. Thus in order to determine whether the proposed method is practical, we carried out a variety of characteristic evaluation experiments under a variety of shooting conditions to determine the robustness of the location detection precision and these experiments included the following: (i) shift in photographed location, (ii) shooting accident and (iii) variation in sunlight. In this paper, we define the noise for the changes in the basic conditions under which the database was made, and then theoretically and experimentally investigate the effect of noise location precision in (i), (ii) and (iii). ISBN:

2 II. A METHOD OF LOCALIZATION BY IMAGE MATCHING With the correlation function method, the video database and mobile phone images are taken and then matching is performed. The video database is taken from 10m away from the building while moving at constant speed parallel to the building. Each of the database video and the input image are matched by calculating the correlation coefficients. The matching method is described as follows. Correlation coefficient 1. Each frame of the database video and the input image are converted to grayscale images. 2. Perform edge detection on each image. 3. Calculate the correlation coefficient by Equation (1). Frame Fig. 1: An example of correlation coefficients graph (horizontal direction). Correlation Coefficient where, N:Number of vertical pixels (240 pixels) M:Number of horizontal pixels (240 pixels) f(m, n):mobile phone camera image pixel values (still image) g(m, n):pixel value for each frame of the database video. As for the correlation coefficients, first, the mobile phone camera image is compared with the images in the database, i.e., in the horizontal direction and for a specified location, calculation is performed separately in the vertical direction to a certain depth. 4. For the vertical direction, and for the purpose of performing detailed analysis, the mobile phone image is enlarged using Equation (2) and pattern matching is done. u ( x, y) M 1 N 1 i 0 j 0 u( i, j) ( x i, y j) (2) where the interpolation function is given by ( x, y) ( x) ( y) φ:1-d interpolation function. 5. Perform localization from the experimentally obtained correlation coefficients. Figures 1 and 2 are the correlation coefficient graphs when the input image and the database image are matched. Figure 1 shows the correlation coefficient value on the vertical axis while the horizontal axis represents the frame number of the video. It is plot of the correlation coefficients between each frame and the input image in the horizontal direction. Figure 2 is a plot of the vertical correlation coefficients for the frame which corresponds to the peak value in Figure 1 and the enlarged input image. The vertical axis is the magnification factor and the horizontal axis represents the correlations coefficient. Magnification Factor Fig. 2: An example of correlation coefficients graph (vertical direction). When the input image matches the database location, in many cases a mountain like shape appears in the graph, so it can be estimated that the part where the mountain exists (shaded area in Fig. 1) is the one being photographed. Therefore, the product of the time of the frames in the shaded area and the speed of motion while shooting the video gives the distance, which determines the horizontal range. The same applies for the vertical direction where the matched part appears in the shape of a mountain. Thus, the product of the correlation coefficient graph's magnification factor and the corresponding distance the point being shot, which is equal to 10m, determines the vertical range. It can be observed that from the pattern matching experimental result that when a match between the input image and the database image exists, both the peak value of the correlation coefficients and the average value tend to be high. In fact, this is true for all input images used in the experiment. That is, when there is a match in the photographed location, the maximum correlation coefficient and the average correlation values are high while for a mismatch both are expected to be low. Therefore, we applied multivariate analysis method, the discriminant analysis, such that from the 2-D plane of maximum value of the correlation coefficients and the average value, the region of "match" and "mismatch" of the image could be discriminated. With the x-xis as the maximum value and the y-axis as the average value, a graph is plotted. From this, the boundary line between the two regions, ax + by + c = 0 (the ISBN:

3 discriminant), as in Figure 3, is obtained. The discriminant indicating which side of this line the given points are is Z=ax+ by+ c (3) whose positive and negative values determines the "match" and "mismatch." (The values a, b, c are derived by the least-squares method.) We formulate the following mountain detection method and from the width of the mountain, image localization is done. Peak detection For mountain detection, Gaussian fitting is carried out. The following equation is used for fitting. 2 ( x c) y a b exp 2 d (4) Average Value (y) Mismatch Match Discriminant This time around, the horizontal axis, x, represents the frame number while the vertical axis is the magnification factor. Least-squares fitting is performed to get a, b, c and d. Figure 4 shows the result of performing fitting on Figure 1. Maximum Value (x) Fig. 3: Graph of the maximum and average correlation coefficients. Correlation Coefficient Frame TABLE I: Correct discrimination rate and detection accuracy Fig. 4: Result of performing fitting on Figure 1 using the discriminant. (a= ,b=0.271, c=134.2, d=83.33). Match Mismatch Correct Discrimination Ratio 75.0% 92.0% Detection Accuracy 80.0% 89.6% Localization From the correlation coefficient data obtained experientially is shown in Figure 3, and the discriminant results are shown in TABLE I. Correct discrimination ratio is the ratio which indicates that the photographed place's match and mismatch were correctly determined by the discriminant. The detection accuracy is the ratio that indicates that within the data that has been determined, correctly determined data exists. From the discriminant, when it is decided that there is a match between the database image and the photographed place, and in order to determine the central frame used and range error within which the still image is contained, detection of the correlation coefficients graph s mountain (around the peak) was performed. The detection was done by least-squares fitting using the Gaussian function. A deviation of ±2σaround the mountain peak was chosen. By this choice the image similarity is high enough and the time interval is the one in which a match occurs. In general, the total probability of 68.3%, 95.4% and 99.7% is concentrated in the interval ±σ, ±2σ, ±3σ around the mean, respectively. However, ±σ is so narrow that there is a possibility of misses in detection, while ±3σ is too wide and could result in large errors, so ±2σ was chosen. Using the discriminant line for the data where the correct determination has been made, we next proceed to perform mountain detection and then plot the correlation coefficients graphs as shown in Figs 1 and 2. When the mountain is detected, the width of the mountain can be determined from d, and then the localization range is calculated. For the horizontal graph, the mountain width is the number of frames while for the vertical graph it is the magnification factor. For a given fitting result, the standard deviation can be calculated from d as follows: d (4) 2 This standard deviation and value that represents the center of the mountain c, are used to compute the width of the mountain as c 2. For the horizontal direction the computed mountain width (=frame number), the frame time is computed from the frame rate and the product of the frame time and speed of motion during shooting gives the distance, which determines the horizontal range. Namely, Horizontal range [m] = detected frame time [s] motion speed [m/s]. For the vertical direction, when there is a mountain, the minimum magnification is set to c 2 while the maximum magnification is set to c 2. The product of these values with distance to the object being pictured (=10 [m]), gives the horizontal range, i.e. ISBN:

4 Horizontal range [m] = (maximum magnification-minimum magnification) shooting distance [m]. Let s consider the case where the database and the input image correspond to the same place as in the graph shown in Figures 1 and 2. The fitting result for Figure 1 (horizontal direction) is a= , b=0.271, c=134.2, d=83.33, while for Figure 2 (the vertical direction) it is a=0.0995, b=0.0955, c=1.17, d= These results are from mountain detection. From Fig.1, the value of d can be determined as 58.92, the frame is 30fps, and speed of motion 1.5 [m/s]. The horizontal range can then be calculated as: Horizontal range =4σ[frame] 1/30[s/frame] 1.5[m/s] =11.78[m]. From Fig. 2, σ=0.101 can be determined, minimum magnification is and maximum magnification is From these values, the vertical range can be computed as: A. Shift in photographed location The degradation in matching when there is a shift in the photographed location is validated. Specifically, two patterns are used in the validation, i.e. 1. the case in which the vertical direction changes and 2. the case in which the image of the buildings could not be taken in parallel as previously described. (a) Effect of variation in vertical direction distance at the time of shooting We are working on the assumption that the video database was shot at 10 [m] away from the building. Keep this in mind, we take the mobile phone images at 10[m] and then increase the distance in steps of 1[m]. Matching is then performed and the changes in the detection rate and the average localization area are observed. The images used are shown in Fig. 5. Vertical range = ( ) 10[m] = 4.04[m]. From this result, the product of the vertical range and horizontal range, namely 47.58[m 2 ], gives the location area. This means that the location range can be narrowed down to about 1/650 of that of a circle of radius 100[m]. III. LOCALIZATION OF NOISY IMAGES Distance 10[m] (basic data) Distance 11[m] In Chapter 2, the matching method was validated and its usability was confirmed. In this chapter, we validate the degree to which the localization is affected by changes in the shooting conditions of the video database when he correlation function is used. That is, the mobile phone camera image is taken under various conditions and the robustness of the localization is evaluated. In this validation, three buildings are used, and the following three mostly likely causes of image change are considered: (i) shift in photographed location, (ii) shooting incident and (iii) variation in sunlight. The effect of the changes in these three conditions is evaluated. The database used is the one that has already been mentioned and the input image is the still image obtained under each of the shooting conditions. When the conditions under which the input image and the database image are taken are the same, the detection rate (successful localization) is 91.7%, while the average localization area (the average value of the area detected by localization) is 159.1[m 2 ]. This means that the area is 1/197 of that for mobile phone GPS error which is the area under the circle of radius 100[m]. Using this result as the basis, the detection rate and the average localization area are determined under (i), (ii) and (iii). That is, we validate the effect of noise on localization. Distance 12[m] Distance 14[m] Distance 13[m] Distance 15[m] Fig. 5: Example of image difference by distance As can be seen in Fig.5 at 15[m], as the distance increases, the roof and other parts of the building are captured by the camera. From these kinds of images, the vertical and horizontal ranges were determined and the result the detection rate and the localization area were computed. The results are shown in Figs. 6 and 7. ISBN:

5 From Fig. 6, it can be observed that there is very little change in the horizontal range and on the overall, the result shows no big errors. Average Distance [m] Average horizontal distance Average vertical distance Even so, by using the database video and an image taken at 5[m] from the building, the detection rate is 72.7%, while the localization area is reduced 1/2 of that by the mobile phone GPS. (b) Effect of shift in angle When taking the mobile phone image, the shift of the camera from the parallel orientation has the effect of degrading localization and we perform the validation of this phenomenon. The angle relative to the building was increased in steps of 10 degrees up to a maximum of 40 degrees. Fig. 8 is the image taken in parallel, while Figs 9 and 10 are the images when the angle was shifted. Shooting Location [m] Fig. 6: Average determined distance according shooting distance. Fig. 8: Parallel image shooting (shooting angle 0 o ). Left 10 o Left 20 o Left 30 o Left 40 o Fig. 7: Detection rate and average area by distance differentiation. In the vertical direction, it can be observed that as the distance from the building increases, the determined distance becomes small. As can be seen in Fig.7, as the distance from the building increases, the localization area also becomes small. This is because as one moves away from the building, the image detail increases, and the building characteristics are emphasized during edge detection. This could be due to the enlargement processing in the vertical direction whereby the correlation between the input image and the database becomes high. However, if the increase is too large as is the case at 15[m], obstacles such the roof are also photographed and it has been observed that the localization failure probability increases. That is, as the distance from the building becomes large, the probability of adding a new type of noise from obstacles increases and on the overall, localization precision decreases in some cases. Fig. 9: Shooting angle shifted to the left. Right 10 o Right 20 o Right 30 o Right 40 o ISBN:

6 Fig. 10: Shooting angle shifted to the right. From these images, their respective detection rates and localization areas are shown in Fig. 11. We investigate the extent to which the camera shake affects matching characteristics. The method used is to shake the hand during shooting until the image blur can be clearly visible and then perform matching. Figure 12 shows the images without blur while Fig. 13 shows blurred images. Fig. 12: Images without blur. Fig. 13: Blurred images. Fig. 11: Detection rate and average localization area according to the shooting angle. As can be understood from Fig. 11, as the angle increases, localization becomes almost impossible. Shifting the angle of the image by merely 10 degrees reduced the detection rate to 13%, while a shift of 30 degrees and above result in detection rate of zero. As can be seen from Figs 9 and 10, when there is a shift in the angle, the positional relationship of the recognizable marks (pillars, etc) of the building are greatly shifted. This results in a shift in the features of the image that are used for edge detection, and image could be recognized as a different image during matching. Therefore, for images taken at an angle to the building, localization by matching is difficult. B. Shooting accident We investigate the matching characteristics when an accident occurs during image shooting. This results in noise that doesn t exist in the database being added to the image. Specifically, we consider 2 categories of noise, i.e., 1 image blur due trembling of the hand (camera shake) and 2 photographer finger obstructing the image, a person or other obstacles getting into the camera s view, etc. (a) Effect of camera shake (b) Obstacle effect When taking a picture, obstacles sometimes get into the view of the camera. Two cases that are especially common are the photographer s finger and a person passing by or other objects obstructing the camera s view. We investigate the matching characteristics of these two cases. Cases when the finger obstructs the image and when a person gets in to the camera s view are shown in Figs. 14 and 15 respectively. The impact of these obstacles is largely dependent on the total area of the image that they occupy. Therefore, for each photo, the area occupied by the obstacle is denoted by (A/B). Little finger (~1/8) Half finger (~1/2) Fig. 14: Cases of obstruction by the finger. Far (~ 1/15) Middle (~ 1/5) Near (~1/2) Fig. 15: Cases of obstruction by a person. ISBN:

7 Figure 16 shows the detection rate and the localization areas when camera blur and obstacle noise are included. Compared to the no noise image, the camera blur has a detection rate close to 70%, which is considerably higher than for images with obstacle noise. Additionally, the determination area is almost the same as that for the image without noise. A: The side of Seikei University's Building No. 8 B: The front Seikei University's Building 10 C: The front Seikei University's Building 8 Photographs of each of these buildings were taken. Even for the same building, due to sunlight and shadows, it can be seen that a completely different picture is obtained. A B C Fig. 17: Images taken at 10 hundred hours. A B C Fig. 18: Images taken at 13 hundred hours. Fig. 16: The detection rate and average localization area for the shooting accident. When an obstacle gets into the camera s view, for little finger (~1/8) and far person (~ 1/15), there is a very small reduction in the detection rate compared to the no noise situation. Even so, the detection rate is over 50%, which is within the usable range. However, for near (~1/5 - ~1/2) the detection rate is noticeably reduced. That is, when the obstacle area is big, the degradation in detection is also big. In the case of half finger (~1/2), the localization area is over 50% higher than when there is no noise. Even so, the error is about 77.5 m2, which is lower than the area of a 100 [m] radius circle for the GPS. For localization area, it can be said that the obstacle noise has little effect. C. Effect of variation in sunlight. To investigate the effect of sunlight changes, the mobile camera images were taken at 3-hour intervals on the same day. Also, in order to clearly observe the effect of sunlight changes, a day with no clouds, i.e., totally clear sky was chosen. Thus it is safe to say that the effect of lighting on the images was strong. The images taken are shown in Fig. 17, 18 and 19. Since the direction of the sunlight changes with time of the day, we not only observed the variations with time but also observed the changes with different photographed locations. The different photographed locations are shown as A, B and C. The places A, B, C, were buildings with different outward appearances. A B C Fig. 19: Images taken at 16 hundred hours. Table Ⅱ: Brightness categories. 10Hr-Place 10-A 10-B 10-C Sunlight strong shadow strong 13Hr- Place 13-A 13-B 13-C Sunlight shadow strong strong 16Hr-Place 16-A 16-B 16-C Sunlight weak weak weak ISBN:

8 as image noise and performed experiments to evaluate the effect of the noise on localization accuracy. The results can be summarized as follows: Fig. 20: The detection rate and average localization by brightness. Photographing time (3 time slots) X photographed places (3 places) gives a total of 9 patterns. The sunlight in these places was divided into three categories, i.e. "strong" "weak" and shadow. For each pattern, "shooting time-shooting place" was specified. For example, "10-A" denotes the shooting time of 10 o'clock at place A. Table 2 shows the contents of this information. Figure 20 shows the results of detection rate and the localization area for these nine patterns. From Fig.20, the detection rate for 10-B, 13-A and 16-C was about 70%, while the rest of the patterns were around 50%. The images that had high sunlight content showed less detection rate than the shadowy ones. This means the absence or presence of sunlight makes a difference. On the other hand, for strong and weak sunlight, there was not much of a difference the detection rate and localization area. As with camera shake, edge detection changes the whole of he image such that the effect on detection rate is small. In addition, all the localization areas were within 200 m2 so there was not much of effect by this noise. IV. CONCLUSION Presently, the mobile phone, which is the most widely used communication terminal that can be carried by anybody 24 hours a day is a necessary component of the ubiquitous environment. Especially, terminals equipped with both the GPS functionality and a camera would be very useful for the further development of technology for localization. By building a video database and matching the mobile phone camera s still image the database, more detailed location information can be obtained compared to when the GPS is independently used. On the other hand, it can be predicted that when the mobile phone camera image s shooting conditions are different from those of the database, degradation in the precision of localization occurs. These changes in conditions include (i) shift in photographed location, (ii) shooting accident and (iii) sunlight changes. Thus, we defined these changes in conditions A. Shift in photographed location In the horizontal direction, if the disturbances are within the database range, then it is possible to perform high precision localization 13. In the vertical direction, even if the distance to the pictured building is large, normally a detection rate of 91.7% can be obtained. This means that there is not much of a reduction in the precision and moreover the lowest detection rate was about 72%. Localization was also found not to be affected. However, when the shooting angle was changed there was a noticeable reduction in the detection rate. From the above observations, it can be said that as long as the buildings can be photographed in parallel and within the range of the database, high precision localization is possible. B. Shooting accident Camera shake results in a reduction in the detection rate but even so, a rate of close to 70% was obtained. Localization was mostly unaffected. This noise can be said to have no effect on localization. However, for obstacle noise, as the area of the image occupied by the obstacle increases, the detection rate decreases. For example, when the area is about 1/15, the detection rate is about 58%, while for 1/5 the detection rate falls to 27%. Thus, the detection rate is proportional to the obstacle area. When the finger occupies half of the image area, localization area becomes large but in other cases, there was very little difference. From the above, it can be said that as long as the obstacle occupies less than half of the image area, the correlation function method can be used for matching. C. Sunlight changes A detection rate of 50% or more was maintained for various changes in sunlight and shadow. The location area was less than 200 m2. As in the case of camera shake, the proposed correlation function method showed a very high robustness to changes in lighting. From A, B and C, a lot of knowledge has been gained about how the changes in shooting conditions and noise affect localization by the correlation function method. From detection rates close to 70% in practical noisy cases to impractical zero detection rates; the results demonstrated the usability of the method. Moreover, on success detection, all the results showed a localization area below 200 m2, which is approximately 1/157 of that by mobile phone GPS at a error range of 100m radius. This demonstrates the superiority of the proposed method. ISBN:

9 ACKNOWLEDGMENT A part of this work was supported by MEXT Grant-in-Aid for Building Strategic Research Infrastructures. We are very grateful for their support. References 1) Year 2009 WHITE PAPER Information and Communications in Japan: 2) Bernhard Hofmann Wellenhof, Herbert Lichtenegger, James Collins, Global Positioning Systems, Spriger Japan ) Hirano, et al., Reduction of GPS information error, The Institute of Image Information and Television Engineers (ITE), 2008 ITE Winter Symposium ) Tukada, et al.: Performance Error in the GPS Available on Mobile Phones, ITE Technical Report Vol.33, No.11, pp.29~32, Feb ) In et al., Location Identification by GPS and Images of Mobile Phones [in Japanese], ITE Technical Report, Vol33, No.11, pp.25~28, Feb ) Hirano, et al., Construct of ubiquitous environment by integration of camera and GPS of mobile phones and map information [in Japanese], ITE Technical Report, Vol.32, No.39, pp.1~4, Sep ) Kitazume, et al., GPS Precision Improvement System by Mobile Phone Camera Images, ICMU2010 ISBN: