True Single View Point Cone Mirror Omni-Directional Catadioptric System 1

Transcription

1 True Single View Point Cone Mirror Omni-Directional Catadioptric System 1 Shih-Schön Lin, Ruzena ajcsy GRASP Laoratory, Computer and Information Science Department University of Pennsylvania, shschon@grasp.cis.upenn.edu, rajcsy@nsf.gov (also, ajcsy@grasp.cis.upenn.edu) Astract Pinhole camera model is a simplified suset of geometric optics. In special cases like the image formation of the cone (a degenerate conic section) in an omnidirectional view catadioptric system, there are more complex optical phenomena involved that the simple pinhole model can not explain. We show that using the full geometric optics model a true single viewpoint cone omni-directional system can e uilt. We show how such system is uilt first, and then show in detail how each optical phenomenon works together to make the system true single viewpoint. The new system requires only simple off-the-shelf components and still outperforms other single viewpoint omni-systems for many applications. 1. Introduction With the ideal pinhole camera model in mind, it has een shown that conic section convex s can e used to create single viewpoint catadioptric omnidirectional view systems that can e unwarped perspectively without systematic distortions [1]. It has also een shown that cone is a degenerate shape of the conic section family that still possesses a single viewpoint [1]. The prolem was that, with a true pinhole camera the single viewpoint of a cone (located at its tip) seems not useale ecause all the visile rays seem to e locked y the cone itself [1], while if the virtual pinhole was placed at some distance away from the tip of the cone, the whole system no longer possesses a single viewpoint [10;1;8]. With this prolem in mind, cone s have not een used to generate precisely unwarped perspective images. However, cone s have een used to aid navigation, collision avoidance, and pipe inspections without single viewpoint [9;10;2;6;8] In those applications, cone images are used as is, and no attempt was made to perform a perspectively correct image unwarping. A cone is also used to construct mosaics [5], ut that requires a special line scan camera, time consuming scanning, and mechanical moving parts. In real world applications we use finite aperture lens systems to simulate the projective properties of a theoretical pinhole camera. The real cameras have more useful properties that a pure pinhole lacks. We show in this work that under the geometric optics image formation model, the single viewpoint of the cone is actually usale, i.e. we can physically place the effective pinhole position at the single viewpoint and still get the exact true single viewpoint image we need. The geometric optics image formation model is more powerful than pinhole camera model, i.e. it is a more accurate description of what really happens in the physical world. For example, when an oject is placed inside the front focal point of a real convex lens, it forms an enlarged virtual image. This is completely unexplainale if you treat the lens system as a perspective pinhole. A perspective pinhole camera always gets real images regardless of the oject distance and image plane distance, and it can never e out of focus. aker and ayar [1] used a simple optical model to analyze some omni-s ut they did not use any optical modeling on cone s. This turns out to e the key deficiency in modeling the cone /perspective camera comination as we will elaorate shortly. The cone system proposed here uses only off-the-shelf ordinary lens and CCD camera. The cone is among the simplest shape to produce, and it has much higher resolution for scenes around the horizon [8]. It adds the least optical distortion to the resulting image ecause it is the only omni-view with a longitudinally flat surface. All other omni- types are curved in oth longitudinal and lateral directions. The aility to pair with readily availale perspective camera instead of expensive and complex orthographic cameras is also a very important advantage over existing omni-view 1 This work has een supported in part y SF-IIS , ARO/MURI-DAAH , SF-CDS , DARPA-ITO-DAT , and Advanced etwork and Services. Special thanks to memers of MOOSE Project: V. J. Kumar, Kostas Daniilidis, Elli Angelopoulou, Oleg aroditsky, and Geoffrey Egnal for their invaluale comments and support. Also grateful thanks to memers of GRASP La for their wonderful feedack and help.

2 systems. For example, when you want to uild an omnisystem outside the visile range, it is often the case that only perspective cameras are availale. That is an added reason why we go through all the troule to prove and uild a cone omni-directional camera system. 2. True single viewpoint cone omniview system As proved y aker and ayar [1], when the viewpoint of a perspective camera coincides with the tip of the cone, you have a single viewpoint omni-view system. The only question then is whether or not you can see an image in such configuration. The COPIS y Yagi [10] did not put the lens camera viewpoint at the tip of the cone ecause of this concern and thus did not form single viewpoint omni-system. It turns out that we can indeed see an image in this single viewpoint configuration. We put the viewpoint of our lens camera right at the tip of the cone and have a working true single viewpoint cone omni-view system. See Figure 1(geometry), Figure 2, Figure 3(full models) and Figure 8(real experimental setup). of lens camera a image plane Figure 1 True Single Viewpoint Cone Mirror Geometry In geometric optics image formation model [3], plane is an optical system y itself. The cross section of a cone through its axis of rotation consists of two plane s joined at the tip of the cone. The image formation property of a plane is that it always forms virtual images ehind its surface. otice that the right side does not actually extends to point, ut that does not prevent the virtual image from forming. Within geometric optics, the position of A M the virtual image point does not change, either. As long as there is some portion of the plane (ideally infinitely large) is occupied with piece(s) of real, the virtual images corresponding to any world point on the front of the plane will e formed on the ack of the plane. The positions of the virtual images are not effected y the size of the actual, ut its visiility will depend on the size and position of the actual. Since the virtual images are formed y the real plane s, the second converging optical system must see the virtual image point through the real plane. One can regard real plane as a window to look at the virtual world on the other side of the. The surface of the cone, after it accomplishes the task of forming a virtual world ehind it, acts as a window opening for the 2 nd converging optical system, i.e. the perspective lens camera, to look into the virtual world. So the cone is not locking our view, it IS the view itself. Whether the lens camera has its effective pinhole at the cone tip or not, the lens camera sees the virtual image points just like any other real world point. These virtual points get imaged perspectively the same way as if there is a real point there. In Figure 1, the virtual images a and formed y the catoptric system are imaged y the dioptric system to form real images α and β respectively. As proved y aker and ayar [1], if we can put the effective pinhole of the lens camera at the tip of the cone and still see image perspectively we have a single viewpoint omni-view catadioptric system. We have just shown that we can do this. We put the effective pinhole of the lens camera right at tip of the cone and construct a true single viewpoint cone omni-directional view catadioptric system. See Figure 8. Full geometric optic ray tracing for the cone comined with Gaussian optics ray tracing for the lens camera will yield exactly the same results we just showed. otice here in geometric optics the term ray tracing has very different meaning than the same term used in computer graphics literatures. Computer graphics ray tracing traces only one ray for one world point while geometric optics ray tracing traces no less than 2 rays (ideally infinitely many) for one world point. The computer graphic ray tracing is a simplified way of finding image position valid only for focused real images. Analyses ased on tracing single rays alone can not model how the single viewpoint of cone omniview system works. Since we need to trace many rays for each world point, we trace only point in Figure 2 and Figure 3 to avoid confusion (too many rays in one small graph if we trace oth points). Interested reader can trace point A and find it imaged exactly at the image position α.

3 In Figure 2 the tip of the cone is cut out a little so the cone tip is not physically present. This does not alter the single viewpoint geometry. Only the rightness of the single viewpoint image is reduced a little. The two configurations (Figure 2 and Figure 3) yield exactly the same image for scenes points that are in focus vertically (geometry-wise, the physical focus settings are different). We shall explain details in the following section. 3. System Characteristics 3.1 Perfect vertical imaging Curved surfaces, do not form exact image points for all scene points. For example, paraolic forms perfect image only for points at infinity, while hyperolic forms perfect virtual image only for one point, its outside focal point. aker and ayar [1] has good analysis of vertical defocus lur caused y vertically curved surfaces. ecause cone is the only vertically flat omni-view, the cone ased omni-view device is the only device free of vertical defocus lur. of lens camera #1 ack focal point image plane lens iris O is projected to I via ray 2. In image space the two image positions are always the same ecause the lengths CH and VI are always the same. The lengths are the same ecause under Gaussian optics conditions ray 1 goes parallel to VW after refracted y the lens and will hit point I when the oject is in-focus. Here the conceptual camera at F does not produce any physical image ut the real image seen y camera at C will e exactly the same since they are 1:1 orthographic projection of each other. Gauss developed the theory for paraxial rays only and founded Gaussian optics [3]. Within Gaussian optics all the rays that come from the same oject point in the world and enter the focused optical system converge at the same point. This is an approximation ut in practice is very close to perfect in many situations. Given that, trace any two such rays from a world point and we can find its image point. of lens camera #2 front focal point f lens real image plane (CCD position) Virtual image plane So Figure 3 Optical ray tracing using front focal point of the lens as its virtual pin-hole Figure 2 Optical ray tracing using lens center as virtual pin-hole O Oject Lens(Aperture) ack Focus Screen (CCD Chip) 3.2 Dual effective pinholes for lens camera This applies to most ordinary perspective lens cameras that are in focus. In Figure 4, with a conceptual perspective camera with viewpoint at F and image plane at C(perpendicular to the axis WV), an aritrary oject point O is projected to H via ray 1. While with a second conceptual perspective camera with viewpoint at C and image plane at V(perpendicular to WV), the same oject W F Front Focus (Pin Hole#2) H C Lens Center (Pin Hole #1 I Image of Oject Figure 4 Thin lens geometry implies dual effective viewpoints V

4 For convenience, we usually pick the two most easily traced rays. 1. The ray that passes through the front focal point will e refracted y the lens and then advances parallel to the optical axis until it hits the image plane. 2. The ray that passes through the center of the lens will pass straight through unaltered. The Gaussian Thin Lens formula:[3] can e derived directly from the same similar triangles we just used to derive the dual effective pinholes. Equation 1 Gaussian Thin Lens Formula s + o s = i f where s o oject distance, s i is image distance, and f is focal length. The term focal length may cause some confusion as many computer vision literature uses focal length to mean the distance from the viewpoint to the image plane. The term focal length in Gaussian optics is a lens parameter that remains fixed while we change the position of image plane. The two conceptual cameras has different oject distance / image plane distance parameters ut their resulting images are the same. This is why we can use the setup of either Figure 2 or Figure 3 and get the same results. The choice of effective pinhole effects only the value of s o, s i, and f, ut not the pixel position of any image point (compare Figure 2, Figure 3, and Figure 4). Krishna and Ahuja [4] also use the front focal point as the viewpoint for their panoramic imaging device. 3.3 Image rightness and visiility Since the profile of a cone consists of two plane s, each point inside the cone should e the virtual image point of two world points simultaneously, one on each side, see Figure 5. In practice, we only see images on the same side of the. Figure 5 shows clearly that the point is also the position of the virtual image of point on the other side. ut in the same figure we can also see that very few rays from point can actually reach the image plane ecause of occlusion. Also, due to the symmetry of the geometry, for a point that is to the right of the central axis, there is always more area of the on the right side (from tip of the cone p to point R) than the area of on the opposite side (from p to L) that is useful in terms of imaging. So we always see instead of in practice ecause the image intensity of is far weaker than that of and thus the image of always dominates in practice. of lens camera #1 L ack focal point image plane R lens iris Figure 5 World points on the opposite side form extremely weak images which are not visile in most practical situations FOV with cone 16.7 /2 cone edge Cone Omnisystem Vertical FOV cone tip 53.3 / / /2 Real Perspective Camera FOV without cone Cone Mirror Figure 6 Cone Shape and Vertical FOV Mirror Image of Half of the Camera 4. Unwarping algorithm Unlike most existing omni-view system, the vertical field of view for cone omni-system is not continuous across the zenith. As shown in Figure 6, if the tip of the cone sutends an angle α, and the normal angular FOV(Field Of View) of the perspective camera is φ, then the upper limit of viewale elevation is α-π/2. The elevation here is defined to e 0 at level, 90 degree straight up and -90 degree straight down. The same as the system used for a gun turret. The lower ound of viewale elevation is α-(π/2)-(φ/2). When unwarping, assume the camera is perfectly positioned and lined up, we estalish a 2D image polar coordinate system. For a given azimuth and elevation

5 angle in the unwarped view, the azimuth match the polar angle directly. The polar radius variale r is related to the elevation as Equation 2 r = f p tan( α π 2 θ ) where f p is the effective focal length(image plane to viewpoint distance) of real camera in real camera pixels(i.e. the pixel unit in the original omni-view image, not the pixels in the unwarped image) and θ is the elevation angle of a point in the unwarped view we want to create pix^2 in the horizontal direction and pix^2 in the vertical direction. The square root of mean squared distance differences is pix. This is equivalent to 0.5 degree in view angle. Figure 10 Cone and Paraolic placed side y side; Right: Test pattern 5. Experiments Figure 7 Cone prototype. Right: Overview of the prototype setup. Figure 11 Omni-views, Left: Cone/perspective, Right: paraolic/orthographic Figure 8 The tip of the cone actually coincides with the effective pinhole. Right: cone omni-view seen y the camera. The left side is a test pattern with dots 4 apart. Figure 9 Actual perspective image taken y real perspective camera (left) The unwarped perspective image from the aove looking at the same test pattern is shown at the right. To demonstrate the aility to correctly unwarp image into perspective view, a test pattern of rectangular grids is imaged in a omni-view picture. The right picture in Figure 8 shows the omni-view containing the grids. Figure 9 shows the unwarped result compared to an actual picture taken y a normal perspective camera. There is no visile radial distortion in Figure 9, which means the radial distortion within our FOV is very small. Tsai s algorithm is suitale for compensating small radial lens distortions. [7] For the pattern of 45 points grids, the est match etween the two pictures has mean squared error of Figure 12 Unwarped images. Left: Cone, more stripes preserved. Right: Paraolic, stripes tend to merge and not separale due to lower vertical resolution. To demonstrate the high vertical angular resolution we put a custom made integrated paraolic/orthographic system along side the cone prototype and let them look at the same color foam pattern placed aout 1m away. In traditional non-omni cameras, there is a preferred direction one can move the camera closer to the scene to improve resolution, ut in our omni camera case there is no special direction to move closer. If you move the camera closer toward one direction, the scene in the opposite direction ecomes farther at the same time. In other words, cone gets high resolution in all direction easily in normal environment while other omni cam system gets high resolution only in very confined space where everything is close y. The lens used is COSMICAR 6 mm 1:1.2 TV lens stopped down to f/11 or f/16. A SOY XC-77 camera is connected through a video ox model PS-12SU made y CHORI AMERICA. If your lens cannot get proper focus within normal adjustment range, using extender rings and/or use smaller aperture settings. Small aperture setting in a dark room may need to digitally enhance the rightness of the image. Some lens set has effective

6 viewpoints surrounded y light shield which locks most FOV. Avoid using such lens or remove the light shield efore use. 6. Discussions The cone is the only system that can get perfect image vertically when you use a perfect lens system. All other longitudinally curved s introduce defocus lurs efore the light enter any lens system [1]. So even with a perfect lens system you can not get a perfect image in those s. In the rotational symmetric direction, all omni-s are curved with the same horizontal cross-section shape: circles. As a degenerate case of conic section, the cone does not have a full hemispherical view. It sees all 360 degrees in the horizontal direction, ut the vertical field of view (FOV) is not continuous over the zenith, see Figure 6. Other omni- system may in theory have complete hemispherical view ut in practice the lens camera or secondary s are always locking the view around the zenith, so in practice no similar system can see the whole hemispherical view anyway. Smaller vertical angular FOV is actually good for many applications ecause we get higher angular resolution with the same numer of pixels concentrating on the most interesting scene. The region not visile is occupied y the lens camera, the sky or the floor, all of which are of secondary interests in many applications like driving a ground vehicle. 7. Conclusion The Cone has een identified as having a single viewpoint efore ut that single viewpoint has een regarded as useless ecause previous analysis was done with the idealized pinhole camera geometric model. This work proves that with more accurate modeling of the optical properties of real cameras used in the real world, the single viewpoint inherent in the cone geometry is usale. Prototype system ased on our theory is constructed and experiments confirm our theoretical predictions. The total field of view is smaller ut more flexile. The new system is simple to uild and maintain, easier to analyze, costs a lot less, and yet yields much higher performance in the peripheral region where most interesting scenes reside. References [2] ogner, S. Introduction to panoramic imaging Proceedings of the IEEE SMC Conference. [3] Hecht, E., Optics, 3 ed. Reading, MA, USA: Addison Wesley Longman, Inc., 1998, pp [4] Krishna, A. and Ahuja,. Panoramic image acquisition San Francisco, CA. International Conference on Computer Vision and Pattern Recognition. [5] ayar, S. K. and Karmarkar, A. 360 x 360 Mosaics Hilton Head Island, South Carolina. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. [6] Southwell, D., Vandegriend,., and asu, A. A Conical Mirror Pipeline Inspection System. 4, Minneapolis, Minnesota, USA. Proceedings of the 1996 IEEE International Conference on Rootics and Automation. [7] Tsai, R. Y., "A Versatile Camera Caliration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the- Shelf TV Cameras and Lenses," IEEE Journal of Rootics and Automation, vol. RA-3, no. 4, pp , Aug [8] Yagi, Y., "Omnidirectional sensing and its applications," IEICE TRASACTIOS O IFORMATIO AD SYSTEMS, vol. E82D, no. 3, pp , Mar [9] Yagi, Y. and Kawato, S. Panoramic scene analysis with conic projection Proceedings of the International Conference on Roots and Systems. [10] Yagi, Y., Kawato, S., and Tsuji, S., "Real-Time Omnidirectional Image Sensor (COPIS) for Vision-Guided avigation," IEEE TRASACTIOS O ROOTICS AD AUTOMATIO, vol. 10, no. 1, pp , Fe [1] aker, S. and ayar, S. K., "A Theory of Single-Viewpoint Catadioptric Image Formation," International Journal of Computer Vision, vol. 35, no. 2, pp , ov.1999.