Design and Implementation of a Scanner with Stitching of Multiple Image Capture

Transcription

1 Y.-W. Bai and C.-C. Cheng: Design and Implementation of a Scanner with Stitching of Multiple Image Capture 1501 Design and Implementation of a Scanner with Stitching of Multiple Image Capture Ying-Wen Bai and Chien-Chung Cheng Abstract Instead of the flatbed scanner which moves slowly as it scans, we propose in this paper a new design of a high-speed and low-power scanner with a stitching of multiple image capture. We divide the area to be scanned into four parts, capture each image and then stitch these into a whole. Our design uses the compact camera module (CCM) advantages of high integrity, white balance, advanced calibration, and auto exposure for color and luminance, all of which reduce the software post-processing time. To solve the light reflection problem we propose an interleaved light source compensation method. To reduce power consumption and prevent light reflection among the four scan areas, we scan the four images one by one, during which a single CCM is turned on while the others are turned off. We use field of view (FOV) calculation to obtain the overlap area used by the stitching algorithms 1. emitting diodes (LED) instead of CCFL, thus reducing both power consumption and mechanical noise [2]. This paper is organized as follows. Section II introduces the design and implementation of the image stitching scanner. Section III draws our conclusion and further work. II. DESIGN AND IMPLEMENTATION OF THE IMAGE STITCHING SCANNER The architecture of the scanner with stitching of multiple image capture is shown in Fig. 1. Our design includes four compact camera modules (CCM) which are commonly used in the camera modules of cellular phones. In addition our design includes a USB video controller and a USB 2.0 HUB. The design takes four images separately through four USB camera modules and stitches the four images in a PC. Index Terms Image Stitching, Compact Camera Module (CCM), Field of View (FOV). I. INTRODUCTION A typical flatbed scanner causes some inconvenience during use, such as low speed and high power consumption. For example, the scanner s cold cathode fluorescent lamp (CCFL) needs to be preheated for several dozen seconds until it reaches normal color temperature. While scanning, the traditionally designed scanner also produces a mechanical noise because of the stepping motor. In addition, the moving mechanical parts of the flatbed scanner may produce the phenomenon of picture discontinuity if, as a result of wear, the stepping motor loses a step. As the flatbed scanner also consumes much power, it requires an extra adaptor to maintain a sufficient supply, thus increasing the weight of the machine and making it more inconvenient. In order both to achieve a good enough picture quality and to scan at the desired speed, we propose a new design of a high-speed and low-power scanner with multiple image capture stitching by using the CCM that is commonly used in mobile phones, thus providing both a small volume (6 mm 3 ) and sufficient resolution. This new design uses the camera capture method instead of the contact image sensor (CIS) of the flatbed scanner to obtain the desired image [1]. We use four CCMs to take the four images and then stitch them into a whole. This improves the scanning speed because the scanner requires neither any moving nor any stepping motor and uses light- Fig. 1. The architecture of the scanner with stitching of multiple image capture. The USB video controller is mainly for transmitting the image which the CCM processes and then encodes in YUV 4:2:2 format. As the USB video controller exports the image to a PC without image processing, the design needs no extra memory buffer for image processing. Thus the result is an overall low hardware cost. Fig. 2 shows the USB video controller. 1 Ying-Wen Bai is with the Department of Electronic Engineering, Fu-Jen Catholic University, Taipei, Taiwan, 242, R.O.C. ( bai@ee.fju.edu.tw) Chien-Chung Cheng is a graduate student of Fu-Jen Catholic University, Taipei, Taiwan, 242, R.O.C. ( @mail.fju.edu.tw) Contributed Paper Manuscript received October 8, /08/$ IEEE Fig. 2. The architecture of the USB video controller.

2 1502 Table I shows the power consumption of the USB video controller. TABLE I POWER CONSUMPTION OF THE USB VIDEO CONTROLLER Item Voltage Current consumption Vcc-IO 3.3 V 60 ma mw Vcc-CORE 1.8 V 15 ma 2.7 mw IEEE Transactions on Consumer Electronics, Vol. 54, No. 4, NOVEMBER 2008 the white balance, and in this way the image processing time at the PC end can be reduced. In our design, the CMOS image sensor in a CCM contains an ISP (Image Signal Processor) which provides an Auto White Balance and Auto Exposure Control together with the necessary image processing functions. Fig. 3 shows the CMOS sensor hardware architecture. Currently the commonly used image sensor is one of two kinds, CCD or CMOS, which we compare below and in Table II. A. CCD image sensor The operating principle of the CCD is to store the electric charge as detected by a photo diode by way of a potential well. In order to take out the charge generated by the photo-electric effect of every photodiode, we first adjust the bias voltage of the high voltage and then remove the charge by way of transmission. Because the amount of the electric charge is small it must be enlarged during the exporting process by means of floating diffusion amplifier (FDA). The removed electric charges therefore, are compatible with the adjustable voltage level [3]. B. CMOS image sensor The CMOS image sensor process is the same as that of the general CMOS IC, and its main difference from the CCD is that for every pixel of the CMOS sensor the charge is exported by amplification after the necessary A/D conversion. Because its process is the same as that of the general CMOS IC it can deal with both the image DSP and the CMOS image sensor as one integrated IC [4]-[6]. Since the CMOS image sensor has greater sensitivity, highintegration, low cost and low power consumption, we use it. To improve the sensitivity, we enhance the luminance by means of an auxiliary light source. TABLE II COMPARISON OF CMOS AND CCD IMAGE SENSORS Item CMOS Image Sensor CCD Image Sensor Sensitivity Cost Has ADC with each pixel, which reduces the effective area of each pixel and affects the sensitivity Can be integrated with the DSP, which reduces cost Outputs the analog signal directly with a larger effective area than the CMOS image sensor Only a raw sensor, which requires AFE (Analog Front End), Back End and Vertical Driver IC for the output signal process Consumption About 200 mw About 500 mw Requirement 2 power sources 4 power sources Because the light source of the environment is quite regular, the color difference can be prevented from affecting the different camera modules by means of advance correction of Fig. 3. The architecture of the CMOS sensor. When we use 4 CCMs of 3M pixels to scan A4 size paper, the converted resolution is equivalent to 300 dpi (deducting about 25% for image overlap). This resolution is calculated based on paper size, resolution and overlap area. A4 size = inches equals dpi dpi = pixels after converting. And 3 M 4 = 12M pixels reduced by about 25% for image overlap equals = 9M pixels. Therefore, 9M M pixels are equivalent to a resolution of 300 dpi. We need the image overlap to offer enough eigenvalue for the image stitching program to reconstruct and stitch the four images. Currently we use 25% of the whole area as the overlap area to obtain 3M pixels from the CMOS image sensor with a resolution of 300 dpi. The image capture area has been found by calculating the FOV of the lens, which is 60 degrees. Our design uses four cameras and defines each image capture area as > A of an A4 paper area, with a focal length of about mm. Our calculation procedure is shown from (1) to (5), and Fig. 4 shows the relationship of both the focal length and the physical parameters. 0.5E = = mm, A = θ = 60 (1) D = tan30 = 3. 14mm (2) Q Area = 0.25 A4 size = 147.5mm 104mm (3) ( ) tan30 = 128. mm C = 6 (4) Focal Length = = mm (5)

3 Y.-W. Bai and C.-C. Cheng: Design and Implementation of a Scanner with Stitching of Multiple Image Capture 1503 Fig. 4. FOV calculation. calculation. To reduce the height, we can increase the number of CCM modules and reduce the image capture area of each CCM module to shorten the focal length. Figs. 5 and 6 and Table III show the relation between the number of cameras and the object distance. If we adopt design I as shown in Table III, we can reduce the thickness of the image stitching scanner to 64.3 mm when it is equipped with 8 cameras. Our design of the position of the light sources achieves the goal of power saving by adopting white light LEDs. However, as shown in Fig. 7, the light source may be reflected by the glass which holds the paper. Fig. 8 shows how our design avoids light reflection by obtaining the image capture through timesharing, thus reducing power consumption at the same time. Fig. 8 shows the physical arrangement of the interleaved light source. 208mm G H I 297mm A4 Paper Fig. 5. A4 paper relative to the number of cameras. Fig. 7. The white light spot from the reflection of white light LEDs. TABLE III NUMBER OF CAMERAS VS OBJECT DISTANCE Number of Design Object Distance Cameras G mm Object Distance ( mm ) H mm I mm Number of Cameras vs Object Distance Number of Cameras Fig. 6. The relation between number of cameras and object distance. The minimum height of the image stitching scanner can be decided by calculating the focal length as in the above CMOS CCM Fig. 8. The design improved to deal with the reflection problem. The image stitching procedure consist of three parts: The first is the geometrical registration. This step consists mainly of the inputting of the image, thus causing the scale to be adjusted to reach the same level, after which we obtain the characteristics value of the sewing-up area and sew it up by using the algorithm of the feature base [7]. The second part is the procedure of stitching the overlapping areas of the images by using the blend algorithm. The third is the photometric registration, which is mainly the even melting of the differences between color and luminance among the different images. The algorithm SIFT (Scale Invariant Feature Transform) is used in the shareware Auto Stitch. SIFT adapts the Feature Base Method. Compared to the Direct Method which considers the whole picture as characteristics, the Feature Base Method only considers specific characteristics of the picture.

4 1504 SIFT creates the scale spaces using the Difference of Gaussian (DoG) filter. The calculation includes the comparison of the 2 continuous scale spaces and the finding of the extrema by utilizing the DoG process which requires 2 continuous images smaller than 2 σ and larger thanσ. Each image must be processed by Gaussian filtering. Step 1: Convolution with a variable-scale Gaussian L(x, y, σ ) = G(x, y, σ ) I(x, y) 1 (6) G(x, y, σ ) = y )/ (2 )e -(x + σ πσ Step 2: Difference of the DoG filter G(x, y,kσ ) - G(x, y, σ ) (7) Step 3: Convolution with the DoG filter D(x, y, σ ) = [G(x, y,kσ ) - G(x, y, σ )] I(x.y) (8) = L(x, y,kσ ) - L(x, y, σ ) SIFT calculates the scale space extrema by using (6), (7) and (8). To obtain the extrema, the black spot pixel needs to be compared with the other 8 pixels in the same scale space and also with the corresponding 9 2 pixels in the correlative scale-space. We use (9) to calculate the gradient of the neighborhood of the extrema and use (10) to find the spanning angle as the reference for stitching the images. 2 2 m(x, y) = [L(x + 1, y)- L(x -1, y)] + [L(x,y + 1) - L(x,y -1)] (9) -1 θ (x, y) = tan 2{[L(x,y + 1) - L(x,y -1)/L(x + 1, y)- L(x -1, y)]} (10) By using the gradient characteristics to indicate the parameter of direction, the characteristics of the image are not changed even when the image rotates. Fig. 9 shows the flowchart of the image stitching scanner. The first step captures the images, the second loads the images, the third, known as SIFT feature matching, finds the major characteristics of the images for image matching, the fourth adjusts the bundle, the fifth blends the multi-band, and the final step compresses the images. IEEE Transactions on Consumer Electronics, Vol. 54, No. 4, NOVEMBER 2008 The system carries out the image stitching procedure by using the shared software Auto Stitch [8], [9]. This procedure uses the SIFT algorithm to detect the image characteristics [10], [11]. Figs. 10 A-D show the four images captured from the 4 CCMs respectively. We see the result in the whole sewing-up area in Fig. 11a. After enlarging the result by 800 % as shown in Fig. 11b, the whole picture shows smooth boundaries between the four original images. A B C D Fig. 10. A, B, C, D The clockwise direction of a picture. Fig. 11a. The white circle shows the stitching area. Fig. 11b. 800 % enlargement in the white circle of Fig. 5a shows no obvious stitches. Fig. 9. The flowchart of the image stitching scanner. Fig. 12 shows the inside arrangement of the prototype of the image stitching scanner. We use a traditional scanner or a flatbed scanner and add the A, B, C, and D cameras with the related chips. Fig. 13 shows the dimensions of the prototype of the image stitching scanner. The length and width of our design are similar to that of the flatbed scanner, but our design is thicker in comparison. Table IV shows the comparison of the scanning speed, the power consumption, the resolution, the audible noise and the weight between the traditional flatbed scanner and our design. Our design has many advantages over the flatbed scanner, including high-speed scanning, low power consumption and an absence of preheating and noise.

5 Y.-W. Bai and C.-C. Cheng: Design and Implementation of a Scanner with Stitching of Multiple Image Capture 1505 Camera B Camera C very quiet during operation. However, there still are many problems to be addressed, such as the improvement of the resolution beyond 300 dpi and a reduction of the height of the prototype machine to less than 140 mm. In the future we will design an all-in-one image stitching scanner which can be used without a PC. Fig. 12. The inside arrangement of the prototype of the image stitching scanner. Fig. 13. The dimensions of the prototype for the image stitching scanner. TABLE IV COMPARISON BETWEEN FLATBED SCANNER AND OUR IMAGE STITCHING SCANNER Item Product A Image Stitching Scanner Scanning Speed Camera A Pre-scan: 7 secs Scan: 20 secs (Condition: 300dpi, A4) Camera D Pre-scan: 2 secs Scan: 2 secs (Condition: 300dpi, A4) Consumption 16 W 1.5 W Audible Noise Yes No Preheating Yes No Dimension L=430 mm, W=280 mm, L=400 mm, W=270 mm, H=57 mm H=140 mm Weight 2.6 kg 1.8 kg III. CONCLUSION In this paper we have designed and implemented an image stitching scanner which scans an A4 picture within 2 seconds and only consumes 0.9 W, and is supported by a single USB port [2]. In comparison with the flatbed scanner our design is REFERENCES [1] H. Kakinuma, M. Mohri, M. Sakamoto, H. Sawai, S. Shibata, Y. Kasuya, Y. Ohnuki and W. Chonan, Direct-contact type image sensors using a novel amorphous-silicon photodiode array, IEEE Electron Device Letters, vol. 12, issue 8, pp , Aug [2] Universal Serial Bus Specification, [3] P. P. Suni, CCD Wafer Scale Integration, Seventh Annual IEEE International Conference on Wafer Scale Integration, pp , Jan [4] Hung-Lung Tu and Wen-Hung Su, CMOS image sensors using feedback reset circuits for high speed operation, 8th International Conference on Solid-State and Integrated Circuit Technology, pp , Oct [5] Jen-Chuan Wang, Der-Song Su, Den-Jen Hwung and Ji-Chien Lee, A single-chip CCD signal processor for digital still cameras, IEEE Transaction on Consumer Electronics, vol. 40, issue 3, pp , Aug [6] M. A. Bayoumi, VLSI architectures for DSP applications: current trends, Proceedings of the 35 th Midwest Symposium on Circuits and Systems, vol.1, pp , 9-12 Aug [7] O. Chum, T. Werner and J. Matas, Two-view geometry estimation unaffected by a dominant plane, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol.1, pp , June [8] M. Brown and D. G. Lowe, Autostitch V2.187-DEMO VERISION copyright 2004, University of British Columbia, [9] Jun Luo, Y.Ma, E. Takikawa, S. Lao, M. Kawade and Bao-Liang Lu, Person-Specific SIFT Feature for Face Recognition, IEEE International Conference on Acoustic, Speech and Signal Processing 2007, vol. 2, pp. II II-596, April [10] M. Brown and D. G. Lowe, Recognizing Panoramas, Proceedings of Ninth IEEE International Conference on Computer Vision, vol. 2, pp , Oct [11] A. E. Abdel-Hakim and A. A. Farag, CSIFT: A SIFT Descriptor with Color Invariant Characteristics, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp , Ying-Wen Bai is a professor in the Department of Electronic Engineering at Fu-Jen Catholic University. His research focuses on mobile computing and microcomputer system design. Ying-Wen Bai obtained his M.S. and Ph.D. degrees in electrical engineering from Columbia University, New York, in 1991 and 1993, respectively. Between 1993 and 1995, he worked at the Institute for Information Industry, Taiwan. Chien-Chung Cheng is currently studying toward the M.S. degree in Electronic Engineering at Fu-Jen Catholic University, Taiwan. His received his A. E. in Electrical Engineering from Lunghwa University of Science and Technology, Taiwan, in He serves as Supervisor in Chicony Electronics Co., Ltd., Taiwan. His major research is focus on camera module design.