Development of an optical inspection platform for surface defect detection in touch panel glass

International Journal of Optomechatronics ISSN: 1559-9612 (Print) 1559-9620 (Online) Journal homepage: http://www.tandfonline.com/loi/uopt20 Development of an optical inspection platform for surface defect detection in touch panel glass Ming Chang, Bo-Cheng Chen, Jacque Lynn Gabayno & Ming-Fu Chen To cite this article: Ming Chang, Bo-Cheng Chen, Jacque Lynn Gabayno & Ming-Fu Chen (2016) Development of an optical inspection platform for surface defect detection in touch panel glass, International Journal of Optomechatronics, 10:2, 63-72, DOI: 10.1080/15599612.2016.1166304 To link to this article: https://doi.org/10.1080/15599612.2016.1166304 Accepted author version posted online: 17 Mar 2016. Published online: 17 Mar 2016. Submit your article to this journal Article views: 458 View related articles View Crossmark data Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalinformation?journalcode=uopt20 Download by: [37.44.204.250] Date: 30 November 2017, At: 02:31

INTERNATIONAL JOURNAL OF OPTOMECHATRONICS 2016, VOL. 10, NO. 2, 63 72 http://dx.doi.org/10.1080/15599612.2016.1166304 Development of an optical inspection platform for surface defect detection in touch panel glass Ming Chang a,b, Bo-Cheng Chen b, Jacque Lynn Gabayno c,d, and Ming-Fu Chen e a College of Mechanical Engineering and Automation, Huaqiao University, Fujian, P. R. China; b Department of Mechanical Engineering, Chung Yuan Christian University, Chung Li, Taiwan; c Center for Biomedical Technology, Chung Yuan Christian University, Chung Li, Taiwan; d Mapua Institute of Technology, Manila, Philippines; e Instrument Technology Research Center, National Applied Research Laboratory, Hsinchu Science Park, Hsinchu, Taiwan ABSTRACT An optical inspection platform combining parallel image processing with high resolution opto-mechanical module was developed for defect inspection of touch panel glass. Dark field images were acquired using a 12288-pixel line CCD camera with 3.5 µm per pixel resolution and 12 khz line rate. Key features of the glass surface were analyzed by parallel image processing on combined CPU and GPU platforms. Defect inspection of touch panel glass, which provided 386 megapixel image data per sample, was completed in roughly 5 seconds. High detection rate of surface scratches on the touch panel glass was realized with minimum defects size of about 10 µm after inspection. The implementation of a custom illumination source significantly improved the scattering efficiency on the surface, therefore enhancing the contrast in the acquired images and overall performance of the inspection system. 1. Introduction KEYWORDS Custom-illumination; line CCD; parallel computing; surface defect inspection; touch panel glass The growing industry of touch panel technologies found in ubiquitous digital electronic devices such as smart phones, ticket vending machines, ATMs, and tablets is driving a demand for a fast and nondestructive defect inspection system to ensure a top quality product during the many stages of the manufacturing process. Surface defects on glass substrates generally include scratches, cracks, bubbles, inclusions, dirt and other markings [1 3]. In resistive touch panel glass, the presence of these defects can result in low light transmittance and low levels of visibility. Manual inspection by trained experts is usually impractical to examine surface defects because of their reduced size. However, most of the current manufacturing lines still rely on visual inspection by manpower due to the low contrast images from touch panel glass defects as opposed, for instance, to the popular optical automated systems used in film circuit defect inspection because the image contrast between the defect and background is higher than the image contrast between tiny touch panel glass defect and glass substrate. Therefore, fast inspection and material handling employ automated systems by means of multi-axis positioning platforms for motion-control and machine-vision-based inspection systems. These solutions seek to address the high volume in-line inspection demand with fast detection, high accuracy, and measurement stability. In machine vision technology, optical images of the surface under inspection are used to examine defects and assess product quality. The images are often characterized by reduced resolution as a compromise to save on data processing capacity [4] or suffer low contrast between defects and other CONTACT Ming Chang ming@cycu.edu.tw Department of Mechanical Engineering, Chung Yuan Christian University, 200 Chung Pei Road, Chung Li City, Taoyuan, 32023, Taiwan. Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/uopt. 2016 Taylor & Francis

64 M. CHANG ET AL. Nomenclature ATM Automated teller machine CCD Charge-coupled device CPU Central processing unit CUDA Compute unified device architecture FOV Field of view f x,y Raw image g x,y Histogram equalized image GPU Graphics processing unit L Length of test object in mm LED Light emitting diode MPI Message passing interface n Pixel intensity values, 0 < n < 255 N Total number of pixels N a Number of actual defects N d Number of detected defects PCB Printed circuit board p N Normalized histogram ROI Region of interest TDR True detection rate W Width of test object in mm surface features because of poor illumination. In both cases, the gray level intensity of the defects against the background is hard to distinguish and thus results to difficulty in defect inspection. Complex algorithms [5,6] for image processing can be used to enhance the defects based on low resolution and poor contrast images. However, image processing and defect inspection using low contrast images can be time-consuming. Conversely, large area or line scan cameras can be employed to improve on the resolution. But this approach would also require a compromise between resolution and detection speed because of the large image data that needs to be processed. By combining established techniques in digital image processing [7 9] with a high-resolution optomechanical module, we developed an optical inspection platform that can achieve significant and reliable performance in both speed and resolution for surface defect inspection in touch panel glass. A novel imaging module which combined a linear CCD with a custom LED illumination and a self-developed parallel computing algorithm for simultaneous detection and labeling of surface defects were utilized in the inspection platform. The scattering caused by the defects and the modification in the light source enhanced the appearance of defects against a dark background regardless of the inclination of a defect. The defects were reconstructed from the light scattered on the surface of the touch panel glass and a line CCD connected to a host CPU was used for image acquisition. From the host CPU, large image data were transmitted to a GPU device where computations using Compute Unified Device Architecture (CUDA) kernel functions for image processing were completed in parallel to speed up the processing time. The system presents an algorithm with a capability to intelligently process big data. The information in each block of GPU was parallel analyzed with automatically adjusted parameters. Besides increasing the detection rate, this upgrades the software intelligence to improve the accuracy of defect identification. After processing, the data were sent back to the CPU allowing the user to assess the surface quality of the touch panel glass based on reconstructed defect maps. Because the illumination design also affects the quality of the image reconstruction of the test object, development of the light module was optimized. The illumination source was able to provide uniform light intensity and low power consumption thus maintaining efficient heat dissipation and low fabrication costs. 2. Materials and methods 2.1. Design of inspection platform Figure 1 shows the optical inspection system consisting of an opto-mechanical module, a motorized carrier stage, and image processing software. The imaging module consists of a custom LED array (Figure 1(c)) as illumination source, a lens (Schneider MRV 4.5/85), and a 12288 pixel-line CCD camera (Basler ral12288-66 km). The requirement from the industry is that a defect with a width bigger than 10 µm should be detected for touch panel glass inspection. In our algorithm, a defect was set to be determined with a width at least 2 or 3 pixels. The pixel resolution of the adopted CCD is 3.5 µm, thus, the lens magnification is set to 1.0x and the detectable width is roughly 43 mm. At 12 khz line rate, the image data that can be acquired by the CCD is approximately 147 megapixels per second.

INTERNATIONAL JOURNAL OF OPTOMECHATRONICS 65 Figure 1. (a) Schematic of hardware components of the optical inspection platform. (b) Design and construction of the illumination source from LED arrays. (c) Actual installation of the opto-mechanical components. The chosen line rate is dependent on the intensity of the light source. The line rate can be higher if the intensity of the light source is adjusted to a higher value. In order to fully automate the process, the inspection stage was motorized where it can be operated at different carrier velocity depending on the spatial resolution and line rate selection of the optomechanical module. A built-in carrier plate was installed on the stage for mounting the test object. The camera and linear stage were connected to a common host computer. The images acquired were also stored on the same computer. To demonstrate the fast detection speed and resolution capability of the inspection system, this study focuses on only one imaging module. However, depending on the size of the test object, additional imaging modules or high pixel number line CCDs can be used to expand the detectable area. Huge data capacity can be expected from either method, but this can also be easily addressed by parallel image processing on a common integration server with a CPU and multiple GPU devices. 2.2. Design of illumination device The line CCD was oriented parallel to the width of the test object as shown. The custom illumination source consisting of two LED arrays was mounted on either side of the CCD thus illuminating the sample at an angle from the normal. In principle, light incident on the surface of the sample is reflected, transmitted, or absorbed. Thus, the normal orientation of the line CCD with respect to the test object was optimized to receive the most light scattered by artificial (e.g., dust) or inherent defects (e.g., cracks, bubbles, scratches, etc.) on the surface. The principle for the optical inspection design is illustrated in Figure 2. For a perfectly smooth surface the camera will record a mostly dark image since the majority of the light reflected is specular. Figure 2. Detection principle of surface defects on touch panel glass based on dark-field illumination. (a) Reflection from a perfectly smooth surface will be recorded as a dark image. (b)-(c) Light scattering from surface defects will appear as bright areas against a dark background. Scattering which are nearly parallel to the light axis of the opto-mechanical module can reach the image plane of the line CCD. Irradiation from slanted parallel light sources are reflected away from the camera.

66 M. CHANG ET AL. Meanwhile, the presence of defect on the surface will scatter light in the direction of the camera which will appear as a bright area in a dark background on the recorded image. The reconstruction and identification of the surface defects will depend on the intensity of the scattered light and the collection efficiency of the imaging module. In order to optimize the detected signal from light scattered on the bottom surface of the touch panel glass, a high reflective coating [10] was added on the carrier plate such as shown. The coating effectively created two reflective layers from the glass substrate. Part of the incident light will be reflected at the top surface of the glass substrate, and some will be refracted into glass and then reflected by the reflective coating. Only the scattered light from defects which are nearly parallel to the light axis of the opto-mechanical module can reach the image plane of the line CCD camera. Light irradiated on the glass substrate from slanted parallel light sources will be reflected away from the camera, and the gray level of pixels on the dark field image will be zero if there are no defects. The scattered light perpendicular to the surface of the glass substrate will pass through the light entrance slit and into the camera sensors, which as mentioned above will appear as bright pixels on the dark field image and correspond to a defects area. The structure of the illumination source and arrangement of the LEDs are schematically shown in Figure 1(b). The device was constructed from four tightly arrayed LEDs soldered on two aluminumbased PCBs. A total number of 24 LEDs with power output of 3 watt for each LED was used. By applying the feature of LED fan-shape lighting, two PCBs were installed with symmetric and adjustable inclination to form the multi-directional lighting on the glass substrates. Based on side-lighting effect and tightly arrayed LEDs, multi-directional and uniform illumination on the glass substrates can be derived from the device and thus address the issue of weak scattering from the defects. The scattering caused by the defects and the modification in the design of the light source enhance the appearance of defects against a dark background regardless of the inclination of a defect. In other systems, the scattering intensity from a crack with an inclination parallel to the moving direction of the translation stage, i.e., perpendicular to the CCD pixel direction, is usually weak. The proposed lighting system overcomes this problem. In addition, high intensity is required for line image acquisition with very short exposure time. In order to satisfy this requirement, the distance between LEDs and glass substrate was designed a separation of around 10 mm with an illumination angle of 30 0 to get the sufficient illumination for dark field image acquisition. The separation distance and illumination angle were determined from experiments. If the separation is bigger, the grabbed intensity from the specimen for the line CCD is weak. However, experimental results show that the 10 mm separation distance allows for easy operation and the 30 0 to 60 0 oblique incidences provide better and acceptable illumination for the CCD. The resolution and variable magnification of the Schneider MRV 4.5/855 lens used are 2.5 µm and 0.5 2, respectively. Magnification of 1 was set to meet the spatial resolution of 3.5 µm and the FOV of about 43 mm for the imaging module. From the lens specification offered by Schneider Company, MRV 4.5/85 lens can provide the relative illumination of greater than 97% and the geometrical distortion is zero for the magnification of 1.0x. This omits the need for image correction. 2.3. Systems architecture The algorithm for surface inspection was self-developed where the system flow chart is shown in Figure 3. Initially, the user was prompted to input the parameter settings which include the capture rate of the CCD, speed of the motorized stage, and desired acquisition time. A command string was then sent out to initialize the CCD and the linear stage. The line CCD acquires the surface scattered light from the glass object as it moves on the translation stage. The image data was stored on the host server (CPU) after the set acquisition time has elapsed. The image pre-processing and defect inspection tasks were assigned as CUDA-based kernel functions to a GPU device. With this design and depending on the size of the object, multiple GPU devices can also be configured to perform the same kernel operations in parallel.

INTERNATIONAL JOURNAL OF OPTOMECHATRONICS 67 Figure 3. System flowchart from image acquisition to data transmission and detection of surface defects. In one GPU device, the image pre-processing and defect detection tasks were performed as pixelwise operations on a designed block of CUDA threads. The image pixel was represented by a CUDA thread which can be varied depending on the resolution requirements. Initially, the raw data was copied from the CPU s main memory to the GPU s global memory through Message Passing Interface (MPI) commands. Separate memory was also allocated in the GPU to contain the processed data after each kernel function, i.e., dst_1, dst_2, etc. in Figure 3. The kernel functions were initiated by the MPI commands. The processed data (i.e., dst_out) was then transferred back to the CPU after the CUDA threads has completed the kernels. The detected image was divided into several sub-images based on GPU for parallel image processing. Each block in the GPU can be seen as a ROI (region of interest) in CPU. The information in each block was analyzed with variable parameters, which was automatically adjusted according to the grabbed image intensity histogram. The processed data from multiple GPUs were also combined to generate a defect map where the size, number, and features of surface defects were displayed for defect classification or final evaluation of the test object. 2.4. Surface defect inspection The kernel functions were carried out by the CUDA threads, which included image pre-processing operations and four successive defect detection algorithms. The kernels can be completed iteratively on a single GPU or in parallel on multiple devices depending on the size of the image data. The first kernel processed histogram equalization which was implemented to enhance the contrast and flatten out the intensity range to obtain a more uniform intensity distribution. First, the normalized histogram p N (i.e., p N ¼ number of pixels with intensity value n N 1 ) of the raw image f x,y for each possible intensity value was computed where N is the total number of pixels with intensity values

68 M. CHANG ET AL. n from 0 to 255. Next, the histogram equalized n image g x,y for each pixel coordinate (x,y) was rounded off to the nearest integer, i.e., g x;y ¼ round ð255þ P o f x;y. The image was stored in the allocated n¼0 p N memory dst_1 after the histogram was equalized. The second kernel implements a Gaussian pyramid scaling operation which was implemented to reduce the computation time. The processed data in dst_1 was resized by convolving with the row kernel g R ðx; y; rþ ¼ 1 16 ½14641 and subsequently, column kernels with the same basis. The computation was repeated until the pixel size approached a set threshold, e.g., 10 µm. After convolution, all the even rows and columns were removed and the processed output was stored in the allocated memory dst_2. For defect detection, four kernel functions were carried out in succession by the CUDA threads. The kernels include binarization, labeling, size detection, and edge detection. The latter two operations can be combined into a single kernel. During binarization, the processed data in dst_2 was binarized based on a set threshold such that if a pixel value is larger than the threshold, the value was set to 255. Otherwise, the pixel value was set to zero and the binarized output was stored in the allocated memory dst_3. In order to obtain the number of defects in an image, each was assigned a label. The idea was to locate the starting and terminal points of defects, and then eliminate the redundant starting and terminal points. After elimination, the number of defects should be the same as the number of starting or terminal points. The processed output after labeling was stored in the allocated memory dst_4. Finally, image size and edge detection were performed. These operations were implemented to obtain the size and geometric shapes or contours of defects, i.e., the left, upper, right, and bottom edges of defects. In addition, all nonzero pixel values were changed back to 255 after labeling. An array of zeroes which has the same size as the image was allocated in the CUDA global memory for the edge detect algorithm. The size of the defects was calculated from the bounded edges. The processed output was stored in the allocated memory dst_out. This memory was directly accessible to the CPU. Figure 4 show the output images after executing the kernel functions on the raw image of a touch panel glass. Figure 4. (a) Input raw image, (b) histogram equalized image, (c) binarized image, (d) labeled image.

INTERNATIONAL JOURNAL OF OPTOMECHATRONICS 69 Figure 5. (a) Dark-field (raw) image of a flat mirror. (b) (d) Reconstruction of defect maps localizing regions with cracks, bubbles, and edge defects. 3. Results and discussion 3.1. Surface defect detection on flat mirrors Initially, we performed the defect inspection using test mirrors to characterize the performance of the opto-mechanical module. The speed of the motorized stage was 42 mm/sec to get a line image pitch of 3.5 µm. Figure 5 shows the dark-field image of a back-coated mirror object with an actual dimension of 43 mm (W) 110 mm (L). The raw image data of the test mirror was about 386 megapixels which was stored on the CPU s main memory after acquisition. To enable fast image processing, the image data was transferred to the GPU s global memory and then defect inspection algorithms were performed in parallel following the kernel functions outlined above. The processed output in dst_out is shown in Figure 5(b) 5(d). Once a defect has been identified, the algorithm constructs a defect map to show the relative location of the defect on the test specimen. The size of the map can be selected by the user to highlight the shape, size and orientation of the detected defects. The constructed defect map was stored on the CPU. For instance, Figure 5(d) shows a defect map of the upper right corner of the back-coated mirror where cracks and edge defects were detected. In the final assessment of the test mirror, the size and location of the detected flaws can be used as judgment criteria for defect classification. Among the defects found on the mirror were as follows: I. Bubbles: These are characterized by their point-like shapes and circular dimensions. The measured diameter ranged from 35 µm to 49 µm. II. Cracks: These are characterized by thin and elongated lines which were mostly located away from the edge of the mirror. The measured width and length ranges from 7 µm to 21 µm and 1 mm to 10 mm, respectively. III. Edge defects: These are characterized by the irregular shapes and jagged contours which were detected on the perimeter of the mirror. The dimensions are comparably larger than the cracks and thus could eventually compromise the mirror quality. For assessment purposes during manufacturing, the shape and size of these defects can help gauge the quality of the tooling/cutting processes on the mirror or other test objects. The detection sensitivity of the algorithm for each defect category is summarized in Table 1. The calculation compared the number of defects correctly detected by the algorithm with that of the actual defects. The results were significant as these showed that the accuracy of the algorithm was 94% 99% on all defects category. This is slightly lower than the industrial standard of 95% positive detection for crack inspection. However, the result was caused by the missing judgment of software in defect classification. In case we did not perform defect classification, 100% defect detection rate was achieved.

70 M. CHANG ET AL. Table 1. True detection rates (TDR) of defects on test mirrors. Surface defects TDR (%) Bubbles 96 Cracks 94 Edge defects 99 *TDR ¼ 100 x (N d /N a ); N d ¼ detected defects, N a ¼ actual defects ¼ 100. 3.2. Surface defect detection on touchscreen glass panel Figure 6 shows a sample optical inspection result of a touch panel glass. The dimension of the detected glass area was also 43 mm (W) 110 mm (L). The raw image data was also about 386 megapixels which had been stored on the CPU s main memory after acquisition. Similar to the test mirror, the data was transferred to the GPU s global memory for parallel image processing and defect inspection. Point-like surface artifacts which can be attributed to dusts have been numerically removed but the position and size of scratches or possible cracks were highlighted on the defect maps after image processing. The defect maps in Figure 6(c)-6(d) show that the size of the scratches in S1 and S2 has a minimum width of 14.8 µm and 2.65 mm in length. If undetected, the presence of these scratches can significantly reduce the transmittance and visibility of information displayed on the touch panel product. Comparison with the manufacturer s independent inspection system provided 100% detection rate of the surface scratches on the touch panel glass with about 10 µm as the minimum size of detectable defect. 3.3. Detection and image processing time The transfer rate of image data from CPU to GPU and computation time of each kernel function on a single GPU is listed in Table 2. The time to complete the exportation of 386 megapixels image data and completion of image pre-processing tasks was 2.606 s. The histogram and Gaussian pyramid kernel functions were both completed in sub-millisecond time scale. Meanwhile, the total time to complete defect detection tasks and then return the data to the CPU was 2.496 s. Only about 5.6 ms was spent to perform the four kernel functions for defect inspection. In total, the data transfer from CPU Figure 6. (a) Dark-field (raw) image of a touch panel glass. (b) Final image reconstruction of touch panel glass. (c)-(d) Localization of surface defects due to scratches or cracks (S1, S2) on the touch panel glass.

INTERNATIONAL JOURNAL OF OPTOMECHATRONICS 71 Table 2. Time duration of image data transfer and image processing of touch panel glass (W ¼ 43 mm, L ¼ 110 mm, Image size ¼ 386 megapixels). Processing commands Processing time (ms) Image pre-processing Copy memory from CPU to GPU 1309.421 GPU kernel functions: Histogram analysis 0.102 and Gaussian pyramid Copy memory from GPU to CPU 1296.922 Defect detection Copy memory from CPU to GPU 1285.545 GPU kernel functions: Binarization, Labeling, Size and Edge detections 5.62 Copy memory from CPU to GPU 1205.390 to GPU, pre-processing, and defect detection times yield significantly faster results if completed by parallel processing on a GPU. Similar tasks completed on a single CPU resulted to detection and processing time that exceeded more than 10 minutes for the same test object. 4. Conclusions This article presented an automated optical inspection system that combines a versatile, high resolution opto-mechanical module with parallel computing platform to achieve fast surface defect detection in touch panel glass. Fast processing time of large image data of about 386 megapixels per 43 mm (W) 110 mm (L) test objects acquired by the imaging module was realized by utilizing GPU machines to execute image processing functions in parallel. High resolution surface flaws such as bubbles, cracks, scratches, and edge defects were detected accurately from the image reconstruction of back-coated test mirrors and touch panel glass. The width of the smallest defect was measured at 10 µm. The calculated sensitivity of the detection algorithm was above 94% for each defect category. The detection and processing time to complete defect inspection on a single CPU-GPU platform was at least 5.1 s, which was significantly faster compared to defect inspection completed only on a CPU. The custom illumination used consisting of two 24-LED arrays achieved uniform and intense illumination as well as multi-directional lighting effect on the test objects to help improve the image contrast and resolution of the inspection system. The versatility of the opto-mechanical module and the image processing algorithm can be applied to examine even test objects. Funding This research was financially supported by Chung Yuan Christian University and the Ministry of Science and Technology under grant No. 102-2218-E-033-002-MY2. References [1] Adamo, F.; Savino, M. A low-cost inspection system for online defects assessment in satin glass. Measurement 2009, 42, 1304 1311. [2] Shimizu, M.; Ishii, A.; Nishimura, T. Detection of foreign material included in LCD panels. In Proceedings of the 26th Annual Conference of the IEEE Industrial Electronics Society, Nagoya, Japan, October 22 28, 2000; IEEE: New York, 2000; pp. 836 841. [3] Tsai, D.-M.; Tsai, H.-Y. Low-contrast surface inspection of mura defects in liquid crystal displays using optical flow-based motion analysis. Mach. Vis. Appl. 2011, 22, 629 649. [4] Zhao, J.; Kong, Q.J.; Zhao, X.; Liu, J.; Liu, Y. A method for detection and classification of glass defects in low resolution images. In Proceedings of the 6th International Conference on Image and Graphics, Anhui, China, August 12 15, 2011; IEEE: New York, 2000; pp. 642 647. [5] Tsai, D.-M.; Lin, P.-C.; Lu, C.-J. An independent component analysis based filter design for defect detection in low-contrast surface images. Pattern Recogn. 2006, 39, 1679 1694.

72 M. CHANG ET AL. [6] Tsai, D.-M.; Chao, S.-M. Anisotropic diffusion-based defect detection for low-contrast glass substrates. Image Vision Comput. 2008, 26, 187 200. [7] Gonzalez, R.C.; Woods, R.E. Digital Image Processing 3rd ed., Prentice Hall: New Jersey, 2008. [8] Malamas, E.N.; Petrakis, E.G.M.; Zervakis, M.; Petit, L.; Legat, J.D. A survey on industrial vision systems, applications; and tools. Image Vision Comput. 2003, 21, 171 188. [9] Canny, J. A computational approach to edge detection. IEEE T. Pattern Anal. 1986, 8, 679 698. [10] Chang, M. Device for panel defect detection. ROC Patent M 457889, 2013.