Quasi one-shot full-field surface profilometry using digital diffractive-confocal imaging correlation microscope Duc Trung Nguyen 1, Liang-Chia Chen* 1,2, Nguyen Dinh Nguyen 1 1 Mechanical Engineering Department, National Taiwan University, No. 1, Sec. 4, Roosevelt Road, Taipei 10617, Taiwan 2 Graduate Institute of Automation Technology, National Taipei University of Technology, No. 1, Section 3, Chung-Hsiao East Road, Taipei 106, 10655, Taiwan Abstract: One-shot full-field surface profilometry using digital diffractive-confocal imaging correlation microscope based on digital micromirror device is developed for one-shot microscopic 3-D surface measurement. Optical configuration applies confocal microscope setup and was building on DMD to generate specific pinhole array arrangement for minimizing cross-talk effect. An innovative method was invented to create normalized cross correlation depth response curve from diffraction patterns of the pinhole. Using this approach, the sub-micrometer scale depth can be detected with high accuracy and precision. References and links 1. T. R. Corle and G. S. Kino, "Chapter 1 - Introduction," in Confocal Scanning Optical Microscopy and Related Imaging Systems, T. R. C. S. Kino, ed. (Academic Press, Burlington, 1996), pp. 1-66. 2. A. Boyd, "Bibliography on confocal microscopy and its applications," Scanning 16, 33-56 (1994). 3. T. Wilson and B. R. Masters, "Confocal microscopy," Appl. Opt. 33, 565-566 (1994). 4. M. Rajadhyaksha, R. R. Anderson, and R. H. Webb, Video-rate confocal scanning laser microscope for imaging human tissues in vivo, Appl. Opt. 38(10), 2105 2115 (1999). 5. M. Petran, M. Hadravsky, M. D. A. V. I. D. Egger, and R. O. B. E. R. T. Galambos, Tandem-scanning reflected-light microscope, J. Opt. Soc. Am. 58(5), 661 664 (1968). 6. C.H. Lee, J.P. Wang, Noninterferometric differential confocal microscopy with 2-nm depth resolution, Opt. Commun. 35 (1997) 233 237 7. Zhao Weiqian, Tan Jiubin, Qiu Lirong, Zou Limin, A new laser heterodyne confocal probe for ultraprecision measurement of discontinuous contours, Meas. Sci. Technol. 16 (2005) 497 504. 8. Z. Weiqian, T. Jiubin, Q. Lirong, Tri-heterodyne confocal microscope with axial superresolution and higher SNR. Opt. Express 12 (2004) 5191 5197 http://www.opticsexpress.org/ 9. J. Liu, J. Tan, H. Bin, and Y. Wang, "Improved differential confocal microscopy with ultrahigh signal-to-noise ratio and reflectance disturbance resistibility," Appl. Opt. 48, 6195-6201 (2009). 10. T. Jiubin, L. Jian, and W. Yuhang, "Differential confocal microscopy with a wide measuring range based on polychromatic illumination," Measurement Science and Technology 21, 054013 (2010). 11. E. Sánchez-Ortiga, C. J. R. Sheppard, G. Saavedra, M. Martínez-Corral, A. Doblas, and A. Calatayud, "Subtractive imaging in confocal scanning microscopy using a CCD camera as a detector," Opt. Lett. 37, 1280-1282 (2012). 12. Liang-Chia Chen, Duc Trung Nguyen, and Yi-Wei Chang, "Precise optical surface profilometry using innovative chromatic differential confocal microscopy," Opt. Lett. 41, 5660-5663 (2016). 13. A. K. Ruprecht, K. Koerner, T. F. Wiesendanger, H. J. Tiziani, and W. Osten, "Chromatic confocal detection for high-speed microtopography measurements," in 2004), 53-60. 14. K. Shi, P. Li, S. Yin, and Z. Liu, Appl. Opt. 12, 2096 (2004). Chromatic confocal microscopy using supercontinuum light
15. Aiko K. Ruprecht ; Klaus Koerner ; Tobias F. Wiesendanger ; Hans J. Tiziani ; Wolfgang Osten; Chromatic confocal detection for high-speed microtopography measurements. Proc. SPIE 5302, Three-Dimensional Image Capture and Applications VI, 53 (April 16, 2004); doi:10.1117/12.525658. 16. F. Bitte, G. Dussler, T. Pfeifer, G Frankowski MicroScan: a DMD based optical surface profiler, Proceedings. of SPIE 4093, 309 318 (2000). 17. F. Bitte, G. Dussler, and T. Pfeifer, "3-D micro-inspection goes DMD," Optics and Lasers in Engineering 36, 155-167 (2001). 18. W. Neu, M. Schellenberg, E. Peev Time-resolved confocal microscopy using a digital micro-mirror device, Proceedings of SPIE 7596, 75960F-1 (2010). 19. Goodman, J. W. 1968. Introduction to Fourier Optics. New York, McGraw-Hill 20. T. Wilson and C. J. R. Sheppard, Theory and Practice of Scanning Optical Microscopy (Academic, London, 1984) 21. H.H. Hai, L.-C. Chen, D.T. Nguyen, S.-T. Lin, S.L. Yeh, Y. Yao, Accurate submicron edge detection using the phase change of a nano-scale shifting laser spot, Opt. Laser Technol. 92 (2017) 109 119. 1. Introduction Scanning confocal microscopy is a well-known technology for the 3D measurement of surface topography. Due to many advantages over other optical techniques such as having a high numerical aperture, meaning a high lateral resolution and a high measurable local slope or its superior depth discrimination compared to conventional microscopy [1], nowadays confocal microscopes are one of those instruments can dominate the market for optical surface topography instruments. In order to improve its property, many efforts have been made [2]. Many different methods have been developed into confocal systems to improve not only spatial and axial resolution but also operating speed of the whole system. From traditional point-scan confocal microscope, a point-scan confocal microscope using Nipkow disk was developed so the scanning speed can be significantly improved [3]. For further parallelization, based on the color dispersion of optic element, point-scan chromatic confocal can eliminate vertical scanning process and also improve its axial resolution and signal-to-noise ratio [4]. Chromatic confocal point sensors are already commercially available but it is still necessary to scan in the x- and the y-direction to get a surface topography, reducing the applications of in-situ profilometry. By changing from pinholes to slits and using line spectrometer, both vertical scanning and one spatial scanning can be neglect [5]. However, line-scan chromatic confocal will suffered from cross-talk effect between adjacent points on measuring line. Going in a different direction, the concept of differential confocal microscopy was proposed that utilizes the linear variance ratio of intensity at the slopes of the axial response curve to measure surface contours with a high axial resolution [6]. A laser heterodyne confocal probe was proposed for use in making ultra-precision bipolar absolute measurements [7]. Based on this device, a tri-heterodyne confocal microscope was developed to improve its axial resolution and signal-to-noise ratio [8]. A lateral super resolution differential confocal technology with phase-only pupil filter is proposed in order to further improve DCM's lateral resolution [9]. Shaped annular beam confocal measurement system (SABCMS) is proposed to increase lateral resolution and the measurement range [10]. Improved differential confocal microscopy was proposed to improve axial resolution and to enhance disturbance resistibility of confocal microscopy [11]. A differential confocal microscopy based on polychromatic illumination was proposed to achieve both a wide range and a high lateral resolution during measurement of a 3D deep-etching microstructure [10]. Differential laser confocal principle is capable of real-time measurements and doesn t need vertical scanning, but its depth measurement range is relatively small and still needs scanning for both X and Y axes [12-14]. Moreover, in scanning microscope systems, the movements of scanning mechanisms can contribute too many errors in measured results. Applying Digital Micromirror Device, a 3D micro-inspection using confocal microscope was invented [15]. Without mechanical lateral scanning, the speed of this system is significantly increased. However, this system still
requires vertical scanning and lacking real pinholes lead to reduce in measurement accuracy and precision. Although special lateral scanning strategy already developed, without real pinhole, the system still suffers from cross-talk effect between adjacent imaging points. In order to speed up the measuring speed of confocal system but to remain high sectioning capability and spatial resolution, our idea is to use the diffraction patterns of real pinholes, which are created by micro-mirrors on DMD to encode depth information. Then, using the reverse procedure to decode the depth information from pinhole's diffraction patterns to depth. Applying this principle will strengthen measurement efficiency by many factors. First, by building diffraction database and extract the depth information from this database, the vertical scanning process can be totally omitted. Second, based on DMD, there are a large number of pinholes matrix can be created, so in in-situ application, spatial scanning processes can be skipped. For applications which require higher spatial resolution, spatial scanning processes can be performed using different specific pinhole matrix arrangement which created by DMD. In this method, there are no requirements for both axial and lateral scanning; therefore, the developed system can improve in both measurement speed and quality. Before, to omit one scanning direction in a confocal microscope, we always need to stick with the complex setup of a differential confocal microscope or specific spectrometer in a chromatic confocal microscope. In our method, we use simple optical configuration with a 12-bit mono CCD. With this approach, confocal microscope system can be optimistic for in-situ surface measurement because of quasi-one-shot full-field capability. 2. Experiment Setup To achieve the above-mentioned approach, this article proposes one novel optical configuration of confocal microscopy using DMD. The system applied a pinhole confocal setup to generate confocal sectioning capability. The configuration uses a digital micro-mirror device (DMD; Texas Instruments, Dallas, Tex.) as a spatial light modulator in confocal microscope configuration with a white-light source. DMD is loaded with a specially designed pinhole pattern to generate the pinhole array. A white-light light source beam is collimated by a collimating lens to illuminate the DMD. After passing through a tube lens, the reflected light from DMD is imaged on the focus plane of the objective. The mirror (or tested sample) is placed perpendicular to the optical axial of objective, so the reflected light from the mirror goes through the objective and tube lens again. The reflected light after that will hit with DMD the second time and finally forms an image on the high-speed 12-bit mono camera by the imaging lens. The light goes through DMD twice, so the pinhole array created by DMD has the conjugate confocal effect on both detection and illumination paths. To build a database of diffraction patterns in every single layer along the optical axis, the reference mirror is moved by an accurate pre-calibrated piezoelectric linear stage to record the diffractive patterns. To build a database of diffraction patterns, the reference mirror is moved by piezoelectric stage. Diagram of optical system setup is described in Fig.1.
Fig. 1. Diagram of optical system setup. 3. Pinhole s diffraction pattern In digital diffractive-confocal imaging correlation microscope, those micro-mirrors of the DMD are used as square micro-pinholes and as square aperture, which can create diffraction patterns needed for encoding and decoding depth information. In this configuration, the light from point light source is collimated by the collimating lens and then hit to DMD. Those micro-mirrors of the DMD act as square aperture, distance from DMD to sample are rather large compared to the size of the square aperture, so the diffraction patterns created by DMD micro-mirrors need to follow Fraunhofer diffraction theory [16 17, 19, 21]. Consider a Fraunhofer diffraction with a distance, apart from a simple rectangular aperture,, its complex amplitude can be written as below: (1) Generally, consider a rectangular aperture of sides of leading to: respectively, hence
(2) where is the Bessel function of order of zero; is the wavelength; is the wave number. The diffraction pattern for square pinhole was illustrated by Fig. 3 and Fig. 4. (3) Fig. 2. 2D and 3D visualization for Fraunhofer pattern of a square aperture (computer generated). Fig. 3. 2D and 3D visualization of the same pattern but with increased exposure time to bring out some of the faint terms (computer generated). Now, the effect of a focus lens to this pattern should be considered.
Figure 4. Simple diagram for the defocus effect of a focus lens; is the focal length; where is the defocus term. The equation (3) shows the complex amplitude before lens after the pattern spread a distance of. This amplitude will be affected by the focus lens and become: (4) where is the function of thin lens [19, 20]. At the distance apart from the lens, the complex amplitude becomes: (5) Replacing of (4) into (5) results in:
(6) So, the axial intensity will be the square of equation (6). For far field diffraction, when the distance between aperture and screen is increased, the diffraction pattern will become larger and less contrasted but it tends to keep the same pattern. However, by applying confocal microscope setup, with in- and out-focus effect the diffraction pattern we see on camera will change when the distance between sample and objective is increased or decreased. Therefore, with each depth position, we will have the corresponding imaged diffraction pattern. This will be the signature pattern for this depth and can be used for depth encoding and decoding process. In our application, because of light intensity, those pinholes created from DMD micro-mirrors consist of 4 micro-mirrors arranged in a a symmetric configuration as shown in Fig. 5. The diffraction pattern generated by this configuration is different from simple single rectangular aperture. The diffraction pattern will be a superposition of 4 Fraunhofer diffraction patterns of 4 square micro-mirrors. Fig. 5. Pinhole s configuration 4. Confocal effect In the traditional confocal principle, pinholes are used to prevent stray light from getting into the optical system, also to prevent cross-talk effect between adjacent measuring points and between adjacent detecting sensors. When the object locates on the focal plane of a confocal layout, the reflected light from the object s surface can pass through a pinhole and reach to the detector with a maximum light intensity. For the traditional confocal microscope, a vertical scanning process is required to generate a depth response curve of the light intensity. The light intensity, which is detected in the above conjugative optical configuration, can be represented by Eq. (7). where (7)
where is the wave - length of the light source; z is the vertical distance; and is the numerical aperture. In the developed system, these pinholes being created by controlling micro-mirrors on DMD are used to form a one-to-one conjugate relation between each incident light and its corresponding detecting pixels. The conjugate relation will help to avoid the cross-talk problem in the full-field measurement. The DMD will be loaded with a specific pinhole array pattern, which is designed to spatially filter unfocused light and other possible stray lights away from the corresponding detecting sensor, thereby minimizing the lateral cross-talk between the detected image sensors. When one lens is used for both objective and imaging lens, the light intensity function of acquired signals can be described as follows. (8) where denotes the pupil function of objective; is the zero-order Bessel function, and are the normalized optical radii. (9) Furthermore, this configuration will cause the change of diffraction pattern when the incident light goes through the objective, hits with measuring sample's surface or reference mirror's surface, then reflects back and finally is imaged on the camera's sensor. Based on this principle, we can replace the intensity-depth response curve by using normalized cross correlation-depth response curve and achieve a quasi one-shot full-field confocal microscope system. First, the specific pinhole array will be loaded into DMD, and then a reference mirror used to build a reference database. Vertical scanning calibration procedure was performed by using PZT. At each step each vertical position, the 12-bit mono camera will take one corresponding picture. When measuring a sample, we just need to take one picture, and with the help of multiple pinholes array configuration generated by controlling DMD, we can achieve full-field measurement with one picture imaging. With the reference database being established, we can perform a special searching algorithm to find the best focus imaged position, size and location of each pinhole on the database images. The sample result of this step is shown in Fig. 7. Because of in- and out-focus effect of the confocal configuration, each diffraction pattern of each pinhole is unique for each step height and can be used as a signature identifier for this corresponding step. The change of diffraction pattern by in- and out-focus of confocal microscope setup is shown in Fig. 8.
Fig. 7. Autolocate pinholes algorithm result. Fig. 8. Diffraction pattern change by in- and out-focus effect of confocal configuration With this process, each pinhole will have an array of sub-image being contained with diffraction patterns corresponding to step heights. When measuring a sample, the camera just needs to take only one shot and based on pinhole's locations which can get from the previous step, the sub-image contains diffraction patterns of those pinholes can be extracted. The extracted sub-images can be used to calculate the normalized cross-correlation value based on the reference sub-images array of each pinhole by the Eq. (10). (10) Therefore, each pinhole, as well as each measurement point, will have its own normalized cross correlation depth response curve, illustrated in Fig. 9. When the normalized cross-correlation factor reaches to peak, this means the diffraction pattern of the measuring point is identical with the diffraction pattern of the reference point at this specific step. From that, we can convert the normalized cross-correlation factor to depth information. For the previous confocal microscopes, an important factor affecting measurement results is light intensity variation caused by undesired light source s power fluctuating and reflectivity variation of sample s surface. In our system, those disturbances can occur in both
reference database image and measured image. In here, we assume that our light spot is small enough, so in this area, the sample s surface has uniform reflectivity. Therefore, those two disturbances can be modeled as one multiplicative mode disturbance in Eq. (11); is disturbance in measured image and is disturbance in references database image. Eq. (11) will make our system immune to light intensity and surface reflectivity variations. This method can be practical to the object surface having a different absorb rate of optics from point to point. (11) where x, y are coordinate of pixels, I is intensity one measured image and R is the intensity in diffraction pattern database images. To illustrate the advantage of using normalized cross-correlation factor to extract depth information from measured diffraction patterns, a 5-step simulation was done as shown below: Step 1: Build the database by creating a set of 2D diffractive images shifted by an equal stepwise. Step 2: Create a random image suffered from noise. Step 3: Calculate N.C.C coefficient of the random image to each of the database. Step 4:Find the maximum coefficient to be represented the perfect matching of the random pattern to the database. Step 5: Derive the corresponding depth of the tested pattern by interpolation technique.
Figure 9. Description for template matching procedure Figure 10. Simulation result for template matching verification Fig. 11. Normalized cross correlation depth response curve
5. Experiment results and analysis We already developed the complete system being consisting of both software and hardware parts for quasi-one-shot full-field surface profilometry using digital diffractive-confocal imaging correlation microscope with the digital micro-mirror device. The flow chart of the developed system is shown in Fig. 10. First, a system calibration procedure needs to be performed in order to optimize light intensity. A specific DMD pinhole pattern was designed and the best light exposure time was chosen to achieve best image contrast. After this, a reference database is built by using reference mirror and PZT device as revealed in confocal effect section. The software will automatically find pinholes location from achieved database. When measuring the sample, the position of the object, light source intensity and camera exposure time also need to be calibrated to achieve the best image contrast and put the sample in the measurement range. The normalized cross correlation depth response curve can be determined with respect to the surface property. The developed system can work in quasi-one-shot or multiple shots for measuring more points by considering the requirement of applications. From the measured picture, 2-D and 3-D view of the sample can be generated and provide the reconstructed line profile or areal surface information. Fig. 10. Flowchart of digital diffractive-confocal imaging correlation microscope To check accuracy and precision of developed system, we designed a DMD pinhole pattern with a pinhole size of 2x2 pixels and the pitch between two pinholes as eight pixels. The smaller the pitch is the better lateral resolution to be achieved. However, the pitch also needs to be chosen, so a cross-talk should be minimized to ensure a satisfactory vertical resolution in profile measurement. With the configuration, we can measure almost 10000 points with quasi-one-shot in FOV. Using this pinhole pattern and a standard flat mirror, we build a reference database consisting of 400 images for 400 steps, in which the distance between each step is one μm. After this, we change the mirror to an empty wafer and use an accurate linear PZT to move the wafer into measurement range. First, we take one image shot and move the wafer to a new location of 50 μm apart from the previous location, and take the other shot. From two consecutive measured images, we can calculate the position of every measuring point and the distance of this measuring point between two measurement times. We repeat this testing procedure for 30 times and then perform statistical analysis to determine the sample mean value and random standard deviation.
For different measure times, the sample mean values were in a range from 49.6 to 50.1 μm and random standard deviations were in a range from 0.02 to 0.03 μm calculated on a group of 9852 measured points. Because our developed method is immune to not only light intensity but also sample surface reflectivity, so the precision of this method is extremely good. Moreover, the measurement accuracy still can be further enhanced by adapting an imaging device with cooling temperature control and high dynamic range for maximizing the signal to noise ratio of images. 3-D images of the wafer surface at different positions were shown in Fig. 11. Fig. 11. 3-D images of wafer surface at different position To attest the measurement accuracy of the developed measurement approach, we conducted an experimental measurement on a calibrated step-height surface target with a step height of 70 μm. With the same setup with above experiment, the average height of target was 70.16 μm. 2-D and 3-D views of this measured step height were shown in Fig. 12.
Fig. 12. 2-D and 3-D views of step-height sample Furthermore, the accuracy and precision tests also performed on diffuse surface sample as described in Fig. 13. The system was tested in diffuse surface in grinding and horizontal milling surface corresponding to Ra 0.4 and 1.6, respectively. Fig. 13. Diffuse surface sample Using the same setup as in section 5, when measuring diffuse surface sample, there are two different type of NCC response curve being received. The first one will provide valuable information for depth measurement as in Fig. 14. The other cannot find any correlation with reference database pattern because of light intensity is too weak at this measuring point as all the light have been scatter away. It may be also caused by the depth change frequency in this measured spot was two heights, so the pattern was alter significantly as described in Fig. 15.
Fig.14. NCC response for averaging case on rough sample Fig. 15. NCC response for mismatch case on rough sample From the data, we found out surface characteristics have important impacts in our measurement results. There are two factors has major role in our method efficiencies: Surface reflectance and roughness. Employing normalized cross correlation for diffractive pattern matching and spectrum response matching, our system was proved immune to light variation and surface reflectance disturbance. However, the formula just can work in case in spot size area when surface reflectance is uniform. On the other hand, surface roughness will affect our measure results when in spot size area, the depth change with high frequency. In this case, the
measured diffractive pattern will be mixed up with diffractive pattern of different heights. Therefore, there are 2 cases happened: averaging case and mismatch case. In the averaging situation, the shape of NCC-Depth response curve is not smooth curve. At the peak of this curve, there are some flat areas, however we still can detect the right peak location and can extract the right depth information from it (Fig. 14). In the mismatch case, the shape of NCC-Depth response curve is not right because we cannot find the correlation coefficients between database and measured pattern, so the right peak location cannot be detected (Fig. 15). In our rough samples testing and system characterization experiment as described above, the average case happened when we used 20-times Mitutoyo objective, which has spot size = 1.3 um on the sample with Ra = 0.4 um. The mismatch case happened when we used 20-times objective which has spot size = 1.3 um on a rough sample with Ra = 3.2 um 6. Discussion One-shot full-field surface profilometry using digital diffractive-confocal imaging correlation microscope based on DMD was successfully developed to achieve one-shot full-field surface profilometry of micro structures without vertical and lateral scanning operation. By employing an innovative method to generate normalized cross correlation depth response curve and using this to encode and decode profilometry information of sample surface, we can totally omit vertical scanning operation in confocal microscope without using special spectrometer device in chromatic confocal system or complex setup or differential system. With this new approach, the developed system is immune to the light intensity of light source and reflectivity of sample s surface. Furthermore, specific pinhole arrangement created by DMD was designed in order to minimize cross-talk effect between adjacent measuring points. The approach significantly improves accuracy and precision of the whole system. On the other hand, this system is really flexible and easy to setup the desirable measurable depth range and field of view can be obtained by choosing suitable objective and size of sensor camera and it can be ranged from a few to several hundred micrometers. From the experimental tests, it was verified that the ±3σ repeatability of the height measurement can be controlled within 0.01% of the overall measurement range. The measurement speed of the system, currently limited to capturing speed of the camera and can be further enhanced by adopting a high-speed cooling CCD with a high dynamic range and low dark current. With one-shot full-field measurement capability, this system will be a suitable solution for high speed in-situ optical inspection.