A Comparison Between Camera Calibration Software Toolboxes

2016 International Conference on Computational Science and Computational Intelligence A Comparison Between Camera Calibration Software Toolboxes James Rothenflue, Nancy Gordillo-Herrejon, Ramazan S. Aygün Computer Science Department University of Alabama Huntsville Huntsville, Alabama, USA e-mail: {jar0030, ngg0002, aygunr}@uah.edu Abstract Camera calibration has many applications in various computer vision fields such as pose estimation, robot navigation, trajectory tracking, and object recognition. Camera calibration involves (mostly) determining the intrinsic parameters of a camera so that problems or distortions caused by the camera s optics or manufacturing could be estimated for proper projection. There are already a number of toolboxes available for camera calibration. In this paper, we explain our experience using these toolboxes in terms of installation, usability, time to calibrate, and performance with respect to the resolution of images. Our experience is helpful for other researchers on how to select or develop camera calibration toolboxes. Keywords- camera calibration I. INTRODUCTION Camera calibration is the process of finding the intrinsic and, optionally, the extrinsic parameters of a camera. The intrinsic parameters of a camera include parameters related to the camera itself such as focal length and distortion. The extrinsic parameters indicate the pose of the camera with respect to a world coordinate system or other cameras. While not necessary, calibration is typically achieved by taking pictures of a calibration object. This object has a known pattern with known distances between recognizable points. The parameters of a camera are calculated by locating and comparing those points across multiple images to calculate the intrinsic parameters. Normally, this would be a long and tedious process. However, the development of software toolboxes to automate this process can make camera calibration significantly easier. The general approach of these toolboxes is to locate the points of interest on the image and then automate the process of calculating the parameters from those points. Calibration toolboxes have different approaches for camera calibration. The types of points needed, how the points are located, and how the parameters are calculated are all dependent entirely on the toolbox chosen. Some toolboxes may not perform these steps as another. Even the steps for running a calibration toolbox may be much easier than running another toolbox. We believe it would be helpful to know in advance which toolboxes make steps easier or harder, and the circumstances that these toolboxes are most accurate. In this research, we analyze the accuracy of usability of three toolboxes: 1) Jean-Yves Bouguet s Camera Calibration Toolbox for Matlab (from CalTech) [3], 2) the DLR Camera Calibration Toolbox, consisting of the programs CalDe version 0.99.42 and CalLab version 1.97 [5], and 3) the 3DF Lapyx version 1.2 toolbox from 3DFlow [1]. These are great toolboxes for camera calibration. We believe many research studies have benefited from these toolboxes. The developers and authors of these toolboxes have implemented the code for camera calibration and made it available for researchers all over the world. Their efforts are commendable. Our goal is not to show rank them or show their weaknesses. Our goal is rather to increase the efficiency of users benefiting from camera calibration toolboxes. Nevertheless, comparison of these toolboxes may help users to get the best benefit from these toolboxes for their purpose. We will first compare how easy or hard it is to get the toolkits operational. Then we will compare how hard or convenient the toolboxes are to use. Then finally, we will compare the accuracy of the results. For simplicity, we will only consider the calibration of the intrinsic properties of two single cameras on a windows operating system in this paper. We do not explore multiple camera calibration functionality (if available) in the experiments in this paper. This paper is organized as follows. The following section gives information about the installation processes of these toolboxes. Section 3 explains the methodology used in these toolboxes. Section 4 provides our experiments and evaluations. Section 5 covers the comparison of toolboxes. The last section concludes our paper. II. INSTALLATION The installation process should be as simple as possible. Difficulties that come up during installation may deter users from using those software applications. Moreover, availability of the environment to run these toolboxes is also important to work with these available toolboxes. The process of installing Jean-Yves Bouguet s Vision toolbox for Matlab is relatively straightforward. Matlab is required to operate this toolkit [3]. The toolkit is available free. If Matlab is available, the folder containing the toolbox should be added to Matlab s path and then the toolbox is ready for use [3]. The availability of Matlab is the major limitation for usability of this toolbox. The DLR toolbox is not difficult to install. The two components of software that make up the toolbox are free for 978-1-5090-5510-4/16 $31.00 2016 IEEE DOI 10.1109/CSCI.2016.149 768 772

non-commercial use. The process of installing them is as simple as downloading the files [5]. The software can only run through the IDL virtual machine. This IDL virtual machine is available from a site through a registration process [5]. The registration is not automatic and goes through an approval process. The 3DF Lapyx toolbox is the easiest of the three to install. The tool is available at [2]. A calibration pattern is available from the software itself and the software is ready to calibrate once images for calibration are ready. All these toolboxes are fairly easy to install for anyone who has interest and background in computer vision. The availability of Matlab could be issue for Bouguet s toolbox for some users. III. METHODOLOGY AND USABILITY In this section, we briefly provide information about the methodology of camera calibration used by these toolboxes. We also mention how much human intervention could be needed during the calibration process. Jean-Yves Bouguet s camera calibration toolbox [3] for Matlab is primarily based off Zhengyou Zhang s work in [6]. Zhang uses Heikkila and Silven s camera model [7] with some alterations [3]. The toolbox solves for the parameters of the camera using an analytical technique before optimizing them through nonlinear iterative methods [6]. Using the toolkit is simple, but it requires some user input for corner detection. The software needs the user to point out the corners of the checkerboard pattern on the calibration object before it can attempt to locate the other corners [3]. Assuming distortion is minimal, the software can detect the rest of the necessary points relatively easily. The user has to provide only an estimate on the distortion in the image if the distortion in the initial corner guesses it generates is significantly off [3]. Otherwise, the process is largely automated. Its results can be further refined [3]. For the sake of this comparison with the other toolboxes, we will not further refine the results beyond the initial calibration to prevent alteration to the data. It has separate functions for calibrating two cameras together, but that functionality will not be used in this comparison [3]. The DLR Camera Calibration Toolbox uses reprojection error minimization and closed formed solutions to calculate the properties of the camera, but focuses on the originality of its methods to calculate the extrinsic parameters of a camera on a robot [5]. In this paper, we do not use it to calculate these parameters. It can support multiple-camera calibration. Here, we test how well its algorithms work for single camera calibration. The DLR Calibration Toolbox has two software applications: CalDe and CalLab. CalDe is used to locate the corners on the images, whereas CalLab uses the data from CalDe to perform the actual calibration. CalDe has an automatic detection algorithm for finding the circles on the calibration pattern that define its coordinate system, and then finds the corners of the checkerboard automatically [5]. The only user input needed in the ideal case is the configuration file needed to define the calibration object s size and pattern. In case the automatic detection fails, the user can point out the central points, or even manually select corners that are mistakenly identified or missed [5]. After the corner detection, the actual calibration is easy. The user needs only to look over the corners to find undetected corners and remove any incorrect corners by CalDe. The user starts the calibration and then saves the results to a file [5]. There are additional phases of calibration such as providing the location of the camera in relation to a fixed origin point, but those phases are unnecessary to find the intrinsic parameters alone, and are thus beyond what we use for experiments in this paper. 3DF Lapyx provides very little information on the methodology it uses to find intrinsic parameters [2]. It uses Brown s distortion model, which is the same basic model used by Bouguet s Matlab toolbox and the DLR toolbox [1,3,5]. Using 3DF Lapyx is not complicated. The user only needs to select the pictures for calibration, review the fully automatic corner detection for errors, and then let the toolbox perform the calibration [1]. If the user finds an error in any picture, the picture should be rejected by the user. The corner detection stage asks for the number of columns and rows. IV. EXPERIMENTS A. Experimental Setup In our experiments, we used a pair of Logitech C525 cameras fixed on a tripod (Fig. 1). We refer these cameras as left and right cameras. We ran 5 calibration trials for each toolbox on 3264x2448, 1600x1200, 640x480, and 320x240 resolution settings with each camera. In each trial, 20 pictures per camera were taken for Bouguet s Matlab and the 3DF Lapyx toolboxes in order to obtain the most accurate results [3]. A total of 5 trials were performed per resolution setting, so that each setting had a total of 100 different pictures. Figure 1. Logitech C525 model cameras To isolate the resolution as the only changing factor, the picture of each calibration object pose was taken in 4 different resolution settings. This gave a total of 400 pictures per camera. The pictures with both cameras were taken 769 773

without moving the object or the cameras until each pose was captured in the 4 resolutions, resulting in a total of 800 pictures of the object for Bouguet s Matlab toolbox. The same calibration object was also usable for the 3DF Lapyx, so we used the same set of pictures for both toolboxes (Fig. 2a). The DLR toolbox required its own calibration pattern, but only requests 10 pictures for each trial. Its calibration object is provided in Fig. 2b. Therefore, we took a total of 400 pictures of its pattern using the same procedure as the previous set, bringing our final picture count to 1200. a) b) Figure 2. a) Matlab and 3DF Lapyx calibration object and b) DLR calibration object B. Calibration Experiments We calibrated each camera separately with each toolbox producing 5 sets of intrinsic parameters per combination of resolution, camera, and toolbox. 60 calibrations were performed for each camera. 1) Calibration with DLR Our experience with performing the DLR toolbox calibrations varied dramatically from the ideal case. While the user input needed for corner detection is theoretically low, in practice the process was neither simple nor easy. For our setup and our configuration, the automatic detection of the origin control points was not successful. After trying to use it 5 times per resolution and at least once on each camera, CalDe did not detect the control points with its auto-detect functionality. As a result, manual selection of the control points was needed for all the images. While the actual corner detection was generally more accurate, it still was noticeably less reliable than the other toolboxes. The nature and degree of its issues were dependent on the image resolution. For 3264x2448 images, the corner detection had a tendency to miss points, forcing the user to manually select the points that it missed. It missed at least 1 point on approximately 40% of the images on this resolution, and missed 10 or more points on about 14% of the images. For 320x240 and 640x480 resolution images, it missed far fewer points, but had a much higher chance to misidentify their locations. The rates of incorrect identification of at least one point on an image was around 27% and 18% for 320x240 and 640x480 resolutions, respectively. It incorrectly identified 10 or more points on 6% and 4% of the images, respectively for the corresponding resolutions. These numbers are significantly lower than those for 3264x2448, but they still indicate a notable amount of human intervention. The optimal resolution for corner detection accuracy with DLR CalDe was 1600x1200, at which the miss rate for at one or more points was only 16% and the miss rate for 10 or more points was 3%. These numbers are the lowest of all the resolutions for the image samples we tested, indicating that the corner detection algorithm works best at this resolution. The time for the corner detection algorithm increases significantly as the resolution of the image decreases. Calibrations for a set of ten 3264x2448 images took about 20-30 minutes including corrections. Calibrations on the lower settings, however, typically took closer to 10 minutes. Additionally, resolutions of 3264x2448 and 1600x1200 were more susceptible to light glare causing incorrect corner detection than the lower resolution images. The DLR calibration software worked effectively without any significant issues. Once the corner detection is complete, the DLR toolbox is just as easy to use as the other toolboxes. The corner detection is the only true difficulty in the use of this toolbox. 2) Calibration with 3DF Lapyx The 3DF Lapyx did not work well for 320x240 resolution images. It was unable to detect the corners correctly in most of the pictures and those pictures were discarded. As a result, it either could not perform the calibration or it gave an extremely high projection error after processing the pictures. The number of pictures that could be processed was not more than two or three. The low resolution pictures completely crashed the program in one case as well. The calibration for the pictures with 640x480 resolution using 3DF was better than for 320x240 resolution images. In the first experiment it only incorrectly detected points in one of twenty pictures. In the second all pictures were detected correctly and the third test correctly identified eighteen out of twenty pictures. In the fourth test only one picture had an error, and in the last only two pictures were incorrect. The results were consistent in all experiments unlike the ones with the low resolution. For the 3264x2448 and 1600x1200 resolutions, all corners were detected correctly with 3DF. 3) Calibration with Bouguet s Toolbox using Matlab Bouget s toolbox using Matlab performed remarkably better than 3DF for 320x240 resolution. Manual corner detection is one of the drawbacks of this toolbox. The low resolution made the program run a lot smoother and faster. Unlike 3DF it was able to detect all the corners correctly and the results were consistent. However, the corners of the checkerboard appeared fuzzy and were somewhat hard to detect. The calibration with this toolbox for images of 640x480 resolution was also performed successfully. In this case, the quality was acceptable. There were no problems with the corner detection, and it was running as fast as with the low resolution. The calibration took longer for 3264x2448 resolution images. The images were also taking longer time to load. 770 774

Table I provides the average time for calibration for these toolboxes regardless of correct calibration. TABLE I. AVERAGE CALIBRATION TIMES. 20 PICTURES PER 3DF AND BOUGUET S CALIBRATION AND 10 PICTURES PER DLR CALIBRATION. Toolbox Resolution Avg. Time(s) 3DF 320X240 156 Bouguet s 320X240 648 DLR 320X240 708 3DF 640X480 324 Bouguet s 640X480 714 DLR 640X480 594 3DF 1600X1200 258 Bouguet s 1600X1200 936 DLR 1600X1200 666 3DF 3264X2448 246 Bouguet s 3264X2448 1014 DLR 3264X2448 1518 C. Validating Calibration Results After we finished all of the calibrations, we then proceeded to validate them. We projected points from a selection of test images onto the image plane with a modified function from the Mirage pose estimation software [4]. Then, we implemented a program to find the distance between the projected points and the actual locations of the points on the image. Since the distance the software finds is measured in pixels, we normalized the results across all the resolutions by dividing the average distance for a given calibration by the x component of the resolution at which the calibration was taken and tested, allowing us to directly compare the accuracies of the calibrations. The top ten calibrations for the right camera are provided in Table II. The top calibrations for the left camera are provided in Table III. We only tested calibrations with images taken at the same resolution the camera was calibrated for. While the calibration itself should not change with resolution, the components of the calibration measured in pixels would need to be converted every time the resolution changes. Therefore, we only used the raw data for the same resolution tests. We took pictures of six poses of the testing rig with each camera at each resolution to perform the projections. We measured the coordinates of the points used for the tests, as well as the coordinates of the camera itself in millimeters. We did not have a method for directly measuring the rotation angles of the cameras, so we had to estimate the rotation such that the point projections produced were as close as possible to the correct values. TABLE II. TOP TEN CALIBRATIONS FOR THE RIGHT CAMERA. Toolbox Resolution Experiment Error Number Value 3DF 640x480 1 0.028802 3DF 1600x1200 1 0.032562 Bouguet s 640x480 1 0.032909 Bouguet s 3264x2448 1 0.036029 3DF 3264x2448 1 0.037356 Bouguet s 1600x1200 1 0.038729 Bouguet s 640x480 2 0.042048 3DF 1600x1200 5 0.042696 Bouguet s 1600x1200 5 0.043044 Bouguet s 320x240 1 0.043572 TABLE III. TOP TEN CALIBRATIONS FOR THE LEFT CAMERA. Toolbox Resolution Experiment Error Number Value Bouguet s 3264x2448 2 0.004355 Bouguet s 1600x1200 2 0.004558 3DF 1600x1200 2 0.004582 3DF 3264x2448 2 0.004727 3DF 3264x2448 5 0.004770 DLR 320x240 4 0.005285 Bouguet s 3264x2448 5 0.005305 DLR 320x240 2 0.005457 Bouguet s 640x480 2 0.006022 3DF 1600x1200 5 0.006065 D. Discussion The DLR method did not perform as good as the others based on top ten rankings. In fact, for the right camera, the DLR calibrations have some of the worst results of all the calibrations. The left camera has a slightly more even spread, but still did not have DLR method in the top 5 calibrations. Therefore, it would seem that DLR is the least accurate of the toolboxes overall under the circumstances we performed our calibrations. The absolute worst calibrations were obtained using 3DF for 320x240 trials. About 50% of the trials crashed the software, and the remainder produced error values two to three times as high as the highest errors from other toolboxes. 3DF is obviously not designed to work on 320x240 resolution. If 3DF is used, resolutions 640x480 or more should be used. The accuracies of Bouguet s toolbox and 3DF toolbox seem to be similar. Bouguet s Matlab toolbox appeared in the top ten calibrations more than 3DF. However, 3DF produced the best calibration by a notable margin for the right camera. They seem to favor the same image sets, as well as the same resolutions, so the difference between them seems to be relatively small for resolutions above 320x240. The top 3DF calibration is obtained both for a lower resolution calibration and has a much smaller error. However, this could be an outlier as well. Its other results are 771 775

still close to the Bouguet s toolbox results suggesting the two accuracies are nearly the same. Another point to be compared is the effect of resolution on the error values. Bouguet s toolbox and 3DF both seem to favor high resolutions, but 1600x1200 resolution performs nearly as well as 3264x2448 suggesting both resolutions almost perform equally well. On the other hand, DLR s low error results seem to be for low resolution images suggesting it is more accurate at such resolutions. There are some lower resolutions for DLR that appear in the top calibration results. However, it even produces good results for images of 640x480 resolution. 320x240 resolution rarely seems to produce accurate results under any circumstances. Finally, in terms of ease of use, the comparisons are not so hard to make as some of our other points. Assuming that less user intervention is more convenient than more, we can easily rank the toolboxes by the degree of effort required to operate them. 3DF requires almost no user input. It is the easiest to use among these three. Bouguet s Matlab toolbox requires slightly more effort, but is still relatively user friendly. The DLR toolbox, on the other hand, requires a large amount of user input, greatly increasing the amount of effort needed to complete a calibration. While this one metric is not the only one that can be used to measure user friendliness, it is a useful and important one, that clearly ranks the toolboxes into different categories. Bouguet s Matlab toolbox and 3DF seem to produce similarly accurate results. They work especially well for resolutions 1600x1200 and above. 3DF is easier to use than the Bouguet s toolbox. However, 3DF does not produce good results for 320x240 resolution images. DLR, on the other hand, is normally both harder to use and less accurate, but tends to produce better calibrations for very low resolutions. Its low accuracy could be due to only calibrating one camera at a time. For single camera calibration, it is the least accurate of the three. If a user would like to do quick camera calibration, the user may start tests with 3DF for 1600x1200 resolution as this resolution may provide accurate results. It gives fast results with less human intervention. comparison, or further explore the more advanced capabilities of each toolbox. ACKNOWLEDGEMENT This material is based upon work supported by the National Science Foundation under Grant No. EEC- 1359311. We would like to thank Semih Dinc and Khomsun Singhirunnusorn for allowing us to use Mirage pose estimation software and their help for planning the experiments. REFERENCES [1] 3Dflow. (n.d.-a). 3DF Lapyx. Verona: 3Dflow. Retrieved from http://www.3dflow.net/tools/samantha-structure-from-motion/ [2] 3Dflow. (n.d.-b). 3DF Samantha - Structure from Motion at its finest. Retrieved June 29, 2016, from http://www.3dflow.net/tools/samantha-structure-from-motion/ [3] Bouguet, J.-Y. (2015). Camera Calibration Toolbox for Matlab. Retrieved June 9, 2016, from http://www.vision.caltech.edu/bouguetj/calib_doc/ [4] Dinc, S., Fahimi, F., & Aygun, R. (2016). Vision based trajectory tracking for mobile robots using Mirage pose estimation method. IET Computer Vision. University of Alabama in Huntsville. http://doi.org/10.1049/iet-cvi.2015.0153 [5] Strobl, K. H., Sepp, W., Fuchs, S., Paredes, C., Smisk, M., & Arbter, K. (n.d.). DLR CalDe and DLR CalLab. Retrieved June 9, 2016, from http://www.robotic.dlr.de/callab/ [6] Zhang, Z. (2000). A Flexible New Technique for Camera Calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330 1334. Retrieved from http://research.microsoft. [7] Janne Heikkila and Olli Silven. 1997. A Four-step Camera Calibration Procedure with Implicit Image Correction. In Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97) (CVPR '97). IEEE Computer Society, Washington, DC, USA, 1106-. V. CONCLUSION In this paper, we have compared the accuracy and usability of three toolboxes. We analyzed the circumstances under which they produce the best results. This will allow users to better judge which toolbox is best suited to their needs, and help them make better decisions regarding their choice of toolbox. We believe each toolbox may have their advantages under some circumstances. However, 3DF for 1600x1200 could be tested for initial calibration experiments. There are still ways that this comparison can be improved. More precise methods of measuring distances is needed especially for camera rotation angles. We could also use more types of cameras than the two Logitech C525 models we used here to avoid dependency on the cameras. We could also test the multi-camera calibration tools in the toolboxes and compare the accuracy with the single camera calibrations. We could use different toolboxes to perform the 772 776