Image Quality Testing of Fire Service Thermal Imaging Cameras

Transcription

1 Image Quality Testing of Fire Service Thermal Imaging Cameras Final Report Prepared by: Francine Amon, Ph.D. Borås, Sweden September 2011 Fire Protection Research Foundation THE FIRE PROTECTION RESEARCH FOUNDATION ONE BATTERYMARCH PARK QUINCY, MASSACHUSETTS, U.S.A WEB:

2 (This page left intentionally blank)

3 FOREWORD Thermal imaging cameras are a relatively new technology being increasingly used by today s fire service. Standardized performance requirements are being addressed by a new NFPA 1801, Standard on Thermal Imagers for the Fire Service. This document addresses the design, performance, testing, and certification requirements for thermal imagers used by fire service personnel during emergency incident operations. One key set of tests in NFPA 1801 is for image quality, and additional validation of these test methods in a round robin format is pending. These image quality tests are complicated and the degree to which they are reproducible from testing laboratory to testing laboratory has not been fully established. This is a critical step for the purposes of testing and certification of products. ASTM E691 09, Standard Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method is a recognized existing methodology to evaluate the repeatability and reproducibility of the NFPA 1801 test methods. The objective of this study is to establish Lab-to-Lab repeatability and reproducibility of the four image quality tests in NFPA 1801 that address: Spatial Resolution, Non-uniformity, Effective Temperature Range, and Thermal Sensitivity. The final results indicate acceptable levels of repeatability and reproducibility for certain test characteristics but not for others. The Research Foundation expresses gratitude to the report author Francine Amon, Ph.D. Dr. Amon is currently located in Borås, Sweden. She started this project while employed with the National Institute of Standards and Technology (NIST) located in Gaithersburg, MD, and has finished it while serving as a member of staff at SP Technical Research Institute of Sweden. NIST facilities were used for one half of the inter-laboratory test series, and their support and involvement is appreciated. The other half of the test series was conducted by Intertek Testing Services of Cortland, NY under the direction of Mr. Chad Morey. The in-kind support and involvement of Intertek Testing Services and Mr. Morey has been a critical aspect of completing this study and is appreciated. The Fire Protection Research Foundation appreciates the guidance provided by the Project Technical Panelists, the funding provided by the sponsors, and all others that contributed to this research effort. The content, opinions and conclusions contained in this report are solely those of the author.

4 (This page left intentionally blank)

5 PROJECT TECHNICAL PANEL Robert Athanas, FDNY / SAFE-IR (NY) Bill Haskell, NIOSH (MA) Jeff Hull, District of Columbia Fire & EMS Dept (DC) Stephen Sanders, Safety Equipment Institute (VA) Dave Trebisacci, NFPA (MA) Bruce Varner, Chair of NFPA TC on Electronic Safety Equipment (AZ) PROJECT SPONSORS Drager Safety AG & Co. (Germany) Honeywell First Responder Products (OH) Intertek Testing Services (NY) ISG Thermalsys (GA) MSA (PA) NFPA (MA) Tyco/Scott Health and Safety (NC)

6 (This page left intentionally blank)

7 Fire Protection Research Foundation Image Quality Testing of Fire Service Thermal Imaging Cameras Francine Amon, Ph.D. Borås, Sweden

8 ABSTRACT This document reports on the evaluation of test methodology developed to measure the image quality of thermal imaging cameras (TIC * ) used by the fire service. These tests were conducted in accordance with selected sections of the National Fire Protection Association (NFPA) 1801 Standard on TICs for the Fire Service. These tests measured 1) nonuniformity, 2) spatial resolution, 3) effective temperature range, and 4) thermal sensitivity of fire service TIC and were conducted independently by two testing laboratories in order to investigate the reproducibility of the test methods. Six TIC were tested. Each testing laboratory began with three TIC and, when the tests of the initial three TIC were complete, exchanged TIC and performed the same tests on the remaining three TIC. Each component of the four major test methods was conducted three times on each TIC in order to investigate the repeatability of the test methods. The results show that the repeatability of all the tests is poor. Some of the nonuniformity test results and most of the thermal sensitivity test results showed good reproducibility, but the rest of the test results leave much room for improvement. Overall, the test methods require revision to improve both repeatability and reproducibility of the test results. Suggested revisions that apply to multiple tests are: specify the alignment of the target components with respect to the TIC under test in more detail; require TIC manufacturers to provide a mounting device and lens for the Nikon camera; and specify the selection of the region of interest in more detail. 1. INTRODUCTION Six different TICs were evaluated by the National Institute of Standards and Technology (NIST) Fire Research Division and by Intertek Commercial & Electrical, of Cortland, NY using performance metrics and testing protocols that were developed specifically for evaluating TIC utilized by firefighters, and that have been integrated into NFPA 1801 Standard on TICs for the Fire Service (NFPA 1801) [1]. The performance metrics for these tests were Nonuniformity (NU), Spatial Resolution (SR), Effective Temperature Range (ETR), and Thermal Sensitivity (TS). The TIC tested were considered black boxes in the sense that a target was placed in the field of view and the resulting image that appeared on the TIC s display was captured by a high resolution Nikon D3 visible camera and processed using MATLAB and Excel software; the performance of the individual imaging components of the TIC was not measured. This testing procedure was performed in accordance with the standard practice specified in ASTM E691 Standard Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method [2], with some deviation due to there being only two testing laboratories participating * In this report the term TIC refers to both the singular and plural forms of thermal imaging camera(s), as is appropriate to the context in which it is used in the text. Certain companies and commercial properties are identified in this paper in order to specify adequately the source of information or of equipment used. Such identification does not imply endorsement or recommendation by the National Institute of Standards and Technology, nor does it imply that this source or equipment is the best available for the purpose. 1

9 in this study. The methods of measurement and the results of these limited tests are presented in this report. The results of each individual test will be presented in the following sections. A multivariate model was created from firefighter perception test results collected at US Army s Night Vision and Electronic Sensors Directorate (NVESD) and is used to evaluate the imaging performance (P) of the TIC in accordance with NFPA In this way, human perception is used to determine the quality of the image while the TIC are tested using objective laboratory test methods. One of the laboratories did not provide raw test data for most of the tests, however, both the raw data and the model variable P are presented and compared whenever possible. 2. PERFORMANCE METRICS AND TEST METHODS The six TIC used for this project were labeled TIC A through TIC F to preserve the anonymity of the TIC manufacturers, likewise, the participating laboratories were labeled Lab 1 and Lab 2. The characteristics of TIC A through TIC F are not germane to the purpose of this project, which is to evaluate the testing methodology rather than the actual performance of the TIC so descriptions of the TIC characteristics are not included in this report. However, it should be mentioned that TIC D is a scientific-grade TIC with adjustable optics and a much narrower field of view than the other TIC. TIC D was included in the specimen collection in order to provide a comparison between fire service and scientific TIC; an effort which may be conducted in the future. TIC D test results are not presented when the narrow field of view or other design limitations prohibit testing without introducing special protocols into the test methods. 2.1 Test order The order in which the individual tests were performed for this evaluation was randomized as much as possible to minimize the effects of instabilities in the testing environment and test operator. As mentioned above, each laboratory tested three TIC as a batch, then swapped batches and tested the remaining three TIC. Within each batch of three TIC, the order of the nonuniformity tests having a target temperature of 30 o C (NU30) was randomized and these tests were conducted first because the results of this test were needed as input to the perception model for other tests. The order of the remaining tests was randomized jointly as a single group, with the caveat that the effective temperature range (ETR) tests could only be conducted a maximum of 3 times per day because they require extra time between tests for the equipment to cool down. 2.2 Nonuniformity Nonuniformity (NU) is a measure of the TIC s response to a uniform thermal target [3, 4]. Three tests for each TIC were conducted at each of five target temperatures: 1 C, 30 C, 100 C, 160 C, and 260 C, which span the temperature range of interest to the fire service [5]. Wellcharacterized extended-area blackbodies were used as targets. The TIC under test was positioned directly in front of the blackbody such that the image of the blackbody surface completely filled the TIC s field of view and the TIC was normal to the plane of the blackbody surface. 2

10 A Nikon D3 digital visible camera was used to take ten 16-bit digital photos of the images displayed by the TIC under test for each target temperature. These images were initially stored in flash memory on the Nikon, and then transferred to a personal computer (PC) for further processing. The processing steps were as follows: 1. Convert the images from the proprietary Nikon NEF format to 16-bit TIF files. 2. Convert the TIF files to grayscale as defined per IEC (1999) [6]. 3. Use a MATLAB program to define a region of interest (ROI) that encompasses at least 90 % of the FOV of the TIC under test. Exclude or remove symbols, icons, and text from this ROI. 4. Filter out the high-frequency noise created by oversampling the TIC s display. This was accomplished with a filter which averaged over an area whose width and height was determined by the size of the TIC s display pixels. 5. Calculate the NU for each image using the following equation [3] σ NU = (1) µ Where σ is the standard deviation and µ is the mean, respectively, of the pixel intensity values in the ROI. Retain the original pixel intensity values for later use. 6. Rank the ten NU values collected at each target temperature, then discard the highest NU value from the data set. 7. Average the NU values for the three individual tests conducted at each target temperature. 8. Record the NU30 test result for use as input to the perception model for other tests. 9. When data is available from the spatial resolution (SR) test, apply the model to each result, producing the P value for the nonuniformity tests. This is the pass/fail value for the nonuniformity tests. The nonuniformity results at 30 o C (NU30) are of particular interest because they are used as input for the perception model in other tests, therefore these results are presented in Figure 1 and discussed separately. The results of all of the other nonuniformity tests, in terms of P, are presented in Figures 2 through 6, with discussion following presentation of all the NU figures. Throughout this document, when less than three tests per TIC were conducted for a test, a note is placed on the graph to differentiate between having insufficient data to create error bars and situations where very small values of uncertainty cause the error bars to be obscured by the data point marker. 3

11 TIC A TIC B TIC C TIC D TIC E TIC F Figure 1. NU30 test results, note that these results are not presented in terms of the perception model parameter P NU30 discussion Both laboratories conducted the expected three NU30 tests for each of the six TIC. Repeatability of the tests, as indicated by the length of the error bars, is somewhat better (shorter error bars) in Laboratory 1 than in Laboratory 2 for most of the test specimens. For Laboratory 2, test specimens 4 and 5 were particularly troublesome, however these TIC did not show the same trend for Laboratory 1. With respect to reproducibility, the NU30 values were virtually identical for TIC 1 and within the measurement uncertainty for TIC 2, but diverged considerably for the other TIC, particularly test specimens 4 and 5. As previously mentioned, the NU30 test is of particular interest because the raw NU30 data is used as input to the perception model for some of the other tests. If there is poor repeatability and/or reproducibility in the results of this test, it will propagate into the model results (P) for the other tests that use NU30 as input. 4

12 Figure 2. Nonuniformity results for TIC A. Note that the Lab 2 value at 30 o C is NU30, rather than P (which is unavailable), and is included only as an indication of the uncertainty of the measurement TIC A nonuniformity The nonuniformity test results for target temperatures of 1 o C, 30 o C, 100 o C, 160 o C, and 260 o C for TIC A, in terms of the perception model P, are shown in Figure 2 above. Both laboratories conducted all of the expected tests for TIC A, however, Laboratory 2 provided results at the target temperature of 30 o C only in terms of NU30. Repeatability of the tests, as indicated by the length of the error bars, is significantly better (shorter error bars) for Laboratory 1 than for Laboratory 2 for most of the target temperatures. The repeatability of the test results does not appear to be a function of target temperature. With respect to reproducibility, the results at target temperatures of 1 o C, 160 o C, and 260 o C are all within the uncertainty of the measurements, however, in all cases this is a consequence of very large measurement uncertainties. The reproducibility of the test results does not appear to be a function of target temperature. 5

13 Figure 3. Nonuniformity results for TIC B. Note that the Lab 2 value at 30 o C is NU, rather than P (which is unavailable), and is included only as an indication of the uncertainty of the measurement TIC B nonuniformity The nonuniformity test results for target temperatures of 1 o C, 30 o C, 100 o C, 160 o C, and 260 o C for TIC B, in terms of the perception model P, are shown in Figure 3 above. Both laboratories conducted all of the expected tests for TIC B, however, Laboratory 2 provided results at the target temperature of 30 o C only in terms of NU30. Repeatability of the tests, as indicated by the length of the error bars, is significantly better (shorter error bars) for Laboratory 2 than for Laboratory 1 for two of the target temperatures. The repeatability of the test results does not appear to be a function of target temperature. With respect to reproducibility, the results at all of the target temperatures are outside the uncertainty of the measurements, particularly at the target temperature of 260 o C. The reproducibility of the test results does not appear to be a function of target temperature. 6

14 Figure 4. Nonuniformity results for TIC C. Note that the Lab 2 value at 30 o C is NU, rather than P (which is unavailable), and is included only as an indication of the uncertainty of the measurement TIC C nonuniformity The nonuniformity test results for target temperatures of 1 o C, 30 o C, 100 o C, 160 o C, and 260 o C for TIC C, in terms of the perception model P, are shown in Figure 3 above. Both laboratories conducted all of the expected tests for TIC C, however, Laboratory 2 provided results at the target temperature of 30 o C only in terms of NU30. Repeatability of the tests, as indicated by the length of the error bars, is significantly better (shorter error bars) for Laboratory 1 than for Laboratory 2 for all of the target temperatures. The repeatability of the test results does not appear to be a function of target temperature. With respect to reproducibility, the results at all of the target temperatures are within the uncertainty of the measurements, however, in all cases this is a consequence of very large measurement uncertainties. The reproducibility of the test results does not appear to be a function of target temperature TIC D nonuniformity 7

15 Laboratory 2 did not collect nonuniformity data for TIC D at target temperatures other than 30 o C so no data is available for analysis of repeatability or reproducibility. Figure 5. Nonuniformity results for TIC E. Note that the Lab 2 value at 30 o C is NU, rather than P (which is unavailable), and is included only as an indication of the uncertainty of the measurement TIC E nonuniformity The nonuniformity test results for target temperatures of 1 o C, 30 o C, 100 o C, 160 o C, and 260 o C for TIC E, in terms of the perception model P, are shown in Figure 5 above. Laboratory 1 conducted all of the expected tests for TIC E, however, Laboratory 2 provided results at the target temperature of 30 o C only in terms of NU30 and conducted only one test at 260 o C. Repeatability of the tests, as indicated by the length of the error bars, is significantly better (shorter error bars) for Laboratory 1 than for Laboratory 2 for all of the target temperatures. The repeatability of the test results does not appear to be a function of target temperature. With respect to reproducibility, the results at target temperatures of 1 o C and 100 o C are within the uncertainty of the measurements, however, this is a consequence of very large measurement uncertainties. The reproducibility of the test results does not appear to be a function of target temperature. 8

16 Figure 6. Nonuniformity results for TIC F. Note that the Lab 2 value at 30 o C is NU, rather than P (which is unavailable), and is included only as an indication of the uncertainty of the measurement TIC F nonuniformity The nonuniformity test results for target temperatures of 1 o C, 30 o C, 100 o C, 160 o C, and 260 o C for TIC F, in terms of the perception model P, are shown in Figure 6 above. Laboratory 1 conducted all of the expected tests for TIC E, however, Laboratory 2 provided results at the target temperature of 30 o C only in terms of NU30 and conducted only one test at target temperatures of 1 o C, 160 o C, and 260 o C. Repeatability of the tests, as indicated by the length of the error bars, is difficult to analyze thoroughly due to the absence of data collected by Laboratory 2. Based on Laboratory 1 results, the repeatability of the tests does not appear to be a function of target temperature. With respect to reproducibility, the results at the target temperature of 100 o C are within the uncertainty of the measurements, however, this is a consequence of very large measurement uncertainties in spite of the near identical average P values. The reproducibility of the test results does not appear to be a function of target temperature. 9

17 2.2.8 Suggestions for improvement Given the state of repeatability and reproducibility of the nonuniformity test results, improvements to this set of tests are recommended. These improvements are listed below (not prioritized). Specify position of the TIC in front of the blackbody surface in more detail. Require TIC manufacturers to provide a mounting device and lens for the Nikon camera to ensure that it always views the TIC display from the same position. Specify the selection of the ROI, including the method of removing symbols, icons, and text, in more detail. Rotate the test/target configuration 90 degrees, such that the TIC is looking upward to view the blackbody surface. This arrangement may mitigate target temperature gradients driven by buoyancy at high temperatures. 2.3 Spatial Resolution The spatial resolution (SR) performance metric is a measure of the ability of TIC to reproduce the details of a scene or target [7]. In NFPA 1801, the spatial resolution test is used both to measure an TIC s spatial resolution and to determine whether a design robustness test, e.g., the vibration test, has impacted the TIC s ability to produce images of acceptable quality. This test was performed three times for each TIC and the results were averaged to determine the SR. The TIC viewed a thermal target comprised of two sets of converging lines, as shown in Figure 7. This target is a portion of the target used by the ISO Spatial Resolution Standard ISO [8]. The target foreground and background were coated with well-characterized black paint having an emissivity of 0.94 ± The characterization of the black paint was performed by the paint manufacturer. ROI (within dashed red lines) Figure 7. Spatial resolution thermal target. The foreground (black markings) is held at a constant temperature of 3 C above ambient. The target size is 61 cm measured along the centerline of each of the two converging line sets. An example of the ROI for each of the line groups is shown on the right. 10

18 The TIC under test was placed 1 m from the target, normal to the plane of the target, and oriented to focus on the center of the target. A Nikon D3 digital visible camera was used to take ten 16-bit digital photos of the images displayed by the TIC under test. These images were initially stored in flash memory on the Nikon, then transferred to a PC for further processing. The processing steps were as follows: 1. Convert the images from the proprietary Nikon NEF format to 16-bit TIF files. 2. Convert the TIF files to grayscale as defined per IEC (1999) [6]. 3. Filter out the high-frequency noise created by oversampling the TIC s display. This was accomplished with a filter which averaged over an area whose width and height was determined by the size of the TIC s display pixels. 4. Use a MATLAB program to rotate the image 45 degrees and select an ROI that encompasses the converging lines, as shown in the right image in Figure 7, and calculate the contrast transfer function (CTF) at least at each of the index numbers along the two sets of converging lines; a process that calculates the CTF at each row in the ROI is also acceptable. The CTF is calculated using the following equation: I max I CTF = min (2) where I max and I min are the highest and lowest pixel intensity values, respectively, along a row of pixels cut through the pattern at least at each index line as indicated by the dotted lines in the right image in Figure 7. Retain the original pixel intensity values for later use. Pixels that represent symbols, icons, or text are excluded from the analysis. 5. The CTF values are multiplied by pi/4 and paired with the frequencies of each index number or row of frequencies to construct a Modulation Transfer Function (MTF) curve. 6. The MTF curve is normalized to the highest CTF value obtained at the low-frequency end of the target between index 1 and index The average NU30 value is subtracted from the MTF curve. This adjustment corrects for the effects of high frequency noise. 8. The area under the normalized and adjusted MTF curve is the spatial resolution. 9. Apply the model to each result, producing the P value for the spatial resolution test. This is the pass/fail value for the spatial resolution tests. 11

19 TIC A TIC B TIC C TIC E TIC F Figure 8. Spatial resolution results, in terms of P, for all specimens except TIC D, which was excluded due to conflicts between its field of view and the image processing algorithms of Lab 2. The spatial resolution test results, in terms of the perception model P, are shown in Figure 8 above. Both laboratories conducted all of the expected spatial resolution tests; there were no exceptions or anomalies. Repeatability of the tests, as indicated by the length of the error bars, is significantly better (shorter error bars) for Laboratory 1 than for Laboratory 2 for all except the TIC B tests. With respect to reproducibility, the results for all TICs except TIC B and TIC E are outside the uncertainty of the measurements, which is due to a very large Laboratory 2 measurement uncertainty for TIC E, although the results for TIC B are quite acceptable Suggestions for improvement Given the state of repeatability and reproducibility of the spatial resolution test results, improvements to these tests is recommended. These improvements are listed below (not prioritized). Require TIC manufacturers to provide a mounting device and lens for the Nikon camera to ensure that it always views the TIC display from the same position. Specify the selection of the ROI in more detail. 12

20 Change the orientation of the target so that the converging sets of lines are vertical and horizontal instead of at 45 degrees. This may improve the filtering of high frequency noise. 2.4 Effective Temperature Range The Effective Temperature Range (ETR) test measures the ability of a firefighter TIC to see relatively small temperature differences in cases when large temperature differences exist in the field of view [1]. In this test, the TIC is positioned such that it views a set of contrast bars having constant temperature while the temperature of a surface of equal size in the field of view is increased from near ambient to 550 C. In general, as the hot surface temperature increases, the contrast of the bars decreases. This test, as described in NFPA 1801, also has a color component in which the colorization corresponding to certain surface temperature ranges is verified. The colorization employed by most of the TICs tested for this work was not designed to conform with NFPA 1801, therefore the color component of this test was not included in this study. The TICs were placed 1 m from the bar target, which were comprised of four vertical bars having a surface temperature of 30 o C ± 1 C. The background temperature was 22 o C ± 3 C. The bars and background were coated with black paint having an emissivity of 0.94 ± A blackbody was positioned such that its radiation impinged on a mirror and was directed to the TIC under test. The temperature of this blackbody s surface was initially set at 50 o C and was increased to 550 o C during the test. It is important that the emitting surface of the blackbody appear in the center of the image displayed by the TIC, while the heated bars appear at one side. The size of the blackbody radiation in the TIC s field of view was at least as large as the bars, but varied depending on the field of view of the TIC under test. A Nikon D3 digital visible camera was used to take 16-bit digital photos of the image displayed by the TIC at every 2 o C increment of the blackbody surface temperature. These images were initially stored in flash memory on the Nikon, then transferred to a PC for further processing. The processing steps were as follows: 1. Convert the images from the proprietary Nikon NEF format to 16-bit TIF files. 2. Convert the TIF files to grayscale as defined per International Electrotechnical Commission (IEC) (1999) [6]. 3. Filter out the high-frequency noise created by oversampling the TIC s display. This was accomplished with a filter which averaged over an area whose width and height was determined by the size of the TIC s display pixels. 4. Use a MATLAB program to define an ROI that encompasses the four vertical bars. Exclude or remove symbols, icons, and text from this ROI. Retain the original pixel intensity values for later use. 5. Use the MATLAB program to calculate the CTF of the bars within the ROI. The CTF is calculated using equation 2. 13

21 6. The perception equation is applied using the average brightness and CTF of the bar ROI, and the spatial resolution and NU30 results obtained in prior tests. The ETR results for test specimens except TIC D are presented in the following Figures 9 through 13. Discussion of all the ETR results is given following the figures. Figure 9. ETR results for TIC A, in terms of P, are shown as a function of the blackbody target temperature. 14

22 Figure 10. ETR results for TIC B, in terms of P, are shown as a function of the blackbody target temperature. 15

23 Figure 11. ETR results for TIC C, in terms of P, are shown as a function of the blackbody target temperature. 16

24 Figure 12. ETR results for TIC E, in terms of P, are shown as a function of the blackbody target temperature. 17

25 Figure 13. ETR results for TIC F, in terms of P, are shown as a function of the blackbody target temperature. Test results from Lab 2 were not available. The effective temperature range test results, in terms of the perception model P, are shown in Figures 9 through 13 above. Laboratory 1 conducted all of the expected effective temperature range tests, while Laboratory 2 had incomplete data for one of the TIC C tests and did not collect data for TIC F. Repeatability of the tests, as indicated by the deviation of the red lines for Lab 1 and the blue lines for Lab 2, shows that the only satisfactory set of tests was the Lab 1 testing of TIC B. The TIC E data is also somewhat consistent for both laboratories. In all other cases the data diverged severely. With respect to reproducibility, only the results for TIC E show a similarity between testing laboratories in data trends as the blackbody temperature increased. The absolute values of the model variable P were not consistent between laboratories for any of the test specimens Suggestions for improvement 18

26 Given the state of repeatability and reproducibility of the effective temperature range test results, improvements to these tests is recommended. These improvements are listed below (not prioritized). Require TIC manufacturers to provide a mounting device and lens for the Nikon camera to ensure that it always views the TIC display from the same position. Require a 100 mm X 100 mm blackbody viewed directly by the TIC under test (no mirror). Specify the alignment of the target components with respect to the TIC under test in more detail. 2.5 Thermal Sensitivity The thermal sensitivity (TS) test measures the response of TICs to changes in temperature [1]. A pair of well-characterized extended area blackbodies were used as targets. The TIC under test was positioned such that the image of two blackbody surfaces equally filled as much of the field of view as possible. The amount of the field of view filled by the blackbody surfaces depended on the field of view of the TIC under test. The TIC was positioned normal to the plane of the blackbody surfaces. The purpose of having two blackbody surfaces in the field of view was to force the automatic gain control (AGC) function of the imagers to stabilize during the test. A significant portion of a typical image used to calculate TS consists of surfaces at ambient temperature, which also contributed to the stabilization of the AGC. The perception model variable P is not used for this test. For this test, one blackbody was held at a constant temperature of 30 o C. The other blackbody was initially set at 17 o C and increased to 27 o C. Images were collected at increments of approximately o C. The images were initially stored in flash memory on the Nikon, then transferred to a PC for further processing. The processing steps were as follows: 1. Convert the images from the proprietary Nikon NEF format to 16-bit TIF files. 2. Convert the TIF files to grayscale as defined per IEC (1999) [6]. 3. Use a MATLAB program to define two ROIs of similar area, one of which encompasses the changing blackbody surface and the other encompasses the ambient surface. Exclude or remove symbols, icons, and text from these ROIs. 4. Filter out the high-frequency noise created by oversampling the TIC s display. This was accomplished with a filter which averaged over an area whose width and height was determined by the size of the TIC s display pixels. 5. Calculate the mean pixel intensity (µ) of each ROI. Retain the original pixel intensity values for later use. 6. Calculate the contrast between the changing blackbody surface and the ambient surface. Calculate a linear fit to the contrast as a function of the temperature difference between the changing blackbody surface and the ambient surface, this is the response slope. 7. Calculate the correlation coefficient of the data to the linear fit. 19

27 TIC A TIC B TIC C TIC D TIC E TIC F Figure 14. Thermal sensitivity test results for all test specimens, in terms of response slope and correlation coefficient. The thermal sensitivity test results, in terms of the response slope and correlation coefficient, are shown in Figure 14 above. Both laboratories conducted all of the thermal sensitivity tests for each test specimen. Repeatability of the tests, as indicated by the length of the error bars, is very similar for both laboratories, except for the TIC D correlation coefficients measured by Laboratory 1. With respect to reproducibility, roughly half of the correlation coefficient data and half of the response slope data falls within the uncertainty of the measurements Suggestions for improvement The repeatability and reproducibility of the thermal sensitivity results is much better than the other tests in this report, however, there is still room for improvement. Some recommendations are listed below. Require TIC manufacturers to provide a mounting device and lens for the Nikon camera to ensure that it always views the TIC display from the same position. 20

28 Specify the alignment of the target components with respect to the TIC under test in more detail. 3.0 UNCERTAINTY ANALYSIS There are different components of uncertainty in the measurements made with the equipment used in the tests discussed in this report. Type A uncertainties are those which are evaluated using statistical methods and Type B uncertainties are those which are estimated using other means, such as experience with a particular type of equipment. Type B uncertainties are evaluated by estimating the upper and lower limits for the quantity in question such that the probability that the quantity would fall within the upper and lower limits is essentially 100 %. After estimating the uncertainties by Type A and Type B analysis, the results are combined in quadrature to yield the combined standard uncertainty. When the combined standard uncertainty is multiplied by a coverage factor of two, the result is the expanded uncertainty which correspond to a 95 % confidence interval (2σ). The components of uncertainty are listed in Table 1. Some of these components, such as the blackbody temperature measurements, are derived from instrument specifications. The error bars and measurement accuracies shown in the figures in this report reflect values calculated using the uncertainties listed in Table 1. Table 1. Components used in uncertainty analysis. Component SR-800 Blackbody SR-80-7HT Blackbody IRCon Blackbody Type J thermocouples Type K Thermocouples Type A Uncertainty σ, pixel intensities in ROI σ, pixel intensities in ROI σ, pixel intensities in ROI σ σ Type B Uncertainty T < 50 o C: ± 8 mk T > 50 o C: ± 15 mk Combined Uncertainty σ + 8 mk σ + 15 mk Total Expanded Uncertainty 2(σ + 8 mk) 2(σ + 15 mk) ± 0.5 o C σ o C 2(σ o C) T < 316 o C: 2 o C T > 316 o C: 0.5 % Greater of 1.1 o C or 0.4 % Greater of 1.1 o C or 0.4 % σ + 2 o C σ % σ o C or σ % σ o C or σ % 2(σ + 2 o C) 2(σ %) 2(σ o C or σ %) 2(σ o C or σ %) 4.0 CONCLUSIONS 21

29 Repeatability is a term used in this report to indicate the ability of a test operator to follow a test procedure and get consistent, accurate results. With the exception of the effective temperature range test, the length of error bars calculated from the three data points collected for each component of each test is used as the measure of repeatability. The self-consistency of the sets of red and blue lines in Figures 9 through 13 is used as the measure of repeatability for the effective temperature range tests. In most cases, but not all, the Laboratory 1 data was more repeatable than the Laboratory 2 data. However, the robustness of the tests is dependent on the ability of all laboratories to produce repeatable test results, therefore improvement is needed. Reproducibility is a term used in this report to indicate the precision of the test results. For all of the tests examined in this report, the consistency of the test results between the participating laboratories is used as the measure of reproducibility. Some of the NU30 test results and most of the thermal sensitivity test results showed good reproducibility. Unfortunately, the rest of the test results leave much room for improvement. Suggestions for improvements to the test procedures that apply to multiple tests are: Specify the alignment of the target components with respect to the TIC under test in more detail. Require TIC manufacturers to provide a mounting device and lens for the Nikon camera to ensure that it always views the TIC display from the same position. Specify the selection of the ROI, including the method of removing symbols, icons, and text, in more detail. References [1] Amon, F., et al., Performance Metrics for Fire Fighting Thermal Imaging Cameras Small- and Full-Scale Experiments [2] ASTM E691-09e1 Standard Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method [3] Holst, G.C., Testing and Evaluation of Infrared Imaging Systems. Winter Park, FL: JCD Publishing [4] Lock, A. and F. Amon. Measurement of the Nonuniformity of First Responder Thermal Imaging Cameras. Proceedings of SPIE Defense + Security. Orlando, FL: SPIE [5] Donnelly, M.K., et al., Thermal Environment for Electronic Equipment used by First Responders. NIST Technical Note [6] IEC, Multimedia systems and equipment - Colour measurement and management. International Electrotechnical Commission. IEC

30 [7] Lock, A. and F. Amon. Application of Spatial Frequency Response as a Criterion for Evaluating Thermal Imaging Camera Performance. Proceedings of SPIE Defense and Security Conference. Orlando, Florida: SPIE [8] ISO, Photography - Electronic still-picture cameras - Resolution measurements. International Standards Organization. ISO