Statistics for Quality Assurance of Ambient Air Monitoring Data

Size: px

Start display at page:

Download "Statistics for Quality Assurance of Ambient Air Monitoring Data"

Tyrone Pearson
6 years ago
Views:

1 Statistics for Quality Assurance of Ambient Air Monitoring Data Lesson 8: Precision and Bias QA Handbook Vol. II, Sections 10 & 17 and Appendices D & I 1

2 What We Will Cover In Lesson #8 Precision and Bias Non Collocated Samples Precision % Difference Accuracy Average Bias Collocated Samples Precision Relative % Difference Bias Maximum Average Bias Precision many values, how reproducible the experiment is - Quality Accuracy single value, how close experimental agrees with true/accepted value 2

3 Guidance Document Guideline on the Meaning and The Use of Precision and Bias Data Required by 40CFR58, Appendix A ( 3

4 Development of Statistics MQOs be based on confidence intervals whether the bias and precision variables met the measurement quality objectives (MQOs) Different to say the bias is 5% ± 10% compared to saying the bias is 5% ± 1% Reduction of QC checks frequency for organizations with tight acceptable results, therefore providing more freedom in their QC program 4

5 Some Typical Measurement Variation Assessments * *100% completeness for data is the goal 5

6 What Is Performed Bias = 15% APTI Course #470 6

B DASC, in the Precision and Accuracy Reporting System Within Quality

7 Data Assessment Statistical Calculator (DASC) DASC has been produced specifically for the data user community Found under the filename P & B DASC, in the Precision and Accuracy Reporting System Within Quality Assurance of AMTIC Web Site: ( 7

8 DASC Data Quality Indicators Calculated for Measured Pollutant 8

9 DASC Main Menu Worksheet 9

10 Precision truth truth Precise Less Precise One can say that a measurement is accurate but not precise; precise but not accurate; neither or both. An example of bad precision and good accuracy can be: Suppose a lab refrigerator holds a constant temperature of 38.0 F. A temperature sensor is tested 10 times in the refrigerator. The temperatures from the test yield the temperatures of: 37.8, 38.3, 38.1, 38.0, 37.6, 38.2, 38.0, 38.0, 37.4, This distribution shows no impressive tendency toward a particular value (lack of precision) but each value does come close to the actual temperature (high accuracy). 10

11 A Graphical Look At Precision Green = Least Precise Red = More Precise Black = Most Precise probability % difference

12 Bias truth truth Unbiased Biased Bias is a term which refers to how far the average statistic lies from the parameter it is estimating, that is, the error which arises when estimating a quantity. Errors from chance will cancel each other out in the long run, those from bias will not. 12

13 A Graphical Look At Bias Red Is Unbiased Black Is Negatively Biased Green Is Positively Biased probability relative % difference

14 Calculations When a known standard is available (% difference) Precision Estimate d i ± observed known = 100 known ( ) ( t ) s n 1,. 975 d Bias: Maximum Average Bias Direction of Bias is the sign of d ( ) d t +.95,n 1 s n 14

15 SO 2 Measurement Example Observed Known conc conc d i % % % n=24 SO 2 Measurements Known Standards Used % % The table is abbreviated, showing first and last row entries. 15

16 Dotplot of %Difference (d i ) The measures of spread of the data are indicative of the precision of the measurement system. The measures of center of the data are most likely indicative of the accuracy (mean = 0 is good). 16

17 Precision Calculation Precision Estimate s = 4.18 t 24-1,.975 = 2.07 Precision estimate is 2.07*4.18 = ± 8.5% ± t s 24 1,. d ( 975 )( ) APTI Course #470 17

18 Bias Calculation Bias Estimate: d + t n 1,. 95 * sd n d Where: = 3.1 s d t 23,.95 n = 4.2 = 1.7 = 4.8 Bias estimate is: (1.7*4.2)/4.8 = 4.6% 18

19 Testing Bias Is this bias of 4.6% significant? 1-sample t test (assumes no bias) p-value.703 No statistically significant bias 19

20 Collocated Samplers For Collocated Samples (Relative % Difference) Precision Estimate Bias Estimate Maximum Average Bias d i xi y = i 100 ( xi + yi) 2 ( t ) n 1,. 975 ± d + t. 95 * s 2 sd n 20

21 PM2.5 Measurement Study S1 S2 RPD PM 2.5 concentration measured daily at each of two collocated sites n=30 measurements at each site The table is abbreviated, showing first and last row entries. 21

22 Dotplot of Relative % Difference 22

23 Precision Calculation Precision Estimate: t 29,.975 = 2.05 s d = 11.7 Precision estimate is ± 2.05*8.3 = ± 17% 23

24 Bias Calculation Bias Estimate d + t. 95 * sd n d Where: = 3.1 s d t 29,.95 n = 11.7 = 1.7 = 5.39 Bias Estimate Is (1.7*11.7)/(5.4) = 6.8% 24

25 Testing Bias Is this bias of 6.8% significant? 1-sample t test (assumes no bias) p-value.162 No statistically significant bias 25

26 Lesson 8: Precision and Bias Summary Non Collocated Samples Precision % Difference Accuracy Maximum Average Bias Collocated Samples Precision Relative % Difference Bias Maximum Average Bias Can statistically test for significant bias EPA DASC Tool 26

Stat472/572 Sampling: Theory and Practice Instructor: Yan Lu Albuquerque, UNM

Stat472/572 Sampling: Theory and Practice Instructor: Yan Lu Albuquerque, UNM 1 Chapter 1: Introduction Three Elements of Statistical Study: Collecting Data: observational data, experimental data, survey