How can it be right when it feels so wrong? Outliers, diagnostics, non-constant variance

Size: px

Start display at page:

Download "How can it be right when it feels so wrong? Outliers, diagnostics, non-constant variance"

Elvin Sims
5 years ago
Views:

1 How can it be right when it feels so wrong? Outliers, diagnostics, non-constant variance D. Alex Hughes November 19, 2014 D. Alex Hughes Problems? November 19, / 61

2 1 Outliers Generally Residual Plots Assessing Leverage Hat-Values Studentized Residuals Measuring Influence DFBETEA(S) 2 Non-normality & Nonconstant Error Variance Non-normal Errors D. Alex Hughes Problems? November 19, / 61

3 Thanks to Christina and Simeon Agreement with our Best Practices What did they do that we know we should do? What did they do that we haven t talked about? What if anything was unclear in their presentations? D. Alex Hughes Problems? November 19, / 61

4 1 Outliers Generally Residual Plots Assessing Leverage Hat-Values Studentized Residuals Measuring Influence DFBETEA(S) 2 Non-normality & Nonconstant Error Variance Non-normal Errors D. Alex Hughes Problems? November 19, / 61

5 Outliers, generally Definition An outlier is an observation whose response-variable value is conditionally unusual given the value of the explanatory variable. Recall: What are we minimizing? What will be the effect of an outlier on our estimated regression coefficients? Influence on Coef = Leverage Discrepancy D. Alex Hughes Problems? November 19, / 61

6 Outliers, generally Problems May unduly influence estimation results; or, May identify model is missing important features of data D. Alex Hughes Problems? November 19, / 61

7 An Example y[ c(5, 6, 7)] x[ c(5, 6, 7)] D. Alex Hughes Problems? November 19, / 61

8 An Example y vals x vals D. Alex Hughes Problems? November 19, / 61

9 An Example y vals x vals D. Alex Hughes Problems? November 19, / 61

10 An Example y vals x vals D. Alex Hughes Problems? November 19, / 61

11 An Example Definition An outlier is an observation whose response-variable value is conditionally unusual given the value of the explanatory variable. So, in these cases, how can we talk about the leverage of the outlying data point? D. Alex Hughes Problems? November 19, / 61

12 An Example y[ c(5, 6, 7)] y vals x[ c(5, 6, 7)] x vals y vals y vals D. Alex Hughes Problems? November 19, / 61

13 Another Example Reported and Measured Weight weight reportedweight D. Alex Hughes Problems? November 19, / 61

14 Another Example Coefficients: (1 not defined because of singularities) Estimate Std. Error t value Pr(> t ) (Intercept) weight e-11 *** I(sex == "F")TRUE < 2e-16 *** weight:sexm < 2e-16 *** D. Alex Hughes Problems? November 19, / 61

15 Another Example Coefficients: (1 not defined because of singularities) Estimate Std. Error t value Pr(> t ) (Intercept) weight <2e-16 *** I(sex == "F")TRUE weight:sexm D. Alex Hughes Problems? November 19, / 61

16 1 Outliers Generally Residual Plots Assessing Leverage Hat-Values Studentized Residuals Measuring Influence DFBETEA(S) 2 Non-normality & Nonconstant Error Variance Non-normal Errors D. Alex Hughes Problems? November 19, / 61

17 Regression Diagnostics Goal: Assess whether the regression results are meaningful, stable, and comply with the assumptions underlying the regression model. 1 Are regression assumptions met? 2 Are there any influential or outlying values messing things up? D. Alex Hughes Problems? November 19, / 61

18 Recall Regression Assumptions Assumptions: 1 There is a linear relationship between X and Y 2 ɛ N(0, σ 2 ) 3 X are fixed (OR!) if X are random, X is orthogonal to ɛ Keep these in mind as we move through tools for regression diagnostics. The first two are sometimes easy to see visually. The third is more complicated. D. Alex Hughes Problems? November 19, / 61

19 Overview Residual Plots Regular residuals Studentized residuals Externally studentized residuals Leverages DFBetas Cook s Distance DFITS Partial Regression Plots D. Alex Hughes Problems? November 19, / 61

20 Residual Plots Recall: Residuals are variance in Y unexplained by the X s e i = Y i Ŷ i D. Alex Hughes Problems? November 19, / 61

21 Residual Plots Calculate residuals: e i = Y i Ŷ i = Y i Xˆβ = Y i â + ˆbx i There are two (at least) plots that we ll use to diagnose outliers using residuals 1 AFitted vs. Residual Plot plots ŷ on the x-axis and e on the y-axis. Are there places in the prediction of y that we re more off? 2 A Fitted vs. X Plot plots X (k) on the x-axis and e on the y-axis. Why would we choose one or the other? D. Alex Hughes Problems? November 19, / 61

22 Residual Plots: What to Watch For 1 Ideal: cloud, normal distribution 2 The Wedge 3 Nonlinearity 4 Outliers D. Alex Hughes Problems? November 19, / 61

23 Residual Plots: Ideal Scatterplot of XvY x y D. Alex Hughes Problems? November 19, / 61

24 Residual Plots: Ideal Residuals of Regression x y D. Alex Hughes Problems? November 19, / 61

25 Residual Plots: Ideal Residuals v X x resid(m3) D. Alex Hughes Problems? November 19, / 61

26 Residual Plots: Ideal hat(y) v resid predict(m3) resid(m3) D. Alex Hughes Problems? November 19, / 61

27 Residual Plots: Wedge Scatterplot of XvY x y D. Alex Hughes Problems? November 19, / 61

28 Residual Plots: Wedge X v Resid x resid(m4) D. Alex Hughes Problems? November 19, / 61

29 Residual Plots: Curve curvy? y x D. Alex Hughes Problems? November 19, / 61

30 Residual Plots: Curve curvy? x resid(m5) D. Alex Hughes Problems? November 19, / 61

31 Residual Plots: Outliers Reported and Measured Weight weight reportedweight D. Alex Hughes Problems? November 19, / 61

32 1 Outliers Generally Residual Plots Assessing Leverage Hat-Values Studentized Residuals Measuring Influence DFBETEA(S) 2 Non-normality & Nonconstant Error Variance Non-normal Errors D. Alex Hughes Problems? November 19, / 61

33 We could think of our fitted values as coming from the statement Ŷ j = h 1j Y 1 + h 2j Y h nj Y n = n h ij Y i i=1 Definition The hat-value h ij captures the contribution of Y i to the fitted value Ŷ j. If h ij is large, then the i th value can have a large impact on the j th fitted value. Theorem The hat-value h i h ii summarizes the impact of some Y i on all the Ŷ. And so, n h i h ii = j=1 h 2 ij D. Alex Hughes Problems? November 19, / 61

34 Hat properties Additional Properties The hat values are bounded on the range ( ) 1 n, 1 The average hat-value is h = (k + 1)/n where k is the number of regressors in the model. In multiple regression, h i measured distance from the centroid of the Xs. Then, multivariate outliers in the X-space are high-leverage observations. D. Alex Hughes Problems? November 19, / 61

35 Leverages - Matrix Version Matrix Version (the diagonals of the hat matrix ) Ŷ = X ˆβ = X (X X ) 1 X Y = Hy H X (X X ) 1 X Definition The hat-matrix, H = X (X X ) 1 X, when postmultiplied by y turns y into ŷ. Nothing about Y here just about unusual combinations of independent variable values Could be a large X, could be an odd combination of X s D. Alex Hughes Problems? November 19, / 61

36 library(car) data(duncan) attach(duncan) lm.out <- lm(prestige ~ income + education) plot(hatvalues(lm.out)) abline(h= c(2,3)*mean(hatvalues(lm.out))) identify(1:45, hatvalues(lm.out), row.names(duncan)) D. Alex Hughes Problems? November 19, / 61

37 Studentized Residuals The errors fed into regression ɛ i may have constant variance, but the residuals do not. In particular,. V (E i ) = σ 2 ɛ (1 h i ) Data-points with high leverage (hat-values) tend to have smaller residuals. Intuitive? Pull the regression line toward them. We could do a pretty simple standardization to figure out if a residual is strange by scaling it over the V Ei. E i E i S E 1 hi But, this is a drag because the numerator and denominator are not independent; so doesn t follow a t-distribution... D. Alex Hughes Problems? November 19, / 61

38 Studentized Residuals Solution: Calculate the i th residual, standardizing from a regression that excludes it. Ei E i = S E( i) 1 hi which follows a t-distribution with df = n k 2. S E( i) : our estimate of the standard deviation of the unobserved errors 1 h i : A measure of each observation s influence Follows a t-distribution with n k 2 degrees of freedom D. Alex Hughes Problems? November 19, / 61

39 Studentized Residuals Example x resid(m6) rstudent(m6) D. Alex Hughes Problems? November 19, / 61

40 Warning Language is Inconsistent Externally Studentized Residuals Internally studentized residuals, Deleted studentized residuals studentized residuals... D. Alex Hughes Problems? November 19, / 61

41 Leverage Remember how we defined influence on the regression coefficients? Definition Influence on Coef = Leverage Discrepancy We ve built mechanisms to identify discrepancy (the residual plots, hat-values, and studentized residuals from the last section). Here, we assess leverage D. Alex Hughes Problems? November 19, / 61

42 DFBETA(S) What is the impact on each coefficient of a regression (D i ) as a result of deleting some observation j? D ij = B j B ( i) j, i, j Where B j is the least squares coefficient, and B ( i) j is that same coefficient without observation j. Like most things...it s nice to scale this by the SE. And so... D ij = D ij SE ( i) (B j ) D. Alex Hughes Problems? November 19, / 61

43 DFBETA(S) D. Alex Hughes Problems? November 19, / 61

44 Cook s Distance D i = E i 2 k + 1 h i 1 h i Provides a summary index of influence on the coefficients Distance between the vector of β s including and excluding observation i. Combines all the coefficients into a single measure. Kind of like F-test - based on SSR -sum of squared (standardized) residuals D i = E i h i 1 h i D. Alex Hughes Problems? November 19, / 61

45 Cook s Distance D. Alex Hughes Problems? November 19, / 61

46 What s the big deal? In previous examples, outliers were very obvious All these, however, were bivariate regression With multivariate regression, strange combinations of x values can make hidden values influential. A regular residual plot won t pick these up! D. Alex Hughes Problems? November 19, / 61

47 Not always so obvious D. Alex Hughes Problems? November 19, / 61

48 Not always so obvious D. Alex Hughes Problems? November 19, / 61

49 Not always so obvious D. Alex Hughes Problems? November 19, / 61

50 Go to the code! require(mass) model1 <- lm(y~x) sresm1 <- studres(model1) cooksdm1 <- cooks.distance(model1) leveragesm1 <- hatvalues(model1) dfbetasm1 <- dfbetas(model1) rstudent(model1) plot(model1) D. Alex Hughes Problems? November 19, / 61

51 Cuttoffs? Generally, they re a poor idea, and should be handled on a case-by-case basis. Buuut... Look for h i > 2 h. Look for E i > 2 Look for D ij > 1or2 unless in large sample In a large sample, scale that by the root of number of observations: D ij > 2/ n. D. Alex Hughes Problems? November 19, / 61

52 Practical Diagnostics Computation with large n? Effectiveness and sample size Publishing More than one outlier? D. Alex Hughes Problems? November 19, / 61

53 What to do with outliers? Outliers might be random Learn more Make a subjective decision Full disclosure D. Alex Hughes Problems? November 19, / 61

54 1 Outliers Generally Residual Plots Assessing Leverage Hat-Values Studentized Residuals Measuring Influence DFBETEA(S) 2 Non-normality & Nonconstant Error Variance Non-normal Errors D. Alex Hughes Problems? November 19, / 61

55 Non-normal Errors The assumption of normal errors is nearly always arbitrary BUT! The CLT says that under hella broad conditions inference is good (unless we re in small samples). So, why do we care about non-normal variance? Let me count the ways? 1 The levels and of tests will be approximately correct in large samples; but, without normal errors they will not be maximally efficient. Indeed, for distributions with heavy tails, OLS will be pretty inefficient. 2 OLS is a conditional statement of means. This doesn t make sense in heavily skewed distributions. 3 A multimodal error distribution suggests that a discrete explanatory variable may be missing. D. Alex Hughes Problems? November 19, / 61

56 Graphical Diagnosis To diagnose non-normality, we typically use a quantile-comparison plot, or sometimes a Q-Q plot. t-distribution of the theoretical quantiles on the x-axis (or z-distribution). Studentized residuals on the y-axis Especially good at identifying problems in the tail-behavior of the distributions. Visible as deviations from the 45-degree line. Supplements Histogram of studentized residuals; or, Kernel density plot of studentized residuals. D. Alex Hughes Problems? November 19, / 61

57 Interpreting a Q-Q plot All but a few points fall on a line Left end below line Right end above line Left end above Right end below Curved pattern: slope increasing Curved pattern: slope decreasing Steps Outliers in the data Long tail at low side Long tail at high side Short tail at low side Short tail at high side Skew right Skew left Discrete data D. Alex Hughes Problems? November 19, / 61

58 1 Outliers Generally Residual Plots Assessing Leverage Hat-Values Studentized Residuals Measuring Influence DFBETEA(S) 2 Non-normality & Nonconstant Error Variance Non-normal Errors D. Alex Hughes Problems? November 19, / 61

59 Assumptions about Errors ɛ i N(0, σ 2 ) i How do things go wrong? 1 The mean isn t zero 2 The errors are not normally distributed 3 The variance of errors is different across observations 4 The covariance of errors across observations is not zero D. Alex Hughes Problems? November 19, / 61

60 What if the errors are not normally distributed? Normality: we either assume it or we rely on the central limit theorem and big datasets. Even without normality, regression is the best linear unbiased estimator (BLUE) of the model parameters β. This comes from the Gaus-Markov Theorem. However, the normality assumption provides t statistics for hypothesis testing - so inference is suspect without normality. If you don t feel comfortable being normal...you could specify an alterative distribution and come up with your own standard errors for hypothesis testing or you could use a nonparametric method to run the regresion, depending on your beliefs about the errors. Check for normality: formal test, or quantile plots of residuals. What s a quantile plot? D. Alex Hughes Problems? November 19, / 61

61 What if the mean of the error distribution isn t zero? This is the easy case. Why? Y = Xβ + ɛ ɛ N(θ, σ 2 ) D. Alex Hughes Problems? November 19, / 61

STAB22 section 2.4. Figure 2: Data set 2. Figure 1: Data set 1

STAB22 section 2.4. Figure 2: Data set 2. Figure 1: Data set 1 STAB22 section 2.4 2.73 The four correlations are all 0.816, and all four regressions are ŷ = 3 + 0.5x. (b) can be answered by drawing fitted line plots in the four cases. See Figures 1, 2, 3 and 4. Figure