BIOS 312: MODERN REGRESSION ANALYSIS

Transcription

1 BIOS 312: MODERN REGRESSION ANALYSIS James C (Chris) Slaughter Department of Biostatistics Vanderbilt University School of Medicine james.c.slaughter@vanderbilt.edu biostat.mc.vanderbilt.edu/coursebios312 Copyright JC Slaughter All Rights Reserved Updated April 5, 2010

2 Contents 12 Bayesian Analysis of Regression Models Overview The posterior distribution Example 1: Estimating the mean Bayesian Approach (WinBUGS) Frequentist Approach (Stata) Bayesian Linear Model Non-informative priors Sampling from the joint posterior distribution Example 2: Linear Regression R Code WinBUGS Code Comparison of Bayesian and Frequentist Results Example 3: Hierarchical Model, Linear Regression Rats data, assuming independence

3 CONTENTS Rats data, modeling correlation

4 Chapter 12 Bayesian Analysis of Regression Models 12.1 Overview Scientific hypotheses are typically refined in statistical hypotheses by identifying some parameter, θ, measuring differences in the distribution of the response variable Often we are interested in if θ differs across of levels of categorical (e.g. treatment/control) or continuous (e.g. age) predictor variables θ could be any summary measure such as Difference/ratio of means Difference/ratio of medians Ratio of geometric means Difference/ratio of proportions 4

5 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 5 Odds ratio, relative risk, risk difference Hazard ratio Statistics is concerned with making inference about population parameters, (θ), based on a sample of data Frequentist estimation includes both point estimates (ˆθ) and interval estimates (confidence intervals) Bayesian analysis estimates the posterior distribution of θ given the sampled data, p(θ data). The posterior distribution can then be summarized by quantities like the posterior mean and 95% credible interval. Bayesian process of data analysis can be divided into three steps Defining the full probability model A joint probability distribution for all the observable and unobservable quantities in a problem The model should be chosen based on an understanding of the underlying scientific problem Same issues here as in frequentist analysis: Linearity, homoscedasticity, inclusion of additional covariates, etc. With the Bayesian approach, more flexibility in how you model departures from the underlying assumptions Often, it is also more obvious what assumptions you are and are not making Conditioning on observed data Calculating and interpreting the appropriate posterior distribution of θ given the sampled data, p(θ data)

6 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 6 The posterior will be a function of both the Likelihood (same as frequentist) and the Prior distribution (additional for Bayes) Evaluating the fit of the model Does the model fit data? Can evaluate using posterior prediction intervals. Are the substantive conclusion reasonable? How sensitive are the results to model assumptions? Assumptions about the joint probability distribution Assumptions about prior distributions The posterior distribution In order to make probability statements about θ given the data (y), must specify a joint probability distribution for θ and y p(θ, y) = p(θ)p(y θ) p(θ, y) is the joint distribution of θ and y p(θ) is the prior distribution of θ p(y θ) is the sampling distribution of the data given θ Bayes rule states that we can express the joint distribution of θ and y as the product of marginal and conditional distributions p(θ, y) = p(y θ) p(θ) = p(θ y) p(y)

7 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 7 Using Bayes rule, we can then express the posterior distribution of θ as p(θ, y) p(θ y) = p(y) p(y θ) p(θ) = p(y) p(y θ) p(θ) The preceding expression captures the core of Bayesian analysis The posterior distribution of the parameter(s) θ given the data is the product of The likelihood: p(y θ) The prior: p(θ) The goal of all Bayesian analyses is to develop an appropriate model to answer the scientific questions and summarize the posterior distribution, p(θ y) in appropriate ways 12.2 Example 1: Estimating the mean Purpose Learn to use WinBUGS Compare results from WinBUGS and STATA Background Consider a sample of continuous data, n = 20 These data come from a Normally distributed population with σ = 1 (standard deviation known)

8 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 8 Our goal is to estimate µ, the mean Bayesian Approach (WinBUGS) The distribution of the data (Normal, with variance known) has already been specified We need a prior distribution for µ The prior will be combined with the distribution of the data to produce a posterior distribution for µ The mean of this posterior distribution is our estimate for µ With a vague prior The standard deviation of the posterior will be similar to the frequentist estimate of the standard error of the mean The credible interval for the posterior will be similar to the frequentist confidence interval WinBUGS will incorporate the correct uncertainty in estimating the parameters Let s say that a priori believe that µ is Normally distributed with mean 0 However, we are uncertain about this prior belief so we attach a variance of 10,000 to it A prior variance of 10,000 may seem big (and will be for most problems), but you have to be careful about the units of the application Example: If we use a prior with mean 0 and variance 10,000 (standard deviation 100) and the true mean is 3000, it will be 30 prior standard

9 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 9 deviations from 0 and the prior will not really be uninformative. Therefore, when selecting a large variance, you need to pay attention to the location and scale of the data Notation for the model The data: x 1,..., x 20 p(x i µ) N(µ, 1) p(µ) N(0, 10000)

10 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 10 WinBUGS code The following program will implement our specifications and give us the posterior mean of µ model { for (i in 1:n) { x[i] ~ dnorm(mu,tau); } mu ~ dnorm(0,0.0001); tau <- 1; sigma <- 1/sqrt(tau); xbar <- mean(x[]); } # Data list(x=c( , , , , , , , , , , , , , , , , , , , ), n=20) #initial values list(mu=0) #tau is the precision of x; sigma is the standard deviation. #standard deviation is 1. In this case, the While this syntax looks pretty difficult, you will not be required to program WinBUGS from scratch In fact, WinBUGS provides several examples that can often be modified to fit the type of model that you desire Running the program in WinBUGS 1. Open WinBUGS 2. Close the license agreement; go to File and choose New 3. Copy and paste the program into WinBUGS

11 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS Highlight the word model in the program, click the Model menu, and choose Specification. Click check model on the dialog box. Look at the bottom lefthand corner for model is syntactically correct. 5. Highlight the word list in the program that precedes the data and click load data in the dialog box. Look in the bottom left-hand corner for data loaded. 6. Click compile in the dialog box. Look for model compiled in the bottom lefthand corner. 7. Highlight the word list in the program that precedes the initial values and click load inits in the dialog box. Look in the bottom left-hand corner for initial values loaded; model initialized. 8. Click Inference in the menu, and choose Samples. 9. Type mu in the dialog box next to node. Click set. 10. Click Model in the menu and choose update. Type a large number (ie, 11000) for number of updates. Click update in the dialog box. 11. Go back to the Sample Monitor Tool to check out the output. 12. Make sure mu is showing as the node. Click sample stats Burn in It takes anywhere from a few to many iterations for the distributions to stabilize This is referred to as the burn in period I chose to use updates starting from 1001 so that the first 1000 are burn in The final estimate is actually a weighted average of the sample mean and the prior mean, with the weights being the reciprocals of the variance of the data (assumed to be 1) and the prior variance of the mean

12 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 12 The reciprocal of the prior variance is very small (0.0001), which means that the data are weighted much more heavily than the prior I you found these steps confusing, you can watch WinBUGS-the movie! at:

13 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS Frequentist Approach (Stata) With only one variable, there are many simple ways to get STATA to calculate the mean and standard error Because this is a regression course, I will use simple linear regression. clear input x end regress x. regress x Source SS df MS Number of obs = F( 0, 19) = 0.00 Model 0 0. Prob > F =. Residual R-squared = Adj R-squared = Total Root MSE = x Coef. Std. Err. t P> t [95% Conf. Interval] _cons Moral: Although WinBUGS uses Bayesian methods and STATA uses standard (frequentist) methods, they can produce very similar results.

14 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 14 Additional Exercise: Change the prior mean and variance in WinBUGS to get a feel for how the prior changes the results 12.3 Bayesian Linear Model Non-informative priors Under standard non-informative prior distributions, Bayesian analysis coincides with classical, frequentist results Even in the non-informative case, posterior simulations are useful for predictive inference and model checking In ordinary linear regression model with homoscedasticity, the likelihood in vector notation can be specified as I is the identity matrix (rank n) p(y β, σ 2, X) N(Xβ, σ 2 I) X is the design matrix containing covariates (rank k) β is a k 1 vector of parameters σ 2 is the common variance term A convenient non-informative prior distribution is uniform on (β, log σ) p(β, σ 2 ) σ 2 With this choice of prior, we can calculate the posterior distribution of β given σ 2 and y p(β σ 2, y) N( ˆβ, V β σ 2 )

15 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 15 where ˆβ = (X X) 1 X y (12.1) V β = (X X) 1 (12.2) We can also calculate the posterior distribution of σ 2 given y where p(σ 2 y) Inv-χ 2 (n k, s 2 ) s 2 = 1 n k (y X ˆβ) (y X ˆβ) Note that in classical linear regression, the standard non-bayesian estimates of β and σ 2 are ˆβ and s 2 as just defined Than classical standard error estimate is found by setting σ 2 = s Sampling from the joint posterior distribution To estimate the joint posterior distribution, we need to draw samples from the joint posterior distribution p(β, σ 2 y) With the non-informative prior, we have expressions for p(β σ 2, y) and p(σ 2 y) Complete conditionals Can factor the joint posterior: p(β, σ 2 y) = p(β σ 2, y)p(σ 2 y) To draw samples from the joint posterior p(β, σ 2 y), we can use Gibbs sampling Compute ˆβ = (X X) 1 X y

16 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 16 Compute V β = (X X) 1 Compute s 2 = 1 n k (y X ˆβ) (y X ˆβ) Draw random vector ˆβ from a multivariate Normal: ˆβ N( ˆβ, V β σ 2 ) Draw random σ 2 from scaled inverse χ 2 : σ 2 Inv-χ 2 (n k, s 2 ) Repeat the previous steps as many times as necessary After an initial burn-in period, the Gibbs sampling procedure will be producing draws from the joint posterior distribution p(β, σ 2 y) that you need For most simple models, a few thousand iterations while discarding the first 10% as burn-in will be more than sufficient For complex models, many more iterations may be required due to slow convergence of the Gibbs sampler Efficient programming may be important for complex models and/or large datasets There are a variety of convergence diagnostics that can be checked (see the CODA package in R) Note that we could have specified a different prior distribution for β and σ 2 With a different prior choice, the general steps of the Gibbs sampler will be the same, but the computations and random draws will differ Some times priors are chosen for Mathematical convenience and ease of programming

17 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 17 Priors could be chosen to incorporate data from previous studies All models should be checked to verify that they are robust to prior assumptions Frequentist models also have prior assumptions 12.4 Example 2: Linear Regression FEV predicted by height and age Will be conducted in class R Code # Read in the dataset fevdata <- read.csv(" attach(fevdata) # Create the Design Matrix Xmat <- matrix(na, nrow=length(fev), ncol=3) Xmat[,1] <- 1 Xmat[,2] <- age Xmat[,3] <- height #Number of parameter, and number of observations k <- 3 n <- length(fev) # Matrix algebra to estimate betas and variance beta.hat <- solve(t(xmat) %*% Xmat) %*% t(xmat) %*% fev V.beta <- solve(t(xmat) %*% Xmat) s2 <- t(fev - Xmat %*% beta.hat) %*% (fev - Xmat %*% beta.hat) / (n - k) # Library that has function for drawing multivaiate Normals library(mass) # Set up matrices to store results reps < beta.gibbs <- matrix(na, nrow=reps, ncol=k) sigma2.gibbs <- rep(na, reps) s2.cur <- 1

18 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 18 # Run the Gibbs Sampler for (g in 1:reps) { beta.cur <- t(mvrnorm(1, beta.hat, as.numeric(s2.cur) * V.beta)) s2.cur <- 1/rchisq(1, n-k) *(n-k)*s2 } beta.gibbs[g,] <- beta.cur sigma2.gibbs[g] <- s2.cur # Compare to linear regression summary(beta.gibbs) summary(lm(fev ~ age + height)) t(apply(beta.gibbs, 2, quantile, c(.025,.975))) confint(lm(fev ~ age + height)) WinBUGS Code Can also be downloaded from class web site model { for (i in 1:n) { fev[i] ~ dnorm(mu[i], tau) mu[i] <- beta0 + beta1*age[i] + beta2*height[i] } beta0 ~ dnorm(0,.00001) beta1 ~ dnorm(0,.00001) beta2 ~ dnorm(0,.00001) # Prior 1: uniform on SD sigma ~ dunif(0, 100) tau <- 1/(sigma * sigma) # Prior 2: (traditional prior, but not recommended now) # tau ~ dgamma(.001,.001) # sigma <- 1/sqrt(tau) } # Initial Values list(beta0=0, beta1=0, beta2=0, sigma=1) # Data

19 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 19 list(n=654, age=c(9, 8, 7, 9, 9, 8, 6, 6, 8, 9, 6, 8, 8, 8, 8, 7, 5, 6, 9, 9, 5, 5, 4, 7, 9, 3, 9, 5, 8, 9, 5, 9, 8, 7, 5, 8, 9, 8, 8, 8, 9, 8, 5, 8, 5, 9, 7, 8, 6, 8, 5, 9, 9, 8, 6, 9, 9, 7, 4, 8, 8, 8, 6, 4, 8, 6, 9, 7, 5, 9, 8, 8, 9, 9, 9, 7, 5, 5, 9, 6, 7, 6, 8, 8, 7, 8, 7, 9, 5, 9, 9, 9, 7, 8, 8, 9, 9, 9, 7, 8, 8, 7, 9, 4, 9, 6, 8, 6, 7, 7, 8, 7, 7, 7, 7, 8, 7, 5, 8, 7, 9, 7, 7, 6, 8, 8, 8, 9, 7, 8, 9, 8, 8, 9, 8, 6, 6, 8, 9, 5, 7, 9, 6, 9, 9, 9, 6, 8, 9, 8, 8, 9, 9, 9, 7, 8, 6, 9, 9, 9, 7, 8, 5, 8, 9, 6, 9, 6, 8, 5, 7, 7, 4, 9, 8, 9, 9, 9, 5, 9, 7, 6, 9, 9, 9, 7, 5, 8, 9, 7, 9, 8, 9, 6, 6, 8, 9, 5, 6, 6, 9, 7, 9, 8, 5, 7, 6, 9, 7, 9, 9, 8, 9, 7, 9, 4, 9, 5, 8, 9, 8, 3, 9, 8, 6, 9, 8, 8, 7, 6, 8, 9, 4, 7, 8, 8, 9, 6, 8, 6, 8, 9, 8, 7, 9, 8, 7, 9, 8, 9, 6, 8, 9, 8, 9, 9, 8, 7, 5, 7, 8, 9, 9, 6, 8, 7, 9, 7, 7, 5, 9, 9, 8, 8, 9, 6, 7, 5, 9, 5, 7, 6, 8, 7, 8, 4, 8, 5, 8, 7, 7, 9, 9, 8, 9, 6, 8, 9, 4, 6, 7, 9, 8, 6, 8, 7, 5, 8, 7, 11, 10, 14, 11, 11, 12, 10, 11, 10, 14, 13, 14, 12, 12, 10, 13, 10, 11, 10, 11, 10, 13, 14, 11, 10, 11, 13, 10, 10, 12, 10, 10, 10, 11, 11, 11, 10, 11, 11, 13, 13, 11, 11, 14, 11, 10, 10, 10, 14, 13, 10, 14, 10, 11, 13, 12, 13, 10, 13, 11, 14, 11, 13, 11, 11, 10, 11, 11, 10, 11, 13, 12, 10, 10, 14, 11, 10, 11, 10, 11, 13, 13, 10, 11, 11, 12, 10, 10, 11, 10, 11, 14, 13, 12, 11, 11, 11, 14, 12, 10, 12, 11, 10, 11, 13, 10, 10, 11, 13, 10, 11, 10, 13, 11, 10, 11, 11, 14, 11, 13, 11, 11, 10, 13, 10, 13, 10, 12, 10, 14, 12, 10, 11, 14, 12, 10, 10, 10, 10, 12, 13, 11, 12, 11, 12, 11, 11, 12, 12, 13, 11, 12, 10, 12, 13, 10, 12, 10, 12, 10, 11, 10, 12, 14, 10, 10, 12, 10, 10, 13, 12, 12, 11, 13, 12, 10, 11, 11, 13, 12, 13, 13, 10, 12, 12, 14, 11, 10, 13, 11, 11, 13, 12, 10, 10, 12, 13, 11, 10, 11, 11, 11, 11, 11, 14, 12, 13, 13, 10, 12, 10, 10, 12, 11, 12, 11, 11, 12, 12, 14, 11, 10, 11, 12, 13, 12, 11, 11, 11, 14, 11, 13, 12, 10, 12, 13, 10, 10, 10, 10, 14, 12, 11, 11, 12, 14, 14, 10, 11, 11, 10, 10, 12, 12, 11, 12, 10, 12, 13, 10, 12, 10, 13, 12, 10, 12, 10, 11, 12, 11, 12, 10, 13, 12, 11, 11, 11, 11, 12, 14, 11, 11, 12, 14, 11, 13, 11, 10, 13, 12, 11, 13, 14, 10, 11, 11, 15, 15, 18, 19, 19, 16, 17, 15, 15, 15, 15, 15, 19, 18, 16, 17, 16, 15, 15, 15, 18, 17, 15, 17, 17, 16, 17, 15, 15, 16, 16, 15, 18, 15, 16, 17, 16, 16, 15, 18, 15, 16, 17, 16, 16, 15, 18, 16, 15), height=c(57, 67.5, 54.5, 53, 57, 61, 58, 56, 58.5, 60, 53, 54, 58.5, 60.5, 58, 53, 50, 53, 59, 61.5, 49, 52.5, 48, 62.5, 65, 51.5, 60, 52, 60, 60, 49, 65.5, 60, 57.5, 52, 59, 59, 55, 57, 57, 60, 59, 51, 57, 50, 57, 54, 52.5, 55, 60.5, 54.5, 61.5, 62, 54.5, 54, 57, 62.5, 60, 50, 57.5, 60, 60, 51, 49, 59, 57.5, 58, 55.5, 52, 60.5, 57, 58, 60, 59, 59.5, 56, 53, 51, 69, 53, 57, 53, 63, 54.5, 58, 63, 52, 61.5, 52.5, 62, 61.5, 64.5, 57.5, 60, 60, 61, 58, 61.5, 55, 55, 52, 54, 63, 47, 60.5, 56, 58.5, 55, 58.5, 51, 59.5, 54, 58, 58, 52.5, 60.5, 54.5, 46.5, 56.5, 55, 57, 54.5, 54, 53, 57, 59, 55, 55.5, 51.5, 61, 63.5, 57.5, 53, 56.5, 54, 52, 53, 61.5, 58.5, 52, 58, 61, 51, 62, 64.5, 58.5, 52.5, 58, 66, 63.5, 64, 58.5, 64, 60.5, 55, 53, 48, 58.5, 64.5, 60.5, 56, 58.5, 54, 58, 64.5, 50.5, 60, 51.5, 59, 53, 53, 53.5, 52, 64, 63, 58, 58.5, 59.5, 51, 57.5, 47, 55.5, 57, 61, 55.5, 53.5, 50, 61.5, 61.5, 55, 58, 56.5, 57, 52, 52.5, 62.5, 65, 50, 57.5, 56.5, 57.5, 54, 59.5, 59.5, 58, 57, 55, 60, 56.5, 60, 59, 63, 59, 53, 61.5, 48, 63, 55.5, 64.5, 59, 58.5, 46, 62, 53, 49.5, 68, 59.5,

20 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS , 55, 51.5, 56, 63, 48, 56.5, 59, 60.5, 65, 55, 55, 51, 52.5, 64, 59, 53.5, 60.5, 59.5, 56.5, 63, 56.5, 61.5, 52, 57, 63.5, 63, 65, 61.5, 62, 55, 51, 54, 57, 56, 61.5, 53, 59, 56, 59, 52.5, 52.5, 50, 59.5, 64, 61, 57.5, 60, 55, 56, 51, 59.5, 50.5, 57.5, 54, 57, 58, 60, 49, 57, 52, 52, 59.5, 58, 64, 57, 53, 62, 53, 56.5, 57, 48, 49.5, 51, 59.5, 56, 54, 57.5, 55.5, 50, 60, 57, 69, 64, 63, 58, 66.5, 60.5, 57, 64, 61, 68.5, 69.5, 72, 71, 63, 61.5, 63.5, 62, 65, 61, 63, 66, 68, 66, 72, 66, 67, 64, 60.5, 63, 64, 60.5, 62, 60, 64.5, 67, 61.5, 66, 60, 64, 68.5, 67, 67, 66, 62.5, 61, 62, 66.5, 60, 64, 62, 58, 68.5, 61.5, 64.5, 67, 68.5, 61, 65, 70, 60, 72, 59, 69, 67.5, 65.5, 65.5, 64.5, 62, 65.5, 67.5, 68, 63.5, 58.5, 65.5, 65, 70.5, 63, 63.5, 66, 63, 67.5, 61.5, 60, 64.5, 68.5, 64.5, 61, 57, 62.5, 58, 62.5, 72.5, 65.5, 69.5, 66.5, 59, 68, 67, 63, 64.5, 62, 62, 66, 62, 62.5, 57, 68.5, 59, 61.5, 65, 66.5, 65, 68, 67, 64.5, 70, 60, 64, 62.5, 61, 64, 65.5, 61, 65.4, 63.5, 67.5, 66, 62, 61.5, 69, 66, 66, 69, 71, 60.5, 58, 63, 69, 59.5, 64, 74, 64.5, 70, 67, 64.5, 65, 62.5, 68, 68.5, 67.5, 60, 68, 65, 64, 73, 59, 70.5, 66, 61.5, 67, 62.5, 52.5, 72, 71, 65.5, 60, 60, 60.5, 59, 68, 62, 64.5, 63, 66, 67, 57, 63, 62, 65, 68.5, 69, 71, 58, 63.5, 66.5, 67, 64, 61, 67, 67.5, 68, 66.5, 67, 60.5, 70, 63.5, 65, 63, 61, 65.5, 67.5, 61, 63, 63.5, 70, 71.5, 65.5, 74, 68, 61, 61, 61, 61.5, 67, 63, 67.5, 70.5, 70.5, 62, 62, 61.5, 62, 60.5, 62.5, 63, 63, 62, 65.5, 65, 70, 66, 63, 68, 59, 68.5, 63, 63, 66, 62, 68, 72.5, 64, 65.5, 61.5, 57, 64, 66, 64.5, 67, 72, 65, 63, 62, 71, 61, 65.5, 62, 65.5, 70.5, 62, 65.5, 65, 61, 69.5, 65, 65.5, 63.5, 60.5, 65, 67, 63.5, 59.5, 63.5, 61, 66, 64, 62, 60.5, 71, 64.5, 61, 62.5, 64.5, 68, 68.5, 62, 60, 59.5, 61, 63, 67.5, 69, 68, 58, 65, 69, 70, 71, 66, 72, 66, 68, 70, 67.5, 70, 64, 66, 62, 65.5, 64.5, 66.5, 67, 63, 64, 69, 71, 68, 70.5, 67, 69, 62, 67.5, 73, 64, 68.5, 62, 73.5, 66.5, 67, 63, 69.5, 70, 66, 66, 63, 70.5, 60, 72, 70, 72, 67, 68, 60, 63, 66.5), fev=c(1.708, 1.724, 1.72, 1.558, 1.895, 2.336, 1.919, 1.415, 1.987, 1.942, 1.602, 1.735, 2.193, 2.118, 2.258, 1.932, 1.472, 1.878, 2.352, 2.604, 1.4, 1.256, 0.839, 2.578, 2.988, 1.404, 2.348, 1.755, 2.98, 2.1, 1.282, 3, 2.673, 2.093, 1.612, 2.175, 2.725, 2.071, 1.547, 2.004, 3.135, 2.42, 1.776, 1.931, 1.343, 2.076, 1.624, 1.344, 1.65, 2.732, 2.017, 2.797, 3.556, 1.703, 1.634, 2.57, 3.016, 2.419, 1.569, 1.698, 2.123, 2.481, 1.481, 1.577, 1.94, 1.747, 2.069, 1.631, 1.536, 2.56, 1.962, 2.531, 2.715, 2.457, 2.09, 1.789, 1.858, 1.452, 3.842, 1.719, 2.111, 1.695, 2.211, 1.794, 1.917, 2.144, 1.253, 2.659, 1.58, 2.126, 3.029, 2.964, 1.611, 2.215, 2.388, 2.196, 1.751, 2.165, 1.682, 1.523, 1.292, 1.649, 2.588, 0.796, 2.574, 1.979, 2.354, 1.718, 1.742, 1.603, 2.639, 1.829, 2.084, 2.22, 1.473, 2.341, 1.698, 1.196, 1.872, 2.219, 2.42, 1.827, 1.461, 1.338, 2.09, 1.697, 1.562, 2.04, 1.609, 2.458, 2.65, 1.429, 1.675, 1.947, 2.069, 1.572, 1.348, 2.288, 1.773, 0.791, 1.905, 2.463, 1.431, 2.631, 3.114, 2.135, 1.527, 2.293, 3.042, 2.927, 2.665, 2.301, 2.46, 2.592, 1.75, 1.759, 1.536, 2.259, 2.048, 2.571, 2.046, 1.78, 1.552, 1.953, 2.893, 1.713, 2.851, 1.624, 2.631, 1.819, 1.658, 2.158, 1.789, 3.004, 2.503, 1.933, 2.091, 2.316, 1.704, 1.606, 1.165, 2.102, 2.32, 2.23, 1.716, 1.79, 1.146, 2.187, 2.717, 1.796, 1.953, 1.335, 2.119, 1.666, 1.826, 2.709, 2.871, 1.092, 2.262, 2.104,

21 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS , 1.69, 2.973, 2.145, 1.971, 2.095, 1.697, 2.455, 1.92, 2.164, 2.13, 2.993, 2.529, 1.726, 2.442, 1.102, 2.056, 1.808, 2.305, 1.969, 1.556, 1.072, 2.042, 1.512, 1.423, 3.681, 1.991, 1.897, 1.37, 1.338, 2.016, 2.639, 1.389, 1.612, 2.135, 2.681, 3.223, 1.796, 2.01, 1.523, 1.744, 2.485, 2.335, 1.415, 2.076, 2.435, 1.728, 2.85, 1.844, 1.754, 1.343, 2.303, 2.246, 2.476, 3.239, 2.457, 2.382, 1.64, 1.589, 2.056, 2.226, 1.886, 2.833, 1.715, 2.631, 2.55, 1.912, 1.877, 1.935, 1.539, 2.803, 2.923, 2.358, 2.094, 1.855, 1.535, 2.135, 1.93, 2.182, 1.359, 2.002, 1.699, 2.5, 2.366, 2.069, 1.418, 2.333, 1.514, 1.758, 2.535, 2.564, 2.487, 1.591, 1.624, 2.798, 1.691, 1.999, 1.869, 1.004, 1.427, 1.826, 2.688, 1.657, 1.672, 2.015, 2.371, 2.115, 2.328, 1.495, 2.884, 2.328, 3.381, 2.17, 3.47, 3.058, 1.811, 2.524, 2.642, 3.741, 4.336, 4.842, 4.55, 2.841, 3.166, 3.816, 2.561, 3.654, 2.481, 2.665, 3.203, 3.549, 2.236, 3.222, 3.111, 3.49, 3.147, 2.52, 2.292, 2.889, 2.246, 1.937, 2.646, 2.957, 4.007, 2.386, 3.251, 2.762, 3.011, 4.305, 3.906, 3.583, 3.236, 3.436, 3.058, 3.007, 3.489, 2.864, 3.428, 2.819, 2.25, 4.683, 2.352, 3.108, 3.994, 4.393, 3.208, 2.592, 3.193, 1.694, 3.957, 2.346, 4.789, 3.515, 2.754, 2.72, 2.463, 2.633, 3.048, 3.111, 3.745, 2.384, 2.094, 3.183, 3.074, 3.977, 3.354, 3.411, 2.387, 3.171, 3.887, 2.646, 2.504, 3.587, 3.845, 2.971, 2.891, 1.823, 2.417, 2.175, 2.735, 4.273, 2.976, 3.835, 4.065, 2.318, 3.596, 3.395, 2.751, 2.673, 2.556, 2.542, 2.608, 2.354, 2.599, 1.458, 3.795, 2.491, 3.06, 2.545, 2.993, 3.305, 4.756, 3.774, 2.855, 2.988, 2.498, 3.169, 2.887, 2.704, 3.515, 3.425, 2.287, 2.434, 2.365, 3.086, 2.696, 2.868, 2.813, 4.309, 3.255, 3.413, 4.593, 4.111, 1.916, 1.858, 2.975, 3.35, 2.901, 2.241, 4.225, 3.223, 5.224, 4.073, 4.08, 2.606, 3.169, 4.411, 3.791, 3.089, 2.465, 3.343, 3.2, 2.913, 4.877, 2.358, 3.279, 2.581, 2.347, 2.691, 2.827, 1.873, 3.751, 2.538, 2.758, 3.05, 3.079, 2.201, 1.858, 2.216, 3.403, 3.501, 2.578, 3.078, 3.186, 1.665, 2.081, 2.974, 3.297, 4.073, 4.448, 3.984, 2.25, 2.752, 2.304, 3.68, 3.102, 2.862, 2.677, 3.023, 3.681, 3.255, 3.692, 2.356, 4.591, 3.082, 3.297, 3.258, 2.216, 3.247, 4.324, 2.362, 2.563, 3.206, 3.585, 4.72, 3.331, 5.083, 3.498, 2.417, 2.364, 2.341, 2.759, 2.953, 3.231, 3.078, 3.369, 3.529, 2.866, 2.891, 3.022, 3.127, 2.866, 2.605, 3.056, 2.569, 2.501, 3.32, 2.123, 3.78, 3.847, 3.785, 3.924, 2.132, 2.752, 2.449, 3.456, 3.073, 2.688, 3.329, 4.271, 3.53, 2.928, 2.689, 2.332, 2.934, 2.276, 3.11, 2.894, 4.637, 2.435, 2.838, 3.035, 4.831, 2.812, 2.714, 3.086, 3.519, 4.232, 2.77, 3.341, 3.09, 2.531, 2.822, 3.038, 2.935, 2.568, 2.387, 2.499, 4.13, 3.001, 3.132, 3.577, 3.222, 3.28, 2.659, 2.822, 2.14, 4.203, 2.997, 3.12, 2.562, 3.082, 3.806, 3.339, 3.152, 2.458, 2.391, 3.141, 2.579, 3.104, 4.045, 4.763, 2.1, 3.069, 2.785, 4.284, 4.506, 2.906, 5.102, 3.519, 3.688, 4.429, 4.279, 4.5, 2.635, 2.679, 2.198, 3.345, 3.082, 3.387, 3.082, 2.903, 3.004, 5.793, 3.985, 4.22, 4.724, 3.731, 3.406, 3.5, 3.674, 5.633, 3.122, 3.33, 2.608, 3.645, 3.799, 4.086, 2.887, 4.07, 3.96, 4.299, 2.981, 2.264, 4.404, 2.278, 4.504, 5.638, 4.872, 4.27, 3.727, 2.853, 2.795, 3.211) )

22 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS Comparison of Bayesian and Frequentist Results The estimates are very similar numerically Estimate of both the βs and σ 2 With Gibbs sampling, there is some random simulation (Markov chain) error so the estimates will not be identical (unless you fix the random number seed, number of iterations, and burn-in period) With frequentist results, the estimation error is so small that it rarely noticeable Frequentist and Bayesian results would be noticeably different if the Bayesian analysis used informative priors and the data disagreed with the prior Interpretation of β 2 (height coefficient) Frequentist: The expected change in FEV comparing individuals who differ by one unit in height while holding age constant Bayesian: The mean change in FEV comparing individuals who differ by one unit in height while holding age constant Interpretation of the confidence interal and credible interval for β 2 Frequentist 95% confidence interval We are 95% confident that the true value of β 2 lies within the CI If we repeated the same experiment many, many times and calculated the CI each time, on average, we would expect approximately 95% of those CIs to contain the true β 2 Each individual CI either contains the true β 2 (with probability 1) or it doesn t (with probability 0); we just don t know which is the case

23 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 23 Bayesian 95% credible interval β 2 has a posterior distribution conditional on the observed data There is 95% probability that β 2 falls within the credible interval Statistical significance (p-value) Frequentist The probability under the null hypothesis of observing data as extreme or more extreme that we observed in this experiment Compares the data in this study to (potential) future studies and asks Is it unusual? Bayesian Can calculate posterior probabilities, which have direct interpretations Example: The probability that β 2 > 0 Simply count the number of iterations of the Gibbs sampler that β 2 > 0 then divide by the total number of iterations For significance, could just observe if the credible interval contains the null value However, Bayesian analyses are not set up in the classical null and alternative hypothesis framework (you don t reject the null in a Bayesian analysis) More emphasis on scientific relevance and estimation of effect size than determining if results are significant or not significant

24 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS Example 3: Hierarchical Model, Linear Regression Rats data, assuming independence For this example, we will use the Rats data available on the WinBUGS examples Examines the association of weight (outcome) with age (predictor) This is a classic growth curve model, examining how some outcome changes over time Before we do the hierarchical, growth curve model we will consider a simple linear regression model This analysis will assume that all observations are independent, even though each rat is measured at multiple time points The WinBUGS code to fit the model is model { for( i in 1 : N ) { for( j in 1 : T ) { Y[i, j] ~ dnorm(mu[i, j],tau.c) mu[i, j] <- alpha + beta * (x[j] - xbar) } } tau.c ~ dgamma(0.001,0.001) sigma <- 1 / sqrt(tau.c) alpha ~ dnorm(0.0,1.0e-6) beta ~ dnorm(0.0,1.0e-6) alpha0 <- alpha - xbar * beta } #data: same as before list(x = c(8.0, 15.0, 22.0, 29.0, 36.0), xbar = 22, N = 30, T = 5, Y = structure(.data = c(151, 199, 246, 283, 320, 145, 199, 249, 293, 354, 147, 214, 263, 312, 328, 155, 200, 237, 272, 297,

25 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS , 188, 230, 280, 323, 159, 210, 252, 298, 331, 141, 189, 231, 275, 305, 159, 201, 248, 297, 338, 177, 236, 285, 350, 376, 134, 182, 220, 260, 296, 160, 208, 261, 313, 352, 143, 188, 220, 273, 314, 154, 200, 244, 289, 325, 171, 221, 270, 326, 358, 163, 216, 242, 281, 312, 160, 207, 248, 288, 324, 142, 187, 234, 280, 316, 156, 203, 243, 283, 317, 157, 212, 259, 307, 336, 152, 203, 246, 286, 321, 154, 205, 253, 298, 334, 139, 190, 225, 267, 302, 146, 191, 229, 272, 302, 157, 211, 250, 285, 323, 132, 185, 237, 286, 331, 160, 207, 257, 303, 345, 169, 216, 261, 295, 333, 157, 205, 248, 289, 316, 137, 180, 219, 258, 291, 153, 200, 244, 286, 324),.Dim = c(30,5))) #init list(alpha =150, beta = 10, tau.c = 1) In this case, mu[i,j] is like E(y) in frequentist linear regression Notice that each observation has the same variance, the usual assumption of linear regression I got the following results node mean sd MC error 2.5% median 97.5% start sample alpha beta sigma Which we can compare to the following Stata output

26 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 26 Source SS df MS Number of obs = F( 1, 148) = Model Prob > F = Residual R-squared = Adj R-squared = Total Root MSE = y Coef. Std. Err. t P> t [95% Conf. Interval] x _cons Rats data, modeling correlation As mentioned earlier, each rat was measured at multiple time points We might expect that weights measured on the same rat will be more alike than weights measured on different rats To model this correlation, we will add a random intercept b i to the model For the Bayesian analysis, wee need to add a prior distribution for b i b i N(0, σ 2 b) Each rat is allowed to have a different baseline weight Heavy rats at baseline will have large b i, light rats will have low b i model { for( i in 1 : N ) { for( j in 1 : T ) { Y[i, j] ~ dnorm(mu[i, j],tau.c) mu[i, j] <- alpha + beta * (x[j] - xbar) + b[i] } b[i] ~ dnorm(0.0,tau.b)

27 CHAPTER 12. BAYESIAN ANALYSIS OF REGRESSION MODELS 27 } tau.c ~ dgamma(0.001,0.001) tau.b ~ dgamma(0.001,0.001) sigma <- 1 / sqrt(tau.c) sigma.b <- 1 / sqrt(tau.b) alpha ~ dnorm(0.0,1.0e-6) beta ~ dnorm(0.0,1.0e-6) alpha0 <- alpha - xbar * beta } list(alpha =150, beta = 10, tau.c = 1, b=c(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0), tau.b=1) Here is the output I obtained for the hierarchical model The output: node mean sd MC error 2.5% median 97.5% start sample alpha beta E sigma sigma.b To facilitate comparison, the results from the model where we assumed independence are below node mean sd MC error 2.5% median 97.5% start sample alpha beta sigma Note the smaller standard deviation for β when we assume independence When outcomes are positively correlated, and we ignore that correlation, our results are anti-conservative Standard errors are too small, CIs too narrow Will declare that results are significant when they are not Not just a Bayesian result; correlated outcomes are critical to consider in all analyses when they arise