Permutation inference for the General Linear Model

Size: px

Start display at page:

Download "Permutation inference for the General Linear Model"

Luke Carpenter
5 years ago
Views:

1 Permutation inference for the General Linear Model Anderson M. Winkler fmrib Analysis Group 3.Sep.25 Winkler Permutation for the glm / 63

2 in jalapeno: winkler/bin/palm Winkler Permutation for the glm 2 / 63

3 Winkler Permutation for the glm 3 / 63

4 Winkler Permutation for the glm 4 / 63

5 Why parametric tests were useful? Practical difficulty Often it is impossible to repeat the same experiment multiple times. Example: Consider comparing the effect of the sex of the subjects on school scores. The sex cannot be simply changed and the test repeated. The same for, e.g., presence of a disorder. In some cases, costs associated with data collection are high, or the samples are unique. Winkler Permutation for the glm 5 / 63

6 Why parametric tests were useful? Instead: Sample subjects from a population, randomly and independently, and assume a certain distribution for the data. Compute a statistic. All else being equal, an extreme value suggest a difference in the parameter that is allowed to vary (usually the mean). Winkler Permutation for the glm 6 / 63

7 Introduction Principles GLM Examples Why parametric tests were useful? Computations involved are quite simple: Calculate the statistic just once, then refer to a table to find what the critical value is for a given significance level. Winkler Permutation for the glm 7 / 63

8 However... - Random sampling is rare in biomedical research. - Population model is often violated, along with the assumptions that come with it. Winkler Permutation for the glm 8 / 63

9 Introduction Principles GLM Examples Why parametric tests were useful? Also good for imaging in the first years: Thousands of voxels need speed, and parametric is much faster. Winkler Permutation for the glm 9 / 63

10 However... - Assumptions can be invalid, and will be invalid in many voxels. - Different imaging modalities have different properties. A single set of assumptions will be invalid for some of them. Winkler Permutation for the glm / 63

11 Permutation tests are superior - No underlying theoretical distribution. - No stringent assumptions (normality, independence, homogeneous variances). - Non-random samples are permitted (but need exchangeability if there is no randomisation). - Wide variety of statistics. - Good for small datasets. - All information needed is in the dataset itself, not on an idealised population. - Resilient to outliers (and robust statistics can be used, even without a limiting distribution). Winkler Permutation for the glm / 63

12 Permutation tests are superior Only assumption: Data are exchangeable. i.e., the joint distribution remains unchanged if data are swapped. Winkler Permutation for the glm 2 / 63

13 Principles Winkler Permutation for the glm 3 / 63

14 General Linear Model Y = Mψ + ɛ Permute the data (Y) or the rows of the design matrix (M), re-estimate the parameters (ψ) and residuals (ɛ), compute a new statistic. Repeat this many times, count how often a permuted statistic is greater or equal than the non-permuted, and divide by the number of tests. That produces the p-value. Winkler Permutation for the glm 4 / 63

15 Two-sample t test Winkler Permutation for the glm 5 / 63

16 Two-sample t test Y = Mψ + ɛ = [ ψ ψ 2 ψ i =? ɛ i =? ] + ɛ ɛ 2 ɛ 3 ɛ 4 ɛ 5 ɛ 6 Estimation: ˆψ = (M M) M Y, ˆɛ = Y M ˆψ. Winkler Permutation for the glm 6 / 63

17 Two-sample t test Y = Mψ + ɛ = [ t = ] How likely is a value at least as large as this if there is no effect? Winkler Permutation for the glm 7 / 63

18 Two-sample t test t = +.33 t =.9 t = +.25 t =.93 t = +.42 t = t = +.4 t =.45 t = 2.4 t =.47 t = +.47 t = +2.4 t = +.45 t =.4 t = 2.24 t =.42 t = +.93 t =.25 t = +.9 t =.33 Winkler Permutation for the glm 8 / 63

19 Two-sample t test t = +.33 t =.9 t = +.25 t =.93 t = +.42 t = t = +.4 t =.45 t = 2.4 t =.47 t = +.47 t = +2.4 t = +.45 t =.4 t = 2.24 t =.42 t = +.93 t =.25 t = +.9 t =.33 Winkler Permutation for the glm 9 / 63

20 Two-sample t test There were 3 cases of a statistic at least as large as the one observed. We have run 2 permutations. Thus: p = 3 2 =.5 Winkler Permutation for the glm 2 / 63

21 One-sample t test Winkler Permutation for the glm 2 / 63

22 One-sample t test Y = Mψ + ɛ = [ ] ψ + ψ i =? ɛ i =? ɛ ɛ 2 ɛ 3 ɛ 4 Estimation: ˆψ = (M M) M Y, ˆɛ = Y M ˆψ. Winkler Permutation for the glm 22 / 63

23 One-sample t test Y = Mψ + ɛ = [ +.29 ] + t = How likely is a value at least as large as this if there is no effect? Winkler Permutation for the glm 23 / 63

24 One-sample t test t = +.65 t = +.56 t =.45 t =.54 t =.42 t =.5 t = 2.7 t = t = +3.3 t = +2.7 t = +.5 t = +.42 t = +.54 t = +.45 t =.56 t =.65 Winkler Permutation for the glm 24 / 63

25 One-sample t test t = +.65 t = +.56 t =.45 t =.54 t =.42 t =.5 t = 2.7 t = t = +3.3 t = +2.7 t = +.5 t = +.42 t = +.54 t = +.45 t =.56 t =.65 Winkler Permutation for the glm 25 / 63

26 One-sample t test There were 3 cases of a statistic at least as large as the one observed. We have run 6 sign flippings. Thus: p = 3 6 =.875 Winkler Permutation for the glm 26 / 63

27 Single test Parametric Relative frequency Test statistic Winkler Permutation for the glm 27 / 63

28 Single test Permutation Relative frequency Test statistic Winkler Permutation for the glm 28 / 63

29 Many tests (maximum) Parametric Relative frequency Test statistic Winkler Permutation for the glm 29 / 63

30 Many tests (maximum) Permutation Relative frequency Test statistic Winkler Permutation for the glm 3 / 63

31 Exactness - If all permutations are done exact. - If many permutations are done exact. In either sense there are no distributional assumptions. However, in the second sense it means that P(p α) = α. Winkler Permutation for the glm 3 / 63

32 A bit more on the assumptions - Exchangeable errors (ee) Permutations. - Independent and symmetric errors (ise) Sign-flippings. - Both (ee + ise) Permutations + Sign-flippings. Winkler Permutation for the glm 32 / 63

33 How many permutations are possible? - Depends on the design: repeated rows reduce the number of possible permutations. - Regardless, the number grows very quicky, and can be astronomical even with small samples. ex.: 24 subjects, single continuous EV, J = 24! Winkler Permutation for the glm 33 / 63

34 How many permutations should we run? - The p-value itself is an estimate, and more permutations narrow confidence interval around it, giving more credence to the results. - More permutations do not increase power, although too few may not be powerful: smallest p = /J. Winkler Permutation for the glm 34 / 63

35 Permutation for the glm Winkler Permutation for the glm 35 / 63

36 Problem: Nuisance variables. What do we do? Y = Mψ + ɛ Winkler Permutation for the glm 36 / 63

37 Model partitioning Y = Xβ + Zγ + ɛ Winkler Permutation for the glm 37 / 63

38 Model partitioning: Before y y 2 y 3 y 4 y 5 y 6 y 7 y 8 = Y = Mψ + ɛ ψ ψ 2 ψ 3 ψ 4 + ɛ ɛ 2 ɛ 3 ɛ 4 ɛ 5 ɛ 6 ɛ 7 ɛ 8 Winkler Permutation for the glm 38 / 63

39 Model partitioning: After y y 2 y 3 y 4 y 5 y 6 y 7 y 8 = Y = Xβ + Zγ + ɛ [.59 ] β γ γ 2 γ 3 + ɛ ɛ 2 ɛ 3 ɛ 4 ɛ 5 ɛ 6 ɛ 7 ɛ 8 Winkler Permutation for the glm 39 / 63

40 Model partitioning Features: - ˆβ = C ˆψ - Ĉov( ˆβ) = C Ĉov( ˆψ)C - span(x) span(z) = span(m) - X Z = D = (M M) C u = null (C) C v = C u C(C DC) C DC u X = MDC ( C DC ) ( Z = MDC v C ) v DC v In other words: The two models are equivalent. Winkler Permutation for the glm 4 / 63

41 Permutation matrix (P) Permutation matrix Original data Permuted data = Winkler Permutation for the glm 4 / 63

42 Sign-flipping matrix (S) Sign-flipping matrix Original data Sign-flipped data = Winkler Permutation for the glm 42 / 63

43 Permutation and sign-flipping matrix (B = PS) Permutation and sign-flipping matrix Original data Permuted & sign-flipped = Winkler Permutation for the glm 43 / 63

44 Method Model Draper Stoneman Y = PXβ + Zγ + ɛ Still White PR Z Y = Xβ + ɛ Freedman Lane (PR Z + H Z ) Y = Xβ + Zγ + ɛ Manly PY = Xβ + Zγ + ɛ ter Braak (PR M + H M ) Y = Xβ + Zγ + ɛ Kennedy PR Z Y = R Z Xβ + ɛ Huh Jhun PQ R Z Y = Q R Z Xβ + ɛ Smith Y = PR Z Xβ + Zγ + ɛ Parametric Y = Xβ + Zγ + ɛ, ɛ N (, σ 2 I) H is the hat matrix, such that Ŷ = HY, R is the residual-forming matrix, such that ˆɛ = RY. I = H + R. The subscript indicates whether these consider the full model (M) or just the nuisance (Z). P is a permutation or sign-flipping matrix. Winkler Permutation for the glm 44 / 63

45 Freedman Lane method. Regress the data against the full model and compute the unpermuted statistic. 2. Regress the data against the nuisance-only model and compute the residuals. 3. Shuffle the residuals, then add back the estimated nuisance effects. 4. Regress this data against the full model and compute the permuted statistic. 5. Repeat the Steps 2 4 many times. 6. Count how many times the permuted statistic was found to be equal to or larger than the unpermuted, and divide by the number of permutations. The result is the p-value. There are computational simplifications that can be used. Winkler Permutation for the glm 45 / 63

46 The F -statistic Consider the statistic: F = ˆψ / C (C (M M) C) C ˆψ rank (C) = ˆβ / (X X) ˆβ rank (C) ˆɛ ˆɛ N rank (M) ˆɛ ˆɛ N rank (X) rank (Z) When rank(c) =, ˆβ is a scalar and the t statistic can be expressed as a function of F as t = sign( ˆβ) F. Winkler Permutation for the glm 46 / 63

47 Need pivotality The statistic must be pivotal. - But if there are heterogeneous variances and groups of different sizes, the F -statistic is no longer pivotal. - And with it, the t statistic, the correlation coefficient r, and the R 2, are no longer pivotal either. Winkler Permutation for the glm 47 / 63

48 The G-statistic Consider the statistic: G = ˆψ C (C (M WM) C) C ˆψ Λ s where W is a diagonal matrix that has elements: n W nn = g n R n n ˆɛ g nˆɛ gn Λ = + 2(s ) s(s + 2) g and where s = rank (C). n g R nn ( n g W ) 2 nn trace (W) Winkler Permutation for the glm 48 / 63

49 Variance groups (vgs) W can be seen as a weighting matrix, the square root of which normalises the model such that the errors have then unit variance and can be ignored. σ 2 σ 2 σ 2 σ 2 σ 2 σ 2 VG ϵ σ 2 W = σ 2 σ 2 σ 2 σ 2 σ 2 σ 2 Winkler Permutation for the glm 49 / 63

50 Variance groups (vgs) W can be seen as a weighting matrix, the square root of which normalises the model such that the errors have then unit variance and can be ignored. 3 VGs ϵ σ 2 ϵ 2 ϵ 2 σ 2 2 σ 2 3 W = σ 2 σ 2 σ 2 σ 2 σ 2 2 σ 2 2 σ 2 2 σ 2 2 σ 2 3 σ 2 3 σ 2 3 σ 2 3 Winkler Permutation for the glm 5 / 63

51 The G-statistic G generalises a number of well known statistical tests: Homoscedastic errors, unrestricted exchangeability Homoscedastic within vg, restricted exchangeability rank (C) = rank (C) > Square of Student s t Square of Aspin Welch v F -ratio Welch s v 2 Winkler Permutation for the glm 5 / 63

52 Heterogeneous variances: the G-statistic - When rank (C) =, the t-equivalent to the G-statistic is v = ˆψ C (C (M WM) C) 2, which is the same v-statistic for the Behrens-Fisher problem. - In the absence of variance groups, G and v are equivalent to F and t respectively. - Although not typically necessary, approximate parametric p-values for the G-statistic can be computed from an F -distribution with degrees of freedom ν = s and ν 2 = 2(s ) 3(Λ ). Winkler Permutation for the glm 52 / 63

53 Examples Winkler Permutation for the glm 53 / 63

54 Example : Mean effect Scenario Consider a multi-subject fmri study to investigate the bold response associated with a novel experimental task. After the first-level analysis (within subject), maps of contrasts of parameter estimates for each subject are used in a second level analysis. The regressor for the effect of interest (the mean effect) is simply a column of ones; nuisance variables, such as handedness, can be included in the model. Winkler Permutation for the glm 54 / 63

55 Example : Mean effect Coded data (Y) eb vg Model (M) m m 2 Subject h Subject 2 h 2 Subject 3 h 3 Subject 4 h 4 Subject 5 h 5 Subject 6 h 6 Subject 7 h 7 Subject 8 h 8 Subject 9 h 9 Subject h Subject h Subject 2 h 2 Contrast (C ) + Contrast 2 (C 2) Winkler Permutation for the glm 55 / 63

56 Example : Mean effect Solution Permutations of the data or of the design matrix do not change the model with respect to the regressor of interest. However, by treating the errors as symmetric, instead of permutation, the signs of the ones in the design matrix, or of each datapoint, can be flipped randomly to create the empirical distribution from which inference can be performed. The procedure can be performed as in either the Freedman Lane or Smith methods. Winkler Permutation for the glm 56 / 63

57 Example 2: Multiple regression Scenario Consider the analysis of a study that compares patients and controls with respect to brain cortical thickness, and that recruiting process ensured that all selected subjects are exchangeable. Elder subjects may, however, have thinner cortices, regardless of the diagnosis. To control for the confounding effect of age, it is included in the design as a nuisance regressor. Sex is also included. Winkler Permutation for the glm 57 / 63

58 Example 2: Multiple regression Coded data (Y) eb vg Model (M) m m 2 m 3 m 4 Subject a s Subject 2 a 2 s 2 Subject 3 a 3 s 3 Subject 4 a 4 s 4 Subject 5 a 5 s 5 Subject 6 a 6 s 6 Subject 7 a 7 s 7 Subject 8 a 8 s 8 Subject 9 a 9 s 9 Subject a s Subject a s Subject 2 a 2 s 2 Contrast (C ) + Contrast 2 (C 2) + Winkler Permutation for the glm 58 / 63

59 Example 2: Multiple regression Solution The permutation strategy follows the Freedman Lane or Smith methods, with the residuals of the reduced model being permuted under unrestricted exchangeability. Winkler Permutation for the glm 59 / 63

60 Example 4: Unequal group variances Scenario Consider a study using fmri to compare whether the bold response associated with a certain cognitive task would differ among subjects with autistic spectrum disorder (asd) and control subjects, while taking into account differences in age and sex. In this hypothetical example, the cognitive task is known to produce more erratic signal changes in the patient group than in controls. Therefore, variances cannot be assumed to be homogeneous with respect to the group assignment of subjects. Winkler Permutation for the glm 6 / 63

61 Example 4: Unequal group variances Coded data (Y) eb vg Model (M) m m 2 m 3 m 4 Subject a s Subject 2 a 2 s 2 Subject 3 a 3 s 3 Subject 4 a 4 s 4 Subject 5 a 5 s 5 Subject 6 a 6 s 6 Subject a 7 s 7 Subject a 8 s 8 Subject a 9 s 9 Subject 2 2 a s Subject 2 2 a s Subject a 2 s 2 Contrast (C ) + Contrast 2 (C 2) + Winkler Permutation for the glm 6 / 63

62 Example 4: Unequal group variances Solution This is an example of the classical Behrens Fisher problem. To accommodate heteroscedasticity, two permutation blocks are defined according to the group of subjects. Under the assumption of independent and symmetric errors, the problem is solved by means of random sign-flipping, using the well known Welch s v statistic, a particular case of the G statistic. Winkler Permutation for the glm 62 / 63

63 That s all folks. Blog: brainder.org Winkler Permutation for the glm 63 / 63

Non-parametric Approaches to Inference

Non-parametric Approaches to Inference Dr. Martyn McFarquhar Division of Neuroscience & Experimental Psychology The University of Manchester 1 Edinburgh SPM course, 217 The parametric approach The SPM