6. Methods of Experimental Control. Chapter 6: Control Problems in Experimental Research

Transcription

1 6. Methods of Experimental Control Chapter 6: Control Problems in Experimental Research 1

2 Goals Understand: Advantages/disadvantages of within- and between-subjects experimental designs Methods of controlling for group differences in between-subjects experimental designs. Counter-balancing techniques for controlling sequence effects in within-subjects experimental designs 2

3 Between-Subjects Experimental Design (single factor, 2 levels) Population Sampling Sample Random Assignment Condition 1 Condition 2 Group 1 Group 2 3

4 Within-Subjects Experimental Designs (single factor, 2 levels) Population Sampling Sample 1/2 1/2 Condition 1 Condition 2 Sample Sample 4

5 Between subjects designs Different groups of people assigned to each level of IV Requires more participants, but avoids sequence effects (e.g., practice or fatigue). Potential Validity Problem: Are the people in the groups the same to begin with? If not, group differences = a confound 5

6 Simple Random Assignment Technique for minimizing group differences. Works simultaneously on ALL variables that might lead to group differences. Very powerful. Simply divide P s among levels of IV in a random (not arbitrary!) way. NOT to be confused with random sampling! 6

7 An Aside: Randomness Random is not the same as arbitrary. Randomness can be thought of as systematically non-systematic. That is, you set up a procedure to eliminate any possible biases. Arbitrary procedures, such as deciding haphazardly who goes in which group, may contain unknown biases 7

8 An Aside: Randomness Two flavours of randomness: Random with replacement: All options are there on every trial (dice, coin tosses) Random without replacement: When an option is picked on a given trial, it is no longer available for later ones (cards) Use the latter with random sampling and random assignment 8

9 Simple Random Assignment Typically, subjects are shuffled randomly using computer-generated random numbers. Physical mixing can also be used. Careful! Must use a method that is random without replacement. Example: drawing cards from a deck without putting them back in the deck for the next P. Counter-Example: Flipping a coin is right out! That is random with replacement 9

10 Random Assignment in Excel: One IV List all levels of IV in column A, with n repeats, where n is the # of individuals who will be in each group. Create a list of random numbers, using =rand() in column B, then sort according to column B (cut and paste values) If all participants are known ahead of time, just paste the list into column C If participants are not known ahead of time, test them as they come in, in the order of the list. 10

11 Random Assignment in Excel: Two IVs List all levels of IV1 in column A, with n repeats, where n is the # of individuals who will be in each level of IV1. List all levels of IV2 in column B, with n repeats within each level of IV1, where n is the number of individuals in each condition. Rest is as for one IV. 11

12 Discussion / Questions 12

13 Block Random Assignment Technique used in between-subjects designs to avoid clumping of conditions at particular times. (Also used in within-subjects designs, but more on this later... ) In sequential testings, simple RA may create a confound: Example: One might end up doing most of Level 1 before any of Level 2. Whether such clumping is a problem depends on the likelihood of history confounds, and the size of the sample, but it s never a bad idea to avoid it. 13

14 Block Random Assignment In block RA, the set of all conditions is shuffled several times, and a series of shuffled sets of conditions is created Example: Experiment with three conditions might produce a sequence like Counter-example: With simple RA, same experiment might produce a sequence like Note that, when using block RA, number of participants should ideally be an even multiple of the number of conditions 14

15 Example of Block RA E.g., Four conditions with n=10 each. In simple RA: Shuffle 10, 10, 10, 10 But might (just by chance) end up drawing most of the s before any s are drawn (for example). Block RA: Instead create 10 block decks of four cards each: 1, 1, 1, 1 in each deck. Shuffle each block deck, then stack all the block decks on top of one-another. 15

16 Block Random Assignment With simple randomization, you might end up with a sequence like this: Start of Study Middle of Study End of Study But with block randomization, you end up with a sequence like so: Start of Study Middle of Study End of Study 16

17 Random Assignment & Number of Participants RA works well with large N. What is large? Ideally 30, but as little as 10 is acceptable for small studies. But, chance of non-equivalent groups rises as N drops. What to do if you re stuck with small N? 17

18 Matching Instead of random assignment, test participants on matching variable(s) Then assign P s to groups such that groups have equal means (or frequency distributions if MVs are nominal) on the matching variable(s) Must have theoretical reason to expect an effect of matching variable Matching variable must be testable practically and without introducing testing effects. 18

19 Matching Easiest to do with variabless that can be assessed without lengthy testing Examples: Age, Gender, Weight... How many factors to match on? Can get complicated. May result in having to turn participants away if no match can be made May be simpler to test more subjects and let random assignment do its magic. 19

20 Step 1: Order Values GPA Step 2: Create pairs of adjacent values Step 3: From each pair, randomly assign one to each group Group 1 Group µ = 7.42 µ =

21 Discussion / Questions 21

22 A Between-Subjects Instructions: Experiment In a moment I m going to show you a video. It shows 6 people playing basketball. I want you to watch the video and keep a silent mental count of the number of passes between players. But to make it a little more difficult, I want you to keep two separate counts, one for the number of passes through the air, and another for number of bounce-passes, that is, the number of times they bounce the ball to one another. If you re on the right side of the class (your right, my left), do this for the white-shirted players only. If you re on the left side of the class (your left, my right), do this for the black-shirted players only. When we re done, I ll ask you to write down the two numbers (number of bounce passes and number of air passes) and give the data to me. Remember, just keep a silent count, don t make any noise or marks with your pen or anything like that. 22

23 23

24 What to Take Away From This? Perception: You don t actually see what s out there, just a reconstruction Cognition: There are limits to human attentional load Phil. of Science: It pays to observe the same thing several times, sometimes looking at details (=experiment), sometimes looking at the big picture (= naturalistic observation) RM&E: Some things can t be repeated within-subjects 24

25 Within-Subjects Experimental Designs Population Sampling Sample 1/2 1/2 Condition 1 Condition 2 Sample Sample 25

26 Within-Subjects Designs a.k.a. Repeated measures designs Same group goes through all levels of the IV. Often used when time to do one condition is small, or when available population is small. Fewer participants needed, no group effects, but may be impractical for some tasks. 26

27 Within-subjects Designs Allows more statistical power More participants per condition for a given grand N Don t have to deal with between-groups variance (even with RA, there s always some difference between groups that can obscure experimental effects) BUT, must be careful of sequence effects 27

28 Sequence Effects Going through level A of the IV may affect performance on level B. Progressive Sequence Effects: Practice effect: Participant gains knowledge, warms up, focuses, etc. Fatigue effect: Participant gets tired, bored, overwhelmed, etc. 28

29 Sequence Effects Carry-over effects: Non-symmetrical sequence effects. Doing Level A then Level B not the same as doing Level B then Level A. Common when levels vary in difficulty: simple then hard is easier than hard then simple. In this case, best to switch to between-subjects design. 29

30 Counterbalancing Group of techniques for minimizing progressive sequence effects in within-subjects experiments Complete counterbalancing Partial counterbalancing Sequence randomization Sequence randomization with constraints Latin square General idea is to equalize the number of participants who do each level in each order 30

31 Complete Counterbalancing Equal number of participants goes through each possible order of conditions Example: With two condition, half of Ps do 1-then-2, other half do 2-then-1 The ultimate form of counterbalancing, but not always practical. Number of orders of levels is n factorial or n!, where n is number of levels. 31

32 Factorial N! = N N-1 N # of Levels # of Orders

33 Factorial Why is it N!? Consider a case where you have 4 chairs and need to seat 4 people. How many people can you choose from to go in the st chair? 4...2nd chair? 3 (because one is in the 1st)...3rd chair? 2 (other 2 already seated)...4th chair? 1 33

34 Discussion / Questions 34

35 Partial Counterbalancing Sequence randomization: Necessary when large number of levels, but adds noise. Sequence randomization with constraints: Fellows Numbers: Ensure that correct answer is not the same more than X times in a row Same stimulus does not appear on sequential trials 35

36 Latin Square From an ancient roman game: Given an X by X grid, and X different symbols, can you place the symbols in the grid so that each appears only once per row and once per column? Similar to Sudoku puzzles. Even harder: Can you make it so that each symbol appears directly to the right of each other symbol once and only once? (only possible when X is even). This is called a balanced latin square 36

37 Latin Square A completed Balanced Latin Square can be used as a form of partial counter-balancing Each participant runs through the conditions in the order indicated by one row of the BLS Number of participants must be evenly divisible by number of levels 37

38 6x6 Balanced Latin Square Sequential Position Order # A B F C E D 2 B C A D F E 3 C D B E A F 4 D E C F B A 5 E F D A C B 6 F A E B D C 38

39 Latin Square Design If you run one participant through each of the 6 orders, then: Each of the 6 levels will have been done once in each of the 6 possible sequential positions. Each of the 6 levels will have been immediately preceded by each of the other 5 levels once and only once. If you run 60 people, then 10 will have gone through each order 39

40 Latin Square Design For example, if you test 60 people (6 x 10), then: 10 will have done A first, 10 B first, 10 C first will have done A second, 10 B second, will have done level A immediately preceded by B, 10 will have done A immediately preceded by C, etc... 40

41 Creating a Balanced Latin Square (if X is even) Build the first row according to the pattern: A B (x) C (x-1) D (x-2) E (x-3) etc... where x is the highest letter you re using (e.g., F if doing a 6 6). With 6 levels, row 1 is: A B F C E D Build the remaining rows by incrementing the letters by 1 (i.e., A becomes B, B becomes C...). Row 2 is B C A D F E Note that we wrap back to A when incrementing F 41

42 Latin Square With Odd Number of Levels Previous system only works with even # of levels. A 5 5 or 7 7 latin square cannot be balanced. With uneven # of levels, create latin square plus a leftright mirror of it. Run an equal number of participants through each of the orders in these two latin squares Order Sequential Position A B E C D 2 B C A D E 3 C D B E A 4 D E C A B 5 E A D B C D C E B A 7 E D A C B 8 A E B D C 9 B A C E D 10 C B D A E 42

43 Summary: Counterbalancing 2-3 levels: Complete counterbalancing 4-8 levels: Latin square 4+ levels: Sequence randomization, possibly with constraints 43

44 Counterbalancing w/ Multiple Exposures What if subjects experience each condition more than once? Reverse counterbalancing: ABCD-DCBA-ABCD-DCBA... Block randomization: BADC-CBAD-DCAB-ADCB... Block randomization with constraints 44

45 Within vs. Between in Developmental Psych Cross-sectional study Between groups: Test 5, 7, 9 years olds Faster than following from 5-9 Problem: Cohort effects. Longitudinal study Within groups: Follow 5 year olds until 9 years old. Takes a long time! Problem: Attrition Other methods combine the two 45

46 Summary: Confounds & Controls Participant differences: random assignment, block randomization, matching Order effects: Full & partial counterbalancing Participant bias: Blind procedures. Removal of demand characteristics Experimenter bias: Automation, double-blind procedures Floor & ceiling effects: Use procedures that are neither too difficult nor too easy. 46

47 Think Twice... Carpenter s adage: Measure twice, cut once Scientist s adage: Think twice, measure once Do not rush experimental design, there are many pitfalls to be avoided and careful design will save time in the long run. 47

48 Discussion / Questions What is the difficulty with reverse counterbalancing? What method of counterbalancing would you use for an experiment with 5 levels? 48