Starting Experimental Design

Transcription

1 Starting Experimental Design Exam 3 will emphasize Experimental Design. Design is the plan for manipulating Independent Variables and analyzing the data. Design determines what you cam learn from your data. One design-option is manipulating your independent variable within subjects rather than between subjects (today s concept). Other options include using more than one level of the IV, and more than one IV (two important concepts to come). Changing how the IV is manipulated Within-subject designs In the standard design you have learned, the IV is manipulated between subjects different subjects receive different treatments (levels of the IV). Anything unique to persons is error (as you know!). Consider an IV people are tasting black tea versus green tea. Assume people taste the IV and give a rating of how they like it (the DV). Some people might hate tea in general, rating any tea low, whereas others might love tea, rating any tea fairly high. In the between-subject design, each person gives one rating. The problem is that a rating may be high or low because the person loves/hates tea in general (rather than because the tea is green or black). Personal differences in general taste become error! We can eliminate such error by using smart design. In a within-subject design, every subject receives both treatment levels each person tastes green tea and black tea. Thus, there would be two ratings for each person. A person might rate both teas fairly high, or both low. But in the within-subject design, this is not error. The new type of error is how the ratings change within the person. Do most people like one tea a little better? General personal differences do not count as error! This type of design is crucial in many situations in which there are strong differences between people. Examples include research on taste (which tea, which cola, which beer?), research on preferences (which color scheme is relaxing? which politician do you like better?); in research on vision or sensation (some people see better or worse, e.g.), and in research on abilities (e.g., some people are better at learning stats, or better at comprehending language, or more socially skilled). In each case, the within-subject design can be a more sensitive way to examine behavior. We can rephrase the terms within-subject and between-subject with more general categories: Between subject design is an example of independent samples each sample in your experiment is independent of the other sample (e.g., different people). Within subject design is an example of dependent samples there is a relation between the people in the two samples (i.e., they are the same person!). In more advanced design, other relations are also used (e.g., matching based on taste or ability). See pages 85-6 of SFS for reading on this! An example with numbers follows.

2 Independent vs. dependent samples L1 Ho Dependent t example.xls Note: Examples use the SAME scores; the difference is design how the scores are arranged (IV manipulated). A) Standard case (you've learned this) -- Independent Samples t (Each group & score independent of others) There are 4 subjects and 4 scores: Sample 1 (Exper.) Sample 2 (Control) ERROR ERROR Score Mean Devtn. Square Score Mean Devtn. Square 1) ) ) ) Mean = 3.5 Sum = 4.5 Mean 5 Sum = 8 SD = SD2 = V 4.5 SD = SD2 = Var = 8 SEdiff= t = B) Dependent samples t -- each subject provides *2* scores There are now 2 subjects and 4 scores. This is a new version of the t-test Differences (D) ERROR Sample (IV Effect: (new) Score 1 - Score 2) Mean Deviation from Score1 Score2 Difference Difference the Mean Diff 1) ) Mean = 3.5 Mean = 5 MeanD = -1.5 SD = t = Works well if pairings (matching variable) related to DV scores SEdiff = SD/sqrt(n) = 0.5 C) Dependent samples t -- same as B) EXCEPT LOW CONSISTENCY WITHIN INDIVIDUALS Sample Differences (D) ERROR Score1 Score2 Difference Mean D Dev from D 1) ) MeanD = -1.5 SD = 4.95 t = NOTE: There is now a larger SEdiff than in example B, because the IV Effect is changing so much from person to person (I.e., differences of 2 and -5) SEdiff = SD/sqrt(n) = 3.5

3 Changing How the IV is Manipulated II Muli-level Designs and ANOVA Design is the plan for manipulating variables and analyzing the data. Today s distinction involves how many levels of the IV you use the number of different treatment conditions. Imagine this research problem. You want to compare the effectiveness of different types of vacations for producing relaxation. Your subjects have just completed Psychological Statistics, and they need a vacation. You decide to manipulate location of the vacation. You will use a between-subject design for good reasons (one level of vacation could influence responses to other levels!). You will measure ratings of relaxation after the trip (15=best, 1=worst). Consider first a familiar design with 2 levels of the IV (no different from the main t-test design). There are two levels of the IV: One group stays in Tampa Bay (control) while another group goes to Paris (experimental). You pay each subject their standard salary during the experiment. Although this is a reasonable design, you do not get much information from it. You could get more information from a multi-level design. This design has more than 2 levels of the IV. For example, you could have 4 levels of vacation: Tampa Bay (control), Miami Beach, London, and Paris. Each new level provides additional information about vacations. A multi-level design requires a new type of analysis, called Analysis of Variance. A multi-level ANOVA tells you if there are differences anywhere in your experiment. If there are differences, additional statistics would be used to pinpoint where the differences might be. Examples of multi-level results follow.

4 Interpreting Multilevel Results (one IV with > 2 levels) 1. After significant F, the important differences could be anywhere. E.g., vacation experiment; DV = happiness rating (7 best) Level 1 Level 2 Level 3 a) Tampa $50/day Paris $50/day Paris $200/day (control) (exper.) (more experimental) b) Tampa $50/day Paris $50/day Paris $200/day (control) (exper.) (more experimental) c) Tampa $50/day Paris $50/day Paris $200/day (control) (exper.) (more experimental) (Further stats needed to pinpoint difference; they can be "planned," or "post hoc") 2. Multilevel designs are great for testing for "trends", or functions. E.g., Four level design, testing a new pain reliever -- Dosage = IV, which dosage is most effective? DV = relief from pain (7 best) Level 1 Level 2 Level 3 Level 4 a) 0 mg (placebo) 25 mg 50 mg 75 mg (linear) b) Level 1 Level 2 Level 3 Level 4 0 mg (placebo) 25 mg 50 mg 75 mg (step) c) Level 1 Level 2 Level 3 Level 4 0 mg (placebo) 25 mg 50 mg 75 mg (quadratic) (c is less likely but happens; it is aka "inverted U" function)

5 Sanocki/Research Methods L3 One-Way Multi-Level ANOVA's.xls One-Way, Multi-Level ANOVA -- Example Problem 1 -- Vacation Experiment 1) The data: DATA: One IV: IVA A1 Stay home A2 Home w/ $$ A3 Paris w/ $$ DV = "Relaxation Score" on last day (1 15 best) 3 Levels of A> A1 A2 A3 1) 3 4) 12 7) 11 2) 4 5) 8 8) 7 3) 2 6) 13 9) 12 3 Means> Means Grand Mean> Grand Mean = 8 STEP 2) Dev. Table MODEL: Total = IV Effect + Error (X - X ) = (X - X ) + (X - X ) DEVIATION TABLE: TOTAL IV (BETWEEN) ERROR (WITHIN) LEVEL A1 check: Scores Total Total Total IV IV IV Error Error Error t iv+e Dev Dev Square Dev Dev Square Dev Dev Square 3 1) ) ) LEVEL A2 12 4) ) ) LEVEL A3 11 7) ) ) SUMS OF SQUARES CHECK 144 = 144 STEP 3) Source table (aka Summary Table) SOURCE TABLE: Source SS df MS F IV >> IV/Between Err >> Error/Within Check: Total Fcrit(2,6) = 5.14 Last, COMPARE F you've obtained to F critical

6 Sanocki/Research Methods L3 One-Way Multi-Level ANOVA's.xls Do this!! EXAMPLE 3 -- NEW Vacation Experiment (answers other side) A1 Stay home A2 Home w/ $$ A3 Paris w/ $$ DV = "Relaxation Score" on last day DATA: One IV: IVA A1 A2 A3 1) 3 4) 4 7) 11 2) 4 5) 5 8) 7 3) 2 6) 6 9) 12 Means Grand Mean = MODEL: Total = IV Effect + Error (X - X ) = (X - X ) + (X - X ) DEVIATION TABLE: TOTAL IV (BETWEEN) ERROR (WITHIN) LEVEL A1 check: Scores Total Total Total IV IV IV Error Error Error t iv+e Dev Dev Square Dev Dev Square Dev Dev Square 3 1) ) ) 0 0 LEVEL A2 4 4) ) ) 0 0 LEVEL A3 11 7) ) ) 0 0 SUMS OF SQUARES CHECK SOURCE TABLE: Source SS df MS F IV >> IV/Between 2 Err >> Error/Within 6 Check: Total 8 Fcrit(2,6) = 5.14

7 Sanocki/Research Methods L3 One-Way Multi-Level ANOVA's.xls ANSWERS (DO OTHER SIDE FIRST!! then check here) A1 Stay home A2 Home w/ $$ A3 Paris w/ $$ DV = "Relaxation Score" on last day DATA: One IV: IVA A1 A2 A3 1) 3 4) 4 7) 11 2) 4 5) 5 8) 7 3) 2 6) 6 9) 12 Means Grand Mean = 6 MODEL: Total = IV Effect + Error (X - X ) = (X - X ) + (X - X ) DEVIATION TABLE: TOTAL IV (BETWEEN) ERROR (WITHIN) LEVEL A1 check: Scores Total Total Total IV IV IV Error Error Error t iv+e Dev Dev Square Dev Dev Square Dev Dev Square 3 1) ) ) LEVEL A2 4 4) ) ) LEVEL A3 11 7) ) ) SUMS OF SQUARES CHECK 96 = 96 SOURCE TABLE: Source SS df MS F IV >> IV/Between Err >> Error/Within Check: Total 96 8 Fcrit(2,6) = 5.14

8 Sanocki/Research Methods L3 One-Way Multi-Level ANOVA's.xls More about the source table: Summarize in the source table to obtain F-value SOURCE TABLE: Source SS df MS F IV >> IV/Between 2 Err >> Error/Within 6 Check: Total 8 Fcrit(2,6) = 5.14 REMEMBER: What does a significant F tell us? 1) Result rare by chance, 2) In multi-level, there is a difference "somewhere" -- need further tests to establish exactly where

9 Factorial ANOVA Overview More than one IV (2 or more), with all levels combined (factorial) Simplest is 2 X 2 (our focus) Produces more information, including information about each IV (main effects) and information about their interaction Information in 2 X 2: (4 cell means) 2 Main effects 1 Interaction effect 3 statistical decisions in 2 x 2 ANOVA Model for 2 x 2 More complex designs possible (the beauty and mystery of ANOVA) Additional factors and levels (beyond 2), more interactions

10 Factorial Design Example (START HERE) 2 X 2 Example. Some of the vacation designs we looked at previously need clean-up. They used two factors, Place and Money, and mixed them up. It is better to look at each factor separately in a Factorial Design. Moreover, the factorial design gives us something very special it tests for synergy between the two factors. For example, the effect of traveling can change, depending on how much money one has to spend. This is an interaction. Let s say that London produced the most relaxation in the multi-level designs. We can factorially combine London (location) and money: 2 X 2 Factorial Design IV B: Money (per Day) $10 B 1 IV A: Location Tampa Bay A 1 London A 2 $160 B 2 (Table repeated) IV B: Money (per Day) $10 B 1 IV A: Location Tampa Bay A 1 London A 2 $160 B 2

11 Sanocki/Research Methods Two-Way Analysis of Variance -- Example Problem L6 2X2 Analysis Overview.xls DESIGN (2 X 2 FACTORIAL): OVERALL VIEW: CONDITIONS AND CELL MEANS The 2X2 Produces Three Independent Effects: key: IVA A1 A2 Main effect of A B1 A1B1 A2B1 Main effect of B IVB Interaction B2 A1B2 A2B2 Model used in Deviation Table (to get sums of squares) Cell A1 A2 Means: B1 5 3 IVB (x - X ) = (X - X ) + (X - X ) + (X - X - X + X ) + ( x - X ) B DATA, WITH 2 SUBJECTS PER CELL / CONDITION IVA Summarize data with source table: 3 sources of IV Effects + error: A1 A2 B1 1) 4 5) 4 B Main Obtain 2) 6 6) 2 Eff. Means SOURCE TABLE: 3 F's Cell Mns Source SS df MS F IVB IVA < A B2 3) 6 7) 14 4) 8 8) 16 IVB < B Cell Mns AXB < AxB A Main 6 9 Eff. Means Grand Mean > 7.5 ERROR TOTAL Compare obtained F's to F-critical from F-table

12 Sanocki/Research Methods L6 2X2 Analysis Overview.xls DATA VIEW WITH 2 SUBJECTS PER CONDITION IVB IVA A1 A2 B1 1) 4 5) 4 B Main 2) 6 6) 2 Eff. Means Cell Mns B2 3) 6 7) 14 4) 8 8) 16 Cell Mns A Main 6 9 Eff. Means Grand Mean 7.5 DEVIATION TABLE: CELL A1B1 TOTAL IVA (BETWEEN) IVB (BETWEEN) A X B INTERACTION ERROR (WITHIN) Total Total Total IVA IVA IVA IVB IVB IVB A X B A X B A X B Error Error Error Dev Dev Square Dev Dev Square Dev Dev Square Dev Dev Square Dev Dev Square 4 1) ) CELL A1B2 6 3) ) CELL A2B1 4 5) ) CELL A2B2 14 7) ) SUMS OF SQUARES CHECK 174 =

13 Sanocki/Research Methods L6 2X2 Analysis Overview.xls DATA VIEW WITH 2 SUBJECTS PER CONDITION IVB IVA A1 A2 B1 1) 4 5) 4 B Main 2) 6 6) 2 Eff. Means Cell Mns B2 3) 6 7) 14 4) 8 8) 16 Cell Mns A Main 6 9 Eff. Means Grand Mean 7.5 DEVIATION TABLE (fill in the rest): CELL A1B1 TOTAL IVA (BETWEEN) IVB (BETWEEN) A X B INTERACTION ERROR (WITHIN) Total Total Total IVA IVA IVA IVB IVB IVB A X B A X B A X B Error Error Error Dev Dev Square Dev Dev Square Dev Dev Square Dev Dev Square Dev Dev Square 4 1) ) CELL A1B2 6 3) ) CELL A2B1 4 5) ) 14 7) 16 8) CELL A2B2 SUMS OF SQUARES CHECK 174 =

14 Psych Stats 2 X 2 Interpretation Exercise: For each of 12 datasets, determine main effects and interaction (yes, no); check answers on other side 1 IVB1 IVB2 4 IVB1 IVB2 IVA IVA IVA IVA IVB1 IVB2 5 IVB1 IVB2 IVA IVA IVA IVA IVB1 IVB2 6 IVB1 IVB2 IVA IVA IVA IVA IVA1 IVA IVA1 IVA IVB1 IVB2 0 IVB1 IVB IVA1 IVA IVA1 IVA IVB1 IVB2 0 IVB1 IVB IVA1 IVA IVA1 IVA IVB1 IVB2 0 IVB1 IVB2

15 ANSWERS 1 Main Eff A? yes 4 Main Eff A? yes Main Eff B? yes Main Eff B? yes AxB Interactyes AxB Interactyes 2 Main Eff A? yes 5 Main Eff A? yes Main Eff B? NO Main Eff B? yes AxB InteractNO AxB InteractNO 3 Main Eff A? NO 6 Main Eff A? yes Main Eff B? NO Main Eff B? NO AxB Interactyes! AxB Interactyes 7 Main Eff A? yes 10 Main Eff A? yes Main Eff B? NO Main Eff B? yes AxB InteractNO AxB InteractNO 8 Main Eff A? yes 11 Main Eff A? yes Main Eff B? yes Main Eff B? NO AxB Interactyes AxB Interactyes 9 Main Eff A? NO 12 Main Eff A? yes Main Eff B? NO Main Eff B? yes AxB Interactyes! AxB Interactyes ^ tricky! (Classic crossover interaction)