Sampling I214 21 Oct 2008
Why the need to understand sampling? To be able to read and use intelligently information collected by others: Marketing research Large surveys, like the Pew Internet and American Life Project, or government studies, trade groups data
From USA Today: Yet for all the headaches associated with cell phones, who's willing to go without? The number of wireless subscribers in the USA hit 180.5 million at the end of last year, up from 158.8 million the year before, according to CTIA, the wireless trade association.
Why the need for UX researcheers to understand sampling? To be able to read and use intelligently information collected by others: Marketing research Large surveys, like the Pew Internet and American Life Project, government studies, trade groups data To understand the limits of even data collected using good sampling methods To understand: How we may try to approximate good sampling The further limits of our studies and data
Sampling in usability: when do we sample?
Sampling in usability: when do we sample? When we can t observe, measure everything When we don t need to observe everything just some Some examples: Usability testing Interviews Observation Surveys
Sampling in usability: what do we sample?
Sampling in usability: what do we sample? People (users, potential users, members of target populations of various sorts) Events (transactions, occurrences of various activities) Things (documents, images, records)] Time periods (minutes, hours, days, weeks)
Representative samples Want the sample to be roughly proportional to the population in terms of groups/characteristics that matter Exception: oversampling small groups Which characteristics matter?
Kinds of samples Probability samples random selection Simple Random Sampling Principle: every unit has an equal chance of being sampled Stratified Random Sampling Systematic Random Sampling Cluster (Area) Random Sampling Multi-Stage Sampling Non-probability not random selection Quota samples Convenience samples Purposive samples Snowballing
Kinds of samples public opinion polls Probability samples random selection Simple Random Sampling Principle: every unit has an equal chance of being sampled Stratified Random Sampling Systematic Random Sampling Cluster (Area) Random Sampling Multi-Stage Sampling Non-probability not random selection Quota samples Convenience samples Purposive samples Snowballing
Kinds of samples user experience research Probability samples random selection Simple Random Sampling Principle: every unit has an equal chance of being sampled Stratified Random Sampling Systematic Random Sampling Cluster (Area) Random Sampling Multi-Stage Sampling Non-probability not random selection Quota samples Convenience samples Purposive samples Snowballing
Sampling terminology Unit of analysis: the unit about which info is collected; E.g., people in the US; transactions Sampling unit: the unit selected as part of the sample, from which information is collected, e.g. households. Respondent or reporting unit: unit (person, organization) providing the information, e.g., any household member 14 or older who answers the phone. Sampling frame: a list of sampling units from which a sample may be drawn, e.g. a list of all households... Survey or study population: aggregate of elements from which the sample is actually selected. Households in US etc. with phones (if a telephone survey)... on October 10, 2007. Population: theoretically specified aggregation of survey elements; The complete group of units to which survey results are to apply. I.e., all US residents (legal and otherwise, temporary and permanent) in the US over age x.
Sampling terminology Unit of analysis: the unit about which info is collected person who is likely to vote in November election Registered voter; self-reported plan to vote; voted in last election Sampling unit: the unit selected as part of the sample, from which information is collected: registered voters; households. Respondent or reporting unit: unit (person, organization) providing the information; the voter; any household member 14 or older who answers the phone. Sampling frame: a list of sampling units from which sample drawn: list of registered voters with phone #s; list of phone #s Survey or study population: aggregate of elements from which the sample is actually selected: registered voters as of X date for Y area with listed phone #s; random digit dialing Population: theoretically specified aggregation of survey elements; The complete group of units to which survey results are to apply: voters in November election
Operationalizing the sampling methods What info do you need? What info can you get and how? From whom can you get the information you need? Non-users are hard to reach Who will answer the phone? Can t ask 10-year-olds what their parents know; whom can you ask about others? Who/what can you reach? Likely voters reachable by phone Website users who agree to a short survey Do you have a sampling frame? Households vs individuals; listed phone #s Registered users
CNN/Opinion Research Corporation Poll. Oct. 15, 2008 N=620 adults nationwide who watched the presidential debate. Survey respondents were first interviewed as part of a random national sample on October 13-14, 2008 respondents indicated they planned to watch tonight's debate and were willing to be re-interviewed after the debate. Some questions were asked of each respondent both in the pre-debate questionnaire on October 13-14 and on tonight's questionnaire. 30% of the respondents who participated in tonight's survey identified themselves as Republicans, 40% identified themselves as Democrats, and 29% identified themselves as independents."
Sample Size Statistically valid sample: Formulas for sample sizes are based on probability samples from very large populations Size depends on expected results and desired methods of analysis, incl breakdown x groups Guiding principle: how large an error (how imprecise an estimate) can you live with? More likely in usability: Convenience sample: As many as possible (over a limited period of time) As varied as possible Representation from desired groups Keep going till things don t change But need to understand how this may affect your results
Sample size 95% C. I. = M ± (1.96 * SE) SE = SD / n where SD is the standard deviation of our sample and n is the number of cases. That is, the standard error is estimated by the dividing the obtained standard deviation by the square root of the number of cases. The standard error becomes smaller as the size of the sample increases. As we increase our sample size the standard error and hence the confidence interval becomes smaller. In other words, we can detect smaller differences between means if we have larger sample sizes. We can increase the power to detect any difference by increasing the sample size. http://web.uccs.edu/lbecker/spss/confintervals.htm#4.%20confidence%20interval%20of%20a%20mean
CNN/Opinion Research Corporation Poll. Oct. 15, 2008. N=620 MoE ± 4. "Regardless of which candidate you happen to support, who do you think did the best job in the debate: Barack Obama or John McCain?" Obama McCain Neither Both Unsure 58% 31% 2% 8% 1% 58±4 31±4 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 27 ---- 35 McCain 54 ---- 62 Obama
But what if.. N=620 MoE ± 4. "Regardless of which candidate you happen to support, who do you think did the best job in the debate: Barack Obama or John McCain? Obama 53% 47% McCain 53±4 47±4 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 43 -- 51 McCain 49 -- 57 Obama
Los Angeles Times/Bloomberg Poll. Oct. 10-13, 2008. N=1,446 registered voters nationwide. MoE ± 3 (for all registered voters).
http://www.measuringusability.com/wald.htm http://www.surveysystem.com/sscalc.htm
Needs, usability, and sampling Requirements specification Convenience sample of current users Purposive sample of employees, users Quota sample E.g., x from each location, department Prototype evaluation Questionnaire as a way of getting consistent data from test population probably in entirety; but could be any of the above User feedback User surveys; comments solicitations