Social Robots and Human-Robot Interaction Ana Paiva Lecture 12. Experimental Design for HRI

Transcription

1 Social Robots and Human-Robot Interaction Ana Paiva Lecture 12. Experimental Design for HRI

2 Scenarios we are interested.. Build Social Intelligence d) e) f) Focus on the Interaction

3 Scenarios we are interested.. Build Social Intelligence d) e) f) Interactions between humans and robots are inherently present in all of robotics Focus on the Interaction

4 Scenarios we are interested.. Build Social Intelligence d) So how can we evaluate the effects of such interaction?? e) f) Interactions between humans and robots are inherently present in all of robotics Focus on the Interaction

5 Summary General aspects of designing experiments for HRI Practical and Methodological Aspects of Experimental Design in HRI Typical Metrics for HRI Some concrete examples of studies Results: statistical analysis Reporting and writing up studies in HRI

6 General Aspects on Designing Experiments for HRI Three types of studies we may conduct: 1. Study to collect data to use in the construction of a social robot; 2. Investigate the responses people give to social robots (even if the capabilities and features of such social robots do not exist yet); 3. Show that a technique built for a social robot has the desired effect in users.

7 General Aspects on Designing Experiments for HRI Three types of studies we may conduct: 1. Study to collect data to use in the construction of a social robot; 2. Investigate the responses people give to social robots (even if the capabilities and features of such social robots do not exist yet); 3. Show that a technique built for a social robot has the desired effect in users.

8 Designing an experiment: What is my Problem? Start by asking a research question! Examples: - The use of a ToM component for a robot increases the level of trust in that robot - The motion type of a robot will positively affect the participants perception of the collaboration, with legible motion being the best, and functional motion being the worst. But.What is a good question for a HRI study?

9 Experimental Design: stages in the research Our interest and motivation Bibliography on the topic Other similar research: moral dilemmas and robots Moral issues Research on Moral dilemmas No research was done investigating stealing in HRI Research question

10 Hypotheses What is a hypothesis? NOT an educated guess! Should allow to prove the impact of the techniques developed for the social robot. H2. people will (H1a) feel more negatively after the reception of a video torturing a robot compared to the normal video, and (H1b) people will show higher physiological arousal during the reception of the video torturing a robot compared to the nice video. H1. The motion type of a robot will positively affect the participants perception of the collaboration, with legible motion being the best, and functional motion being the worst Should be based on literature and theory

11 Designing an experiment : Variables To draw conclusions and answer our research questions we need to have variables which should be measured in the study Example: - Galvanic Skin Conductance can be measured in an experiment Types of variables: - Nominal (categorical)- scale with equivalent aspects given a name - Ordinal - Interval - Ratio Non parametric Parametric

12 In a study we have the Independent variables and Dependent variables: Dependent versus Independent Variables The Independent variable (the causes ) is the variable that is purposely changed. It is the manipulated variable. The Dependent variable (the effects ) changes in response to the independent variable. It is the responding variable.

13 Depends on what our research question is. Choice of the Dependent Variable What do we want to measure? - trust in a robot - speed of task execution (human and robot) - emotional reaction towards a robot - social proximity with the robot Often to measure these attitudes, we need to rely on variables that we can measure (through questionnaires, behaviour analysis, physiological measures, etc).

14 How to manipulate the Independent Variable? Examples: 1) People trust more a robot that addresses the user by his or her name ; 2) Children learning more in a simple math task together with a robot, when it has a social learning mechanism to make the robot adapt to the child. 3) Humanoid robots with a face are more friendly It s not easy. Independent Variable 1 ) Calling the user by the name 2) Social learning mechanism for personalisation and adaptation 3) Have a face or not (in a humanoid robot) Dependent Variable 1 ) Trust the robot 2) Learn more 3) Perception of friendliness

15 Set-up, Task and Materials in an Experiment Materials 1 ) ROBOT/S 2) Software used (and algorithms that are used in particular) 3) Pictures etc Set up and Task: example

16 Characteristics of a Study: the three aims of a study Reliability Validity Importance

17 Characteristics of a Study: the three aims of a study, Reliability Yielding the same or compatible results in different experiments or statistical trials How to achieve? 1. Measure the dependent variable as precisely as possible 2. Depends on the definition of the variable (use an operational definition of what we want to measure)

18 Characteristics of a Study: the three aims of a study Validity Internal validity: If the results obtain are not due to the manipulation, but rather to some other factors, then the study lacks internal validity. To get validity a good experimental design must be created. External validity: If the results obtained are not representative of humanity, but rather valid for the specific situation. Threats to internal validity: Group threats: if the experimental and control groups are different to start with (then we may be measuring those differences and not the result of the manipulation) Regression to the mean: if the participants produce extreme scores in a pretest that by chance are likely to be the score of a post-test. Time threats: when the passage of time, with events happening produce changes in behaviour of participants History: events of the participants lives may lead to results that are unrelated to the manipulation.

19 Characteristics of a Study: the three aims of a study Validity Internal validity: If the results obtain are not due to the manipulation, but rather to some other factors, then the study lacks internal validity. To get validity a good experimental design must be created. External validity: If the results obtained are not representative of humanity, but rather valid for the specific situation. Threats to external validity: Over-use of special participant groups (most HRI is tested with males, aged between 20-24, studying computer science and electrical engineering, most psychology studies are carried out with females, aged between studying psychology). Too few participants

20 Characteristics of a Study: the three aims of a study Importance/ Generality Related to external validity How do the results generalize to other populations.

21 Quasi-experimental designs One group post-test design Interaction of Human and Robot One group pre-test/post-test design Interaction of Human and Robot

22 Between-Group and Within Subjects Experimental Designs Between groups (or independent measures) Use separate groups of users (participants) for each of the different conditions (associated with the manipulation done). Each participant is tested only once. Within-subjects (or repeated measures) Participants are exposed to the different conditions of the experiment. Hybrid design (mixed) Designs that involve a combination of a between groups and within subjects variables.

23 Randomization Between groups (or independent measures) It is essential that we allocate participants randomly to our experimental conditions Within-subjects (or repeated measures) It is essential that participants done all experience our experimental conditions in the same order.

24 A post-test only/control group design Between-Groups experimental design Random allocation Group1 Group2 C1: Interaction of Human and Robot C2: Interaction of Human and Robot However, randomization may fail.

25 Between-Groups experimental design A pre-test post-test control group design Group1 C1: Interaction of Human and Robot Random allocation Group2 C2: Interaction of Human and Robot

26 Between-Groups experimental design Advantages - Simplicity - Less chance of practice and fatigue effects - useful when it is impossible for a user to participate in all experimental conditions Disadvantages - Expense in time, effort and participants - Insensitivity to experimental manipulations

27 Within-Subjects (Repeated measures) experimental design In a within-subjects, all the user pass through the interaction. It is often desirable, for many reasons: - economical: in terms of time, effort as we are using the same users - sensitivity: sometimes the differences are hard to perceive and there is random noise in the data.

28 Within-Subjects (Repeated measures) experimental design Disadvantages - Carry over effects from one condition to another - Need for the conditions to be reversible (due to the randomization of the order of the conditions)

29 Single subject Experimental Design Sometimes it is not possible to do a large experiment with a normal population. Single subject design have a long history in psychology. Single subject techniques use a different strategy as they compare the effects of an experimental manipulation to same baseline steady state in one individual. Example (ABA): State 1 Time State 2 State 3

30 Practical and Methodological Aspects of Conducting Studies in HRI

31 Planning, Legal and Safety Usually before running any study involving humans and robots that will be interacting physically, certain legal, safety and ethical issues need to be considered. Ethical Approval (many organisations require an Ethic Committee to give approval for any research done with humans); Creation of a study protocol The Ethics committee must look into: Privacy of the data (videos, photos, records must be kept private) Protection of minors and vulnerable adults Mental or emotional stress and humiliation Physical harm

32 RESEARCH QUESTIONS Study Protocol Creation: Example RQ 1: Understand how children envision a robotic assistant to integrate futuristic classrooms. PARTICIPANTS Participants will be: Children from years old Participants will be grouped in N to perform the activity. MATERIAL Drawing material (e.g., pen, pencil) One audio recorder One video camera Consent forms Robot NAO running the application

33 PROCEDURE Study Protocol Creation: Example The activity will be conducted in a Portuguese school. To participate in this study, users are required to have the consent form signed Users will be grouped in groups of 2. The activity is divided into three different phases: PHASE 1 QUANTITATIVE DATA COLLECTION PHASE 2 THE INTERACTION PHASE PHASE 3 THE REFLECTION PHASE

34 Typical Metrics/ Measures for HRI

35 Task Measures Robot task metrics Human task metrics Physiological Measures Behavioural Measures Different types of Measures Human responses to the robot (usually obtained by observing the human) Subjective Measures (self reported) Perception of the robot or task by the human, and measures by the use of concrete questionnaires

36 Task Metrics Often we want to measure the performance of the robot and the user in a task. - Performance of the robot (must be related to what the user also does). Example: navigation, or if the robot performs better when the user gave the robot certain instructions - Performance of the user (for example, how well the user did when interacting with the robot) - Performance of the team (which includes the user and the robot)

37 Example: Navigation Task Metrics: Example 1, navigation - Global navigation: The system needs to have an overall understanding of the locale in which it is working. - Local navigation: This is a finer granularity of information that is essential for smoothly moving in an area. - Obstacle encounter: Not all navigation is without problems. Breaking down the variable: effectiveness measures how well the task is completed. Potential measures include: Percentage of navigation tasks successfully completed Coverage of area Deviation from planned route Obstacles that were successfully avoided Obstacles that were not avoided, but could be overcome

38 Task Metrics: Example 2, Efficiency measures the time needed to complete the task. Efficiency Efficiency measures include: Time to complete the task User time for completing the task Average time for obstacle extraction Number of user interventions and requests per unit time. Interactions can be planned or unplanned. Ratio of user time to robot time. For example, if the User spends 5 minutes to input a navigation plan that allows the robot to successfully navigate for an hour, we have a 1:12 ratio.

39 Dragan, A. D., Bauman, S., Forlizzi, J., & Srinivasa, S. S. (2015, March). Effects of Robot Motion on Human-Robot Collaboration. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (pp ). ACM.

40 Example: Results Dragan, A. D., Bauman, S., Forlizzi, J., & Srinivasa, S. S. (2015, March). Effects of Robot Motion on Human-Robot Collaboration. In Proceedings of the Tenth Annual ACM/ IEEE International Conference on Human-Robot Interaction (pp ). ACM.

41 Physiological Measures Use of heart-rate and blood pressure Galvanic Skin response Hormone levels fmri

42 Physiological Measures Difficulty: Often the need to have expensive equipment The signal processing and analysis is sometimes non trivial The changes detected in the signals may not be a result of the experimental manipulation Difficult to draw conclusions (in terms of HRI).

43 Behavioral Measures - Include the analysis of behaviour of users (aspects such as gaze, actions, posture, etc) - Often relies on the annotation of videos (or use of tools to extract behaviours).

44 Example: analisys of students Coding Scheme and teachers behaviours GAZE STUDENT TEACHER Student looks at Task Teacher looks at Task Student looks at Teacher Teacher looks at Student Mutual gaze + Transcrip3ons + Informal verbal and non-verbal behavior analysis INTERACTION STUDEN TEACHER T Ask a question Answer a question Help Other Ask a question Provide Feedback Explain Pose Problem Give Clue Instruction Other

45 Example: analisys of students and teachers behaviours Student and teacher spent most of the 3me looking at the task. Average percentage of the interac3on SW Teacher UK SW Student UK Look at Task Look at Student Look at Task Look at Teacher

46 Example: Gaze behaviour of users interacting with Robovie

47 Self reported measures They are subjective Scales: Yes/No scale Do you consider the robot friendly? Yes/No Likert scales

48 What is Likert scale? It is a psychometric scale commonly involved in research that employs questionnaires. It consists of a statement to which the user can express different degrees of agreement. It is the most widely used approach to scaling responses in survey research. Likert scales are a non-comparative scaling technique and are one-dimensional in nature.

49 The format of a typical five-level Likert item 1. Strongly disagree 2. Disagree 3. Neither agree nor disagree 4. Agree 5. Strongly agree

50 Questionnaires Need to find validated questionnaires (but most are not) Typical questionnaires used: Godspeed ALMERE

51 Example of Typical Questionnaires: ALMERE ALMERE (Heerink et al. ): [t]he Almere model is a technology acceptance model: it can be used to predict and explain usage of a system by observing the influences on the Intention to Use this system. Dimension: Anxiety 1. If I should use the robot, I would be afraid to make mistakes with it 2. If I should use the robot, I would be afraid to break something 3. I find the robot scary 4. I find the robot intimidating

52 ATTITUDE 5. I think it s a good idea to use the robot 6. The robot would make my schoolwork more interesting 7. It s good to make use of the robot PERCEIVED ENJOYMENT 16. I enjoy the robot talking to me 17. I enjoy doing things with the robot 18. I find the robot enjoyable 19. I find the robot fascinating PERCEIVED SOCIABILITY 24. I consider the robot a nice partner to talk to 25. I find the robot nice to interact with 26. I feel the robot understands me 27. I think the robot is nice Example of Typical Questionnaires: ALMERE

53 Example of Typical Questionnaires: ALMERE PERCEIVED USEFULNESS 28. I think the robot is useful to me 29. It would be convenient for me to work with the robot in school 30. I think the robot can help me with many things in school SOCIAL INFLUENCE 31. I think the teachers would like me working with the robot 32. I think many people would like me working with the robot SOCIAL PRESENCE 33. When interacting with the robot I felt like I m talking to a real person 34. It sometimes felt as if the robot was really looking at me 35. I can imagine the robot to be a living creature 36. I often think the robot is not a real person 37. Sometimes the robot seems to have real feelings

54 Example of a Typical Questionnaire: The Godspeed questionnaire

55 Data analysis

56 Analyzing the data Depends on the type of data captured Self-reported measures: analysis of questionnaires data (statistical analysis) Behavioural measures: annotation of videos followed by statistical analysis of the data.

57 The Subjects: characterizing the population To answer general questions about the studies performed we need to look at the population that will test our robot. Ideally we would like for everyone to test it! Of course that is not possible So, we will rely on a sample that we will assume represents the population.

58 Descriptive Statistics Tell us about the population and the distribution of the results. What we need to look at is: Mean (M): the sum of all the scores divided by the number of scores; Median (Mdn): is the middle score of a distribution of scores, when they are ranked in order of magnitude. Sum of the squared errors (SS) Variance (s2): average squared deviation between the mean and the observed score: tells us on average how much a given data point differs from the mean of all data points. Standard Deviation (SD) is the square root of the variance. s = (xi x) 2 N 1

59 But We are interested not in statistics that describe the population sample but in statistics that allow us to infer things about the users in the groups (conditions) interacting with our social robots: we need inferential statistics.

60 Inferential statistics To be used with an experimental hypothesis (that is, the prediction that the experiment manipulation will have an effect) Basically we need to prove that the sample population that we observe of one condition is different from the other condition. We will calculate the probability that the results obtained are a chance result- and as this result decreases we become more confident that the experimental hypothesis is correct and the null hypothesis can be rejected.

61 Test statistics A test statistics is a statistic that has known properties (we know its frequency distribution). By knowing that, then we can compare with the results we got and calculate the systematic variation (the variation due us doing something in one condition and not in the other) and the unsystematic variation which is due to the natural differences between people in different samples (such as differences in intelligence or motivation) TestStatistic = SystematicVariance UnsystematicVariance

62 Simple Experiments staesecal analysis (between groups) Random allocation Group1 Group2 C1: Interaction of Human and Robot C2: Interaction of Human and Robot The simplest form of experiment that can be done is one with only one independent variable that is manipulated in only two ways and only one outcome is measured. More often than not the manipulation of the independent variable involves having an experimental condition and a control one. E.g., Is the interacting with NAO more engaging than interacting with BAXTER? We could use a measure of engagement (for example based on how long the users spend with the robot) and compare them. This situation can be analysed with a t-test

63 Independent T-test Independent t-test Compares two means based on independent data E.g., data from different groups of users Significance teseng TesEng the significance of Pearson s correla.on coefficient

64 Simple Experiments staesecal analysis (Repeated measures) C1: Interaction of Human and Robot C2: Interaction of Human and Robot Dependent t-test Compares two means based on related data. E.g., Data from the same users measured at different Emes (people interact with both robots). Significance teseng TesEng the significance of Pearson s correla.on coefficient

65 Rationale to Experiments Group 1 Group 2 NAO Engagement BAXTER Variance created by our manipulaeon InteracEon with NAO or with BAXTER Variance created by unknown factors E.g. Differences in ability (unsystemaec variance) Slide 65

66 RaEonal for the t-test Two samples of data are collected and the sample means calculated. These means might differ by either a liwle or a lot. If the samples come from the same populaeon, then we expect their means to be roughly equal. Although it is possible for their means to differ by chance alone, we would expect large differences between sample means to occur very infrequently.

67 RaEonal for the t-test We compare the difference between the sample means that we collected to the difference between the sample means that we would expect to obtain if there were no effect (i.e. if the null hypothesis were true).

68 RaEonal for the t-test As the observed difference between the sample means gets larger, the more confident we become that the null hypothesis should be rejected. If the null hypothesis is incorrect, then we gain confidence that the two sample means differ because of the different experimental manipulaeon imposed on each sample.

69 RaEonale to the t-test t = observed difference between sample means expected difference between population means (if null hypothesis is true) estimate of the standard error of the difference between two sample means

70 AssumpEons of the t-test Both the independent t-test and the dependent t-test are parametric tests based on the normal distribueon. Therefore, they assume: The sampling distribueon is normally distributed. In the dependent t--test this means that the sampling distribueon of the differences between scores should be normal, not the scores themselves. Data are measured at least at the interval level. The independent t-test, because it is used to test different groups of people, also assumes: Variances in these populaeons are roughly equal (homogeneity of variance). Scores in different treatment condieons are independent (because they come from different people).

71 Other tests The t-test is limited to situations where there only two levels of the independent variable. However, often we have experiments (given the types of experiments with our robots) that have three or four levels of the independent variable. Other tests are ANOVA: analysis of variance, which produces a test statistic that compares the amount of systematic variance in the data.

72 A case

73 Research Question People feel more close with others if they share emotional states So, if a robot shares its emotional states with the user, what can happen?

74 Material FAtiMA Emotional Agent Architecture; Thalamus Framework; Unity3D

75 Material: The FAtiMA Agent Architecture for Emotional Robots

76 Set-up and Task Emys played the game Coins and Strings against a participant while expressing some social behaviours. 5 Board Games

77 Variables Independent Variable: Sharing or not sharing the emotions Dependent Variable: Perception of being human-like

78 The study C1 No Sharing Condition Behaviours: Gaze tracking using Microsoft Kinect; Emotion expression actions; Small Talk. C2 Sharing Condition Behaviours: Gaze tracking using Microsoft Kinect; Emotion expression actions; Small Talk; Emotional sharing.

79 Hypothesis Participants allocated in the Sharing Condition will perceive the robot as more humanlike (less artificial), closer to them and with more friendlier characteristics than the ones allocated in the No Sharing Condition.

80 Experimental Design Between groups (independent measures) design experiment. Each subject only interacted with one condition.

81 Study: sample 30 university students, with ages ranging from 19 to 30 years old (M=23.4; SD=2.99). The sessions had the duration of approximately 20 minutes per participant (with 10 minutes of interaction with Emys).

82 Measures The Godspeed Questionnaire (Anthropomorphism; Animacy; Likeability and Perceived Intelligence); Connection Questionnaire adapted from a consumer product attachment scale; Example: I feel very close to Emys McGill Friendship Questionnaire. Example: If I was worried, Emys would make me feel better

83 Results Godspeed Questionnaire There were no significant differences between conditions for each dimension. But some items reported significant differences.

84 Results Anthropomorphism Dimension: Unconscious/ C o n s c i o u s i t e m (U=58, p=.010) Perceived as more conscious in the No Sharing Condition

85 Results Anthropomorphism Dimension: Artificial/Lifelike (U=65, p=.042) Perceived as slighlty less artificial in the No Sharing Condition

86 RESULTS Likeability Dimension: Awful/Nice (U=67.5, p=.05) Perceived as nicer in the No Sharing Condition

87 Results Perceived Intelligence Dimension No significant differences but means show across conditions that participants perceived Emys as very competent (M=4.40; M=4.47), knowledgeable (M=3.87; M=4.13) and intelligent (M=4.13; M=4.33). Only 4 out of 30 participants were able to beat Emys in the game.

88 Discussion