A Data Collection of Infants Visual, Physical, and Behavioral Reactions to a Small Humanoid Robot

Transcription

1 A Data Collection of Infants Visual, Physical, and Behavioral Reactions to a Small Humanoid Robot Rebecca Funke 1, Naomi T. Fitter 1, Joyce T. de Armendi 2, Nina S. Bradley 2, Barbara Sargent 2, Maja J. Mataric 1, and Beth A. Smith 2 Abstract Exploratory movements during infancy help typically developing infants learn the connections between their own actions and desired outcomes. In contrast, infants who are at risk for developmental delays often have neuromotor impairments that negatively influence their motivation for movement. The goal of this work is to expand our understanding of infant responses to non-contact interactions with a small humanoid robot. In the initial work presented here, we focus on understanding how this type of robotic system might help to encourage typically developing infant motor exploration. A data collection with N = 9 infants compared infant reactions to four robot conditions: saying yay with arm movement, saying kick with leg movement, saying yay with no movement, and saying kick with no movement. The results indicate that infants visually gazed at the robot while it moved, looking specifically to the part of the robot that was moving. Infants tended to move more during periods of robot inactivity. When the robot was moving, the infants also seemed more alert. Overall, these results can inform future studies of how to develop interventions to encourage movement practice by typically developing and at-risk infants. I. INTRODUCTION Typically developing infants engage in exploratory movements that help them learn how their own actions are connected to different outcomes, from making a caregiver smile to grasping a favorite toy. Through this perceptionaction learning process, infants learn to control their bodies and interact with the environment. In contrast to typically developing (TD) infants, infants at risk (AR) for developmental delays often have neuromotor impairments involving strength, proprioception, and coordination. These challenges can lead to greater difficulty with movement and decreased motivation for motor babbling. The goal of this work is to expand our understanding of infant responses to noncontact interactions with a small humanoid robot. Initially, these efforts focus on how this type of robotic system might help to encourage TD infant movement. This foundational work also will equip us with appropriate strategies for future interventions with the more vulnerable population of AR infants. A recent estimate determined that approximately 9% of all infants in the United States are AR and could benefit from early intervention services to address motor, cognitive, and/or 1 Interaction Lab, Department of Computer Science, University of Southern California, Los Angeles, CA 90089, USA {rfunke, nfitter, mataric}@usc.edu 2 Division of Biokinesiology and Physical Therapy, University of Southern California, Los Angeles, CA 90089, USA {jdearmen, nbradley, bsargent, beth.smith}@usc.edu social development [1]. Because of connections between these different types of development, motor, cognitive, and social domains can all be positively impacted by an intervention in just one of these areas [2]. The current standard of care for early intervention practice is to provide infrequent, low-intensity movement therapy or no intervention in infancy [3], [4]. However, early, intense, and targeted therapy intervention has the potential to improve neurodevelopmental structure and function [5]. Despite this potential gain, it can be challenging to find feasible and resource-efficient ways to deliver this type of intervention to AR infants. Noncontact infant-robot interactions that provide demonstrations and feedback are one such possible solution for encouraging infant motor exploration. Before working with AR infants, we first aim to understand TD infant responses to non-contact interactions with a humanoid robot. This paper summarizes related work (Section II), describes our methods for conducting a data collection on TD infant interactions with a small humanoid robot (Section III), presents the results of the data collection (Section IV), and discusses our findings (Section V). II. RELATED WORK Overall, the work presented in this paper aims to explore how TD infants respond to non-contact interactions with a small humanoid robot. Because of the humanoid form of the robot, one goal of this work is to determine if infants might be inclined to imitate the robot s motions. Related literature also suggests that robot motion might capture the attention of infants and provide contingent rewards for motion exploration. The related work discussed in the following sections motivates the design of the data collection, demonstrates the potential of socially assistive robots, and explains possible future applications of this work. A. Infant Motor Learning and Adaptation Infants acquire motor skills through a dynamic process of exploration and discovery during which the spontaneous movements of infancy modulate into task-specific actions such as reaching, crawling, and walking [6], [7]. The process by which task-specific action emerges from spontaneous movement is a fundamental topic of study in infant development. One previously studied paradigm of infant behavior is infant replication of demonstrated actions. This replication qualifies as imitation if the infant repeats the action more with the demonstration than without the demonstration [8].

2 Although some debate exists on when and why imitation behavior emerges in humans [9], several previous works suggest that infants use imitation as one mechanism for the acquisition of new behaviors, skills, and actions [10], [11], [12]. It is not yet known if infants perceive human actions differently from humanoid robot actions resembling human behavior. Infants appear more likely to initiate certain actions toward objects if they are first modeled by humans. For example, in one object-directed action study, six-month-old infants who observed a researcher removing and replacing a mitten on a puppet were more likely to perform this action than infants who did not observe this behavior [10]. On the other hand, infants appear unlikely to imitate a select objectdirected action if it is performed by one object on another object. For example, in a study of nine-month-old infants, participants who watched a claw grasping and moving an object did not imitate this action, but infants who watched a human demonstration did imitate this action [13]. In this initial data collection, we aim to understand infant responses to robot sound and motion. After we understand the infant imitation tendencies in this scenario, we may also be able to design appropriate robot-based feedback. Contingent feedback is one technique used to study the emergence of task-specific learning from spontaneous movement. A mobile paradigm is one example of contingent feedback where specific arm or leg movements of an infant are reinforced with sound and motion from an overhead mobile [14]. Historically, this technique was used to study learning and memory in early infancy [15], but we may be able to leverage this paradigm to reward and encourage particular types of infant exploratory motion. B. Robot Design for Intervention Human-robot interaction is a rapidly expanding research area. Robots have demonstrated capabilities to assist people in applications ranging from socially-motivated physical therapy for stroke survivors [16] to behavioral therapy for children with autism spectrum disorder [17]. Robots present opportunities to assist people as broadly replicable platforms for effective, personalized, and socially engaging intervention. Past work has also shown that physically present robots can persuade and motivate people more than on-screen agents do [18], [19]. Behavioral cues produced by objects having humanoid morphological features (e.g., a face and eyes) appear to be salient stimuli for motivating infant action [20]. Abstract behavioral cues such as biological motion [21] and selfpropulsion [22] trigger infants innate detection of objectdirected actions. These past results, in combination with the knowledge that infants of a certain age tend to imitate nearby people, led to our decision to use a humanoid robot for this work. This research aims to leverage a small humanoid robot s embodiment to engage infants in imitating specific physical movements. Personalized models appropriate for each infant participant will be developed based on findings in this initial work. C. Infant-Robot Intervention Motion demonstrations from a humanoid robot have several unique advantages for studying infant motion adaptation. Since infants attend preferentially to faces [23], interactive humanoid robots may capture and maintain the attention of infants more than inanimate toys. Past work has shown that children from six to fourteen months old attend to a humanoid robot for longer than an android or a person [24]. Furthermore, small humanoid robots can produce motions similar to those of infants. This ability may help the robot to inspire infants to imitate desired patterns of motion. Since each infant is unique in their development, interactions should be personalized to each user. The benefits of personalized one-on-one instruction are well supported in psychology research [25], and these results have been replicated in human-computer interaction work [26] and human-robot interaction studies [19]. The data collected by our infant-robot system will enable us to achieve real-time personalization and adaptation of robot behaviors. The infants studied in this data collection can help us to understand how infant responses and behaviors might vary across system user. Post-hoc analyses of these infants behaviors will help us to equip future iterations of the infant-robot system with appropriate personalization and adaptation capabilities. III. METHODS To better understand infant responses to a small humanoid robot, we conducted an exploratory data collection. We brought a robot and an infant together in a laboratory setting, varied the robot s actions, and tracked the following aspects of the infant s state: visual responses (i.e., eye gaze), physical responses (i.e., motion measured by inertial sensors), and behavioral responses (i.e., infant alertness level). For this work, we chose to use the Aldebaran Nao robot shown in Fig. 1 because of its humanoid form. During the data collection, each infant sat across from the Nao robot in Fig. 1. The Nao robot used in this data collection.

3 TABLE I INFORMATION ABOUT THE DEMOGRAPHICS AND OPENING ASSESSMENT MEASUREMENTS FOR THE SIX INFANTS WHO SUCCESSFULLY COMPLETED THE EXPERIMENT. Infant Gender Age Gestation Age Healthy AIMS AIMS Head Weight (kg) Height (cm) (months, days) at Birth (weeks) at Delivery? Score Percentile Circ. (cm) TD2 M 4m 13d 38 No TD3 M 4m 29d 41 Yes TD4 M 4m 24d 40 Yes TD6 M 4m 26d 40 Yes TD7 F 5m 11d 40 Yes TD9 M 2m 28d 40 Yes Fig. 2. The experimental setup in which an infant observes and reacts to a Nao robot. The labeled sensors capture information about the infant-robot interaction for post-hoc analyses. A camcorder and Kinect behind the robot (not shown in image) provide additional infant behavior data. a small white room. Participants interacted with the robot as described in the following sections. The data collection procedure was approved by the University of Southern California Institutional Review Board under protocol #HS A. Participants We recruited nine TD infants between the ages of 2 months 28 days and 5 months 11 days. The infants were recruited from the Greater Los Angeles Area. The data presented in this paper excludes three infants: two were excluded due to excessive crying (nonstop for more than two minutes) and one was excluded for having an Alberta Infant Motor Scales (AIMS) score below the tenth percentile. Table I displays the age, size, and development information for each infant included in the data analysis. Each family received $20 of compensation for participating in the data collection. B. Procedure When the infant first entered the experiment space, the child s parent or legal guardian received a written overview and verbal explanation of the procedures. This caregiver signed an informed consent form prior to their infant s participation. A researcher administered the AIMS assessment to quantify infant motor development status [27] and measured the weight, length, and head circumference of the infant. We affixed one Opal inertial movement sensor [28] on each infant leg and arm using custom-made leg warmers with pockets. Past work has validated that these sensors can accurately record the quantity of infant limb movements [29]. The infant also wore a head-mounted eye tracker that allowed us to analyze their visual gaze. Infants came to the lab for a single session that lasted approximately one hour. At the start of the infant-robot interaction, the robot remained still for ten seconds while baseline visual, physical, and behavioral measurements were recorded from the infant. Figure 2 illustrates the setup of this interaction. For the next eight minutes, the infant engaged with the Nao robot in the procedure described by Fig. 3. The caregiver was seated next to the infant in the experiment setup, but they were asked to refrain from socially interacting with the infant during the data collection procedure. C. Conditions All infants experienced four interaction conditions during which the Nao robot behaved in each of the following ways: 1) raising arms and saying yay 2) kicking legs and saying kick 3) stationary and saying yay 4) stationary and saying kick The LED lights in the robot s eyes flashed during each robot behavior (movement and/or speech). The button on the robot s chest was dimly lit throughout the full eightminute experiment procedure. In this initial data collection, the robot s actions were preprogrammed and not contingent on the infant s actions so that we might gain some initial understanding of how infants respond to robot activity. As illustrated in Fig. 3, during conditions 1, 3, and 4, robot behavior lasted six seconds and inactivity lasted nine seconds per interaction cycle. During condition 2, robot behavior lasted seven seconds and inactivity lasted eight seconds. Thus, the duration of a full cycle of the active and inactive state together for any given condition was fifteen seconds; this sequence was repeated four times to make up each minute-long interaction condition trial. The experiment was composed of two four-minute blocks during which each condition was presented in a random counterbalanced order. D. Data Collection In addition to the recordings captured by the inertial movement sensors and head-mounted eye tracker, an RGB camera and a Microsoft Kinect One sensor (both positioned behind the robot) captured information about the infant s body and face throughout child-nao interactions. A trained video coder annotated the visual gaze of the infant using the eye tracker data. There were seven possible gaze annotations: robot face, robot arms, robot legs, robot trunk, not robot, eyes

4 Study Session (8 min) Trial (1 min) Block 1 (4 min) Block 2 (4 min) Cycle (15 sec) Active (6 sec) Inactive (9 sec) or Active (7 sec) Inactive (8 sec) Fig. 3. Graphic explaining the flow of the experiment and the makeup of trials within each experiment session. The full study session lasted eight minutes and was divided into two four-minute blocks. Each condition occurred once during a block in a random counterbalanced order. Condition behaviors repeated in fifteen-second cycles of robot activity (6-7 seconds) and inactivity (8-9 seconds) that repeated four times to make up a condition trial. closed, and eye movement. A different trained video coder completed annotations of the infant s behavioral state using footage of the infant from the camera behind the robot. This rater used a five-point arousal scale with the following anchor points: drowsy, alert and inactive, alert and active, fussy, and crying. This coding system is consistent with previous infant behavior research [30]. IV. RESULTS The goal of this data collection was to determine how a humanoid robot can elicit visual, physical, and behavioral responses of an infant in preparation for future interventions. A. Visual Responses Infants spent a larger percentage of their time looking at the robot when the robot was active (74.69%) compared to when the robot was inactive (53.13%). Figure 4 displays the breakdown of the infants average attention on and off of the robot for the active and inactive periods of each condition. Overall, during the active state of either condition involving robot motion (moving arms or legs in conditions 1 and 2), the infants exhibited more visual attention on the robot compared to when the robot was active in the non-moving conditions (saying yay or kick in conditions 3 and 4) and all inactive periods. The infants spent more time observing the robot during and after robot motion than during conditions when the robot was not moving. Figure 5 illustrates that the infant gaze was directed to the specific limb that was moving; when the legs moved, the infant was drawn to watch the legs, and arm motion promoted a similar tendency. After the motion of the respective limbs stopped, infant visual focus continued to be directed to the limbs that had been moving. The infants visual attention was more focused on the robot s face when the condition was purely verbal, and attention on the face continued during the inactive state of the robot in these conditions. B. Physical Responses We found that the infants tended to move more when the robot was immobile or inactive. We detected leg movements Fig. 4. Percent of time that infants focused on any part of the robot during its activity and inactivity over the discussed conditions. For the rest of the time, the infant was blinking, moving their eyes, or focusing on something other than the robot. The error bars represent one standard deviation. using an algorithm that has been validated for typically developing infants aged one to twelve months old using the same Opal sensor [29]. Figure 6 illustrates infant motion during the active and inactive phases of each condition. We did not find a relationship between what limb the robot moved and what limb the infant moved. Infant arm movement data were not analyzed; leg movement activity was used to represent overall physical activity as infants tended to concurrently move all limbs or not move. C. Behavioral Response Using the collected camera data, a member of the research team re-watched the experiment footage and conducted qualitative and quantitive analyses of the infant s behavioral state during the data collection. Typically, infants were more likely to be alert and engaged when the robot was active compared to when the robot was not active. More specifically, infants maintained an alert and focused behavioral state for longer when the robot was active with movements compared to when the robot was active but immobile. Infants tended to become more fussy during the intervals of robot inactivity. It is not clear if this was triggered by the cessation of robot behavior. It was possible, for example, that the infant was

5 Fig. 5. The average amount of time the infants directed their gaze to a specific body region of the robot during each 6-7 seconds of robot behavior and the following 8-9 seconds of robot inactivity for each of the behavioral conditions. The error bars represent one standard deviation. Fig. 6. The average number of right- and left-legged movements per second during the active and inactive phase of each condition. The error bars represent one standard deviation. Fig. 7. Visualization of the percent of time infants spent being alert while the robot was active and inactive in each condition. The error bars represent one standard deviation. not comfortable in the chair used in the experiment setup. When the robot stopped moving, the infants also started to move their legs more. With the exception of the two excluded infants, the participating infants spent little time crying. Overall, the infants were more alert when the robot was active (64.50%) compared to when the robot was inactive (47.50%). Figure 7 shows the average percent the infant was alert during each robot condition. V. DISCUSSION In this work, we investigated the visual, physical, and behavioral responses of infants to a humanoid Nao robot. Understanding infant reactions to this type of robot is essential for designing future robot-mediated therapy interventions to encourage AR infants to make exploratory movements. One of our hypotheses was that infants would be inclined to imitate the small humanoid robot s motions. We also believed the robot motion might capture the attention of infants. In future work, this focus may help us to provide infants with contingent rewards for motion exploration. We found that infants directed their gaze to the Nao when it was active. Participants eyes focused on the part of the robot that was moving. This visual tendency could promote both the success of robot motion demonstrations and the future strategy of robot motion as a contingent reward. In future studies, it will be important to design interactions that help us discern whether infant gaze varies in this way because of salient events in their environment or because of the robot s form specifically. Infants tended to move more during periods of robot inactivity. This may mean that future intervention interactions should include pauses in robot activity to promote infant movement. We did not find a correlation between what limb the robot moved and what limb the infant moved. Thus, the studied duration of infant-robot interaction does not seem appropriate for an imitation paradigm in infants this young. It is possible that infants would adjust their movements to match the robot s demonstration during a longer interaction with a clear contingent reward. The infants behavioral state generally appeared more focused when the robot was active. This is important because we hope to use the humanoid robot to model new behaviors or movements for the infant to repeat. Infants tended to be

6 physically still and visually engaged when the robot was moving, and when the robot stopped moving, the infants started moving their legs more. We hypothesize that they may have been either attempting to mimic the robot or expressing a desire to see the robot move again. This observation provides support for using the robot in a contingent manner to encourage infant movement. As this data collection involved very young infants, the session duration and number of conditions were constrained to minimize the possibility of participant stress or fatigue. We were also limited in the number of conditions included in the experimental design; additional pairings of present and absent LED lights, speech, and motion would help us determine what types of robot action are most salient for infants. Smaller and younger infants seemed to experience discomfort in the selected chair, which could result in crying and fussing. We will consider other chair options in future studies. We also may need to reconsider the appropriate spacing between the robot and infant. The moderate distance between the infant and robot may have caused the infant to lose focus and look around the room at other lights and objects. In the future, recruiting infants aged six months and older will allow us to test whether children imitate robot movements at a developmental time period during which this behavior typically begins to emerge. These results offer a first step into the exploration of infant-robot interactions, particularly focusing on two- to five-month-old infants perceptions of a humanoid robot. Specifically, we understand that infants direct their gaze toward the robot when it is active. Infants are more alert during robot activity. Infants also seem motivated to move during periods of robot inactivity. In the future, we will apply these findings to develop robot behavior than can motivate infant movement. ACKNOWLEDGMENT The authors acknowledge Elizabeth Cha for her assistance and advice preparing this paper and Edward Kaszubski and Jeong-ah Kim for their contributions to the data collection. REFERENCES [1] S. A. Rosenberg, C. C. Robinson, E. F. Shaw, and M. C. Ellison, Part c early intervention for infants and toddlers: Percentage eligible versus served, Pediatrics, vol. 131, no. 1, pp , [2] M. A. Lobo and J. C. Galloway, Assessment and stability of early learning abilities in preterm and full-term infants across the first two years of life, Research in Developmental Disabilities, vol. 34, no. 5, pp , [3] G. Roberts, K. Howard, A. J. Spittle, N. C. Brown, P. J. Anderson, and L. W. Doyle, Rates of early intervention services in very preterm children with developmental disabilities at age 2 years, Journal of Paediatrics and Child Health, vol. 44, no. 5, pp , [4] B. G. Tang, H. M. Feldman, L. C. Huffman, K. J. Kagawa, and J. B. Gould, Missed opportunities in the referral of high-risk infants to early intervention, Pediatrics, pp. peds 2011, [5] R. L. Holt and M. A. Mikati, Care for child development: Basic science rationale and effects of interventions, Pediatric Neurology, vol. 44, no. 4, pp , [6] E. J. Gibson and A. D. Pick, An ecological approach to perceptual learning and development. Oxford University Press, USA, [7] E. Thelen and L. Smith, A dynamic systems approach to the development of cognition and action. Cambridge, MA: The MIT Press, [8] A. N. Meltzoff, Infant imitation after a 1-week delay: long-term memory for novel acts and multiple stimuli, Developmental Psychology, vol. 24, no. 4, p. 470, [9] S. S. Jones, The development of imitation in infancy, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 364, no. 1528, pp , [10] R. Barr, A. Dowden, and H. Hayne, Developmental changes in deferred imitation by 6-to 24-month-old infants, Infant Behavior and Development, vol. 19, no. 2, pp , [11] R. Barr and H. Hayne, It s not what you know, it s who you know: Older siblings facilitate imitation during infancy, International Journal of Early Years Education, vol. 11, no. 1, pp. 7 21, [12] H. Hayne, J. Boniface, and R. Barr, The development of declarative memory in human infants: Age-related changes in deffered imitation, Behavioral Neuroscience, vol. 114, no. 1, p. 77, [13] T. Hofer, P. Hauf, and G. Aschersleben, Infant s perception of goaldirected actions performed by a mechanical device, Infant Behavior and Development, vol. 28, no. 4, pp , [14] C. K. Rovee-Collier and M. J. Gekoski, The economics of infancy: A review of conjugate reinforcement, in Advances in Child Development and Behavior. Elsevier, 1979, vol. 13, pp [15] R. Barr, C. Rovee-Collier, and A. Learmonth, Potentiation in young infants: The origin of the prior knowledge effect? Memory & Cognition, vol. 39, no. 4, pp , [16] M. J. Matarić, J. Eriksson, D. J. Feil-Seifer, and C. J. Winstein, Socially assistive robotics for post-stroke rehabilitation, Journal of NeuroEngineering and Rehabilitation, vol. 4, no. 1, p. 5, [17] J. Greczek, E. Kaszubski, A. Atrash, and M. Matarić, Graded cueing feedback in robot-mediated imitation practice for children with autism spectrum disorders, in IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2014, pp [18] W. A. Bainbridge, J. Hart, E. S. Kim, and B. Scassellati, The effect of presence on human-robot interaction, in IEEE International Symposium on Robot and Human Interactive Communication (RO- MAN), 2008, pp [19] D. Leyzberg, S. Spaulding, M. Toneva, and B. Scassellati, The physical presence of a robot tutor increases cognitive learning gains, in Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 34, no. 34, [20] S. Carey and E. Spelke, Domain-specific knowledge and conceptual change, Mapping the Mind: Domain Specificity in Cognition and Culture, vol. 169, p. 200, [21] S. Baron-Cohen, Mindblindness: An essay on autism and theory of mind. MIT press, [22] A. M. Leslie, Tomm, toby, and agency: Core architecture and domain specificity, Mapping the mind: Domain specificity in cognition and culture, pp , [23] M. C. Frank, E. Vul, and S. P. Johnson, Development of infants attention to faces during the first year, Cognition, vol. 110, no. 2, pp , [24] G. Matsuda, H. Ishiguro, and K. Hiraki, Infant discrimination of humanoid robots, Frontiers in Psychology, vol. 6, p. 1397, [25] B. S. Bloom, The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring, Educational Researcher, vol. 13, no. 6, pp. 4 16, [26] A. Corbett, Cognitive computer tutors: Solving the two-sigma problem, in International Conference on User Modeling. Springer, 2001, pp [27] M. C. Piper, J. Darrah, T. O. Maguire, and L. Redfern, Motor assessment of the developing infant. Saunders Philadelphia, 1994, vol. 1. [28] APDM Wearable Technologies, Portland, OR, USA, Opals, [29] B. A. Smith, I. A. Trujillo-Priego, C. J. Lane, J. M. Finley, and F. B. Horak, Daily quantity of infant leg movement: Wearable sensor algorithm and relationship to walking onset, Sensors, vol. 15, no. 8, pp , [30] B. Sargent, N. Schweighofer, M. Kubo, and L. Fetters, Infant exploratory learning: influence on leg joint coordination, PLoS One, vol. 9, no. 3, p. e91500, 2014.