Bio-inspired Computing for Robots and Music. Jim Tørresen Research group Robotics and Intelligent Systems

Transcription

1 Bio-inspired Computing for Robots and Music Jim Tørresen Research group Robotics and Intelligent Systems

2 Robotics and Intelligent Systems (ROBIN) Jim Tørresen Professor, Group leader Mats Høvin Assoc. Prof. Kyrre Glette Assoc. Prof. Kristian Nymoen Assoc. Prof. (shared with music dep) Yngve Hafting Senior engineer Adjunct positions (20%): Alexander Wold (assoc.prof.) Jørgen Norendal (lecturer) Roar Skogstrøm (lecturer) Ståle Skogstad (assoc.prof.) Ole Jakob Elle (assoc.prof.) Postdocs: Charles Martin Enrique Garcia-Ceja Kai Olav Ellefsen Md. Zia Uddin Weria Khaksar PhD students: Arash Ahmadi (MedFak) Asbjørn Danielsen (HiN) Eivind Samuelsen Flavia Dias Casagrande (HIOA) Justinas Miseikis Jørgen Nordmoen Kim Mathiesen Sondre Engebråten (FFI) Tønnes Nygaard Students Bachelor ~60; Master: ~30 Robotics and Intelligent Systems program Students hired on hourly basis Visiting researchers

3 Robotics and Intelligent Systems group ROBIN Creating systems for demanding run-time environments Hardware Electronics 3D-printing Prototyping Robotics and Intelligent Systems Applications Robotics Music Health ++ Biology Apply principles from nature Web page: Google for ROBIN IFI

4 ROBIN Methods: Bio-inspired Evolutionary Computation Machine learning Self-awareness Prediction Applications: Adaptive and evolutionary robotics Robotic surgery and robots for elderly Music technology Mental health support Reconfigurable (evolvable) hardware

5 Robot surgery Intervention Centre Oslo University Hospital (former Rikshospitalet)

6 Environment Aware Robot for Hospitals Goal: allow the fluoroscopy (Carm) and ultrasound robots to coexist in surgery room without having any direct communication 3D cameras are used to identify and detect robots in the surgery room UR5 robot tries to adapt to changes in the environment and move out of the way when needed

7 State-of-the-art Rapid Prototyping Facilities 3D printers and milling machines Large potential for developing innovative robot systems.

8 Robot Design, Simulation, Assembly and Evaluation - Work with real robots and simulations. - Reduce gap between simulation and reality. - Create novel methods for design (e.g., evolution) and dynamic body shapes (morphology).

9 Evolved Control Systems We can evolve movement patterns! Parameterize periodic functions for each joint Evolve all those parameters 9

10 Evolved Robot Design Robot bodies could be difficult to design by hand. We use evolutionary algorithm to evolve both body and control system simultaneously. 10

11 Video Reuters 11

12 EPEC: Prediction and Coordination for Robots and Interactive Music 1 PhD (Tønnes Nygaard) + 2 post-docs (Charles Martin and Kai Olav Ellefsen) RCN grant Goal: Design, implement and evaluate multi-sensor systems that are able to sense, learn and predict future actions and events.

13 Dyret: A low-cost self-modifying quadruped - Our most advanced legged robot to date - Used for evolutionary experiments and research in self modeling and control - Used in master s projects.

14 MECS: Multi-sensor Elderly Care Systems 1 PhD (Trenton Schulz) + 2 postdocs (Weria Khaksar and Zia Uddin) ( ) Goal: Create and evaluate multimodal mobile human supportive systems that are able to sense, learn and predict future events.

15 Navigation without a Map - Having a mobile robot with 3D camera and/or Lidar. - Moving in a completely unknown environment. - Using the sensory information to build the path and navigate. - Employing several AI tools including Fuzzy Logic and Genetic Algorithm Challenge: - Finding computationally cheap solutions with high quality.

16 Real-time Tracking, Segmentation, and Modelling - Deep Learning for body skeleton tracking in real-time. - 3-d body modeling in real-time.

17 Real-time Face Tracking - Face tracking in realtime using RGB camera on a robot. - Works well when there is enough light. - Face tracking in realtime using thermal camera on a robot. - Works well even in the dark.

18 Bio-inspired Computing for Music 18

19 State-of-the-art Motion Capture Facilities Allows precise tracking of human and robot motion Camera-based and on-body motion capture

20 Music and Motion - Using motion capture and sensors to analyse musical movements. - Use cases: creating music, assisting performers, analysing dance, understanding musical motion.

21 (Inter) Active Music Direct Control o Navigate within the song o Control certain instruments (e.g. keep playing the chorus drumbeat in the verse) o Change the tempo of the song Indirect Control o Use on-body sensors to adapt the music to the mood of the user o Listen to music that pushes you to work out harder o Fuse the musical preferences of multiple users into one song 21

22 Ant Colony Optimization (ACO) Ants find shortest path to food source from nest. Ants deposit pheromone along traveled path which is used by other ants to follow the trail. This kind of indirect communication via the local environment is called stigmergy. 22

23 Funky Sole Music 23

24 Combination of sound samples and synthesis Hierarchy of loops, controlled by the user Various sound effects are controllable when in different states (i.e., for different walking patterns) 24

25 Video Reuters 25

26 PheroMusic: Navigating a Musical Space 26

27 Direct + Indirect Control Combined PheroMusic app Control by ant colony optimization 27

28 Evolutionary Music Algorithms - Evolutionary Algorithms are efficient optimisers. Shown to be useful for art! - Tricky to measure quality for art and music. - Idea: Crowdsource in an app! - Use EA to create new music in microjam, ask users for ratings.

29 Predictive smartphone music making - MicroJam: Social music making app - Gathered 1000s of performances - Use ML to help users create music, playback appropriate sounds to their actions. Data ML Interaction!

30 Embedded Smart Musical Devices - Raspberry/Beaglebone can be used to make selfcontained interactive / intelligent music systems. - Use sensors to create and modify music. - Challenge: Use machine learning to generate more effective music and control it more effectively. - Create systems that users want to use everyday!

31 Centre of Excellence for Interdisciplinary Studies in Rhythm, Time and Motion The center will study the perceptual and cognitive mechanisms underlying our ability to experience the rhythm and act rhythmically. Interdisciplinary collaboration between musicology, phycology and informatics. Machine learning and robotics to be applied Hiring new PhDs and postdocs (start 2018) 31

32 INTROMAT: INtroducing personalized TReatment Of Mental health problems using Adaptive Technology ( ) Goal: Increase access to mental health services for common mental health problems by developing smartphone technology which can guide patients. Funding: IKTPLUSS Lighthouse, Research Council of Norway

33 Mental health monitoring (INTROMAT) Analysis of sensor and behavioral data with machine learning. Mental states prediction for bipolar, anxiety and attention-deficit/hyperactivity disorders. Use of smartphones, wristwatches and virtual reality devices to monitor users behavior. Adapt clinical follow up and activate automatic treatments when needed. Mental state Speech, motion, heartbeat, phone usage etc Machine learning

34 International Collaboration 34

35 35

36 36