Virtual reality: a tool for the highly quantitative study of animal behavior

Size: px

Start display at page:

Download "Virtual reality: a tool for the highly quantitative study of animal behavior"

Jocelin Young
5 years ago
Views:

1 Virtual reality: a tool for the highly quantitative study of animal behavior Andrew D. Straw Laboratory of Michael H. Dickinson Bioengineering, California Institute of Technology August 20, 2009 SciPy 2009 Conference

Introduction Vision Egg and Motmot Vision Egg and Motmot http://visionegg.org/ http://code.astraw.

2 Introduction Vision Egg and Motmot Vision Egg and Motmot Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

Introduction Motivation: Fly neuroscience Motivation: behavior and neurobiology of fly flight Photo: Clifton Beard (CC-BY-NC) Behavior emerges from interactions between the nervous system,

3 Introduction Motivation: Fly neuroscience Motivation: behavior and neurobiology of fly flight Photo: Clifton Beard (CC-BY-NC) Behavior emerges from interactions between the nervous system, musclo-skeletal system and the environment. Goal 1: algorithmic level understanding Goal 2: implementation level understanding Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

4 Introduction Motivation: Fly neuroscience Fine-scale sensory-motor control Andrew D. Straw (Caltech) Virtual Video: reality Gwyneth for the Card study & Michael of animal Dickinson behavior SciPy / 30

5 Introduction Motivation: Fly neuroscience Large-scale sensory-motor control Straw, Branson, Neumann & Dickinson (in review). Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

6 Introduction Altitude control Altitude control in insects Vision used to estimate height what algorithm(s)? Changes in height implemented through wingstroke modification (frequency, kinematics) Franceschini et al. proposed model unifying several observations Franceschini, Ruffier, Serres (2007). Current Biology Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

7 Introduction Altitude control Free-flight arena 100Hz cameras 120Hz DLP projector mirrors Straw, Lee, Dickinson (In preparation) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

8 Introduction Altitude control Closed loop and open loop Closed loop: movement affects sensory feedback Open loop: movement does not affect sensory feedback Virtual open loop on freely moving animals A B v v world z λ λ world z world z stim λ stim x v stim x Straw, Lee, Dickinson (In preparation) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

9 Introduction Altitude control Virtual open loop experiment Straw, Lee, Dickinson (In preparation) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

Introduction Altitude control Ventral visual motion is not used to regulate altitude Altitude change (cm) x acceleration (m s 2 ) 5.0 2.

10 Introduction Altitude control Ventral visual motion is not used to regulate altitude Altitude change (cm) x acceleration (m s 2 ) n=37 n=42 n=39 n=41 n=37 n=47 n=44 * * * * * n=0 n=38 * z velocity (m s 1 ) Visual velocity (deg s 1 ) CL jump Straw, Lee, Dickinson (In preparation) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

11 Motmot Motmot overview Realtime camera control, image acquisition and analysis BSD license Modular, decoupled pieces Pluggable, unified viewer/analysis app FView 57 pages of documentation Straw, A.D. and Dickinson, M.H. (2009) Motmot, an open-source toolkit for realtime video acquisition and analysis. Source Code for Biology and Medicine 4:1 doi: / Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

12 Motmot demo (run FView externally) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

13 Motmot Motmot relationships OpenGL pyglet libdc1394 libprosilica QuickTime wxpython pygarrayimage libcamiface (C) FlyMovieFormat wxglvideo cam_iface (Python) Intel IPP or AMD Framewave fview FastImage realtime_image_analysis fview_ext_trig flytrax trackem Filled box: GUI App/plugin dotted arrow: optional dependency Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

14 Motmot Example: empty FView plugin Extracted from plugin tutorial (online, in PDF docs). import motmot.fview.traited_plugin as traited_plugin class FviewHistogram(traited_plugin.HasTraits_FViewPlugin): plugin_name = image histogram def camera_starting_notification(self,cam_id, pixel_format=none, max_width=none, max_height=none): # This function gets called from FView when a camera is # initialized. return def process_frame(self,cam_id,buf,buf_offset,timestamp,framenumber): draw_points = [] draw_linesegs = [] # This function gets called from FView immediately after # acquisition of each frame. Implement your image processing # logic here. return draw_points, draw_linesegs Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

15 Motmot CamTrig USB Device Hard realtime Firmware for $30 USB device AT90USBKEY Traits-based FView plugin to trigger cameras, measure latency, synchronize with other devices, acquire analog data Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

16 Motmot Motmot in action flight stage IR lights flight stopper flight puffer IR sensor IR sensitive camera LED display Computer running: - FView - custom realtime image processing - Electrophysiology data acquisition - Stimulus generation Maimon, Straw & Dickinson (submitted) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

17 Motmot Motmot in action Maimon, Straw & Dickinson (submitted) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

18 Motmot Motmot in action 5 mv 1 s VS cell membrane voltage Upward motion Mean amplitude of L & R wings (deg) R wing amplitude (deg) L wing amplitude (deg) Maimon, Straw & Dickinson (submitted) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

19 Vision Egg Vision Egg overview Realtime visual stimulus generation library High-level interface for OpenGL LGPL license Old (1st release in 2002), stable Many demos Straw, Andrew D. (2008) Vision Egg: An Open-Source Library for Realtime Visual Stimulus Generation. Frontiers in Neuroinformatics 2:4. doi: /neuro Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

20 Vision Egg demo (run multi_stim.py demo externally) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

21 Vision Egg Vision Egg relationships Windows GUI Mac OS X GUI X11 GUI SDL OpenGL pygame PyOpenGL Vision Egg Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

22 Vision Egg Example 1: sine grating, conventional flow control # Import modules import VisionEgg VisionEgg.start_default_logging(); VisionEgg.watch_exceptions() from VisionEgg.Core import get_default_screen, Viewport, \ swap_buffers from VisionEgg.Gratings import SinGrating2D # Initialize OpenGL window/screen screen = get_default_screen() # Create sinusoidal grating object stimulus = SinGrating2D(spatial_freq = 10.0 / screen.size[0], temporal_freq_hz = 1.0, ) # Create viewport - intermediary between stimuli and screen viewport = Viewport( screen=screen, stimuli=[stimulus] ) # Use our own main loop tstart = VisionEgg.time_func() while (VisionEgg.time_func() - tstart) <= 5.0: screen.clear() viewport.draw() swap_buffers() Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

23 Vision Egg Example 2: sine grating, VE flow control # Import modules import VisionEgg VisionEgg.start_default_logging(); VisionEgg.watch_exceptions() from VisionEgg.Core import get_default_screen, Viewport from VisionEgg.FlowControl import Presentation from VisionEgg.Gratings import SinGrating2D # Initialize OpenGL window/screen screen = get_default_screen() # Create sinusoidal grating object stimulus = SinGrating2D(spatial_freq = 10.0 / screen.size[0], temporal_freq_hz = 1.0, ) # Create viewport - intermediary between stimuli and screen viewport = Viewport( screen=screen, stimuli=[stimulus] ) # Create presentation object and go p = Presentation(go_duration=(5.0, seconds ),viewports=[viewport]) p.go() Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

24 Vision Egg Future of the Vision Egg Planned Maintain backward compatibility OpenGL 3.1 and 3.2 support (maintaining backward compatibility with VE code) Export parameters to emerging meta-data formats (e.g. odml) Possible integration with RELACS electrophysiology software Not planned enthought.traits parameter encapsulation (Maybe in VE2 or something with a different name) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

25 Latency Measuring latency - Vision Egg Straw (2008) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

26 Latency Measuring latency - Motmot Computer CamTrig USB device Camera time pre x 1 = get_time() request timestamp receive timestamp post x 1 = get_time() request handler camera sync trigger emit trigger pulse... y 1 =get_time() capture pre x 2 = get_time() request timestamp receive timestamp post x 2 = get_time() y 2 =get_time() Synchronization flowchart Latency measured 3 ways Straw & Dickinson (2009) Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

27 Conclusion Putting it all together - virtual reality Flydra - 3D realtime markerless tracking of multiple flying animals I I I I A 2D feature extraction, data association, Extended Kalman Filtering to estimate (x, y, z, x, y, z ) Built from Motmot components, Numpy/Scipy and lots of other stuff Not open source Straw, Branson, Neumann & Dickinson (in review) simulated cylinder B Tracking cameras C virtual reality simulated cylinder static gray stimulus Computer projector Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

28 Conclusion VR confinement of freely flying flies A virtual reality simulated cylinder y y x x B control: simulated cylinder appears, remains static y y x x C control: gray background (no stimulus) y y x x Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

29 Conclusion Conclusions Science High throughput virtual reality enables a detailed investigation of sensory integration and motor control. Repetability of specific dynamic configurations allows investigations of active behaviors (e.g. self induced motion parallax). Large volumes of data enable sophisticated modeling. in conjunction with electrophysiology on tethered flying flies in the genetic model organism Drosophila melanogaster Python Motmot and Vision Egg libraries enable this Flexible, modular, low-latency, open-source Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

30 Conclusion Thanks Michael Dickinson, David O Carroll NSF, NIH, AFOSR my collaborators developers of open source software Andrew D. Straw (Caltech) Virtual reality for the study of animal behavior SciPy / 30

Nature Methods: doi: /nmeth Supplementary Figure 1. VR Assays for Flies, Fish, and Mice

Nature Methods: doi: /nmeth Supplementary Figure 1. VR Assays for Flies, Fish, and Mice Supplementary Figure 1 VR Assays for Flies, Fish, and Mice (a) The Flycave assay, a 1m diameter 1m high cylindrical VR arena. Three projectors create a panoramic VR, each projecting directly onto the surface