Nature Methods: doi: /nmeth Supplementary Figure 1. VR Assays for Flies, Fish, and Mice

Transcription

1 Supplementary Figure 1 VR Assays for Flies, Fish, and Mice (a) The Flycave assay, a 1m diameter 1m high cylindrical VR arena. Three projectors create a panoramic VR, each projecting directly onto the surface of the cylinder, and simultaneously via 6 mirrors placed in 3 pairs around the setup. Flies are tracked in 3D from above using custom software and multiple cameras. (b) The FishVR fishbowl assay, in which fish swim in 9cm deep water in a 32cm diameter hemispherical bowl. A panoramic VR is created by projecting onto the surface of the bowl, via a mirror, from below. Fish position is tracked in 3D in real time using custom software and multiple cameras. (c) The MouseVR mouse floor assay. A 0.5m diameter elevated circular platform is placed on top of a 1.9m consumer television. Mouse head position is tracked in real time using custom software and a single camera placed above.

2 Supplementary Figure 2 FreemoVR software and system architecture (a) The Flycave assay, a 1m diameter 1m high cylindrical VR arena. Three projectors create a panoramic VR, each projecting directly onto the surface of the cylinder, and simultaneously via 6 mirrors placed in 3 pairs around the setup. Flies are tracked in 3D from above using custom software and multiple cameras. (b) The FishVR fishbowl assay, in which fish swim in 9cm deep water in a 32cm diameter hemispherical bowl. A panoramic VR is created by projecting onto the surface of the bowl, via a mirror, from below. Fish position is tracked in 3D in real time using custom software and multiple cameras. (c) The MouseVR mouse floor assay. A 0.5m diameter elevated circular platform is placed on top of a 1.9m consumer television. Mouse head position is tracked in real time using custom software and a single camera placed above.

3 Supplementary Figure 3 Post avoidance behavior as a function of latency (a) The FreemoVR rendering pipeline. From left to right; a virtual world is first rendered as a cube map based on the observer position.

4 Using a model of the display geometry, a geometry texture is rendered. Based on a calibration of the display elements which maps their pixels to geometry coordinates, a VR is created. (b) The FreemoVR software allows distributed operation across multiple computers such as in situations where multiple PCs are required for real-time tracking, or for generating or displaying the stimulus on sufficient displays to suit the experiment. (c) Schematic describing the information flow between the system components in a FreemoVR assay for a freely moving observer. Dark grey indicates computation occurs in real time, light grey represents a calibration step specific to the particular display geometry.

5 Supplementary Figure 4 Drosophila post avoidance behavior trajectories (a) All trials for post avoidance and latency measurement analysis. In trials with a real post (RW) N flies = 40, n trials = 240. In all VR trials, N=80, n=1890. In all VR trials, total added latency is indicated in text.

6 Supplementary Figure 5 MouseVR arena and analysis (a) The elements of the mouse VR experiments were located in the laboratory as indicated. The television lies in the corner of the room surrounded by black walls on two sides, and a black curtain on the other. Mice are introduced from the front side and are shielded from the rest of the lab by a black cardboard divider. (b) Mouse preference (cumulative) for the shallow side. (c) Cumulative preference of mice for the wall side of the laboratory. (d) Cumulative distance walked by the mice for all trials. Mean ± s.d. for all plots. N=15 mice for RW trials, N=16 for VR, N=16 for ST.

7 Supplementary Figure 6 MouseVR head dip analysis (a) Occurrence of head dips during the RW, ST and VR trials. Black bars indicate the time the mouse head was scored to be below the platform; start of bar is head-down, end of bar is head-up. (b) Distribution of the duration of all head dips for all scored trials. (c)

8 Summary plots per mouse quantifying the number of head dips. Circles indicate each scored mouse. Mann-Whitney test; RW vs VR; p=0.15, RW vs ST; p=0.59, ST vs VR, p= mice in each group. (d) Summary plot per mouse quantifying the total head dip duration. Circles indicate each scored mouse. Mann-Whitney test; RW vs VR; p=0.53, RW vs ST; p=0.29, ST vs VR, p= mice in each group. (b,c,d) Box plots indicate median, upper- and lower-quartile. Whiskers extend to 1.5 IQRs of the lower and upper quartile, observations outside this range are indicated with diamonds. (e) Spatial location of each head dip. Arrows indicate dip location; downward arrows indicate head-down phase, upward arrows indicate head-up phase.

9 Supplementary Figure 7 Eliciting path following of freely flying Drosophila

10 (a) Fly trajectories for path following task with different gain values. Path following is elicited in a gain-dependent manner. The gain (text in top-left of each panel) describes how strong the virtual world is biased to bring the fly location closer to a target location on the path. As the fly position approaches the target, it is advanced around the path. (b) Distance from the current fly position to current target position on the path. The accuracy of the path following task can be quantified by considering the distance from the fly to the target point on the infinity path. When the distance between fly and target is less than 0.1m the target is advanced to the next point on the path - yielding a theoretical lower bound of 0.1m between the fly and the target in the case of perfect path following. N=50 flies.

11 Supplementary Figure 8 Flight performance for glued fly experiments (a) Pearson correlation between angular velocity of stimulus and angular velocity of the fly for free flight path following experiments under different glue preparations; no glue (N=36 flies), head-free glue (N=39), and head-fixed glue (N=78). Mean and ± 68% c.i. plotted. (b) Distribution of vertical speed for freeflight trails. (c) Distribution of altitude for freeflight trials. (d) Pearson correlation between angular velocity of stimulus and simulated angular velocity of the fly under head-free (N=6) and head-fixed (N=6) tethered preparations. (e) Distribution of simulated horizontal speed of tethered trials. (f,g) Distribution of simulated angular velocity for tethered head-free and head-fixed preparations for different gain couplings between wingbeat amplitude and simulated turning torque.

12 Supplementary Figure 9 Fish trajectories for path-following experiments (a-c) All experimental data for AB and mitf-a -/- path following experiments in the a large-dots, b small-dots and c grey conditions. All conditions were tested for all fish. N=56 AB fish, N=62 mitf-a -/-.

13 Supplementary Figure 10 Swimming behavior of AB and mitf-a -/- during path experiments (a-c) Large dot condition statistics. (d-f) Small dot condition statistics (g-i) Grey condition statistics. Distribution of fish forward velocity in all trials a,e,g. during the a large-dots Distribution of control action the magnitude of the dot velocity shown to the fish in order to elect path following in all trials b,e,h. Distribution of distance to target the distance between the target point on the path and the fish position in all trials c,f,i. All conditions were tested for all fish. N=56 AB fish, N=62 mitf-a -/-.

14 Supplementary Figure 11 Teleportation and changing environments in virtual reality (a) Zebrafish could visit a checkerboard scene or (b) a plant scene by entering a virtual teleportation portal. See also Supplementary Video 12. (c) Each of two portals was coupled to a constant destination per fish but different fish had different couplings. Upon entering a portal, the portals were rearranged to equalize distance required for subsequent portal entry. Additionally, for the portal coupled to the other scene, the fish was virtually teleported to that new scene. (d) Decisions, operationally defined as portal entry (vertical marks) and current scene (horizontal line), over time for each fish. (e) Top view of occupancy. (f) Fraction of all decisions per fish that teleported it to the plant scene (one-sample t-test difference from 0.5, p=0.81). (g) Fraction of all portal entries per fish that were magenta (onesample t-test difference from 0.5, p=2.7e-7). (h) Fraction of 30 minutes per fish in which the fish was in the plant scene (one-sample t- test difference from 0.5, p=0.047). (i) Mean horizontal speed per fish in each coupling condition (two-related-sample t-test, p=0.0411). (d-i) N=12 fish, AB strain. Box plots indicate median, upper- and lower-quartile. Whiskers extend to 1.5 IQRs of the lower and upper quartile.

15 Supplementary Figure 12 Animation of the tail beat of the zebrafish larvae To obtain kinematic information during swimming, 3 zebrafish larvae of the control group were recorded in a square arena (20cm width, 1 cm water depth) and filmed with a Basler 2040-um camera at 180 frames per second for a duration of 5 minutes. The curvature, C(s), of the animal is computed along the curvilinear abscissa of the median line, s, from head to tail of the fish. (a) Curvature along the median line of an individual as a function of time. Three different tail beats are shown for a single animal. The movement is highly stereotyped, and can be described by an inflexible part of the body ( from the head) and an oscillation of the body that propagates from the end of the inflexible part to the tail (with temporal frequency, and a velocity of propagation ). 0.1s after the beginning of the tail beat, no movement is observed. (b) Movement of the recorded fish (18 frames at 180 fps). (c) Animation of the median line of the fish. (d) Animation generated of the virtual fish. The rendering of FreemoVR is done at 120fps, so only 12 frames are represented.

16 Supplementary Figure 13 Burst and glide movement of the zebrafish larvae in absence of VR stimuli (a) The velocity,, in the direction of the movement is displayed for a single animal as a function of time. Burst and glide events are clearly visible with phase of high acceleration followed by a decay of in speed resulting from drag. We first identified the position of the local maxima of the velocity (in red). (b) For each individual of the control group we plotted the distribution of the time between two successive burst and glide events,. (c) The median value for all individuals is given by. We selected for our animation to allow sufficient frames for effective animation. (d) For each individual of the control group we plotted the distribution of the characteristic time of decay of the velocity,. This time has been computed by the fit of an exponential function during the 0.3 s after the detection of each velocity peak. (e) The median value for all individuals is given by. (f) For each individual of the control group we plotted the distribution of the value of the maximal value of the velocity for each burst and glide event, (in red on a.). (g) The median value for all individuals is. For simplicity, and due to its extremely short nature, the approach we use to approximate the burst and glide animation does not account for the first part of acceleration of the burst and glide movement. Furthermore, since the time between two successive beats is on the higher end of the range, the distance travelled for each event should be longer. The value used for animation were taken slightly higher to account for those discrepancies,.

17 Supplementary Figure 14 Individual fish data from social feedback experiment Histogram of each individual real fish s distance from the periphery of the arena, r, as a function of the strength of the goal-oriented tendency, ω, of the virtual fish. The virtual fish s internal preferred trajectory was fixed at r = 0.07m (dotted white). These individual fish data were used (together with control experiments on 15 real fish with no virtual fish) to create panel Fig. 5k.

18 Supplementary Figure 15 Empirically derived measurements of group splitting and shared travel direction in social feedback experiment (a) Empirically measured probability of the group (real and virtual zebrafish) splitting as defined by the distance between the two fish exceeding a threshold value of 0.05 m. (b) Empirically measured probability that the real fish is traveling on the virtual fish s internal preferred trajectory, r = 0.07 ±0.01m. N=16 fish, same experiment as shown in Fig. 5i-l.

19 Wildtype Head-free Head-fixed Number of experiments Total fly in apparatus duration (hours) Number of flies Pre trajectory filter Time spent in flight (seconds) Fraction time spent in flight 0.02% 0.03% 0.003% Number of trials Post trajectory filter Time spent in flight (seconds) Fraction time spent in flight 0.02% 0.03% 0.001% Number of trials Supplementary Table 1: Trial and experiment details for glued freeflight experiments

20 Item Number Description Cameras x10 Basler-aca m POE Network Switch x1 24-port to suit IR Filters x10 Lee Folienfilter IR-Filter 100 x 100 mm 87 Infrarot Lens x10 Computar CS-Mount mm Varifocal Lens 3 Computers are needed 2 for tracking and one for Display Graphics Cards x2 nvidia GeForce GTX 670 CPU x3 Intel Core i5/i7 Motherboard x3 ASUS P9X79-E WS Network Cards x3 Intel PRO/1000 PT Quad Port Low Profile Server Adapter Projector x3 Mitsubushi WD385U-EST Memory x3 > 4Gb per PC, to suit The arena should be constructed to suit your space requirements Flight arena x1 1m diameter x 1m high (described in text) Aluminium profiles ad. lib. 80/20 brand or similar Arena top x1 Perspex, with ventilation holes Mirrors x6 Required if using 3 projectors. IR Leds ad. lib. 1W per LED, wired as to illuminate entire arena Lab power supply x1 Linear, for driving LEDS Triggerbox x1 Software Flydra Tracking Software x1 Supplementary Software, strawlab.org/freemovr FreemoVR x1 Supplementary Software, strawlab.org/freemovr Supplementary Table 2: Equipment list for Fly VR arena

21 Item Number Description Cameras x4 Basler-aca m POE Network Switch x1 12-port to suit IR Filters x4 Lee Folienfilter IR-Filter 100 x 100 mm 87 Infrarot Lens x4 Computar CS-Mount mm Varifocal Lens One computer needed Graphics Card x1 nvidia GeForce GTX 670 CPU x1 Intel Core i7 Motherboard x1 ASUS P9X79-E WS Network Cards x1 Intel PRO/1000 PT Quad Port Low Profile Server Adapter Projector x1 Optoma ML1500e Memory x1 > 8Gb per PC, to suit SSD x1 Samsung SSD 850 PRO 512GB, SATA The arena should be constructed to suit your space requirements Bowl x1 Described in text Aluminium profiles ad. lib. Bosch 20mm profiles or similar Enclosure x1 Black PVC sized to enclose bowl + cameras Mirror x1 60cm x 60cm IR Leds ad. lib. 1W per LED, wired as to illuminate entire arena Lab power supply x1 Linear, for driving LEDS Triggerbox x1 Software Flydra Tracking Software x1 Supplementary Software, strawlab.org/freemovr FreemoVR x1 Supplementary Software, strawlab.org/freemovr Supplementary Table 3: Equipment list for Fish VR arena

22 Item Number Description Cameras x1 Basler-aca m POE Injector x1 PoE Single Port Injector POE21U IR Filters x1 Lee Folienfilter IR-Filter 100 x 100 mm 87 Infrarot Lens x1 Computar CS-Mount mm Varifocal Lens One computer needed Graphics Card x1 nvidia GeForce GTX 670 CPU x1 Intel Core i7 Motherboard x1 ASUS P9X79-E WS Network Cards x1 Intel PRO/1000 PT Quad Port Low Profile Server Adapter Memory x1 > 4Gb per PC, to suit SSD x1 Samsung SSD 850 PRO 512GB, SATA Television x1 Sony KDL-75W855CBAEP IR Leds ad. lib. 1W per LED, wired as to illuminate entire arena Lab power supply x1 Linear, for driving LEDS Software Mouse Tracking Software 1x Supplementary Software FreemoVR 1x Supplementary Software, strawlab.org/freemovr Supplementary Table 4: Equipment list for Mouse VR arena