Reducing confounding factors in automatic acoustic recognition of individual birds

Size: px

Start display at page:

Download "Reducing confounding factors in automatic acoustic recognition of individual birds"

Jasmine Elfrieda Green
5 years ago
Views:

1 Reducing confounding factors in automatic acoustic recognition of individual birds Dan Stowell Machine Listening Lab Centre for Digital Music Acoustic recognition of birds 1 / 31

2 versus

3 Machine listening and bird sounds - why? dan.stowell@qmul.ac.uk Acoustic recognition of birds 3 / 31

4 Machine listening and bird sounds - why? Changes in populations, in migration patterns monitoring is important Intrusive vs. passive monitoring behavioural impact of catching/ringing birds Many birds are most easily observed by sound Manual (volunteer) monitoring common, but not scalable dan.stowell@qmul.ac.uk Acoustic recognition of birds 4 / 31

5 In this talk... Classification-based approaches to: 0. Bird species recognition 1. Bird sound detection (presence/absence) 2. Bird individual ID (By the way: we do more than just classification!) Acoustic recognition of birds 5 / 31

6 Species classification of bird sounds In 2014: feature-learning approach to bird sound recognition Dataset Location Total duration Num items Num classes Labelling lifeclef Brazil 77.8 hours (12M frames) singlelabel 100 lifeclef2014 Classifier: multilabel 90 AUC (%) mfcc-ms mfcc-maxp mfcc-modul melspec-ms melspec-maxp melspec-modul melspec-kfl1-ms melspec-kfl2-ms Feature learning melspec-kfl3-ms melspec-kfl4-ms Acoustic recognition of birds 6 / 31 melspec-kfl8-ms c-kfl4pl8kfl4-ms

7 Bird species classification: Warblr Warblr app for Android and ios Acoustic recognition of birds 7 / 31

8 Bird species classification: Warblr Over 45,000 recordings submitted to our database ( 80/day) Submission geolocations dan.stowell@qmul.ac.uk Acoustic recognition of birds 8 / 31

9 Some of our users... Acoustic recognition of birds 9 / 31

10 Some of our users... Acoustic recognition of birds 9 / 31

12 Part 1: Bird Audio Detection challenge Many projects need reliable detection of bird sounds e.g. in long unattended recordings But existing methods are not robust, not general-purpose enough, and need lots of manual tweaking/post-processing Acoustic recognition of birds 11 / 31

Bird Audio Detection challenge We designed the Bird Audio

audio from around the UK (Warblr phone app) Dev set 2: 7k

Testing set: 10k items, remote monitoring, Chernobyl

13 Bird Audio Detection challenge We designed the Bird Audio Detection challenge Dev set 1: 10k items, crowdsourced audio from around the UK (Warblr phone app) Dev set 2: 7k items, crowdsourced audio from misc field recordings Testing set: 10k items, remote monitoring, Chernobyl Exclusion Zone Acoustic recognition of birds 12 / 31

14 Bird Audio Detection challenge Training/testing sets differ in: location recording eqpt species class balance background sounds time of day time of year weather... How is a classifier meant to work in such mismatched conditions??? dan.stowell@qmul.ac.uk Acoustic recognition of birds 13 / 31

15 Bird Audio Detection challenge: outcomes 30 teams submitted Strong results (up to 89% AUC) Domain adaptation strategies Pseudo-labelling, test mixing Though not always needed Acoustic recognition of birds 14 / 31

16 So why do we evaluate using matched conditions? To study the classifier s behaviour Sometimes a practical application is in matched conditions Pragmatic reasons: only one dataset available; free choice of bootstrap/n-fold crossvalidation...because our algorithms aren t good enough at avoiding confounds? dan.stowell@qmul.ac.uk Acoustic recognition of birds 15 / 31

17 Machine learning workflow train validate test Acoustic recognition of birds 16 / 31

18 Machine learning workflow train validate test reallytest Acoustic recognition of birds 16 / 31

20 Part 2: Identifying individual bird ID Motivation: reduce intrusive monitoring (capturing/tagging/ringing) Many birds do have individual signature Acoustic recognition of birds 18 / 31

21 Identifying individual bird ID Data collection: Acoustic recognition of birds 19 / 31

22 Identifying individual bird ID Data collection: Acoustic recognition of birds 19 / 31

23 Identifying individual bird ID Data collection: Bird ID: categorical label. Is this the same task as species classification? Acoustic recognition of birds 19 / 31

24 Identifying individual bird ID Acoustic recognition of birds 20 / 31

25 Identifying individual bird ID Acoustic recognition of birds 20 / 31

26 Making use of silence (1) Training set: Acoustic recognition of birds 21 / 31

27 Making use of silence (1) Training set: Testing set: Acoustic recognition of birds 21 / 31

Making use of silence (1) dan.stowell@qmul.

28 Making use of silence (1) Acoustic recognition of birds 21 / 31

29 Making use of silence (1) Training set: Testing set: Acoustic recognition of birds 21 / 31

30 Making use of silence (1) Training set: Testing set: Acoustic recognition of birds 21 / 31

31 Analogy: the album effect in music artist ID Training set: Express Yourself Bad Acoustic recognition of birds 22 / 31

32 Analogy: the album effect in music artist ID Training set: Express Yourself Bad Testing set: Like a Prayer Smooth Criminal dan.stowell@qmul.ac.uk Acoustic recognition of birds 22 / 31

33 Analogy: the album effect in music artist ID Training set: Express Yourself Bad Testing set: Like a Prayer Smooth Criminal dan.stowell@qmul.ac.uk Acoustic recognition of birds 22 / 31

34 Analogy: the album effect in music artist ID Training set: Express Yourself Bad Testing set: Like a Prayer Smooth Criminal dan.stowell@qmul.ac.uk Acoustic recognition of birds 22 / 31

35 Territorial birds: the territory is the album mel spec features 100 skfl features AUC (%) owl cross-year t within-year t across-year f chaff within-year f chaff across-year aug little owl cross-year 40 pipit within-year pipit across-year chiff chaff within-year chiff chaff across-year 30 standard aug dan.stowell@qmul.ac.uk Acoustic recognition of birds 23 / 31

36 Making use of silence (2) Data augmentation of the TESTING set (adversarial) Measure the distractability of the classifier when mismatched silence is added Measure RMSE in classifier decisions Acoustic recognition of birds 24 / 31

Making use of silence (2) Confusion matrix: linhart2015marcutday1day2_melspec-kfl4pe8kfl4-ms_nr05_pk0_heq0_pool0_rfall_max PC1101 5 PC1102 PC1103 4 PC1104

37 Making use of silence (2) Confusion matrix: linhart2015marcutday1day2_melspec-kfl4pe8kfl4-ms_nr05_pk0_heq0_pool0_rfall_max PC PC1102 PC PC1104 PC True PC1106 PC1107 PC1108 PC PC1110 PC PC1112 PC Estimated dan.stowell@qmul.ac.uk Acoustic recognition of birds 25 / 31

38 Making use of silence (3) Data augmentation of the TRAINING set Each item gets new versions with added silence from each class... Acoustic recognition of birds 26 / 31

39 Making use of silence (4) Finally we can add a new wastebasket class NB not using the a/b labels here... dan.stowell@qmul.ac.uk Acoustic recognition of birds 27 / 31

40 Results Plus silence-test result: 50% AUC Acoustic recognition of birds 28 / 31

42 Conclusions Outdoor bird sound recognition is tricky: The sounds (classes) are highly variable Many potential confounding factors for black-box ML 1. Bird Audio Detection Challenge: Good detection, even in strongly mismatched conditions Adaptation methods useful though, not always needed? 2. Recognising individual bird ID: Strong recognition possible (depending on species) Silence is surprisingly useful for sound recognition! Generally: make more use of mismatched-condition testing Acoustic recognition of birds 30 / 31

43 Thank you Collaborators: 1. Bird Audio Detection Challenge: Mike Wood (U of Salford), Yannis Stylianou (U of Crete), Herve Glotin (U of Toulon), IEEE Signal Processing Society 2. Recognising individual bird ID: Pavel Linhart (Adam Mickiewicz U / Praha U) Machine Listening Lab: dan.stowell@qmul.ac.uk Acoustic recognition of birds 31 / 31

LifeCLEF Bird Identification Task 2016

LifeCLEF Bird Identification Task 2016 The arrival of deep learning Alexis Joly, Inria Zenith Team, Montpellier, France Hervé Glotin, Univ. Toulon, UMR LSIS, Institut Universitaire de France Hervé Goëau,