Diet Networks: Thin Parameters for Fat Genomics

Transcription

1 Institut des algorithmes d apprentissage de Montréal Diet Networks: Thin Parameters for Fat Genomics Adriana Romero, Pierre Luc Carrier, Akram Erraqabi, Tristan Sylvain, Alex Auvolat, Etienne Dejoie, Marc-André Legault, Marie-Pierre Dubé, Julie G. Hussin, Yoshua Bengio

2 Outline Motivation & Challenges Deep Learning Architectures Diet Networks Results Wrap up and future research directions

3 Motivation & Challenges

4 Motivation Deep Learning Zhou et al., 2015 from from

5 Motivation Deep Learning Genomics from from from

6 Genomic Data as Fat Data Target millions of simple variants across the genome (SNPs). Number of participants limited, even for large datasets. # participants (samples) # SNP (features) Fat Data Dire imbalance between # samples and # input features

7 Challenges: Parameter explosion Linear classifier: Naive setup for SNP data: W (parameters) # inputs = hundreds of thousands # samples = thousands # samples << # parameters In deep networks: the # parameters in the 1st layer grows linearly with the # inputs. # inputs = # parameters

8 Challenges: Overfitting from

9 Challenges: The curse of dimensionality Considering: - 300K SNPs - 3 possible values (0, 1, 2) from K combinations!!

10 Why deep learning? Capturing information directly from the raw input data is not trivial and often involves complex and non-linear functions. Many problems become easier if the input data is transformed into a representation that emphasizes its most relevant characteristics.

11 Multi-Layer Perceptron (MLP) Supervised learning: desired output values are provided Describe data as a hierarchy of concepts Unsupervised learning: aims to discover hidden structure in the input data.

12 CNN - reducing the number of parameters Parameter sharing. Exploit spatially local correlations. Suitable for data with a grid-like topology. Problem: When the full DNA sequence is unavailable, other type of methods seem more appropriate.

13 Diet Networks

14 The idea Use a novel neural network reparametrization, which considerably reduces the number of free parameters when the input is very high-dimensional and orders of magnitude larger than the number of training samples.

15 The model Input data: Fx100 NxF, N << F 100 MLP MLP MLP 50K 500 MLP 100 Emb. Emb. Fx100 30M 300K Input = 1 sample 1xF Input = 1 feature (SNP) 1xN

16 Embeddings Raw (learnt embedding, end to end training) MLP MLP Per class histograms MLP Emb.

17 Per class histogram Individuals Class 1: 1 x 0, 2 x 1, 1 x SNPs Class 2: 3 x 0, 0 x 1, 1 x

18 The 1000 Genomes Project (1) Large-scale comparison of DNA sequences from populations, thanks to the presence of genetic variations. Represents 26 populations from 5 geographical regions, in total 3,450 individuals SNP inclusion/exclusion criteria: Genetic variants with frequencies of at least 5% Excluded SNPs positioned on sex chromosomes Only included SNPs in approximate linkage equilibrium with each other As a result, we obtained 315,345 SNPs, encoded as having 0, 1 or 2 copies of a genetic mutation (non-reference nucleotide).

19 Experimental setup Ethnicity prediction from SNPs on 1000 Genomes data. Metric: misclassification error and number of free parameters. 5-fold crossvalidation.

20 Quantitive results (1) Embedding Misclassification error (%) # free parameters Without reconstruction Basic MLP M Diet Networks (raw end2end) k Diet Networks (histograms) k With reconstruction Basic MLP M Diet Networks (raw end2end) k Diet Networks (histograms) k

21 Quantitive results (2) Embedding Misclassification error (%) Diet Networks (histograms) PCA (10 PCs) PCA (50 PCs) PCA (100 PCs) PCA (200 PCs)

22 Quantitive results (3)

23 What is the network learning? Layer 2 MLP Layer 1 MLP Input

24 What is the network learning? Raw input Layer 1 Layer 2 Ethnicities

25 What is the network learning? Raw input Layer 1 Layer 2 Continents

26 Wrap up and future research directions

27 Wrap up We demonstrated the potential of deep learning models to tackle genomicspecific tasks. The parameter explosion introduced by high dimensional genomic data can be mitigated by smart model parameterization, such as Diet Networks.

28 What comes next Conducting genetic association studies, with emphasis on population-aware analyses of SNP data in disease cohorts. Identify the genetic basis of common diseases to achieve a better patient risk prediction and improve our overall understanding of disease etiology.

29 Institut des algorithmes d apprentissage de Montréal Diet Networks: Thin Parameters for Fat Genomics Adriana Romero, Pierre Luc Carrier, Akram Erraqabi, Tristan Sylvain, Alex Auvolat, Etienne Dejoie, Marc-André Legault, Marie-Pierre Dubé, Julie G. Hussin, Yoshua Bengio Thank Code: