Hierarchical clustering

Transcription

1 Hierarchical clustering François Husson Applied Mathematics Department - Rennes Agrocampus husson@agrocampus-ouest.fr 1 / 42

2 Hierarchical clustering 1 Introduction 2 Principles of hierarchical clustering 3 Example 4 K-means : a partitioning algorithm 5 Extras Making more robust partitions Clustering in high dimensions Qualitative variables and clustering Combining with factor analysis - clustering 6 Describing classes of individuals 1 / 42

3 Hierarchical clustering 1 Introduction 2 Principles of hierarchical clustering 3 Example 4 Partitioning algorithm : K-means 5 Extras 6 Characterizing classes of individuals 2 / 42

4 Introduction Definitions : Clustering is : making or building classes Class : set of individuals (or objects) with similar shared characteristics Examples of clustering : animal kingdom, computer hard disk, geographic division of France, etc. of classes : social classes, political classes, etc. Two types of clustering : hierarchical : tree partitioning methods 3 / 42

5 Hierarchical example : the animal kingdom 4 / 42

6 Hierarchical clustering 1 Introduction 2 Principles of hierarchical clustering 3 Example 4 Partitioning algorithm : K-means 5 Extras 6 Characterizing classes of individuals 5 / 42

7 A C B D E F H What data? What goals? Clustering is for data tables : rows of individuals, columns of quantitative variables Goals : build a tree structure that : shows hierarchical links between individuals or groups of individuals detects a natural number of classes in the population G 6 / 42

8 Critères Measuring similarity of individuals : Euclidean distance similarity indices etc. Similarity between groups of individuals : minimum jump or single linkage (smallest distance) complete linkage (largest distance) Ward criterion x x x x x x x x x x x x x x 7 / 42

9 Algorithm 7 th grouping 6 th grouping 5 th grouping 4 th grouping 3 rd grouping ABC DEFGH 4.07 ABC DE DE 4.72 FGH ABC DE FG DE 4.72 FG H ABC D E FG D 4.72 E FG H ABC D E F G D 4.72 E F G H A C B D E F G H {ABCDEFGH} {ABC},{DEFGH} {ABC},{DE},{FGH} {ABC},{DE},{FG},{H} {ABC},{D},{E},{FG},{H} {ABC},{D},{E},{F},{G},{H} {AC},{B},{D},{E},{F},{G},{H} {A},{B},{C},{D},{E},{F},{G},{H} 2 nd grouping AC B D E F G B 0.50 D E F G H A B C D E F G B 0.50 C st grouping D E F G H / 42

10 Trees and partitions Hierarchical Clustering Trees always end up... cut through! Click to cut the tree inertia gain Choosing a height to cut at gives a partition Casarsa Parkhomenko YURKOV Lorenzo NOOL BOURGUIGNON MARTINEAU Karlivans BARRAS Uldal HERNU Turi Karpov Clay Sebrle Schoenbeck Ojaniemi Barras Qi Smirnov Gomez Zsivoczky Macey Smith McMULLEN Bernard ZSIVOCZKY Hernu KARPOV SEBRLE Terek Pogorelov Korkizoglou CLAY BERNARD Nool Warners Drews WARNERS Schwarzl Averyanov Remark : given how it was made, the partition is interesting but not optimal 9 / 42

11 Partition quality When is a partition a good one? If individuals placed in the same class are close to each other If individuals in different classes are far from each other Mathematically speaking? small within-class variability large between-class variability = Two criteria. Which one to use? 10 / 42

12 Partition quality x k the mean of the x k, x qk the mean of the x k in class q K Q k=1 q=1 i=1 I (x iqk x k ) 2 = } {{ } total inertia K Q I K Q (x iqk x qk ) 2 + k=1 q=1 i=1 } {{ } within-class inertia k=1 q=1 i=1 I ( x qk x k ) 2 } {{ } between-class inertia x 2 x x x 1 x 3 = 1 criterion only! 11 / 42

13 Partition quality is measured by : Partition quality 0 between-class inertia total inertia 1 inertia between inertia total = 0 = k, q, x qk = x k by variable, classes have the same means Doesn t allow us to classify inertia between inertia total = 1 = k, q, i, x iqk = x qk individuals in the same class are identical Ideal for classifying Warning : don t just accept this criteria at face value : it depends on the number of individuals and classes 12 / 42

14 Ward s method Initialize : 1 class = 1 individual = Between-class inertia = total inertia At each step : combine classes a and b that minimize the decrease in between-class inertia Inertia(a) + Inertia(b) = Inertia(a b) Group together objects with small weights and avoid chain effects xx x x x x x x Saut minimum Ward m am b m a + m b d 2 (a, b) } {{ } to minimize Group together classes with similar centers of gravity xx x x x x *************************** x x Saut minimum Ward Direct use for clustering / 42

15 Hierarchical clustering 1 Introduction 2 Principles of hierarchical clustering 3 Example 4 Partitioning algorithm : K-means 5 Extras 6 Characterizing classes of individuals 14 / 42

16 Temperature data 23 individuals : European capitals 12 variables : mean monthly temperatures over 30 years Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Area Amsterdam West Athens South Berlin West Brussels West Budapest East Copenhagen North Dublin North Elsinki North Kiev East Krakow East Lisbon South London North Madrid South Minsk East Moscow East Oslo North Paris West Prague East Reykjavik North Rome South Sarajevo South Sofia East Stockholm North Which cities have similar weather patterns? How to characterize groups of cities? 15 / 42

17 Temperature data : hierarchical tree Hierarchical clustering Cluster Dendrogram Athens Lisbon Madrid Rome Reykjavik Moscow Minsk Elsinki Oslo Stockholm Dublin Paris London Amsterdam Brussels Copenhagen Kiev Krakow inertia gain Budapest Berlin Prague Sarajevo Sofia 16 / 42

18 Temperature data Loss in between-inertia when going from 23 clusters to 22 clusters: clusters to 21 clusters: clusters to 20 clusters: clusters to 8 clusters: clusters to 7 clusters: clusters to 6 clusters: clusters to 5 clusters: clusters to 4 clusters: clusters to 3 clusters: clusters to 2 clusters: clusters to 1 clusters: 6.76 Sum of losses of inertia = inertia gain Important loss when going from2 clusters to a unique cluster thuswepreferto keep 2 custers 17 / 42

19 Using the tree to build a partition Should we make 2 groups? 3? 4? Cut into 2 groups : between-class inertia total inertia What can we compare this percentage with? = = 56% Athens Lisbon Madrid Rome Reykjavik Moscow Minsk Hierarchical clustering Elsinki Oslo Cluster Dendrogram Stockholm Dublin Paris London Amsterdam Brussels Copenhagen Kiev Krakow Budapest Berlin Prague Sarajevo Sofia 18 / 42

20 Using the tree to build a partition 66 % of the information is contained in this 2-class cut What can we compare this percentage with? Dim 2 (15.40%) Moscow Kiev Budapest Minsk Krakow Sofia Elsinki Oslo Prague Sarajevo Stockholm Berlin Copenhagen Paris Brussels London Amsterdam Madrid Rome Lisbon Athens Reykjavik Dublin Dim 1 (82.90%) 19 / 42

21 Using the tree to build a partition Athens Lisbon Madrid Rome Reykjavik Moscow Minsk Hierarchical clustering Elsinki Oslo Cluster Dendrogram Stockholm Dublin Paris London Amsterdam Brussels Copenhagen Kiev Krakow Budapest Berlin Prague Sarajevo Sofia Separate cold cities into 2 groups : between-class inertia total inertia = = 20% 19 / 42

22 Using the tree to build a partition The move from 23 cities to 3 classes : 56 % + 20 % = 76 % of the variability in the data Dim 2 (15.40%) Moscow Kiev Minsk Krakow Elsinki Oslo Sarajevo Berlin Stockholm Copenhagen Brussels London Budapest Prague Sofia Paris Amsterdam Madrid Rome Lisbon Athens Reykjavik Dublin Dim 1 (82.90%) 20 / 42

23 Determining the number of classes Starting from the tree Depends on the use (survey, etc.) Athens Lisbon Madrid Rome Reykjavik Moscow Hierarchical clustering Minsk Elsinki Cluster Dendrogram Oslo Stockholm Dublin Paris London Amsterdam Brussels Copenhagen Kiev Krakow Budapest Berlin Prague Sarajevo Sofia Plot with the bars Ultimate criterion : interpretability of the classes inertia gain 20 / 42

24 Hierarchical clustering 1 Introduction 2 Principles of hierarchical clustering 3 Example 4 Partitioning algorithm : K-means 5 Extras 6 Characterizing classes of individuals 21 / 42

25 Reykjavik Reykjavik Reykjavik Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Budapest Copenhagen Paris Brussels Amsterdam London Dublin Madrid Rome Lisbon Athens Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Dim 1 ( 82.9 %) Budapest Copenhagen Paris Brussels Amsterdam London Dublin Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Dim 1 ( 82.9 %) Copenhagen Paris Brussels Amsterdam London Dublin Budapest Madrid Rome Lisbon Athens Dim 1 ( 82.9 %) Madrid Rome Lisbon Athens Reykjavik Reykjavik Reykjavik Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Budapest Copenhagen Paris Brussels Amsterdam London Dublin Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Dim 1 ( 82.9 %) Budapest Copenhagen Paris Brussels Amsterdam London Dublin Madrid Rome Lisbon Athens Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Dim 1 ( 82.9 %) Copenhagen Paris Brussels Amsterdam London Dublin Budapest Madrid Rome Lisbon Athens Dim 1 ( 82.9 %) Madrid Rome Lisbon Athens Reykjavik Reykjavik Reykjavik Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Budapest Copenhagen Paris Brussels Amsterdam London Dublin Madrid Rome Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Dim 1 ( 82.9 %) Budapest Copenhagen Paris Brussels Amsterdam London Dublin Lisbon Athens Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Dim 1 ( 82.9 %) Copenhagen Paris Brussels Amsterdam London Dublin Budapest Madrid Rome Lisbon Athens Dim 1 ( 82.9 %) Madrid Rome Lisbon Athens Reykjavik Reykjavik Reykjavik Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Budapest Copenhagen Paris Brussels Amsterdam London Dublin Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Dim 1 ( 82.9 %) Budapest Copenhagen Paris Brussels Amsterdam London Dublin Madrid Rome Lisbon Athens Moscow Minsk Elsinki Oslo Stockholm Kiev Krakow Prague Sofia Sarajevo Berlin Dim 1 ( 82.9 %) Copenhagen Paris Brussels Amsterdam London Dublin Budapest Madrid Rome Lisbon Athens Dim 1 ( 82.9 %) Madrid Rome Lisbon Athens Partitioning algorithm : K-means Algorithm for aggregating around moving centers (K-means) Choose randomly Q centers of gravity Assign the points to the closest center Calculate anew the Q centers of gravity Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) Dim 2 ( 15.4 %) / 42

26 Hierarchical clustering 1 Introduction 2 Principles of hierarchical clustering 3 Example 4 Partitioning algorithm : K-means 5 Extras 6 Characterizing classes of individuals 23 / 42

27 Robustifying a partition obtained using hierarchical clustering The partition obtained by hierarchical clustering is not optimal and can be improved or made robust using K-means Algorithm : use the obtained hierarchical partition to initialize K-means run a few iterations of K-means = potentially improved partition Advantage : more robust partition Disadvantage : loss of hierarchical structure 24 / 42

28 Hierarchical clustering in high dimension If many variables : do PCA and keep only first axes = takes us to classical case If many individuals, hierarchical algorithm is too long Use K-means to partition into around 100 classes Build tree using these classes (weighted by the number of individuals in each class) Gives us the top of the tree 25 / 42

29 Hierarchical clustering in high dimension If many variables : do PCA and keep only first axes = takes us to classical case If many individuals, hierarchical algorithm is too long Use K-means to partition into around 100 classes Build tree using these classes (weighted by the number of individuals in each class) Hierarchical clustering Hierarchical Clustering Gives us the top of the tree Cluster Dendrogram Hierarchical Classification Tree from original data Tree using classes 25 / 42

30 Hierarchical clustering on qualitative data Two strategies : Transform them to quantitative data Do MCA and keep only the first dimensions Do hierarchical clustering using the principal axes of the MCA Use measures/indices suitable for qualitative variables : similarity indices, Jaccard index, etc. 26 / 42

31 Doing factor analysis followed by clustering Qualitative data : MCA outputs quantitative principal components Factor analysis eliminates the last components, which are just noise = more stable clustering x.1 x.k x.k F 1 F Q F K Data PCA Structure Noise 27 / 42

32 Doing factor analysis followed by clustering Representation of the tree and classes on two factor axes = FA gives continuous information, the tree gives discontinuous information. The tree hints at information hidden in further axes Hierarchical clustering on the factor map height cluster 1 cluster 2 cluster 3 Moscow Minsk Elsinki Krakow Oslo Berlin Stockholm Copenhagen Reykjavik Dublin London Kiev Prague Sofia Sarajevo Paris Brussels Amsterdam Dim 1 (82.9%) Budapest Madrid Rome Lisbon Athens / 42

33 Hierarchical clustering 1 Introduction 2 Principles of hierarchical clustering 3 Example 4 Partitioning algorithm : K-means 5 Extras 6 Characterizing classes of individuals 29 / 42

34 The class make-up : using model individuals Model individuals : the ones closest to each class center Cluster 1: Oslo Helsinki Stockholm Minsk Moscow Cluster 2: Berlin Sarajevo Brussels Prague Amsterdam Cluster 3: Rome Lisbon Madrid Athens Dim 2 (15.40%) cluster 1 cluster 2 cluster 3 Moscow Kiev Budapest Minsk Krakow cluster 1 Prague Elsinki Oslo Sarajevo Sofia Berlin Stockholm Copenhagen cluster 2 Paris Brussels Amsterdam London Athens Madrid Rome cluster 3 Lisbon Reykjavik Dublin Dim 1 (82.90%) 30 / 42

35 Characterizing/describing classes Goals : Find the variables which are most important for the partition Characterize a class (or group of individuals) in terms of quantitative variables Sort the variables that best describe the classes Questions : Which variables best characterize the partition How can we characterize individuals in the 1st class? Which variables describe them best? 31 / 42

36 Characterizing/describing classes Which variables best represent the partition? For each quantitative variable : build an analysis of variance model between the quantitative variable and the class variable do a Fisher test to detect class effect Sort the variables by increasing p-value Eta2 P-value October e-10 March e-10 November e-09 September e-09 April e-08 February e-08 December e-07 January e-06 August e-06 July e-05 May e-04 June e / 42

37 Characterizing classes using quantitative variables January February March April May June July August September October November December Elsinki Kiev Minsk Moscow Oslo Reykjavik Stockholm Amsterdam Berlin Brussels Budapest Copenhagen Dublin Krakow London Paris Prague Sarajevo Sofia Athens Lisbon Madrid Rome / 42

38 Characterizing classes using quantitative variables 1st idea : if the values of X for class q seem to be randomly drawn from all the values of X, then X doesn t characterize class q. Random January Temperature 2nd idea : the more a random draw appears unlikely, the more X characterizes class q. 34 / 42

39 Characterizing classes using quantitative variables Idea : use as reference a random draw of n q values from N What values can x q take? (i.e., what is the distribution of X q?) ( ) E( X q ) = x V( X q ) = s2 N nq n q N 1 L( X q ) = N because X q is a mean = Test statistic = s 2 n q x q x ( N nq N 1 ) N (0, 1) If test statistic 1.96 then X characterizes class q and the more the test statistic is large, the better X characterizes class q. Idea : rank the variables by decreasing test statistic 35 / 42

40 Characterizing classes using quantitative variables $quanti$ 1 v.test Mean in Overall sd in Overall p.value category mean category sd July June August May September January December November April February October March / 42

41 Characterizing classes using quantitative variables $ 2 NULL $ 3 v.test Mean in Overall sd in Overall p.value category mean category sd September October August November July April March February June December January May / 42

42 Characterizing classes using qualitative variables Which variables best characterize the partition? For each qualitative variable, do a χ 2 test between it and the class variable Sort the variables by increasing p-value $test.chi2 p.value df Area / 42

43 Characterizing classes using qualitative variables Does the South category characterize the 3rd class? Cluster 3 Other cluster Total South n mc = 4 1 n m = 5 Not south Total n c = 4 19 n = 23 n Test : H 0 : mc n c = nm n versus H 1 : m abnormally overrepresented in c Under H 0 : L(N mc ) = H(n c, nm n, n) P H 0 (N mc n mc ) Cluster 3 Cla/Mod Mod/Cla Global p.value v.test Area=South = 80 ; = 100 ; = ; P H(4, 23 5,23) [Nmc 4] = = H 0 rejected, South is overrepresented in the 3rd class Sort the categories in terms of p-values 39 / 42

44 Characterizing classes using factor axes These are also quantitative variables $ 1 v.test Mean in Overall sd in Overall p.value category mean category sd Dim $ 2 v.test Mean in Overall sd in Overall p.value category mean category sd Dim $ 3 v.test Mean in Overall sd in Overall p.value category mean category sd Dim / 42

45 Conclusions Clustering can be done on tables of individuals vs quantitative variables MCA transforms qualitative variables into quantitative ones hierarchical clustering gives a hierarchical tree number of classes K-means can be used to make classes more robust Characterize classes by active and supplementary variables, quantitative or qualitative 41 / 42

46 Introduction Principles of hierarchical clustering Example K-means Extras Describing the classes found More nalysis Series Husson Lê Pagès Exploratory Multivariate Analysis by Example Using R Second Edition Chapman & Hall/CRC Computer Science & Data Analysis Series Exploratory Multivariate Analysis by Example Using R SECOND EDITION Husson F., Lê S. & Pagès J. (2017) Exploratory Multivariate Analysis by Example Using R 2nd edition, 230 p., CRC/Press. François Husson Sébastien Lê Jérôme Pagès The FactoMineR package for performing clustering : Movies on Youtube : a Youtube channel: youtube.com/hussonfrancois a playlist with 11 movies in English a playlist with 17 movies in French 42 / 42