15-826: Multimedia Databases and Data Mining. Outline. Must-read Material. Motivation: (Q1) Find patterns in data Motion capture data: broad jumps

Transcription

1 Must-read Material : Multimedia Databases and Data Mining Lecture #22: Independent Component Analysis (ICA) Jia-Yu Pan and Christos Faloutsos AutoSplit: Fast and Scalable Discovery of Hidden Variables in Stream and Multimedia Databases, Jia-Yu Pan, Hiroyuki Kitagawa, Christos Faloutsos and Masafumi Hamamoto, PAKDD 2004, Sydney, Australia (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #2 (Q1) Find patterns in data Motion capture data: broad jumps Energy exerted Left Knee Energy exerted Right Knee Take-off Landing (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #4 1

2 (Q1) Find patterns in data Human would say Pattern 1: along diagonal Pattern 2: along vertical axis How to find these automatically? R:L=60:1 R:L=1:1 Take-off Landing Each point is the measurement at a time tick (total 550 points) (c) C. Faloutsos and J-Y Pan (2017) #5 Alcoa American Express Boeing Citi Group (Q2) Find hidden variables Stock prices Hidden variables (= topics = concepts) General trend Internet bubble (c) C. Faloutsos and J-Y Pan (2017) #6 (Q2) Find hidden variables (Q2) Find hidden variables Caterpillar Intel Caterpillar Intel B1,CAT B1,INTC? B2,CAT B2,INTC General trend Hidden variables Internet bubble?? Hidden variable 1 Hidden variable (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #8 2

3 Find hidden variables There are two sound sources in a cocktail party = blind source separation (= we don t know the sources, nor their mixing) (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #10 Formulation: Finding patterns Linear representation Given n data points, each with m attributes. Find patterns that describe data properties the best (c) C. Faloutsos and J-Y Pan (2017) #11 Find vectors that describe the data set the best. Each point: linear combination of the vectors (patterns): xi = h i, 1b1 + h i,2b (c) C. Faloutsos and J-Y Pan (2017) #12 3

4 Patterns as data vocabulary Good pattern sparse coding Only b 1 is needed to describe x i. (Q) Given data x i s, compute h i,j s and b i s that are sparse? (c) C. Faloutsos and J-Y Pan (2017) #13 Patterns in motion capture data b 2 b 1 n=550 ticks Left x x 1,1 2,1 Right x n,1 x n,2 Data matrix x1,2 h1,1 h1,2 x 2,2 h2,1 h2,2 b1 = "? " b2? " " h n,1 h n,2 = H B X nx 2 nx 2 2x2 Hidden variables Sparse ~ non-gaussian ~ Independent Basis vectors Independent : e.g., minimize mutual (c) information. C. Faloutsos and J-Y Pan (2017) #14 Patterns in motion capture data x x 1,1 2,1 x n,1 x n,2 Data matrix x1,2 h1,1 h1,2 x 2,2 h2,1 h2,2 b1 = "? " b2? " " h n,1 h n,2 = H B X nx 2 nx 2 2x2 Hidden variables Sparse ~ non-gaussian ~ Independent Basis vectors X = (U Λ ) V T (c) C. Faloutsos and J-Y Pan (2017) #15 Find topics in documents Hidden variables in stock prices (c) C. Faloutsos and J-Y Pan (2017) #16 4

5 Pattern discovery with ICA: AutoSplit [PAKDD 04][WIRI 05] Video frames Stock prices Text documents or or (Q) Different modalities Step 1: Data points (matrix) Step 2: Compute patterns Step 3: Interpret patterns Data mining (Case studies) (Q) What pattern? (Q) How? Finding patterns in highdimensional data PCA finds the hyperplane. Dimensionality reduction details ICA finds the correct patterns (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #18 Find topics in documents Hidden variables in stock prices Visual vocabulary for retinal images (c) C. Faloutsos and J-Y Pan (2017) #19 Topic discovery on text streams Data: CNN headline news (Jan.-Jun. 1998) Documents of 10 topics in one single text stream Documents are sorted by date/time Subsequent documents may have different topics Topic 1 Topic 3 Topic 1 Date/Time (c) C. Faloutsos and J-Y Pan (2017) #20 5

6 Topic discovery on text streams Data: CNN headline news (Jan.-Jun. 1998) Documents of 10 topics in one single text stream FIND: the document boundaries AND: the terms of each topic Topic discovery on text streams Known: number of topics = 10 Unknown: (1) topic of each document (2) topic description Date/Time (c) C. Faloutsos and J-Y Pan (2017) #21?? Topic 1 Topic 3 Topic 1 Date/Time (c) C. Faloutsos and J-Y Pan (2017) #22 Step 1 Step 2 Topic discovery in documents New stories Windowing X [nxm ] = H [nxm ] B [m xm ] (1) Find hyperplane (m =10) (2) Find patterns (n=1659) (30 words) X [nxm] Step 3 aaron x i = [1, 5,, 0] (c) C. Faloutsos and J-Y Pan (2017) #23 zoo m=3887 (dictionary size) ' b 1 ' b2 ' b10 (Q) B [10x3887] What does b i mean? aaron animal zoo b i = [0, 0.7,, 0.6] ' b 1 ' b2 ' b10 Topics found Step 3: Interpret the patterns ID aaron animal b i = [0, 0.7,, 0.6] zoo m=3887 (dictionary size) Sorted word list Top words : animal, zoo, A hidden topic A Mckinne Sergeant sexual Major Armi B bomb Rudolph Clinic Atlanta Birmingham C Winfrei Beef Texa Oprah Cattl D Viagra Drug Impot Pill Doctor E Zamora Graham Kill Former Jone F Medal Olymp Gold Women Game General idea: related to the data attributes G Pope Cube Castro Cuban Visit H Asia Economi Japan Econom Asian I Super (c) Bowl C. Faloutsos and Game J-Y Pan (2017) Team Re #24 J Peopl Tornado Florida Re bomb 6

7 Step 3: Evaluate the patterns ID True Topic 1 Sgt. Gene Mckinney is on trial for alleged sexual misconduct 2 A bomb explodes in a Birmingham, AL abortion clinic 3 The Cattle Industry in Texas sues Oprah Winfrey for defaming beef 4 New impotency drug Viagra is approved for use 5 Diane Zamora is convicted of helping to murder her lover s girlfriend 6 ID 1998 Winter Olympic games Sorted word list 7 A The mckinne Pope s historic sergeant visit to sexual Cube in Winter major 1998 armi 8 B The bomb economic rudolph crisis in Asia clinic atlanta birmingham 9 C Superbowl winfrei XXXII beef texa oprah cattl 10 D Tornado viagra in Florida drug Impot pill doctor E zamora graham kill former jone AutoSplit finds correct topics (c) C. Faloutsos and J-Y Pan (2017) #25 Step 3: Evaluate the patterns ID AutoSplit A mckinne sergeant sexual major armi B bomb rudolph clinic atlanta birmingham C winfrei beef texa oprah cattl D viagra drug Impot pill doctor E zamora graham kill former jone ID PCA A mckinne bomb women sexual sergeant B bomb mckinne rudolph clinic atlanta C winfrei viagra texa beef oprah D viagra winfrei drug texa beef E zamora viagra winfrei graham olymp AutoSplit s topics are better than PCA (c) C. Faloutsos and J-Y Pan (2017) #26 Step 3: Evaluate the patterns AutoSplit A Topic 1 B C D Topic 2 E PCA A B C D PCA vectors mix the topics. E AutoSplit s topics are better than PCA (c) C. Faloutsos and J-Y Pan (2017) #27 Find topics in documents Hidden variables in stock prices (c) C. Faloutsos and J-Y Pan (2017) #28 7

8 Find hidden variables (DJIA stocks) Formulation: Find hidden variables Weekly DJIA closing prices 01/02/ /05/2002, n=660 data points A data point: prices of 29 companies at the time Alcoa American Express Boeing AA1,, XOM1 AAn,, XOMn Date = H11, H12,, H1m? Hn1, Hn2,, Hnm B11, B12,, B1m Hidden variable? Bm1, Bm2,, Bmm Caterpillar Citi Group Date (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #30 Caterpillar Characterize hidden variable by the companies it influences B1,CAT B1,INTC Intel B2,INTC B2,CAT Companies related to hidden variable 1 B1,j Highest Lowest Caterpillar AT&T Boeing WalMart MMM Intel Coca Cola Home Depot Du Pont Hewlett-Packard General trend Internet bubble (c) C. Faloutsos and J-Y Pan (2017) #31 General trend (c) C. Faloutsos and J-Y Pan (2017) #32 8

9 Companies related to hidden variable 1 B1,j General trend (and outlier) Highest Lowest Caterpillar AT&T Boeing WalMart MMM Intel Coca Cola Home Depot Du Pont Hewlett-Packard All companies are affected by the general trend variable (with weights 0.6~0.9), except AT&T. AT&T United Technologies Walmart Exxon Mobil General trend (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #34 Companies related to hidden variable 2 Companies related to hidden variable 2 B2,j Highest Lowest Intel Philip Morris Hewlett-Packard International Paper GE Caterpillar American Express Procter and Gamble Disney Du Pont Tech company Internet bubble (c) C. Faloutsos and J-Y Pan (2017) #35 Highest B2,j Lowest Intel Philip Morris Hewlett-Packard International Paper GE Caterpillar American Express Procter and Gamble Tech company Disney Du Pont Companies affected by the internet bubble variable (with weights 0.5~0.6) are tech-related. Other companies are un-related (weights < 0.15) (c) C. Faloutsos and J-Y Pan (2017) #36 9

10 Find topics in documents Hidden variables in stock prices Visual vocabulary for retinal images Conclusion ICA: more flexible than PCA in finding patterns. Many applications Find topics and vocabulary for images Find hidden variables in time series (e.g., stock prices) Blind source separation Rule of thumb: plot after PCA; if chicken-feet, try ICA ICA PCA (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #38 Citation AutoSplit: Fast and Scalable Discovery of Hidden Variables in Stream and Multimedia Databases, Jia-Yu Pan, Hiroyuki Kitagawa, Christos Faloutsos and Masafumi Hamamoto PAKDD 2004, Sydney, Australia References Jia-Yu Pan, Andre Guilherme Ribeiro Balan, Eric P. Xing, Agma Juci Machado Traina, and Christos Faloutsos. Automatic Mining of Fruit Fly Embryo Images. In Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Arnab Bhattacharya, Vebjorn Ljosa, Jia-Yu Pan, Mark R. Verardo, Hyungjeong Yang, Christos Faloutsos, and Ambuj K. Singh. ViVo: Visual Vocabulary Construction for Mining Biomedical Images. In Proceedings of the Fifth IEEE International Conference on Data Mining (ICDM), Masafumi Hamamoto, Hiroyuki Kitagawa, Jia-Yu Pan, and Christos Faloutsos. A Comparative Study of Feature Vector-Based Topic Detection Schemes for Text Streams. In Proceedings of International Workshop on Challenges in Web Information Retrieval and Integration (WIRI), 2005, pp Jia-Yu Pan, Hiroyuki Kitagawa, Christos Faloutsos, and Masafumi Hamamoto. AutoSplit: Fast and Scalable Discovery of Hidden Variables in Stream and Multimedia Databases. In Proceedings of the The Eighth Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #40 10

11 References Aapo Hyvärinen, Juha Karhunen, Erkki Oja: Independent Component Analysis, John Wiley & Sons, 2001 Software Open source software: fastica Or autosplit : (c) C. Faloutsos and J-Y Pan (2017) # (c) C. Faloutsos and J-Y Pan (2017) #42 11