MonetDB & R. amst-r-dam meet-up, Hannes Mühleisen

Size: px

Start display at page:

Download "MonetDB & R. amst-r-dam meet-up, Hannes Mühleisen"

Phyllis Johnson
5 years ago
Views:

1 MonetDB & R amst-r-dam meet-up, Hannes Mühleisen

2 Collect data Growing Load data Filter, transform & aggregate data Analyze & Plot Not really Analysis features Publish paper/ Profit

3 Problem: #BiggeR 00 R CSV Loading Time (s) MB 690 MB 6 GB Dataset X R

4 Running Example Say you are Starfleet Research and want to analyze warp drive performance (Coil Flux) Lots of data (~G CSV, 68M records) class,speed,flux NX,, Constitution,,5 Galaxy,, Defiant,,3 Intrepid,, NX,,5

5 Solution? Use optimized data management system for data loading & retrieval... like a relational database... like a analytics-optimized database

6 Solution? Use optimized data management system for data loading & retrieval... like a relational database... like a analytics-optimized database... like MonetDB!

7 Relational DBs 0 class speed flux NX Constitution Galaxy Defiant Intrepid

8 Postgres, Oracle, DB2, etc.: Conceptional class speed flux NX Constitution Galaxy Defiant Intrepid Physical (on Disk) NX 3 Constitution 8 Galaxy 3 Defiant 6 Intrepid

9 Column Store: class speed flux NX Constitution Galaxy Defiant Intrepid NX Constitution Galaxy Defiant Intrepid Peter A. Boncz, Martin L. Kersten, and Stefan Manegold Breaking the memory wall in MonetDB. Commun. ACM 5, 2 (December 2008), DOI=0.45/

10 Why Columns? TPC H SF 00 Hot runs Average time (s) monetdb postgres log! Query

11 First Gains CSV Loading Time (s) MB 690 MB 6 GB Dataset X R MonetDB

12 But then... data <- dbgetquery(conn," SELECT t,count(t) AS ct FROM ( SELECT CAST(flux as integer) AS t FROM starships WHERE ( (speed = 5) ) AND ( (class = 'NX') ) ) AS t WHERE t > 0 GROUP BY t ORDER BY t LIMIT 00; ") normalized <- data$ct/sum(data$ct)...do we really want this?

13 Enter monet.frame The virtual data object for R data <- monet.frame(conn,"starships") nxw5 <- subset(data,class=="nx" & speed==5)$flux t <- tabulate(nxw5,00) normalized <- t/sum(t) R-style data manipulation & aggregation

14 Meanwhile Behind the scenes: data <- monet.frame(conn,"starships") SELECT * FROM starships; nxw5 <- subset(data,class=="nx" & speed==5)$flux SELECT * FROM starships WHERE class = 'NX' AND speed = 5; SELECT flux FROM starships WHERE class = 'NX' AND speed = 5; t <- tabulate(nxw5,00) SELECT t,count(t) AS ct FROM (SELECT CAST(flux as integer) AS t FROM starships WHERE class = 'NX' AND speed = 5) AS t WHERE t > 0 GROUP BY t ORDER BY t LIMIT 00; Actually executed

15 Implementation # R core unique <- function(x, incomparables = FALSE,...) UseMethod("unique") # MonetDB.R unique.monet.frame <- function (x, incomparables = FALSE,...) as.vector(.col.func(x,"distinct",num=false,aggregate=true)) # On Shell unique(wcflux$flux) # result query: SELECT DISTINCT(flux) FROM starships;

16 Flux Analysis Script wcflux <- read.table("starships.csv",sep=",",header=t) classes <- sort(unique(wcflux$class)) wcflux5 <- subset(wcflux,speed==5)[c("class","flux")] plot(0,0,ylim = c(0,0.),xlim = c(0,00),type = "n") for(i in :length(classes)){ tclass <- classes[[i]] ct <- tabulate(subset(wcflux5,class==tclass)$flux,00) normalized <- ct/sum(ct) lines(data.frame(x=seq(,00),y=normalized)) }

17 Density Plot of Warp Coil Flux per Starship Class (Warp 5) Density n= Starship Class Constitution Defiant Galaxy Intrepid NX Warp Coil Flux (mc)

18 Flux Analysis Script (2) wcflux <- monet.frame(conn,"starships") changed! classes <- sort(unique(wcflux$class)) wcflux5 <- subset(wcflux,speed==3)[c("class","flux")] plot(0,0,ylim = c(0,0.2),xlim = c(0,60),type = "n") for(i in :length(classes)){ tclass <- classes[[i]] ct <- tabulate(subset(wcflux5,class==tclass)$flux,60) normalized <- ct/sum(ct) lines(data.frame(x=seq(,60),y=normalized)) }

19 Generated SQL SELECT DISTINCT(class) FROM starships; SELECT t,count(t) AS ct FROM (SELECT CAST(flux as integer) AS t FROM starships WHERE ( (speed = 3) ) AND ( (class = 'Constitution') ) ) AS t WHERE t > 0 GROUP BY t ORDER BY t LIMIT 60; -- [...]

Performance 50 20 2,5 44 Time (s) 75 20 86 37,5 0 70 0 Plain R

20 Performance ,5 44 Time (s) , Plain R PostgreSQL + R monet.frame System Execution Loading Import

21 Demo

22 Collect data Load data Filter, transform & aggregate data Analyze & Plot Publish paper

23 sd() ^ range() log() subset() exp() / na.omit() sin() summary() sample() trunc() + str() $ * sort() Thank You! Questions? min() abs() sum() max() - round() names() dim() sign() merge() sqrt() tabulate() floor() ceiling() [] tail() range() head() quantile() length() == aggregate() signif() print() var() CRAN: MonetDB.R

New Assessment Tool for AT-Fieldtest and Monitoring

31 st Conference of the European Working Group on Acoustic Emission (EWGAE) Th.1.A.4 More Info at Open Access Database www.ndt.net/?id=17549 New Assessment Tool for AT-Fieldtest and Monitoring Abstract