Working with data. Garrett Grolemund. PhD Student / Rice Univeristy Department of Statistics

Size: px

Start display at page:

Download "Working with data. Garrett Grolemund. PhD Student / Rice Univeristy Department of Statistics"

Bethanie Ferguson
5 years ago
Views:

1 Working with data Garrett Grolemund PhD Student / Rice Univeristy Department of Statistics Sept 2010

2 1. Loading data 2. Data structures & subsetting 3. Strings vs. factors 4. Combining data 5. Exporting data

3 The data Global school based healthy survey Three countries: Uganda, The Philippines and the United Arab Emirates Variables related to diet and hand washing.

4 Loading data

5 1. Plain text 2. Excel 3. Other stats packages 4. Databases

6 Plain text data stored as text + some type of delimiter to separate the entries

7 tab separated Houston Dallas El Paso read.delim("filepath")

8 separated Houston Dallas El Paso read.delim("filepath", sep = " ")

9 , separated Houston,15000,43,19 Dallas,30000,21,45 El Paso,12000,78,02 read.csv("filepath")

10 fixed width Houston Dallas El Paso read.fwf("filepath")

11 # load our data set gshs <- read.csv("gshs.csv") # Hint: make sure gshs.csv is in your working directory!

12 Tips # If you know what is used for missing values, use it read.csv(file, na.string = ".") read.csv(file, na.string = "-99") # Use count.fields to check the number of # observations in each column. The following # call uses the same default as read.csv count.fields(file, sep = ",", quote = "", comment.char = "")

13 Your turn Practice loading all the files in the tricky directory. Use?read.csv, etc. for help.

14 read.csv("tricky-1.csv") read.csv("tricky-2.csv", header = FALSE) read.delim("tricky-3.csv", sep = " ") count.fields("tricky-4.csv", sep = ",")

15 This is what I always do Excel Save as csv. (Use VBA to automate) RODBC::odbcConnectExcel R-data.html#RODBC (uses excel) xlsx::read.xlsx (uses java) gdata::read.xls (uses perl)

16 Other stats packages library(foreign) help(package = foreign)?read.spss?read.dbf?read.xport # If that doesn't work, I've heard good # things about stattransfer: #

17 Databases library(rodbc) library(rmysql) library(rpostgresql) library(roracle) library(rsqlite) # A complete list: # index.html

18 Your turn Work through the code in 02-sqlite.r to create a database, load data and then pull some out.

19 Data structures

20 Data types Class Example numeric 3.14 character logical Rice TRUE etc....

21 1d Vector List 2d Matrix Data frame nd Array Same types Different types

22 str()

23 Variables Size to append 1d names() length() c() 2d colnames() rownames() ncol() nrow() cbind() rbind() nd dimnames() dim() abind() (special package)

24 Subsetting Vectors x[1:4] Matrices Arrays Lists x[1:4, ] x[, 2:3, ] x[[1]] x$name x[1:4,, drop = F] x[1]

25 blank include all integer logical character +: include -: exclude include TRUEs e.g, x[v1 > 50, ] lookup by name

26 Aside: never use attach! Non-local effects; not symmetric; implicit, not explicit. Makes it very easy to make mistakes. Use with() instead. with(bnames, table(year, length))

27 Your turn What type of data structure is gshs? Which variables of gshs as saved as numerics? Display (only) the first 5 rows of gshs.

28 Strings vs factors

29 Factors R s way of storing categorical data Have ordered levels() which: Control order on plots and in table() Are preserved across subsets Affect contrasts in linear models

30 # Creating a factor x <- sample(5, 20, rep = T) a <- factor(x) b <- factor(x, levels = 1:10) c <- factor(x, labels = letters[1:5]) levels(a); levels(b); levels(c) table(a); table(b); table(c)

31 # Subsets b2 <- b[1:5] levels(b2) table(b2) # Remove extra levels b2[, drop=t] factor(b2) # Convert to character b3 <- as.character(b) table(b3) table(b3[1:5])

32 as.numeric(a) as.numeric(b) as.numeric(c) d <- factor(x, labels = 2^(1:5)) as.numeric(d) as.character(d) as.numeric(as.character(d))

33 Character vs. factor Characters don t remember all levels. Tables of characters always ordered alphabetically By default, strings converted to factors when loading data frames. Use stringsasfactors = F to turn off for one data frame, or options(stringsasfactors = F) for all

34 Character vs. factor Use a factor when there is a well-defined set of all possible values. Use a character vector when there are potentially infinite possibilities.

35 Possible values Order Character Anything Alphabetical Factor Ordered factor Fixed and finite Fixed, but arbitrary (default is alpbpbetical Fixed and meaningful

36 Quiz Take one minute to decide which data type is most appropriate for each of the following variables collected in a medical experiment: Subject id, name, sex, address, race, eye colour, birth city, birth country, age, amount of fruit eaten every day.

37 gshs$age <- factor(gshs$age, levels = c("11-", "12", "13", "14", "15", "16+")) gshs$fruit <- factor(gshs$fruit, levels = c("0", "<1", "1", "2", "3", "4", "5+")) gshs$vegetables <- factor(gshs$vegetables, levels = c("0", "<1", "1", "2", "3", "4", "5+")) gshs$teeth <- factor(gshs$teeth, levels = c("0", "<1", "1", "2", "3", "4+")) freq <- c("never", "Rarely", "Sometimes", "Most of the time", "Always") gshs$hands_eating <- factor(gshs$hands_eating, levels = freq) gshs$hands_toilet <- factor(gshs$hands_toilet, levels = freq) gshs$hands_soap <- factor(gshs$hands_soap, levels = freq) gshs$hungry <- factor(gshs$hungry, levels = freq)

38 Combining data

39 Combining datasets Name instrument John guitar Paul bass George guitar Ringo drums Stuart bass Pete drums Name band John T Paul T + George T =? Ringo T Brian F

40 base::merge - fully featured, but complex, slow and reorders output plyr::join - minimalist, but fast and simple.

41 x Name instrument John guitar Paul bass George guitar Ringo drums Stuart bass Pete drums y Name band John T Paul T + George T = Ringo T Brian F Name instrument band John guitar T Paul bass T George guitar T Ringo drums T Stuart bass NA Pete drums NA join(x, y, type = "left")

42 x Name instrument John guitar Paul bass George guitar Ringo drums Stuart bass Pete drums y Name band John T Paul T + George T = Ringo T Brian F Name instrument band John guitar T Paul bass T George guitar T Ringo drums T Brian NA F join(x, y, type = "right")

43 x Name instrument John guitar Paul bass George guitar Ringo drums Stuart bass Pete drums y Name band John T Paul T + George T = Ringo T Brian F Name instrument band John guitar T Paul bass T George guitar T Ringo drums T join(x, y, type = "inner")

44 x Name instrument John guitar Paul bass George guitar Ringo drums Stuart bass Pete drums y Name band John T Paul T + George T = Ringo T Brian F Name instrument band John guitar T Paul bass T George guitar T Ringo drums T Stuart bass NA Pete drums NA Brian NA F join(x, y, type = "full")

45 Type "left" "right" "inner" "full" Action Include all of x, and matching rows of y Include all of y, and matching rows of x Include only rows in both x and y Include all rows

46 Saving data

47 Your turn Guess the name of the function you might use to write an R object back to a csv file on disk. Use it to save gshs to gshs-2.csv. What happens if you now read in gshs-2.csv with read.csv?

48 write.csv(gshs, "gshs-2.csv") gshs2 <- read.csv("gshs-2.csv") head(gshs) head(gshs2) str(gshs) str(gshs2) # Better, but still loses factor levels write.csv(gshs, file = "gshs-3.csv", row.names = F) gshs3 <- read.csv("gshs-3.csv")

49 Saving data # For long-term storage write.csv(data.frame, file = "path", row.names = F) # For short-term caching save(gshs, file = "gshs.rdata")

50 .csv read.csv() write.csv( row.names = F) Only data frames Can be read by any program Long term storage.rdata load() save() Any R object Only by R Short term caching of expensive computations

51 My workflow One file that cleans up all input and saves both a csv for long-term storage and rdata file for short-term use. Track csv file with source code control so it s easy to see what changes over time.

52 Compression Easy to store compressed files to save space: write.csv(gshs, file = bzfile("gshs.csv.bz2"), row = F) gshs4 <- read.csv("gshs.csv.bz2") Files stored with save() are automatically compressed.

53 Resources

54 Other resources Always a good place to start. Data Manipulation with R, Phil Spector. A different perspective. Full review at

56 This work is licensed under the Creative Commons Attribution-Noncommercial 3.0 United States License. To view a copy of this license, visit 3.0/us/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.

dplyr: manipulating your data

dplyr: manipulating your data Washington University in St. Louis September 14, 2016 (Washington University in St. Louis) dplyr: manipulating your data September 14, 2016 1 / 44 1 Overview Data manipulation