Literary Data: Some Approaches. it depends. Andrew Goldstone

Transcription

1 Literary Data: Some Approaches Andrew Goldstone March 12, Higher-order functions and dplyr. it depends every program has dependencies software packages (library) data files (readlines, read.csv, scan, dir ) good programs document their dependencies clearly at the start nice programs allow their users to meet dependencies in a controlled fashion. Which is better as a file dependency: "/Users/agoldst/jockers/data/plainText/austen.txt" "austen.txt" "../../../../data/plaintext/austen.txt"

2 file system, once and for all Every R process has a working directory RStudio defaults to ~ the project directory the containing directory of the file you launched RStudio to open Knitting starts a new R process whose working directory is the containing directory of the R markdown file portable dependencies all file paths relative to the working directory working directory set to the directory containing the program script working directory never subsequently modified Testing portability for console testing, start each session by setting the working directory once to the script-containing directory for knitting, do not modify the working directory read your error messages (including in knit PDFs)

3 ha ha ha, character encoding ef bb bf f 6a T h e P r o j e c t G Moral readlines(filename, encoding="utf-8") read.csv(filename, as.is=t,..., encoding="utf-8") scan(..., encoding="utf-8") first class function definitions look like assignments because they are function (...) {... is a value like any other

4 function as parameter bind <- function (x, f) { f(x) bind(c(1, 2, 3), sum) [1] 6 twice <- function (s) { str_c(s, s) bind("ha", twice) [1] "haha" funny function `%p%` <- function (x, y) { x + y 100 %p% 200 [1] 300 `%b%` <- function (x, f) { f(x) "ha" %b% twice [1] "haha"

5 anonymous function bind("parenthetical", function (s) { str_c("(", s, ")") ) [1] "(parenthetical)" map <- function (f, xs) { result <- list() for (j in seq_along(xs)) { result[[j]] <- f(xs[[j]]) result map(twice, c("well", "now", "no")) [[1]] [1] "wellwell" [[2]] [1] "nownow" [[3]] [1] "nono"

6 filter_vector <- function (f, xs) { result <- c() for (x in xs) { if (f(x)) { result <- c(result, x) result pos <- function (x) (x > 0) filter_vector(pos, (-5):5) [1] curry map_f <- function (f) { function (xs) { result <- list() for (j in seq_along(xs)) { result[[j]] <- f(xs[[j]]) result map_f(twice)(c("well", "now", "no")) [[1]] [1] "wellwell" [[2]] [1] "nownow" [[3]] [1] "nono"

7 how would you write filter_f? filter_f <- function (f) { function (xs) { result <- c() for (x in xs) { if (f(x)) { result <- c(result, x) result filter_f(pos)(c(-1, 1)) [1] 1 built-in lapply(lst, f) # same as map(f, lst) lapply(lst, f, x, y,...) #... passed on to f: # inside the for loop: result[[j]] <- f(xs[[j]], x, y,...) sapply (returns a vector if possible) apply (iterate over rows/columns of a matrix) tapply (iterate over groups identified by a factor) what a mess!

8 dplyr: split-apply-combine split a data frame up into pieces (rows, groups of rows ) do something to each piece put together the result No new functionality (but ) select column indexing by name (or $) filter logical row subscripting arrange order mutate expressions in terms of columns summarize for loops, table, sum, mean

9 surnames <- select(laureates, surname) head(surnames) # data type! surname 1 Modiano 2 Munro 3 Yan 4 Tranströmer 5 Vargas Llosa 6 Müller women <- filter(laureates, gender=="female") women$surname[1:3] [1] "Munro" "Müller" "Lessing" (notionally) women <- filter(laureates, function (laur_row) { laur_row$gender == "female" )

10 modularity, but women_surnames <- select(filter(laureates, gender=="female"), surname, year) # ugh curiouser and curiouser women_last <- laureates %>% filter(gender=="female") %>% select(surname, year) women_last[1:3, ] surname year 1 Munro Müller Lessing 2007

11 pipe x %>% f x %b% f f(x) # equivalent to... # or... x %>% f(y, z) # equivalent to... f(x, y, z) # wow!? a special function laureates %>% filter(gender=="female") %>% summarize(total=length(surname)) # think about it... total 1 12 laureates %>% filter(gender=="female") %>% summarize(total=n()) total 1 12

12 laureates %>% filter(!is.na(diedcountrycode)) %>% filter(borncountrycode!= diedcountrycode) %>% summarize(n_exiles=n()) n_exiles 1 34 f_sws <- laureates %>% filter(gender=="female" borncountry=="sweden") %>% select(firstname, surname) %>% slice(1:5) f_sws firstname surname 1 Alice Munro 2 Tomas Tranströmer 3 Herta Müller 4 Doris Lessing 5 Elfriede Jelinek f_sws %>% mutate(fullname=str_c(firstname, surname, sep=" ")) firstname surname fullname 1 Alice Munro Alice Munro 2 Tomas Tranströmer Tomas Tranströmer 3 Herta Müller Herta Müller 4 Doris Lessing Doris Lessing 5 Elfriede Jelinek Elfriede Jelinek

13 f_sws %>% mutate(fullname=str_c(firstname, surname, sep=" ")) %>% select(-firstname) # minus!! surname fullname 1 Munro Alice Munro 2 Tranströmer Tomas Tranströmer 3 Müller Herta Müller 4 Lessing Doris Lessing 5 Jelinek Elfriede Jelinek f_sws %>% mutate(fullname=str_c(firstname, surname, sep=" ")) %>% arrange(surname) %>% select(fullname) fullname 1 Elfriede Jelinek 2 Doris Lessing 3 Alice Munro 4 Herta Müller 5 Tomas Tranströmer

14 laur_ages <- laureates %>% filter(born!= " ") %>% # Mo Yan mutate(born_year=as.numeric(str_sub(born, 1, 4))) %>% mutate(age=year - born_year) %>% mutate(decade=str_c(str_sub(year, 1, 3), 0)) %>% select(surname, year, decade, age) laur_ages %>% slice(1:3) surname year decade age 1 Modiano Munro Tranströmer split laur_ages %>% group_by(decade) # no-op? Source: local data frame [106 x 4] Groups: decade surname year decade age 1 Modiano Munro Tranströmer Vargas Llosa Müller Le Clézio Lessing Pamuk Pinter Jelinek

15 apply-combine laur_ages %>% group_by(decade) %>% summarize(avg_age=mean(age)) Source: local data frame [12 x 2] decade avg_age input txl <- read.csv("three-percent.csv", as.is=t, encoding="utf-8") # N.B. nrow(txl) [1] 3187 colnames(txl) [1] "ISBN" "Titles" "AuthorFN" [4] "AuthorLN" "TranslatorFN" "TranslatorLN" [7] "Publisher" "Genre" "Price" [10] "Month" "Year" "Lanuage" [13] "Country" "OtherFN" "OtherLN" [16] "Other.Role"

16 txl <- txl %>% filter(year < 2015) %>% select(isbn, Year, Country, Genre, Publisher, Language=Lanuage) split, apply, combine txl %>% group_by(year, Genre) %>% summarize(count=n()) Source: local data frame [14 x 3] Groups: Year Year Genre count Fiction Poetry Fiction Poetry Fiction Poetry Fiction Poetry Fiction Poetry Fiction Poetry Fiction Poetry 93

17 split, apply, combine (more than one row) top_n(x, n, wt): top n rows of x by wt lang_counts <- txl %>% group_by(year, Language) %>% summarize(count=n()) win_place_langs <- lang_counts %>% top_n(2, count) %>% arrange(year, desc(count)) win_place_langs Source: local data frame [14 x 3] Groups: Year Year Language count French Spanish Spanish French French Spanish French Spanish French Spanish French Spanish French German 85

18 variety txl %>% group_by(publisher) %>% summarize(langs=n_distinct(language)) %>% arrange(desc(langs)) %>% top_n(5, langs) Source: local data frame [5 x 2] Publisher langs 1 Dalkey Archive 26 2 Archipelago 21 3 Open Letter 20 4 Knopf 17 5 Other Press 17