DSC 201: Data Analysis & Visualization

Size: px

Start display at page:

Download "DSC 201: Data Analysis & Visualization"

Meredith Silvia Park
5 years ago
Views:

1 DSC 201: Data Analysis & Visualization Data Transformation Dr. David Koop

2 Data Wrangling Data wrangling: transform raw data to a more meaningful format that can be better analyzed Data cleaning: getting rid of inaccurate data Data transformations: changing the data from one representation to another Data reshaping: reorganizing the data Data merging: combining two datasets 2

3 Example: Football Game Data Data about football games, teams, and players - Game is between two Teams - Each Team has Players For each game, we could specify every player and all of their information why is this bad? 3

4 Example: Football Game Data Data about football games, teams, and players - Game is between two Teams - Each Team has Players For each game, we could specify every player and all of their information why is this bad? Normalization: reduce redundancy, keep information that doesn't change separate 3 Relations: Team, Player, Game Each relation only encodes the data specific to what it represents Player Id Name Height Weight Team Id Name Wins Losses Game Id Location Date 3

5 Example: Football Game Data Have each game store the id of the home team and the id of the away team (one-to-one) Have each player store the id of the team he plays on (many-to-one) Player Id Name Height Weight TeamId Team Id Name Wins Losses Game What happens if a player plays on 2+ teams? Id Location Date Home Away 4

6 Concatenation Take two data frames with the same columns and add more rows pd.concat([data-frame-1, data-frame-2, ]) Default is to add rows (axis=0), but can also add columns (axis=1) Can also concatenate Series into a data frame. concat preserves the index so this can be confusing if you have two default indices (0,1,2,3 ) they will appear twice - Use ignore_index=true to get a 0,1,2 5

7 Merges (aka Joins) Need to merge data from one DataFrame with data from another DataFrame Example: Football game data merged with temperature data Game Id Location Date Home Away 0 Boston 9/ Boston 9/ Cleveland 9/ San Diego 9/ No data for San Diego Weather wid City Date Temp 0 Boston 9/ Boston 9/ Boston 9/ Boston 9/ Cleveland 9/

8 Inner Strategy Merged Id Location Date Home Away Temp wid 0 Boston 9/ Boston 9/ Cleveland 9/ No San Diego entry 7

9 Outer Strategy Merged Id Location Date Home Away Temp wid 0 Boston 9/ NaN Boston 9/3 NaN NaN Boston 9/ NaN Boston 9/10 NaN NaN 76 8 NaN Cleveland 9/2 NaN NaN Cleveland 9/ San Diego 9/ NaN NaN 8

10 Left Strategy Merged Id Location Date Home Away Temp wid 0 Boston 9/ Boston 9/ Cleveland 9/ San Diego 9/ NaN NaN 9

11 Right Strategy Merged Id Location Date Home Away Temp wid 0 Boston 9/ NaN Boston 9/3 NaN NaN Boston 9/ NaN Boston 9/10 NaN NaN 76 8 NaN Cleveland 9/2 NaN NaN Cleveland 9/ No San Diego entry 10

12 Data Merging in Pandas pd.merge(left, right, ) Default merge: join on matching column names Better: specify the column name(s) to join on via on kwarg - If column names differ, use left_on and right_on - Multiple keys: use a list how kwarg specifies the type of join ("inner", "outer", "left", "right") Can add suffixes to column names when they appear in both tables, but are not being joined on Can also merge using the index by setting left_index or right_index to True 11

13 Assignment 3 Due after the exam, but useful to practice pandas concepts! Same data and concepts as Assignment 2 but using pandas Due Thursday, October 27 No class on Thursday, October 27 12

14 Midterm Exam Tuesday, October 25 In class: 12:30pm-1:45pm, Dion 114 Format: Multiple Choice and Short Answer me with questions, no office hours next week No class on Thursday, October

15 Midterm Questions? 14

16 Reshaping Data Reshape/pivoting are fundamental operations Can have a nested index in pandas Example: Congressional Districts (Ohio's 1st, 2nd, 3rd, Colorado's 1st, 2nd, 3rd) and associated representative rankings Could write this in different ways: number one two three state Ohio Colorado Out[97]: state number Ohio one 0 two 1 three 2 Colorado one 3 two 4 three 5 Out[99]: state Ohio Colorado number one 0 3 two 1 4 three

17 Stack and Unstack stack: pivots from the columns into rows (may produce a Series!) unstack: pivots from rows into columns unstacking may add missing data stacking filters out missing data (unless dropna=false) can unstack at a different level by passing it (e.g. 0), defaults to innermost level Out[99]: number one two three state Ohio Colorado stack T unstack Out[97]: state number Ohio one 0 two 1 three 2 Colorado one 3 two 4 three 5 state Ohio Colorado number one 0 3 two 1 4 three 2 5 unstack(0) 16

18 Pivot Sometimes, we have data that is given in "long" format and we would like "wide" format Long format: column names are data values Wide format: more like spreadsheet format Example: Out[116]: date item value realgdp infl unemp realgdp infl unemp realgdp infl unemp realgdp pivot('date', 'item', 'value') Out[118]: item infl realgdp unemp date

19 Tidy Data Each variable forms a column Each observation forms a row Each type of observational unit forms a table (DataFrame) 18

20 Melting year artist track time date.entered wk1 wk2 wk Pac Baby Don t Cry 4: Ge+her The Hardest Part Of... 3: Doors Down Kryptonite 3: ^0 Give Me Just One Nig... 3: A*Teens Dancing Queen 3: Aaliyah I Don t Wanna 4: Aaliyah Try Again 4: Adams, Yolanda Open My Heart 5: Table 7: The first eight Billboard top hits for Other columns not shown are wk4, wk5,...,wk75. year artist time track date week rank Pac 4:22 Baby Don t Cry Pac 4:22 Baby Don t Cry Pac 4:22 Baby Don t Cry Pac 4:22 Baby Don t Cry Pac 4:22 Baby Don t Cry Pac 4:22 Baby Don t Cry Pac 4:22 Baby Don t Cry Ge+her 3:15 The Hardest Part Of Ge+her 3:15 The Hardest Part Of Ge+her 3:15 The Hardest Part Of Doors Down 3:53 Kryptonite Doors Down 3:53 Kryptonite Doors Down 3:53 Kryptonite Doors Down 3:53 Kryptonite Doors Down 3:53 Kryptonite [Wickham, 2014] 19

21 Melting Pandas also has a melt function: In [41]: cheese = pd.dataframe({'first' : ['John', 'Mary'],...: 'last' : ['Doe', 'Bo'],...: 'height' : [5.5, 6.0],...: 'weight' : [130, 150]})...: In [42]: cheese Out[42]: first height last weight 0 John 5.5 Doe Mary 6.0 Bo 150 In [43]: pd.melt(cheese, id_vars=['first', 'last']) Out[43]: first last variable value 0 John Doe height Mary Bo height John Doe weight Mary Bo weight In [44]: pd.melt(cheese, id_vars=['first', 'last'], var_name='quantity') Out[44]: first last quantity value 0 John Doe height Mary Bo height John Doe weight Mary Bo weight

22 More Tidy Data Have sex (m,f) and age range(0-14,15-24, ) encoded in the column names country year m014 m1524 m2534 m3544 m4554 m5564 m65 mu f014 AD AE AF AG AL AM AN AO AR AS [Wickham, 2014] 21

23 Melting + Splitting country year column cases AD 2000 m014 0 AD 2000 m AD 2000 m AD 2000 m AD 2000 m AD 2000 m AD 2000 m65 0 AE 2000 m014 2 AE 2000 m AE 2000 m AE 2000 m AE 2000 m AE 2000 m AE 2000 m65 10 AE 2000 f014 3 (a) Molten data country year sex age cases AD 2000 m AD 2000 m AD 2000 m AD 2000 m AD 2000 m AD 2000 m AD 2000 m AE 2000 m AE 2000 m AE 2000 m AE 2000 m AE 2000 m AE 2000 m AE 2000 m AE 2000 f (b) Tidy data [Wickham, 2014] 22

24 Data Transformation Includes filtering, cleaning, and other transformations String methods help with data transformation Dropping duplicates Custom operations 23

25 Removing Duplicates Often, have data that is reported more than once Example: daily digests of sensor data may include a 24+-hour window that might overlap from time to time Two methods: - duplicated: returns a boolean Series that indicates if a row is duplicated or not (1st occurrence is False, 2nd occurrence is True) - drop_duplicates: returns a DataFrame where the duplicates are dropped (computes duplicated and drops True rows) Options: - subset: only check a subset of columns when checking - keep: {"first", "last", False (=delete all)} 24

26 Removing Duplicates df Out[127]: k1 k2 0 one 1 1 one 1 2 one 2 3 two 3 4 two 3 5 two 4 6 two 4 df.duplicated() Out[128]: 0 False 1 True 2 False 3 False 4 True 5 False 6 True dtype: bool df.drop_duplicates() Out[129]: k1 k2 0 one 1 2 one 2 3 two 3 5 two 4 df.drop_duplicates(['k1']) k1 k2 0 one 1 3 two 3 k1 k2 1 one 1 2 one 2 4 two 3 6 two 4 df.drop_duplicates(keep="last") 25

27 Custom Transformations Take any value and transform it (e.g. based on a lookup table) map: takes a dictionary as an argument - df['animal'] = df['food'].str.lower().map(meat_to_animal) Out[134]: food ounces 0 bacon pulled pork bacon Pastrami corned beef Bacon pastrami honey ham nova lox 6.0 map can also take a function as an argument - df['food'].map(translate_meat_to_animal)' Works on the index as well meat_to_animal = { 'bacon': 'pig', 'pulled pork': 'pig', 'pastrami': 'cow', 'corned beef': 'cow', 'honey ham': 'pig', 'nova lox': 'salmon' } Out[137]: food ounces animal 0 bacon 4.0 pig 1 pulled pork 3.0 pig 2 bacon 12.0 pig 3 Pastrami 6.0 cow 4 corned beef 7.5 cow 5 Bacon 8.0 pig 6 pastrami 3.0 cow 7 honey ham 5.0 pig 8 nova lox 6.0 salmon 26

28 Replacing Values Works just like with string replacement Works on Series (or DataFrame columns) or DataFrame - df.replace(-999, np.nan) Can do multiple values at once - df.replace([-998,-999], np.nan) Can do multiple mappings at once (via dictionary) - df.replace({-998: 0, -999: np.nan}) 27

29 Chicago Food Inspections Using unstack with a groupby 28

DSC 201: Data Analysis & Visualization

DSC 201: Data Analysis & Visualization Data Transformation Dr. David Koop Example: Football Game Data Have each game store the id of the home team and the id of the away team (one-to-one) Have each player