CS 147: Computer Systems Performance Analysis Mistakes in Graphical Presentation CS 147: Computer Systems Performance Analysis Mistakes in Graphical Presentation 1 / 45
Overview Excess Information Multiple Scales Symbols for Text Poor Scales Bad Line Usage Overview Overview Excess Information Multiple Scales Symbols for Text Poor Scales Bad Line Usage Non-Zero Origins Double Whammy No Confidence Intervals Height Scaling Histogram Problems Graphical Integrity Special-Purpose Charts A Few Examples Non-Zero Origins Double Whammy No Confidence Intervals Height Scaling Histogram Problems Graphical Integrity Special-Purpose Charts A Few Examples 2 / 45
3 / 45 Excess Information Excess Information Excess Information Excess Information Excess Information Sneaky trick to meet length limits Rules of thumb: 6 curves on line chart 1 bars on bar chart 8 slices on pie chart (But note that Tufte hates pie charts) Extract essence; don t cram things in Sneaky trick to meet length limits Rules of thumb: 6 curves on line chart 1 bars on bar chart 8 slices on pie chart (But note that Tufte hates pie charts) Extract essence; don t cram things in
4 / 45 Excess Information Way Too Much Information Excess Information Way Too Much Information Way Too Much Information 4 3 Time 2 1 1 REPL 2 3 4 5 6 7 8 CP FIND FINDGREP GREP LS MAB RCP RM 4 3 Time 2 1 CP FIND FINDGREP GREP LS MAB RCP RM What s important on that chart? Times for cp and rcp rise with number of replicas Most other benchmarks are near constant Exactly constant for rm 1 REPL 2 3 4 5 6 7 8
Excess Information The Right Amount of Information Excess Information The Right Amount of Information The Right Amount of Information 4 3 cp Time 2 compile rm 1 1 2 3 4 5 6 7 8 Replicas 4 Time 3 2 1 cp compile rm 1 2 3 4 5 6 7 8 Replicas 5 / 45
6 / 45 Multiple Scales Multiple Scales Multiple Scales Multiple Scales Multiple Scales Another way to meet length limits Basically, two graphs overlaid on each other Confuses reader (which line goes with which scale?) Misstates relationships Implies equality of magnitude that doesn t exist Another way to meet length limits Basically, two graphs overlaid on each other Confuses reader (which line goes with which scale?) Misstates relationships Implies equality of magnitude that doesn t exist
7 / 45 Multiple Scales Some Especially Bad Multiple Scales Multiple Scales Some Especially Bad Multiple Scales Some Especially Bad Multiple Scales 45 1 4 35 3 25 1 2 15 Throughput 1 Response Time 5 1 1 2 3 4 45 4 35 3 25 2 15 1 5 Throughput Response Time 1 2 3 4 1 1 1
8 / 45 Symbols for Text Using Symbols in Place of Text Symbols for Text Using Symbols in Place of Text Using Symbols in Place of Text Graphics should be self-explanatory Remember that the graphs often draw the reader in So use explanatory text, not symbols This means no Greek letters! Unless your conference is in Athens... Graphics should be self-explanatory Remember that the graphs often draw the reader in So use explanatory text, not symbols This means no Greek letters! Unless your conference is in Athens...
Symbols for Text It s All Greek To Me... 12 Symbols for Text It s All Greek To Me... It s All Greek To Me... 12 1 8 w 6 4 2..1.2.3.4.5.6.7.8 ρ 1 8 w 6 4 2..1.2.3.4.5.6.7.8 ρ 9 / 45
1 / 45 Symbols for Text Explanation is Easy 12 Waiting Time as a Function of Offered Load Symbols for Text Explanation is Easy Explanation is Easy Waiting Time as a Function of Offered Load 12 1 8 Waiting Time 6 4 2..1.2.3.4.5.6.7.8 Offered Load 1 Waiting Time 8 6 4 2..1.2.3.4.5.6.7.8 Offered Load
11 / 45 Poor Scales Poor Scales Poor Scales Poor Scales Poor Scales Fiddle with axis ranges (and logarithms) to get your message across But don t lie or cheat Sometimes trimming off high ends makes things clearer Brings out low-end detail Fiddle with axis ranges (and logarithms) to get your message across But don t lie or cheat Sometimes trimming off high ends makes things clearer Brings out low-end detail
12 / 45 Poor Scales A Poor Axis Range Poor Scales A Poor Axis Range A Poor Axis Range 12 1 8 6 4 2 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 12 1 8 6 4 2 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
13 / 45 Poor Scales A Logarithmic Range Poor Scales A Logarithmic Range A Logarithmic Range 1 1 1 1 1 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 1 1 1 1 1 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
14 / 45 Poor Scales A Truncated Range 1 Poor Scales A Truncated Range A Truncated Range 1 5 4 3 2 1 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 5 4 3 2 1 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
15 / 45 Bad Line Usage Using Lines Incorrectly Bad Line Usage Using Lines Incorrectly Using Lines Incorrectly Don t connect points unless interpolation is meaningful Don t smooth lines that are based on samples Exception: fitted non-linear curves Don t connect points unless interpolation is meaningful Don t smooth lines that are based on samples Exception: fitted non-linear curves
Bad Line Usage Incorrect Line Usage Bad Line Usage Incorrect Line Usage Incorrect Line Usage 4 3 cp Time 2 compile rm 1 1 2 3 4 5 6 7 8 Replicas 4 Time 3 2 1 cp compile rm 1 2 3 4 5 6 7 8 Replicas 16 / 45
17 / 45 Non-Zero Origins Non-Zero Origins and Broken Scales Non-Zero Origins Non-Zero Origins and Broken Scales Non-Zero Origins and Broken Scales People expect (,) origins Subconsciously So non-zero origins are great way to lie More common than not in popular press Also very common to cheat by omitting part of scale Really, Your Honor, I included (,) People expect (,) origins Subconsciously So non-zero origins are great way to lie More common than not in popular press Also very common to cheat by omitting part of scale Really, Your Honor, I included (,)
18 / 45 Non-Zero Origins Non-Zero Origins Non-Zero Origins Non-Zero Origins Non-Zero Origins 27 1 Us Us 26 Them 8 Them 25 24 6 23 4 22 2 21 2 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 27 26 25 24 Us Them 1 8 6 Us Them 23 22 21 4 2 2 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
19 / 45 Non-Zero Origins The Three-Quarters Rule Highest point should be 3/4 of scale or more Non-Zero Origins The Three-Quarters Rule The Three-Quarters Rule Highest point should be 3/4 of scale or more 3 25 2 Us 15 Them 1 5 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 3 25 2 15 1 Us Them 5 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
2 / 45 Double Whammy Double-Whammy Graphs Double Whammy Double-Whammy Graphs Double-Whammy Graphs Put two related measures on same graph One is (almost) function of other Hits reader twice with same information And thus overstates impact 6 Sales ($) Units Shipped 4 2 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr Put two related measures on same graph One is (almost) function of other Hits reader twice with same information And thus overstates impact 6 4 2 Sales ($) Units Shipped 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
21 / 45 No Confidence Intervals Omitting Confidence Intervals No Confidence Intervals Omitting Confidence Intervals Omitting Confidence Intervals Statistical data is inherently fuzzy But means appear precise Giving confidence intervals can make it clear there s no real difference So liars and fools leave them out Statistical data is inherently fuzzy But means appear precise Giving confidence intervals can make it clear there s no real difference So liars and fools leave them out
22 / 45 No Confidence Intervals Graph Without Confidence Intervals No Confidence Intervals Graph Without Confidence Intervals Graph Without Confidence Intervals 7 6 5 4 3 2 1 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 7 6 5 4 3 2 1 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
23 / 45 No Confidence Intervals Graph With Confidence Intervals No Confidence Intervals Graph With Confidence Intervals Graph With Confidence Intervals 7 6 5 4 3 2 1 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 7 6 5 4 3 2 1 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
24 / 45 Height Scaling Scaling by Height Instead of Area Clip art is popular with illustrators: Height Scaling Scaling by Height Instead of Area Scaling by Height Instead of Area Clip art is popular with illustrators: Women in the Workforce 196 198 Women in the Workforce 196 198
25 / 45 Height Scaling The Trouble with Height Scaling Height Scaling The Trouble with Height Scaling The Trouble with Height Scaling Previous graph had heights of 2:1 But people perceive areas, not heights So areas should be what s proportional to data Tufte defines lie factor: size of effect in graphic divided by size of effect in data Not limited to area scaling But especially insidious there (quadratic effect) Previous graph had heights of 2:1 But people perceive areas, not heights So areas should be what s proportional to data Tufte defines lie factor: size of effect in graphic divided by size of effect in data Not limited to area scaling But especially insidious there (quadratic effect)
26 / 45 Height Scaling Scaling by Area Same graph with 2:1 area: Height Scaling Scaling by Area Scaling by Area Same graph with 2:1 area: Women in the Workforce 196 198 Women in the Workforce 196 198
Histogram Problems Poor Histogram Cell Size Histogram Problems Poor Histogram Cell Size Poor Histogram Cell Size Picking bucket size is always problem Prefer 5 or more observations per bucket Choice of bucket size can affect results: 12 1 8 6 4 2 5 1 15 2 25 3 Picking bucket size is always problem Prefer 5 or more observations per bucket Choice of bucket size can affect results: 12 1 8 6 4 2 5 1 15 2 25 3 Note that green bars are steadily decreasing, but blue bars rise, fall, and rise again. It s not clear which is correct (given small counts in the smaller buckets). 27 / 45
Graphical Integrity Principles of Graphics Integrity (Tufte) Graphical Integrity Principles of Graphics Integrity (Tufte) Principles of Graphics Integrity (Tufte) Proportional representation of numbers Clear, detailed, thorough labeling Show data variation, not design variation Use deflated money units Don t have more dimensions than data has Don t quote data out of context Proportional representation of numbers Clear, detailed, thorough labeling Show data variation, not design variation Use deflated money units Don t have more dimensions than data has Don t quote data out of context 28 / 45
29 / 45 Graphical Integrity Proportional Representation of Numbers Graphical Integrity Proportional Representation of Numbers Proportional Representation of Numbers Maintain lie factor of 1. Use areas, not heights, with clip art Avoiding decorative graphs will do wonders Not too hard for most engineers! Maintain lie factor of 1. Use areas, not heights, with clip art Avoiding decorative graphs will do wonders Not too hard for most engineers!
Graphical Integrity Clear, Detailed, Thorough Labeling Graphical Integrity Clear, Detailed, Thorough Labeling Clear, Detailed, Thorough Labeling Goal is to defeat distortion and ambiguity Write explanations on graphic itself Label important events in the data Goal is to defeat distortion and ambiguity Write explanations on graphic itself Label important events in the data 3 / 45
Graphical Integrity Show Data Variation, Not Design Variation Graphical Integrity Show Data Variation, Not Design Variation Show Data Variation, Not Design Variation Use one design for entire graphic In papers, try to use one design for all graphs Again, artistic license is big culprit Use one design for entire graphic In papers, try to use one design for all graphs Again, artistic license is big culprit 31 / 45
32 / 45 Graphical Integrity Use Deflated Money Units Graphical Integrity Use Deflated Money Units Use Deflated Money Units Often necessary to show money over time Even in computer science E.g., price/performance over time Or expected future cost of a disk Nominal dollars are meaningless Derate by some standard inflation measure That s what the WWW is for! Often necessary to show money over time Even in computer science E.g., price/performance over time Or expected future cost of a disk Nominal dollars are meaningless Derate by some standard inflation measure That s what the WWW is for!
33 / 45 Graphical Integrity Don t Have More Dimensions Than Data Has This gets back to the Lie Factor 1-D data (e.g., money) should occupy one dimension on the graph: not Clip art is prohibited by this rule But if you have to, use an area measure Graphical Integrity Don t Have More Dimensions Than Data Has Don t Have More Dimensions Than Data Has This gets back to the Lie Factor 1-D data (e.g., money) should occupy one dimension on the graph: not Clip art is prohibited by this rule But if you have to, use an area measure $1. $2. $1. $2.
34 / 45 Graphical Integrity Don t Quote Data Out of Context Tufte s example: Graphical Integrity Don t Quote Data Out of Context Don t Quote Data Out of Context Tufte s example: Traffic Deaths and 35 Enforcement of Speed Limits After stricter 325 enforcement 3 Before stricter enforcement 275 25 1954 1955 1956 1957 35 Traffic Deaths and Enforcement of Speed Limits 325 After stricter enforcement 3 275 Before stricter enforcement 25 1954 1955 1956 1957
Graphical Integrity The Same Data in Context Connecticut Traffic Deaths, 1951-1959 Graphical Integrity The Same Data in Context The Same Data in Context Connecticut Traffic Deaths, 1951-1959 35 3 25 2 15 1 5 195 1951 1952 1953 1954 1955 1956 1957 1958 1959 196 35 3 25 2 15 1 5 195 1951 1952 1953 1954 1955 1956 1957 1958 1959 196 35 / 45
Special-Purpose Charts Special-Purpose Charts Special-Purpose Charts Special-Purpose Charts Special-Purpose Charts Tukey s box plot Histograms Scatter plots Gantt charts Kiviat graphs Tukey s box plot Histograms Scatter plots Gantt charts Kiviat graphs 36 / 45
37 / 45 Special-Purpose Charts Tukey s Box Plot Special-Purpose Charts Tukey s Box Plot Tukey s Box Plot Shows range, median, quartiles all in one: minimum quartile median quartile Tufte can t resist improvements: or or even maximum Shows range, median, quartiles all in one: minimum quartile median quartile Tufte can t resist improvements: maximum or or even
Special-Purpose Charts Histograms Tufte improves everything about them: Special-Purpose Charts Histograms Histograms Tufte improves everything about them: 1 8 6 4 2 1st 2nd 3rd 4th Quarter 1 8 6 4 2 1st 2nd 3rd 4th Quarter 38 / 45
39 / 45 Special-Purpose Charts Scatter Plots Useful in statistical analysis Also excellent for huge quantities of data Can show patterns otherwise invisible Special-Purpose Charts Scatter Plots Scatter Plots Useful in statistical analysis Also excellent for huge quantities of data Can show patterns otherwise invisible 2 15 1 5 5 1 2 15 1 5 5 1
4 / 45 Special-Purpose Charts Better Scatter Plots Again, Tufte improves the standard But it can be a pain with automated tools Can use modified Tukey box plot for axes: Special-Purpose Charts Better Scatter Plots Better Scatter Plots Again, Tufte improves the standard But it can be a pain with automated tools Can use modified Tukey box plot for axes: 4 3 2 1 2 4 6 8 4 3 2 1 2 4 6 8
41 / 45 Special-Purpose Charts Gantt Charts Shows relative duration of Boolean conditions Arranged to make lines continuous Each level after first follows FTTF pattern (Possibly repeated) CPU I/O Network Special-Purpose Charts Gantt Charts Gantt Charts Shows relative duration of Boolean conditions Arranged to make lines continuous Each level after first follows FTTF pattern (Possibly repeated) CPU I/O Network 2 4 6 8 1% Gantt charts are any chart with horizontal lines showing spans on the X axis. Also useful for scheduling; shows simultaneous tasks. Lines are divided in mid-true; any vertical line shows one unique combo of conditions. Length of line with particular condition shows percentage of time system spends in that state. 2 4 6 8 1%
Special-Purpose Charts Kiviat Graphs Also called star charts or radar plots Special-Purpose Charts Kiviat Graphs Kiviat Graphs Also called star charts or radar plots Useful for looking at balance between HB and LB metrics Useful for looking at balance between HB and LB metrics 42 / 45
43 / 45 A Few Examples A Very Bad Graph A Few Examples A Very Bad Graph A Very Bad Graph
44 / 45 A Few Examples A Good Graph: Sunspots A Few Examples A Good Graph: Sunspots A Good Graph: Sunspots Vertical scale is latitude of sunspot; length of bar is extent of latitude width of sunspot (longitude width is not in the graph). The 11-year cycle is easily visible. The horizontal scale is empty in a few places where sunspot data extends into it. This graph was drawn in 194 by Edward Walter Maunder (1851-1928). It is commonly called a butterfly diagram for obvious reasons.
45 / 45 A Few Examples A Superb Graph: DEC Traces A Few Examples A Superb Graph: DEC Traces A Superb Graph: DEC Traces X axis is time (instructions executed). Y axis is memory address referenced, modulo 4 MB. Red lines are data accesses, blue instructions. Green is perhaps stack? Note how parallel access to arrays is easy to see, as well as occasional faster access and reverse-order access.