Analyzing Data Properties using Statistical Sampling Techniques
|
|
|
- Whitney Cross
- 6 years ago
- Views:
Transcription
1 Analyzing Data Properties using Statistical Sampling Techniques Illustrated on Scientific File Formats and Compression Features Julian M. Kunkel
2 Outline 1 Introduction 2 Exploring a Subset of Data 3 Statistical Sampling 4 Summary Julian Kunkel ISC-HPC Research Posters, / 18
3 Motivation Understanding data characteristics is useful Relation of file types to optimize relevant file formats Conducting what-if analysis Influence of compression, deduplication Performance expectations Analysing large quantities of data is time consuming and costly Scanning petabytes of data in > 100 millions of files With 50 PB of data and 5 GiB/s read, 115 node days are needed The complete experiment for this paper would have cost 4000 e Working on a representative data set reduces costs Conducting analysis on representative data is difficult What data makes up a representative data set? How can we infer knowledge for all data based on the subset? Based on file numbers (i.e. a typical file is like X) Based on capacity (i.e. 10% of storage capacity is like Y) Many studies simply select a data set and claim it is representative Julian Kunkel ISC-HPC Research Posters, / 18
4 Contribution Goal Investigation of statistical sampling to estimate file properties Can we trust the results? What are typical mistakes when sampling data? Conduct a simple study to investigate compression and file types Approach 1 Scanning a fraction of data on DKRZ file systems Analyzing file types, compression ratio and speed 2 Investigating characteristics of the data set 3 Statistical simulation of sampling approaches We assume the population (full data set) is the scanned subset 4 Discussion of the estimation error for several approaches Julian Kunkel ISC-HPC Research Posters, / 18
5 1 Introduction 2 Exploring a Subset of Data Sampling Approach Distribution of File Sizes Scientific File Formats Compression Ratio Compression Speed Differences Between Projects 3 Statistical Sampling 4 Summary Julian Kunkel ISC-HPC Research Posters, / 18
6 Sampling of the Test Data DKRZ usage: 320 million files in 12 PB, 270 project dirs Scan of user accessible data (scan is done by a regular user) Accessible data: 58 million files, 160 project dirs Scanned files: 380 k files (0.12%) in 53.1 TiB (0.44%) capacity Scanning process is described on the poster For now, we analyze characteristics for all scanned files Julian Kunkel ISC-HPC Research Posters, / 18
7 Distribution of File Sizes File size follows a heavy tailed distribution 90% of files consume roughly 10% capacity (a) Histogram (logarithmic x-axis) (b) Cumulative file sizes (y-axis in log scale) Julian Kunkel ISC-HPC Research Posters, / 18
8 Scientific File Formats The computation by file count and capacity differs The heavy-tailed distribution skews analysis file often determines wrong formats (c) CDO types (d) File types Julian Kunkel ISC-HPC Research Posters, / 18
9 Compression Ratio Arithmetic mean compression of the full data set for each scientific file format computed on file number. The column all shows the mean values for the whole data set. Yellow diamonds show compress % computed by file size Julian Kunkel ISC-HPC Research Posters, / 18
10 Compression Speed Measured user-time for the execution of each tool (ignores I/O) Again difference between compression by size and count (a) Compression (b) Decompression Boxplots showing compression/decompression speed per file, mean shown under the plot Julian Kunkel ISC-HPC Research Posters, / 18
11 Differences Between Projects Properties vary significantly proper sampling requires to pick data from all projects File count Project size in GiB NetCDF % of files GRIB % of files NetCDF % of size LZMA compressed % by file count LZMA compressed % by size LZMA compr. speed by size in MiB/s LZMA decompr. speed by size in MiB/s Analyzing 125 individual projects, each point represents the arithmetic mean value of one Julian Kunkel ISC-HPC Research Posters, / 18
12 1 Introduction 2 Exploring a Subset of Data 3 Statistical Sampling Overview Demonstration of the Strategies 4 Summary Julian Kunkel ISC-HPC Research Posters, / 18
13 Statistical Sampling Can we determine the error when analyzing only a fraction of data? We simulate sampling by drawing samples from the totally analyzed files Statistics offers methods to determine confidence interval and sample size We analyze random variables for quantities that are continuous or proportions Proportions: fraction of samples for which a property holds Sample size and confidence intervals For proportions Cochran s sample size formula estimates sample size (Similar number) works for extremely large population sizes Error bound ±5% requires 400 samples (95% confidence) Error bound ±1% requires 10,000 samples For continuous variables Models require to know the distribution of the value A-priori unknown, usually not Gaussian, difficult to apply out-of-scope (here) Nevertheless, we will demonstrate convergence Julian Kunkel ISC-HPC Research Posters, / 18
14 Sampling Strategies Sampling to Compute by File Count 1 Enumerate all files 2 Create a simple random sample Select a random number of files to analyze without replacement For proportional variables, the number of files can be computed with Cochran s formula Sampling to Compute by File Size 1 Enumerate all files AND determine their file size 2 Pick a random sample based on the probability filesize with replacement totalsize Large files are more likely to be chosen (even multiple times) 3 Create a list of unique file names and analyze them 4 Compute the arithmetic mean for the variables If a file has been picked multiple times in Step 2., its value is used multiple times Julian Kunkel ISC-HPC Research Posters, / 18
15 Investigating Robustness: Computing by File Count Apply the approach with an increasing number of samples Compare true value with the estimated value Running the simulation 100 times to understand the variance of the estimate Clear convergence: thanks to Cochran s formula the total file count is irrelevant GRIB files % of files Random samples Simulation of sampling by file count to compute compr.% by file count Julian Kunkel ISC-HPC Research Posters, / 18
16 Investigating Robustness: Computing by File Size Using the correct sampling by weighting probability with file size GRIB files % of size Random samples Simulation of sampling to compute proportions of types by size Julian Kunkel ISC-HPC Research Posters, / 18
17 Investigating Robustness: Computing by File Size Using the WRONG sampling by just picking a simple random sample Almost no convergence behavior; you may pick a file with 99% file size at the end Random samples GRIB files % of size Simulation of sampling to compute proportions of types by size Julian Kunkel ISC-HPC Research Posters, / 18
18 Summary We investigated statistical sampling to estimate data characteristics for a system The approach is demonstrated for analyzing scientific file formats and compression Several sources of error have been discussed Estimation of values that should be computed by file size requires proper sampling Statistic simulation helps to understand the error when analyzing continuous vars Julian Kunkel ISC-HPC Research Posters, / 18
Univariate Descriptive Statistics
Univariate Descriptive Statistics Displays: pie charts, bar graphs, box plots, histograms, density estimates, dot plots, stemleaf plots, tables, lists. Example: sea urchin sizes Boxplot Histogram Urchin
Lesson Sampling Distribution of Differences of Two Proportions
STATWAY STUDENT HANDOUT STUDENT NAME DATE INTRODUCTION The GPS software company, TeleNav, recently commissioned a study on proportions of people who text while they drive. The study suggests that there
USE OF BASIC ELECTRONIC MEASURING INSTRUMENTS Part II, & ANALYSIS OF MEASUREMENT ERROR 1
EE 241 Experiment #3: USE OF BASIC ELECTRONIC MEASURING INSTRUMENTS Part II, & ANALYSIS OF MEASUREMENT ERROR 1 PURPOSE: To become familiar with additional the instruments in the laboratory. To become aware
Chapter 3 Monday, May 17th
Chapter 3 Monday, May 17 th Surveys The reason we are doing surveys is because we are curious of what other people believe, or what customs other people p have etc But when we collect the data what are
Math 58. Rumbos Fall Solutions to Exam Give thorough answers to the following questions:
Math 58. Rumbos Fall 2008 1 Solutions to Exam 2 1. Give thorough answers to the following questions: (a) Define a Bernoulli trial. Answer: A Bernoulli trial is a random experiment with two possible, mutually
Laboratory 1: Uncertainty Analysis
University of Alabama Department of Physics and Astronomy PH101 / LeClair May 26, 2014 Laboratory 1: Uncertainty Analysis Hypothesis: A statistical analysis including both mean and standard deviation can
A Boxcar Kernel Filter for Assimilation of Discrete Structures (and Other Stuff)
A Boxcar Kernel Filter for Assimilation of Discrete Structures (and Other Stuff) Jeffrey Anderson NCAR Data Assimilation Research Section (DAReS) Anderson: NWP/WAF 27: Park City 1 6/18/7 Background: 1.
This page intentionally left blank
Appendix E Labs This page intentionally left blank Dice Lab (Worksheet) Objectives: 1. Learn how to calculate basic probabilities of dice. 2. Understand how theoretical probabilities explain experimental
ECON 214 Elements of Statistics for Economists
ECON 214 Elements of Statistics for Economists Session 4 Probability Lecturer: Dr. Bernardin Senadza, Dept. of Economics Contact Information: bsenadza@ug.edu.gh College of Education School of Continuing
Chapter 1: Stats Starts Here Chapter 2: Data
Chapter 1: Stats Starts Here Chapter 2: Data Statistics data, datum variation individual respondent subject participant experimental unit observation variable categorical quantitative Calculator Skills:
Chapter 20. Inference about a Population Proportion. BPS - 5th Ed. Chapter 19 1
Chapter 20 Inference about a Population Proportion BPS - 5th Ed. Chapter 19 1 Proportions The proportion of a population that has some outcome ( success ) is p. The proportion of successes in a sample
Skip Lists S 3 S 2 S 1. 2/6/2016 7:04 AM Skip Lists 1
Skip Lists S 3 15 15 23 10 15 23 36 2/6/2016 7:04 AM Skip Lists 1 Outline and Reading What is a skip list Operations Search Insertion Deletion Implementation Analysis Space usage Search and update times
AP STATISTICS 2015 SCORING GUIDELINES
AP STATISTICS 2015 SCORING GUIDELINES Question 6 Intent of Question The primary goals of this question were to assess a student s ability to (1) describe how sample data would differ using two different
MITOCW watch?v=sozv_kkax3e
MITOCW watch?v=sozv_kkax3e The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To
Chapter 1. Statistics. Individuals and Variables. Basic Practice of Statistics - 3rd Edition. Chapter 1 1. Picturing Distributions with Graphs
Chapter 1 Picturing Distributions with Graphs BPS - 3rd Ed. Chapter 1 1 Statistics Statistics is a science that involves the extraction of information from numerical data obtained during an experiment
1. Why randomize? 2. Randomization in experiental design
Statistics 101 106 Lecture 3 (22 September 98) c David Pollard Page 1 Read M&M 3.1 and M&M 3.2, but skip bit about tables of random digits (use Minitab). Read M&M 3.3 and M&M 3.4. A little bit about randomization
Displaying Distributions with Graphs
Displaying Distributions with Graphs Recall that the distribution of a variable indicates two things: (1) What value(s) a variable can take, and (2) how often it takes those values. Example 1: Weights
What are the chances?
What are the chances? Student Worksheet 7 8 9 10 11 12 TI-Nspire Investigation Student 90 min Introduction In probability, we often look at likelihood of events that are influenced by chance. Consider
Some Parameter Estimators in the Generalized Pareto Model and their Inconsistency with Observed Data
Some Parameter Estimators in the Generalized Pareto Model and their Inconsistency with Observed Data F. Ashkar, 1 and C. N. Tatsambon 2 1 Department of Mathematics and Statistics, Université de Moncton,
One-Sample Z: C1, C2, C3, C4, C5, C6, C7, C8,... The assumed standard deviation = 110
SMAM 314 Computer Assignment 3 1.Suppose n = 100 lightbulbs are selected at random from a large population.. Assume that the light bulbs put on test until they fail. Assume that for the population of light
Unit Nine Precalculus Practice Test Probability & Statistics. Name: Period: Date: NON-CALCULATOR SECTION
Name: Period: Date: NON-CALCULATOR SECTION Vocabulary: Define each word and give an example. 1. discrete mathematics 2. dependent outcomes 3. series Short Answer: 4. Describe when to use a combination.
Proportions. Chapter 19. Inference about a Proportion Simple Conditions. Inference about a Proportion Sampling Distribution
Proportions Chapter 19!!The proportion of a population that has some outcome ( success ) is p.!!the proportion of successes in a sample is measured by the sample proportion: Inference about a Population
Section 1.5 Graphs and Describing Distributions
Section 1.5 Graphs and Describing Distributions Data can be displayed using graphs. Some of the most common graphs used in statistics are: Bar graph Pie Chart Dot plot Histogram Stem and leaf plot Box
Describing Data Visually. Describing Data Visually. Describing Data Visually 9/28/12. Applied Statistics in Business & Economics, 4 th edition
A PowerPoint Presentation Package to Accompany Applied Statistics in Business & Economics, 4 th edition David P. Doane and Lori E. Seward Prepared by Lloyd R. Jaisingh Describing Data Visually Chapter
Chapter 10. Re-expressing Data: Get it Straight! Copyright 2012, 2008, 2005 Pearson Education, Inc.
Chapter 10 Re-expressing Data: Get it Straight! Copyright 2012, 2008, 2005 Pearson Education, Inc. Straight to the Point We cannot use a linear model unless the relationship between the two variables is
Real Time Jitter Analysis
Real Time Jitter Analysis Agenda ı Background on jitter measurements Definition Measurement types: parametric, graphical ı Jitter noise floor ı Statistical analysis of jitter Jitter structure Jitter PDF
Supplementary Information for Social Environment Shapes the Speed of Cooperation
Supplementary Information for Social Environment Shapes the Speed of Cooperation Akihiro Nishi, Nicholas A. Christakis, Anthony M. Evans, and A. James O Malley, David G. Rand* *To whom correspondence should
Poverty in the United Way Service Area
Poverty in the United Way Service Area Year 2 Update 2012 The Institute for Urban Policy Research At The University of Texas at Dallas Poverty in the United Way Service Area Year 2 Update 2012 Introduction
Image Enhancement in Spatial Domain
Image Enhancement in Spatial Domain 2 Image enhancement is a process, rather a preprocessing step, through which an original image is made suitable for a specific application. The application scenarios
Comparing Means. Chapter 24. Case Study Gas Mileage for Classes of Vehicles. Case Study Gas Mileage for Classes of Vehicles Data collection
Chapter 24 One-Way Analysis of Variance: Comparing Several Means BPS - 5th Ed. Chapter 24 1 Comparing Means Chapter 18: compared the means of two populations or the mean responses to two treatments in
Lecture 2: Chapter 2
Lecture 2: Chapter 2 C C Moxley UAB Mathematics 3 June 15 2.2 Frequency Distributions Definition (Frequency Distribution) Frequency distributions shows how data are distributed among categories (classes)
EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY
EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY HIGHER CERTIFICATE IN STATISTICS, 2011 MODULE 3 : Basic statistical methods Time allowed: One and a half hours Candidates should answer THREE questions. Each
Bandwidth Scaling in Ultra Wideband Communication 1
Bandwidth Scaling in Ultra Wideband Communication 1 Dana Porrat dporrat@wireless.stanford.edu David Tse dtse@eecs.berkeley.edu Department of Electrical Engineering and Computer Sciences University of California,
Statistical Analysis of Modern Communication Signals
Whitepaper Statistical Analysis of Modern Communication Signals Bob Muro Application Group Manager, Boonton Electronics Abstract The latest wireless communication formats like DVB, DAB, WiMax, WLAN, and
VARIANCE AS APPLIED TO CRYSTAL OSCILLATORS
VARIANCE AS APPLIED TO CRYSTAL OSCILLATORS Before we can discuss VARIANCE AS APPLIED TO CRYSTAL OSCILLATORS we need to understand what a Variance is, or is trying to achieve. In simple terms a Variance
Understanding Apparent Increasing Random Jitter with Increasing PRBS Test Pattern Lengths
JANUARY 28-31, 2013 SANTA CLARA CONVENTION CENTER Understanding Apparent Increasing Random Jitter with Increasing PRBS Test Pattern Lengths 9-WP6 Dr. Martin Miller The Trend and the Concern The demand
Statistics 101: Section L Laboratory 10
Statistics 101: Section L Laboratory 10 This lab looks at the sampling distribution of the sample proportion pˆ and probabilities associated with sampling from a population with a categorical variable.
Name: Exam 01 (Midterm Part 2 Take Home, Open Everything)
Name: Exam 01 (Midterm Part 2 Take Home, Open Everything) To help you budget your time, questions are marked with *s. One * indicates a straightforward question testing foundational knowledge. Two ** indicate
DIGITAL COMMUNICATION
DEPARTMENT OF ELECTRICAL &ELECTRONICS ENGINEERING DIGITAL COMMUNICATION Spring 00 Yrd. Doç. Dr. Burak Kelleci OUTLINE Quantization Pulse-Code Modulation THE QUANTIZATION PROCESS A continuous signal has
Grades 6 8 Innoventure Components That Meet Common Core Mathematics Standards
Grades 6 8 Innoventure Components That Meet Common Core Mathematics Standards Strand Ratios and Relationships The Number System Expressions and Equations Anchor Standard Understand ratio concepts and use
Correlation of Model Simulations and Measurements
Correlation of Model Simulations and Measurements Roy Leventhal Leventhal Design & Communications Presented June 5, 2007 IBIS Summit Meeting, San Diego, California Correlation of Model Simulations and
EE EXPERIMENT 3 RESISTIVE NETWORKS AND COMPUTATIONAL ANALYSIS INTRODUCTION
EE 2101 - EXPERIMENT 3 RESISTIVE NETWORKS AND COMPUTATIONAL ANALYSIS INTRODUCTION The resistors used in this laboratory are carbon composition resistors, consisting of graphite or some other type of carbon
Chapter 19. Inference about a Population Proportion. BPS - 5th Ed. Chapter 19 1
Chapter 19 Inference about a Population Proportion BPS - 5th Ed. Chapter 19 1 Proportions The proportion of a population that has some outcome ( success ) is p. The proportion of successes in a sample
Sampling distributions and the Central Limit Theorem
Sampling distributions and the Central Limit Theorem Johan A. Elkink University College Dublin 14 October 2013 Johan A. Elkink (UCD) Central Limit Theorem 14 October 2013 1 / 29 Outline 1 Sampling 2 Statistical
MULTIPLE CHOICE. Choose the one alternative that best completes the statement or answers the question. B) Blood type Frequency
MATH 1342 Final Exam Review Name Construct a frequency distribution for the given qualitative data. 1) The blood types for 40 people who agreed to participate in a medical study were as follows. 1) O A
Chapter 25. One-Way Analysis of Variance: Comparing Several Means. BPS - 5th Ed. Chapter 24 1
Chapter 25 One-Way Analysis of Variance: Comparing Several Means BPS - 5th Ed. Chapter 24 1 Comparing Means Chapter 18: compared the means of two populations or the mean responses to two treatments in
Histogram equalization
Histogram equalization Contents Background... 2 Procedure... 3 Page 1 of 7 Background To understand histogram equalization, one must first understand the concept of contrast in an image. The contrast is
Chapter 5 - Elementary Probability Theory
Chapter 5 - Elementary Probability Theory Historical Background Much of the early work in probability concerned games and gambling. One of the first to apply probability to matters other than gambling
Lesson 6.1 Linear Equation Review
Name: Lesson 6.1 Linear Equation Review Vocabulary Equation: a math sentence that contains Linear: makes a straight line (no Variables: quantities represented by (often x and y) Function: equations can
Organizing Data 10/11/2011. Focus Points. Frequency Distributions, Histograms, and Related Topics. Section 2.1
Organizing Data 2 Copyright Cengage Learning. All rights reserved. Section 2.1 Frequency Distributions, Histograms, and Related Topics Copyright Cengage Learning. All rights reserved. Focus Points Organize
Do It Yourself 3. Speckle filtering
Do It Yourself 3 Speckle filtering The objectives of this third Do It Yourself concern the filtering of speckle in POLSAR images and its impact on data statistics. 1. SINGLE LOOK DATA STATISTICS 1.1 Data
Section 6.4. Sampling Distributions and Estimators
Section 6.4 Sampling Distributions and Estimators IDEA Ch 5 and part of Ch 6 worked with population. Now we are going to work with statistics. Sample Statistics to estimate population parameters. To make
Guess the Mean. Joshua Hill. January 2, 2010
Guess the Mean Joshua Hill January, 010 Challenge: Provide a rational number in the interval [1, 100]. The winner will be the person whose guess is closest to /3rds of the mean of all the guesses. Answer:
BE540 - Introduction to Biostatistics Computer Illustration. Topic 1 Summarizing Data Software: STATA. A Visit to Yellowstone National Park, USA
BE540 - Introduction to Biostatistics Computer Illustration Topic 1 Summarizing Data Software: STATA A Visit to Yellowstone National Park, USA Source: Chatterjee, S; Handcock MS and Simonoff JS A Casebook
Consumer Behavior when Zooming and Cropping Personal Photographs and its Implications for Digital Image Resolution
Consumer Behavior when Zooming and Cropping Personal Photographs and its Implications for Digital Image Michael E. Miller and Jerry Muszak Eastman Kodak Company Rochester, New York USA Abstract This paper
Digital Imaging and Multimedia Point Operations in Digital Images. Ahmed Elgammal Dept. of Computer Science Rutgers University
Digital Imaging and Multimedia Point Operations in Digital Images Ahmed Elgammal Dept. of Computer Science Rutgers University Outlines Point Operations Brightness and contrast adjustment Auto contrast
Chapter 3 Exponential and Logarithmic Functions
Chapter 3 Exponential and Logarithmic Functions Section 1 Section 2 Section 3 Section 4 Section 5 Exponential Functions and Their Graphs Logarithmic Functions and Their Graphs Properties of Logarithms
Sampling Terminology. all possible entities (known or unknown) of a group being studied. MKT 450. MARKETING TOOLS Buyer Behavior and Market Analysis
Sampling Terminology MARKETING TOOLS Buyer Behavior and Market Analysis Population all possible entities (known or unknown) of a group being studied. Sampling Procedures Census study containing data from
Compound Probability. Set Theory. Basic Definitions
Compound Probability Set Theory A probability measure P is a function that maps subsets of the state space Ω to numbers in the interval [0, 1]. In order to study these functions, we need to know some basic
Tutorial on the Statistical Basis of ACE-PT Inc. s Proficiency Testing Schemes
Tutorial on the Statistical Basis of ACE-PT Inc. s Proficiency Testing Schemes Note: For the benefit of those who are not familiar with details of ISO 13528:2015 and with the underlying statistical principles
Introduction INTRODUCTION TO SURVEY SAMPLING. General information. Why sample instead of taking a census? Probability vs. non-probability.
Introduction Census: Gathering information about every individual in a population Sample: Selection of a small subset of a population Census INTRODUCTION TO SURVEY SAMPLING Sample February 14, 2018 Linda
Module 7. Accounting for quantization/digitalization e ects and "o -scale" values in measurement
Module 7 Accounting for quantization/digitalization e ects and "o -scale" values in measurement Prof. Stephen B. Vardeman Statistics and IMSE Iowa State University March 4, 2008 Steve Vardeman (ISU) Module
A Novel Technique or Blind Bandwidth Estimation of the Radio Communication Signal
International Journal of ISSN 0974-2107 Systems and Technologies IJST Vol.3, No.1, pp 11-16 KLEF 2010 A Novel Technique or Blind Bandwidth Estimation of the Radio Communication Signal Gaurav Lohiya 1,
Lecture Start
Lecture -- 4 -- Start Outline 1. Science, Method & Measurement 2. On Building An Index 3. Correlation & Causality 4. Probability & Statistics 5. Samples & Surveys 6. Experimental & Quasi-experimental Designs
Real Time Word to Picture Translation for Chinese Restaurant Menus
Real Time Word to Picture Translation for Chinese Restaurant Menus Michelle Jin, Ling Xiao Wang, Boyang Zhang Email: mzjin12, lx2wang, boyangz @stanford.edu EE268 Project Report, Spring 2014 Abstract--We
TO PLOT OR NOT TO PLOT?
Graphic Examples This document provides examples of a number of graphs that might be used in understanding or presenting data. Comments with each example are intended to help you understand why the data
DIGITAL SIGNAL PROCESSING TOOLS VERSION 4.0
(Digital Signal Processing Tools) Indian Institute of Technology Roorkee, Roorkee DIGITAL SIGNAL PROCESSING TOOLS VERSION 4.0 A Guide that will help you to perform various DSP functions, for a course in
The challenges of sampling in Africa
The challenges of sampling in Africa Prepared by: Dr AC Richards Ask Afrika (Pty) Ltd Head Office: +27 12 428 7400 Tele Fax: +27 12 346 5366 Mobile Phone: +27 83 293 4146 Web Portal: www.askafrika.co.za
Confidence Intervals. Class 23. November 29, 2011
Confidence Intervals Class 23 November 29, 2011 Last Time When sampling from a population in which 30% of individuals share a certain characteristic, we identified the reasonably likely values for the
Statistics Intermediate Probability
Session 6 oscardavid.barrerarodriguez@sciencespo.fr April 3, 2018 and Sampling from a Population Outline 1 The Monty Hall Paradox Some Concepts: Event Algebra Axioms and Things About that are True Counting
Midterm 2 Practice Problems
Midterm 2 Practice Problems May 13, 2012 Note that these questions are not intended to form a practice exam. They don t necessarily cover all of the material, or weight the material as I would. They are
Lecture - 06 Large Scale Propagation Models Path Loss
Fundamentals of MIMO Wireless Communication Prof. Suvra Sekhar Das Department of Electronics and Communication Engineering Indian Institute of Technology, Kharagpur Lecture - 06 Large Scale Propagation
BIOMEDICAL SIGNAL PROCESSING (BMSP) TOOLS
BIOMEDICAL SIGNAL PROCESSING (BMSP) TOOLS A Guide that will help you to perform various BMSP functions, for a course in Digital Signal Processing. Pre requisite: Basic knowledge of BMSP tools : Introduction
SAMPLE. This chapter deals with the construction and interpretation of box plots. At the end of this chapter you should be able to:
find the upper and lower extremes, the median, and the upper and lower quartiles for sets of numerical data calculate the range and interquartile range compare the relative merits of range and interquartile
Efficiency and detectability of random reactive jamming in wireless networks
Efficiency and detectability of random reactive jamming in wireless networks Ni An, Steven Weber Modeling & Analysis of Networks Laboratory Drexel University Department of Electrical and Computer Engineering
LESSON 2: FREQUENCY DISTRIBUTION
LESSON : FREQUENCY DISTRIBUTION Outline Frequency distribution, histogram, frequency polygon Relative frequency histogram Cumulative relative frequency graph Stem-and-leaf plots Scatter diagram Pie charts,
What can evolution tell us about the feasibility of artificial intelligence? Carl Shulman Singularity Institute for Artificial Intelligence
What can evolution tell us about the feasibility of artificial intelligence? Carl Shulman Singularity Institute for Artificial Intelligence Artificial intelligence Systems that can learn to perform almost
Assignment 4: Permutations and Combinations
Assignment 4: Permutations and Combinations CS244-Randomness and Computation Assigned February 18 Due February 27 March 10, 2015 Note: Python doesn t have a nice built-in function to compute binomial coeffiecients,
TenMarks Curriculum Alignment Guide: EngageNY/Eureka Math, Grade 7
EngageNY Module 1: Ratios and Proportional Relationships Topic A: Proportional Relationships Lesson 1 Lesson 2 Lesson 3 Understand equivalent ratios, rate, and unit rate related to a Understand proportional
Why Should We Care? Everyone uses plotting But most people ignore or are unaware of simple principles Default plotting tools are not always the best
Elementary Plots Why Should We Care? Everyone uses plotting But most people ignore or are unaware of simple principles Default plotting tools are not always the best More importantly, it is easy to lie
Synthesis Algorithms and Validation
Chapter 5 Synthesis Algorithms and Validation An essential step in the study of pathological voices is re-synthesis; clear and immediate evidence of the success and accuracy of modeling efforts is provided
PASS Sample Size Software
Chapter 945 Introduction This section describes the options that are available for the appearance of a histogram. A set of all these options can be stored as a template file which can be retrieved later.
Descriptive Statistics II. Graphical summary of the distribution of a numerical variable. Boxplot
MAT 2379 (Spring 2012) Descriptive Statistics II Graphical summary of the distribution of a numerical variable We will present two types of graphs that can be used to describe the distribution of a numerical
Development of an improved flood frequency curve applying Bulletin 17B guidelines
21st International Congress on Modelling and Simulation, Gold Coast, Australia, 29 Nov to 4 Dec 2015 www.mssanz.org.au/modsim2015 Development of an improved flood frequency curve applying Bulletin 17B
Data fusion for traffic flow estimation at intersections
Data fusion for traffic flow estimation at intersections Axel WOLFERMANN Masao KUWAHARA Babak MEHRAN German Aerospace Center (DLR e. V.) Tohoku University Germany Japan Canada Outline Part I Motivation
Botswana - Botswana AIDS Impact Survey III 2008
Statistics Botswana Data Catalogue Botswana - Botswana AIDS Impact Survey III 2008 Statistics Botswana - Ministry of Finance and Development Planning, National AIDS Coordinating Agency (NACA) Report generated
Mining for Statistical Models of Availability in Large-Scale Distributed Systems: An Empirical Study of
Mining for Statistical Models of Availability in Large-Scale Distributed Systems: An Empirical Study of SETI@home Bahman Javadi 1, Derrick Kondo 1, Jean-Marc Vincent 1,2, David P. Anderson 3 1 Laboratoire
How Will the Changing U.S. Census Affect Decision-Making?
How Will the Changing U.S. Census Affect Decision-Making? David A. Swanson University of California Riverside David.swanson@ucr.edu Prepared for the Lewis Seminar May 15, 2008 ACKNOWLEDGMENTS In addition
Evaluation of image quality of the compression schemes JPEG & JPEG 2000 using a Modular Colour Image Difference Model.
Evaluation of image quality of the compression schemes JPEG & JPEG 2000 using a Modular Colour Image Difference Model. Mary Orfanidou, Liz Allen and Dr Sophie Triantaphillidou, University of Westminster,
On the GNSS integer ambiguity success rate
On the GNSS integer ambiguity success rate P.J.G. Teunissen Mathematical Geodesy and Positioning Faculty of Civil Engineering and Geosciences Introduction Global Navigation Satellite System (GNSS) ambiguity
Analysis of a Metropolitan-Area Wireless Network
Wireless Networks 0 (2000)?? 1 Analysis of a Metropolitan-Area Wireless Network Diane Tang and Mary Baker Department of Computer Science, Stanford University E-mail: [dtang,mgbaker]@cs.stanford.edu We
Introduction INTRODUCTION TO SURVEY SAMPLING. Why sample instead of taking a census? General information. Probability vs. non-probability.
Introduction Census: Gathering information about every individual in a population Sample: Selection of a small subset of a population INTRODUCTION TO SURVEY SAMPLING October 28, 2015 Karen Foote Retzer
How can it be right when it feels so wrong? Outliers, diagnostics, non-constant variance
How can it be right when it feels so wrong? Outliers, diagnostics, non-constant variance D. Alex Hughes November 19, 2014 D. Alex Hughes Problems? November 19, 2014 1 / 61 1 Outliers Generally Residual
INF3430 Clock and Synchronization
INF3430 Clock and Synchronization P.P.Chu Using VHDL Chapter 16.1-6 INF 3430 - H12 : Chapter 16.1-6 1 Outline 1. Why synchronous? 2. Clock distribution network and skew 3. Multiple-clock system 4. Meta-stability
MITOCW mit_jpal_ses06_en_300k_512kb-mp4
MITOCW mit_jpal_ses06_en_300k_512kb-mp4 FEMALE SPEAKER: The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational
Permutations and Combinations. MATH 107: Finite Mathematics University of Louisville. March 3, 2014
Permutations and Combinations MATH 107: Finite Mathematics University of Louisville March 3, 2014 Multiplicative review Non-replacement counting questions 2 / 15 Building strings without repetition A familiar
CS 445 HW#2 Solutions
1. Text problem 3.1 CS 445 HW#2 Solutions (a) General form: problem figure,. For the condition shown in the Solving for K yields Then, (b) General form: the problem figure, as in (a) so For the condition
Using Figures - The Basics
Using Figures - The Basics by David Caprette, Rice University OVERVIEW To be useful, the results of a scientific investigation or technical project must be communicated to others in the form of an oral
Two Factor Full Factorial Design with Replications
Two Factor Full Factorial Design with Replications Raj Jain Washington University in Saint Louis Saint Louis, MO 63130 Jain@cse.wustl.edu These slides are available on-line at: 22-1 Overview Model Computation
Example 1. An urn contains 100 marbles: 60 blue marbles and 40 red marbles. A marble is drawn from the urn, what is the probability that the marble
Example 1. An urn contains 100 marbles: 60 blue marbles and 40 red marbles. A marble is drawn from the urn, what is the probability that the marble is blue? Assumption: Each marble is just as likely to