A Bayesian Approach to Data Base Integration. NYC INFORMS March 17, 2004 James N. Arvesen, Ph.D.

Transcription

1 A Bayesan Approach to Data Base Integraton NYC INFORMS March 17, 2004 James N. Arvesen, Ph.D.

2 The Basc Problem Have a data seres-spatal/temporal-that s ether a census or a statstcal random sample. Have a correlated data seres that s a statstcal convenence sample. e.g. The random sample mght be quarterly household demographcs at the zp code level. The statstcal convenence sample mght be the number of credt cards by ssuer at zp from several processors e.g. In the pharma ndustry, the random sample mght be monthly drug Rx s (at the product level) by zp code. The convenence sample mght be patents purchasng the partcular drug each month for each zp code

3 Varety of data sources The Pharmaceutcal ndustry uses a varety of data sources to analyze markets, establsh promotonal strateges, deploy feld organzatons, measure representatves performance, etc. Some wdely used data types nclude: Sales volume data (shpments, nternal sales) Prescrpton delvery data Call reportng data (nternal reports, 3 rd party audts) Patent level data

4 Benefts Each data source fulflls dfferent needs and presents a dfferent advantage/dsadvantage profle Some sources enjoy a hgh level of coverage, but provde lttle nsght n drug usage (shpment, call reportng) Some sources provde better trackng of drug usage, but lack n coverage Indvdually they don t provde a complete pcture of the drug market Collectvely, they suffer from dsparate levels of coverage (by geography), makng t dffcult to compare and combne them

5 Other felds face ncomplete data Baseball llustraton: 1970 Battng Averages for 18 MLB Players and Estmates Player Battng average frst 45 at bats Battng avg remander of season Bayesan Estmate 1. Clemente (Ptts, NL) F. Robnson (Balt, AL) F. Howard (Wash, AL) Johnstone (Cal, AL) Berry (Ch, AL) Spencer (Cal, AL) Kessnger (Ch, NL) L. Alvarado (Bos, AL) Santo (Ch, NL) Swoboda (NY, NL) Unser (Wash, AL) Wllams (Ch, AL) Scott (Bos, AL) Petrocell (Bos, AL) E. Rodrguez (KC, AL) Campaners (Oak, AL) Munson (NY, AL) Alvs (Ml, NL)

6 Searchng for mproved estmates X θ ~ N( θ,1), = 1,..., k( = 18) after standard arcsne transformaton. IMPROVED ESTIMATOR X + (1 ( k 3) / V )( X X ) V = ( X X 2 ) Ths s a standard shrnkage estmator. See B. Efron and C. Morrs, Data analyss usng Sten s estmator and ts generalzatons, JASA, 70, (1975)

7 Example: Patent level data Newer type of data, lnkng prescrptons to patents (undentfed) offers opportuntes for mproved understandng of market dynamcs Complance trackng New patent measurement Brand swtchng

8 Example: Patent level data Data presented by supplers poses several challenges to analysts, lmtng ts value Overall capture rate of samples may be low Capture rate of samples may vary by regon Capture rate of samples may vary wthn a wde range at lower level of geography (5-dgt zp codes) Capture rate of samples may vary over tme Data often suppled wth no projecton at low geographc level Stuaton s comparable to early days of prescrpton data

9 Projectng patent level data Applyng smple projecton methodology to patent-level data results n relable estmates at a hgher geographc level (naton, regon), but estmates at lower geographc levels (5- dgt zp code) are less robust. Large samples, coverng a hgh percentage of actual transactons, lower level of uncertanty n estmates Small samples, coverng a small percentage of actual transactons, lead to projectons wth larger range of uncertanty

10 Symptoms of data ssues Cty State Zp Code Total Populaton Pop over 60 Product Shpment Rx Capture Smple Projecton BROOKLYN NY % 2767 CHICAGO IL % 2627 LOS ANGELES CA % 2281 SAN JOSE CA % 716 GREENSBORO NC % 0 COLUMBUS OH % 1431 Smlar populaton and shpment, very dfferent projectons Populaton and shpment values, no projecton W. TOWNSEND MA % 235 ATHENS TN % 0 SHADY DALE GA % 39 ROSE HILL MS % 0 Range n capture rates may explan strange projecton results FLORENCE MA % 6 CONCORD MA % 6 PETALUMA CA % 7 NOVATO CA % 16 GILROY CA % 16 Dfferent populaton, shpment, same projecton ABILENE TX % 0

11 Symptoms of data ssues Cty ST Total Pop Pop > 60 Product Shp $ Smple Projecton Rank by Pop Rank by Pop > 60 Rank by Prod Shp Rank by Proj BROOKLYN NY CHICAGO IL LOS ANGELES CA SAN JOSE CA GREENSBORO NC COLUMBUS OH W. TOWNSEND MA ATHENS TN SHADY DALE GA ROSE HILL MS Smple projecton method leads to estmates that are nconsstent wth other data profles FLORENCE MA CONCORD MA PETALUMA CA NOVATO CA GILROY CA ABILENE TX

12 Smple projecton Product Shpments By quntles Smple projecton By quntles Quntle 5 Quntle 4 Quntle 3 Quntle 2 Quntle 1

13 Goals of advanced projecton Advanced data projecton methodology s requred to mprove the value of patent level data for analyss that s geographcally drven Reduce the range of uncertanty around projectons based on low capture rates of samples Make varous data sources more geographcally comparable Overcome sample varatons over tme

14 Bayesan estmaton Use Bayesan approach to mprove estmates Bayesan smoothng estmaton technque (borrowed strength) appled to mprove estmates at zp code level when low capture samples are avalable Bayesan approach s superor to rato estmaton, especally when samples represent dverse percentages of transactons Bayesan approach can be used over tme to further mprove estmates

15 Bayesan estmaton Methodology concepts Bayesan Smoothng allows one to use estmates from regons that are smlar n geography, demographcs, and other mportant varables to mprove estmates n regons where data s scarce or mssng. Appled to patent level data, the approach conssts n usng other, more robust data varables to provde better estmates of patent data and reduce the range of uncertanty n areas wth low capture rates Example: zp codes wth smlar demographc profle, shpments, or n smlar geographc areas

16 Bayesan estmaton Ths methodology s used n a varety of applcatons, for example: Improve estmates of dsease occurrence L. A. Waller, B. P. Carln, H, Xa, and A. E. Gelfand, Herarchcal spato-temporal mappng of dsease rates JASA (1997) D. G. Clayton and L. Bernardnell Bayesan methods for mappng dsease rsk n Geographcal and Envronmental Epdemology: Methods for Small-Area Studes, eds P. Ellot, J. Cuzck, D. Englsh, and R. Stern. Oxford Unversty Press, (1992) Desgn spam flters n emal systems

17 Apply Bayesan Smoothng model to geographc estmates Let Y, =1,...,36875 Be the true number of patents n zp code. θ ϕ ϕ ϕ = Covarate data n zp code : Total populaton Populaton over 60 Product Shpments ($), = 1,...,36875 ϕ λ ~ CAR( λ), = 1,...,36875, ~ N( ϕ,1/( λn )), j 1 n ϕ j δ j, as n the baseball example Where the summaton s over the set of neghbors of regon. Then the model s:, Y = X β + θ + ϕ, =1,...,36875

18 Bayesan estmates added to data Cty State Zp Code Total Populaton Populaton over 60 Product Shp $ Smple Projecton Bayesan Estmate BROOKLYN NY CHICAGO IL LOS ANGELES CA SAN JOSE CA GREENSBORO NC COLUMBUS OH WEST TOWNSEND MA ATHENS TN SHADY DALE GA ROSE HILL MS Estmates mproved n areas where smple projectons appeared faulty FLORENCE MA CONCORD MA PETALUMA CA NOVATO CA GILROY CA ABILENE TX

19 Bayesan mprovements Cty Total Populaton Product Shp $ Smple Proj Bayesan Estmate Rank by Pop Rank by Prod Shp Rank by Proj Rank by Bayesan BROOKLYN CHICAGO LOS ANGELES SAN JOSE GREENSBORO COLUMBUS W. TOWNSEND ATHENS SHADY DALE ROSE HILL Bayesan smoothng reduces the gap between estmates for records that are related FLORENCE CONCORD PETALUMA NOVATO GILROY ABILENE

20 Improved projectons Product Shpments By quntles Smple projecton By quntles Bayesan projecton By quntles Quntle 5 Quntle 4 Quntle 3 Quntle 2 Quntle 1

21 Impact on terrtory algnment Shaded terrtores: based on Bayesan estmates Yellow boundary terrtores: based on Smple projectons Algnment based on Bayesan estmates recommends deployng 6 terrtores n SE Wsconsn and NE Illnos, when smple estmates suggest 4 terrtores (ncludng 1 large terrtory extendng across northern Illnos).

22 Conclusons Bayesan smoothng approach helps mprove projectons at a fner geographc level Local estmates are more relable Estmates from varous regons are more comparable Estmates mprove over tme Improved estmates provde stronger bass for geography senstve tasks Market analyss Targetng Sales force deployment