Data Compression. Prof. Ja-Ling Wu. Department of Computer Science and Information Engineering National Taiwan University

Transcription

1 ata Coresso Prof. Ja-Lg Wu eartet of Couter Scece ad Iforato Egeerg Natoa Tawa Uversty

2 ata Coresso ca be acheved by assgg short descrtos to the ost frequet outcoes of the data source ad og descrto to the ess frequet outcoes. efto: A Source code C for a rado varabe s a ag fro the rage to * the set of fte egth strgs of sybos fro a -ary ahabet. Let c() deote the codeword corresodg to ad et () deote the egth of c(). E: C(Red) = 00 C(Bue) = s a source code for = {Red Bue} wth ahabet * = {0 }.

3 efto: The eected egth L(c) of a source code c() for a r.v. wth robabty ass fucto () s gve by L c where () s the egth of the codeword assocated wth. w..o.g.: * = {0 -}. 3

4 efto: A code s sad to be o-sguar f every eeet of the rage of as to a dfferet strg *.e. j c( ) c( j ) No-sguarty suffces for a uabguous descrto of a sge vaue of. But we usuay wsh to sed a sequece of vaues of. I such cases we ca esure decodabty by addg a seca sybo ( coa ) betwee ay two codewords. But ths s a effcet use of the seca sybo; oe ca do better Sef-uctuatg or stataeous codes 4

5 efto: A eteso C* of a code C s a ag fro fte egth strgs of to fte egth strgs of * defed by c( ) = c( )c( ) c( ) where c( )c( ) c( ) dcates cocateato of the corresodg codewords. E: If c( ) = 00 ad c( ) = the c( ) = 00 5

6 efto: A code s caed uquey decodabe f ts eteso s o-sguar. I other words ay ecoded strg a uquey decodabe code has oy oe ossbe source strg roducg t. efto: A coded s caed a ref code or a stataeous code f o codeword s a ref of ay other codeword. A stataeous code ca be decoded wthout referece to the future codewords sce the ed of a codeword s edatey recogzabe. For a stataeous code the sybo ca be decoded as soo as we coe to the ed of the codeword corresodg to t. sef-uctuatg A codes No-sguar codes Uquey decodabe codes Istataeous codes 6

7 KRAFT INEQUALITY: Goa of source codg: costructg stataeous codes of u eected egth to descrbe a gve source. Theore (Kraft equaty) For ay stataeous code (ref code) over a ahabet of sze the codeword egths ust satsfy the equaty Coversey gve a set of codeword egths that satsfy ths equaty there ests a stataeous code wth these word egths. 7

8 Proof: Cosder a -ary tree whch each ode has chdre. Let the braches of the tree rereset the sybos of the codeword. For eae the braches fro the root rereset the ossbe vaues of the frst sybo of the codeword. The each codeword s rereseted by a eaf o the tree. The ath fro the root traces out the sybos of the codeword. 8

9 The ref codto o the codewords es that o codeword s a acestor of ay other codeword o the tree. Hece each codeword eates ts descedats as ossbe codewords. Let a be the egth of the ogest codeword of the set of codewords. Cosder a odes of the tree at eve a. Soe of the are codewords soe are descedats of codewords ad soe are ether. a A codeword at eve has descedats at eve a. Each of these descedat set ust be dsjot. Aso the tota uber of odes these sets ust be ess tha or equa to a. Hece sug over a codewords we have 9

10 or a a whch s the Kraft equaty. Coversey gve ay set of codewords whch satsfy the Kraft quaty we ca aways costruct a -ary tree. Labe the frst ode (ecograhcay) of deth as codeword ad ove ts descedats fro the tree. The abe the frst reag ode of deth as codeword etc. Processg ths way we costruct a ref code wth the secfed. 0

11 0 0 Tota # of odes at eve a = 4 Root Tota # of descedats of ode ode (eve ) at eve a = 4 = ode Leve 0 Leve Leve Leve 3 La = Leve 4

12 Note that: A fte ref code aso satsfes the Kraft equaty. Theore (Eteded Kraft equaty) For ay coutaby fte set of codewords that for a ref code the codeword egths satsfy the eteded Kraft equaty Coversey gve ay satsfyg the eteded Kraft equaty we ca costruct a ref code wth these codeword egths.

13 Proof: Let the -ary ahabet be {0 -}. Cosder the -th codeword y y y. Let 0.y y be the rea y uber gve by the -ary easo O y Ths codeword corresods to the terva O y y y O y y y y the set of a rea uber whose -ary easo begs wth O y y y. Ths s a subterva of the ut terva [0]. j y j j 3

14 By the ref codto those tervas are dsjot. Hece the su of ther egths has to be ess tha or equa to. Ths rove that Just as the fte case we ca reverse the roof to costruct the code for a gve that satsfes the Kraft equaty. Frst reorderg the deg so that. Ths sy assg the tervas order fro the ow ed of the ut terva. 4

15 Ota codes: Questo: How to fd the ref code wth the u eected egth? Fd the set of egths satsfyg the Kraft equaty ad whose eected egth L= s ess tha the eected egth of ay other ref code. Mze L= Over the tegers satsfyg 5

16 We egect the teger costat o ad assue equaty the costrat. Lagrage uter J J wefd og og substtutg ths the costrat yedg ota code egths * og og 0 ad hece to fd 6

17 Ths o-teger chose of codeword egths yeds eected codeword egth * * L og H but sce the ust be tegers we w ot aways be abe to set the codeword egths as eq. (). Istead we shoud choose a set of codeword egths cose to the ota set. 7

18 Theore: The eected egth L of ay stataeous -ary code for a r.v. s greater tha or equa to the etroy H ().e. L H () wth equaty ff. Proof: The dfferece betwee the eected egth ad the etroy ca be wrtte as: L H og og og 8

19 9. a teger for a s og ff.e. wth equaty ff 0 og og og we obta ad Lettg H L c r c r H L C r j j

20 Bouds o the Ota Codeegth Sce og / ay ot equa a teger we roud t u to gve teger word egth assgets og where s thesaest teger. These egths satsfy the Kraft equaty sce Ths choce og of og Mutyg by H codeword egths og og ad sug over L H satsfes we oba 0

21 Theore: Let * * * be the ota codeegths for a source dstrbuto P ad a -ary ahabet ad et L* be the assocated eected egth of the ota code ( L*= * ). The H * L H

22 Fro the recedg theore there s a overhead whch s at ost bt due to the fact that og/ s ot aways a teger. Oe ca reduce the overhead er sybo by sreadg t out over ay sybos: Let s cosder a syste whch we sed a sequece of sybos fro. The sybos are assued to draw..d. accordg to (). We ca cosder these sybos to be a suersybo fro the ahabet. efe L to be the eected codeword egth er ut sybo.e. f ( ) s the egth of the codword assocated wth ( ) the

23 3 H L H H L H H H H H E H E L are ot..d. f cose to the etroy. er syboarbtrary codeegth a eected we ca acheve bock egths usg arge by are..d. Sce

24 Theore: The u eected codeword egth er sybo satsfes: H * H L Moreover f s a statoary stochastc rocess. L * H() where H() s the etroy rate of the rocess. The eected uber of bts er sybo requred to descrbe the rocess. 4

25 Theore: The eected egth uder () of the code assget og satsfes q H q E H q Proof: The eected egth s E og q og q og q q H og og q The crease descrtve coety due to correct forato (dstrbuto) 5

26 We have roved that ay stataeous code ust satsfy the Kraft equaty. The cass of uquey decodabe codes s arger tha the cass of stataeous codes so oe eects to acheve a ower eected codeegth f L s zed over a uquey decodabe codes. I the foowg we rove that the cass of uquey decodabe codes does ot offer ay further ossbtes for the set of codeword egths tha do stataeous codes. 6

27 Theore: The codeword egths of ay uquey decodabe code ust satsfy the Kraft equaty coversey gve a set of codeword egths that satsfy ths equaty t s ossbe to costruct a uquey decodabe code wth these codeword egths. Proof: Cosder C K the K-th eteso of the code.e. the code fored by the cocateato of K reettos of the gve uquey decodabe code C. By the defto of uque decodabty the K-th eteso of the code s o-sguar. Sce there are oy dfferet -ary strgs of egth uque decodabty es that the uber of code sequeces of egth the K-th eteso of the code ust be o greater tha. 7

28 8 Let the codeword egths of the sybos be deoted by (). For the eteso code the egth of the code-sequece s: Let s cosder the Kth ower of the Kraft equaty: k k a k k a K K k K k k k k (or k)

29 where a s the au codeword egth ad a() s the uber of source sequeces k ag to codewords of egth. For uquey decodabe code there s at ost oe sequece ag to each code -sequece ad there are at ost code -sequeces. a k k a a k a k a 9

30 Hece j j k k a Sce ths equaty s true for a k t s true the t as k. Sce (k a ) /k we have j j whch s the Kraft equaty. Coversey gve ay set of satsfyg the Kraft equaty we ca costruct a stataeous code as rescrbed. Sce every stataeous code s uquey decodabe we have aso costructed a uquey decodabe code. 30

31 Coroary: A uquey decodabe code for a fte source ahabet aso satsfes the Kraft equaty. Ay subset of uquey decodabe code s aso uquey decodabe; hece ay fte subset of the fte set of codewords satsfes the Kraft equaty. Hece N N The set of achevabe codeword egths s the sae for uquey decodabe ad stataeous codes 3

32 Huffa Codes. A. Huffa A ethod for the costructo of u redudacy codes Proc. IRE vo. 40; Eae: Code word egth codeword Probabty

33 We eect the ota bary code for to have the ogest codewords assged to the sybos 4 ad 5. Both these egths ust be equa sce otherwse we ca deete a bt fro the oger codeword ad st have a ref code but wth a shorter eected egth. I geera we ca costruct a code whch the two ogest codewords dffer oy the ast bt. 33

34 Eae: If 3 we ay ot have a suffcet uber of sybos so that we ca cobe the at a te. I such a case we add duy sybos to the ed of the set of sybos. The duy sybos have robabty 0 ad are serted to f the tree. Sce at each stage of the reducto the uber of sybos s reduced by - we wat the tota uber of the sybos to be +k(-) where k s the uber of eves the tree. Codeword Probabty uy

35 Otaty of Huffa codes There are ay ota codes: vertg a the bts or echagg two codewords of the sae egth w gve aother ota code. w..o.g. we assue P P P Lea: For ay dstrbuto there ests a ota stataeous code (wth u eected egth) that satsfes the foowg roertes:. If P j > P k the j k. The two ogest codewords have the sae egth. 3. The two ogest codewords dffer oy the ast bt ad corresod to the two east key sybos. 35

36 Proof: If P j > P k the j k Cosder C wth the codewords j ad k of C terchaged. The L C LC But P P k Hece we uct have roerty. j j k P P k j P j P j k P 0 ad sce C k j k P P P j s ota L.Thus C k k tsef j C LC satsfes 0 36

37 If the two ogest codewords are ot of the sae egth the oe ca deete the ast bt of the oger oe reservg the ref roerty ad achevg ower eected codeword egth. Hece the two ogest codewords ust have the sae egth. By roerty the ogest codewords ust beog to the east robabe source sybos. If there s a aa egth codeword wthout a sbg the we ca deete the ast bt of the codeword ad st satsfy the ref roerty. Ths reduces the average codeword egth ad cotradcts the otaty of the code. Hece every aa egth codeword ay ota code has a sbg. 37

38 Now we ca echage the ogest egth codewords so that the two owest robabty source sybos are assocated wth two sbg o the tree. Ths does ot chage the eected egth P. Thus the codewords for the two owest robabty source sybos have aa egth ad agree a but the ast bt. If P P P the there ests a ota code wth - = ad codewords c( - ) ad C( ) dffer oy the ast bt. 38

39 39 For a code C satsfyg the roertes of the above ea we ow defe a erged code C - for - sybos as foows: take the coo ref of the two argest codewords (corresodg to the two east key sybos) ad aot t to a sybo wth robabty P - ad P. A the codewords rea the sae. 0 w w w w w P P w w w P w w w P w w w P C C

40 40 where w deotes a bary codeword ad deotes ts egth. The eected egth of the code C s C L C L

41 Thus the eected egth of the code C dffers fro the eected egth of C - by a fed aout deedet of C -. Thus zg the eected egth L(C ) s equvaet to zg L(C - ). Thus we have reduced the robe to oe wth - sybos ad robabty asses (P P P - + P ). we aga ook for a code whch satsfes the roertes of the ea for there - sybos ad the reduce the robe to fd the ota code for - sybos wth the arorate robabty asses obtaed by ergg the two owest robabtes o the revous erged st. 4

42 Proceedg ths way we fay reduce the robe to two sybos for whch the souto s obvous.e. aot 0 for oe of the sybos ad for the other. Sce we have ataed otaty at every stage the reducto the code costructed for sybos s ota. 4

43 Theore: Huffa codg s ota.e. f C * s the Huffa code ad C s ay other code the L(C * )L(C ). Huffa codg s a greedy agorth that t coaesces ( 合併 ) the two east key sybo at each stage. oca otaty goba otaty 43