Compression Programs. Compression Outline. Multimedia. Lossless vs. Lossy. Encoding/Decoding. Analysis of Algorithms

Transcription

1 Aalysis of Algorithms Compressio Programs File Compressio: Gzip, Bzip Archivers :Arc, Pkzip, Wirar, File Systems: NTFS Piyush Kumar (Lecture 5: Compressio) Welcome to 453 Source: Guy E. Blelloch, Emad, Tseg HDTV (Mpeg 4) Soud (Mp3) Images (Jpeg) Multimedia Compressio Outlie Itroductio: Lossy vs. Lossless Iformatio Theory: Etropy, etc. Probability Codig: Huffma + Arithmetic Codig Ecodig/Decodig Will use message i geeric sese to mea the data to be compressed Iput Message Ecoder Compressed Message Decoder Output Message CODEC The ecoder ad decoder eed to uderstad commo compressed format. Lossless vs. Lossy Lossless: Iput message = Output message Lossy: Iput message Output message Lossy does ot ecessarily mea loss of quality. I fact the output could be better tha the iput. Drop radom oise i images (dust o les) Drop backgroud i music Fix spellig errors i text. Put ito better form. Writig is the art of lossy text compressio.

2 Lossless Compressio Techiques LZW (Lempel-Ziv-Welch) compressio Build dictioary Replace patters with idex of dict. Burrows-Wheeler trasform Block sort data to improve compressio Ru legth ecodig Fid & compress repetitive sequeces Huffma code Use variable legth codes based o frequecy How much ca we compress? For lossless compressio, assumig all iput messages are valid, if eve oe strig is compressed, some other must expad. Model vs. Coder To compress we eed a bias o the probability of messages. The model determies this bias Messages Model Ecoder Probs. Coder Bits Example models: Simple: Character couts, repeated strigs Complex: Models of a huma face Quality of Compressio Rutime vs. Compressio vs. Geerality Several stadard corpuses to compare algorithms Calgary Corpus 2 books, 5 papers, bibliography, collectio of ews articles, 3 programs, termial sessio, 2 object files, geophysical data, bitmap bw image The Archive Compariso Test maitais a compariso of just about all algorithms publicly available Compariso of Algorithms Program Algorithm Time BPC Score BOA PPM Var PPMD PPM IMP BW BZIP BW GZIP LZ77 Var LZ77 LZ77? 3.94? Iformatio Theory A iterface betwee modelig ad codig Etropy A measure of iformatio cotet Etropy of the Eglish Laguage How much iformatio does each character i typical Eglish text cotai? 2

3 Etropy (Shao 948) For a set of messages S with probability p(s), s S, the self iformatio of s is: i( s) log log p( s) p ( s ) Measured i bits if the log is base 2. The lower the probability, the higher the iformatio Etropy is the weighted average of self iformatio. H ( S ) p ( s )log p ( s ) ss Etropy Example p( S) {. 25,. 25,. 25,. 25,. 25} H( S) 3. 25log log p( S) {. 5,. 25,. 25,. 25,. 25} H( S). 5log log8 2 p( S) {. 75,. 0625,. 0625,. 0625,. 0625} H( S). 75log( 4 3) log 6 3. Etropy of the Eglish Laguage How ca we measure the iformatio per character? ASCII code = 7 Etropy = 4.5 (based o character probabilities) Huffma codes (average) = 4.7 Uix Compress = 3.5 Gzip = 2.5 BOA =.9 (curret close to best text compressor) Must be less tha.9. Shao s experimet Asked humas to predict the ext character give the whole previous text. He used these as coditioal probabilities to estimate the etropy of the Eglish Laguage. The umber of guesses required for right aswer: # of guesses > 5 Probability From the experimet he predicted H(Eglish) =.6-.3 Data compressio model Iput data Reduce Data Redudacy Reductio of Etropy Etropy Ecodig Codig How do we use the probabilities to code messages? Prefix codes ad relatioship to Etropy Huffma codes Arithmetic codes Implicit probability codes Compressed Data 3

4 Assumptios Commuicatio (or file) broke up ito pieces called messages. Adjacet messages might be of a differet types ad come from a differet probability distributios We will cosider two types of codig: Discrete: each message is a fixed set of bits Huffma codig, Shao-Fao codig Bleded: bits ca be shared amog messages Arithmetic codig Uiquely Decodable Codes A variable legth code assigs a bit strig (codeword) of variable legth to every message value e.g. a =, b = 0, c = 0, d = 0 What if you get the sequece of bits 0? Is it aba, ca, or, ad? A uiquely decodable code is a variable legth code i which bit strigs ca always be uiquely decomposed ito its codewords. Prefix Codes A prefix code is a variable legth code i which o codeword is a prefix of aother word e.g a = 0, b = 0, c =, d = 0 Ca be viewed as a biary tree with message values at the leaves ad 0 or s o the edges. a 0 b 0 0 c d Some Prefix Codes for Itegers Biary Uary Split May other fixed prefix codes: Golomb, phased-biary, subexpoetial,... Average Bit Legth For a code C with associated probabilities p(c) the average legth is defied as ABL ( C) p( c) l( c) cc We say that a prefix code C is optimal if for all prefix codes C, ABL(C) ABL(C ) Relatioship to Etropy Theorem (lower boud): For ay probability distributio p(s) with associated uiquely decodable code C, H( S) ABL ( C) Theorem (upper boud): For ay probability distributio p(s) with associated optimal prefix code C, ABL( C) H( S) 4

5 Kraft McMilla Iequality Theorem (Kraft-McMilla): For ay uiquely decodable code C, l( c) 2 cc Also, for ay set of legths L such that 2 l ll there is a prefix code C such that l( c ) l ( i,..., L ) i i Proof of the Upper Boud (Part ) Assig to each message a legth l( s) log p( s) We the have l( s) log / p( s) 2 2 ss ss ss ss So by the Kraft-McMilla ieq. there is a prefix code with legths l(s). 2 log / p( s) p( s) Proof of the Upper Boud (Part 2) Now we ca calculate the average legth give l(s) ABL( S) Ad we are doe. ss ss ss p( s) l( s) p( s) log/ p( s) p( s) ( log(/ p( s))) ss H ( S) p( s)log(/ p( s)) Aother property of optimal codes Theorem: If C is a optimal prefix code for the probabilities {p,, p } the p i > p j implies l(c i ) l(c j ) Proof: (by cotradictio) Assume l(c i ) > l(c j ). Cosider switchig codes c i ad c j. If l a is the average legth of the origial code, the legth of the ew code is ' la la p j ( l( ci ) l( c j )) pi ( l( c j ) l( ci )) la ( p j pi )( l( ci ) l( c j )) la This is cotradictio sice l a was supposed to be optimal Corollary The p i is smallest over the code, the l(c i ) is the largest. Huffma Codig Biary trees for compressio 5

6 Huffma Code Approach Variable legth ecodig of symbols Exploit statistical frequecy of symbols Efficiet whe symbol probabilities vary widely Priciple Use fewer bits to represet frequet symbols Use more bits to represet ifrequet symbols A A A B A A B A Huffma Codes Iveted by Huffma as a class assigmet i 950. Used i may, if ot most compressio algorithms gzip, bzip, jpeg (as optio), fax compressio, Properties: Geerates optimal prefix codes Cheap to geerate codes Cheap to ecode ad decode l a =H if probabilities are powers of 2 Huffma Code Example Symbol Dog Cat Bird Fish Frequecy /8 /4 /2 /8 Origial Ecodig Huffma Ecodig Expected size bits 2 bits 2 bits 2 bits bits 2 bits bit 3 bits Origial /82 + /42 + /22 + /82 = 2 bits / symbol Huffma /83 + /42 + /2 + /83 =.75 bits / symbol Huffma Codes Huffma Algorithm Start with a forest of trees each cosistig of a sigle vertex correspodig to a message s ad with weight p(s) Repeat: Select two trees with miimum weight roots p ad p 2 Joi ito sigle tree by addig root with weight p + p 2 Example p(a) =., p(b) =.2, p(c ) =.2, p(d) =.5 a(.) (.3) Step a(.) b(.2) c(.2) d(.5) b(.2) (.5) (.0) 0 (.3) (.5) d(.5) c(.2) 0 a(.) b(.2) (.3) c(.2) 0 Step 2 a(.) b(.2) Step 3 a=000, b=00, c=0, d= Ecodig ad Decodig Ecodig: Start at leaf of Huffma tree ad follow path to the root. Reverse order of bits ad sed. Decodig: Start at root of Huffma tree ad take brach for each bit received. Whe at leaf ca output message ad retur to root. (.0) There are eve faster methods that ca process 8 or 32 bits at a time 0 (.5) d(.5) 0 0 (.3) c(.2) a(.) b(.2) 6

7 Lemmas L : Let p i be the smallest over the code, the l(c i ) is the largest ad hece a leaf of the tree. ( Let its paret be u ) L2 : If p j is secod smallest over the code, the l(c j ) is the child of u i the optimal code. L3 : There is a optimal prefix code with correspodig tree T*, i which the two lowest frequecy letters are sibligs. Huffma codes are optimal Theorem: The Huffma algorithm geerates a optimal prefix code. I other words: It achieves the miimum average umber of bits per letter of ay prefix code. Proof: By iductio Base Case: Trivial (oe bit optimal) Assumptio: The method is optimal for all alphabets of size k-. Proof: Let y* ad z* be the two lowest frequecy letters merged i w*. Let T be the tree before mergig ad T after mergig. The : ABL(T ) = ABL(T) p(w*) T is optimal by iductio. Proof: Let Z be a better tree compared to T produced usig Huffma s alg. Implies ABL(Z) < ABL(T) By lemma L3, there is such a tree Z i which the leaves represetig y* ad z* are sibligs (ad has same ABL as Z). By previous page ABL(Z ) =ABL(Z) p(w*) Cotradictio! Adaptive Huffma Codes Huffma codes ca be made to be adaptive without completely recalculatig the tree o each step. Ca accout for chagig probabilities Small chages i probability, typically make small chages to the Huffma tree Used frequetly i practice Huffma Codig Disadvatages Itegral umber of bits i each code. If the etropy of a give character is 2.2 bits,the Huffma code for that character must be either 2 or 3 bits, ot

8 Towards Arithmetic codig A Example: Cosider sedig a message of legth 000 each with havig probability.999 Self iformatio of each message -log(.999)=.0044 bits Sum of self iformatio =.4 bits. Huffma codig will take at least k bits. Arithmetic codig = 3 bits! Arithmetic Codig: Itroductio Allows bledig of bits i a message sequece. Ca boud total bits required based o sum of self iformatio: l 2 s i i Used i PPM, JPEG/MPEG (as optio), DMM More expesive tha Huffma codig, but iteger implemetatio is ot too bad. Arithmetic Codig (message itervals) Assig each probability distributio to a iterval rage from 0 (iclusive) to (exclusive). e.g i f ( i) p( j) j f(a) =.0, f(b) =.2, f(c) =.7 The iterval for a particular message will be called the message iterval (e.g for b the iterval is [.2,.7)) Arithmetic Codig (sequece itervals) To code a message use the followig: l f l l s f s p s s p i i i i i i i Each message arrows the iterval by a factor of p. i Fial iterval size: s i The iterval for a message sequece will be called the sequece iterval p i Arithmetic Codig: Ecodig Example Uiquely defiig a iterval Codig the message sequece: bac The fial iterval is [.27,.3) Importat property:the sequece itervals for distict message sequeces of legth will ever overlap Therefore: specifyig ay umber i the fial iterval uiquely determies the sequece. Decodig is similar to ecodig, but o each step eed to determie what the message value is ad the reduce iterval 8

9 Arithmetic Codig: Decodig Example Decodig the umber.49, kowig the message is of legth 3: RealArith Ecodig ad Decodig RealArithEcode: Determie l ad s usig origial recurreces Code usig l + s/2 trucated to +-log s bits RealArithDecode: Read bits as eeded so code iterval falls withi a message iterval, ad the arrow sequece iterval. Repeat util messages have bee decoded. The message is bbc. Boud o Legth Theorem: For messages with self iformatio {s,,s } RealArithEcode will geerate at most 2 s i i bits. log s log pi i log pi i si i 2 s i i Applicatios of Probability Codig How do we geerate the probabilities? Usig character frequecies directly does ot work very well (e.g. 4.5 bits/char for text). Techique : trasformig the data Ru legth codig (ITU Fax stadard) Move-to-frot codig (Used i Burrows-Wheeler) Residual codig (JPEG LS) Techique 2: usig coditioal probabilities Fixed cotext (JBIG almost) Partial matchig (PPM) Ru Legth Codig Code by specifyig message value followed by umber of repeated values: e.g. abbbaacccca => (a,),(b,3),(a,2),(c,4),(a,) The characters ad couts ca be coded based o frequecy. This allows for small umber of bits overhead for low couts such as. Facsimile ITU T4 (Group 3) Stadard used by all home Fax Machies ITU = Iteratioal Telecommuicatios Stadard Ru legth ecodes sequeces of black+white pixels Fixed Huffma Code for all documets. e.g. Ru legth White Black Sice alterate black ad white, o eed for values. 9

10 Move to Frot Codig Trasforms message sequece ito sequece of itegers, that ca the be probability coded Start with values i a total order: e.g.: [a,b,c,d,e,.] For each message output positio i the order ad the move to the frot of the order. e.g.: c => output: 3, ew order: [c,a,b,d,e, ] a => output: 2, ew order: [a,c,b,d,e, ] Codes well if there are cocetratios of message values i the message sequece. Residual Codig Used for message values with meaigfull order e.g. itegers or floats. Basic Idea: guess ext value based o curret cotext. Output differece betwee guess ad actual value. Use probability code o the output. JPEG-LS JPEG Lossless (ot to be cofused with lossless JPEG) Just completed stadardizatio process. Codes i Raster Order. Uses 4 pixels as cotext: NW N NE W * Tries to guess value of * based o W, NW, N ad NE. Works i two stages JPEG LS: Stage Uses the followig equatio: mi( N, W ) if NW max( N, W ) P max( N, W ) if NW mi( N, W ) N W NW otherwise Averages eighbors ad captures edges. e.g * * * 40 JPEG LS: Stage 2 Uses 3 gradiets: W-NW, NW-N, N-NE Classifies each ito oe of 9 categories. This gives 9 3 =729 cotexts, of which oly 365 are eeded because of symmetry. Each cotext has a bias term that is used to adjust the previous predictio After correctio, the residual betwee guessed ad actual value is foud ad coded usig a Golomblike code. Usig Coditioal Probabilities: PPM Use previous k characters as the cotext. Base probabilities o couts: e.g. if see th 2 times followed by e 7 times, the the coditioal probability p(e th)=7/2. Need to keep k small so that dictioary does ot get too large. 0

11 Ideas i Lossless compressio That we did ot talk about specifically Lempel-Ziv (gzip) Tries to guess ext widow from previous data Burrows-Wheeler (bzip) Cotext sesitive sortig Block sortig trasform LZ77: Slidig Widow Lempel-Ziv a a c a a c a b c a b a b a c Dictioary (previously coded) Cursor Lookahead Buffer Dictioary ad buffer widows are fixed legth ad slide with the cursor O each step: Output (p,l,c) p = relative positio of the logest match i the dictioary l = legth of logest match c = ext char i buffer beyod logest match Advace widow by l + Lossy compressio Scalar Quatizatio Give a camera image with 2bit color, make it 4-bit grey scale. Uiform Vs No-Uiform Quatizatio The eye is more sesitive to low values of red compared to high values. Vector Quatizatio How do we compress a color image (r,g,b)? Fid k represetative poits for all colors For every pixel, output the earest represetative If the poits are clustered aroud the represetatives, the residuals are small ad hece probability codig will work well. Trasform codig Trasform iput ito aother space. Oe form of trasform is to choose a set of basis fuctios. JPEG/MPEG both use this idea.

12 Other Trasform codes Wavelets Fractal base compressio Based o the idea of fixed poits of fuctios. 2