Entropy Coding. Outline. Entropy. Definitions. log. A = {a, b, c, d, e}

Transcription

1 Outline efinition of ntroy Three ntroy coding techniques: Huffman coding rithmetic coding Lemel-Ziv coding ntroy oding (taken from the Technion) ntroy ntroy of a set of elements e,,e n with robabilities, n is: n { H i log i ntroy (in our context) - smallest number of bits needed, on the average, to reresent a symbol (the average on all the symbols code lengths). Note: log i is the uncertainty in symbol e i (or the surrise when we see this symbol). ntroy average surrise ssumtion: there are no deendencies between the symbols aearances i 4 efinitions lhabet: finite set containing at least one element: = {a, b, c, d, e} Symbol: n element in the alhabet: s string over the alhabet: sequence of symbols, each of which is an element of that alhabet: ccdabdcaad odeword: sequence of bits reresenting a coded symbol or string: i : The occurrence robability of symbol s i in the inut string. i Åi L i : The length of the codeword of symbol s i in bits. 3

2 ntroy examles ntroy of e, e n is maximized when = = = n =/n Æ H(e,,e n )=log n RV\PROLV HWWHU WKQWKHRWKHURUFRQWLQV PRUHLQIRUPWLRQ k symbols may be reresented by k bits ntroy of, n is minimized when =, = = n = Æ H(e,,e n )= 6 ntroy examle ntroy calculation for a two symbol alhabet. xamle : =.5 =.5 H, log log.5log.5.5log.5 xamle : =.8 =. H, log.8log.8.log. #.79 log It requires one bit er symbol on the average to reresent the data. It requires less than one bit er symbol on the average to reresent the data. How can we code this? 5 ode tyes Fixed-length codes - all codewords have the same length (number of bits),,,,, F Variable-length codes - may give different lengths to codewords ntroy coding ntroy is a lower bound on the average number of bits needed to reresent the symbols (the data comression limit). ntroy coding methods: sire to achieve the entroy for a given alhabet, PSÆntroy code achieving the entroy limit is otimal,,,,, F - PS : bits er symbol PS encoded original message message 8 7

3 Huffman coding ach symbol is assigned a variable-length code, deending on its frequency. The higher its frequency, the shorter the codeword Number of bits for each codeword is an integral number refix code variable-length code Huffman code is the otimal refix and variable-length code, given the symbols robabilities of occurrence odewords are generated by building a Huffman Tree ode tyes (cont.) Prefix code - No codeword is a refix of any other codeword. = ; = ; = ; = Uniquely decodable code - Has only one ossible source string roducing it. Unambigously decoded xamles: Prefix code - the end of a codeword is immediately recognized without ambiguity: Æ Fixed-length code 9 Huffman encoding Huffman tree examle Use the codewords from the revious slide to encode the string : String: ncoded: Number of bits used: 9 The PS is (9 bits/4 symbols) =.5 ntroy: -.5log.5 -.5log.5 -.log. -.5log.5 -.5log.5 =.854 PS is lower than the entroy. WHY? ach codeword is determined according to the ath from the root to the symbol. When decoding, a tree traversal is erformed, starting from the root. Probabilities codewords: xamle: decoding inut ()

4 Huffman encoding uild a table of er-symbol encodings (generated by the Huffman tree). Globally known to both encoder and decoder Sent by encoder, read by decoder ncode one symbol after the other, using the encoding table. ncode the seudo-eof symbol. 4 Huffman tree construction Initialization: Leaf for each symbol x of alhabet with weight=x. Note: One can work with integer weights in the leafs (for examle, number of symbol occurrences) instead of robabilities. while (tree not fully connected) do begin Y, Z Å lowest_root_weights_tree() r Å new_root r->attachsons(y, Z) // attach one via a, the other via a (order not significant) weight(r) = weight(y)+weight(z) 3 Symbol robabilities How are the robabilities known? ounting symbols in inut string o ata must be given in advance o Requires an extra ass on the inut string ata source s distribution is known o ata not necessarily known in advance, but we know its distribution Huffman decoding onstruct decoding tree based on encoding table Read coded message bit-by-bit: Travers the tree to to bottom accordingly. When a leaf is reached, a codeword was found Æ corresonding symbol is decoded Reeat until the seudo-eof symbol is reached. No ambiguities when decoding codewords (refix code) 6 5

5 Huffman ntroy analysis est results (entroy wise) - only when symbols have occurrence robabilities which are negative owers of (i.e. ½, ¼, ). Otherwise, PS > entroy bound. xamle: Symbol Probability odeword ntroy =.75 reresenting robabilities inut stream : ode: PS = (4 bits/8 symbols) =.75 8 Letter F G H I J K L M xamle Global nglish frequencies table: Prob Total:. Letter N O P Q R S T U V W X Y Z Prob Huffman summary Huffman tree construction comlexity chieves entroy when occurrence robabilities are negative owers of lhabet and its distribution must be known in advance Given the Huffman tree, very easy (and fast) to encode and decode Huffman code is not unique (because of some arbitrary decisions in the tree construction) Simle imlementation - o(n ). Using a Priority Queue - o(n log(n)): Inserting a new node o(log(n)) n nodes insertions - o(n log(n)) Retrieving smallest node weights o(log(n)) 9

6 rithmetic coding ssigns one (normally long) codeword to entire inut stream Reads the inut stream symbol by symbol, aending more bits to the codeword each time rithmetic coding odeword is a number, reresenting a segmental subsection based on the symbols robabilities ncodes symbols using a non-integer number of bits Æ very good results (entroy wise) Mathematical definitions xamle L The smallest binary value consistent with a code reresenting the symbols rocessed so far. R The roduct of the robabilities of those symbols L i+ o R i+ R i oding of = =.5, =., = = ny number in this range reresents. L i o

7 rithmetic encoding (cont.) rithmetic encoding Two ossibilities for the encoder to signal to the decoder end of the transmission: Initially L =, R =. When encoding next symbol, L and R are refined. L m L R j i i R m R j. Send initially the number of symbols encoded.. ssign a new OF symbol in the alhabet, with a very small robability, and encode it at the end of the message. Note: The order of the symbols in the alhabet must remain consistent throughout the algorithm. t the end of the message, a binary value between L and L+R will unambiguously secify the inut message. The shortest such binary string is transmitted. In the revious examle: ny number between and 3875 (discard the. ). Shortest number - 386, in binary: PS = (9 bits/4 symbols) = o o rithmetic decoding examle o.375 ecoding of ecoding: rithmetic decoding In order to decode the message, the symbols order and robabilities must be assed to the decoder. The decoding rocess is identical to the encoding. Given the codeword (the final number), at each iteration the corresonding sub-range is entered, decoding the symbol reresenting the secific range o

8 istributions issues rithmetic entroy analysis Until now, symbol distributions were known in advance What haens if they are not known? Inut string not known Huffman and rithmetic odings have an adative version o istributions are udated as the inut string is read o an work online rithmetic coding manages to encode symbols using non integer number of bits! One codeword is assigned to the entire inut stream, instead of a codeword to each individual symbol This allows rithmetic oding to achieve the ntroy lower bound 3 9 Lemel-Ziv concets What if the alhabet is unknown? Lemel-Ziv coding solves this general case, where only a stream of bits is given. LZ creates its own dictionary (strings of bits), and relaces future occurrences of these strings by a shorter osition string: In simle Huffman/rithmetic coding, the deendency between the symbols is ignored, while in the LZ, these deendencies are identified and are exloited to erform better encoding. When all the data is known (alhabet, robabilities, no deendencies), it s best to use Huffman (LZ will try to find deendencies which are not there ) Lemel-Ziv concets 3 3

9 Lemel-Ziv algorithm. Initialize the dictionary to contain an emty string (={ }).. W Å longest block in inut string which aears in. 3. Å first symbol in inut string after W 4. ncode W by its index in the dictionary, followed by 5. dd W+ to the dictionary. 6. Go to Ste. Lemel-Ziv comression Parses source inut (in binary) into the shortest distinct strings: Æ,,,,,, ach string includes a refix and an extra bit ( = + ), therefore encoded as: (refix string lace, extra bit) Requires asses over the inut (one to arse inut, second to encode). an be modified to one ass. omression: (n number of distinct strings) log(n) bits for the refix lace + bit for the added bit Overall n log(n) bits comressed omression comarison omressed to (ercentage): html (5k) Token based ascii file df (69k) inary file (.5k) Random ascii file Lemel-Ziv (unix gzi) % 75% 33% Huffman (unix ack) 65% 95% 8.% (5k) Random ascii file 9% 8.% { =.5, =.5, =.5, =.5} Lemel-Ziv is asymtotically otimal 36 ictionary Index ntry xamle Inut string: W ncoded string: Pairs: (,) (,) (,) (,) (4,) (,) (,) ncoding: 35

10 Probabilities lhabet ata loss Symbols deendency Prerocessing ntroy odewords Intuition omarison Huffman Known in advance Known in advance None Not used Tree building O(n log n) If robabilities are negative owers of One codeword for each symbol Intuitive rithmetic Known in advance Known in advance None Not used None Very close One codeword for all data Not intuitive Lemel-Ziv Not known in advance Not known in advance None Used better comression First ass on data (can be eliminated) est results when alhabet not known odewords for set of alhabet Not intuitive 37