Entropy, Coding and Data Compression
|
|
- Maude Wade
- 6 years ago
- Views:
Transcription
1 Entropy, Coding and Data Compression
2 Data vs. Information yes, not, yes, yes, not not In ASCII, each item is 3 8 = 24 bits of data But if the only possible answers are yes and not, there is only one bit of information per item
3 Compression = Squeezing out the Air Suppose you want to ship pillows in boxes and are charged by the size of the box To use as few boxes as possible, squeeze out all the air, pack into boxes, fluff them up at the other end Lossless data compression = pillows are perfectly restored Lossy data compression = some damage to the pillows is OK (MP3 is a lossy compression standard for music) Loss may be OK if it is below human perceptual threshold Entropy is a measure of limit of lossless compression
4 Example: Telegraphy Source English letters -> Morse Code Sender: from Hokkaido D D Receiver: in Tokyo
5 Coding Messages with Fixed Length Codes Example: 4 symbols, A, B, C, D A=, B=, C=, D= In general, with n symbols, codes need to be of length lg n, rounded up For English text, 26 letters + space = 27 symbols, length = 5 since 2 4 < 27 < 2 5 (replace all punctuation marks by space)
6 Modeling the Message Source Source Destination Characteristics of the stream of messages coming from the source affect the choice of the coding method We need a model for a source of English text that can be described and analyzed mathematically
7 Uniquely decodable codes If any encoded string has only one possible source string producing it then we have unique decodablity Example of uniquely decodable code is the prefix code
8 Prefix Coding A prefix code is defined as a code in which no codeword is the prefix of some other code word. A prefix code is uniquely decodable. Prefix Code Source Symbol Code A Symbol Codeword Code B Symbol Codeword Code C Symbol Codeword s s s 2 s 3 Uniquely Decodable Codes
9 Decoding of a Prefix Code Decision Tree for Code B Code B Initial State s s s 2 Source Symbol s k s s s 2 Symbol Codeword c k s 3 s 3 Example : Decode Answer : s s 3 s 2 s s
10 Prefix Codes Only one way to decode left to right when message received Example Symbol A B C D Probability.7... Code Received message:
11 Prefix Codes Example 2 Source Symbol s k A B C D Code E Symbol Codeword c k IS CODE E A PREFIX CODE? NO WHY? Code of D is a prefix to code of C
12 Average Code Length Information Source s k Source Encoder c k Source has K symbols Each symbol s k has probability p k Each symbol s k is represented by a codeword c k of length l k bits Average codeword length L K = k = pl k k
13 Shannon s First Theorem: The Source Coding Theorem L H( S ) The outputs of an information source cannot be represented by a source code whose average length is less than the source entropy
14 Average Code Length Example Average bits per symbol: L= =.6 bits/symbol (down from 2) A.7 B. C. D. Another prefix code that A B C D is better.7... L= =.5
15 Robot Example Source Entropy Examples 4-way random walk prob( x = S) =, prob( x = N) = 2 prob( x = E) = prob( x = W ) = 8 H ( X ) = ( log 2 + log 2 + log 2 + log 2 ) =. 75bps W N S E
16 Source Entropy Examples Robot Example symbol k S N E p k fixed-length codeword variable-length codeword W 5 symbol stream : S S N W S E N N N W S S S N E S S fixed length: variable length: 32bits 28bits 4 bits savings achieved by VLC (redundancy eliminated)
17 Entropy, Compressibility, Redundancy Lower entropy <=> More redundant <=> More compressible Higher entropy <=> Less redundant <=> Less compressible A source of yes s and not s takes 24 bits per symbol but contains at most one bit per symbol of information
18 Entropy and Compression First-order entropy is theoretical minimum on code length when only frequencies are taken into account A B C L= = First-order Entropy =.353 D. First-order Entropy of English is about 4 bits/character based on typical English texts
19 Bits You are watching a set of independent random samples of X You see that X has four possible values P(X=A) = /4 P(X=B) = /4 P(X=C) = /4 P(X=D) = /4 So you might see output: BAACBADCDADDDA You transmit data over a binary serial link. You can encode each reading with two bits (e.g. A =, B =, C =, D = ) 2 bits on average per symbol
20 Fewer Bits Someone tells you that the probabilities are not equal P(X=A) = /2 P(X=B) = /4 P(X=C) = /8 P(X=D) = /8 Is it possible to invent a coding for your transmission that only uses.75 bits on average per symbol. How?
21 Fewer Bits Someone tells you that the probabilities are not equal P(X=A) = /2 P(X=B) = /4 P(X=C) = /8 P(X=D) = /8 It s possible to invent a coding for your transmission that only uses.75 bits on average per symbol. How? A B C D (This is just one of several ways)
22 Fewer Bits Suppose there are three equally likely values P(X=A) = /3 P(X=B) = /3 P(X=C) = /3 Here s a naïve coding, costing 2 bits per symbol A B C Can you think of a coding that would need only.6 bits per symbol on average? In theory, it can in fact be done with bits per symbol.
23 Kraft-McMillan Inequality K 2 lk k = If codeword lengths of a code satisfy the Kraft McMillan s inequality, then a prefix code with these codeword lengths can be constructed. For code D = 9/8 This means that Code D IS NOT A PREFIX CODE Source Symbol s k Code D Symbol Codewor d C k s s 2 s 2 3 s 3 2 Codeword Length l k
24 Use of Kraft-McMillan Inequality We may use it if the number of symbols are large such that we cannot simply by inspection judge whether a given code is a prefix code or not WHAT Kraft-McMillan Inequality Can Do: It can determine that a given code IS NOT A PREFIX CODE It can identify that a prefix code could be constructed from a set of codeword lengths WHAT Kraft-McMillan Inequality Cannot Do: It cannot guarantee that a given code is indeed a prefix code
25 Example Source Symbo l s k Symbol Codewor d c k Code E Codeword Length l k s s 3 s 2 3 For code E s = IS CODE E A PREFIX CODE? NO WHY? s 3 is a prefix to s 2
26 Code Efficiency? η ( ) = H S L An efficient code means?
27 Source Symbol s k Symbol Probability p k Examples Code I Symbol Codeword c k Codeword Length l k Symbol Codeword c k s /2 2 s /4 2 2 s 2 /8 2 3 s 3 /8 2 3 Source Entropy H(S) =/2log 2 (2)+/4log 2 (4)+ /8log 2 (8)+/log 2 (8) = ¾ bits/symbol Code I L = = η = = Code II Code II Codeword Length l k 7 L = = η = = 74
28 For a Prefix Code Shannon s First Theorem ( S ) L ( S ) H < H + L = H( S ) if pk l = 2 k k p k 2 l k for some What is the Efficiency???= if?< However, we may increase efficiency by extending the source k
29 Increasing Efficiency by Source By extending the source we may potentially increase efficiency The drawback is Increased decoding complexity Extension ( n ) ( n S L ) n S H < H + ( ) ( ) nh S L < nh S + H η η n L < + n ( ) n S H( S ) = H L ( S ) n n n when n
30 Extension of a Discrete Memoryless Source Treats Blocks of n successive symbols Information Source Extended Information Source S = { } { } s,s,...,s K Pr s = p,k =,,...,K k k K- k = p k = { } n n S = σ, σ,..., σ K { } n Pr σ = q,i =,,...,K i K i n - i = p i =
31 Example 2 S={s,s,s 2 }, p =/4, p =/4, p 2 =/2 H(S)=(/4)log 2 (4)+ (/4)log 2 (4)+ (/2)log 2 (2) H(S)=3/2 bits Second-Order Extended Source Symbols of S 2 s s s 2 s 3 s 4 s 5 s 6 s 7 s 8 Sequence of Symbols from S s s s s s s 2 s s s s s s 2 s 2 s s 2 s s 2 s P{s i }, i=,,,8 /6 /6 /8 /6 /6 /8 /8 /8 /4 By Computing: H(S 2 )=3 bits
32 Example 3 Calculate the English of English language if. All alphabet letters are equally probable 2. For a, e, o, t P{s k }=. For h, i, n, r, s P{s k }=.7 For c, d, f, l, m, p, u, y P{s k }=.2 For b, g, j, k, q, v, w, x, z P{s k }=.. H(S)=4.7 bits 2. H(S)=4.7 bits
33 Source Encoding Efficient representation of information sources Source Coding Requirements Uniquely Decodable Codes Prefix Codes No codeword is a prefix to some other code word Code Efficiency η ( ) = H S L Kraft s Inequality K lk k = 2 Source Coding Theorem ( S ) L ( S ) H < H +
34 Source Coding Techniques. Huffman Code. 2. Two-path Huffman Code. 3. Lemple-Ziv Code. 4. Shannon Code. 5. Fano Code. 6. Arithmetic Code.
35 Source Coding Techniques. Huffman Code. 2. Two-path Huffman Code. 3. Lemple-Ziv Code. 4. Shannon Code. 5. Fano Code. 6. Arithmetic Code.
36 Source Coding Techniques. Huffman Code. With the Huffman code in the binary case the two least probable source output symbols are joined together, resulting in a new message alphabet with one less symbol
37 Huffman Coding: Example Compute the Huffman Code for the source shown H( S ) = ( 4log. ) ( 2log. ) ( log. ) 2. = L Source Symbol s k. s s s 2.4 s 3. s 4 Symbol Probability p k
38 Solution A Source Symbol s k s 2 s s 3 s s 4 Stage I.4..
39 Solution A Source Symbol s k s 2 s s 3 s s 4 Stage I.4.. Stage II.4
40 Solution A Source Symbol s k Stage I Stage II Stage III s s.4 s 3 s. s 4.
41 Solution A Source Symbol s k Stage I Stage II Stage III Stage IV s s.4.4 s 3 s. s 4.
42 Solution A Source Symbol Stage I Stage II Stage III Stage IV s s k.6 s.4.4 s 3 s. s 4.
43 Solution A Source Symbol Stage I Stage II Stage III Stage IV s s k Code.6 s.4.4 s 3 s. s 4.
44 Source Symbol Symbol Probability p k Solution A Cont d s. s s 2.4 s 3. s 4 Code word c k s k ( ) = H S L = = 22. ( S ) L ( S ) H < H + THIS IS NOT THE ONLY SOLUTION!
45 Alternate Solution B Source Symbol Stage I Stage II Stage III Stage IV s s k Code.6 s.4.4 s 3 s. s 4.
46 Source Symbol Alternative Solution B Cont d Symbol Probability p k s. s s 2.4 s 3. s 4 Code word c k s k ( ) = H S L = = 22. ( S ) L ( S ) H < H +
47 What is the difference between the two solutions? They have the same average length They differ in the variance of the average code length Solution A s 2 =.6 Solution B s 2 =.36 σ 2 K ( ) 2 p l L k k k = =
48 Source Coding Techniques. Huffman Code. 2. Two-path Huffman Code. 3. Lemple-Ziv Code. 4. Shannon Code. 5. Fano Code. 6. Arithmetic Code.
49 Source Coding Techniques 2. Two-path Huffman Code. This method is used when the probability of symbols in the information source is unknown. So we first can estimate this probability by calculating the number of occurrence of the symbols in the given message then we can find the possible Huffman codes. This can be summarized by the following two passes. Pass : Measure the occurrence possibility of each character in the message Pass 2 : Make possible Huffman codes
50 Source Coding Techniques 2. Two-path Huffman Code. Example Consider the input: ABABABABABACADABACADABACADABACAD
51 Source Coding Techniques. Huffman Code. 2. Two-path Huffman Code. 3. Lemple-Ziv Code. 4. Shannon Code. 5. Fano Code. 6. Arithmetic Code.
52 Lempel-Ziv Coding Huffman coding requires knowledge of a probabilistic model of the source This is not necessarily always feasible Lempel-Ziv code is an adaptive coding technique that does not require prior knowledge of symbol probabilities Lempel-Ziv coding is the basis of well-known ZIP for data compression
53 Lempel-Ziv Coding Example Codebook Index Subsequence Representation Encoding
54 Lempel-Ziv Coding Example Codebook Index Subsequence Representation Encoding
55 Lempel-Ziv Coding Example Codebook Index Subsequence Representation Encoding
56 Lempel-Ziv Coding Example Codebook Index Subsequence Representation Encoding
57 Lempel-Ziv Coding Example Codebook Index Subsequence Representation Encoding
58 Lempel-Ziv Coding Example Codebook Index Subsequence Representation Encoding
59 Lempel-Ziv Coding Example Codebook Index Subsequence Representation Encoding
60 Lempel-Ziv Coding Example Codebook Index Subsequence Representation Encoding
61 Lempel-Ziv Coding Example Information bits Source encoded bits Codebook Index Subsequence Representation Source Code
62 How Come this is Compression?! The hope is: If the bit sequence is long enough, eventually the fixed length code words will be shorter than the length of subsequences they represent. When applied to English text Lempel-Ziv achieves approximately 55% Huffman coding achieves approximately 43%
H(X,Y) = H(X) + H(Y X)
Today s Topics Iformatio Theory Mohamed Hamada oftware gieerig ab The Uiversity of Aizu mail: hamada@u-aizu.ac.jp UR: http://www.u-aizu.ac.jp/~hamada tropy review tropy ad Data Compressio Uiquely decodable
More informationIntroduction to Source Coding
Comm. 52: Communication Theory Lecture 7 Introduction to Source Coding - Requirements of source codes - Huffman Code Length Fixed Length Variable Length Source Code Properties Uniquely Decodable allow
More informationComm. 502: Communication Theory. Lecture 6. - Introduction to Source Coding
Comm. 50: Communication Theory Lecture 6 - Introduction to Source Coding Digital Communication Systems Source of Information User of Information Source Encoder Source Decoder Channel Encoder Channel Decoder
More informationCommunication Theory II
Communication Theory II Lecture 13: Information Theory (cont d) Ahmed Elnakib, PhD Assistant Professor, Mansoura University, Egypt March 22 th, 2015 1 o Source Code Generation Lecture Outlines Source Coding
More informationA Brief Introduction to Information Theory and Lossless Coding
A Brief Introduction to Information Theory and Lossless Coding 1 INTRODUCTION This document is intended as a guide to students studying 4C8 who have had no prior exposure to information theory. All of
More informationLECTURE VI: LOSSLESS COMPRESSION ALGORITHMS DR. OUIEM BCHIR
1 LECTURE VI: LOSSLESS COMPRESSION ALGORITHMS DR. OUIEM BCHIR 2 STORAGE SPACE Uncompressed graphics, audio, and video data require substantial storage capacity. Storing uncompressed video is not possible
More information# 12 ECE 253a Digital Image Processing Pamela Cosman 11/4/11. Introductory material for image compression
# 2 ECE 253a Digital Image Processing Pamela Cosman /4/ Introductory material for image compression Motivation: Low-resolution color image: 52 52 pixels/color, 24 bits/pixel 3/4 MB 3 2 pixels, 24 bits/pixel
More informationMultimedia Systems Entropy Coding Mahdi Amiri February 2011 Sharif University of Technology
Course Presentation Multimedia Systems Entropy Coding Mahdi Amiri February 2011 Sharif University of Technology Data Compression Motivation Data storage and transmission cost money Use fewest number of
More informationLecture5: Lossless Compression Techniques
Fixed to fixed mapping: we encoded source symbols of fixed length into fixed length code sequences Fixed to variable mapping: we encoded source symbols of fixed length into variable length code sequences
More informationInformation Theory and Huffman Coding
Information Theory and Huffman Coding Consider a typical Digital Communication System: A/D Conversion Sampling and Quantization D/A Conversion Source Encoder Source Decoder bit stream bit stream Channel
More informationInformation Theory and Communication Optimal Codes
Information Theory and Communication Optimal Codes Ritwik Banerjee rbanerjee@cs.stonybrook.edu c Ritwik Banerjee Information Theory and Communication 1/1 Roadmap Examples and Types of Codes Kraft Inequality
More informationCoding for Efficiency
Let s suppose that, over some channel, we want to transmit text containing only 4 symbols, a, b, c, and d. Further, let s suppose they have a probability of occurrence in any block of text we send as follows
More informationChapter 1 INTRODUCTION TO SOURCE CODING AND CHANNEL CODING. Whether a source is analog or digital, a digital communication
1 Chapter 1 INTRODUCTION TO SOURCE CODING AND CHANNEL CODING 1.1 SOURCE CODING Whether a source is analog or digital, a digital communication system is designed to transmit information in digital form.
More informationModule 8: Video Coding Basics Lecture 40: Need for video coding, Elements of information theory, Lossless coding. The Lecture Contains:
The Lecture Contains: The Need for Video Coding Elements of a Video Coding System Elements of Information Theory Symbol Encoding Run-Length Encoding Entropy Encoding file:///d /...Ganesh%20Rana)/MY%20COURSE_Ganesh%20Rana/Prof.%20Sumana%20Gupta/FINAL%20DVSP/lecture%2040/40_1.htm[12/31/2015
More information2.1. General Purpose Run Length Encoding Relative Encoding Tokanization or Pattern Substitution
2.1. General Purpose There are many popular general purpose lossless compression techniques, that can be applied to any type of data. 2.1.1. Run Length Encoding Run Length Encoding is a compression technique
More informationSOME EXAMPLES FROM INFORMATION THEORY (AFTER C. SHANNON).
SOME EXAMPLES FROM INFORMATION THEORY (AFTER C. SHANNON). 1. Some easy problems. 1.1. Guessing a number. Someone chose a number x between 1 and N. You are allowed to ask questions: Is this number larger
More informationMAS160: Signals, Systems & Information for Media Technology. Problem Set 4. DUE: October 20, 2003
MAS160: Signals, Systems & Information for Media Technology Problem Set 4 DUE: October 20, 2003 Instructors: V. Michael Bove, Jr. and Rosalind Picard T.A. Jim McBride Problem 1: Simple Psychoacoustic Masking
More informationComputing and Communications 2. Information Theory -Channel Capacity
1896 1920 1987 2006 Computing and Communications 2. Information Theory -Channel Capacity Ying Cui Department of Electronic Engineering Shanghai Jiao Tong University, China 2017, Autumn 1 Outline Communication
More informationDEPARTMENT OF INFORMATION TECHNOLOGY QUESTION BANK. Subject Name: Information Coding Techniques UNIT I INFORMATION ENTROPY FUNDAMENTALS
DEPARTMENT OF INFORMATION TECHNOLOGY QUESTION BANK Subject Name: Year /Sem: II / IV UNIT I INFORMATION ENTROPY FUNDAMENTALS PART A (2 MARKS) 1. What is uncertainty? 2. What is prefix coding? 3. State the
More informationMAS.160 / MAS.510 / MAS.511 Signals, Systems and Information for Media Technology Fall 2007
MIT OpenCourseWare http://ocw.mit.edu MAS.160 / MAS.510 / MAS.511 Signals, Systems and Information for Media Technology Fall 2007 For information about citing these materials or our Terms of Use, visit:
More informationCompression. Encryption. Decryption. Decompression. Presentation of Information to client site
DOCUMENT Anup Basu Audio Image Video Data Graphics Objectives Compression Encryption Network Communications Decryption Decompression Client site Presentation of Information to client site Multimedia -
More informationThe Lempel-Ziv (LZ) lossless compression algorithm was developed by Jacob Ziv (AT&T Bell Labs / Technion Israel) and Abraham Lempel (IBM) in 1978;
Georgia Institute of Technology - Georgia Tech Lorraine ECE 6605 Information Theory Lempel-Ziv Lossless Compresion General comments The Lempel-Ziv (LZ) lossless compression algorithm was developed by Jacob
More informationSolutions to Assignment-2 MOOC-Information Theory
Solutions to Assignment-2 MOOC-Information Theory 1. Which of the following is a prefix-free code? a) 01, 10, 101, 00, 11 b) 0, 11, 01 c) 01, 10, 11, 00 Solution:- The codewords of (a) are not prefix-free
More informationHuffman Coding - A Greedy Algorithm. Slides based on Kevin Wayne / Pearson-Addison Wesley
- A Greedy Algorithm Slides based on Kevin Wayne / Pearson-Addison Wesley Greedy Algorithms Greedy Algorithms Build up solutions in small steps Make local decisions Previous decisions are never reconsidered
More informationDigital Communication Systems ECS 452
Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong prapun@siit.tu.ac.th 2. Source Coding 1 Office Hours: BKD, 6th floor of Sirindhralai building Monday 10:00-10:40 Tuesday 12:00-12:40
More informationCommunication Theory II
Communication Theory II Lecture 14: Information Theory (cont d) Ahmed Elnakib, PhD Assistant Professor, Mansoura University, Egypt March 25 th, 2015 1 Previous Lecture: Source Code Generation: Lossless
More informationPROBABILITY AND STATISTICS Vol. II - Information Theory and Communication - Tibor Nemetz INFORMATION THEORY AND COMMUNICATION
INFORMATION THEORY AND COMMUNICATION Tibor Nemetz Rényi Mathematical Institute, Hungarian Academy of Sciences, Budapest, Hungary Keywords: Shannon theory, alphabet, capacity, (transmission) channel, channel
More informationModule 3 Greedy Strategy
Module 3 Greedy Strategy Dr. Natarajan Meghanathan Professor of Computer Science Jackson State University Jackson, MS 39217 E-mail: natarajan.meghanathan@jsums.edu Introduction to Greedy Technique Main
More informationModule 3 Greedy Strategy
Module 3 Greedy Strategy Dr. Natarajan Meghanathan Professor of Computer Science Jackson State University Jackson, MS 39217 E-mail: natarajan.meghanathan@jsums.edu Introduction to Greedy Technique Main
More informationECE Advanced Communication Theory, Spring 2007 Midterm Exam Monday, April 23rd, 6:00-9:00pm, ELAB 325
C 745 - Advanced Communication Theory, Spring 2007 Midterm xam Monday, April 23rd, 600-900pm, LAB 325 Overview The exam consists of five problems for 150 points. The points for each part of each problem
More informationDigital Communication Systems ECS 452
Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong prapun@siit.tu.ac.th Source Coding 1 Office Hours: BKD 3601-7 Monday 14:00-16:00 Wednesday 14:40-16:00 Noise & Interference Elements
More informationSpeech Coding in the Frequency Domain
Speech Coding in the Frequency Domain Speech Processing Advanced Topics Tom Bäckström Aalto University October 215 Introduction The speech production model can be used to efficiently encode speech signals.
More informationCOMM901 Source Coding and Compression Winter Semester 2013/2014. Midterm Exam
German University in Cairo - GUC Faculty of Information Engineering & Technology - IET Department of Communication Engineering Dr.-Ing. Heiko Schwarz COMM901 Source Coding and Compression Winter Semester
More information1 This work was partially supported by NSF Grant No. CCR , and by the URI International Engineering Program.
Combined Error Correcting and Compressing Codes Extended Summary Thomas Wenisch Peter F. Swaszek Augustus K. Uht 1 University of Rhode Island, Kingston RI Submitted to International Symposium on Information
More informationLecture 1 Introduction
Lecture 1 Introduction I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw September 22, 2015 1 / 46 I-Hsiang Wang IT Lecture 1 Information Theory Information
More information4. Which of the following channel matrices respresent a symmetric channel? [01M02] 5. The capacity of the channel with the channel Matrix
Send SMS s : ONJntuSpeed To 9870807070 To Recieve Jntu Updates Daily On Your Mobile For Free www.strikingsoon.comjntu ONLINE EXMINTIONS [Mid 2 - dc] http://jntuk.strikingsoon.com 1. Two binary random
More informationMULTIMEDIA SYSTEMS
1 Department of Computer Engineering, Faculty of Engineering King Mongkut s Institute of Technology Ladkrabang 01076531 MULTIMEDIA SYSTEMS Pk Pakorn Watanachaturaporn, Wt ht Ph.D. PhD pakorn@live.kmitl.ac.th,
More informationThe Need for Data Compression. Data Compression (for Images) -Compressing Graphical Data. Lossy vs Lossless compression
The Need for Data Compression Data Compression (for Images) -Compressing Graphical Data Graphical images in bitmap format take a lot of memory e.g. 1024 x 768 pixels x 24 bits-per-pixel = 2.4Mbyte =18,874,368
More informationA SURVEY ON DICOM IMAGE COMPRESSION AND DECOMPRESSION TECHNIQUES
A SURVEY ON DICOM IMAGE COMPRESSION AND DECOMPRESSION TECHNIQUES Shreya A 1, Ajay B.N 2 M.Tech Scholar Department of Computer Science and Engineering 2 Assitant Professor, Department of Computer Science
More informationCSE 100: BST AVERAGE CASE AND HUFFMAN CODES
CSE 100: BST AVERAGE CASE AND HUFFMAN CODES Recap: Average Case Analysis of successful find in a BST N nodes Expected total depth of all BSTs with N nodes Recap: Probability of having i nodes in the left
More informationB. Tech. (SEM. VI) EXAMINATION, (2) All question early equal make. (3) In ease of numerical problems assume data wherever not provided.
" 11111111111111111111111111111111111111111111111111111111111111III *U-3091/8400* Printed Pages : 7 TEC - 601! I i B. Tech. (SEM. VI) EXAMINATION, 2007-08 DIGIT AL COMMUNICATION \ V Time: 3 Hours] [Total
More informationWednesday, February 1, 2017
Wednesday, February 1, 2017 Topics for today Encoding game positions Constructing variable-length codes Huffman codes Encoding Game positions Some programs that play two-player games (e.g., tic-tac-toe,
More informationChapter 3 LEAST SIGNIFICANT BIT STEGANOGRAPHY TECHNIQUE FOR HIDING COMPRESSED ENCRYPTED DATA USING VARIOUS FILE FORMATS
44 Chapter 3 LEAST SIGNIFICANT BIT STEGANOGRAPHY TECHNIQUE FOR HIDING COMPRESSED ENCRYPTED DATA USING VARIOUS FILE FORMATS 45 CHAPTER 3 Chapter 3: LEAST SIGNIFICANT BIT STEGANOGRAPHY TECHNIQUE FOR HIDING
More informationFAST LEMPEL-ZIV (LZ 78) COMPLEXITY ESTIMATION USING CODEBOOK HASHING
FAST LEMPEL-ZIV (LZ 78) COMPLEXITY ESTIMATION USING CODEBOOK HASHING Harman Jot, Rupinder Kaur M.Tech, Department of Electronics and Communication, Punjabi University, Patiala, Punjab, India I. INTRODUCTION
More informationError Control Coding. Aaron Gulliver Dept. of Electrical and Computer Engineering University of Victoria
Error Control Coding Aaron Gulliver Dept. of Electrical and Computer Engineering University of Victoria Topics Introduction The Channel Coding Problem Linear Block Codes Cyclic Codes BCH and Reed-Solomon
More informationImages with (a) coding redundancy; (b) spatial redundancy; (c) irrelevant information
Images with (a) coding redundancy; (b) spatial redundancy; (c) irrelevant information 1992 2008 R. C. Gonzalez & R. E. Woods For the image in Fig. 8.1(a): 1992 2008 R. C. Gonzalez & R. E. Woods Measuring
More informationCourse Developer: Ranjan Bose, IIT Delhi
Course Title: Coding Theory Course Developer: Ranjan Bose, IIT Delhi Part I Information Theory and Source Coding 1. Source Coding 1.1. Introduction to Information Theory 1.2. Uncertainty and Information
More informationKINGS COLLEGE OF ENGINEERING DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING QUESTION BANK
KINGS COLLEGE OF ENGINEERING DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING QUESTION BANK SUB.NAME : COMMUNICATION THEORY SUB.CODE: EC1252 YEAR : II SEMESTER : IV UNIT I AMPLITUDE MODULATION SYSTEMS
More informationMonday, February 2, Is assigned today. Answers due by noon on Monday, February 9, 2015.
Monday, February 2, 2015 Topics for today Homework #1 Encoding checkers and chess positions Constructing variable-length codes Huffman codes Homework #1 Is assigned today. Answers due by noon on Monday,
More information6.450: Principles of Digital Communication 1
6.450: Principles of Digital Communication 1 Digital Communication: Enormous and normally rapidly growing industry, roughly comparable in size to the computer industry. Objective: Study those aspects of
More information6.004 Computation Structures Spring 2009
MIT OpenCourseWare http://ocw.mit.edu 6.004 Computation Structures Spring 2009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms. Welcome to 6.004! Course
More informationTarek M. Sobh and Tarek Alameldin
Operator/System Communication : An Optimizing Decision Tool Tarek M. Sobh and Tarek Alameldin Department of Computer and Information Science School of Engineering and Applied Science University of Pennsylvania,
More informationCOURSE MATERIAL Subject Name: Communication Theory UNIT V
NH-67, TRICHY MAIN ROAD, PULIYUR, C.F. - 639114, KARUR DT. DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING COURSE MATERIAL Subject Name: Communication Theory Subject Code: 080290020 Class/Sem:
More informationProblem Sheet 1 Probability, random processes, and noise
Problem Sheet 1 Probability, random processes, and noise 1. If F X (x) is the distribution function of a random variable X and x 1 x 2, show that F X (x 1 ) F X (x 2 ). 2. Use the definition of the cumulative
More informationECE/OPTI533 Digital Image Processing class notes 288 Dr. Robert A. Schowengerdt 2003
Motivation Large amount of data in images Color video: 200Mb/sec Landsat TM multispectral satellite image: 200MB High potential for compression Redundancy (aka correlation) in images spatial, temporal,
More informationMulticasting over Multiple-Access Networks
ing oding apacity onclusions ing Department of Electrical Engineering and omputer Sciences University of alifornia, Berkeley May 9, 2006 EE 228A Outline ing oding apacity onclusions 1 2 3 4 oding 5 apacity
More information15.Calculate the local oscillator frequency if incoming frequency is F1 and translated carrier frequency
DEPARTMENT OF ELECTRONICS & COMMUNICATION ENGINEERING SUBJECT NAME:COMMUNICATION THEORY YEAR/SEM: II/IV SUBJECT CODE: EC 6402 UNIT I:l (AMPLITUDE MODULATION) PART A 1. Compute the bandwidth of the AMP
More informationEntropy Coding. Outline. Entropy. Definitions. log. A = {a, b, c, d, e}
Outline efinition of ntroy Three ntroy coding techniques: Huffman coding rithmetic coding Lemel-Ziv coding ntroy oding (taken from the Technion) ntroy ntroy of a set of elements e,,e n with robabilities,
More informationLecture #2. EE 471C / EE 381K-17 Wireless Communication Lab. Professor Robert W. Heath Jr.
Lecture #2 EE 471C / EE 381K-17 Wireless Communication Lab Professor Robert W. Heath Jr. Preview of today s lecture u Introduction to digital communication u Components of a digital communication system
More informationRab Nawaz. Prof. Zhang Wenyi
Rab Nawaz PhD Scholar (BL16006002) School of Information Science and Technology University of Science and Technology of China, Hefei Email: rabnawaz@mail.ustc.edu.cn Submitted to Prof. Zhang Wenyi wenyizha@ustc.edu.cn
More informationMATHEMATICS IN COMMUNICATIONS: INTRODUCTION TO CODING. A Public Lecture to the Uganda Mathematics Society
Abstract MATHEMATICS IN COMMUNICATIONS: INTRODUCTION TO CODING A Public Lecture to the Uganda Mathematics Society F F Tusubira, PhD, MUIPE, MIEE, REng, CEng Mathematical theory and techniques play a vital
More informationSYLLABUS of the course BASIC ELECTRONICS AND DIGITAL SIGNAL PROCESSING. Master in Computer Science, University of Bolzano-Bozen, a.y.
SYLLABUS of the course BASIC ELECTRONICS AND DIGITAL SIGNAL PROCESSING Master in Computer Science, University of Bolzano-Bozen, a.y. 2017-2018 Lecturer: LEONARDO RICCI (last updated on November 27, 2017)
More informationDCSP-3: Minimal Length Coding. Jianfeng Feng
DCSP-3: Minimal Length Coding Jianfeng Feng Department of Computer Science Warwick Univ., UK Jianfeng.feng@warwick.ac.uk http://www.dcs.warwick.ac.uk/~feng/dcsp.html Automatic Image Caption (better than
More informationCHAPTER 5 PAPR REDUCTION USING HUFFMAN AND ADAPTIVE HUFFMAN CODES
119 CHAPTER 5 PAPR REDUCTION USING HUFFMAN AND ADAPTIVE HUFFMAN CODES 5.1 INTRODUCTION In this work the peak powers of the OFDM signal is reduced by applying Adaptive Huffman Codes (AHC). First the encoding
More informationKeywords Audio Steganography, Compressive Algorithms, SNR, Capacity, Robustness. (Figure 1: The Steganographic operation) [10]
Volume 4, Issue 5, May 2014 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Audio Steganography
More informationBell Labs celebrates 50 years of Information Theory
1 Bell Labs celebrates 50 years of Information Theory An Overview of Information Theory Humans are symbol-making creatures. We communicate by symbols -- growls and grunts, hand signals, and drawings painted
More informationError Detection and Correction: Parity Check Code; Bounds Based on Hamming Distance
Error Detection and Correction: Parity Check Code; Bounds Based on Hamming Distance Greg Plaxton Theory in Programming Practice, Spring 2005 Department of Computer Science University of Texas at Austin
More informationREVIEW OF IMAGE COMPRESSION TECHNIQUES FOR MULTIMEDIA IMAGES
REVIEW OF IMAGE COMPRESSION TECHNIQUES FOR MULTIMEDIA IMAGES 1 Tamanna, 2 Neha Bassan 1 Student- Department of Computer science, Lovely Professional University Phagwara 2 Assistant Professor, Department
More informationLossless Grayscale Image Compression using Blockwise Entropy Shannon (LBES)
Volume No., July Lossless Grayscale Image Compression using Blockwise ntropy Shannon (LBS) S. Anantha Babu Ph.D. (Research Scholar) & Assistant Professor Department of Computer Science and ngineering V
More informationImage Processing Computer Graphics I Lecture 20. Display Color Models Filters Dithering Image Compression
15-462 Computer Graphics I Lecture 2 Image Processing April 18, 22 Frank Pfenning Carnegie Mellon University http://www.cs.cmu.edu/~fp/courses/graphics/ Display Color Models Filters Dithering Image Compression
More informationGENERIC CODE DESIGN ALGORITHMS FOR REVERSIBLE VARIABLE-LENGTH CODES FROM THE HUFFMAN CODE
GENERIC CODE DESIGN ALGORITHMS FOR REVERSIBLE VARIABLE-LENGTH CODES FROM THE HUFFMAN CODE Wook-Hyun Jeong and Yo-Sung Ho Kwangju Institute of Science and Technology (K-JIST) Oryong-dong, Buk-gu, Kwangju,
More informationChapter 6: Memory: Information and Secret Codes. CS105: Great Insights in Computer Science
Chapter 6: Memory: Information and Secret Codes CS105: Great Insights in Computer Science Overview When we decide how to represent something in bits, there are some competing interests: easily manipulated/processed
More informationChannel Concepts CS 571 Fall Kenneth L. Calvert
Channel Concepts CS 571 Fall 2006 2006 Kenneth L. Calvert What is a Channel? Channel: a means of transmitting information A means of communication or expression Webster s NCD Aside: What is information...?
More informationLossless Image Compression Techniques Comparative Study
Lossless Image Compression Techniques Comparative Study Walaa Z. Wahba 1, Ashraf Y. A. Maghari 2 1M.Sc student, Faculty of Information Technology, Islamic university of Gaza, Gaza, Palestine 2Assistant
More informationFREDRIK TUFVESSON ELECTRICAL AND INFORMATION TECHNOLOGY
1 Information Transmission Chapter 5, Block codes FREDRIK TUFVESSON ELECTRICAL AND INFORMATION TECHNOLOGY 2 Methods of channel coding For channel coding (error correction) we have two main classes of codes,
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 1: Introduction & Overview Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 1 / 26 Outline 1 Course Information 2 Course Overview
More informationInformation Theory: the Day after Yesterday
: the Day after Yesterday Department of Electrical Engineering and Computer Science Chicago s Shannon Centennial Event September 23, 2016 : the Day after Yesterday IT today Outline The birth of information
More informationMasters of Engineering in Electrical Engineering Course Syllabi ( ) City University of New York--College of Staten Island
City University of New York--College of Staten Island Masters of Engineering in Electrical Engineering Course Syllabi (2017-2018) Required Core Courses ELE 600/ MTH 6XX Probability Theory and Stochastic
More informationScheduling in omnidirectional relay wireless networks
Scheduling in omnidirectional relay wireless networks by Shuning Wang A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Master of Applied Science
More informationComputer Science 1001.py. Lecture 25 : Intro to Error Correction and Detection Codes
Computer Science 1001.py Lecture 25 : Intro to Error Correction and Detection Codes Instructors: Daniel Deutch, Amiram Yehudai Teaching Assistants: Michal Kleinbort, Amir Rubinstein School of Computer
More informationSHANNON S source channel separation theorem states
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 55, NO. 9, SEPTEMBER 2009 3927 Source Channel Coding for Correlated Sources Over Multiuser Channels Deniz Gündüz, Member, IEEE, Elza Erkip, Senior Member,
More informationBackground Dirty Paper Coding Codeword Binning Code construction Remaining problems. Information Hiding. Phil Regalia
Information Hiding Phil Regalia Department of Electrical Engineering and Computer Science Catholic University of America Washington, DC 20064 regalia@cua.edu Baltimore IEEE Signal Processing Society Chapter,
More informationError-Correcting Codes
Error-Correcting Codes Information is stored and exchanged in the form of streams of characters from some alphabet. An alphabet is a finite set of symbols, such as the lower-case Roman alphabet {a,b,c,,z}.
More informationBSc (Hons) Computer Science with Network Security, BEng (Hons) Electronic Engineering. Cohorts: BCNS/17A/FT & BEE/16B/FT
BSc (Hons) Computer Science with Network Security, BEng (Hons) Electronic Engineering Cohorts: BCNS/17A/FT & BEE/16B/FT Examinations for 2016-2017 Semester 2 & 2017 Semester 1 Resit Examinations for BEE/12/FT
More informationFundamentals of Digital Communications and Data Transmission
Fundamentals of Digital Communications and Data Transmission 29 th October 2008 Abdullah Al-Meshal Overview Introduction Communication systems Digital communication system Importance of Digital transmission
More informationPooja Rani(M.tech) *, Sonal ** * M.Tech Student, ** Assistant Professor
A Study of Image Compression Techniques Pooja Rani(M.tech) *, Sonal ** * M.Tech Student, ** Assistant Professor Department of Computer Science & Engineering, BPS Mahila Vishvavidyalya, Sonipat kulriapooja@gmail.com,
More informationTCET3202 Analog and digital Communications II
NEW YORK CITY COLLEGE OF TECHNOLOGY The City University of New York DEPARTMENT: SUBJECT CODE AND TITLE: COURSE DESCRIPTION: REQUIRED COURSE Electrical and Telecommunications Engineering Technology TCET3202
More informationArithmetic Compression on SPIHT Encoded Images
Arithmetic Compression on SPIHT Encoded Images Todd Owen, Scott Hauck {towen, hauck}@ee.washington.edu Dept of EE, University of Washington Seattle WA, 98195-2500 UWEE Technical Report Number UWEETR-2002-0007
More information6.02 Introduction to EECS II Spring Quiz 1
M A S S A C H U S E T T S I N S T I T U T E O F T E C H N O L O G Y DEPARTMENT OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE 6.02 Introduction to EECS II Spring 2011 Quiz 1 Name SOLUTIONS Score Please
More informationByte = More common: 8 bits = 1 byte Abbreviation:
Text, Images, Video and Sound ASCII-7 In the early days, a was used, with of 0 s and 1 s, enough for a typical keyboard. The standard was developed by (American Standard Code for Information Interchange)
More informationEngineering Scope and Sequence Student Outcomes (Objectives Skills/Verbs)
The World of Modern Engineering What do Scientists and Engineers do? What is the difference between analog and digital devices? How do Engineers organize their designs? Introduction to LabView software
More informationChapter 1 Coding for Reliable Digital Transmission and Storage
Wireless Information Transmission System Lab. Chapter 1 Coding for Reliable Digital Transmission and Storage Institute of Communications Engineering National Sun Yat-sen University 1.1 Introduction A major
More informationThe ternary alphabet is used by alternate mark inversion modulation; successive ones in data are represented by alternating ±1.
Alphabets EE 387, Notes 2, Handout #3 Definition: An alphabet is a discrete (usually finite) set of symbols. Examples: B = {0,1} is the binary alphabet T = { 1,0,+1} is the ternary alphabet X = {00,01,...,FF}
More informationEENG 444 / ENAS 944 Digital Communication Systems
EENG 444 / ENAS 944 Digital Communication Systems Introduction!! Wenjun Hu Communication Systems What s the first thing that comes to your mind? Communication Systems What s the first thing that comes
More informationUNIT 7C Data Representation: Images and Sound
UNIT 7C Data Representation: Images and Sound 1 Pixels An image is stored in a computer as a sequence of pixels, picture elements. 2 1 Resolution The resolution of an image is the number of pixels used
More informationError Detection and Correction
. Error Detection and Companies, 27 CHAPTER Error Detection and Networks must be able to transfer data from one device to another with acceptable accuracy. For most applications, a system must guarantee
More informationUCSD ECE154C Handout #21 Prof. Young-Han Kim Thursday, April 28, Midterm Solutions (Prepared by TA Shouvik Ganguly)
UCSD ECE54C Handout #2 Prof. Young-Han Kim Thursday, April 28, 26 Midterm Solutions (Prepared by TA Shouvik Ganguly) There are 3 problems, each problem with multiple parts, each part worth points. Your
More informationA Hybrid Technique for Image Compression
Australian Journal of Basic and Applied Sciences, 5(7): 32-44, 2011 ISSN 1991-8178 A Hybrid Technique for Image Compression Hazem (Moh'd Said) Abdel Majid Hatamleh Computer DepartmentUniversity of Al-Balqa
More information5984 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 12, DECEMBER 2010
5984 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 12, DECEMBER 2010 Interference Channels With Correlated Receiver Side Information Nan Liu, Member, IEEE, Deniz Gündüz, Member, IEEE, Andrea J.
More informationIntroduction to Error Control Coding
Introduction to Error Control Coding 1 Content 1. What Error Control Coding Is For 2. How Coding Can Be Achieved 3. Types of Coding 4. Types of Errors & Channels 5. Types of Codes 6. Types of Error Control
More information