Lecture 1 Introduction
|
|
- Kerry Tyler
- 6 years ago
- Views:
Transcription
1 Lecture 1 Introduction I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw September 22, / 46 I-Hsiang Wang IT Lecture 1
2 Information Theory Information Theory is a mathematical theory of information Information is usually obtained by getting some messages (speech, text, images, etc.) from others. When obtaining information from a message, you may care about: What is the meaning of a message? How important is the message? How much information can I get from the message? 2 / 46 I-Hsiang Wang IT Lecture 1
3 Information Theory Information Theory is a mathematical theory of information. Information is usually obtained by getting some messages (speech, text, images, etc.) from others. When obtaining information from a message, you may care about: What is the meaning of a message? How important is the message? How much information can I get from the message? Information theory is about the quantification of information. 3 / 46 I-Hsiang Wang IT Lecture 1
4 Information Theory Information Theory is a mathematical theory of information (primarily for communication systems) that Establishes the fundamental limits of communication systems (Quantifies the amount of information that can be delivered from a party to another) Built upon probability theory and statistics Main concern: ultimate performance limit (usually the rate of information processing) as certain resources (usually the total amount of time) scales to the asymptotic regime, given that the desired information is delivered reliably. 4 / 46 I-Hsiang Wang IT Lecture 1
5 In this course, we will 1 Establish solid foundations and intuitions of information theory, 2 Introduce explicit methods to achieve information theoretic limits, 3 Demonstrate further applications of information theory beyond communications. Later, we begin with a brief overview of information theory and the materials to be covered in this course. 5 / 46 I-Hsiang Wang IT Lecture 1
6 1 Course Information 2 6 / 46 I-Hsiang Wang IT Lecture 1
7 Logistics 1 Instructor: I-Hsiang Wang 王奕翔 ihwang@ntu.edu.tw Office: MD-524 明達館 524 室 Office Hours: 17:00 18:00, Monday and Tuesday 2 Lecture Time: 13:20 14:10 (6) Tuesday, and 10:20 12:10 (34) Wednesday 3 Lecture Location: EE2-225 電機二館 225 室 4 Course Website: 5 Prerequisites: Probability, Linear Algebra. 7 / 46 I-Hsiang Wang IT Lecture 1
8 Logistics 6 Grading: Homework (35%), Midterm (30%), Final (35%) 7 References T. Cover and J. Thomas, Elements of Information Theory, 2nd Edition, Wiley-Interscience, R. Gallager, Information Theory and Reliable Communications, Wiley, I. Csiszar and J. Korner, Information Theory: Coding Theorems for Discrete Memoryless Systems, 2nd Edition, Cambridge University Press, S. M. Moser, Information Theory (Lecture Notes), 4th edition, ISI Lab, ETH Zürich, Switzerland, R. Yeung, Information Theory and Network Coding, Springer, A. El Gamal and Y.-H. Kim, Network Information Theory, Cambridge University Press, / 46 I-Hsiang Wang IT Lecture 1
9 Homework 1 Roughly 5 6 problems every two weeks, in total 7 times. 2 Homework (HW) is usually released on Monday. Deadline of submission is usually on the next Wednesday in class. 3 Late homework = 0 points. (Let me know in advance if you have difficulties.) 4 Everyone has to develop detailed solution for one HW problem, documented in L A TEX and submitted 1 week after the HW due. We will provide L A TEX templates, and you should discuss with the instructor about the homework problem that you are in charge of, making sure the solution is correct. 5 This additional effort accounts for part of your homework grades. 9 / 46 I-Hsiang Wang IT Lecture 1
10 Reading and Lecture Notes 1 Slides: Slides are usually released/updated every Sunday evening. 2 Readings: Each lecture has assigned readings. Reading is required: it is not enough to learn from the slides! 3 Go through the slides and the assigned readings before our lectures. It helps you learn better. 4 I recommend you get a copy of the textbook by Cover and Thomas. It is a good reference, and we will often assign readings in the book. 5 Other assigned readings could be Moser s lecture note (can be obtained online) and relevant papers. 10 / 46 I-Hsiang Wang IT Lecture 1
11 Interaction 1 In-class: Language: This class is taught in English. However, to encourage interaction, feel free to ask questions in Mandarin. I will repeat your question in English (if necessary), and answer it in English. Exercises: We put some exercises on the slides to help you learn and understand. Occasionally, I will call for volunteer to solve the exercises in class. Volunteers get bonus. 2 Out-of-class: Office Hours: Both TA and myself have 2-hour office hours per week. You are more than welcome to come visit us and ask questions, discuss about research, chat, complain, etc. If you cannot make it to the regular office hours, send us s to schedule a time slot. My schedule can be found in my website. Send us s with a subject starting with [NTU Fall15 IT]. Feedback: There will be online polls during the semester to collect your feedback anonymously. 11 / 46 I-Hsiang Wang IT Lecture 1
12 Course Outline Measures of Information: entropy, conditional entropy, relative entropy (KL divergence), mutual information. Lossless Coding: lossless source coding theorem, discrete memoryless sources, asymptotic equipartition property, typical sequences, Fano s inequality, converse proof, ergodic sources, entropy rate. Noisy Coding: noisy channel coding theorem, discrete memoryless channels, random coding, typicality decoder, threshold decoder, error probability analysis, converse proof, channel with feedback. Coding over Continuous Valued s: channel coding with cost constraints, discretization technique, differential entropy, Gaussian channel capacity. 12 / 46 I-Hsiang Wang IT Lecture 1
13 Course Outline Lossy Coding (Rate Distortion Theory): distortion, rate-distortion tradeoff, typicality encoder, converse proof. - Separation and Joint - Coding Information Theory and Statistics: method of types, Sanov s theorem, large deviation, hypothesis testing, estimation, Cramér-Rao lower bound, non-parametric estimation. Data Compression: prefix-free code, Kraft s inequality, Huffman code, Lempel-Ziv compression. Capacity Achieving Codes: polar codes, LDPC codes. Selected Advanced Topics: network coding, compressed sensing, community detection, non-asymptotic information theory, etc. 13 / 46 I-Hsiang Wang IT Lecture 1
14 Tentative Schedule Week Date Content Remark 1 09/15, 16 Introduction; Measures of Information 2 09/22, 23 Measures of Information 3 09/29, 30 Lossless Coding HW1 out 4 10/06, 07 Lossless Coding HW1 due 5 10/13, 14 Noisy Coding HW2 out 6 10/20, 21 Noisy Coding HW2 due 7 10/27, 28 Continuous-Valued Coding HW3 out 8 11/03, 04 Continuous-Valued Coding HW3 due 9 11/10, 11 Midterm Exam 14 / 46 I-Hsiang Wang IT Lecture 1
15 Tentative Schedule Week Date Content Remark 10 11/17, 18 Lossy Coding HW4 out 11 11/24, 25 Joint - Coding HW4 due 12 12/01, 02 Information Theory and Statistics HW5 out 13 12/08, 09 Information Theory and Statistics HW5 due 14 12/15, 16 Data Compression HW6 out 15 12/22, 23 Data Compression; Polar Code HW6 due 16 12/29, 30 Polar Code HW7 out 17 01/05, 06 Advanced Topics HW7 due 18 01/12, 13 Final Exam 15 / 46 I-Hsiang Wang IT Lecture 1
16 1 Course Information 2 16 / 46 I-Hsiang Wang IT Lecture 1
17 17 / 46 I-Hsiang Wang IT Lecture 1
18 Claude E. Shannon ( ) 18 / 46 I-Hsiang Wang IT Lecture 1
19 Information theory is a mathematical theory of communication 19 / 46 I-Hsiang Wang IT Lecture 1
20 20 / 46 I-Hsiang Wang IT Lecture 1
21 Information theory is the mathematical theory of communication 21 / 46 I-Hsiang Wang IT Lecture 1
22 Origin of Information Theory 22 / 46 I-Hsiang Wang IT Lecture 1
23 Origin of Information Theory 23 / 46 I-Hsiang Wang IT Lecture 1
24 Origin of Information Theory Shannon s landmark paper in 1948 is generally considered as the birth of information theory. In the paper, Shannon set it clear that information theory is about the quantification of information in a communication system. In particular, it focuses on characterizing the necessary and sufficient condition of whether or not a destination terminal can reproduce a message generated by a source terminal. 24 / 46 I-Hsiang Wang IT Lecture 1
25 What is Information Theory about? 25 / 46 I-Hsiang Wang IT Lecture 1
26 It is about the analysis of fundamental limits 1 Stochastic modeling It is a unified theory based on stochastic modeling (information source, noisy channel, etc.). 2 Theorems, not only definitions It provides mathematical theorems on optimal performance of algorithms (coding schemes), rather than merely definitions. 3 Sharp phase transition It draws the boundary between what is possible to achieve and what is impossible, leading to math-driven system design. 26 / 46 I-Hsiang Wang IT Lecture 1
27 Engineering)Design)Driver Engineering)Design)Driver It is about the design driven by theory Information theory not only gives fundamental limits, but also provides guidelines suggesting how to achieve them Information theory not only gives fundamental limits, but also provides Some guidelines examples: suggesting how to achieve them Some examples: river Universal lossless data compression compression mental lossless limits, but compression w to achieve them Error correcting code satellite communication satellite communication Cellular system DSL modems DSL modems Wireless accessetc. network mobile communication DSL modems etc. mobile communication DSL modem wireless access networks wireless access networks 10 and much more! / 46 I-Hsiang Wang IT Lecture 1 etc. Cryptography
28 Communication System Noise Encoder Decoder Destination Above is an abstract model of communication system: 1 The source would like to deliver some message to the destination, where the message includes speech, image, video, audio, text, etc. 2 The channel is the physical medium that connects the source and the destination, such as cable, optical fiber, EM radiation, etc., and is usually subject to certain noise disturbances. 3 The encoder can carry out any processing of the source output, including compression, modulation, insertion of redundancy, etc. 4 The decoder can carry out any processing of the channel output to reproduce the source message. 28 / 46 I-Hsiang Wang IT Lecture 1
29 A primary concern of information theory is on the encoder and the decoder, both in terms of How the encoder and the decoder function, and The existence or nonexistence of encoders and decoders that achieve a given level of performance 29 / 46 I-Hsiang Wang IT Lecture 1
30 Prior to the 1948 paper, design of communication systems followed the analog paradigm if the source produces a electromagnetic waveform, the destination should try its best to reconstruct this waveform, in order to extract useful information (usually, voice). This line of research was based on Fourier analysis and gave birth to sampling theory. Shannon asked: If the receiver knows that a sine wave of unknown frequency is to be communicated, why not simply send the frequency rather than the entire waveform? Prior to Shannon, theorist and engineers were able to analyze the performance of certain choice of encoders/decoders, but had little knowledge about what is the ultimate limit. Shannon asked: For all possible encoders/decoders, what is the necessary and sufficient condition for the destination to be able to reconstruct the message sent from the source? 30 / 46 I-Hsiang Wang IT Lecture 1
31 Shannon s View Noise Encoder Decoder Destination Key new insights due to Shannon s work: Shannon: Information is the resolution of uncertainty. Indeed, the set of possible source outputs, rather than any particular output, is of primary interest. Introduction of an abstract mathematical model of communication system based on random processes (hence, a stochastic model) Creation of the digital paradigm of communication system design bit as the universal currency of information, by proposing and proving the source-channel separation theorem 31 / 46 I-Hsiang Wang IT Lecture 1
32 Stochastic Modeling Noise Encoder Decoder Destination The stochastic modeling of communication system comprises: : model the information source by random processes, where the data to be conveyed is drawn randomly from a given distribution : model the noisy channel by random processes, where the impact of noise is drawn randomly from a given distribution Why use random processes to model communication system? Shannon: The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design. 32 / 46 I-Hsiang Wang IT Lecture 1
33 - Separation Encoder Bits Encoder Binary Interface Noisy Destination Decoder Bits Decoder Shannon showed that by splitting the coders into source coders and channel coders, the fundamental limit of the system remains the same. In other words, introducing a digital (binary) interface does not incur any loss of optimality, in terms of whether or not the destination can reproduce the source data. Separation of source coding and channel coding simplifies engineering design source coder design (data compression) and channel coder design (data transmission). 33 / 46 I-Hsiang Wang IT Lecture 1
34 I have been always wondering how on earth Shannon came up with the brilliant idea of separating source coding and channel coding. A very likely answer: Shannon is simply a genius. A more down-to-earth one: Shannon saw the essence of research: seek for simplification first. 34 / 46 I-Hsiang Wang IT Lecture 1
35 Original Block Diagram Noise Encoder Decoder Destination Simplification: Remove the Noise! Noise Encoder Decoder Destination This step makes life much easier. Yet, it is still a non-trivial problem. 35 / 46 I-Hsiang Wang IT Lecture 1
36 Coding (Data Compression) Encoder Encoder Noisy Destination Decoder Decoder Features of source messages: Uncertainty: the destination has no idea what message is chosen by the source a priori. Redundancy: though randomly chosen, some choices are more likely, while others are less likely. Goal: Remove redundancy of the source message and represent it by a bit sequence, so that it can be delivered to the destination reliably. 36 / 46 I-Hsiang Wang IT Lecture 1
37 Coding (Data Compression) Destination s[1],...,s[n] bs[1],...,bs[n] Encoder Decoder Encoder b[1],...,b[k] Decoder Noisy Notations: {s[1],..., s[n]} represent the source message; each s[t] is called a source symbol. {b[1],..., b[k]} represent the codeword, generated by the source encoder; each bit b[t] is called a source codeword symbol (bit). {ŝ[1],..., ŝ[n]} represent the reproduced source message at the destination. 37 / 46 I-Hsiang Wang IT Lecture 1
38 Coding (Data Compression) Destination s[1],...,s[n] bs[1],...,bs[n] Encoder Decoder Encoder b[1],...,b[k] Decoder Noisy Question: For a given N (# of source symbols), what is the minimum K (# of bits) to recover s[1],..., s[n] at the decoder? It is not hard to show that the smallest K = Θ (N). (check!) The right (non-trivial) question to ask is: What is the minimum value of K N? 38 / 46 I-Hsiang Wang IT Lecture 1
39 Coding (Data Compression) Destination s[1],...,s[n] bs[1],...,bs[n] Encoder Decoder Encoder b[1],...,b[k] Decoder Noisy Shannon answered the above question and characterized the necessary and sufficient condition for (lossless) source coding: A Coding Theorem The destination can reconstruct the source message losslessly code rate R := K N > the entropy rate of the source, H(S) We will define entropy in Lecture 2; it is a quantity that can be computed from the distribution of the source random process {S[t] t N}. 39 / 46 I-Hsiang Wang IT Lecture 1
40 Original Block Diagram Noise Encoder Decoder Destination Simplification : Remove the Redundancy! i.i.d. Bernoulli(1/2) i.e., Random Bits Noise Encoder Decoder Destination It remains a highly non-trivial problem. 40 / 46 I-Hsiang Wang IT Lecture 1
41 Coding (Data Transmission) Encoder Encoder Noisy Destination Decoder Decoder Features of noisy channel: Noise: channel input sent by the channel encoder is corrupted by the noise randomly, and produce the channel output. Uniform messages: input of channel encoders are assumed WLOG to be bit sequences with no redundancy, since source coding already removes all redundancy and convert to bit sequences. Goal: add minimum redundancy so that messages (bit sequences) can be communicated over the noisy channel and decoded reliably. 41 / 46 I-Hsiang Wang IT Lecture 1
42 Coding (Data Transmission) Encoder b[1],...,b[k] Encoder x[1],...,x[n] p (y x) Noisy Destination Decoder b b[1],..., b b[k] Decoder y[1],...,y[n] Notations: {x[1],..., x[n]} represent the codeword; each x[t] is called a coded symbol. {b[1],..., b[k]} represent the message; each bit b[t] is called a data symbol (bit). {y[1],..., y[n]} represent the channel output. 42 / 46 I-Hsiang Wang IT Lecture 1
43 Coding (Data Transmission) Encoder b[1],...,b[k] Encoder x[1],...,x[n] p (y x) Noisy Destination Decoder b b[1],..., b b[k] Decoder y[1],...,y[n] Question: For a given K (# of input bits), what is the minimum N (# of coded symbols) to recover b[1],..., b[k] at the decoder? It turns out that N = Θ (K). However, proving this is already non-trivial. Shannon further ask: What is the maximum value of K N? 43 / 46 I-Hsiang Wang IT Lecture 1
44 Coding (Data Transmission) Encoder b[1],...,b[k] Encoder x[1],...,x[n] p (y x) Noisy Destination Decoder b b[1],..., b b[k] Decoder y[1],...,y[n] Shannon gave the necessary and sufficient condition for channel coding: A Coding Theorem The (channel) decoder can decode the message reliably code rate R := K N < the channel capacity of the channel, C We will define channel capacity later; it is a quantity that can be computed by maximizing the mutual information between X and Y, which can be computed from the conditional distribution of the channel. 44 / 46 I-Hsiang Wang IT Lecture 1
45 Summary 45 / 46 I-Hsiang Wang IT Lecture 1
46 Information theory focuses on the quantitative aspects of information, not the qualitative aspects Information theory is mainly about what is possible and what is impossible in communication systems In information theory, one investigates problems in communication systems through the lens of probability theory and statistics In the course, we mainly focus on discrete-time signals, not on continuous-time signals -channel separation forms the basis of digital communication, where binary digits (bits) become the universal currency of information 46 / 46 I-Hsiang Wang IT Lecture 1
ECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 1: Introduction & Overview Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 1 / 26 Outline 1 Course Information 2 Course Overview
More informationECE Advanced Communication Theory, Spring 2007 Midterm Exam Monday, April 23rd, 6:00-9:00pm, ELAB 325
C 745 - Advanced Communication Theory, Spring 2007 Midterm xam Monday, April 23rd, 600-900pm, LAB 325 Overview The exam consists of five problems for 150 points. The points for each part of each problem
More informationMasters of Engineering in Electrical Engineering Course Syllabi ( ) City University of New York--College of Staten Island
City University of New York--College of Staten Island Masters of Engineering in Electrical Engineering Course Syllabi (2017-2018) Required Core Courses ELE 600/ MTH 6XX Probability Theory and Stochastic
More informationENSC327/328 Communication Systems Course Information. Paul Ho Professor School of Engineering Science Simon Fraser University
ENSC327/328 Communication Systems Course Information Paul Ho Professor School of Engineering Science Simon Fraser University 1 Schedule & Instructor Class Schedule: Mon 2:30 4:20pm AQ 3159 Wed 1:30 2:20pm
More informationCommunication Theory II
Communication Theory II Lecture 13: Information Theory (cont d) Ahmed Elnakib, PhD Assistant Professor, Mansoura University, Egypt March 22 th, 2015 1 o Source Code Generation Lecture Outlines Source Coding
More informationCOMM901 Source Coding and Compression Winter Semester 2013/2014. Midterm Exam
German University in Cairo - GUC Faculty of Information Engineering & Technology - IET Department of Communication Engineering Dr.-Ing. Heiko Schwarz COMM901 Source Coding and Compression Winter Semester
More informationEENG 444 / ENAS 944 Digital Communication Systems
EENG 444 / ENAS 944 Digital Communication Systems Introduction!! Wenjun Hu Communication Systems What s the first thing that comes to your mind? Communication Systems What s the first thing that comes
More informationECE 8771, Information Theory & Coding for Digital Communications Summer 2010 Syllabus & Outline (Draft 1 - May 12, 2010)
ECE 8771, Information Theory & Coding for Digital Communications Summer 2010 Syllabus & Outline (Draft 1 - May 12, 2010) Instructor: Kevin Buckley, Tolentine 433a, 610-519-5658 (W), 610-519-4436 (F), buckley@ece.vill.edu,
More informationInformation Theory and Huffman Coding
Information Theory and Huffman Coding Consider a typical Digital Communication System: A/D Conversion Sampling and Quantization D/A Conversion Source Encoder Source Decoder bit stream bit stream Channel
More informationEntropy, Coding and Data Compression
Entropy, Coding and Data Compression Data vs. Information yes, not, yes, yes, not not In ASCII, each item is 3 8 = 24 bits of data But if the only possible answers are yes and not, there is only one bit
More informationInformation Theory and Communication Optimal Codes
Information Theory and Communication Optimal Codes Ritwik Banerjee rbanerjee@cs.stonybrook.edu c Ritwik Banerjee Information Theory and Communication 1/1 Roadmap Examples and Types of Codes Kraft Inequality
More informationInformation Theory: the Day after Yesterday
: the Day after Yesterday Department of Electrical Engineering and Computer Science Chicago s Shannon Centennial Event September 23, 2016 : the Day after Yesterday IT today Outline The birth of information
More informationDetection and Estimation of Signals in Noise. Dr. Robert Schober Department of Electrical and Computer Engineering University of British Columbia
Detection and Estimation of Signals in Noise Dr. Robert Schober Department of Electrical and Computer Engineering University of British Columbia Vancouver, August 24, 2010 2 Contents 1 Basic Elements
More informationChapter 1 INTRODUCTION TO SOURCE CODING AND CHANNEL CODING. Whether a source is analog or digital, a digital communication
1 Chapter 1 INTRODUCTION TO SOURCE CODING AND CHANNEL CODING 1.1 SOURCE CODING Whether a source is analog or digital, a digital communication system is designed to transmit information in digital form.
More informationComputing and Communications 2. Information Theory -Channel Capacity
1896 1920 1987 2006 Computing and Communications 2. Information Theory -Channel Capacity Ying Cui Department of Electronic Engineering Shanghai Jiao Tong University, China 2017, Autumn 1 Outline Communication
More informationLECTURE VI: LOSSLESS COMPRESSION ALGORITHMS DR. OUIEM BCHIR
1 LECTURE VI: LOSSLESS COMPRESSION ALGORITHMS DR. OUIEM BCHIR 2 STORAGE SPACE Uncompressed graphics, audio, and video data require substantial storage capacity. Storing uncompressed video is not possible
More informationMultimedia Systems Entropy Coding Mahdi Amiri February 2011 Sharif University of Technology
Course Presentation Multimedia Systems Entropy Coding Mahdi Amiri February 2011 Sharif University of Technology Data Compression Motivation Data storage and transmission cost money Use fewest number of
More informationA Brief Introduction to Information Theory and Lossless Coding
A Brief Introduction to Information Theory and Lossless Coding 1 INTRODUCTION This document is intended as a guide to students studying 4C8 who have had no prior exposure to information theory. All of
More informationIntroduction to Source Coding
Comm. 52: Communication Theory Lecture 7 Introduction to Source Coding - Requirements of source codes - Huffman Code Length Fixed Length Variable Length Source Code Properties Uniquely Decodable allow
More informationENGR 4323/5323 Digital and Analog Communication
ENGR 4323/5323 Digital and Analog Communication Chapter 1 Introduction Engineering and Physics University of Central Oklahoma Dr. Mohamed Bingabr Course Materials Textbook: Modern Digital and Analog Communication,
More informationS Coding Methods (5 cr) P. Prerequisites. Literature (1) Contents
S-72.3410 Introduction 1 S-72.3410 Introduction 3 S-72.3410 Coding Methods (5 cr) P Lectures: Mondays 9 12, room E110, and Wednesdays 9 12, hall S4 (on January 30th this lecture will be held in E111!)
More informationTCET3202 Analog and digital Communications II
NEW YORK CITY COLLEGE OF TECHNOLOGY The City University of New York DEPARTMENT: SUBJECT CODE AND TITLE: COURSE DESCRIPTION: REQUIRED COURSE Electrical and Telecommunications Engineering Technology TCET3202
More informationBlock Markov Encoding & Decoding
1 Block Markov Encoding & Decoding Deqiang Chen I. INTRODUCTION Various Markov encoding and decoding techniques are often proposed for specific channels, e.g., the multi-access channel (MAC) with feedback,
More informationInformation & Communication
Information & Communication Bachelor Informatica 2014/15 January 2015 Some of these slides are copied from or heavily inspired by the University of Illinois at Chicago, ECE 534: Elements of Information
More informationLecture #2. EE 471C / EE 381K-17 Wireless Communication Lab. Professor Robert W. Heath Jr.
Lecture #2 EE 471C / EE 381K-17 Wireless Communication Lab Professor Robert W. Heath Jr. Preview of today s lecture u Introduction to digital communication u Components of a digital communication system
More informationSHANNON S source channel separation theorem states
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 55, NO. 9, SEPTEMBER 2009 3927 Source Channel Coding for Correlated Sources Over Multiuser Channels Deniz Gündüz, Member, IEEE, Elza Erkip, Senior Member,
More informationCourse Developer: Ranjan Bose, IIT Delhi
Course Title: Coding Theory Course Developer: Ranjan Bose, IIT Delhi Part I Information Theory and Source Coding 1. Source Coding 1.1. Introduction to Information Theory 1.2. Uncertainty and Information
More informationCommunication Theory II
Communication Theory II Lecture 14: Information Theory (cont d) Ahmed Elnakib, PhD Assistant Professor, Mansoura University, Egypt March 25 th, 2015 1 Previous Lecture: Source Code Generation: Lossless
More informationOutline. Communications Engineering 1
Outline Introduction Signal, random variable, random process and spectra Analog modulation Analog to digital conversion Digital transmission through baseband channels Signal space representation Optimal
More informationComm. 502: Communication Theory. Lecture 6. - Introduction to Source Coding
Comm. 50: Communication Theory Lecture 6 - Introduction to Source Coding Digital Communication Systems Source of Information User of Information Source Encoder Source Decoder Channel Encoder Channel Decoder
More informationUNIT-1. Basic signal processing operations in digital communication
UNIT-1 Lecture-1 Basic signal processing operations in digital communication The three basic elements of every communication systems are Transmitter, Receiver and Channel. The Overall purpose of this system
More informationDigital Communication Systems ECS 452
Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong prapun@siit.tu.ac.th 2. Source Coding 1 Office Hours: BKD, 6th floor of Sirindhralai building Monday 10:00-10:40 Tuesday 12:00-12:40
More informationLecture5: Lossless Compression Techniques
Fixed to fixed mapping: we encoded source symbols of fixed length into fixed length code sequences Fixed to variable mapping: we encoded source symbols of fixed length into variable length code sequences
More informationMulticasting over Multiple-Access Networks
ing oding apacity onclusions ing Department of Electrical Engineering and omputer Sciences University of alifornia, Berkeley May 9, 2006 EE 228A Outline ing oding apacity onclusions 1 2 3 4 oding 5 apacity
More informationCOMMUNICATION SYSTEMS
COMMUNICATION SYSTEMS 4TH EDITION Simon Hayhin McMaster University JOHN WILEY & SONS, INC. Ш.! [ BACKGROUND AND PREVIEW 1. The Communication Process 1 2. Primary Communication Resources 3 3. Sources of
More informationSIGNALS AND SYSTEMS LABORATORY 13: Digital Communication
SIGNALS AND SYSTEMS LABORATORY 13: Digital Communication INTRODUCTION Digital Communication refers to the transmission of binary, or digital, information over analog channels. In this laboratory you will
More information6.450: Principles of Digital Communication 1
6.450: Principles of Digital Communication 1 Digital Communication: Enormous and normally rapidly growing industry, roughly comparable in size to the computer industry. Objective: Study those aspects of
More informationTSKS01 Digital Communication Lecture 1
TSKS01 Digital Communication Lecture 1 Introduction, Repetition, Channels as Filters, Complex-baseband representation Emil Björnson Department of Electrical Engineering (ISY) Division of Communication
More informationRab Nawaz. Prof. Zhang Wenyi
Rab Nawaz PhD Scholar (BL16006002) School of Information Science and Technology University of Science and Technology of China, Hefei Email: rabnawaz@mail.ustc.edu.cn Submitted to Prof. Zhang Wenyi wenyizha@ustc.edu.cn
More informationELEC1200: A System View of. Lecture 1
ELEC1200: A System View of Communications: from Signals to Packets Lecture 1 Course Overview and Mechanics A basic communication system Bits and Bit Sequences The transmitter The channel The receiver ELEC1200
More informationPhysical Layer: Outline
18-345: Introduction to Telecommunication Networks Lectures 3: Physical Layer Peter Steenkiste Spring 2015 www.cs.cmu.edu/~prs/nets-ece Physical Layer: Outline Digital networking Modulation Characterization
More informationSYLLABUS of the course BASIC ELECTRONICS AND DIGITAL SIGNAL PROCESSING. Master in Computer Science, University of Bolzano-Bozen, a.y.
SYLLABUS of the course BASIC ELECTRONICS AND DIGITAL SIGNAL PROCESSING Master in Computer Science, University of Bolzano-Bozen, a.y. 2017-2018 Lecturer: LEONARDO RICCI (last updated on November 27, 2017)
More informationSignal and Information Processing
Signal and Information Processing Alejandro Ribeiro Dept. of Electrical and Systems Engineering University of Pennsylvania aribeiro@seas.upenn.edu http://www.seas.upenn.edu/users/~aribeiro/ January 11,
More informationEDI042 Error Control Coding (Kodningsteknik)
EDI042 Error Control Coding (Kodningsteknik) Chapter 1: Introduction Michael Lentmaier November 3, 2014 Michael Lentmaier, Fall 2014 EDI042 Error Control Coding: Chapter 1 1 / 26 Course overview I Lectures:
More informationChapter-1: Introduction
Chapter-1: Introduction The purpose of a Communication System is to transport an information bearing signal from a source to a user destination via a communication channel. MODEL OF A COMMUNICATION SYSTEM
More informationMAS.160 / MAS.510 / MAS.511 Signals, Systems and Information for Media Technology Fall 2007
MIT OpenCourseWare http://ocw.mit.edu MAS.160 / MAS.510 / MAS.511 Signals, Systems and Information for Media Technology Fall 2007 For information about citing these materials or our Terms of Use, visit:
More informationMAS160: Signals, Systems & Information for Media Technology. Problem Set 4. DUE: October 20, 2003
MAS160: Signals, Systems & Information for Media Technology Problem Set 4 DUE: October 20, 2003 Instructors: V. Michael Bove, Jr. and Rosalind Picard T.A. Jim McBride Problem 1: Simple Psychoacoustic Masking
More informationMultiuser Information Theory and Wireless Communications. Professor in Charge: Toby Berger Principal Lecturer: Jun Chen
Multiuser Information Theory and Wireless Communications Professor in Charge: Toby Berger Principal Lecturer: Jun Chen Where and When? 1 Good News No homework. No exam. 2 Credits:1-2 One credit: submit
More informationCOPYRIGHTED MATERIAL. Introduction. 1.1 Communication Systems
1 Introduction The reliable transmission of information over noisy channels is one of the basic requirements of digital information and communication systems. Here, transmission is understood both as transmission
More informationSHANNON showed that feedback does not increase the capacity
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 57, NO. 5, MAY 2011 2667 Feedback Capacity of the Gaussian Interference Channel to Within 2 Bits Changho Suh, Student Member, IEEE, and David N. C. Tse, Fellow,
More informationThe Lempel-Ziv (LZ) lossless compression algorithm was developed by Jacob Ziv (AT&T Bell Labs / Technion Israel) and Abraham Lempel (IBM) in 1978;
Georgia Institute of Technology - Georgia Tech Lorraine ECE 6605 Information Theory Lempel-Ziv Lossless Compresion General comments The Lempel-Ziv (LZ) lossless compression algorithm was developed by Jacob
More informationLecture 1: Tue Jan 8, Lecture introduction and motivation
Lecture 1: Tue Jan 8, 2019 Lecture introduction and motivation 1 ECE 6602: Digital Communications GEORGIA INSTITUTE OF TECHNOLOGY, SPRING 2019 PREREQUISITE: ECE 6601. Strong background in probability is
More informationDigital Speech Processing and Coding
ENEE408G Spring 2006 Lecture-2 Digital Speech Processing and Coding Spring 06 Instructor: Shihab Shamma Electrical & Computer Engineering University of Maryland, College Park http://www.ece.umd.edu/class/enee408g/
More informationChapter 1 Coding for Reliable Digital Transmission and Storage
Wireless Information Transmission System Lab. Chapter 1 Coding for Reliable Digital Transmission and Storage Institute of Communications Engineering National Sun Yat-sen University 1.1 Introduction A major
More informationMATHEMATICS IN COMMUNICATIONS: INTRODUCTION TO CODING. A Public Lecture to the Uganda Mathematics Society
Abstract MATHEMATICS IN COMMUNICATIONS: INTRODUCTION TO CODING A Public Lecture to the Uganda Mathematics Society F F Tusubira, PhD, MUIPE, MIEE, REng, CEng Mathematical theory and techniques play a vital
More informationInterference Mitigation Through Limited Transmitter Cooperation I-Hsiang Wang, Student Member, IEEE, and David N. C.
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 57, NO 5, MAY 2011 2941 Interference Mitigation Through Limited Transmitter Cooperation I-Hsiang Wang, Student Member, IEEE, David N C Tse, Fellow, IEEE Abstract
More informationDownloaded from 1
VII SEMESTER FINAL EXAMINATION-2004 Attempt ALL questions. Q. [1] How does Digital communication System differ from Analog systems? Draw functional block diagram of DCS and explain the significance of
More informationComm 502: Communication Theory
Comm 50: Communication Theory Prof. Dean of the faculty of IET The German University in Cairo 1 COMM 50: Communication Theory Instructor: Ahmed El-Mahdy Office : C3.319 Lecture Time: Sat. nd Slot Office
More informationECE 457 Communication Systems. Selin Aviyente Assistant Professor Electrical & Computer Engineering
ECE 457 Communication Systems Selin Aviyente Assistant Professor Electrical & Computer Engineering Announcements Class Web Page: http://www.egr.msu.edu/~aviyente/ece 457.htm M, W, F 10:20-11:10 a.m. Office
More informationDEGRADED broadcast channels were first studied by
4296 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 54, NO 9, SEPTEMBER 2008 Optimal Transmission Strategy Explicit Capacity Region for Broadcast Z Channels Bike Xie, Student Member, IEEE, Miguel Griot,
More informationDepartment of Electronics and Communication Engineering 1
UNIT I SAMPLING AND QUANTIZATION Pulse Modulation 1. Explain in detail the generation of PWM and PPM signals (16) (M/J 2011) 2. Explain in detail the concept of PWM and PAM (16) (N/D 2012) 3. What is the
More informationIntroduction to Coding Theory
Coding Theory Massoud Malek Introduction to Coding Theory Introduction. Coding theory originated with the advent of computers. Early computers were huge mechanical monsters whose reliability was low compared
More informationFundamentals of Digital Communication
Fundamentals of Digital Communication Network Infrastructures A.A. 2017/18 Digital communication system Analog Digital Input Signal Analog/ Digital Low Pass Filter Sampler Quantizer Source Encoder Channel
More informationEE303: Communication Systems
EE303: Communication Systems Professor A. Manikas Chair of Communications and Array Processing Imperial College London An Overview of Fundamentals: Channels, Criteria and Limits Prof. A. Manikas (Imperial
More informationModule 8: Video Coding Basics Lecture 40: Need for video coding, Elements of information theory, Lossless coding. The Lecture Contains:
The Lecture Contains: The Need for Video Coding Elements of a Video Coding System Elements of Information Theory Symbol Encoding Run-Length Encoding Entropy Encoding file:///d /...Ganesh%20Rana)/MY%20COURSE_Ganesh%20Rana/Prof.%20Sumana%20Gupta/FINAL%20DVSP/lecture%2040/40_1.htm[12/31/2015
More informationIEEE C /02R1. IEEE Mobile Broadband Wireless Access <http://grouper.ieee.org/groups/802/mbwa>
23--29 IEEE C82.2-3/2R Project Title Date Submitted IEEE 82.2 Mobile Broadband Wireless Access Soft Iterative Decoding for Mobile Wireless Communications 23--29
More informationCSCD 433 Network Programming Fall Lecture 5 Physical Layer Continued
CSCD 433 Network Programming Fall 2016 Lecture 5 Physical Layer Continued 1 Topics Definitions Analog Transmission of Digital Data Digital Transmission of Analog Data Multiplexing 2 Different Types of
More informationDEPARTMENT OF INFORMATION TECHNOLOGY QUESTION BANK. Subject Name: Information Coding Techniques UNIT I INFORMATION ENTROPY FUNDAMENTALS
DEPARTMENT OF INFORMATION TECHNOLOGY QUESTION BANK Subject Name: Year /Sem: II / IV UNIT I INFORMATION ENTROPY FUNDAMENTALS PART A (2 MARKS) 1. What is uncertainty? 2. What is prefix coding? 3. State the
More information6.004 Computation Structures Spring 2009
MIT OpenCourseWare http://ocw.mit.edu 6.004 Computation Structures Spring 2009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms. Welcome to 6.004! Course
More informationThe Z Channel. Nihar Jindal Department of Electrical Engineering Stanford University, Stanford, CA
The Z Channel Sriram Vishwanath Dept. of Elec. and Computer Engg. Univ. of Texas at Austin, Austin, TX E-mail : sriram@ece.utexas.edu Nihar Jindal Department of Electrical Engineering Stanford University,
More informationPhysical Layer. Networks: Physical Layer 1
Physical Layer Networks: Physical Layer 1 Physical Layer Part 1 Definitions Nyquist Theorem - noiseless Shannon s Result with noise Analog versus Digital Amplifier versus Repeater Networks: Physical Layer
More informationA Bit of network information theory
Š#/,% 0/,94%#(.)15% A Bit of network information theory Suhas Diggavi 1 Email: suhas.diggavi@epfl.ch URL: http://licos.epfl.ch Parts of talk are joint work with S. Avestimehr 2, S. Mohajer 1, C. Tian 3,
More information5984 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 12, DECEMBER 2010
5984 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 12, DECEMBER 2010 Interference Channels With Correlated Receiver Side Information Nan Liu, Member, IEEE, Deniz Gündüz, Member, IEEE, Andrea J.
More informationDigital Signal Processing Lecture 1
Remote Sensing Laboratory Dept. of Information Engineering and Computer Science University of Trento Via Sommarive, 14, I-38123 Povo, Trento, Italy Digital Signal Processing Lecture 1 Prof. Begüm Demir
More informationCompression. Encryption. Decryption. Decompression. Presentation of Information to client site
DOCUMENT Anup Basu Audio Image Video Data Graphics Objectives Compression Encryption Network Communications Decryption Decompression Client site Presentation of Information to client site Multimedia -
More informationComputer Networks Chapter 2: Physical layer
Computer Networks Chapter 2: Physical layer Holger Karl Computer Networks Group Universität Paderborn Goals of this chapter Answer the basic question: how can data be transported over a physical medium?
More informationUniversity of Jordan. Faculty of Engineering & Technology. Study Plan. Master Degree. Year plan
University of Jordan Faculty of Engineering & Technology Study Plan Master Degree In Electrical Engineering/Communication (Thesis Track) Year plan 2005 STUDY PLAN MASTER IN Electrical Engineering /Communication
More informationECE 3500: Fundamentals of Signals and Systems (Fall 2015) Lab 4: Binary Phase-Shift Keying Modulation and Demodulation
ECE 500: Fundamentals of Signals and Systems (Fall 2015) Lab 4: Binary Phase-Shift Keying Modulation and Demodulation Files necessary to complete this assignment: none Deliverables Due: Before Dec. 18th
More informationWireless Network Information Flow
Š#/,% 0/,94%#(.)15% Wireless Network Information Flow Suhas iggavi School of Computer and Communication Sciences, Laboratory for Information and Communication Systems (LICOS), EPFL Email: suhas.diggavi@epfl.ch
More informationIntroduction to Telecommunications and Computer Engineering Unit 3: Communications Systems & Signals
Introduction to Telecommunications and Computer Engineering Unit 3: Communications Systems & Signals Syedur Rahman Lecturer, CSE Department North South University syedur.rahman@wolfson.oxon.org Acknowledgements
More informationDigital modulation techniques
Outline Introduction Signal, random variable, random process and spectra Analog modulation Analog to digital conversion Digital transmission through baseband channels Signal space representation Optimal
More informationEE 8510: Multi-user Information Theory
EE 8510: Multi-user Information Theory Distributed Source Coding for Sensor Networks: A Coding Perspective Final Project Paper By Vikrham Gowreesunker Acknowledgment: Dr. Nihar Jindal Distributed Source
More informationSOME EXAMPLES FROM INFORMATION THEORY (AFTER C. SHANNON).
SOME EXAMPLES FROM INFORMATION THEORY (AFTER C. SHANNON). 1. Some easy problems. 1.1. Guessing a number. Someone chose a number x between 1 and N. You are allowed to ask questions: Is this number larger
More informationECE 630: Statistical Communication Theory
ECE 630: Statistical Communication Theory Dr. B.-P. Paris Dept. Electrical and Comp. Engineering George Mason University Last updated: January 23, 2018 2018, B.-P. Paris ECE 630: Statistical Communication
More informationMULTILEVEL CODING (MLC) with multistage decoding
350 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 52, NO. 3, MARCH 2004 Power- and Bandwidth-Efficient Communications Using LDPC Codes Piraporn Limpaphayom, Student Member, IEEE, and Kim A. Winick, Senior
More informationMulti-user Two-way Deterministic Modulo 2 Adder Channels When Adaptation Is Useless
Forty-Ninth Annual Allerton Conference Allerton House, UIUC, Illinois, USA September 28-30, 2011 Multi-user Two-way Deterministic Modulo 2 Adder Channels When Adaptation Is Useless Zhiyu Cheng, Natasha
More information3432 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 10, OCTOBER 2007
3432 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 53, NO 10, OCTOBER 2007 Resource Allocation for Wireless Fading Relay Channels: Max-Min Solution Yingbin Liang, Member, IEEE, Venugopal V Veeravalli, Fellow,
More informationDHANALAKSHMI SRINIVASAN COLLEGE OF ENGINEERING AND TECHNOLOGY CS6304- ANALOG AND DIGITAL COMMUNICATION BE-CSE/IT SEMESTER III REGULATION 2013 Faculty
DHANALAKSHMI SRINIVASAN COLLEGE OF ENGINEERING AND TECHNOLOGY CS6304- ANALOG AND DIGITAL COMMUNICATION BE-CSE/IT SEMESTER III REGULATION 2013 Faculty Name: S.Kalpana, AP/ECE QUESTION BANK UNIT I ANALOG
More informationOFDM Transmission Corrupted by Impulsive Noise
OFDM Transmission Corrupted by Impulsive Noise Jiirgen Haring, Han Vinck University of Essen Institute for Experimental Mathematics Ellernstr. 29 45326 Essen, Germany,. e-mail: haering@exp-math.uni-essen.de
More informationCSCD 433 Network Programming Fall Lecture 5 Physical Layer Continued
CSCD 433 Network Programming Fall 2016 Lecture 5 Physical Layer Continued 1 Topics Definitions Analog Transmission of Digital Data Digital Transmission of Analog Data Multiplexing 2 Different Types of
More informationLanguage of Instruction Course Level Short Cycle ( ) First Cycle (x) Second Cycle ( ) Third Cycle ( ) Term Local Credit ECTS Credit Fall 3 5
Course Details Course Name Telecommunications II Language of Instruction English Course Level Short Cycle ( ) First Cycle (x) Second Cycle ( ) Third Cycle ( ) Course Type Course Code Compulsory (x) Elective
More informationECEn 665: Antennas and Propagation for Wireless Communications 131. s(t) = A c [1 + αm(t)] cos (ω c t) (9.27)
ECEn 665: Antennas and Propagation for Wireless Communications 131 9. Modulation Modulation is a way to vary the amplitude and phase of a sinusoidal carrier waveform in order to transmit information. When
More informationOn Optimum Communication Cost for Joint Compression and Dispersive Information Routing
2010 IEEE Information Theory Workshop - ITW 2010 Dublin On Optimum Communication Cost for Joint Compression and Dispersive Information Routing Kumar Viswanatha, Emrah Akyol and Kenneth Rose Department
More informationPart A: Question & Answers UNIT I AMPLITUDE MODULATION
PANDIAN SARASWATHI YADAV ENGINEERING COLLEGE DEPARTMENT OF ELECTRONICS & COMMUNICATON ENGG. Branch: ECE EC6402 COMMUNICATION THEORY Semester: IV Part A: Question & Answers UNIT I AMPLITUDE MODULATION 1.
More informationDigital Communication Systems ECS 452
Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong prapun@siit.tu.ac.th Source Coding 1 Office Hours: BKD 3601-7 Monday 14:00-16:00 Wednesday 14:40-16:00 Noise & Interference Elements
More informationBell Labs celebrates 50 years of Information Theory
1 Bell Labs celebrates 50 years of Information Theory An Overview of Information Theory Humans are symbol-making creatures. We communicate by symbols -- growls and grunts, hand signals, and drawings painted
More informationFAST LEMPEL-ZIV (LZ 78) COMPLEXITY ESTIMATION USING CODEBOOK HASHING
FAST LEMPEL-ZIV (LZ 78) COMPLEXITY ESTIMATION USING CODEBOOK HASHING Harman Jot, Rupinder Kaur M.Tech, Department of Electronics and Communication, Punjabi University, Patiala, Punjab, India I. INTRODUCTION
More informationSCHEME OF COURSE WORK. Course Code : 13EC1114 L T P C : ELECTRONICS AND COMMUNICATION ENGINEERING
SCHEME OF COURSE WORK Course Details: Course Title : DIGITAL COMMUNICATIONS Course Code : 13EC1114 L T P C 4 0 0 3 Program Specialization Semester Prerequisites Courses to which it is a prerequisite :
More informationBroadcast Networks with Layered Decoding and Layered Secrecy: Theory and Applications
1 Broadcast Networks with Layered Decoding and Layered Secrecy: Theory and Applications Shaofeng Zou, Student Member, IEEE, Yingbin Liang, Member, IEEE, Lifeng Lai, Member, IEEE, H. Vincent Poor, Fellow,
More informationAnnouncement : Wireless Networks Lecture 3: Physical Layer. A Reminder about Prerequisites. Outline. Page 1
Announcement 18-759: Wireless Networks Lecture 3: Physical Layer Peter Steenkiste Departments of Computer Science and Electrical and Computer Engineering Spring Semester 2010 http://www.cs.cmu.edu/~prs/wirelesss10/
More information