ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 1: Introduction & Overview Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 1 / 26
Outline 1 Course Information 2 Course Overview 3 Topics Covered 4 Research Project Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 2 / 26
Outline 1 Course Information 2 Course Overview 3 Topics Covered 4 Research Project Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 2 / 26
Outline 1 Course Information 2 Course Overview 3 Topics Covered 4 Research Project Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 2 / 26
Outline 1 Course Information 2 Course Overview 3 Topics Covered 4 Research Project Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 2 / 26
Outline Course Information 1 Course Information 2 Course Overview 3 Topics Covered 4 Research Project Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 3 / 26
Administrivia Course Information Hours and Location Lectures TuTh 9:30-10:45 ASEC 511 (North Tower) Course Webpage: http://www.ecgf.uakron.edu/~tran/courses/13f-693/ Instructor Information Name: Dr. Nghi Tran Office: ASEC 352 Phone: x7168 email: nghi.tran@uakron.edu Webpage: http://www.ecgf.uakron.edu/~tran Office Hours TuF 10:45-11:45 and (or) by appointment. Teaching Assistant: N/A Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 4 / 26
Course Information Logistics References: Though not required, the following references are highly recommended T. Cover and J. Thomas, Elements of Information Theory, Second Edition, John Wiley & Sons, Inc., 2006. D. Tse and P. Viswanath, Fundamentals of Wireless Communication, Cambridge University Press, 2005 Homework: Bi-weekly, with in total of 6 or 7 Research Project: In-depth study, or original research topic Assessment: 30% Homework, 25% Project, and 45% Final Exam Final Exam: Open-book Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 5 / 26
Outline Course Overview 1 Course Information 2 Course Overview 3 Topics Covered 4 Research Project Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 6 / 26
Course Overview What is Information Theory? Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 7 / 26
Course Overview What is Communication? The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. (Claude Shannon 1948) Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 8 / 26
Course Overview Communication System Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 9 / 26
Course Overview Encoder: Source Coding Source Coding is the process of compressing the data using fewer bits to remove redundancy. Shannon s source coding theorem establishes the limits to possible data compression: Entropy. Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 10 / 26
Course Overview Encoder: Channel Coding Channel coding adds controlled redundancy to protect against channel errors. The Shannon limit or Shannon capacity of a communications channel is the theoretical maximum information transfer rate of the channel, for a particular noise level. Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 11 / 26
Course Overview Decoder Channel decoding: Decode signals, detect/correct errors. Source decoding: Decompress and restore source. Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 12 / 26
Channel Definition Course Overview X p( X/ Y) Y A communication channel is a system in which the output Y depends probabilistically on its input X: Probabilistic relationship between input and output The channel is characterized by a probability transition p(y/x) Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 13 / 26
Course Overview Channel Examples Noiseless binary channel: Input is reproduced exactly at output 0 0 1 1 Binary symmetric channel: Cross probability p 1 p 0 0 p p 1 1 1 p Gaussian channel with Gaussian noise N: Y = X + N Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 14 / 26
Course Overview Information Theory Claude E. Shannon, A Mathematical Theory of Communication. Bell System Technical Journal, 27, 379 423 & 623 656, 1948. A new filed of Information Theory was developed in 1948, motivated by Fundamental Limits in Communication Theory What are these limits? Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 15 / 26
Course Overview Information Theory Motivated by Communication Theory, Information Theory addresses two fundamental questions: Source Coding: Is there a limit to how much data can be compressed? The answer is Entropy Channel Coding: Over a noisy channel, what is the minimum redundancy rate required to achieve a reliable communication, e.g.,the bit error probability can be made as low as desired? The answer is Channel Capacity Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 16 / 26
Course Overview Information Theory Motivated by Communication Theory, Information Theory addresses two fundamental questions: Source Coding: Is there a limit to how much data can be compressed? The answer is Entropy Channel Coding: Over a noisy channel, what is the minimum redundancy rate required to achieve a reliable communication, e.g.,the bit error probability can be made as low as desired? The answer is Channel Capacity Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 16 / 26
Course Overview Two Main Topics in Information Theory Main Topic 1 - Source Coding: How can we reduce the bit rate at the output of source encoding? What is the minimum bit rate required to recover the data at the receiver for two cases: 1) Distortion is NOT allowed; 2) distortion is allowed Main Topic 2: Channel Coding and Capacity How can we reduce the bit error probability by intelligently adding the redundancy for channel coding? What is the Capacity? Joint source and channel coding indeed needed? For single TX-RC channel: source and channel coding should be done separately Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 17 / 26
Course Overview Two Main Topics in Information Theory Main Topic 1 - Source Coding: How can we reduce the bit rate at the output of source encoding? What is the minimum bit rate required to recover the data at the receiver for two cases: 1) Distortion is NOT allowed; 2) distortion is allowed Main Topic 2: Channel Coding and Capacity How can we reduce the bit error probability by intelligently adding the redundancy for channel coding? What is the Capacity? Joint source and channel coding indeed needed? For single TX-RC channel: source and channel coding should be done separately Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 17 / 26
Course Overview What Else in Information Theory Information Theory also finds applications in other disciplines Electrical Engineering - Communication Theory Mathematics Computer Science Physics Economics Others Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 18 / 26
Course Overview Why We Need Information Theory? Understand communications systems well enough to engineer Make a fundamental comparison among different schemes System benchmark to design Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 19 / 26
Outline Topics Covered 1 Course Information 2 Course Overview 3 Topics Covered 4 Research Project Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 20 / 26
Topic Covered Topics Covered Some Mathematical Tools and Probability Theory Information Theory Basis Entropy, mutual information, AEP,... Data Compression Channel Capacity and Random Coding Bound Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 21 / 26
Topics Covered Topic Covered (Continued) Differential Entropy Single-User Gaussian Channel AWGN Fading with CSI, CDI MIMO Practical Coding Schemes Block Codes, Convolutional Codes, Turbo Codes Network Information Theory MAC and Broadcast Channels Interference and Relay Channels (if time permits) Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 22 / 26
Outline Research Project 1 Course Information 2 Course Overview 3 Topics Covered 4 Research Project Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 23 / 26
Research Project Research Project Main goal: get exposed to some interesting topics in Information Theory and its related applications It is your choice to select the topic. You can also pick up from the list of suggested topics The project can either be your original research, or a high quality survey of a relevant topic, or a review of some papers Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 24 / 26
Research Project Research Project - Time Line 1-page proposal: Due Nov. 14 Project report: Due Dec. 9 Project presentation in the end of the term Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 25 / 26
Research Project Thank you! Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Information Theory 26 / 26