Rab Nawaz. Prof. Zhang Wenyi

Transcription

1 Rab Nawaz PhD Scholar (BL ) School of Information Science and Technology University of Science and Technology of China, Hefei Submitted to Prof. Zhang Wenyi 1 P a g e

2 1. A Simple Derivation of the Coding Theorem and some Applications Shannon Source Coding Theorem Derivation of the Coding Theorem For MemoryLess Channel: Continuous channel: Multiaccess Channels Some basics about Multiaccess channels w.r.t Information Theory Multi-access information theory Multiaccess Coding Theorem Broadcast Channel Introduction Mathematical Model of Broadcast Channel Broadcast channel model More on Broadcast channel Concluding Remarks The Capacity of the Gaussian Interference Channel under Strong Interference The Capacity Region of the Discrete Memoryless Interference Channel with Strong Interference Capacity theorem for the Relay Channel A Proof of the Data Compression Theorem of Slepian and Wolf for Ergodic Sources Data Compression A Proof of the Data Compression Theorem Encoding Technique Basic Limits on Protocol Information in Data Communication Networks Introduction Some Assumption in terms of Data Communication Prospect Two Common cases of understanding Basic Model (Assumption) and Calculation Strategies for minimizing protocol information with a delay constraint Description through Flow Chart Acknowledgement References (Papers discussed) P a g e

3 1. A Simple Derivation of the Coding Theorem and some Applications [1] 1.1 Shannon Source Coding Theorem In information theory, Shannon's source coding theorem or noiseless coding theorem establishes the limits to possible data compression, and the operational meaning of the Shannon entropy. The source coding theorem shows that it is impossible to compress the data such that the code rate (average number of bits per symbol) is less than the *Shannon entropy of the source, without it being virtually certain that information will be lost. However it is possible to get the code rate arbitrarily close to the Shannon entropy, with negligible probability of loss. In information theory, systems are modeled by a transmitter, channel, and receiver. The transmitter produces messages that are sent through the channel. The channel modifies the message in some way. The receiver attempts to infer which message was sent. In this context, entropy (more specifically, Shannon entropy) is the expected value (average) of the information contained in each message. 'Messages' can be modeled by any flow of information. The logarithm of the probability distribution is useful as a measure of entropy because it is additive for independent sources. For instance, the entropy of a coin toss is 1 shannon, whereas of m tosses it is m shannons. Generally, you need log2 (n) bits to represent a variable that can take one of n values if n is a power of 2. If these values are equally probable, the entropy (in shannons) is equal to the number of bits. 1.2 Derivation of the Coding Theorem For discrete memoryless channels, the strongest known form of the theorem was stated by Fano [2] in In this result, the minimum probability of error P, for codes of block length N is bounded for any rate below capacity between the limits In this expression, E,(R) and E(R) are positive functions of the channel transition probabilities and of the rate R; O(N) is a function going to 0 with increasing N. For arrange of rates immediately beneath channel capacity, EL(R) = E(R). Let XN be the set of all sequences of length N that can be transmitted on a given channel, and let YN, be the set of all sequences of length N that can be received. We assume that both XN and YN, are finite sets. Let Pr (y I x), for y E YN and x e XN be the conditional probability of receiving sequence y, given that x was transmitted. We assume that we have a code consisting of ii/r code words; that is, a mapping of the integers from 1 to M into a set of code words x1,..., xm ; where xm E XN ; 1<=5<=M. We assume that maximum likelihood de- coding is performed at the receiver; that is, the decoder decodes the output sequence y into the integer m if (1) (2) 3 P a g e

4 Now let Pem be the probability of decoding error when Xm, is transmitted. A decoding error will be made if a y is received such that (2) is not satisfied. Thus we can express Pem as (3) Where we define the function (4) (5) We shall now upperbound Pem by upperbounding the function (6) The reason for using (6) is not at all obvious intuitively, but we can at least establish its validity by noting that the right-hand side of (6) is always non-negative, thereby satisfying the inequality when some term in the numerator is greater than or equal to the denominator, thus the numerator is greater than or equal to the denominator; raising the fraction to the p power keeps it greater than or equal to 1. Substituting (6) in (3), we have, (7) Equation (7) yields a bound to P,, for a particular set of code words. Aside from certain special cases, this bound is too complicated to be useful if the number of code words is large. We will simplify (7) by averaging over an appropriately chosen ensemble of codes. Let us suppose that we define a probability measure P(x) on the set X, of possible input sequences to the channel. (8) (9) Since the code words are chosen with the probability P(x), 4 P a g e

5 (10) the right-hand side of (10) is independent of m, we can substitute (10) in both the m and m term in (9). Since the summation in (9) is over M-1 terms, this yields (11) The above equation (11) bounds to any discrete channel. 1.3 For MemoryLess Channel: We shall now assume that the channel is memoryless so as to simplify the bound in (11). Let x1,, xn..., XN be the individual letters in an input sequence x, and let y1,.,yn,, YN be the letters in a sequence y. By a memoryless channel, we mean a channel that satisfies, (12) Now we restrict the class of ensembles of codes under consideration to those in which each letter of each code word is chosen independently of all other letters with a probability measure p(x); (13) Substituting (12) and (13) in (11), we get (14) The equation 14 can be written in a simplified form as (15) Note that the bracketed term in (15) is a product of sums and is equal to the bracketed term in (14) by the usual arithmetic rule for multiplying products of sums, 5 P a g e

6 (16) Note: (17) If we now upperbound M - 1 by M = e NR, where R is the code rate in nats per channel symbol, (17) can be rewritten as, (18) Since the right-hand side of (18) is independent of m, it is a bound on the ensemble probability of decoding error and is independent of the probabilities with which the code words are used. Application Specific Modeling Note: The binary symmetric channel in which two inputs and two outputs are modeled, it was first modeled by Elias. Moreover the author has extended his work to modeled very noisy channel in the sense that the probability of receiving a given output is almost independent of the input. Parallel channel was modeled through two discrete memoryless channel, first having K inputs and J outputs with transition probability Pik, the second having I inputs and L outputs with transition probabilities Qli as, 1.4 Continuous channel: A time-discrete amplitude-continuous channel is a channel whose input and output alphabets are the set of real numbers. It is usually necessary or convenient to impose a constraint on the code words of such channels to reflect the physical power limitations of the transmitter. In addition to this a detail analysis of the Gaussian noisy channel is also derived from the base line requirements and equation derived above, the detail is presented in [1]. 6 P a g e

7 2. Multiaccess Channels [2] 2.1 Some basics about Multiaccess channels w.r.t Information Theory In information theory, the multiple-access channel (MAC) is a model for communication scenarios where several transmitters wish to communicate to a common receiver. Sometimes these channels are also called multiaccess channels. Among multi terminal networks, multiple-access channels are those for which the strongest results are known, including an exact capacity result for the discrete memoryless setting with an arbitrary number of senders. A typical multicaccess communication system is demonstrated in the following figure 1, Figure 1: Multi-access Communication System There are multiple transmitters and a single receiver. The received signal is corrupted both by noise and by mutual interference between the transmitters. Each of the transmitters is fed by an information source, and each information source generates a sequence of messages, successive messages arriving at random instants of time. There is usually some small amount of feedback from the receiver to the transmitter, but this feedback will not be our main focus. Our major focus, rather, is on the interference, the noise, and the random, or bursty, message arrivals. This type of model is appropriate for the uplink of a satellite network, for a radio network where there is one central repeater, and for the traffic to the central node on a multidrop telephone line. The beginning of the collision resolution approach to multi-access communication came in 1970 with Abramson s ALOHA network. message (or packet) arrived at a transmitter, it would simply be transmitted, ignoring all other transmitters in the network. If another transmitter was transmitting in an overlapping interval, interference would prevent the message from being correctly received, the cyclic redundancy check (CRC) would not check, no acknowledgment would be sent, and the transmitter would try again later; the later time would be pseudo randomly chosen to avoid the certainty of another collision if both transmitters waited the same time. The multi-access information theoretic approach to mul-tiaccess began in 1973 with a coding theorem developed by Ahlswede and Liao. The noise and interference aspects of the multiaccess channels are appropriately modeled, but the random arrivals of the messages are ignored. 7 P a g e

8 2.2 Multi-access information theory The coding theorems of information theory treat the question of how much data can be reliably communicated from one point, or set of points, to another point. The class of channels to be considered is illustrated in Fig. 2. Each unit of time, the first transmitter sends a symbol x from an alphabet X and the second transmitter sends a symbol w from an alphabet W. There is an output alphabet Y and a transmitter probability assignment P(y I x,w) determining the probability of receiving each y E Y for each choice of inputs. Figure 1: Multi-access Channel with two transmitters The channel is memoryless in the sense that if x = (x1,, xn) and w = (w1,,wn) represent the inputs to transmitters one and two, respectively, over N successive time units, then the probability of receiving y = (y1,,yn)for the given x, w, is We assume for the time being that the alphabets are all discrete, but it will soon be obvious that this can be generalized in the same way as for single input channels. As indicated in Fig.1, there are two independent sources which are encoded independently into the two channel inputs. Consider block coding with a given block length N using M code words for transmitter 1, and L code words for transmitter 2; each code word is a sequence of N channel inputs. For convenience we refer to a code with these parameters as an (N, M, L) code. The rates of the two sources are defined as, (1) Each N units of time, source 1 generates an integer m uniformly distributed from 1 to M, and source 2 independently generates an integer I uniformly distributed from 1 to L. The transmitters send x, and w,, respectively, and the corresponding channel output y enters the decoder and is mapped into a decoded and. The decoding is correct and otherwise a decoding error occurs. (2) 8 P a g e

9 2.3 Multiaccess Coding Theorem Let Q1(x) and Q2(w) be probability assignments on the X and W alphabets, respectively, and consider an ensemble of (N, M, L) codes where each code word xm, 1<=m<=M, is independently selected according to the probability assignment X ~ Q1 (x), W ~ Q2 (y) R1 + R2 < = I (x, w, Y) R1 <= I(x; yiw) R2 <= I (w; yix) (3) (4) (5) (6) and each code word is independently selected according to (7) For each code in the ensemble, the decoder uses maximum likelihood decoding, and we want to upper bound the expected value (8) (9) 9 P a g e

10 Now consider Pe1 We first condition this probability on a particular message 1 entering the second encoder, and a choice of code with a particular w, transmitted at the second input. Given wl we can view the channel as a single input channel with input x, and with transition probabilities A maximum likelihood.decoder for that single input channel will make an error (or be ambiguous) if Since this event must occur whenever a type 1 error occurs, the probability of a type 1 error, conditional on wl being sent, is upper bounded by the probability of error or ambiguity on the above single input channel. (10) Taking the expected value of (2.11) over w, and then using the product form of Q1, Q2, and P again, (11) (12) (13) **** Remaining part is continued from next page**** 10 P a g e

11 11 P a g e

12 12 P a g e

13 13 P a g e

14 3. Broadcast Channel [3] 3.1 Introduction A communication channel or simply channel refers either to a physical transmission medium such as a wire, or to a logical connection over a multiplexed medium such as a radio channel in telecommunications and computer networking. A channel is used to convey an information signal, for example a digital bit stream, from one or several senders (or transmitters) to one or several receivers. A channel has a certain capacity for transmitting information, often measured by its bandwidth in Hz or its data rate in bits per second. In information theory, a channel refers to a theoretical channel model with certain error characteristics. In this more general view, a storage device is also a kind of channel, which can be sent to (written) and received from (reading). In broadcasting, a channel is a designated radio frequency (or, equivalently, wavelength), assigned by a competent frequency assignment authority for the operation of a particular radio station, television station or television channel. Broadcasting is the distribution of audio or video content to a dispersed audience via any electronic mass communications medium, but typically one using the electromagnetic spectrum (radio waves), in a one-to-many model. 3.2 Mathematical Model of Broadcast Channel We model a scenario in which a single source attempting to communicate information simultaneously to several receivers. Thus several different channels with a common input alphabet are specified. We shall determine the families of simultaneously achievable transmission rates for many extreme classes of channels. Upper and lower bounds on the capacity region will be found, and it will be shown that the family of theoretically achievable rates dominates the family of rates achievable by previously known time- sharing. The model is also applicable to the situation of compound channels, where the transmitter does not know the true channel characteristics but wishes to transmit at an interesting rate to the receiver. The broadcast channel is modeled on the next page, see on page P a g e

15 3.3 Broadcast channel model 15 P a g e

16 16 P a g e

17 17 P a g e

18 18 P a g e

19 19 P a g e

20 4. More on Broadcast channel [4] A broadcast channel has one sender and many receivers. The object is to broadcast information to the receivers. The information may be independent or nested. We shall treat broadcast channels with two receivers as shown in Fig. 1 Fig.1 Broadcast Channel A broadcast channel consists of an input alphabet X and two output alphabets Y1 and Y2 and a probability transition function p (y1, y2 I x). The broadcast channel is said to be memoryless if A code for a broadcast channel with independent information consists of an encoder And two decoders, 20 P a g e

21 It is often the case in practice that one received signal is a degraded, or corrupted, version of the other. One receiver may be farther away or downstream. Concluding Remarks One of the coding ideas used in achieving good rate regions is superposition, in which one layers, or superimposes, the information intended for each of the receivers. The receiver can then peel off the information in layers. To achieve superposition, one introduces auxiliary random variables that act as virtual signals. These virtual signals participate in the construction of the code, but are not actually sent. One useful idea used in the proof of capacity for the deterministic broadcast channel is random binning of the outputs Y1 and Y2. Another technique is Marton s introduction of correlated auxiliary random variables. Marton s region is the largest known achievable rate region for the general broadcast channel, but the capacity region remains unknown. 21 P a g e

22 5. The Capacity of the Gaussian Interference Channel under Strong Interference [5] The capacity region of an interference channel, when two separate messages are sent has been obtained only for very strong interference and for some rather trivial cases. We will analyze the problem for a wider class of channels with strong interference. A Gaussian interference channel takes the following form after suitable normalization: (1) where xi, yi and ni (i = 1,2) are sampled values of the input signal, the output signal, and superposed noise, respectively. interference does not reduce the capacity when the interference is very strong- that is, when ( alpha and beta satisfy the following conditions simultaneously: (2) The capacity region is a rectangle with side lengths C1 and C2 where C1 and C2 are the capacities of the component channels when there is no interference: The capacity region G is obtained when the following two inequalities are simultaneously satisfied: and that the region is an intersection GJ of the two capacity regions G1 G2, of the two multiple access channels. Gj can be expressed by the following inequalities when the conditions (4) hold: (3) (4) (5) 22 P a g e

23 Let the encoding functions f1(i) and f2( j) map the message i generated at the first information source and the message j generated at the second source into codewords x1i and x2j, both of length n; let the decoding functions g1(y1) and g2( y2) map the received words y1 and y2 into i and j if y, and y, belong to decoding sets D1i and D2j, respectively. An code for the Gaussian two-user channel (1) consists of M1 codewords x1i, M2 codewords x2j M1 decoding sets such that (6) where codewords should be chosen with the following power constraints: (7) (8) The typical shape of the capacity region is depicted in Fig. 1. The dotted curve represents the rate-pair obtained by frequency division multiplexing (FDM). The rate pair of FDM is expressed in the following way: (9) Strong interference can be expressed as, 23 P a g e

24 (10) The above equation can be written in simplified form as, (11) conditions(10) and (11) having strong and very strong interference. It should be noted that these inequalities compare the amount of information flowing in the desired directions with those flowing in the undesired directions. In other words, both conditions represent the situation where leakage occurs. 24 P a g e

25 6. The Capacity Region of the Discrete Memoryless Interference Channel with Strong Interference [6] The discrete memoryless interference channel with strong interference is a discrete memoryless interference channel with inputs X1 and X2, and corresponding outputs Y1 and Y2 which satisfy, And for all product probability distributions on The capacity region of this channel coincides with the capacity region C of the model where both messages are required at both receiving terminals. This region can be expressed as the union of the rate pairs (R,, R,) satisfying (a) (b) (c) (d) (e) where Q is a time-sharing parameter of cardinality 4, and the union is over all probability distributions of the form set by the channel. Let W1 and W2, be two independent information sources uniformly distributed over the integer sets (1,...,M1} and { 1,..., M2}, respectively. Encoder 1 maps W1, into codeword X1 and encoder 2 maps W2, into codeword X2. The interference channel consists of four finite alphabets and conditional probability distributions and for this channel is a set of two encoding functions, And two decoding functions Such that 25 P a g e

26 (f) (g) (h) Note: Achievability and converse (for hand written calculation, kindly move on next page) 26 P a g e

27 27 P a g e

28 28 P a g e

29 7. Capacity theorem for the Relay Channel [7] In information theory, a relay channel is a probability model of the communication between a sender and a receiver aided by one or more intermediate relay nodes. A discrete memoryless single relay channel can be modelled as four finite sets, X1, X2, Y1 and Y and a conditional probability distribution P(Y1, Y1 I X1, X2) on these sets. The probability distribution of the choice of symbols selected by the encoder and the relay encoder is represented by P(X1, X2). 29 P a g e

30 30 P a g e

31 31 P a g e

32 8. A Proof of the Data Compression Theorem of Slepian and Wolf for Ergodic Sources [8] 8.1 Data Compression In signal processing, data compression, source coding, or bit-rate reduction involves encoding information using fewer bits than the original representation. Compression can be either lossy or lossless. Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression. Lossy compression reduces bits by removing unnecessary or less important information. The process of reducing the size of a data file is referred to as data compression. In the context of data transmission, it is called source coding (encoding done at the source of the data before it is stored or transmitted) in opposition to channel coding. Compression is useful because it reduces resources required to store and transmit data. Computational resources are consumed in the compression process and, usually, in the reversal of the process (decompression). Figure 1, shows the data compression process, Fig.1 Data Compression Process and its elements 32 P a g e

33 8.2 A Proof of the Data Compression Theorem 33 P a g e

34 34 P a g e

35 8.3 Encoding Technique Let A be the set of typical (x,y) n-sequences. We shall prove that (Rx, Ry) = (H(X 1 Y),H(Y)) is achievable. The achievability of the pair (H(X),H(Y 1 X)) follows an identical argument. The remainder of the boundary follows by timesharing these two schemes. 35 P a g e

36 36 P a g e

37 37 P a g e

38 9. Basic Limits on Protocol Information in Data Communication Networks [9] 9.1 Introduction The limitations on the amount of protocol information that must be transmitted in a data communication network to keep track of source and receiver addresses and of the starting and stopping of messages. Assuming Poisson message arrivals between each communicating source-receiver pair, we find a lower bound on the required protocol information per message. This lower bound is the sum of two terms, one for the message length information, which depends only on the distribution of message lengths, and the other for the message start information, which depends only on the product of the source-receiver pair arrival rate and the expected delay for transmitting the message. 9.2 Some Assumption in terms of Data Communication Prospect DATA communication network, for the purposes of this paper, is a finite collection of nodes; a finite collection of noiseless two-way communication links, each connecting some pair of nodes; and a finite collection of sources and receivers, each source and each receiver being connected to a node. We view each source as being paired with a receiver (typically at a different node), and the purpose of the network is to transmit messages from each source to its paired receiver. Messages are assumed to arrive from a source at randomly chosen instants of time and consist of binary strings of random length. Any communication network must not only transmit the messages (data) from source to receiver but must also transmit a certain amount of control information indicating, for example, the beginning, the end, and the destination of each message. It is customary in data networks to refer to such control information as protocol information and to refer to the conventions for representing it as protocols. In simple words a protocol is a source code for representing control information. Our major objective in this paper is to define and calculate the amount of protocol information (in the information-theoretic sense) required in the type of network described above. We shall find, rather surprisingly, that this information is related solely to the starting and stopping of messages and has nothing to do with addressing. 9.3 Two Common cases of understanding The following examples provide some idea of the wide variety of ways in which protocol information can be represented in different types of networks. In a message switching network, each message is generally preceded by a binary encoding of the source and receiver address and of the message length. Messages, with their preceding protocol bits, are queued at the individual nodes and passed from one node to another by a variety of routing algorithms which will be of no concern here. In a packet switching network, the messages are first divided into packets of some fixed length. Each packet is preceded by an encoding of the source and receiver address and the position of the packet within the message. The 38 P a g e

39 final packet, of course, might have fewer message bits than the others and this information must also be encoded. The packets are transmitted through the network independently and are reconstructed into a message at the receiver s node. The amount of protocol information is somewhat greater than for message switching, but the delay in transmission is frequently reduced. Another possible system is to use multiplexers on each link of the network and to assign to each source-receiver pair a fixed fraction of the capacity of the links on some path from source to receiver. Frequently in such systems, there is a special string of bits called an idle character, which is used repeatedly when there is no message to be sent, and a special string of bits called a flag, which precedes and follows each message. The messages themselves are slightly encoded to prevent the occurrence of the flag character within the encoded message. The idle characters and flags should be regarded as protocol codewords which together indicate the beginning and the end of each message. Such a system is usually either very inefficient in its use of link capacities or subject to very long queues since it has no flexibility to allocate the network resources to messages as needed. If all the messages from source are of equal length and if they arrive equally spaced in time, then the multiplexer becomes very efficient and, in fact, it is possible to eliminate all control characters. In comparing the multiplexer approach with the message switching or packet switching approach, we see that the former transmits message start information but no addresses, while the latter transmits addresses but no message start information. Simple network communication model is described in fig. 1. Figure 1. Simple Network Communication Model 39 P a g e

40 9.4 Basic Model (Assumption) and Calculation 40 P a g e

41 41 P a g e

42 42 P a g e

43 9.5 Strategies for minimizing protocol information with a delay constraint Description through Flow Chart 43 P a g e

44 Acknowledgement I am really thanks to Prof. Zhang Wenyi for his kind support during this course. He is very kind, humble, supportive and hardworking professor. He is very sound in theoretical aspects of network science. He taught us the course by selecting some good papers from IEEE transaction on information theory journal and discussed all the nitty gritty in those papers in a very detail manner in the class. I have learnt a lot in this course that I never learnt before. References (Papers discussed) [1]. Gallager, R. "A simple derivation of the coding theorem and some applications." IEEE Transactions on Information Theory 11.1 (1965): [2]. Gallager, Robert. "A perspective on multiaccess channels." IEEE Transactions on information Theory 31.2 (1985): [3]. Cover, Thomas. "Broadcast channels." IEEE Transactions on Information Theory 18.1 (1972): [4]. Cover, Thomas M. "Comments on broadcast channels." IEEE Transactions on information theory 44.6 (1998): [5]. Sato, Hiroshi. "The capacity of the Gaussian interference channel under strong interference (corresp.)." IEEE transactions on information theory 27.6 (1981): [6]. Costa, Max HM, and Abbas El Gamal. "The capacity region of the discrete memoryless interference channel with strong interference." IEEE Transactions on Information Theory 33.5 (1987): [7]. Cover, Thomas, and A. EL Gamal. "Capacity theorems for the relay channel." IEEE Transactions on information theory 25.5 (1979): [8]. Cover, T. "A proof of the data compression theorem of Slepian and Wolf for ergodic sources (Corresp.)." IEEE Transactions on Information Theory 21.2 (1975): [9]. Gallager, Robert. "Basic limits on protocol information in data communication networks." IEEE Transactions on Information Theory 22.4 (1976): P a g e