Annex. 1.3 Measuring information

Size: px

Start display at page:

Download "Annex. 1.3 Measuring information"

Jasmin Johns
5 years ago
Views:

1 Annex This appendix discusses the interrelated concepts of information, information source, channel capacity, and bandwidth. The first three concepts relate to a digital channel, while bandwidth concerns an analog channel. e first discuss a mathematical model of a digital channel and its physical realization. e then study the analog channel. Lastly, we relate the digital and analog channels. 1.3 Measuring information A bit is a unit of measurement of information. If you ask me a binary question one whose answer is yes or no and I reply to you, my answer conveys one bit of information. ( Bit is an abbreviation of binary digit. ) Of course we may agree ahead of time to encode the answer. That is, we may agree that I will tell you the answer is 1 meaning yes or 0 meaning no. That is we can agree on a code expressed as a table: Code word Meaning 1 yes 0 no If you know the code, you can decode my answer. Of course, the code that we select will depend on the medium in which the answer is delivered. For example, if the answer is to be conveyed by the value of a voltage on a wire, the code might be: voltage high encodes yes and voltage low encodes no. If you ask me three binary questions, my answers convey three bits of information. You could have combined these three questions into one: Answer Q1 and Q2 and Q3 with one of eight possible alternatives: YYY if the answer to Q1 is Y, to Q2 is Y, to Q3 is Y; YYN if the answer to Q1 is Y, to Q2 is Y, to Q3 is N; and so on. So the answer to three binary questions conveys the same information as the answer to one eight-ary question (this is a question with eight possible answers). More generally if you ask me n binary questions, the same information can be conveyed by an answer to a 2 n -ary question one which has 2 n possible answers. So an answer to a 2 n -ary question conveys log 2 2 n = n bits of information. e can go the other way as well. That is, the information conveyed by an answer to a 2 n -ary question can also be conveyed by the answers to n binary questions. Let s see this for an 8-ary questions, whose possible answers are 0, 1,, 7. Consider the following code: 9

2 10 CHAPTER 1. COMMUNICATION SYSTEM Code word Meaning NNN 0 NNY 1 NYN 2 NYY 3 YNN 4 YNY 5 YYN 6 YYY 7 e can now encode the 8-ary question call it Q as three binary questions Q1, Q2, Q3. The binary question Q1 is: answer N if the answer to Q is 0,1,2, or 3 and Y if the answer to Q is 4,5,6, or 7; the binary question Q2 is: answer N if the answer to Q is 0,1,4, or 5 and answer Y otherwise; Q3 is: answer N if the answer to Q is 0,2,4, or 6, and answer Y otherwise. Evidently, if you know the answers to Q1, Q2, Q3 you know the answer to Q and vice versa. In general, then, an answer to a m-ary question conveys log 2 m bits of information. Example 1.3: Consider a gray scale picture with = 10 6 pixels or picture elements. Suppose each pixel has 64 possible gray scale values (that is, the intensity of black or white is divided into 64 levels). So a picture can have values (the picture is an answer to the question: hat are your 10 6 pixel values? ), and conveys log = bits or 8 Mb. This definition of bits of information has the following attractive additivity property. The answer to a m-ary question and the answer to a k-ary question conveys log 2 m +log 2 k bits. If you combine the two questions into one question, that combined question will have m k possible answers which conveys log 2 (mk) =log 2 m +log 2 k bits of information. So information in bits is preserved, no matter how you reformulate the question. The information you obtain from any set of questions depends on the number of possible answers: if there are m possible answers to the set of questions, the answers provide log 2 m bits of information. 1.4 Source Suppose you answer one binary question every second. Then you are conveying one bit of information every second. e say that you are conveying information at the rate of 1 bit per second or 1 bps. Suppose you answer one 4-ary question (a question with 4 possible answers) every second. Each answer conveys log 2 4=2bits of information, so you are conveying information at the rate of 2 bps.

3 1.5. DIGITAL CHANNELS 11 It is common in communications to call a device that answers questions on a periodic basis a source. In the preceding paragraph you are the source. A source has a certain information rate. Since that rate is measured in bits per second, we say that the source has a certain bit rate. Example 1.4: In digital telephony, voice is sampled 8,000 times per second, and each sample is quantized into one of 64 possible values. So each sample is an answer to a 64-ary question (namely, what is the value of this sample? ). Thus each sample conveys log 2 64 = 8 bits. The bit rate of the voice source is 8 bits/sample 8, 000 samples/sec = 64, 000 bps. If a source delivered 30 gray scale images of the kind described in example1.3, the rate of the source would be = 240 Mbps. Parallel composition of sources Just as the information conveyed by the answers to two questions is the sum of the information conveyed by each answer, the bit rate of two sources in parallel (considered as a single source) is the sum of the bit rates. Let s see how this works. Suppose we have source Source i that conveys information at rate r i bps, for i =1, 2. This means that every second Source i answers a question that has 2 r i possible answers. Consider a composite source Source which, every second, gives an answer to two questions, the first with 2 r 1 possible answers and the second with 2 r 2 possible answers. As we have seen, we could encode the two questions into one question with 2 r 1 2 r 2 possible answers. An answer to this composite question conveys log 2 (2 r 1 2 r 2 )=r 1 + r 2 bits. So the rate of the composite sources is r 1 + r 2 bps. Lastly, suppose we have a source that answers a 2 r -valued question once a second. So its rate is r bps. Now suppose this source becomes twice as fast, i.e. it answers two 2 r -valued questions each second. Then its rate is 2r bps. This is inuitively obvious, but we can also see this by observing that answering two 2 r -valued questions in one second conveys the same information as answering one 2 r 2 r -valued composite question, and such a source s information rate is log 2 (2 r 2 r )=2r bps. Thus the rate of a source is proportional to its speed. 1.5 Digital channels A digital channel is a device with an input port, an output port, and a set of symbols, Symbols, called the channel alphabet. The channel offers the following service: every second it accepts one symbol at its input port and delivers that symbol at its output port without error, and possibly after some delay. See figure 1.4. Suppose the alphabet Symbols contains N symbols. e say that the channel has capacity of log 2 N bits per second. Just as with a source, if the channel could send those symbols twice per second, its capacity would double. So the capacity of a channel can be expressed as Capacity = bits/symbol symbols/second. The first term on the right is log 2 N if there are N symbols in the alphabet. The unit of the second term is baud or baud rate (after a French engineer called Baudot). If the channel is binary, N =2, the first term is 1, and the baud rate of the channel is its bit rate.

4 12 CHAPTER 1. COMMUNICATION SYSTEM... x 1 x 2 x 3 x 4... Channel... x 1 x 2 x 3 x 4... input port output port Figure 1.4: A digital channel accepts a sequence of symbols, x 1 x 2, at its input port, one symbol each second, and delivers it at its output port. Example 1.5: A voiceband modem connected to a telephone line accepts a 2 7 -valued symbol every 1/8000 sec. So its capacity is Modem Capacity = bits/symbol symbols/second = = 56 Kbps. So the modem implements a digital channel with a rate of 8000 baud and bps. An optical fiber channel from San Francisco to New York (5,000 km long) accepts a binary signal (Symbols = {pulse, nopulse}) every 10 9 sec in San Francisco where its input port is located and delivers it to its output port in New York after a delay of 5000/( )=16.6 ms. The capacity of this channel is Optical Capacity = bits/symbol symbols/second = =1Gbps. So a mathematical formulation of a digital channel consists of a symbol set Symbols and a symbol time T. The channel can transfer one symbol every T seconds. So its baud rate is 1/T and its capacity is 1 T log 2 N bps where N is the number of elements in Symbols. Symbol sets The symbol sets of a physical channel consist of distinguished waveforms. A common binary symbol set in an optical channel consists of two waveforms, pulse-on and pulse-off. If the channel has a symbol rate of T 1, the duration of these waveforms is T sec. In the optical channel of example 1.5, the duration is 10 9 s or 1 nanosecond. The waveforms of light intensity (emitted by a laser) are illustrated in the left-most panel of figure 1.5. This form of symbol generation is called on-off keying or OOK. The two symbols in the middle panel are sinusoids of the same frequency, but the lower sinusoid is 180 degrees out of phase from the first. (In an electronic circuit, these sinusoids are voltages.) The symbols are generated by phase-shift keying or PSK. Since there are only two symbols one also says binary PSK or BPSK. The two symbols on the right are sinuoids of different frequency, and are said to be generated by frequency-shift keying or FSK. The rate of a channel can be increased either by increasing the number of different symbols (with the same symbol time) or by decreasing the symbol time T. The number of different symbols is limited by noise, which prevents the receiver (at the output port) from distinguishing between them. The symbol rate T is limited by the speed of electronics. For example, the system bus speed in your computer may be 100 MHz, i.e. the bus can transfer 1 symbol every 10 ns. Parallel composition of channels hen we compose to sources of rates r 1 and r 2 bps in parallel we get a composite source of rate r 1 + r 2 bps. Similarly, if we compose two channels with capacity

5 1.5. DIGITAL CHANNELS 13 pulse-on T T T = 1 ns pulse-off T T T OOK BPSK FSK Figure 1.5: Symbol sets for OOK, BPSK and FSK. C 1 and C 2 bps in parallel, we get a channel of capacity C 1 + C 2 bps. This is an important way of constructing a higher capacity channel from lower capacity channels. Figure1.6 indicates how this is done. two symbols from Symbols 1 everyt 2 sec... x 1 x 2 x 3 x 4... Decoder Channel 1 Coder... x 1 x 2 x 3 x 4... Channel 2 one symbol from Symbols everyt 2 sec one symbol from Symbols 2 everyt 2 sec Figure 1.6: The parallel composition of two channels gives a channel whose capacity is the sum. Suppose channel i has symbol set Symbols i with N i elements and a symbol time of T i seconds, i =1, 2. For simplicity assume that T 2 is a multiple of T 1, say T 2 =2T 1. So channel 1 can transmit one symbol from Symbols 1 every T 1 seconds, and channel 2 can transmit one symbol from Symbols 2 every T 2 =2T 1 seconds. So the capacity of the first channel is 1 T 1 log 2 N 1, the capacity of the second channel is 1 T 2 log 2 N 2. e use a decoder/coder to compose these two in parallel to form a third channel whose symbol set

6 14 CHAPTER 1. COMMUNICATION SYSTEM is Symbols = Symbols 1 Symbols 1 Symbols 2 and which can transmit one symbol from Symbols every T 2 seconds. The number of elements in Symbols is N1 2 N 2. So the capacity of this channel is 1 log T 2 (N1 2 N 2)= 2 log 2 T 2 N log 2 T 2 N 2 = 1 log 2 T 2 N log 1 T 2 N 2, 2 which is the sum of the capacities of the two channels. Every T 2 seconds, the decoder accepts a symbol of the form s =(s 1,s 1,s 2) Symbols 1 Symbols 1 Symbols 2 = Symbols. It then transmits the two symbols s 1,s 1 over the first channel and the symbol s 2 over the second channel. The three symbols are collected from the two output ports and combined together by the coder. Example 1.6: By composing 64 binary channels, each with a rate of 100 Mbps, you can get a bus with a capacity of 6.4 Gbps. Your computer has many such buses as shown in figure 1.7. Suppose we have a source with rate R bps and a channel of capacity C bps. e want to transmit the symbol stream emitted by the source through the channel and receive it from the channel output. Evidently, to do this we must have R C, (1.3) i.e. the channel capacity must exceed the (information) rate of the source. The symbol sets and the symbol periods of the source and the channel may be different. So again we ll need a coder/decoder pair to match the symbols and times as shown in figure 1.8. Suppose there are Nsource symbols in Symbols source with symbol time Tsource, and N channel symbols in Symbols channel with symbol time T channel. By hypothesis, R = 1 Tsource log 2 Nsource C = 1 T channel log 2 N channel. Suppose for simplicity that the symbol times are rationally related, mtsource = nt channel. Since R C, we must have Nsource m N channel n. Hence we can encode blocks of m symbols from the source emitted in time mtsource into blocks of n channel symbols. The coder in figure 1.8 does this, and the decoder takes those length-n blocks of channel symbols and recovers the length-m blocks of source symbols. 1.6 Analog channels An analog channel is an LTI system, characterized by its frequency response. e assume an ideal bandpass channel whose frequency response is constant, so it is characterized by its bandwidth

7 1.6. ANALOG CHANNELS 15 Figure 1.7: Your computer has many buses made of parallel channels. Source Coder Channel Decoder Symbols source Symbol time T source Rate R bps Symbols channel Symbol time T channel Rate C bps Figure 1.8: To compose a source with rate R bps with a channel with capacity C R may require a coder/decoder pair to match the symbols and symbol times.

8 16 CHAPTER 1. COMMUNICATION SYSTEM Hz and its center frequency f c Hz. e also say that the analog channel has capacity of Hz. For a baseband channel (center frequency 0), this means the frequency response { K, f, H baseband (f) =. 0, otherwise For a bandpass channel this means the frequency response H bandpass (f) ={ K, f ± fc 2 0, otherwise. Observe that bandwidth is the width of the positive frequencies. (If you count negative frequencies as well, the bandwidth is 2.) See top of figure 1.9. e ignore the constant K, so the channel may be identified with its spectrum, i.e. the set of frequencies that the channel transmits without change. The spectrum is shown as the shaded area. The bandwidth of an analog source x ContSignals is similarly defined. e consider its Fourier transform (FT) X : Reals Complex. If it is a baseband signal, its bandwidth is the least frequency such that X(f) =0, f >. If it is a bandpass source, with center frequency f c, its bandwidth is the least frequency such that { f fc > X(f) =0, if 2, f > 0 f + f c > 2, f < 0. e can connect an (analog) source x ContSignals to an (analog) channel with bandwidth channel. Suppose the source has FT X, X : Reals Complex, and bandwidth source. and the channel has frequency response H : Reals Complex. Then the output of the channel y : Reals Complex is the signal with FT Y, f Reals, X(f)H(f). In order that x can be recovered from y two conditions must be satisfied. First, we must have source channel. (1.4) It must also be the case that the frequencies in X must be in the spectrum of the channel, i.e. {f Reals X(f) > 0} {f Reals H(f) > 0}. (1.5) Condition (1.4) is the counterpart of the rate condition (1.3) for digital channels. Condition (1.5) is the counterpart of the requirement that to connect a discrete source to a digital channel the two symbol sets should match. If (1.4) holds but (1.5) does not, we can use a modulator to move the spectrum of X so that (1.5) holds. The modulator is the counterpart of the coder in fig 1.8. See figure 1.1.

9 1.7. TRANSMITTING AN ANALOG SOURCE OVER A DIGITAL CHANNEL 17 H baseband H bandpass K f f c f X X f f c f Figure 1.9: Bandwidth of a baseband channel (top, left) and bandpass channel (top, right), and bandwidth of a source (bottom). 1.7 Transmitting an analog source over a digital channel e have an analog real-valued source x ContSignals with FT X and bandwidth x Hz, and we wish to transmit it over a digital channel with capacity C bps. How do we do this? The idea is this. e first sample x and convert it into a discrete-time signal. Next we quantize the samples using an analog-to-digital converter or ADC. This gives a digital source which we connect to the digital channel. At the output port we reconstruct the original analog signal. Let s suppose x : Reals Reals is a baseband signal, so X(f) =0for f >. e sample this at a rate at least 2, i.e. the sampling period must satisfy the Nyquist condition T s 1 2. (1.6) The resulting signal is a discrete-time signal, x d : Integers Reals, where the nth sample is x d (n) =x(nt s ),n Integers. The Nyquist-Shannon theorem guarantees that because of (1.6) we can recover x from the sampled signal x d. More precisely we can use the following procedure to recover x. See figure1.10. e first construct a continuous-time signal x dc ContSignals, t Reals, x dc (t) = n= x(n)δ(t nt s ), and pass it through an low-pass filter with frequency response H lp, { 1, f, H lp (f) = 0, otherwise

10 18 CHAPTER 1. COMMUNICATION SYSTEM Call the resulting signal y. Then there is a constant K such that t Reals, y(t) =Kx(t). t n t t x T s x d x dc impulse sampler generator low pass filter H lp y Figure 1.10: If an analog signal is sampled at twice its bandwidth, the original signal can be recovered from the sampled signal. However, we cannot directly connect the discrete-time signal x d to the digital channel. The symbol set of x d is the set Reals which has infinitely many values, so its rate is infinite. e need to quantize these values. e choose an N-bit quantizer, i.e. we quantize the set Reals into 2 N values. That is, we choose a quantization function A 3-bit quantizer is shown in figure Quantizer : Reals {0, 1} N. hen we pass the discrete-time signal x d through the quantizer, we get a digital source x dq, n Integers, x dq (n) =Quantizer(x d (n)). The symbol set of this source is {0, 1} N and the symbol time is the sampling period T s. So the rate of this source is N T s bps. e require that this rate be smaller than the capacity C bps of the digital channel. If that is the case, the output of the channel is the input x dq (possibly with delay). e decode the symbols and obtain a discrete-time signal y d and pass it through a pulse generator and low-pass filter. The output of the low-pass filter is proportional to the original analog input x, 3-bit quantizer 111 x d x dq Digital channel from sampler x dq Impulse gen & LPF y ADC DAC Figure 1.11: A 3-bit quantizer and the associated digital-to-analog converter (DAC)

11 1.8. TRANSMITING A DIGITAL SOURCE OVER AN ANALOG CHANNEL 19 except that the quantization introduces an error. The finer the quantization the lower this error, but the larger is the rate of the digital source x dq. Thus we have the condition N 1 T s C, for transmitting an analog source with sampling period T s and N-bit quantizer over a digital channel with capacity C bps. If T s = 1 2, this becomes 2N C. 1.8 Transmiting a digital source over an analog channel e have a digital source with symbol set Symbols s and symbol time T that we wish to transmit over an analog channel with bandwidth c Hz. e study this in more detail in chapter 4.

Basic Concepts in Data Transmission

Basic Concepts in Data Transmission EE450: Introduction to Computer Networks Professor A. Zahid A.Zahid-EE450 1 Data and Signals Data is an entity that convey information Analog Continuous values within