Information Theory and Coding

Size: px

Start display at page:

Download "Information Theory and Coding"

Meryl Ellis
6 years ago
Views:

1 Iformato heory ad Codg Itroducto What s t all aout? Refereces: C..hao, A Mathematcal heory of Commucato, he Bell ystem echcal Joural, Vol. 7, pp , , July, Octoer, 948. C..hao Commucato the presece of ose, roc. Ist of Rado geers, Vol 37 o, pp, Jauary 949. James L Massey, Iformato heory: he Coperca ystem of Commucatos, I Commucatos Magaze, Vol. o, pp 6 8, Decemer 984 Mscha chwartz, Iformato rasmsso, Modulato ad ose, McGraw Hll 99, Chapter he purpose of telecommucatos s the effcet ad relale trasmsso of real data text, speech, mage, taste, smell etc over a real trasmsso chael cale, fre, wreless, satellte. I achevg ths there are two ma ssues. he frst s the source the formato. How do we code the real lfe usually aalogue formato, to dgtal symols a way to covey ad reproduce precsely wthout loss the trasmtted text, pcture etc.? hs etals proper aalyss of the source to work out the est relale ut mmum amout of formato symols to reduce the source data rate. he secod s the chael, ad how to pass through t usg approprate formato symols the source. I dog ths there s the prolem of ose all the multfarous ways that t appears radom, shot, urst- ad the prolem of oe symol terferg wth the prevous or ext symol especally as the symol rate creases. At the recever the symols have to e guessed, ased o classcal statstcal commucatos theory. he guessg, ufortuately, s heavly flueced y the chael ose, especally at hgh rates. o get aroud ths prolem, techques are used to mprove the relalty of the guess at the recever. hs etals addtoal symols worked to the formato symols producg a code of symols. he recever ca therefore guess at the level of the symol, ut has also addtoally the posslty of guessg, somewhat more relaly, at the level of the code. Codg s therefore the art ad scece of mapulatg the travelg data symols a way to decrease as far as possle the posslty of passg erroeous formato to the recever (sk). wo mportat methods are used. he frst gves the alty to the recever to detect a error. Automatc repeat request (ARQ) methods are the used to tell the source to retrasmt that porto whch the recever has detected a error. he secod method volves addg eough redudat symols to the formato symols, to allow ot oly

2 the capalty of detectg, ut also that of correctg wrogly receved symols. hs s kow as Forward rror Correcto, ad uses varous types of codes. I the aove the source codg has ee separated out from the chael codg. However there s also the posslty of comg source ad chael codg to reduce symol rate ut keepg the same relalty. ermology: Of course the aove has to traslate to scetfc method ad egeerg jargo. hs wll clude, formato, etropy, source codg Huffma Codg, Arthmetc Codg, Dctoary Codg Methods (LZW ad others), hao s heorems, chael capacty, t rates, symol rates, proalty of t error, lear algera groups, rgs, vector spaces, prmtve roots -, lock codes, covolutoal codes, proalty of lock error etc. etc. Iformato he amout of formato a source s related to ts proalty of occurrece. A message whch a pror has a hgh proalty of occurrg coveys very lttle formato whe t s receved. Based o ths, the measure of formato of a evet X occurrg wth proalty X from amog a set of possle evets, s defed as I X log (.) X If the proalty of occurrece s, the o formato s coveyed, sce the recever already kows what the trasmtter s sedg. he mmum umer of evets ecessary, to requre the recever to make a decso, s two evets. Gve two evets, assumed equproale, the easest symol represetato s the ary pulse, whch s called the t. At the level of source codg the possle evets occurrg from a gve data source are coverted to ts. For evets there s a eed of log ts of formato (eg CM speech). For equproalty I log log ts (.) I practce the evets are read, coded ad trasmtted sequetally. If each evet takes tme ô, ad a terval s cosdered, the formato trasmtted over secods s Iformato log ts (.3) tropy

3 he formato calculated aove s ased o equproalty. I practce, evets that eed to e trasmtted are ot lke ths. x: ext has some characters more proale tha others; voce has more cotet at low frequeces. he formato the message ow depeds o two factors: he relatve frequecy of oe evet defed as umer of tmes evet occurs otal umer of evets he average of the formato due to each evet, also kow as the etropy, H of the source, ad gve y H log (.4) I practce, the value of formato for H (.4) s less or equal to the value of I (.) whch s tself the average for the equproale evet. (see hao s paper, for the graph of H vs p, the proalty of oe evet, gve the other proalty s (-p)). Addg (.4) the ssue of the tme, take for trasmsso, the value for H ecomes H log ts sec ods (.5) he etropy of a source therefore depeds o the relatve frequecy of the evets t s made up of. hao, hs paper defed the rato of the etropy of a source to the maxmum value t could have, whle stll restrcted to the same symols (evets), as ts relatve etropy. For trasmsso the evets eed to e coded, usually to ts, ad ths s the maxmum compresso possle whe we ecode to the same alphaet. Oe mus the relatve etropy s the redudacy. he redudacy of ordary glsh, ot cosderg statstcal structure over greater dstaces tha aout eght letters, s roughly 5%. Huffma, Arthmetc ad other types of data compresso methods, (codes) wll e cosdered to address ths compresso avalale a source, to reduce the t rate. hao s Codg heorems hao s frst theorem s the source codg theorem. It states that every commucato source s characterzed y a sgle umer H, the source rate, ad ca e represeted as accurately as desred y R ary dgts per secod f oly R > H. (ote that H s eg used stead of H as ths s used for oth meags e ts ad ts per secod) hus, the theorem estalshes that the etropy rate s the rate of a optmal lossless data compresso code. he lmt exsts as log as the source s statoary. How to acheve the lmt s the realm of data compresso methods.

4 A commucato system capale of trasmttg ths formato should have the capacty to hadle the average formato ts geerated. C AV log ts per sec od (.6) O the chael tself, the ts ca e trasformed to other symols. hs gves rse to the aud (symol) rate over the chael, ad s characterzed y C log m ts / s (.7) where ô s the tme to trasmt a symol over the chael m s the umer of symols avalale o the chael (eg for 8 QK, there are 8 symols, ad each symol requres 3 ts.) o (.7) must e capale of hadlg (.6) wth a artrary small frequecy of errors..7 ca also e wrtte as C B log m ts / s (.8) where B s the adwdth of the chael hs s hao s frst theorem, whch assumes a oseless, lossless, chael. he ssue of codtoal depedece, as opposed to statst6cal depedece, of the set of evets wll e cosdered la work. he ssue of lossy data compresso ad rate-dstorto theory wll e cosdered later. he secod theorem s the chael codg theorem. I practce commucato chaels are ot oseless. hey all exht ose. he qualty of sgals over practcal chaels s expressed terms of the sgal-to-ose rato expressed decels ad gve y log db.9 he hao-hartley Law states that for a chael wth adwdth B Hz ad a sgal-toose rato of / (ot db) the maxmum umer of ts per secod s gve y B log (+ ) ps. C C s the theoretcal tghtest upper oud o the formato rate. If we have a chael wth capacty C ad a source whose rate of geeratg formato R s less tha or equal to C, t s possle to ecode the source such a way as to trasmt over ths chael wth the fdelty measured y. Cosderg aga the 8_QK system. he ose

5 adds a sphere aroud each pot. he umer of dfferet messages whch must e capale of dstct trasmsso s of the order of the volume of the rego of possle messages dvded y the volume of the small spheres. hao s theorem states that R ca stll e of the order of C (.). What s mportat s ot the / (so log as R<C), ut how the formato s coded to acheve the theoretcal oud. hs mples that formato should ot e trasmtted oe t at a tme, ut as a lock of ts coded to a chael put sequece. Chael ose Chael ose ca e radom or thermal ose or mpulse ose due to lghtg, cross-talk etc. Radom ose gves rse to radom-t errors. Impulse ose gves rse to urst errors (e a umer of adjacet ts error). I practce the mmum sgal level requred for acceptale t error rates s mportat. he eergy measured joules per t a sgal, deoted y, s gve y, where s the sgal power watts ad the t tme. he level of thermal ose a adwdth of Hz s defed as k watts/hz, where k s Boltzma's costat ad the temperature K. Hece /R k I terms of the chael adwdth, the ose eergy W, so that W R whch fally yelds db db log ( ) + log W - log R he sgal power requred to acheve a requred / creases wth temperature ad t rate R.

Module 6. Channel Coding. Version 2 ECE IIT, Kharagpur

Module 6. Channel Coding. Version 2 ECE IIT, Kharagpur Module 6 Chael Codg Lesso 36 Coded Modulato Schemes After readg ths lesso, you wll lear about Trells Code Modulato; Set parttog TCM; Decodg TCM; The modulated waveform a covetoal ucoded carrer modulato