第4章-信道容量教学课件

Review–propertyofmutualinfo.function:Property.1Relationshipbetweenaveragemutualinfo.andchannelinputprobabilitydistributionProperty1:I(X;Y)isanupperconvexfunctionofthechannelinputprobabilitydistributionp(x).I(X;Y)p(x)Review–propertyofmutualinfo.function:Property.2

RelationshipbetweenInfo.contentandchanneltransitionprobabilitydistribution

Property2:I(X;Y)isaconcavefunctionofchanneltransitionprobabilitydistributesp(y/X).I(X;Y)p(y/x)Whatischannel?Thechannelisacarriertransmittingmessages—apassagethroughwhichsignalpasses.Theinformationisabstract,butthechannelisconcrete.Forinstance:Iftwopeopleconverse,theairisthechannel;Ifthetwocalleachother,thetelephonelineisthechannel;Ifwewatchthetelevision,listentotheradio,thespacebetweenthereceiverandthetransmitteristhechannel.4.1.ThemodelandclassificationofthechannelInthispart,wewillmainlyintroducetwoparts:ChannelmodelsChannelclassifications

4.1.1ChannelModelswecantreatchannelasaconverterwhichtransferevents.thechannelmodelcanbeindicatedasthefollowFig:BinarySymmetricChannel(BSC)isthesimplestchannelmodelABSCisshownasbelow:BSCDMCWeassumethatthechannelandthemodulationismemoryless.TheinputsandoutputscanthenberelatedbyasetofconditionalprobabilitiesDMCThischannelisknownasaDiscreteMemorylessChannel(DMC)andisdepictedas4.1.2ChannelclassificationsChannelcanbeclassifiedintoseveraltypes.(对流层)(电离层)4.2Channeldoubtdegreeandaveragemutualinformation4.2.1Channeldoubtdegree4.2.2Averagemutualinformation4.2.3Propertiesofmutualinformationfunction4.2.4Relationshipbetweenentropy,channeldoubtdegreeandaveragemutualinformation4.2.1ChanneldoubtdegreeAssumer.v.Xindicatestheinputsetofchannel,andr.v.Yindicatestheoutputsetofchannel,thechanneldoubtdegreeis:Themeaningof“channeldoubtdegreeH(X|Y)”isthatwhenthereceivingterminalgetsmessageY,theaverageuncertaintystillleavesaboutsourceX.Infact,theuncertaintycomesfromthenoiseinchannel.ThismeansthatiftheaverageuncertaintyofsourceXisH(X),we’llgetmoreorlessinformationwhicheliminatestheuncertaintyofthesourceXwhengettheoutputmessageY.Sowehavethefollowingconceptofaveragemutualinformation.Sincewehave:4.2.2AveragemutualinformationTheaveragemutualinformationistheentropyofsourceXminusthechanneldoubtdegree.TheabovemeaningisthatwhenthereceivergetsamessageY,theaverageinformationhecangetaboutXfromeverysymbolhereceived.4.2.3PropertiesofmutualinformationfunctionProperty1:Relationshipbetweenmutualinformationandchannelinputprobabilitydistribution.I(X;Y)isanupperconvexfunctionofthechannelinputprobabilitydistributionP(X).ThiscanbeshowninFig.4.5andFig.4.6.Fig.4.5.I(X;Y)isconvexfunctionofP(X)Fig.4.6.MessagepassingthroughthechannelE.g.4.1Consideringadualelementchannel,theprobabilitydistributionisandthematrixofchannelisWhereistheprobabilityoftransmissionerror.Thenthemutualinformationis,Andwecangetthefollowingresults,So,TheaveragemutualinformationdiagramisshowninthefollowingFig.4.7.Fig.4.7.MutualinformationofthedualsymmetricchannelFromthediagram,wecanseethatwhentheinputsymbolssatisfy“equalprobabilitydistribution”,theaveragemutualinformationI(X;Y)reachesthemaximumvalue,andonlyatthistimethereceivergetsthelargestinformationfromeverysymbolhereceived.Property2Relationshipbetweeninformationandchanneltransitionprobabilitydistribution.I(X;Y)isaconcavefunctionofchanneltransitionprobabilitydistributionofp(Y|X).Fig.4.8.I(X;Y)isaconcavefunctionofP(X|Y)E.g.4.2(Thisisthefollow-upofE.g.4.1)Consideringdualchannel,nowweknowtheaveragemutualinformationis,whenthesourcedistributionisaveragemutualinformationI(X;Y)istheconcavefunctionofp,justseeitfromthefollowingdiagram,,theMutualinfo.offixedbinarysource

Fromthediagram,wecansee,oncethebinarysourcefixed,whenthechannelpropertypchanges,we’llgetthedifferentmutualinformationI(X;Y),whenp=1/2,I(X;Y)=0,thatmeansthereceivergetthelestinformationfromthischannel,andalltheinformationislostinthewayoftransmission,thischannelhasthemostloudlynoise.Property3Ifthechannelinputisdiscreteandwithoutmemory,wehavethefollowinginequalityProperty4Ifthechannelisdiscreteandwithoutmemory,wehave(Remembertheresults)4.2.4Relationshipbetweenentropy,channeldoubtdegreeandaveragemutualinformation

E.g.4.3Thereisasource

Itsmessagespassthroughachannelwithnoise.Thesymbolsreceivedbytheotherendofthechannelare.Thechannel’stransfermatrixis,pleasecalculate(1)Theself-informationincludedinthesymbolandofevent.(2)Theinformationaboutthereceivergetswhenitobservesthemessage(3)TheentropyofsourceXandreceivedY(4)ThechanneldoubtdegreeH(X|Y)andthenoiseentropyH(Y|X).(5)TheaveragemutualinformationgotbyreceiverwhenitreceivesY.

Solution:(1):(2):(3):(5):(4):4.3DiscretechannelwithoutmemoryThreegroupsofvariablestodescribethechannel:(1)Channelinputprobabilityspace,(2)Channeloutputprobabilityspace,(3)Channeltransferprobability,So,thechannelcanberepresentedbyThiscanbeindicatedbythefollowingillustration,andthechanneltransfermatrixis

WhenK=1,itdegeneratestothesinglemessagechannel;andwhenn=m=2,itdegeneratestothebinarysinglemessagechannel.Ifitsatisfiessymmetry,itconstitutesthemostcommonlyusedBSC.Fig.4.11.Binarymessagesymmetricalchannel4.4Channelcapacity4.4.1Theconceptofchannelcapacity4.4.2Discretechannelwithoutmemoryanditschannelcapacity4.4.3Continuouschannelanditschannelcapacity4.4.1TheconceptofchannelcapacityThecapacityofchannelcanbedefinedasthemaximumvalueofaveragemutualinformation,TheunitofchannelcapacityCisbit/symbolor

nat/symbolFromthepropertymentionedbefore,weknowI(X;Y)isanupperconvexfunctionofprobabilitydistributionp(x)ofinputvariableX.Foraspecificchannel,therealwaysexistsasourcewhichmaximizestheinformationofeverymessagetransmittingthroughthechannel.ThatmeansthemaximumofI(X;Y)exists.Andtheprobabilitydistributionp(x)iscalledtheoptimuminputdistribution.4.4.2DiscretechannelwithoutmemoryanditschannelcapacityClassificationofthediscretemessagesequencechannelFordiscretechannelwithoutmemory,itsatisfiesthefollowingrelationship.Accordingtothe“property4”ofthemutualinformation

I(X;Y)ofthemessagesequence,forthediscretechannelwithoutmemory,wehaveNote:onlywhenthesourceisofwithoutmemory,theequalrelationshipinthisformulamaybesatisfied

SowecangetthefollowingdeductionwhichgetstheformulaofchannelcapacityCTheoremofdiscretechannelwithoutmemory

AssumingthatthetransmissionprobabilitymatrixofthediscretechannelwithoutmemoryisQ,thesufficientconditionsunderwhichtheinputletterprobabilitydistributionp*canmakethemutualinformationI(p;Q)toachievemaximumvalueare

Whereistheaverage

mutualinformationwhensourceletterissent;andCisthechannelcapacityofthischannel.UnderstandingthistheoremFirstly,underthiskindofdistribution,eachletterwhoseprobabilityisabovezeroprovidesmutualinformationC,andeachletterwhoseprobabilityiszeroprovidesmutuallyinformationlowerthanorequaltoC.Secondly,onlyunderthiskindofdistribution,itmaycauseI(p;Q)toobtainthemaximumvalueC.Thirdly,I(X;Y)istheaverageof.Thatistosay,itsatisfiesthisequation

(1)IfwewanttoenhanceI(X;Y),enhancingp(ak)maybeagoodidea.(2)However,oncep(ak)isenhanced,I(x=ak;Y)maybereduced.(3)Toadjustp(ak)repeatedly,makeI(x=ak;Y)allequaltoC

(4)ThistimeI(X,Y)=CThetheoremonlyprovidesasufficientconditionoftomake

distributionandthevalueofC;butitmayhelptogetthevalueofCofseveralkindsofchannelsinsimplesituation.ItdoesnotgivetheconcreteE.g.4.4

Assumethetransmissionmatrixofbinarydiscretesymmetricalchannelis(1)If

pleasecalculate

(2)Pleasecalculatethecapacityofchannel,andtheprobabilitydistributionwhenreachingthecapacityofchannel.(1)(2)ApplicationExample3.6Wheremrepresentsthenumberofoutputsymbolset;Hmiistheentropyoftherowvectorofchannelmatrix.4.4.3ContinuouschannelanditschannelcapacityCharacteristicofcontinuouschannelAnalogchannelBasicknowledgetoaddablechannelShannonformulaUsageofShannonformulaCharacteristicofcontinuouschannelCharacteristic1:Thetimeisdiscrete,thevaluescopeiscontinuous.Characteristic2:Ateachmoment,itisthesinglerandomvariablewhosevalueiscontinuous.AnalogchannelBasicknowledgetoaddablechannel

X:channelinput

N:channelnoiseY:channeloutputIftwoof

X,Y,NareGaussdistributions,thentheotherisalsotheGaussdistribution.Thedifferentialentropyofther.v.satisfyingGaussiandistributiononlyconcernswithitsvariance

andhasnothingtodowiththeaveragevalue.Fig.4.22.AddablechannelTheorem:Whenagenerallystationaryrandomprocesssourcewithlimitedfrequency(F)andtime(T)passesthroughawhiteGaussianchannelwhichhaslimitedpower(PN),thechannelcapacityis:ShannonformulaThisisthefamousShannonformulaforcontinuouschannel.WhenT=1,thecapacityis:ProofofShannonformula:Assume,whereXandNareindependent

discreter.v.’s;andSinceWehaveThebiggestentropytheoremoflimitedaveragepowerProofofShannonformula:DuetothelimitedfrequencyFforandaccordingtotheNyquistsampletheorem,thecontinuoussignalX(t,w)canbeequivalentto2Fdiscretesignalspersecond.Thatis:ConsideringtimedurationT:Fig.4.23.ShannonformulaUsageofShannonformul