一种用于微电网能量管理的多目标强化学习框架 A Multiobjective Reinforcement Learning Framework for Microgrid Energy Management

arXivarXiv2v1[eess.SY]17Jul2023Abstract—Theemergenceofmicrogrids(MGs)hasprovidedgsolutionfordecarbonizinganddecentralizingtheacehelicywithouttheneedforlongtermcoordinativeframeworkontheCornellUniversityMG(CU-MG),whichisaityanalysisI.INTRODUCTIONeldofSystemsEngineeringCornellUniversityhtheFacilitiesandCampusServiceCornellUniversityC.L.AndersoniswiththeFieldofSystemsEngineeringandCornell e hetsII.MULTI-OBJECTIVEENERGYMANAGEMENTFORMULATIONONCU-MGA.IntroductionoftheMicrogridFig.1.sC(p,t,s)=φtη(p,t,s)=a+b·()+c·()2R≤p,t,s−p,t−1,s≤R(3)Pic≤p,t,s≤Pic(4)C(q,t,s)=max(a·q,t,s−b,0)φt(5)Q≤q,t,s≤Q(6)C(q,s)=(ab+bb·(q,s)+c(q)))φ2t(7)Qb≤q,s≤Qb(8)sa·(对iq,t,s+q,s)+bifp,s=〈对iq,t,s+q,s>cs(a·(对iq,t,s+q,s)+botherwise4)PowerExchangeatPCCTheCHPMGcanbuyorC(p,s)=πt,s·p,s(10)hourtinscenarios.p,s>0indicatesthatbuyinggysB.FormulationtheenergymanagementproblemeO1=((C(p,t,s)+C(q,t,s))++C(q,s)+C(p,s)yEF=gt,s·ϵLt,s+HLt,sgt,s=(+max(a·q,t,s−b,0))(13)+(ab+bb·(q,t,s)+cb(q,t,s))2)STO2=(gt,s·Lt,s+EFgrid·max(p,s,0))(14)STO3=It,s(15)It,s=〈(1,if1+对1qs−HLt,s>δ(0,otherwise2)SystemLevelConstraints:InadditiontothephysicalncLt,s=工p,t,s+p,s+ys+y,s+p,i=1ST工工RLt,s≥s=1t=1RLt,s=〈1,if1+HLt,s≥0.95 (RLt,s=〈1,if1+HLt,s≥0.95(0,otherwisePsformulatedasAt=(p,t,s,q,t,s,q,t,s).TheSToutputp,seAt=argminFF=(O1,O2,O3)(20)s.t.(3)−(10),(17)−(19)III.METHODOLOGIESaA.StateVariableseeSt=(Wt,Yt,Ht)Wt=(Tet,s,Wvt,s,Srt,s,Sft,s,π,s)Ht=(Lt,s,y,s,ys,HLt,s,πt,s)Yt=(p,t−1,s,q,t−1,s,q,t−1,s)dsarethetemperatureB.ActionsandfunctionapproximationsstatetotheactionAt=πθ(Wt,Yt).Thebenefitisthree-fC.RewardFunctionsncR1(st,at)=工(C(p,t,s)+C(q,t,s))+C(q,s)+C(p,s)i=1R2(st,at)=gt,s·Lt,s+EFgrid·max(p,s,0)R3(st,at)=It,sD.BorgMOEAnaleyhE.Model-freeRLmodelSt+1=f(St,At).AsshowninFigure3,theagentdTV=Es(工R1(st,at))t=1TV=Es(工R2(st,att=1TV=Es(工R3(st,at))t=1F.TimeVaryingSensitivityAnalysisCost($)HeatWasteCost($)Emission(MetricTon)eatasteNon-dominatedPoliciesCurrentOperation0.800.750.700.650.600.5515200154001i0(cTon)144001460014800150001901851801560015800155Cost($)HeatWasteCost($)Emission(MetricTon)eatasteNon-dominatedPoliciesCurrentOperation0.800.750.700.650.600.5515200154001i0(cTon)144001460014800150001901851801560015800155150Non-dominatedPoliciesCurrentOperation0.90.80.70.60.50.494009600980010000102001040010600140200Low-emissioncompromisepolicyIdealPointLow-emissioncompromisepolicyIdealPointkVar(ut|t)=()Var((Wt)a)+()()Cov((Wt)a,(Wt)b)IV.NUMERICALRESULTSeA.TrainingandTestdatarsyB.TradeoffsinObjectiveSpace C.AdaptiveandCoordinativePerformanceofPolicies7rsPn(a)Observablestates(b)Unobservablestates(c)Operationdecisions(a)Observablestates(b)Unobservablestates(c)OperationdecisionseytsD.TVSAfortheParametricPoliciesneiFig.8:TVSAforwinterV.CONCLUSIONtherACKNOWLEDGMENTSREFERENCES[1]R.H.Lasseter,“Microgrids,”in2002IEEEpowerengineeringsocietywintermeeting.Conferenceproceedings(Cat.No.02CH37309),vol.1.IEEEpp.305–308.distributiongridflexibility,”IEEETransactionsonPowerSystems,in2015IEEEinternationalconferenceonmechatronicsandautomation(ICMA).IEEE,2015,pp.76–81.EEEPESInnovativeSmartGridTechnologiesEurope(ISGTEurope).IEEE,onSmartGrid,vol.4,no.4,pp.2174–2181,2013.researchandmanagementscience,vol.2,pp.331–434,1990.neuralnetworklearning,”IEEETransactionsonSmartGrid,vol.10,EnergySystems,vol.6,no.1,pp.213–225,2019.ing:anoverview,”inProceedingsofSAIIntelligentSystemsConference(IntelliSys)2016:Volume2.Springer,2018,pp.426–440.information,”IEEETransactionsonSmartGrid,vol.11,no.2,pp.WLiuPZhuangHLiangJPengandZ.Huang,“Distributedlearning,”IEEEtransactionsonneuralnetworksandlearningsystems,futurechallenges,”IEEETransactionsonSmartGrid,2022.ropeanJournalofOperationalResearch,vol.275,no.3,pp.795–821,[14]D.Bertsekas,Reinforcementlearningandoptimalcontrol.AthenariveStructuralandMultidisciplinaryOptimization,vol.48,pp.201–219,loringinachangingworld,”WaterResourcesResearch,vol.57,no.12,p.[20]X.Yang,Z.Leng,S.Xu,C.Yang,L.Yang,K.Liu,Y.Song,andRenewableEnergy,vol.172,pp.408–423,2021.ridsElectricPowerSystemsResearchvolpp201,2019. citiesandsociety,vol.45,pp.596–606,2019.ogrids,”IEEETransactionsonPowerDelivery,vol.27,no.3,pp.1243– arwalandTMeyarivanAfastandelitisttionarycomputationvolno,pp.182–197,2002.CC.CoelloandM.S.Lechuga,“Mopso:AproposalformultipleCongressonEvolutionaryComputation.CEC’02(Cat.No.02TH8600),T.T.Teo,T.Logenthiran,W.L.Woo,K.Abidi,T.John,N.S.Wade,IEEEtransactionsoncyberneticsvolnoppanJournalofOperationalResearch,vol.264,no.2,pp.582–606,2018.noPMReedDHadkaJDHermanJRKasprzykandJBKollatsthemsAdvancesinwaterresourcesvolp controlAdvancesinwaterresourcesvolKFBeckersMZLukawski,G.A.Aguirre,S.D.Hillson,andJ.W.yplayer,”inProceedings,40thworkshopongeothermalreservoirengi-neering,StanfordUniversity,Stanford,California,2015.facilities,”SustainableEnergy,GridsandNetworks,vol.11,pp.34–45,arycomputation,vol.20,no.3,pp.423–452,2012.points,”EnvironmentalModelling&Software,vol.92,pp.125–141,ResourcesResearch,vol.54,no.7,pp.4638–4662,2018.bsystemsEnvironmentalModellingSoftwarevolsunderuncertainty,”IEEETransactionsonControlSystemsTechnology,[41]J.D.Quinn,P.M.Reed,M.Giuliani,andA.Castelletti,“WhatissourcesResearchvolnopp5962–5984,2019.NewtransmissioninvestmentsaddanewchapterarXivarXiv2v1[eess.SY]17Jul2023AMultiobjectiveReinforcementLearningFrameworkforMicrogridEnergyManagement:SupplementaryInformation1.MICROGRIDPARAMETERSANDGENERATORMODELINGTheelectricityefficiency,fuelfunctionforextrasteamgeneration,heatratefortheboilerandtheelectricityrecoveryfunctionforthesteamturbinesareestimatedusinghistoricaldataof2019fromtheCornellmicrogriddataarchive[1].TheparameterizedfunctionsaredescribedinSectionII.AinthemanuscriptandarefittedusingtheFitnonlinearregressionmodel(fitnlm)functioninMatlab,whichissolvedbytheIterativelyReweightedLeastSquaresmethod[2,3].TheestimatedparametersaresummarizedinTableS1.TableS1.FittedparametersforgeneratorsParametervaluea1ab1bc1cOthermicrogrid-relatedparametersarereportedinTableS2forreproducibility.2TableS2.MicrogridrelatedparametersParametervalue i iii000KW000KW i iii-5000KW5000KW0klb/h153klb/h0klb/h540klb/hHc293kwh/dthe116.65lb/dth[4]6EFgrid0.932lb/kwh2.PARAMETERSFORMM-BORGIMPLEMENTATIONThemm-BorgisimplementedwithfourmasterstoruninparalleltofindthebestParetooptimalsolutionset.Theepsilonsforcost,emission,andheatwastearesettobe10,1,and0.01tofilteroutthee-dominatesolutions.AstheMOEAsaresensitivetotheinitializationofhyperparameters,10randomseedsareusedtoinitialize10parallelruns,eachwith500,000functionsofevaluations.ThefinalresultisthejointParetofrontieroverthesolutionsfromalltherandomseeds.3.HIGH-EMISSION,LOW-COSTPOLICYANALYSISToanalyzethebehaviorofapolicywithdifferentpreferences,weselectedalow-cost,high-emissionpolicyandcompareditsaveragedailyperformanceacrossthreeobjectiveswithhistoricaloperation,assummarizedinTableS3.Remarkably,anapproximately8%reductionincostwasachievedwithoutcompromisingemissions.ThecorrespondinghourlydecisionsforbothcoldandwarmdaysareillustratedinFigureS1.ItisevidentthatbothCHPsaregeneratingmoreelectricitylocally,enablingahighervolumeofpowertobesoldtotheutilityratherthanpurchased.ThisBuy/Sellbehaviorcloselyalignswiththecurrentoperation,comparedtothehigh-cost,low-emissionpolicy.Thisisasexpectedbecausethecurrentoperationtargetsminimizingcostonly.Thecostreductionisprimarilyattainedbygeneratingslightlymorepower,particularlyduringpricepeaks.However,asthepolicy’spriorityremainshigh-levelgenerationmostofthetime,electricitygenerationcloselyfollowstheelectricityloadcurvetominimizeemissions,especiallyduringperiodsoflowprices.Recallthattheloadprofileisnotavailabletotheagentuntilitmakesthedecisions,whichmeanstheagentlearnedthelatentloadprofilebyusingtheobservableweatherandpricevariables.Steamgenerationexhibitssimilaritiestothehigh-cost,low-emissionpolicy,withthecoordinatedoperationoftheCHPsandtheboileraligningwiththeheatloadwhileprioritizingtheutilizationofCHPsthathavehigherefficiency..TableS3.WinterCompassionbetweenrepresentativepoliciesandthecurrentOperationObjectiveCost($/day)Emission(MT/day)HeatWasteCurrentOperationLowcost,highemission3(a)Observablestates(b)Unobservablestates(c)OperationdecisionsFig.S1.Dailyplotsforlowcosthighemissionpolicyinwinter4.SECOND-ORDERTVSAANALYSISThisnotesummarizesfirst-andsecond-orderTVSAforthewarmandcolddaysinwinterandthehigh-priceandlow-pricedaysinsummerunderthehigh-cost,low-emissionpolicies.CHP1electricitygenerationCHP2electricitygenerationBoilersteamgenerationCHP1steamgenerationCHP2steamgenerationFig.S2.Fullfirst-orderTVSAforwinter-warmday4CHP1electricitygenerationCHP2electricitygenerationBoilersteamgenerationCHP1steamgenerationCHP2steamgenerationFig.S3.Fullsecond-orderpositiveinteractionTVSAforwinter-warmdayCHP1electricitygenerationCHP2electricitygenerationBoilersteamgenerationCHP1steamgenerationCHP2steamgenerationFig.S4.Fullsecond-ordernegativeinteractionTVSAforwinter-warmday5CHP1electricitygenerationCHP2electricitygenerationBoilersteamgenerationCHP1steamgenerationCHP2steamgenerationFig.S5.Fullfirst-orderTVSAforwinter-colddayCHP1electricitygenerationCHP2electricitygenerationBoilersteamgenerationCHP1steamgenerationCHP2steamgenerationFig.S6.Fullsecond-orderpositiveinteractionTVSAforwinter-coldday6CHP1electricitygenerationCHP2electricitygenerationBoilersteamgenerationCHP1steamgenerationCHP2steamgenerationFig.S7.Fullsecond-ordernegativeinteractionTVSAforwinter-colddayCHP1electricitygenerationCHPCHP2electricitygenerationBoilersteamgenerationCHP1steamgenerationCHP2steamgenerationCHP1On/OffCHP2On/OffBoilerOn/OffFig.S8.Fullfirst-orderTVSAforsummer-highpriceday7CHP2electricitygenerationCHP2electricitygenerationCHP1electricitygenerationCHP1electricitygenerationCHP1On/OffCHP2CHP2On/OffBoilerOn/OffBoilerOn/OffFig.S9.Fullsecond-orderpositiveTVSAforsummer-highpricedayCHP1electricitygenerationCHP1electricitygenerationCHP1On/OffC