经济资本模型验证方法

ValidationofEconomicCapitalModels:StateofthePractice,SupervisoryExpectationsandResultsfromaBankStudyMichaelJacobs,Ph.D.,CFASeniorFinancialEconomist/CreditRiskAnalysisDivisionU.S.OfficeoftheComptrolleroftheCurrencyRiskConferenceonEconomicCapital,NYC,February2010TheviewsexpressedhereinarethoseoftheauthoranddonotnecessarilyrepresenttheviewsoftheOfficeoftheComptrolleroftheCurrencyortheDepartmentoftheTreasury.经济资本模型验证方法共37页，您现在浏览的是第1页!Outline Introduction,BackgroundandMotivationFitnessforUseofEconomicCapital(EC)ModelsProvidingConfidenceRegardingECModelAssumptionsAssessingtheValueofValidationMethodologiesQualitativeApproachesQuantitativeApproachesSupervisoryConcernsandExpectationsLimitationsofValidationEffectiveReportingofModelOutputsTechnicalChallengesinTestingtheAccuracyofECModelsAssessingAccuracyintheTailsoftheLossDistributionExample:AlternativeModelsforRiskAggregation经济资本模型验证方法共37页，您现在浏览的是第2页!Introduction,BackgroundandMotivationThevalidationofECmodelsisataverypreliminarystageInternalcapitaladequacyassessmentprocess(ICAAP):notamodelforEC,butanoverallframeworkforassessingifECisappropriateECmaybeaquantitativeponentofICAAP,butitisnotrequiredofallbanksbysupervisors(onlythelargest)ValidationofthequantitativeECponentofICAAP,ifthereisone,isaponentofICAAPECmodelscanbeplex,havingmanyponents,anditmaynotbeimmediatelyobviousthatasuchmodelworkssatisfactorilyModelsmayembodyassumptionsaboutrelationshipsamongstorbehaviorofvariablesthatmaynotalwayshold(e.g.,stress)Validationcanprovideadegreeofconfidencethatassumptionsareappropriate,increasingtheconfidenceofusersinthemodeloutputsAdditionally,validationcanbealsousefulinidentifyingthelimitationsofECmodels(i.e.,whereembeddedassumptionsdonotfitreality)经济资本模型验证方法共37页，您现在浏览的是第3页!Introduction,BackgroundandMotivation(continued)Thereexistsawiderangeofvalidationtechniques,eachprovidingevidenceregardingonlysomeofthedesirablepropertiesofamodelSuchtechniquesarepowerfulinsomeareas(risksensitivity)butnotinothers(accuracyofabsoluteECorquantileestimator)ArangeofvalidationtechniquescanprovidemoresubstantialevidencefororagainsttheperformanceofthemodelParticularlyinanenvironmentofgoodcontrolsandgovernanceThereappearstobescopefortheindustrytoimprovethevalidationpracticesthatshedlightontheoverallcalibrationofmodelsParticularlyincaseswhereassessmentofoverallcapitalisanimportantapplicationofthemodel经济资本模型验证方法共37页，您现在浏览的是第4页!FitnessforPurposeofECModels(continued)ECareconceptuallysimilartoVaRmodels,butseveraldifferencesforcevalidationmethodstodifferinpracticefromthoseusedinVaRLongtimehorizon,highconfidencelevels,andthescarcityofdataFullinternalECmodelsarenotusedforPillar1minimumcapitalrequirements,sofitnessforpurposeneedstocoverarangeofusesMostofwhichandperhapsalltheseusesareinternaltothefirminquestionNotethatECmodelsandregulatorycapitalservedifferentobjectives&mayreasonablydifferinsomedetailsofimplementationBCBS’sValidationPrinciple1referstopredictiveabilityofcreditratingsystems,anemphasisonperformanceofmodelforecastsThenaturalevolutionofthisprincipleforECisthatvalidationisconcernedwiththepredictivepropertiesofthosemodelsI.e.,embodyforward-lookingestimatesofrisk&theirvalidationinvolvesassessingthoseestimates,sothisrelatedprincipleremainsappropriateBroadlyinterpretedvalidationprocessessetouthereinindifferentwaysallprovideinsightintothepredictiveabilityofECmodel经济资本模型验证方法共37页，您现在浏览的是第5页!ProvidingConfidenceRegardingECModelAssumptions(cont.)ThereisgreatdifficultyinvalidatingconceptualsoundnessofanECmodelduetomanyuntestableorhard-to-testassumptionsmade:FamilyofstatisticaldistributionsforriskfactorsEconomicprocessesdrivingdefaultorloss(e.g.,observablevs.latent)Dependencystructureamongrisksorlosses(e.g.,copulae)Behaviorofmanagementoreconomicagents&howthesevaryovertimeSomeECmodelsareofriskaggregationmodelswhereestimatesforindividualcategoriesarebinedtogenerateasingleriskfigureTheremaybenobestoruniquewaytodothisaggregationSincemanyoftheseassumptionsmaybeuntestable,itmaybeimpossibletobecertainthatamodelisconceptuallysoundWhiletheconceptualunderpinningsmayappearcoherentandplausible,theymayinpracticebenomorethanuntestedhypothesesOpinionsmayreasonablydifferaboutthestrengthorweaknessofanyparticularprocessinrespectofanygivenproperty经济资本模型验证方法共37页，您现在浏览的是第6页!ValidationofECModels:RangeofPracticeinQualitativeApproachesThephilosophyoftheusetestasincorporatedintotheBaselIIframework:ifabankisactuallyusingitsriskmeasurementsystemsforinternalpurposes,thenwecanplacemorerelianceonitApplyingtheusetestsuccessfullywillentailgainingacarefulunderstandingofwhichmodelpropertiesarebeingusedandwhicharenotBankstendtosubjecttheirmodelstosomeformofqualitativereviewprocess,whichcouldentail:ReviewofdocumentationordevelopmentworkDialoguewithmodeldevelopersormodelmanagersReviewandderivationofanyformulaeoralgorithmsComparisontootherfirmsorwithpubliclyavailableinformationQualitativereviewisbestabletoanswerquestionssuchas:Doesthemodelworkintheory?Doesthemodelincorporatetherightriskdrivers?Isanytheoryunderpinningitconceptuallywell-founded?Isthemathematicsofthemodelright?经济资本模型验证方法共37页，您现在浏览的是第7页!RangeofPracticeinQualitativeApproachestoValidation(concluded)Asallmodelsrestonpremisesofvariouskinds,varyinginthedegreetowhichobvious,wehaveexaminationofassumptionsCertainaspectsofanECmodelare'built-in'andcannotbealteredwithoutfundamentallychangingthemodelToillustrate,theseassumptionscouldbeabout:Fixedmodelparameters(PDs,correlationsorrecoveryrates)Distributionalassumptions(margins,copulae&shapeoftaildistributions)BehaviorofseniormanagementorofcustomersSomebanksgothroughadeliberateprocessofdetailingtheassumptionsunderpinningtheirmodels,includingexaminationof:ImpactonmodeloutputsLimitationsthattheassumptionsplaceonmodelusageandapplicability经济资本模型验证方法共37页，您现在浏览的是第8页!RangeofPracticeinQuantitativeValidation:ModelReplicationModelreplicationisusefultechniquethatattemptstoreplicateECmodelresultsobtainedbythebankThiscoulduseindependentlydevelopedalgorithmsordatasources,butinpracticereplicationmightleverageabank’sexistingprocessesE.g.,runamodelofthesametypeorclassonathebank'sdata-setHowever,butoncetheeithertheoriginalortestmodelhasbeenvalidatedThistechniqueandthequestionsthatoftenariseinimplementingreplicationcanhelpidentifyif:Definitions&algorithmsthebankclaimstousecorrectlyareunderstoodbystaffwhodevelop,maintain,operateandvalidatethemodelThebankisusinginpracticethemodelingframeworkthatitpurportstoComputercodeiscorrect,efficientandwell-documentedDataclaimedtobeusedbythebanktoobtainitsresultsisinfactbeingusedHowever,thistechniqueisrarelysufficienttovalidatemodels,andinpracticethereislittleevidenceofitbeingusedbybanksforeithervalidationortoexplorethedegreeofaccuracyoftheirmodelsNotethatreplicationsimplybyre-runningasetofalgorithmstoproduceanidenticalsetofresultswouldnotbesufficientmodelvalidationduediligence经济资本模型验证方法共37页，您现在浏览的是第9页!RangeofPracticeinQuantitativeValidation:Benchmarking(continued)Theremaybegoodreasonswhymodelsproduceoutliersinbenchmarking,allofwhichplicateinterpretationoftheresults:MaybedesignedtoperformwellunderdifferingcircumstancesMaybemoreorlessconservativelyparameterizedMaydifferintheireconomicfoundationsComparisonsofinternalECaremadewithvariedalternatives:IndustrysurveyresultsRatingagencyorindustry-widemodelsConsultancymarketedmodelsAcademicpapersRegulatorycapitalmodels经济资本模型验证方法共37页，您现在浏览的是第10页!RangeofPracticeinQuantitativeValidation:BacktestingBacktestingaddressesthequestionofhowwellthemodelforecaststhedistributionofoutesTherearemanyformsofthisthatentailsomedegreeofparisonofoutestoforecasts,andthereisawideliteratureonthesubjectHowever,weakpowerofbacktestingtestsformodelsofriskthatquantifyhighquantileshasbeennotedE.g.,forportfoliocreditmodelsseeBCBS(1999)Variationstothebasicbacktestingapproachwhichcanincreasethepowerofthetestshavebeensuggestedintheliterature:Backtestingmorefrequentlyovershorterholdingperiods(e.g.,inmarketriskusingaone-daystandardversusthe10-dayregulatorycapitalstandardUsingcross-sectionaldataonarangeofreferenceportfoliosUsinginformationinforecastsofthefulldistributionTestingexpectedvaluesofdistributionsasopposedtohighquantiles经济资本模型验证方法共37页，您现在浏览的是第11页!RangeofPracticeinQuantitativeValidation:StressTestingStresstestingcoversbothstressingofthemodelandparisonofmodeloutputstostresslossesTheoutputsofthemodelmightbeexaminedunderconditionswheremodelinputsandmodelassumptionsmightbestressedThisprocesscanrevealmodellimitationsorhighlightcapitalconstraintsthatmightonlybeeapparentunderstressWhilestresstestingofregulatorycapitalmodels,particularlyIRBmodels,ismonlyundertakenbybanks,thereismorelimitedevidenceofstresstestingofeconomiccapitalmodelsThroughaplementaryprogrammeofstresstesting,thebankmaybeabletoquantifythelikelylossesthatthefirmwouldconfrontunderarangeofstresseventsComparisonofstresslossesagainstmodel-basedcapitalestimatesmayprovideamodestdegreeoffortoftheabsolutelevelofcapitalBanksreportsomeuseofthisstresstestingtechniquetovalidatetheapproximatelevelofmodeloutput经济资本模型验证方法共37页，您现在浏览的是第12页!SupervisoryConcernsandExpectationsintheValidationofECModelsGiventheinherentlimitationsofECmodelvalidation,thereisanemphasisoneffectivereportingtoavoidmisuseormisunderstandingArecognitionofintrinsicdifficultyinevaluationofhighquantilesoflossoverlongperiods,datascarcity&technicaldifficultiesClearreportingofsuchdifficulties&limitationstousers&seniormanagementisnecessaryforthemtounderstandthattheremaybegreateruncertaintyaroundtheoutputfromECmodelsModelsnotfullyvalidatedimpliesoutputshouldgenerallybetreatedwithextraconservatismUnderstand&explorethepotentialcostsofusingmodelsinthissituation(i.e.,ifkeyassumptionsprovetobeinaccurate)Whilevalidationpracticesdependonmodeltype&use,weaknessestargetedatevaluationofoverallperformancemightresultinbanksoperatingwithinappropriatelycalibratedmodelsThiscouldbeofconcernifassessmentofoverallcapitaladequacyisanimportantapplicationofthemodelImprovementsintheseareascouldincludefurtherbenchmarking&industry-wideexercises,backtesting,P/Lanalysisandstresstesting经济资本模型验证方法共37页，您现在浏览的是第13页!TechnicalChallengesinAssessingtheAdequacyofanECModelAfundamentaldifficultyfacedinECmodelingisthelackofdatatoestimatehighquantilesinthetailsofthelossdistributionleadstoaveryhighdegreeofuncertaintyOneapproachtodealingwiththisproblemisBayesiantechniques,whichbineexpertassessmentswithavailabledataSeeKiefer&Jacobs(2010,forthing)However,thisisputationallydemanding,andalsorequirestheelicitationofapriordistributionfromanexpect,whichisveryinvolvedButifthepriorisdiffuse,thenmuchuncertaintystillremainsAswedon’tseethisusedinpracticecurrently,wewillnotfurtherpursuethisapproachhereAsnotedpreviously,traditionalbacktestingproceduresasappliedinmarketriskVaRmodelsareimpracticalinanECmodelsettingAnalternativeapproachistotrytoassesstheaccuracyoftheECoutputbyapproximatingastatisticalmeasureofuncertaintyE.g.,throughresamplingorbootstrapmethods(Efronetal,1986)Butthindatainthetailsimpliesconfidenceboundsarelikelytobewide

.经济资本模型验证方法共37页，您现在浏览的是第14页!ValidationExample:AlternativeRiskAggregationModels–RiskProxyDataSummary(LargestBanksAsOf4Q08)经济资本模型验证方法共37页，您现在浏览的是第15页!ValidationExample:AlternativeRiskAggregationModels–CorrelationsofRiskProxies(LargestBanksAsOf4Q08))经济资本模型验证方法共37页，您现在浏览的是第16页!ValidationExample:AlternativeRiskAggregationModels–AbsoluteECComparison(LargestBanksAsof4Q08)经济资本模型验证方法共37页，您现在浏览的是第17页!ValidationExample:AlternativeRiskAggregationModels–%DiversificationComparison(LargestBanksAsof4Q08)经济资本模型验证方法共37页，您现在浏览的是第18页!ValidationExample:AlternativeRiskAggregationModels–ECVariabilityComparison(LargestBanksAsof4Q08)经济资本模型验证方法共37页，您现在浏览的是第19页!ValidationExample:AlternativeRiskAggregationModels–ECVariabilityComparison(LargestBanksAsof4Q08)经济资本模型验证方法共37页，您现在浏览的是第20页!FitnessforPurposeofEconomicCapitalModelsInsomecasesthetermvalidationisusedexclusivelytorefertostatisticalexpostvalidation(e.g.,backtestingofaVaR)InothercasesitisseenasabroaderbutstillquantitativeprocessthatalsoincorporatesevidencefromthemodeldevelopmentstageHerein,“validation”ismeantbroadly,meaningalltheprocessesthatprovideevidence-basedassessmentofamodel'sfitnessforpurposeThisassessmentmightextendtothemanagementandsystemsenvironmentwithinwhichthemodelisoperatedItisadvisablethatvalidationprocessesaredesignedalongsidedevelopmentofthemodels,ratherthanchronologicallyThisinterpretationofvalidationisconsistentwiththeBaselCommittee(2005)inrelationtotheBaselIIFrameworkHowever,thatwasphrasedintermsoftheIRBparameters&developedinthecontextofassessmentofriskestimatesforuseinminimumcapitalrequirementsValidationofECdifferstoanIRBmodelastheoutputisadistributionratherthanasinglepredictedforecastagainstwhichactualoutesmaybepared经济资本模型验证方法共37页，您现在浏览的是第21页!ProvidingConfidenceRegardingECModelAssumptionsPropertiesofanECmodelthatcanbeassessedusingpowerfultools,andhencethatarecapableofrobustassessment,include:IntegrityofmodelimplementationDegreetowhichgroundedinhistoricalexperienceSensitivitytorisk&totheexternalenvironmentGoodmarginalpropertiesRankordering&relativequantificationPropertiesofanECmodelforwhichonlyweakervalidationprocessesareavailableinclude:Conceptualsoundness&validityofassumptionsDegreetowhichmodelisforward-lookingAbsoluteriskquantification&predictiveaccuracyofriskestimateItisimportanttostressthepowerofindividualtests&acknowledgethatviewsastostrengthandweaknessarelikelytodiffer经济资本模型验证方法共37页，您现在浏览的是第22页!ValidationofECModels:IntroductiontoRangeofPracticeWhilewewilldescribethetypesofvalidationprocessesthatareinuseorcouldbeused,notethatthelistisnotprehensiveWedonotsuggestthatalltechniquesshouldorcouldbeusedbybanksWewishtodemonstratethatthereisawiderangeoftechniquespotentiallycoveredbyourbroaddefinitionofvalidationThisiscreatingalayeredapproach,themore(fewer)ofwhichthatcanbeprovided,themore(less)fortthatvalidationisabletoprovideevidencefororagainsttheperformanceofthemodelEachvalidationprocessprovidesevidencefor(oragainst)onlysomeofthedesirablepropertiesofamodelThelistpresentedbelowmovesfromthemorequalitativetothemorequantitativevalidationprocesses,andtheextentofuseisbrieflydiscussed经济资本模型验证方法共37页，您现在浏览的是第23页!RangeofPracticeinQualitativeApproachestoValidation(continued)Extensivesystemsimplementationtestingisstandardforproduction-levelriskmeasurementsystemspriortoimplementationE.g.,useracceptancetesting,checkingofmodelcode,etc.Theseprocessescouldbeviewedaspartoftheoverallvalidationeffort,sincetheywouldassistinevaluatingwhetherthemodelisimplementedwithintegrityManagementoversightistheinvolvementofseniormanagementinthevalidationprocessE.g.,reviewingoutputfromthemodel&usingtheresultsinbusinessdecisionsSeniormanagementknowinghowthemodelisused&outputsareinterpretedThisshouldtakeaccountofthespecificimplementationframeworkadoptedandtheassumptionsunderlyingthemodelanditsparameterizationDataqualitychecksrefertotheprocessesdesignedtoprovideassuranceofthepleteness,accuracyandappropriatenessofdatausedtodevelop,validateandoperatethemodelE.g.,Reviewof:datacollectionandstorage,datacleaningoferrors,extentofproxydata,processesthatneedtobefollowedtoconvertrawdataintosuitablemodelinputs,andverificationoftransactiondatasuchasexposurelevelsWhilenottraditionallyviewedbytheindustryasaformofvalidation,increasinglyformingamajorpartofsupervisorythinking经济资本模型验证方法共37页，您现在浏览的是第24页!RangeofPracticeinQuantitativeApproachestoValidation:InputsApletevalidationofanECmodelwouldinvolvetheinputsandparameters,bothstatisticallyestimatedandnotExamplesofestimated(assumed)parametersarethemainIRBparameters(PDorLGD)(correlations,PDinalowdefaultportfolio)Techniquescouldincludeassessingparametersagainst:HistoricaldatathroughreplicationofestimatorsOutesovertimethroughbacktestingMarket-impliedparameters(e.g.,impliedvolorcorrelation,CDSspreadsforPD)MaterialitythroughsensitivitytestingThistestingofinputparameterscouldplementexaminationofassumptionspreviously&sensitivitytestingtodescribedlaterHowever,thatcheckingofmodelinputsisunlikelytobefullysatisfactorysince,everymodelisbasedonunderlyingassumptionsThemoresophisticatedthemodel,themoresusceptibletomodelerror,socheckinginputparameterswillnothelphereHowever,modelaccuracyandappropriatenesscanbeassessed,atleasttosomedegree,usingtheprocessesdescribedinthissection经济资本模型验证方法共37页，您现在浏览的是第25页!RangeofPracticeinQuantitativeValidation:BenchmarkingBenchmarkingtheparisonofabank’sECmodeltoalternativemodelsonthebank’sportfolioE.g.,toavendormodelafterstandardizationofparametersAmongthemostmonlyusedformsofquantitativevalidationusedinternallyAlimitationofbenchmarkingisitonlyprovidesrelativeassessmentsandprovideslittleassurancethatanymodelaccuratelyreflectsrealityorabouttheabsolutelevelsofcapitalTherefore,asavalidationtechnique,benchmarkingislimitedtoprovidingparisonofonemodelagainstanotheroronecalibrationtoothers,butnottestingagainst‘reality’Itisthereforedifficulttoassessthedegreeoffortprovidedbysuchbenchmarkingmethods,astheymayonlybecapableofprovidingbroadparisonsconfirmingthatinputparametersormodeloutputsarebroadlyparable经济资本模型验证方法共37页，您现在浏览的是第26页!RangeofPracticeinQuantitativeValidation:HypotheticalPortfoliosHypotheticalportfoliotestingisanexaminationofeitherdifferentbank’sECmodelsonareferenceportfolio,ordifferentbanksECoutputfromagivenreferencemodelThisistypicallyaeitherareferencemodelorportfolioexternaltoanyonebankFromasupervisoryperspective:permitsidentificationofmodelsthatproduceoutliersamongstasetofbanksA“modelriskmanagement”toolAlternatively,thishelpssupervisorsidentifybanksthatareoutliersinriskwithrespecttoareferencemodelA“bankportfolioriskmanagement”toolIneithercasethismeansparisonacrossbank’smodelsagainstthesamereferenceportfolio(externaltothebank)orofbanksthemselves(theirECforagivenreferencemodel)Capableofaddressingsimilarquestionstobenchmarking,butbydifferentmeansThetechniqueisapowerfuloneandcanbeadaptedtoanalyzemanyofthepreferredmodelpropertiessuchasrank-orderingandrelativeriskquantification经济资本模型验证方法共37页，您现在浏览的是第27页!RangeofPracticeinQuantitativeValidation:Backtesting(continued)BacktestingisusefulprincipallyformodelswhoseoutputsareaquantifiablemetricwithwhichtopareanouteHowever,someriskmeasurementsystemsinusehaveoutputscannotbeinterpretedinthiswayandcannotbebacktestedSuchriskmeasurementapproachesnotamenabletooutes-basedvalidationmightneverthelessbevaluabletoolsforbanksE.g.,ratingsystems,sensitivitytestsandaggregatedstresslosses.Theroleofbacktestingforsuchmodels,ifused,wouldneedelaborationInpractice,backtestingisnotyetakeyponentofbanks'validationpracticesforeconomiccapitalpurposes经济资本模型验证方法共37页，您现在浏览的是第28页!RangeofPracticeinValidation:AdditionalConsiderationsWehavenotmentionedinternalaudit,butvalidationoftheoverallimplementationframeworkandprocessshouldalsobesubjecttoindependentandperiodicreviewThisworkshouldbemadebypartieswithinthebankingorganizationthatareindependentofthoseaccountableforthedesignandimplementationofthevalidationprocessApossibilityisthatinternalauditwouldbeinchargeofundertakingthisreviewprocess,andassuchitcouldbeviewedasprisingapartofthemanagementoversightprocessdiscussedpreviouslyThelistofvalidationtoolsalsodoesnotaddresstheissueofadequateinternalstandardsrelevantforvalidationExamplesofsuchstandardsinclude:AdescriptionoftheissuesthatneedtobeaddressedaspartofvalidationThestandardsthatcapitalmodelsareexpectedtoachieveAseriesofquantitativethresholdsthatmodelsneedtomeetWarningindicatorsforparticularmonitoringmetricsAssessmentagainstmodeldevelopmentstandards经济资本模型验证方法共37页，您现在浏览的是第29页!SupervisoryConcernsandExpectationsintheValidationofECModels(cont.)Mainareasofimprovementseentobeinbenchmarkingofmodelparametersandtheconductofcross-firmparisonsofmodelsE.g.,IACPMandISDAstudy(2006)onportfoliocreditriskmodelsEvidencethatbanksensureECmodelsaresensitivetothe