跳出 AI 风险“末日概率”：非概率视角下的不确定性量化洞察报告

CenterforSecurityandEmergingTechnology|1

ExecutiveSummary

Manyobserverswonderaboutthepotentialforartificialintelligencetocause

catastrophicrisks,butfortunately,thereislittleempiricalevidenceonwhichtobasethoseassessments.Absentsuchevidence,expertsoftenusetheirbestguessestoestimatetheprobabilityofanAI-inducedcatastropheorapocalypse(i.e.,p-doom).Althoughsubjectiveexpertassessmentmaybethebestevidenceavailable,

policymakersandriskanalystsarenotrestrictedtoaskingforprobabilities.ThisbriefpromotesadditionaltoolsforhandlinguncertaintyinAIriskassessments.

Imagineyouareaskedtorollasix-sideddiebutyouhaveonlyseenthreesides;onesidehasastaretchedonit,twosidesareblank,andtheotherthreeareunknown.

Predictingtheoutcomeinvolvespartrandomnessandpartignorance.Ifyouareaskedtogivetheprobabilityofastar,youmightnotethatoneofthethreesidesyou’veseenhasastarandanswer1/3.Askedhowconfidenttobethatastarwillcomeup,thereisonlyonesidethatyouknowhasastar,so1/6wouldbeareasonableanswer.Askedwhetherastarcouldcomeup,youmightnotethatfoursidescouldhaveastar,so4/6wouldbeareasonableanswer.Thesequestionsappearsimilar,buttheirdifferencesareimportantifyouareadecision-makerwhocaresaboutstars.

InAIrisk,ratherthanindicerolls,ignoranceisthedominantformofuncertainty,notrandomness,sothebesttechniquesarenotalwaysprobabilistic.Therearealternativemathematicaltechniquesthatarejustasrigorousasprobability.Theyalsousefamiliartermsfromcommondiscourse,suchasBeliefandPlausibility,allowingthemtoeasilybecomepartofpopularAIriskvernacularandtobecommunicatedtodecisionmakers.

ThewaytothinkofthemathematicaltermBeliefisthatitexpresseshowconfident

onecanbebasedontheevidence.Forinstance,theevidenceallowsustohavea1/6degreeofbeliefthatthediewillcomeupstars.Plausibilityexpresseswhatisleftafterremovingthecounter-evidence.Twoofthesixsidescannotbestars,sothePlausibilityofstarsis4/6.ThegapbetweenBeliefandPlausibilityisduetoignorance.Without

ignorance,BeliefandPlausibilitybecomethesamenumber,equaltoprobability.

Thisissuebriefexplainswhyanalystsanddecision-makersneedalternativesto

probabilityforhandlingtheuncertaintyinAIrisk.ItexplainsBelief,Plausibility,and

howtheyrelatetoprobabilityinanintuitivelyaccessibleway.Anditdemonstrates

howtocalculateBeliefandPlausibilityinthecontextofexpertassessmentsofAIrisk.

CenterforSecurityandEmergingTechnology|2

Atahighlevel,enactingthechangesoughtbythisbriefiseasy.Policymakersonly

needtoaddtwoadditionalquestionswhendiscussingAIrisks.Thefirstiseither,howcertainareyouthatthisriskwilloccur,orevenbetter,howstrongistheevidence

supportingthishypotheticaloutcome?Thesecondis,howcertainareyouthatthisriskwillnotoccur,orhowstrongistheevidenceagainstthishypotheticaloutcome?

Askingthosetwoadditionalquestionswillforceanalyststoconfronttheirsourcesofuncertaintymoredirectlyanddriveanalyststoexpandtheirriskanalysistoolbox.

Answeringthosequestions,andcommunicatingthoseanswers,isalsoalowlift

becausetheanalyticaltechniquesalreadyexistandbecausethevocabularyisalreadyfamiliar.Thisbriefprovidesanintroductiontothosetechniquesandvocabulary.

CenterforSecurityandEmergingTechnology|3

TableofContents

ExecutiveSummary 1

Introduction 4

ProbabilityIsNottheOnlyOption 5

AleatoricandEpistemicUncertainty 5

AlternativestoProbability 7

Belief,Plausibility,andProbability 8

Belief 8

Plausibility 8

Probability 8

StructureofEvidence 9

Indirect,Subjective,andConflicting 9

AssertingProbabilities 10

ConfidenceTriplets 10

ConfidenceTripletsfromAIRiskProbabilities 11

SourcesofDisagreements 12

CombiningEvidence 13

FullIgnorance 13

FullCertainty 13

MixedIgnoranceandCertainty 14

CalculatingAgreementRatherThanAddingNewEvidence 15

FutureWorkandLimitations 16

Conclusion 18

Author 19

Acknowledgments 19

AppendixA:CombiningExpertstoCalculateBeliefandPlausibility 20

FullIgnorance 21

FullCertainty 22

PartialCertaintyandPartialIgnorance 24

LogarithmicPooling 25

Endnotes 27

Introduction

CenterforSecurityandEmergingTechnology|4

Experts,policymakers,andcitizensaroundtheworlddebatetheexistentialrisksofAI

withastonishinglylittleagreement

.1

OnestudyfoundthatestimatesforthelikelihoodofAI-inducedexistentialcatastrophesdifferedby250timesbetweenagroupof

“skeptics”anda“concerned”group(0.001and0.25,respectively)

.2

Evenwithingroups,suchasamongAIexperts,thevariationcanbejarring

.3

Thisreportaimstofamiliarizeanalystsanddecision-makerswiththetoolsand

vocabularytohandlethisuncertaintymoreexplicitly.Italsosuggeststhatsomeofthisdisagreementmaybeduetoimpreciseterminologyratherthanwhollyfrom

substantivedifferencesinviews.ThisbriefaimstoprovideconceptualclaritytoresolvedisagreementswherediscussantstalkpasteachotherindescribingtheirexpectationsaboutthefutureofAIrisk.Thatclaritymayhelpfindcommongroundandperhaps

helptopinpointthesourcesofthedisagreementsoridentifythedatathatwouldbeneededtoresolvethem.

CenterforSecurityandEmergingTechnology|5

ProbabilityIsNottheOnlyOption

ExistentialAIriskquestionsareusuallyphrasedalongthelinesof,whatisyour

estimateforp(doom)(i.e.,probabilityofAI-inducedapocalypse),orwhatdoyouthinkaretheoddsofamajorAIcatastropheinthenextfiveyears?Settingasidethe

ambiguityabouthowmuchdestructionisrequiredtobeexistentialorcatastrophic,thedeeperproblemisthatprobabilityispoorlysuitedtoansweringthesetypesof

questions.Theupsideisthatthereareadditionaloptions,someofwhicharebasedontermsthatarealreadyfamiliartobothexpertsandlaypeopleeveniftheprecise

quantitativedefinitionsarenot.

Thissectiondiscusseswhyprobabilityisill-suitedtothetopicofexistentialrisksfromAI.Thenextsectionsintroducealternativestoprobability—namely,Beliefand

Plausibility—andexplaintheiradvantagesforhandlingvarioustypesofuncertainty.Thefinalsectionillustrateshowtocombineexpertassessmentsofriskusingtheseconceptstobetterunderstandwheredisagreementsinriskassessmentstrulylie.

AleatoricandEpistemicUncertainty

Famously,thereareknownunknownsandunknownunknowns

.4

Moretechnically,riskanalystsseparateuncertaintyintoaleatoricandepistemiccategories.Aleatoric

uncertaintydealswithvariabilityorindeterminacythatnaturallyoccursinasystem.

Epistemicuncertaintydealswithalackofknowledge

.5

Putanotherway,thereis

randomnessandthereisignorance

.6

Aleatoricuncertaintyisnotknowingwhichfaceofthediewillcomeup.Epistemicuncertaintyisnotknowingwhatiswrittenonthose

faces.Probabilityisexcellentforcalculatingwhichfacewillcomeup,butitisnotidealforestimatingwhatwillbewrittenonit.Withepistemicuncertainty,itmightnotevenbeclearwhatastatedprobabilitymeansorhowtocalculateit.

Asaresult,probabilityisoftenaninappropriateframingofAIriskthatcanskew

perspectivesinwaysthatharmfullydetractfromeffectivediscourseanddecision-

making.Inwritingaboutriskingeneral(notspecificallyaboutAI),TerjeAven,the

formerPresidentoftheSocietyforRiskAnalysis,saidthat“someofthecurrent

perspectivesaresimplymisguidingthedecision-makerinmanycases.”Hehasalsosaidthat“probabilityhastoberemovedfromthedefinitionofriskandthenatural

replacementisuncertainty.

”7

Imagineasix-sideddiewhereyouhaveonlyseenthreesides:onesidehasastar

etchedonit,twosidesareblank,andtheotherthreeareunknown.Youmaywantto

CenterforSecurityandEmergingTechnology|6

knowhowlikelyitisforastartocomeup.Orinstead,youmaywanttoknowhow

confidentlytobelievethatastarwillcomeup.Oryoumaywanttoknowhow

plausibleitisthatastarcouldcomeup.Thoseappeartobesimilarquestionsbuttheyresultindifferentanswersand,inapolicycontext,havedifferentimplications.

Youcanhave1/6(~17%)confidencethatastarwillcomeuponthenextrollbecauseyouknowthatthereisatleastonestar.Thatdoesnotmakeyou83%confidentthatastarwillnotcomeup,becauseseveralmoresidescouldalsohavestars.Butknowingthattwosidesdonothaveastarisusefulinformation.Itmakesyou4/6(~67%)

confidentthatastarcouldplausiblycomeup.Thatwouldrequireallthreeofthe

unknownsidestohavestars,butthatisplausible.Toestimatelikelihood,youcouldusetheinformationthatyoudohave.Youcouldsaythat,becausetherearestarsonone-thirdofthesidesyouhaveseen,thereisa1/3(~33%)chanceofrollingastar.

Theseanswersaredifferentbecausethereisanunderlyingmixofaleatoricand

epistemicuncertainty.Therollofthedieisrandom,butyouarealsoignorantabout

whatiswrittenonthreeofthefacesthatcouldcomeup.Mostreal-worlduncertaintiesareamixofrandomnessandignorance.Incasesthatarewell-understoodorthathaveplentyofevidencetodrawfrom,randomnesscandominate.Asanexample,cyber

intrusionsarecommonenoughtocollectstatisticsandhavecausalchainsofeventsthatcanbedelineatedandreasonedabout.Asaresult,thetechniquesforhandlingaleatoricuncertaintyareoftenappropriateeventhoughanticipatingcyberattacksstillinvolvesplentyofignorancefrommanysources,includingattackermotivations,novelattackmethods,orthecomplexityofnetworkeffects.

InAI,wherefuturetechnologyinteractswithcomplexsocial,economic,orgeopoliticalsystems,thereislittledataandmuchispoorlyunderstood.Thatisespeciallytrueofnewexistentialrisks,suchasfromAI,thatdonothaveahistoryofrepeatedeventstodrawfrom.Ignoranceisthedominantformofuncertainty,notrandomness.For

example,expertsmaybeignorantaboutwhenanAImodelwouldengageindeceptivebehaviorandhowsocietywouldrespond,despitehavingevidencesuggestingthatthemodelsrandomlymaintaintheirdeceptivebehavior85%ofthetime

.8

Asanother

example,ananalysismayshowthatAImodelsadvocatefortacticalnuclearusein

95%ofsimulations

.9

Butexpertsremainignorantabouthowconflictsandcriseswilldevelop,howdifferenttoday’scomplexmultipolarworldisfromthatstudy’sbipolardynamicscirca1958–1962,andhowmuchhumanswilldefertoAIsuggestionsfor

existentialdecision-making.Studiessuchasthesetwoarehelpingtoshiftuncertaintyfromepistemictoaleatoricbutthelion’sshareoftheworkremainstobedone.

CenterforSecurityandEmergingTechnology|7

Uncertaintyaboutexistentialrisks,especiallyfromAI,isgoingtoremainprimarilyepistemic,notaleatoric,forsometime.

AlternativestoProbability

Fortunately,riskanalysisanditstoolboxhasadvancedsignificantlysincedeMoivreintroducedriskasprobabilityandconsequencein1711

.10

Arecentreviewofrelevantriskliteraturenotedthat“itisdifficulttodealwithepistemicuncertaintyeffectivelyonlythroughprobabilitytheory.Therefore,aseriesofuncertaintytheorieshavebeendevelopedtocomplementprobabilitytheory,whichincludeevidencetheory,fuzzy

sets,possibilitytheory,convexmodels,probabilitybox,etc.

”11

Theseconceptsarenotjustobscuremath.Severalhavetheimportantbenefitof

alreadybeingpartofthecommonvernacular.Thatfamiliaritymeansthatthese

conceptscanbeimmediatelyandseamlesslyincorporatedinAIriskdiscussions.Themaththenallowsspecialistsandpractitionerstorefinethoseconcepts,makingthediscussionsprogressivelymorepreciseandnuancedwhileshrinkingtheboundsofuncertainty.

CenterforSecurityandEmergingTechnology|8

Belief,Plausibility,andProbability

ThissectionfocusesonthefamiliartermsBeliefandPlausibility,andtheirrelationtoprobability

.12

Itprovidesinformaldefinitionsbeforedescribinghowanalystscanuseevidencetorefinethem.

Belief

Beliefisthestrengthoftheevidenceinfavorofaproposition.Thatisdifferentfromtheuseof“belief”inareligiousorfaith-basedsense.Inthistechnicalcontext,Beliefisthedegreeofcertaintyinastatement,basedonevidenceorreason.Notably,thatevidencemaybesubjective,asinthecaseofexpertopinions.

Plausibility

Plausibilityistheabsenceofevidencerefutingaproposition.AfteryouremovethedegreeofBeliefthatapropositionisnottrue,itsPlausibilityremains.

Probability

ItistemptingtothinkofBeliefandPlausibilityaslowerandupperboundson

probability,butthatisnotalwaysappropriate

.13

Combiningconflictingevidencefrommultiplesources,suchasfromdissentingexpertjudgments,canbreakthesimple

interpretationofBeliefandPlausibilityasprobabilitybounds.BeliefandPlausibility

areadeptathandlingdistinctionsbetweenconflictingassessmentsbutinwaysthatdonotalwaysadheretoprobabilitytheory.Dependingontheamountofconflictamongexpertsandtheamountofignorance,BeliefandPlausibilitycanbeboundsfor

probability.Belief,Plausibility,andprobabilitycanevenallreducetothesamenumber.ThenextsectiondescribeshowBeliefandPlausibilityhandleconflictingevidenceandhowtheyrelatetoprobability.

CenterforSecurityandEmergingTechnology|9

StructureofEvidence

Thissectionoutlinesseveraldifferentwaystoclassifyuncertaintyandtheseveraldifferenttechniquesforhandlingdifferenttypesofuncertainty.Understandingtheuncertaintyisimportantformakingreasoneddecisionsaboutit.

Indirect,Subjective,andConflicting

BeliefandPlausibilityaremostusefulwhentheevidenceisnotrelateddirectlytothequestionofinterest.Themostfamousexamplehastwowitnessesinacourtroom

.14

Eachmakesconflictingstatementsandyouneedtodeterminethetruth.Youhavenoevidenceaboutthecrimeitself.Instead,youhaveevidence(subjectiveassessments)aboutthereliabilityofthewitnesses.

ImaginethatMrs.Peacockisawitnesswhois80%reliableandthatsheconfidentlyaccusesColonelMustardandhiscandlestick.Thatgivesyou80%BeliefthatColonelMustardisthekiller,butitdoesnotmakeyou20%certainofhisinnocenceasitwouldinbasicprobabilitytheorywhereprobabilitiesmustsumtoone.Theunreliable20%providesnoadditionalinformation.AlthoughtheprobabilityofColonelMustard’sguiltisactuallyhigherthan80%,thereisnobasisfordetermininghowmuchhigher.BeliefandPlausibilitydonotrequireanyadditionalassertions.

SupposenowthatMissScarlettis75%reliableandconfidentlyaccusesProfessorPlumandhiswrench.Whencombiningtheirconflictingtestimonies,thejurymust

recognizethatitisnotpossibleforbothwitnessestobereliable.ThetotalevidenceagainstColonelMustardisthe80%thatMrs.Peacockisreliabletimesthe25%thatMissScarlettisnot.Thatisthesameanswerthatprobabilitytheorywouldgiveifwewronglyignoredthattheprobabilityforeachdefendantshouldbehigherthantheiraccuser’sreliability.BeliefandPlausibilityarebuilttoreasonthroughtheremainingcasesandtodirectlyconsidertheconflictamongaccusations.

IntheAIriskcontext,therearemanyprominent(i.e.,reliable)expertswhoassertthatvariousAIcatastrophesareimminent.Therearealsomanyprominentexpertswhoareskepticaloftheserisks.Bothgroupsmightclaimtobetrustworthy,buttheycannot

bothbecorrect.ThefollowingsectionswillillustratehowBeliefandPlausibilitycanbeusedtounderstand,assess,andcalculateuncertaintiesinthecontextofAIrisk.

CenterforSecurityandEmergingTechnology|10

AssertingProbabilities

Whenexpertsprovidesubjectiveprobabilities,theyareinternallyweighingthe

evidenceinfavorofapropositionandtheevidenceagainstit.Theirprobabilityreflectsabalancebetweenthetwo.Sotheiranswertothequestion“WhatistheprobabilityofanAIcatastrophe?”islikelytobehigherthantheiranswerto“HowcertainareyouthatanAIcatastrophewilloccur?”

Forthedieofstarsdescribedearlier,theevidencewas17%infavorofstars,buttheprobabilitywas33%.Althoughtherewasonlyevidenceofonestaramongthesix

sides,areasonableprobabilityestimatewouldalsoincludethechancethatoneor

moreoftheunknownsidescouldhaveastar.Usingasingleprobabilityconflatesthequestionsoflikelihood,certainty,andplausibility.Itisusefultoseparatethoseideasandtobemoreexplicitabouttheuncertainties.

InonestudyofAI-inducedextinctionbytheyear2100,themediansuperforecaster’slikelihoodwas0.0038

.15

GiventhelargeepistemicuncertaintyaboutexistentialorcatastrophicAIrisks,theirconfidenceincatastrophewaspresumablymuchlower.Itwouldbeusefultoknowhowmuchlower.

ConfidenceTriplets

Ratherthanusingasingleprobabilitytorepresentuncertainty,AIriskanalystscan

createatripletofconfidencethatincludes:certaintyincatastrophe(C),certaintyinnocatastrophe(N),andignorance(I)thatalladduptoone.Thatassessmentcanbe

writtenas:[C,N,I].Forthedieofstarsdescribedearlier,thetripletwouldbe[1/6,2/6,3/6].

Expertscouldbeaskedtoprovidethattripletdirectlyoranalystscantrytocreateit

fromanexpert’sstatementofprobability,butcreatingitfromasingleprobability

introduceschallenges.Therearetwobookendsonhowtodothatconversion.Let’ssaytheexpertis20%certainofcatastrophe.Atoneend,theanalystcanpresumethattheexpertispartlycertainaboutcatastropheandisignorantabouttherest;theirtriplet

wouldbe[0.2,0,0.8].Alternatively,theanalystcanpresumethattheexpertisimplyingcertaintythatnocatastrophewilloccur,foratripletof[0.2,0.8,0].

Eveniftheexpertpresumestohavenoignorance,ananalystcouldchoosetoadjusttheexpertassessmenttoreflectthatexpert’spartialknowledgeoftheissue.The

analystcouldattributea30%reliabilitytotheexpertandadjustthetripleof[0.2,0.8,

CenterforSecurityandEmergingTechnology|11

0]to[0.2x0.3,0.8x0.3,1-0.3]=[0.06,0.24,0.7].Theanalystmayadjustdownward

stillfurtheriftheysuspectthattheexpertisnotassertingcertaintyaboutcatastropheatall,butisratherassertingalikelihood.Inthesamewaythattheconfidenceinrollingstarswasonly17%whentheprobabilitywas33%,convertingprobabilityestimatesforAIrisktocertaintyshouldleadtolowernumbers.

ConfidenceTripletsfromAIRiskProbabilities

Revisitingthestudyfromtheintroduction,the“concerned”group’sprobabilityofAI-inducedextinctionby2100of0.25couldcorrespondtomanypossibletriplets

.*

For

example,bothofthefollowingtripletsgiveaprobabilityof0.25:[0.25,0.75,0]or

[0.001,0.003,0.996].Theformerwouldimplythattheexpertbelievesthecatastropheismostlyimplausible(Plausibility=0.25).Inthatcase,withnoignorance,allthreeofBelief,Plausibility,andprobabilityare0.25.Inthelattertriplet,withhighignorance,

theexpertisclaimingthatcatastropheisalmostcertainlyplausible(0.997),butthattheydonothavemuchevidencetosuggestthatitwillhappen(0.001).Theseare

differentassertionstoapolicymaker.

The0.25estimateisparticularlyinterestingbecauseitisaveryhighnumberforan

existentialrisk,butitisnotespeciallyhighforPlausibilityofanassertionwithsomuchuncertainty.GettingahigherPlausibilitywhilekeepingprobabilityat0.25requires

Beliefincatastrophetobelow.Perhapstheexpertsdomeanthatthereislittle

evidencetosupportclaimsofcatastrophebutthattheirignoranceishigh,asin[0.001,0.003,0.996].Orperhapstheymeanthatcatastropheisbothprobableand

implausibleasin[0.25,0.75,0].Orperhapstheyarenotreallyexpressingprobabilityatall.PerhapstheymeanforbothBeliefandPlausibilitytobehigh,asin[0.25,0,

0.75],buttheyusethevocabularyofprobabilitywhentheymeantorefertoBelief

.†

Determininganexpert’sconfidencetripletcanbedonewithtwoquestions,andto

convertfromprobabilityrequiresoneadditionalquestionandanassumption.Therearethreevariablestodetermine:C,N,andI.Theyhavetosumtoone,soknowingtwoissufficienttocalculatethethird.Forexample,givenaprobabilityestimatethatisan

*ProbabilityiscalculatedfromthevaluesCandNaloneanddoesnotdependonignorance(i.e.,

p=C/(C+N)).FromaBayes’perspective,ignoranceiseffectivelyauniformprior(i.e.,0.5).

†ThisiscommonindiscourseandisabenefitofBeliefandPlausibilitythatevenitscriticsappreciate.

Seepage383ofJudeaPearl,“ReasoningWithBeliefFunctions:AnAnalysisofCompatibility,”

InternationalJournalofApproximateReasoning,Volume4,1990,

/10.1016/0888-

613X(90)90013-R

.

CenterforSecurityandEmergingTechnology|12

honestprobability,ratherthanbeingaBeliefestimatereferredtoasprobability,thenjustonemorequestionisneededtocalculatethefulltriplet.ThatquestioncouldaskforPlausibility:“HowplausibleisAIextinctionby2100?”OritcouldaskforIgnorance:“Whatfractionoftheinformationneededtomakethisassessmentdoestheexpert

know?”

SourcesofDisagreements

Focusingonsourcesofdisagreement,itisalsoatleastmathematicallypossiblefortheskepticsandconcernedgroupsfromthatstudytohavetheexactsameBeliefabout

catastrophedespiteprobabilitiesthatare250timesdifferent.Theskepticscouldhaveatripletof[0.0005,0.4995,0.5]togettheir0.001likelihood,andtheconcernedcouldhaveatripletof[0.0005,0.0015,0.998]fortheir0.25likelihood.Withthosetriplets,theconcernedgroupwouldviewcatastropheasbothmorelikelyandmoreplausibledespiteequivalentlevelsofBelief.Inthatcase,thedisagreementwouldnotbeaboutthecaseforcatastrophe.Thedisagreementwouldfocusonthecaseagainst

catastropheandhowmuchtheexpertsdonotknow(i.e.,ignorance).

Withouthavingaskedtheexpertsmoredirectlyabouttheiruncertainties,ananalystcannotknowwhichtheymeanttoexpress,butthereissomequalitativeinformationaboutthesetwoparticulargroupsfromacorrespondingstudy

.16

Membersofthe

concernedgroupwereswayedbytheprecedentthathigherintelligenceshad

previouslycausedextinctionoflesserintelligences,whereastheskepticsfeltthat

sweepingchangestendtobeslow.Basedonthosearguments,thetwogroupswouldhavedifferentlevelsofBelief.Theconcernedgroupwould,unsurprisingly,havehighvaluesforC,whiletheskepticswouldhavehighvaluesforN.

Moreinterestingly,theconcernedgroupalsowas“morewillingtoplaceweighton

theoreticalargumentswithmultiplestepsoflogic,whiletheskepticstendedtodoubttheusefulnessofsucharguments.”Thatimpliesthatthetwogroupshandled

ignorancedifferently,buttherearetoomanywaystointerpretthatqualitative

statementwithouthavingaskedanadditionalquestionthatcouldbeusedtoquantifytheirignorance,Belief,orPlausibility.WhilewehaveyettoseeBeliefandPlausibilityexploredforAIrisk,theauthorsoftheAIriskstudiesdescribedabovehavemore

recentlystatedthat,goingforward,moreweightshouldbegiventoalternativeframeworks,includingthetechniquesdescribedinthisbrief

.17

CenterforSecurityandEmergingTechnology|13

CombiningEvidence

Therearetwodifferentobjectivesforcombiningexpertassessments.Oneistoadd

newinformationwhenanewindependentexpertisadded.Theotheristofind

consensusamongexpertswhohavedifferingassessmentsofthesameevidence.Thissectionwillstartbydescribingtechniquesforindependentexperts,thenfinishwithtechniquesforaggregatingassessmentsofthesameevidence.

Let’sconsidertwohypotheticalexperts,AandB,whoare15%and20%confidentincatastrophe,respectively.Thosenumbersareveryhighbutmakeforsimple

illustration.Thefirstquestioniswhattodowiththeremaining85%and80%.Shoulditbeassignedtoignoranceordoesitimplythattheexpertsarecertainthatno

catastrophewilloccur?

Thissectionwillconsiderthreedifferentwaystoallocatetheremainder:1)Allocatetheremaining85%and80%fullytoignorance,2)allocatethoseremainingportionsfullytocertaintythatnocatastrophewilloccur,and3)haveapartialallocationto

certaintyandignorance.Thesubsectionsbelowdiscusstheintuitionbehindthose

assignmentsandtheresultingBeliefsandPlausibilities.ThecalculationsthemselvesareinAppendixA.

FullIgnorance

Assigningtheremainingportionsalltoignorancegivesthetriplets[0.15,0,0.85]and[0.2,0,0.8].Awaytoreadthesetripletsisthatbothexpertshavenothingtoconvincethemagainstcatastrophe,buttheyadmitthatthereismuchtheydonotknow.As

calculatedinAppendixA,theircombinedBeliefandPlausibilityarebothquitehigh:

0.32and1,respectively.Thosenumbersarehighbecausetheyeachhavesomereasontosuspectcatastropheandnoreasoningagainstit.

FullCertainty

Attheotherextreme,theexpertsknoweverythingtheyneedtomakethe

assessments,buttheworldisstillrandomandcouldleadtoeithercatastropheornocatastrophe.Thatgivesthetriplets[0.15,0.85,0]and[0.2,0.8,0].Inthiscase,thereisconflictamongtheassessments.Handlingthisconflictisthestrengthofcalculating

BeliefandPlausibility,buttherearedifferenttechniquesforit

.18

AppendixAshowsthecalculationsfortwopopulartechniques:Dempster’sruleandYager’srule.

Bothrulesfirstneedtocalculatetheconflict,whichisthetotalwhereoneexperthassomecertaintyinanoutcomethattheotherhassomecertaintycannotoccur.Thefirstexpertassesses0.15forcatastrophewhenthesecondassesses0.8against,whicharemultipliedtoget0.12.Andt