人工智能安全中的具体问题 Concrete Problems in AI Safety

1

ConcreteProblemsinAISafety

DarioAmodei*GoogleBrain

ChrisOlah*GoogleBrain

JacobSteinhardtStanfordUniversity

PaulChristianoUCBerkeley

arXiv:1606.06565v2[cs.AI]25Jul2016

DanManéGoogleBrain

JohnSchulmanOpenAI

Abstract

Rapidprogressinmachinelearningandartificialintelligence(AI)hasbroughtincreasingatten-tiontothepotentialimpactsofAItechnologiesonsociety.Inthispaperwediscussonesuchpotentialimpact:theproblemofaccidentsinmachinelearningsystems,definedasunintendedandharmfulbehaviorthatmayemergefrompoordesignofreal-worldAIsystems.Wepresentalistoffivepracticalresearchproblemsrelatedtoaccidentrisk,categorizedaccordingtowhethertheproblemoriginatesfromhavingthewrongobjectivefunction(“avoidingsideeffects”and“avoidingrewardhacking”),anobjectivefunctionthatistooexpensivetoevaluatefrequently(“scalablesupervision”),orundesirablebehaviorduringthelearningprocess("safeexploration”

and“distributionalshift”).Wereviewpreviousworkintheseareasaswellassuggestingre-searchdirectionswithafocusonrelevancetocutting-edgeAIsystems.Finally,weconsiderthehigh-levelquestionofhowtothinkmostproductivelyaboutthesafetyofforward-lookingapplicationsofAI.

1Introduction

Thelastfewyearshaveseenrapidprogressonlong-standing,difficultproblemsinmachinelearningandartificialintelligence(AI),inareasasdiverseascomputervision|32,videogameplaying|102,autonomousvehicles|86,andGo|140].TheseadvanceshavebroughtexcitementaboutthepositivepotentialforAItotransformmedicine|126,science|59,andtransportation|36,alongwithconcernsabouttheprivacy|76,security|115|,fairness3,economic32,andmilitaryl16implicationsofautonomoussystems,aswellasconcernsaboutthelonger-termimplicationsofpowerfulAI27167.

TheauthorsbelievethatAItechnologiesarelikelytobeoverwhelminglybeneficialforhumanity,butwealsobelievethatitisworthgivingseriousthoughttopotentialchallengesandrisks.Westronglysupportworkonprivacy,security,fairness,economics,andpolicy,butinthisdocumentwediscussanotherclassofproblemwhichwebelieveisalsorelevanttothesocietalimpactsofAI:theproblemofaccidentsinmachinelearningsystems.Wedefineaccidentsasunintendedandharmfulbehaviorthatmayemergefrommachinelearningsystemswhenwespecifythewrongobjectivefunction,are

*Theseauthorscontributedequally.

2

notcarefulaboutthelearningprocess,orcommitothermachinelearning-relatedimplementation

errors.

Thereisalargeanddiverseliteratureinthemachinelearningcommunityonissuesrelatedtoaccidents,includingrobustness,risk-sensitivity,andsafeexploration;wereviewtheseindetailbelow.However,asmachinelearningsystemsaredeployedinincreasinglylarge-scale,autonomous,open-domainsituations,itisworthreflectingonthescalabilityofsuchapproachesandunderstandingwhatchallengesremaintoreducingaccidentriskinmodernmachinelearningsystems.Overall,webelievetherearemanyconcreteopentechnicalproblemsrelatingtoaccidentpreventioninmachinelearningsystems.

Therehasbeenagreatdealofpublicdiscussionaroundaccidents.Todatemuchofthisdiscussionhashighlightedextremescenariossuchastheriskofmisspecifiedobjectivefunctionsinsuperintelligentagents27.However,inouropiniononeneednotinvoketheseextremescenariostoproductivelydiscussaccidents,andinfactdoingsocanleadtounnecessarilyspeculativediscussionsthatlackprecision,asnotedbysomecritics3885.Webelieveitisusuallymostproductivetoframeaccidentriskintermsofpractical(thoughoftenquitegeneral)issueswithmodernMLtechniques.AsAIcapabilitiesadvanceandasAIsystemstakeonincreasinglyimportantsocietalfunctions,weexpectthefundamentalchallengesdiscussedinthispapertobecomeincreasinglyimportant.ThemoresuccessfullytheAIandmachinelearningcommunitiesareabletoanticipateandunderstandthesefundamentaltechnicalchallenges,themoresuccessfulwewillultimatelybeindevelopingincreasinglyuseful,relevant,andimportantAIsystems.

Ourgoalinthisdocumentstohighlightafewconcretesafetyproblemsthatarereadyforex-perimentationtodayandrelevanttothecuttingedgeofAIsystems,aswellasreviewingexistingliteratureontheseproblems.InSection2weframemitigatingaccidentrisk(oftenreferredtoas“AIsafety”inpublicdiscussions)intermsofclassicmethodsinmachinelearning,suchassupervisedclassificationandreinforcementlearning.Weexplainwhywefeelthatrecentdirectionsinmachinelearning,suchasthetrendtowarddeepreinforcementlearningandagentsactinginbroaderenviron-ments,suggestanincreasingrelevanceforresearcharoundaccidents.InSections3-7,weexplorefiveconcreteproblemsinAIsafety.Eachsectionisaccompaniedbyproposalsforrelevantexperiments.Section8discussesrelatedefforts,andSection9concludes.

2OverviewofResearchProblems

Verybroadly,anaccidentcanbedescribedasasituationwhereahumandesignerhadinmindacertain(perhapsinformallyspecified)objectiveortask,butthesystemthatwasdesignedanddeployedforthattaskproducedharmfulandunexpectedresults..Thisissuearisesinalmostanyengineeringdiscipline,butmaybeparticularlyimportanttoaddresswhenbuildingAIsystems|146.Wecancategorizesafetyproblemsaccordingtowhereintheprocessthingswentwrong.

First,thedesignermayhavespecifiedthewrongformalobjectivefunction,suchthatmaximizingthatobjectivefunctionleadstoharmfulresults,eveninthelimitofperfectlearningandinfinitedata.Negativesideeffects(Section3andrewardhacking(Section4describetwobroadmechanismsthatmakeiteasytoproducewrongobjectivefunctions.In“negativesideeffects”,thedesignerspecifiesanobjectivefunctionthatfocusesonaccomplishingsomespecifictaskintheenvironment,butignoresotheraspectsofthe(potentiallyverylarge)environment,andthusimplicitlyexpressesindifferenceoverenvironmentalvariablesthatmightactuallybeharmfultochange.In“rewardhacking”,theobjectivefunctionthatthedesignerwritesdownadmitsofsomeclever“easy”solutionthatformallymaximizesitbutpervertsthespiritofthedesigner'sintent(i.e.theobjectivefunctioncanbe“gamed”),ageneralizationofthewireheadingproblem.

3

Second,thedesignermayknowthecorrectobjectivefunction,oratleasthaveamethodofevaluatingit(forexampleexplicitlyconsultingahumanonagivensituation),butitistooexpensivetodosofrequently,leadingtopossibleharmfulbehaviorcausedbybadextrapolationsfromlimitedsamples.“Scalableoversight”(Section5)discussesideasforhowtoensuresafebehaviorevengivenlimitedaccesstothetrueobjectivefunction.

Third,thedesignermayhavespecifiedthecorrectformalobjective,suchthatwewouldgetthecorrectbehaviorwerethesystemtohaveperfectbeliefs,butsomethingbadoccursduetomakingdecisionsfrominsufficientorpoorlycuratedtrainingdataoraninsufficientlyexpressivemodel.“Safeexploration”(Section⑥discusseshowtoensurethatexploratoryactionsinRLagentsdon'tleadtonegativeorirrecoverableconsequencesthatoutweighthelong-termvalueofexploration."Robustnesstodistributionalshift"(Section7discusseshowtoavoidhavingMLsystemsmakebaddecisions(particularlysilentandunpredictablebaddecisions)whengiveninputsthatarepotentiallyverydifferentthanwhatwasseenduringtraining.

Forconcreteness,wewillillustratemanyoftheaccidentriskswithreferencetoafictionalrobotwhosejobistocleanupmessesinanofficeusingcommoncleaningtools.Wereturntotheexampleofthecleaningrobotthroughoutthedocument,butherewebeginbyillustratinghowitcouldbehaveundesirablyifitsdesignersfallpreytoeachofthepossiblefailuremodes:

●AvoidingNegativeSideEffects:Howcanweensurethatourcleaningrobotwillnotdisturbtheenvironmentinnegativewayswhilepursuingitsgoals,e.g.byknockingoveravasebecauseitcancleanfasterbydoingso?Canwedothiswithoutmanuallyspecifyingeverythingtherobotshouldnotdisturb?

●AvoidingRewardHacking:Howcanweensurethatthecleaningrobotwon'tgameitsrewardfunction?Forexample,ifwerewardtherobotforachievinganenvironmentfreeofmesses,itmightdisableitsvisionsothatitwon'tfindanymesses,orcoverovermesseswithmaterialsitcan'tseethrough,orsimplyhidewhenhumansarearoundsotheycan'ttellitaboutnewtypesofmesses.

·ScalableOversight:Howcanweefficientlyensurethatthecleaningrobotrespectsaspectsoftheobjectivethataretooexpensivetobefrequentlyevaluatedduringtraining?Forinstance,itshouldthrowoutthingsthatareunlikelytobelongtoanyone,butputasidethingsthatmightbelongtosomeone(itshouldhandlestraycandywrappersdifferentlyfromstraycellphones).Askingthehumansinvolvedwhethertheylostanythingcanserveasacheckonthis,butthischeckmighthavetoberelativelyinfrequent—cantherobotfindawaytodotherightthingdespitelimitedinformation?

·SafeExploration:Howdoweensurethatthecleaningrobotdoesn'tmakeexploratorymoveswithverybadrepercussions?Forexample,therobotshouldexperimentwithmoppingstrategies,butputtingawetmopinanelectricaloutletisaverybadidea.

●RobustnesstoDistributionalShift:Howdoweensurethatthecleaningrobotrecognizes,andbehavesrobustly,wheninanenvironmentdifferentfromitstrainingenvironment?Forexample,strategiesitlearnedforcleaninganofficemightbedangerousonafactoryworkfloor.

Thereareseveraltrendswhichwebelievepointtowardsanincreasingneedtoaddressthese(andother)safetyproblems.Firstistheincreasingpromiseofreinforcementlearning(RL),whichal-lowsagentstohaveahighlyintertwinedinteractionwiththeirenvironment.SomeofourresearchproblemsonlymakesenseinthecontextofRL,andothers(likedistributionalshiftandscalableoversight)gainaddedcomplexityinanRLsetting.Secondisthetrendtowardmorecomplexagentsandenvironments.“Sideeffects”aremuchmorelikelytooccurinacomplexenvironment,andanagentmayneedtobequitesophisticatedtohackitsrewardfunctioninadangerousway.Thismayexplainwhytheseproblemshavereceivedsolittlestudyinthepast,whilealsosuggestingtheir

4

importanceinthefuture.ThirdisthegeneraltrendtowardsincreasingautonomyinAIsystems.Systemsthatsimplyoutputarecommendationtohumanusers,suchasspeechsystems,typicallyhaverelativelylimitedpotentialtocauseharm.Bycontrast,systemsthatexertdirectcontrolovertheworld,suchasmachinescontrollingindustrialprocesses,cancauseharmsinawaythathumanscannotnecessarilycorrectoroversee.

Whilesafetyproblemscanexistwithoutanyofthesethreetrends,weconsidereachtrendtobeapossibleamplifieronsuchchallenges.Together,webelievethesetrendssuggestanincreasingroleforresearchonaccidents.

Whendiscussingtheproblemsintheremainderofthisdocument,wewillfocusforconcretenessoneitherRLagentsorsupervisedlearningsystems.ThesearenottheonlypossibleparadigmsforAIorMLsystems,butwebelievetheyaresufficienttoillustratetheissueswehaveinmind,andthatsimilarissuesarelikelytoariseforotherkindsofAIsystems.

Finally,thefocusofourdiscussionwilldiffersomewhatfromsectiontosection.Whendiscussingtheproblemsthatariseaspartofthelearningprocess(distributionalshiftandsafeexploration),wherethereisasizablebodyofpriorwork,wedevotesubstantialattentiontoreviewingthispriorwork,althoughwealsosuggestopenproblemswithaparticularfocusonemergingMLsystems.Whendiscussingtheproblemsthatarisefromhavingthewrongobjectivefunction(rewardhackingandsideeffects,andtoalesserextentscalablesupervision),wherelesspriorworkexists,ouraimismoreexploratory—weseektomoreclearlydefinetheproblemandsuggestpossiblebroadavenuesofattack,withtheunderstandingthattheseavenuesarepreliminaryideasthathavenotbeenfullyfleshedout.Ofcourse,westillreviewpriorworkintheseareas,andwedrawattentiontorelevantadjacentareasofresearchwheneverpossible.

3AvoidingNegativeSideEffects

SupposeadesignerwantsanRLagent(forexampleourcleaningrobot)toachievesomegoal,likemovingaboxfromonesideofaroomtotheother.Sometimesthemosteffectivewaytoachievethegoalinvolvesdoingsomethingunrelatedanddestructivetotherestoftheenvironment,likeknockingoveravaseofwaterthatisinitspath.Iftheagentisgivenrewardonlyformovingthebox,itwillprobablyknockoverthevase.

Ifwe'reworriedinadvanceaboutthevase,wecanalwaysgivetheagentnegativerewardforknockingitover.Butwhatiftherearemanydifferentkindsof“vase”—manydisruptivethingstheagentcoulddototheenvironment,likeshortingoutanelectricalsocketordamagingthewallsoftheroom?Itmaynotbefeasibletoidentifyandpenalizeeverypossibledisruption.

Morebroadly,foranagentoperatinginalarge,multifacetedenvironment,anobjectivefunctionthatfocusesononlyoneaspectoftheenvironmentmayimplicitlyexpressindifferenceoverotheraspectsoftheenvironment!Anagentoptimizingthisobjectivefunctionmightthusengageinmajordisruptionsofthebroaderenvironmentifdoingsoprovidesevenatinyadvantageforthetaskathand.Putdifferently,objectivefunctionsthatformalize“performtaskX”mayfrequentlygiveundesiredresults,becausewhatthedesignerreallyshouldhaveformalizediscloserto“performtaskXsubjecttocommon-senseconstraintsontheenvironment,"orperhaps"performtaskXbutavoidsideeffectstotheextentpossible.”Furthermore,thereisreasontoexpectsideeffectstobenegativeonaverage,sincetheytendtodisruptthewiderenvironmentawayfromastatusquostatethatmayreflecthumanpreferences.Aversionofthisproblemhasbeendiscussedinformallyby3undertheheadingof“lowimpactagents.”

Intuitively,thisseemsrelatedtotheframeproblem,anobstacleinefficientspecificationforknowledgerepresen-tationraisedby95.

5

Aswiththeothersourcesofmis-specifiedobjectivefunctionsdiscussedlaterinthispaper,wecouldchoosetoviewsideeffectsasidiosyncratictoeachindividualtask—astheresponsibilityofeachindividualdesignertocaptureaspartofdesigningthecorrectobjectivefunction.However,sideeffectscanbeconceptuallyquitesimilarevenacrosshighlydiversetasks(knockingoverfurnitureisprobablybadforawidevarietyoftasks),soitseemsworthtryingtoattacktheproblemingenerality.Asuccessfulapproachmightbetransferableacrosstasks,andthushelptocounteractoneofthegeneralmechanismsthatproduceswrongobjectivefunctions.Wenowdiscussafewbroadapproachestoattackingthisproblem:

·DefineanImpactRegularizer:Ifwedon'twantsideeffects,itseemsnaturaltopenalize"changetotheenvironment.”Thisideawouldn'tbetostoptheagentfromeverhavinganimpact,butgiveitapreferenceforwaystoachieveitsgoalswithminimalsideeffects,ortogivetheagentalimited“budget”ofimpact.Thechallengeisthatweneedtoformalize"changetotheenvironment."

Averynaiveapproachwouldbetopenalizestatedistance,d(si,so),betweenthepresentstateSiandsomeinitialstateso.Unfortunately,suchanagentwouldn'tjustavoidchangingtheenvironment—itwillresistanyothersourceofchange,includingthenaturalevolutionoftheenvironmentandtheactionsofanyotheragents!

Aslightlymoresophisticatedapproachmightinvolvecomparingthefuturestateundertheagent'scurrentpolicy,tothefuturestate(ordistributionoverfuturestates)underahypothet-icalpolicyTnullwheretheagentactedverypassively(forinstance,wherearobotjuststoodinplaceanddidn'tmoveanyactuators).Thisattemptstofactoroutchangesthatoccurinthenaturalcourseoftheenvironment'sevolution,leavingonlychangesattributabletotheagent'sintervention.However,definingthebaselinepolicyTnullisn'tnecessarilystraightforward,sincesuddenlyceasingyourcourseofactionmaybeanythingbutpassive,asinthecaseofcarryingaheavybox.Thus,anotherapproachcouldbetoreplacethenullactionwithaknownsafe(e.g.lowsideeffect)butsuboptimalpolicy,andthenseektoimprovethepolicyfromthere,somewhatreminiscentofreachabilityanalysis|93.100orrobustpolicyimprovement⑦3111.

Theseapproachesmaybeverysensitivetotherepresentationofthestateandthemetricbeing

usedtocomputethedistance.Forexample,thechoiceofrepresentationanddistancemetriccoulddeterminewhetheraspinningfanisaconstantenvironmentoraconstantlychangingone.

●LearnanImpactRegularizer:Analternative,moreflexibleapproachistolearn(ratherthandefine)ageneralizedimpactregularizerviatrainingovermanytasks.Thiswouldbeaninstanceoftransferlearning.Ofcourse,wecouldattempttojustapplytransferlearningdirectlytothetasksthemselvesinsteadofworryingaboutsideeffects,butthepointisthatsideeffectsmaybemoresimilaracrosstasksthanthemaingoalis.Forinstance,bothapaintingrobotandacleaningrobotprobablywanttoavoidknockingoverfurniture,andevensomethingverydifferent,likeafactorycontrolrobot,willlikelywanttoavoidknockingoververysimilarobjects.Separatingthesideeffectcomponentfromthetaskcomponent,bytrainingthemwithseparateparameters,mightsubstantiallyspeedtransferlearningincaseswhereitmakessensetoretainonecomponentbutnottheother.Thiswouldbesimilartomodel-basedRLapproachesthatattempttotransferalearneddynamicsmodelbutnotthevalue-function|155,thenoveltybeingtheisolationofsideeffectsratherthanstatedynamicsasthetransferrablecomponent.Asanaddedadvantage,regularizersthatwereknownorcertifiedtoproducesafebehaviorononetaskmightbeeasiertoestablishassafeonothertasks.

·PenalizeInfluence:Inadditiontonotdoingthingsthathavesideeffects,wemightalsoprefertheagentnotgetintopositionswhereitcouldeasilydothingsthathavesideeffects,eventhoughthatmightbeconvenient.Forexample,wemightpreferourcleaningrobotnot

6

bringabucketofwaterintoaroomfullofsensitiveelectronics,evenifitneverintendstousethewaterinthatroom.

Thereareseveralinformation-theoreticmeasuresthatattempttocaptureanagent'spotentialforinfluenceoveritsenvironment,whichareoftenusedasintrinsicrewards.Perhapsthebest-knownsuchmeasureisempowerment|131,themaximumpossiblemutualinformationbetweentheagent'spotentialfutureactionsanditspotentialfuturestate(orequivalently,theShannoncapacityofthechannelbetweentheagent'sactionsandtheenvironment).Empowermentisoftenmaximized(ratherthanminimized)asasourceofintrinsicreward.Thiscancausetheagenttoexhibitinterestingbehaviorintheabsenceofanyexternalrewards,suchasavoidingwallsorpickingupkeys|103.Generally,empowerment-maximizingagentsputthemselvesinapositiontohavelargeinfluenceovertheenvironment.Forexample,anagentlockedinasmallroomthatcan'tgetoutwouldhavelowempowerment,whileanagentwithakeywouldhavehigherempowermentsinceitcanventureintoandaffecttheoutsideworldwithinafewtimesteps.Inthecurrentcontext,theideawouldbetopenalize(minimize)empowermentasaregularizationterm,inanattempttoreducepotentialimpact.

Thisideaaswrittenwouldnotquitework,becauseempowermentmeasuresprecisionofcontrolovertheenvironmentmorethantotalimpact.Ifanagentcanpressornotpressabuttontocutelectricalpowertoamillionhouses,thatonlycountsasonebitofempowerment(sincetheactionspacehasonlyonebit,itsmutualinformationwiththeenvironmentisatmostonebit),whileobviouslyhavingahugeimpact.Conversely,ifthere'ssomeoneintheenvironmentscribblingdowntheagent'sactions,thatcountsasmaximumempowermenteveniftheimpactislow.Furthermore,naivelypenalizingempowermentcanalsocreateperverseincentives,suchasdestroyingavaseinordertoremovetheoptiontobreakitinthefuture.

Despitetheseissues,theexampleofempowermentdoesshowthatsimplemeasures(evenpurelyinformation-theoreticones!)arecapableofcapturingverygeneralnotionsofinfluenceontheenvironment.Exploringvariantsofempowermentpenalizationthatmorepreciselycapturethenotionofavoidinginfluenceisapotentialchallengeforfutureresearch.

·Multi-AgentApproaches:Avoidingsideeffectscanbeseenasaproxyforthethingwereallycareabout:avoidingnegativeexternalities.Ifeveryonelikesasideeffect,there'snoneedtoavoidit.Whatwe'dreallyliketodoisunderstandalltheotheragents(includinghumans)andmakesureouractionsdon'tharmtheirinterests.

OneapproachtothisisCooperativeInverseReinforcementLearning|66,whereanagentandahumanworktogethertoachievethehuman'sgoals.Thisconceptcanbeappliedtosituationswherewewanttomakesureahumanisnotblockedbyanagentfromshuttingtheagentdownifitexhibitsundesiredbehavior|67(this“shutdown”issueisaninterestingprobleminitsownright,andisalsostudiedin113).Howeverwearestillalongwayawayfrompracticalsystemsthatcanbuildarichenoughmodeltoavoidundesiredsideeffectsinageneralsense.

Anotherideamightbea"rewardautoencoder”whichtriestoencourageakindof“goaltransparency”whereanexternalobservercaneasilyinferwhattheagentistryingtodo.Inparticular,theagent'sactionsareinterpretedasanencodingofitsrewardfunction,andwemightapplystandardautoencodingtechniquestoensurethatthiscandecodedaccurately.Actionsthathavelotsofsideeffectsmightbemoredifficulttodecodeuniquelytotheiroriginalgoal,creatingakindofimplicitregularizationthatpenalizessideeffects.

·RewardUncertainty:Wewanttoavoidunanticipatedsideeffectsbecausetheenvironmentisalreadyprettygoodaccordingtoourpreferences—arandomchangeismorelikelytobeverybadthanverygood.Ratherthangivinganagentasinglerewardfunction,itcouldbe

2ThankstoGregWayneforsuggestingthisidea.

7

uncertainabouttherewardfunction,withapriorprobabilitydistributionthatreflectsthepropertythatrandomchangesaremorelikelytobebadthangood.Thiscouldincentivizetheagenttoavoidhavingalargeeffectontheenvironment.Onechallengeisdefiningabaselinearoundwhichchangesarebeingconsidered.Forthis,onecouldpotentiallyuseaconservativebutreliablebaselinepolicy,similartotherobustpolicyimprovementandreachabilityanalysisapproachesdiscussedearlier|93.100.73111.

Theidealoutcomeoftheseapproachestolimitingsideeffectswouldbetopreventoratleastboundtheincidentalharmanagentcoulddototheenvironment.Goodapproachestosideeffectswouldcertainlynotbeareplacementforextensivetestingorforcarefulconsiderationbydesignersoftheindividualfailure