法律与人工智能研究所2026激进可选性不确定性下变革性人工智能治理研究报告

ChristophWinter&CharlieBullock

RadicalOptionality

GoverningTransformativeAIUnderUncertainty

CONTENTS

Introduction1

ITheChallengeofRegulatingTransformativeAI2

IIApproachestoRegulation4

IIIConcretePolicySuggestions11

IVObjections27

VConclusion34

VIEndnotes34

ChristophWinterisAssistantProfessorofLawandAIattheUniversityofCambridge.BothauthorsareaffiliatedwiththeInstituteforLaw&AI.

1

Introduction

Theprospectof“transformativeAI”appearstopresentpolicymakerswithadilemma:overregulationcouldstifleinnovationandforfeitthepotentialbenefitsofthetechnology,whileafailuretoregulateappropriatelycouldhavedisastrousimplicationsforpublicsafetyandnationalsecurity.

It’struethatsecurityandinnovationaresometimesintension.Somesafetymeasuresdoimposecostsoninnovation,andsomeformsofderegulationdocarrygenuinerisks.Butthereisalsoaclassofpoliciesthatwouldmeaningfullyincreasesafetywithoutimposingsignificantcostsoninnovation.Wearguethatgovernmentsshouldaggressivelyimplementthesepolicies;thisisthemainthrustofthegovernancestrategydiscussedinthisessay,whichwecall“radicaloptionality.”

Atitscore,radicaloptionalityisaboutpreservingdemocraticgovernments’abilitytomakegooddecisionsabouthowtogoverntransformativeAIsystemsascircumstancesevolve.Intheshortterm,thismeansavoidingoverregulationwhilerapidlybuildingtheinstitutions,informationchannelsandlegalauthoritiesneededtorespondcompetentlytoabroadrangeofscenarios.

Theargumentforfocusingonoptionalityissimple,and—ifyouacceptafewreasonableassumptions—compelling.Theseassumptionsare:

1.ThatthereisarealpossibilityoftransformativeAI(definedas“AIthatprecipitatesatransitioncomparableto(ormoresignificantthan)theagriculturalorindustrialrevolution”)beingdevelopedwithinthenexttenyears;

2.ThatprofounduncertaintyexistsastowhatcapabilitiestransformativeAIsystemswillpossess,whatbenefitsandriskstheywillgenerate,andwhatthebestwaysforsocietytocapturethebenefitswhilemitigatingtheriskswillbe;

3.Thatatransformativelyimpactfuldual-usetechnologywithsignificantnationalsecurityimplicationswillinevitablyrequiresomedegreeofgovernmentoversight;and

2

4.ThatbuildingtheinstitutionalcapacityrequiredtoeffectivelygoverntransformativeAIsystemswilltakeyears,andthatsocietythereforecannotaffordtowaituntiltransformativecapabilitieshaveactuallybeendeveloped.

Justifyingthefirstassumptionisbeyondthescopeofthispaper.Whether“AGI”or“superintelligence”or“powerfulAI”or“transformativeAI”willeverarrive,andwhen,arequestionsthathavebeendebatedextensivelyelsewhere.ButifyoubelievethattransformativeAIispossible,wehopetodemonstratethatthecaseforradicaleffortstopreserveoptionalityisoverwhelminglystrong.

TheChallengeofRegulatingTransformativeAI

Theimportanceofgettingtheregulatoryresponsetoatrulytransformativetechnologyrightisobvious.Thecomplexityoftheproblemmaynotbe.Scholarswhostudytheregulationofemergingtechnologieshavelongacknowledgedthedifficultiesposedbythe“pacingproblem.”Inbrief,technologicalprogressoftenoccursatsucharapidpacethatlaws,regulations,andthelegalsystemareunabletoadaptquicklyenoughtokeepup.Thismakesitdifficultforpolicymakerstoeffectivelygovernemergingtechnologies.IntheAIcontext,thisproblemiscompoundedbythefactthatAIsystemsareinsomewaysuniquelydifficulttounderstand,andmaypossesscapabilitiesthateventhepeoplewhocreatedthemareinitiallyunawareof.Capabilityprofilesarealsojagged:amodelmightperformatexpertlevelononetaskwhilefailingatmuchsimplertasksinanotherdomain,makingithardtoinferasystem’soverallcompetencefromanygivenbenchmark.Inotherwords,AIgovernanceinvolvesdecisionmakingunderextremeuncertaintyaboutthefuturecapabilitiesofthetechnology,thenatureandseverityofitsrisksitmightpose,andthebenefitsitmightoffer.

Bysomemeasures,AImodelperformancehasbeenimprovingexponentially,andsomeresearchersbelievethatthistrendwillcontinueinthecomingyears.Oneparticularlyexcitingandconcerningprospectisthepossibilityof

3

recursiveself-improvement.AsincreasinglycapableAIsystemsaredeveloped,perhapswithsuperhumanprogrammingand/orresearchabilities,thesesystemsmightfacilitatethedevelopmentofevenmorecapablesystems,whichmightfacilitatethedevelopmentofstillmorecapablesystems,andsoon.Anearlyversionofthisphenomenonisalreadyoccurring:engineersatsomefrontierlabshavereportedthatasignificantmajorityofthecodeusedtocreatenext-generationAIsystemsisbeingwrittenbycurrent-generationAIsystems.AndbecauseAIsystemswritecodefarmorequicklythanhumanprogrammers,recursiveself-improvementmayresultindramatic,andperhapsexponential,improvementsincapabilitiesintherelativelynearfuture.

Becausehumanbeingsarepsychologicallydisinclinedtoaccepttheimplicationsofexponentialgrowth,theexponentialprogressofAIcapabilitiesresearchfurthercompoundsthedifficultyofgoverningtransformativeAI.Historically,institutionshaveoftenfailedtograpplewiththerealityofexponentialtrendsuntilitwastoolatetorespondeffectively.Forinstance,epidemiologistsinearly2020dismissedCOVID-19asbeinglessprevalentthantheflu,andtheInternationalEnergyAgencysystematicallyunderestimatedsolarpowergrowthforoveradecade,repeatedlypredictingthatitwouldleveloffordecreasewheninfacttheindustrymaintainedroughly25%annualgrowth.

AllthisistosaythatwehaveeveryreasontobelievethatregulatingAIeffectivelywillbeunusuallydifficultevenincomparisontopasteffortstoregulateemergingtechnologies,whichhavebeenfarfromuniversallysuccessful.

4

ApproachestoRegulation

LettheMarketHandleIt

Inlightofthesechallenges,howshouldthegovernmentregulatetransformativeAI?Onepossibleansweristhatitshouldn’t.LibertarianwriterslikeAdamThiererhavemadethecaseforacultureof“permissionlessinnovation”forAIdevelopment,inwhichtheroleofgovernmentwouldbelimitedtoenforcingexistinglawsandfacilitatingindustryself-regulationwith“softlaw”toolssuchasvoluntarystandard-setting.InEurope,MarioDraghi’scompetitivenessreviewstrikesasimilarnote,urgingtheEUto“profoundlyrefocusitscollectiveeffortsonclosingtheinnovationgapwiththeUSandChina,especiallyinadvancedtechnologies.”

Theappealthatthisviewhasfortechno-optimistchampionsofinnovationisn’thardtounderstand.Historically,governmentshaveoftenstruggledtoregulateemergingtechnologiesinawaythatdoesmoregoodthanharm.Ifyousquint,callsforregulationtopreemptivelymitigatespeculativefutureAIriskslookalotlikehistoricalcallsto(over)regulatenuclearenergy,whicharguablyledtodisastrouseconomicandenvironmentalconsequencesintheformofmissedopportunitiestogeneratecheapandabundantcleanenergy.Hundredsofthousandsofliveshavelikelybeenlostprematurelyduetoairpollutionthatcouldhavebeenpreventedbyremovingtheregulatorybarriersthatpreventnuclearenergyfrombeingcost-effective.AndtherespectivetrajectoriesofthetechindustriesofEuropeandtheUnitedStatesoverthecourseofthelastfewdecadesislikesomethingoutofanAynRandnovel;itgivesrisetothesameinstinctivesenseofbewilderedcontemptinthebreastofthelibertarianobserverasasatellitephotographoftheKoreanpeninsulaatnight.It’seasytoseewhysomeonewiththesebeliefswouldbedisinclinedtofollowtheleadofthosewhohavesucceededonlyinregulatingtheirowntechindustryoutofexistence,1andmoredisposedtosidewithbuildersandvisionariesthantakersandbureaucrats.

But,ironically,supportforthistechno-optimistperspectivetypicallydepends(atleasttacitly)onskepticismaboutthefuturetrajectoryofAIcapabilities.Again,considertheexampleofnuclearpower.Howeverwelllibertarianobjectionstotheoverregulationofnuclearpowerplantshaveaged,it’smuchhardertomakethecasethattheadventofthenuclearage

5

requirednonewlawsorregulationswhatsoever.Presumably,reasonablepeoplecanagreethatalaissez-faireapproachtoregulatingtheprivateacquisitionandpossessionofnuclearweaponswouldbeinadvisable.“Everyprivateindividualshouldbeallowedtobuyasmanynuclearweaponsastheywant,freefromgovernmentinterference”isaverydifficultviewtodefendwithastraightface,nomatterhowstrongyourpro-marketpriors.

Atrulytransformativegeneral-purposeAIsystemwouldlikelyhavesignificantmilitaryapplications,perhapsevenassignificantasthemilitaryapplicationsofnuclearfission.ManynationalsecuritycommentatorspredictthattheimportanceofAIonthebattlefieldwillcontinuetorapidlyincrease.Ifthisisthecase,weshouldnotexpectatotallylaissez-faireapproachtoAIgovernancetobeanymorepracticallyorpoliticallyfeasiblethanalaissez-faireapproachtothegovernanceofnuclearweapons.Inotherwords,atrulytransformativedual-use2technologywillalmostcertainlyrequireanonzeroamountofregulation.

AcrucialdifferencebetweennuclearweaponsandtransformativeAIisthatnuclearweaponsunquestionablyexist,whiletransformativeAIisstillonlyapossibility.Promulgatingrigidanddetailedregulationsaddressingnuclearweaponsbeforetheywereevenbeingdevelopedwouldlikelyhavebeenfoolish,giventhewell-documenteddifficultiesthatregulatorshavehistoricallyhadinpredictingthefuturecourseoftechnologicalprogress.Butatsomepoint,onceatransformativedual-usetechnologyisactuallyunderdevelopment,regulationbecomesunquestionablynecessary—andatthatpoint,allstakeholdershaveamutualinterestinensuringthatregulationsarecompetentlydesigned,interpretedandenforced.Insufficientgovernmentregulatorycapacitycouldleadtohamfistedandoverlyharshregulationdowntheline,oncestakeholdersrealizethatsocietyisabouttobefundamentallytransformed.Worseyet,anunpreparedgovernmentmightregulateincompetently,harmingindustrywithouthelpingthepublic.

Ifthat’sthecase,then,giventhestakes,wouldn’titbeagoodideatostartpreparing?Evenif“letthemarkethandleit”istypicallysoundwisdom,thatheuristicalonecan’tfullyresolveissueslikethisonewherethenationalsecurityimplicationsofaparticulartechnologyarelikelytonecessitatesomedegreeofgovernmentoversightatsomepoint.Andgiventhescaleofthecostsandbenefitsatissue,“letthemarkethandleituntilthesenationalsecurityissuesmanifestthemselves,iftheyeverdo”isn’tanadequate

6

solutioneither.Iftherearestepsthatcanbetakentoincreasetheregulatorycapacityoftherelevantgovernmentbodieswithoutsignificantlyinhibitinginnovation,it’sincrediblyimportantthatweshouldtakethemassoonandaswellaspossible.

AnticipatoryGovernanceandthePrecautionaryPrinciple

Ontheothersideoftheaisle/pond,somewillobjecttoradicaloptionalityfromtheoppositedirection,arguingthatweneedtoimplementstrictprescriptiveregulation(suchasalawprohibitingAIresearch,oranFDA-stylelicensingregimethatwouldonlyauthorizecompaniestodevelopsystemsalreadyproventobesafe)assoonaspossible.ThesecriticssometimessuggestthattheregulatoryresponsetothepossibilityoftransformativeAIshouldobservetheprecautionaryprinciple,orthatregulatorsshouldtrytopredictthefutureofAIprogressandimplementananticipatorygovernanceapproach.

Initsmostextremeform,theprecautionaryprincipledictatesthatanyactionwhichmightposearisktopublichealthorsafetyshouldbeprohibitedunlessthepartywishingtoundertaketheactioncanprovethattheactionisnotdangerous.Becauseeverytrulyrevolutionarytechnologycreatesrisksaswellasbenefits,this“hard”precautionaryprinciplewouldprohibitdevelopmentoftransformativeAIfortheforeseeablefuture.Thisversionoftheprecautionaryprincipleissimplybadpolicy,becauseitignoresthepossibilitythatregulationmightitselfcausemoreharminexpectationthantherisksthattheregulationisintendedtoaddress.However,therearealsoanumberoflessunreasonablealternativeformulations.TheEU,forexample,endorsesaversionoftheprecautionaryprinciplethatinvolvesconductingacost-benefitanalysisthattakesboththecostsofregulatingandthecostsoffailingtoregulateintoaccount;thisistheprecautionaryprinciplethattheEU’sCodeofPracticeforGeneral-PurposeAIinvokes.Andsomescholarshavearguedthattheprecautionaryprincipleisjustifiedincaseswherethereisarealdangeroftrulycatastrophicharm,becauseevenaverylowprobabilityofacostthatcanbesaidtobe“infinite”(suchastheextinctionofalllifeontheplanet)outweighsevenaveryhighprobabilityofaverysubstantialfinitebenefit.

7

Aprecautionaryprinciplethatincorporatescost-benefitanalysismightnotbeinconsistentwithradicaloptionality,underreasonableassumptionsaboutthecostsofpreemptivelyregulatingemergingtechnologies.Andwhilethecatastrophicharmargumentistheoreticallysound,itdoesn’thaveobviousimplicationsforAIgovernance.Foronething,itisn’tclearthatrestrictinginnovationinliberaldemocracieswouldlowertheoveralllong-termprobabilityofcatastrophicoutcomes,giventhatAIresearchwouldalmostcertainlycontinuetobeconductedinauthoritarianstatesabroad.Foranother,we’renotconvincedthatthepotentialrisksassociatedwithtransformativeAIarenecessarilymoreinfinitethanthepotentialbenefits.Totakeoneobviousexample,powerfulAIsystemsintherighthandscouldpreventacatastrophethatwouldotherwisehaveoccurred,suchasanuclearwaroraglobalpandemic.Moregenerally,it’splausiblethatawealthiersocietywithmoreaccesstointelligencewillbemorewillingandabletoinvestinavertingcatastrophe,meaningthatgrowthandprogressmayinfactbeanti-correlatedwithcatastrophicrisk.

There’snoscientificallycertainwayofdetermining,fromourcurrentvantagepoint,whetherthepotentialbenefitsoftransformativeAIoutweighthepotentialrisks.Inthefaceofthiskindofuncertainty,relyingonroughheuristicsmaybethebestwecando.Theprecautionaryprincipleisanapplicationofonepossibleheuristic—namely,thatpolicyshouldtreatchangesaslikelytobeharmfulunlesstheycanbeprovenharmless.Thealmostequallyroughheuristicthatwepreferisbasedontheobservationthatthroughouthistorynewtechnologieshavetypically(thoughnotinvariably)producedanetbenefitforsocietyinthelongterm.3

Unlikemostprecautionaryprincipleapproaches,anticipatorygovernancedoesn’trelyontechno-pessimistassumptions.Instead,proponentsofanticipatorygovernanceareoptimisticabouttheabilityofregulatorstopredictthecourseoftechnologicalprogressandimplementeffectiveregulationsbeforethey’reneeded.Thegreatadvantageofthisapproachisthatitcanpreventharmsbeforetheyhappen.It’snotunreasonabletoexpectanounceofpreventiontobeworthapoundormoreofcure.

Ourobjectiontoananticipatorygovernanceapproachisbasedontheobservationthatpredictingthefuturetrajectoryoftechnologicalprogressisdifficult,andthatgovernmentshavehistoricallybeenterribleatit.We’vewrittenaboutthisprobleminmoredetailelsewhere,butessentially,wethink

8

thatthehistoryofregulationofemergingtechnologiesshowsthatattemptstoaddressrisksthatarepoorlyunderstoodoftenresultinlegalregimesthatareineffectiveorevencounterproductive.Thehands-offapproachthattheU.S.governmenttookwhentheInternetwascomingintoexistence,forexample,holdsupbetterinhindsightthanthesamegovernment’sattemptstoanticipatorilyregulatehometapingviatheAudioHomeRecordingActof1992,whichwasrenderedmostlyobsoletealmostassoonasitwaspassedbytheadventofthepersonalcomputer.

Anobviouscorollaryofthefactthatit’sveryhardtopredictthefuturecourseoftechnologicaldevelopmentisthatit’salsoveryhardtopredicthowsocietywillinteractwithunknowablefuturetechnologiesandwhatregulatoryinterventionswillbenecessaryoradvisable.Also,regulatoryregimestendtobemuchstickierandmorepath-dependentthanyoumightexpect,sochangingcourseafterinitiallycommittingtoacertainregulatoryapproachmaybedifficultandcostly.Crucially,institutionalleadtimesarelong;recruitingscarceexpertise,establishingsecuretestingandinformation-sharing,anddesigningflexible,lawfulauthoritiestakesyears.Ifcapacity-buildingbeginsonlyoncerisksandbenefitsareunmistakable,thedecisivewindowforproportionateactionwilllikelyhaveclosed.

Somedegreeofanticipationisnecessary,ofcourse.Evenoursuggestionthatgovernmentsshouldfocusonbuildingcapacityisbasedonpredictions—wepredictthattransformativeAIsystemsmaybedevelopedandthat,ifdeveloped,theywilllikelybedual-use.Decisionmakingunderuncertaintyisunavoidable,andradicaloptionalityisaformulaforminimizinguncertainty,notforeliminatingit.Butwethinkthere’sagreatdealtobegainedbyminimizinguncertaintyandmakinginformeddecisions.

Ifyoufindyourselfunconvincedbythepointsweraiseabove,andstillfavoramorerestrictiveregulatoryapproach,fairenough.Reasonablepeoplecandisagree.Butthefinalargumentinfavorofanoptionality-focusedapproachissimplerandhardertodispute:thepolicieswefavoraremorepoliticallyrealistic,atleastintheshortterm.CaliforniaandNewYorkhavepassedtransparencylawsthatrequirefrontierAIdeveloperstopublishtheirpoliciesforaddressingcatastrophicrisks,andtheEU’sGeneral-PurposeAI(GPAI)CodeofPracticeincludessimilartransparencyandreportingrequirements.Butfarmoresweepingandcostlypolicies,likeaninternationaltreatyprohibitingfrontierAIdevelopmentoramoratoriumondatacenter

9

construction,havenorealisticchanceofbecominglawinthenearfuture,eitherintheU.S.orEurope.Tobeclear,wehaveourdoubtsaboutwhethersuchproposalswouldbedesirableorevenworkable—butifyoudisagree,youshouldstillsupportthepolicieswe’rerecommendingevenasyoucontinuetoadvocateformoreambitiousproposals.

RadicalOptionality

Insteadofregulatingorfailingtoregulate,governmentscanpreparetoregulateinawaythatwillimprovetheirabilitytorespondtoawiderangeofpossiblescenarios,foreseenorunforeseen.Thiscanbedonebybuildingstrongregulatoryinstitutions,equippingthemwithappropriatelyflexibleauthorities,andensuringthattheyhaveaccesstotheinformationthey’llneedtorespondcompetentlyanddecisivelyifiteverbecomesclearthatregulationisneededtoaddresssomeintolerablerisktopublicsafetyornationalsecurity.Unliketheprecautionaryprincipleapproach,thepolicymeasuresinvolvedwouldimposeonlynegligibleburdensonAIcompaniesandwouldhaveanegligibleimpactoninnovation.Instead,thecostsofanoptionality-preservingapproachwouldbemeasurableintaxpayerdollarsandpoliticalcapital.

Thisiswherethe“radical”comesin.Simplyfocusingonpreservingoptionalitysoundslikearathermoderateproposal.Butwearguethatgovernmentsshouldbewillingtospendanextraordinaryamountofmoney,effort,andpoliticalcapitalonpreservingoptionality.

Thisfollowsfromthescaleoftheproblem—again,thepremisefromwhichwe’restartingisthatthereisadistinctpossibilityofatransitionatleastassignificantastheIndustrialRevolutionoccurringoverthecourseofthenextfewyearsordecades.Ifgovernmentsgenuinelyacceptthatdual-usetransformativeAIsystemsmayarriveinthenear-ormedium-termfuture,thelogicalconsequenceisthatagreatdealofwhattheycandotoevenmarginallyimprovetheoddsofthetransitiongoingwellwillbecost-justified.Theprimarycostsatissue,then,arecoststoinnovation.Governmentsshouldbewaryofcounterproductiveinterventions,butnotmuchconcernedwiththeactualpecuniarycostofanyrealisticmeasurethatseemslikelytohavenet-positiveresults.Evenifthere’sa95%chancethatthemoneyspentonagivenpolicymeasureiswasted,afivepercentchanceofsomepositiveimpact

10

intermsofmitigatingtherisksorrealizingthebenefitsofthemostimportantinventioninhumanhistorywouldmeanthatthecostswerejustifiedathousandtimesoverinexpectation.

Radicaloptionalityshouldbedistinguishedfrom“muddlingthrough,”theincrementalistpolicymakingphilosophyassociatedwithCharlesLindblom.Bothapproachesareskepticalofcommittingtograndregulatoryregimesbasedonpresent-dayguessesabouthowthefuturewillunfold,andbothpreferflexibilitytoforeclosure.Butmuddlingthroughtendstopreserveflexibilitylargelybydefault,throughinactionanddelay.Radicaloptionality,bycontrast,requiresproactiveinvestmentinbuildinginstitutionalcapacity—notenoughtolockinaparticularregulatoryfuture,butenoughtomakegoodchoiceswhentheybecomenecessary.

Ofcourse,theremaycomeatimewhenpreparationceaseseithertobenecessaryortobesufficient.IfprogressinAIcapabilitiesresearchplateausanditbecomesclearthattransformativesystemsarenotonthehorizon,thepolicieswe’resuggestingwouldmostlyberedundant.Ontheotherhand,iftransformativeAIisimminentanddireriskstopublicsafetyaremanifesting,asubstantiveregulatoryresponsecouldbebothnecessaryandpoliticallyinevitable.

ConsiderformerOpenAIemployeeLeopoldAschenbrenner’sseriesofessayson“SituationalAwareness.”Aschenbrennerproposesthatitisnecessaryand“inevitable”fortheU.S.government,motivatedbynationalsecurityconcerns,toprohibitprivatecompaniesfromworkingontransformativeAIsystemsandinsteadinvesttrillionsofdollarsinagovernment-run“AGIManhattanProject,”startinginperhaps2027or2028.Unsurprisingly,thistakegeneratedafairamountofcriticismaswellasitsshareofapprobation.Inanotherinfluentialrecentpiece,thecryptopioneerVitalikButerinproposedawayofavoidingafutureinwhichtransformativeAIisdevelopedinaclosed,centralized,securitizedcontext.Hissolutionwasaphilosophythathecallsdefensiveacceleration,or“d/acc,”whichwouldfocusonthedevelopmentofdefense-favoringtechnologies.

Thesetwoperspectivesareprofoundlydifferent,buttheybothgenerallydealwiththeproblemofbalancingtheneedtoencourageinnovationandtheneedtomitigaterisks.Radicaloptionalityisastrategyfordoingexactlythis—orrather,astrategythattakesadvantageofthefactthatsafetyandinnovation

11

aren’tnecessarilyconflictingvalues.Theappealofthisapproachisthatitpromisestoworkwellinawidevarietyoffutures.RegardlessofwhetheryouagreewithAschenbrennerorButerin(or,aswillbethecaseformostpeople,withneither)abouttherightwaytohandlethedevelopmentoftransformativeAIwhenithappens,thecorrectcourseofactionatthispointintimeistoavoidoverregulatingfornowwhilepreparingourinstitutionsforthechallengesthatmaylieahead.

InthescenarioAschenbrennerpredicts,theU.S.governmentwilleventuallyandsuddenlyrealizethenecessityofurgentactionand,likeastudentpullinganall-nighterbeforeanexam,throwtogetherthemostcomplexpublicprojectinhumanhistoryatthelastminute.Wedon’tnecessarilyagreewithAschenbrenner’stimelinesorhiscertaintyabouthowtransformativeAIsystemswillbedeveloped.Butifhedoesturnouttobecorrect,theprojecthecontemplateswillproducebettero