人工智能算法进展：现状分析与未来预测

ResearchReport

CARTERC.PRICE,BRIENALKIRE,MOHAMMADAHMADI

Algorithmic

Advancementin

ArtificialIntelligence

ASurveyofAdvanceswithProjectionsfortheNearFuture

Formoreinformationonthispublication,visit/t/RRA3485-1.

AboutRAND

RANDisaresearchorganizationthatdevelopssolutionstopublicpolicychallengestohelpmakecommunitiesthroughouttheworldsaferandmoresecure,healthierandmoreprosperous.RANDisnonprofit,nonpartisan,andcommittedtothepublicinterest.TolearnmoreaboutRAND,visit.

ResearchIntegrity

Ourmissiontohelpimprovepolicyanddecisionmakingthroughresearchandanalysisisenabledthroughourcorevaluesofqualityandobjectivityandourunwaveringcommitmenttothehighestlevelofintegrityandethicalbehavior.Tohelpensureourresearchandanalysisarerigorous,objective,andnonpartisan,wesubjectourresearchpublicationstoarobustandexactingquality-assuranceprocess;avoidboththeappearanceandrealityoffinancialandotherconflictsofinterestthroughstafftraining,projectscreening,andapolicyofmandatorydisclosure;andpursuetransparencyinourresearchengagementsthroughourcommitmenttotheopenpublicationofourresearchfindingsandrecommendations,disclosureofthesourceoffundingofpublishedresearch,andpoliciestoensureintellectualindependence.Formoreinformation,visit/about/research-integrity.

RAND’spublicationsdonotnecessarilyreflecttheopinionsofitsresearchclientsandsponsors.

PublishedbytheRANDCorporation,SantaMonica,Calif.

©2025RANDCorporation

isaregisteredtrademark.

Cover:MF3d/GettyImages.

LimitedPrintandElectronicDistributionRights

Thispublicationandtrademark(s)containedhereinareprotectedbylaw.ThisrepresentationofRANDintellectualpropertyisprovidedfornoncommercialuseonly.Unauthorizedpostingofthispublicationonlineisprohibited;linkingdirectlytoitswebpageonisencouraged.PermissionisrequiredfromRANDtoreproduce,orreuseinanotherform,anyofitsresearchproductsforcommercialpurposes.Forinformationonreprintandreusepermissions,pleasevisit/about/publishing/permissions.

AboutThisReport

iii

Withrecentadvancementsincommercialproducts,suchasOpenAI’sChatGPT,AnthropicAI’sClaude,Meta’sLlama,andotherlargelanguagemodels,thetopicofartificialintelligence(AI)hasexpandedinthepublicdiscourse.And,asAIcapabilitiesdevelop,therehasbeen

increasingconcernabouttheirsecurityimplications.Inthisreport,wesurveyalgorithmic

improvementsfromnumericalanalysis,operationsresearch,andcomputerscience;identify

somecommonchannelsofadvancement;andthendescribethechannelsbywhichAImight

advance.WealsodescribetheimplicationsthatalgorithmicimprovementmayhaveonAI

advancementoverthenextfewyearsanddiscusssomeindicatorsthatmightpointtosuch

advancements.Thepurposeofthisresearchistopresentissuestoconsiderregardingthefuturealgorithmicadvancement.

Thisworkisintendedtobeofinteresttobothpolicymakersandamoregeneralaudience

lookingforinformationaboutalgorithmicadvancementinAI.However,portionsofthisreportassumethatthereaderhasfamiliaritywithalgorithmsingeneralandmachinelearning

algorithmsinparticular,andsomeofthecontentintheappendixesreliesonanunderstandingofadvancedmathematics,includingnumericalanalysis.

TheresearchinthisreportwasconductedbetweenOctober2023andAugust2024.This

predatestheunveilingofDeepSeek-V3inlateDecember2024.1DeepSeek-V3purportedly

outperformssimilaropen-sourcelanguagemodelsandperformscomparablytoleadingclosed-sourcemodelswhilerequiringlesscomputeforfulltraining;itmayprovideanimportant

exampleofanalgorithmicadvancement.2However,theauthorsmademinorrevisionsandupdatestothereportthroughFebruary12,2025.

TechnologyandSecurityPolicyCenter

RANDGlobalandEmergingRisksisadivisionofRANDthatdeliversrigorousand

objectivepublicpolicyresearchonthemostconsequentialchallengestocivilizationandglobalsecurity.Thisworkwasundertakenbythedivision’sTechnologyandSecurityPolicyCenter,

whichexploreshowhigh-consequence,dual-usetechnologieschangetheglobalcompetitionandthreatenvironment,thendevelopspolicyandtechnologyoptionstoadvancethesecurityofthe

1CadeMetzandMeaghanTobin,“HowChineseA.I.Start-UpDeepSeekIsCompetingwithSiliconValleyGiants,”NewYorkTimes,January23,2025.

2DeepSeek-AI,AixinLiu,BeiFeng,BingXue,BingxuanWang,BochaoWu,ChengdaLu,Chenggang

Zhao,ChengqiDeng,ChenyuZhang,etal.,“DeepSeek-V3TechnicalReport,”arXiv,version1,December27,2024.

iv

UnitedStates,itsalliesandpartners,andtheworld.Formoreinformation,contacttasp@.

Funding

ThisresearchwasindependentlyinitiatedandconductedwithintheTechnologyandSecurityPolicyCenterusingincomefromoperationsandgiftsfromphilanthropicsupporters,whichhavebeenmadeorrecommendedbyDALHAPInvestmentsLtd.,EffektivSpenden,ErgoImpact,

FoundersPledge,CharlottesochFredriksStiftelse,GoodVentures,JaanTallinn,Longview,

OpenPhilanthropy,andWakingUpFoundation.Acompletelistofdonorsandfundersis

availableat/TASP.RANDdonorsandgrantorshavenoinfluenceoverresearchfindingsorrecommendations.

Acknowledgments

Wethankthereviewers,MaryLeeandNeilThompson,fortheirthoughtfulreviewsand

constructivecomments.WealsoappreciatetheguidanceofJeffAlstott,EmmaWesterman,andCaseyDuganandthesupportprovidedfromtheTechnologyandSecurityPolicyCenter

throughoutthisstudy.WealsothankLennartHeim,KonstantinPitz,Mauricio,GabrielKulp,

andtheparticipantsintheComputeWorkingGroupfortheircommentsonanearlierdraftofthiswork,andNickBrown,whoalsoprovidedsubstantivecomments,particularlyduringtheinitialphasesofthiswork.Finally,wethankAlisonHottesandBryanBolingfortheirworkmanagingthequalityassuranceprocessofthisdocument.Whilewehavebenefitedfromtheinsightsfrommanypeople,anyerrorsinthisdocumentaresolelytheresponsibilityoftheauthors.

Summary

v

Withrecentadvancementsincommercialproducts,suchasOpenAI’sChatGPT,AnthropicAI’sClaude,Meta’sLlama,andotherlargelanguagemodels,thetopicofartificialintelligence(AI)hasexpandedinthepublicdiscourse.And,asAIcapabilitiesdevelop,therehasbeen

increasingconcernabouttheirsecurityimplications.Inthisreport,wemakeevidence-based

projectionsaboutthedirectionandpaceofalgorithmicadvancementstohelpinform

policymaking.WedescribeseveralpossiblechannelsforalgorithmicimprovementrelatedtoAIandexploretheimplicationsofhowprogressmightbemadealongeachofthosechannels.3

KeyFindings

Ourresearchonthedirectionandpaceofalgorithmicadvancementsrevealedthefollowingkeyfindings:

•Thetwopotentiallyhigh-impactchannelsforalgorithmicimprovementinvolve(1)

generatingsyntheticdataorpruningexistingdatatoproducedatasetsbettersuitedfor

trainingAIand(2)increasingdataefficiencythroughimprovedalgorithmsthatareeitherlesscomputationallycostlythantransformers(suchasMamba)ormoreeffectiveper

iterationthantransformers(suchasKolmogorov-ArnoldNetworks).4Thereisalsopotentialforbothimprovementstohappenmoreorlesssimultaneously.

•Onewild-cardchannelwouldbethedevelopmentofalternativecriteria(whichwe

looselyrefertointhisreportasobjectivefunctions)fortrainingAIsystemsthatbettermatchcommerciallyusefulperformancemeasures.5

•Therearethreenear-termfuturesthatdependondifferentlevelsofadvancementalongthetwohigh-impactchannels.

-Ifdatalimitationsarebinding:Afuturescenarioispossibleinwhichthe

unavailabilityofadditionaldatacouldpreventmodelsfromcontinuingtoscale

efficiently,andthatcouldleadtosmall,focusedAIsystemsdominatingthemarket.

-Ifalgorithmsfailtoscale:Inafutureinwhichadditionaldatacanbeobtained

throughsyntheticgeneration(orsomeothermechanism)6butnewalgorithmsarenotabletoefficientlyextractmeaningfulperformancegainsbyincludingthoseadditionaldata,thenworkonlargemodelscouldcontinue,butsmallAIsystemswouldlikely

3Forthepurposesofthisreport,changestoanalgorithmareanimprovementiftheyleadtoenhancedperformancemeasuresorreducedeffortandassociatedresourcerequirements(orboth)foragiventask.

4Wemakenoclaimthatalgorithmicimprovementswillorwillnotbewidelyadoptedforcommercialapplications.

5Manymodelsusethecross-entropylossfunctionastheprimaryobjectivefortraining.Somemodelspairthatwithreinforcementlearningorreinforcementlearningthroughhumanfeedbacktoimproveperformance.Alternativestotheseobjectivescouldleadtosubstantialimprovements.

6Forinstance,trainingonnontextmodalities.

vi

dominate.7Essentially,iftherearediminishingreturnstoscalewithadditionaldata,thenlargermodelsmightnotbecommerciallyviable.

-Ifalgorithmscontinuetoadvance:Inafutureinwhichdataareabundantand

algorithmsaremoreefficientinusingthosedata,thenever-largermodelsarelikelytobeasignificantfactorinAIresearchforthenearterm.

•Oneimplicationofalgorithmicadvancementisthatexportcontrolsonhardware—such

asrestrictionsontheexportofhigh-endchipstoChinathatweremadeinOctober2022,October2023,December2024,andJanuaryof20258—couldhavemutedeffects,

dependingonthepathofalgorithmicadvancement.Asdescribedina2024CenterforaNewAmericanSecurityreport,9ifalgorithmicimprovementscontinuetobewidely

available,thenhardware-restrictedactors(suchasChina)willbeabletotrainmodelsandbeonlyafewupgradecyclesbehindthefrontier.

7Foradetaileddiscussionofwhichdomainsmightbenefitfromuseofsyntheticdata,seePabloVillalobos,AnsonHo,JaimeSevilla,TamayBesiroglu,LennardHeim,andMariusHobbhahn,“WillWeRunOutofData?Limitsof

LLMScalingBasedOnHuman-GeneratedData,”arXiv,version2,June4,2024,pp.7–9.

8BureauofIndustryandSecurity,“CommerceStrengthensRestrictionsonAdvancedComputingSemiconductorstoEnhanceFoundryDueDiligenceandPreventDiversiontoPRC,”OfficeofCongressionalAffairs,January15,2025.

9PaulScharre,“Future-ProofingFrontierAIRegulation,”CenterforaNewAmericanSecurity,March13,2024.

Contents

vii

AboutThisReport iii

Summary v

Figures viii

Chapter1.Introduction 1

WhatConstitutesAlgorithmicImprovement? 1

DimensionsofImprovement 2

ApproachandLimitations 2

OrganizationofThisReport 3

Chapter2.LiteratureonAlgorithmicAdvancement 5

Chapter3.MechanismsforAlgorithmicAdvancement 8

ChannelsUnlikelytoLeadtoSubstantialImprovements 8

ChannelswithPotentialtoLeadtoSomeImprovements 9

ChannelswithPotentialtoLeadtoSubstantialImprovements 10

SummaryofAdvancementChannels 13

Chapter4.ConclusionsandEarlyIndicators 14

PossibleFutures 14

RecommendationsforPolicymaking 15

AppendixA.BackgroundontheComputationalEffortAssociatedwithMachineLearning

Algorithms 16

MachineLearningAlgorithms 16

ComputationalEffortandResourcesforTraining 19

OtherFactorsandResourcesContributingtoTrainingTime 22

ComputationalEffortandResourcesRequiredforInference 23

MeasuringPerformanceforTasks 24

AppendixB.SurveyofMechanismsforAlgorithmicAdvancement 26

NumericalAnalysis 26

OperationsResearch 29

ComputerScience 30

AppendixC.ImplicationsforHardwareExportControls 31

AppendixD.CaseStudyofReinforcementLearningfromHumanFeedback 33

BackgroundandContext 33

ReinforcementLearningfromHumanFeedbacktoImproveSampleEfficiency 34

ReinforcementLearningfromHumanFeedbacktoAlignBehaviorswithHumanPreferencesand

Values 36

Summary 39

Abbreviations 40

References 42

Figures

viii

Figures

FigureD.1.ReinforcementLearning 33

FigureD.2.Human-SupervisedReinforcementLearning 34

FigureD.3.ReinforcementLearningwithHumanFeedback 34

FigureD.4.ReinforcementLearningfromHumanFeedbackPerformanceinLearningPong 35

FigureD.5.OverviewofSupervisedFine-TuningandReinforcementLearningfromHuman

FeedbackasAppliedtoaLargeLanguageModel 37

FigureD.6.HumanJudgesPreferredReinforcementLearningfromHumanFeedback

Summaries,EvenwithSmallerModelSizes 38

Chapter1.Introduction

1

Withrecentadvancementsincommercialproducts—suchasOpenAI’sChatGPT(whichwasreleasedin2018),10AnthropicAI’sClaude,Meta’sLlama,andotherlargelanguagemodels

(LLMs)—thetopicofartificialintelligence(AI)hasexpandedinthepublicdiscourse.And,asAIcapabilitiesdevelop,therehasbeenincreasingconcernabouttheirsecurityimplications.ToassessthesecurityimplicationsassociatedwithAI,policymakersneedtohaveestimatesofthedirectionandpaceofalgorithmicadvancements.Tothatend,weseektoaddressthequestion:

HowwillAIcapabilitiesadvanceinthenearfuturebecauseofalgorithms?

WhatConstitutesAlgorithmicImprovement?

Therearemanywaystodefinewhatconstitutesanalgorithmicimprovementandwhat

distinguishesanimprovementfromanewalgorithm,butnoneoftheoptionsareparticularly

robust.AstudybyYashSherryandNielC.Thompsonfocusesonalgorithmsforproblemswithexactsolutionsthataregloballyoptimalanddefinesanimprovementintermsofsolvingthe

sameproblemwithfeweroperations.11Alternatively,researchbyKatjaGracetakesavery

differentapproachandevaluatesavarietyofalgorithms,includingmachinelearningalgorithms,usingavarietyofperformancemeasures.ThesemeasuresincludetheEloratingsystem,12the

timerequiredforproblemsofagivencomplexity,thesizeofproblemsthatcanbesolved,andsamplestatistics,suchasprobabilitiesofdetection.13ThereportbyGracefocusedonthe

empiricalperformanceofalgorithms,includingbothhardwareandsoftwareadvances,whilethefocusofourworkinthisreportisonalgorithmicimprovementsintheabsenceofhardware

improvements.Hence,wearekeenlyinterestedinalgorithmicperformanceonspecifictasksrelativetotheeffortandassociatedresourcesrequired.

Forthepurposesofthisreport,changestoanalgorithmareanimprovementif,foragiven

task,theyleadto(1)enhancedperformancemeasuresor(2)reducedeffortandassociated

resourcerequirements(orboth).Indifferentcases,theimprovementscouldbemoresubjective(e.g.,samplestatisticsonhumanpreferences)ormoreobjective(e.g.,areductioninthenumber

10AnamNazirandZeWang,“AComprehensiveSurveyofChatGPT:Advancements,Applications,Prospects,andChallenges,”Meta-Radiology,Vol.1,No.2,2023.

11YashSherryandNeilC.Thompson,“HowFastDoAlgorithmsImprove?”ProceedingsoftheIEEE,Vol.109,No.11,November1,2021.

12AnEloratingsystemisamethodofcalculatingtherelativeskilllevelofplayersinzero-sumgames,suchaschess,baseball,andpocketbilliards.ThisratingsystemhasmorerecentlybeenappliedtoLLMs.

13KatjaGrace,“AlgorithmicProgressinSixDomains,”MachineIntelligenceResearchInstitute,TechnicalReport2013-3,December9,2013.

2

offloatingpointoperations[FLOPs]14neededtoperformamathematicaloperation).Thejudgmentoftheauthorswillbeusedtoidentifywhatconstitutesatask.

DimensionsofImprovement

ThereareseveralwaystodescribealgorithmicimprovementinAI.Onewaytoframe

improvementwouldbewithregardtotheextensiveorintensivemargin.Theintensivemargin

wouldincludesuchthingsasreducedrequirementsforinputs(e.g.,trainingdata,training

FLOPs,15ormodelparameters)orbetterperformancewiththesameorfewerinputs.Essentially,theintensivemarginisaboutefficiency.Improvementsalongtheextensivemarginwould

includenewcapabilitiesorareasofapplication—forexample,theabilitytosolveanewclassofproblemthatpriormodelswerenotabletosolve.

Improvementscanoccuratdifferentperiods:duringthetrainingphase,whilemakingpost-

trainingadjustments,orduringinference.16Ourfocusinthisreportisontheintensivemargin

duringthetrainingphase.Ourrationaleisthattrainingrequiresupfrontcoststhatcouldbea

barriertothedevelopmentoffuturemodels,andadvancementsalongtheextensivemarginare

generallyhardertoquantify.Thatsaid,somealgorithmicchangesmightresultinimprovementsalongmultipledimensionsorofferimprovementsalongonedimensionattheexpenseofanother.

ApproachandLimitations

Toestimatethepaceofalgorithmicadvancement,wefirstlookedatavarietyofalgorithmsinnumericalanalysis,operationsresearch,andcomputersciencetofindthemechanismsfor

algorithmicadvancement.WethengroupedthesemechanismsintobroadclassesandsearchedthecomputerscienceliteraturefordiscussionsabouttheirapplicabilitytoLLMs.Finally,we

describehowthesemechanismscouldworkinthenearfuturetoimprovethealgorithmsbehindLLMsandotherfoundationmodels.

Ourapproachwasnotcomprehensive,andthealgorithmsthatweassessedwereselectedbyexaminingseveraltextbooks.Thus,wemighthavemissedsomerelevantmechanisms.

Additionally,becauseoftherapidpaceatwhichnewresearchpapersarepublished,our

examinationoftheapplicationofthesemechanismsisnecessarilyincomplete.Whilethisstudy

14Inthisreport,FLOPsisthepluralformoftheabbreviationFLOP,whichreferstooneoperation(e.g.,addition,multiplication)performedondecimal(orfloatingpoint)numbers.FLOPspersecond(FLOP/s)referstothenumberofFLOPsthataprocessorcanperforminonesecond.SeeLennartHeim,“FLOPforQuantity,FLOP/sfor

Performance,”blog,*.xyz,April14,2023,andAppendixAforamoredetaileddiscussionofFLOPsandFLOP/s.

15SeeAppendixA.

16Inferencereferstothepost-trainingperiodwhenanAImodelisintroducedtonewdataandassessedonitsabilitytorecognizepatternsinandmakeinferencesaboutthenewdataset.SeeAppendixAforamoredetaileddiscussionofdifferenttypesoftrainingandinference.

3

cannotbeconsideredexhaustive,wedobelievethattheapproachissufficienttoidentifybroadtrendsandmakeprojectionsusefulforexploringpolicyoptions.

Asdiscussedinthepreface,theresearchinthisreportwasconductedbetweenOctober2023andAugust2024.AnimportantlimitationofthisreportisthattheresearchwasconductedbeforeDeepSeekunveileditsDeepSeek-V3languagemodelinDecember2024,whichappearstobeanimportantexampleofalgorithmicimprovement.17AccordingtoDeepSeek,theirmodel

“outperformsotheropen-sourcemodelsandachievesperformancecomparabletoleadingclosedmodels....[Andit]requiresonly2.788MH800GPUhoursforitsfulltraining.”18DeepSeek-V3isdescribedasamixture-of-experts(MoE)languagemodelthatachievesefficientinferenceandcost-effectivetrainingbyadoptingmulti-headlatentattentionandarchitecturalchangestotheir

previousmodel,implementinganewstrategyforloadbalancing,andperformingamulti-tokenpredictiontrainingobjectiveforstrongerperformance.Modeltrainingwasfollowedby

supervisedfine-tuning(SFT)andreinforcementlearningstagestoalignitsperformancewithhumanpreferences.19

ThisreportdiscussessimilarmechanismsofalgorithmicimprovementbutisnotinformedbythespecificdetailsofDeepSeek-V3.Forinstance,aconsiderationofDeepSeek-V3wasnota

partofourassessmentinAppendixDoftheutilityofreinforcementlearningfromhuman

feedback(RLHF)inadvancingAIalgorithms.DetailsabouttheroleofreinforcementlearningwithDeepSeek-V3aredescribedinatechnicalreportpublishedinJanuary2025.20

OrganizationofThisReport

Chapter2describestherelevantliteratureonalgorithmicadvancementrelatedtoAI.Then,Chapter3presentsthemechanismsthatwehaveidentifiedforalgorithmicadvancementand

discusseshowtheymightapplytoAIsystems.ThefinalchapterdescribeshowAIalgorithmsmightadvanceinthenearfutureandtheimplicationstheseadvancementscouldhave.

Wealsoincludefourappendixes:AppendixAprovidesbackgroundinformationonthe

computationaleffortassociatedwithmachinelearningalgorithms,whichisintendedtoprovideusefulcontextfortheinterestedreader;AppendixBincludesdetailsabouthowweidentifiedthemechanismsofalgorithmicimprovements;AppendixCdescribesthespecificimplicationsof

17CadeMetzandMeaghanTobin,“HowChineseA.I.Start-UpDeepSeekIsCompetingwithSiliconValleyGiants,”NewYorkTimes,January23,2025.

18DeepSeek-AI,AixinLiu,BeiFeng,BingXue,BingxuanWang,BochaoWu,ChengdaLu,Chenggang

Zhao,ChengqiDeng,ChenyuZhang,etal.,“DeepSeek-V3TechnicalReport,”arXiv,version1,December27,2024.

19DeepSeek-AIetal.,2024.

20DeepSeek-AI,DayaGuo,DejianYang,HaoweiZhang,JunxiaoSong,RuoyuZhang,RunxinXu,QihaoZhu,ShirongMa,PeiyiWang,etal.,“DeepSeek-R1:IncentivizingReasoningCapabilityinLLMsvia

ReinforcementLearning”arXiv,version1,January22,2025.

4

algorithmicadvancementonexportcontrolpolicies;andAppendixDcontainsacasestudyrelatedtoRLHF.

Chapter2.LiteratureonAlgorithmicAdvancement

5

WearenotthefirsttoinvestigatealgorithmicimprovementsrelevanttoAI.Inthischapter,weexploreexistingliteratureonalgorithmicimprovement,especiallystudiesthatarerelevanttothetrainingphaseofanAIsystem’sdevelopment.

ThereportbyGracementionedinChapter1examinedahandfulofproblemtypesonwhichtherehasbeenalgorithmicprogress,includingBooleansatisfiability,chessandGo,largenumberfactorization,physicssimulations,mixedintegerprogramming,scheduling,andavarietyof

machinelearningproblems.Foreachoftheseproblems,Gracefoundliteraturesummarizing

performanceprogressandassessedtheshareoftheprogressthatwasattributabletoalgorithmicadvancement.Usingtheseexamples,shedeterminedthatalgorithmicadvancementaccountedfor50to100percentofimprovedperformance.21

SherryandThompsonmeasuredthepaceofalgorithmicinnovationfor128familiesofexactalgorithmsand310algorithmicimprovements.Withexactalgorithms,theresultofaspecific

problemsolvedbydifferentalgorithmswithineachfamilywillbeidentical,soanimprovementwouldbeinthearithmeticoperationcountthatanalgorithmrequirestoreachtheexactsolution.SherryandThompsonfoundthatthepaceandscaleofimprovementvariedsubstantially;somealgorithmfamiliessawnosubstantiveimprovements,andotherssawimprovementsthatwere

substantiallyfasterthanthehardwareadvancementpacedescribedinMoore’sLaw.22While

theirstudyprovidesanempiricalassessmentofalgorithmicadvancement,itdoesnotprovideaforecastthatisrelevanttothepaceofadvancementinAI.23

Usingpublishedcharacteristicsofmodelsfrom2012to2023andapplyingcross-entropylossfunctiontomeasureperformance,Hoetal.estimatedthat5to40percentofLLMperformance

increasesfollowingpretrainingwereattributabletoalgorithmicimprovements.24Thepaperidentifiestwokeyinnovationsthatresultedinthemajorityoftheperformanceincrease:the

21Grace,2013.

22Moore’sLawisaprojection,basedonempiricalobservation,thatthenumberoftransistorspersquareinchonamicrochipwilldoubleeverytwoyears.Thisincreaseindensityrelatestoanincreaseincomputingpower.

23YashMohanSherryandNeilC.Thompson,“MeasuringthePaceofInnovation:EvidenceFromAlgorithms,”conferencepaper,SI2020ITandDigitization,NationalBureauofEconomicResearch,July2020.

24AnsonHo,TamayBesiroglu,EgeErdil,DavidOwen,RobiRahman,ZifanCarlGuo,DavidAtkinson,NeilThompson,andJaimeSevilla,“AlgorithmicProgressinLanguageModels,”arXiv,version1,March9,2024.Across-entropylossfunctionisawaytoevaluatemachinelearningalgorithms.Ingeneral,across-entropyloss

functioncomparesanactualdatapoint(s)totheoutputfromthemachinelearningmodel.Inpractice,these

comparisonsareaggregatedtoelicitspecificbehaviorinamodel.Essentially,thesemeasuresevaluatehowwellthemodelmatchesthetrainingdata.

6

introductionofthetransformer(adeeplearningarchitecture)andthescalinglawfromHoffmannetal.,2022.25

IntheStanfordInstituteforHuman-CenteredAI’s2024AIIndexReport,theauthors

collectedinformationonAIadvancement.26TheynotethatAIperformancehasbeen

approachingorsurpassinghumanperformanceonninetechnicalperformancebenchmarks.

However,theyalsonotethat“[p]erformanceonthesebenchmarkshasstagnatedinrecentyears,indicatingeitheraplateauinAIcapabilitiesorashiftamongresearcherstowardmorecomplexresearchchallenges.”27

LeopoldAschenbrennerreviewedadvancementsinLLMsandprojectedthegrowth

forward.28Heestimatesthattherehasbeenabouthalfanorderofmagnitudeofgainsinmodelimprovementperyearattributabletoalgorithmicadvancementand,ifthistrendcontinuesinto2027,hepredictsthatAIsystemswillbeabletodotheworkofAIresearchers.

Thereisnoclearconsensusamongthesestudiesaboutthepaceordirectionofalgorithmic

advancement.Furthermore,althoughAschenbrennerandtheauthorsofthe2024AIIndexReportdiscussforward-lookingpathsforadvancement,theyhavesomewhatdivergentinterpretationsofthetrends.Specifically,theydisagreeaboutwhetherAIsystemsareplateauingatornearhumanlevelsofperformance.Anotherkeypointofdisagreementisaboutwhethercontinued

improvementsintheperformanceofacross-entropylossfunctionthatisbasedonpredictingthenexttokenissufficienttoachievematerialimprovementsincommerciallyrelevantperformance

29measures.

Weattempttoresolvetheseissuesbyapproachingthisproblemslightlydifferentlythanearlierstudies.Byfocusingonthemechanismsofimprovementratherthanthepaceof

25Inthiscontext,ascalinglawisanempiricalrelationshipbetweenthenumberofparamet