英伟达-自动驾驶的Chat GPT 时刻 The ChatGPT Moment for Autonomous Driving 2026

DVIDIA

TheChatGPTMoment

forAutonomousDriving

MarcoPavone,Sr.DirectorofAutonomousVehicleResearchAlpamayoSummit@CVPR2026|June4,2026

2nVIDIA

AVrepresentsoneofthebestexamplesofPhysicalAltodate

ProgressinAVhasbeenmind-blowing

2005

2026

2nVIDIA

AVrepresentsoneofthebestexamplesofPhysicalAltodate

ProgressinAVhasbeenmind-blowing

2005

2026

KeyPillarsofNVIDIAAutonomousVehicleSolutions

AfullstackprogramtargetingL4autonomy|Billionsofinvestmentover10years

OrinThor

Scalability/OneChipArchitecture

Al/DLInfrastructuretoEdge

DriveSDK-OS,Tools,DriveWorks

Simulation

DRIVEAVStack&Fleet

AlEngineering

FunctionalSafety

AVResearch

nVIDIA

FoundationModelsforPhysicalAl

Afoundationmodelisanymodelthatistrainedonbroaddata(usingself-supervisionatinternetscale)andthatcanbeadaptedtoawiderangeofdownstreamtasks

CosmosWorldFoundationModels(WFMs)

Embed|Predict|Transfer|Reason

CosmosPredict

PhysicalAlSynthesis

CosmosTransfer

PhysicalAl

Understanding

CosmosEmbed

Cosmos

Reason

"Acarovertakingatruck"015-021

GeneratefutureworldstatesfrommultimodalinputsForpost-trainingtodevelopspecializedphysical

Almodels

GeneratePhysics-groundedphotorealoutputsconditionedonstructuralinputsorgroundtruthfromNVIDIAOmniverse

ForsyntheticdatagenerationwithNVIDIAOmniverse

Ajointvideo-textembeddertailoredforphysicalAlFortext-to-videoretrieval,inversevideosearch,

semanticdeduplication,anddatacuration

Chain-of-thoughtreasoningtoprovidethebestresponse

Forout-of-theboxphysicalAlreasoningandpost-trainingforembodiedAlmodels

nVIDIA

FoundationModelsDrivingEnd-to-EndPhysicalAIDevelopment

CosmosWorldFoundationModels(WFMs)Embed|Predict|Transfer|Reason

Long-tailgeneralizationviacommon-sensereasoning

On-vehicleAVstack

FM-empoweredE2Estacks

Interactivevehicleinterface

Explainability:fromvisiontoaction

Run-timemonitoring

AVdevelopmentprocesses

Datageneration

Datacuration

Sensorandtrafficsimulation

Rateestimation&safetyvalidation

Large,intelligentmodelsrunninginthecloudSmall,efficientmodelsrunningonboard

TheEmergenceof

EmbodiedReasoning

Background:AlpamayoVision-Action(VA)Model

FoundationforAlpamayoproduction(NDASE2Eprogram)

Behavior

Cloning

ReconstructionLosses

Action:AutoregressiveTrajectoryDecoder

TrajectoryDecoder

VideoDecoder

(training-only)

·Next-token-predictiontrainingpipelineonthefullmultimodalsequence(images,actions)

TokenProcessing:LLMBackbone

Llama2Backbone

·Trainedoverlarge-scaleAVdata

·Runtime-friendlysizeforembeddeddeploymentbydesign

ViTEncoder

Vision:SpatiotemporalContextEncoder

·Efficientmulti-camera,multi-frametokenizationtoreducetokensequencelengths

Egohistory

Multi-camera,

multi-frameimages

nVIDIA

Wu,AcceleratethefutureofAl-definedvehiclesandautonomousdriving,GTCTalk2025

nVIDIA

AlpamayoVAinAction

Videos2xspeed

Unprotectedleft,withVRUcrossing,

thennudgingforVRUgettingintovehicle

withopendoor

Creepingandyieldingatminortomajoroccludedleftandstopsign

NudgingforDPV,slowingforspeedbump,nudgingfortrashcan

Nudgingforaparkingcar,slowing/waiting

foranoncomingbusinourlane,stopping

atastopsignandtakingoff

Lanechangewithdeadlineintostopandgotrafficforupcomingconstructionzone

NudgeforVRU,yieldbehindparkingtruck,andnudgeintooncominglane

FromAlpamayoVAtoAlpamayoVLA

Assessingthebenefitsofinternet-scalepre-training

SupervisedFine-Tuning

ReconstructionLosses

BehaviorCloning

Action:MultimodalTrajectoryDecoder

TrajectoryDecoder

·Trainingwithautolabeledmeta-actionstoimprove

trajectorymulti-modalityandinterpretability

VideoDecoder

(training-only)Meta-Actions

Language:Internet-PretrainedLLMBackbone

●Pretrainedoverinternet-scaledata

·Distilledintoaruntime-friendlysizeforembeddeddeployment

VILALLMBackbone

(distilled)

Text

Encoder

ViTEncoder

Vision:SpatiotemporalContextEncoder

Egohistory

·Efficientmulti-sensortokenizationtoreducetokensequencelengths

·Nativescalingtohigherresolutionsandsensorcounts

"In400feet,turnright”

Multi-camera,Navigation

multi-frameimages

AlpamayoVLAConvergesFasterandDrivesBetter

Keyresult:Languagepre-trainingsynergizeswellwiththetaskofdriving

·Languagepre-trainingprovidesfasterconvergenceandleadstoabettersteadystatecomparedtotrainingfromscratch,evaluatedon25,600scenariosacrosstheUSandEU*

·Closed-loopevaluationcorroboratestheresults

1.61.51.41.3

1.2

1.1

1

NoLLMpre-training

Step

LLMpre-trained

20k40k60k80k

Onthevalueoflanguagepre-training

*Notthefirsttimewehaveseensucharesult:Mao,Ye,Qian,Pavone,Wang.ALanguageAgentforAutonomousDriving.COLM2024

AlpamayoReasoningVLA:ScalableArchitectureforReasoningVLAs

Action:Reasoning-informedTrajectoryDiffusion

·Efficientaction-experttrajectorydecodingvialightweightconditionalflow-matching

·RLpost-trainingtoimprovereasoning-actionalignmentandactionquality

Reasoning:Internet-PretrainedReasoningBackbone

·Pretrainedtoreasonoverinternet-scaledata

·RLwithverifiablerewardsimprovecausalreasoning

Vision:EfficientContextEncoder

·Handlesmultipleinputmodalities(cameras,text)

·Efficientmulti-camera,multi-timesteptokenizationtoreducetokensequencelengths

·Nativescalingtohigherresolutionsandsensorcounts

TrainingSignals

(IL,SFT,RL)

TrajectoryDecoder

Cosmos-ReasonBackbone

VisionEncoderTextEncoder

Ego

history

"Stopover"In400feet,

there”turnright”

UserNavigationCommands

Imagesfrommultiplecamerasandmultipletimesteps

CoTReasoning

Meta-Actions

nVIDIA

2025-10-28

nVIDIA.

Alpamayo-R1:BridgingReasoningandActionPredictionforGeneralizableAutonomousDrivingintheLongTail

NVIDIA¹

Abstract

End-to-endarchitecturestrainedviaimitationlearninghaveadvancedautonomousdrivingbyscal-ingmodelsizeanddata,yetperformanceremainsbrittleinsafety-criticallong-tailscenarioswheresupervisionissparseandcausalunderstandingislimited.WeintroduceAlpamayo-R1(AR1),avi-sion-language-actionmodelthatintegratesChainofCausationreasoningwithtrajectoryplanningtoenhancedecision-makingincomplexdrivingscenarios.Ourapproachisbuiltonthreekeyinnovations:

NVIDIA,Alpamayo-R1:BridgingReasoningandActionPredictionforGeneralizableAutonomousDrivingintheLongTail,2025;

/publication/2025-10alpamayo-rl

NVIDIA

Alpamayo1

Chain-of-ThoughtReasoning

Images,OverTime

Cosmos-Reason

TrainingSignals

MetaActions

Backbone

TrajectoryDecoder

Navigation

Reasoning

Stopduetopedestriansinthecrosswalk.

Alpamayo1DrivinginAlpaSim

PhysicalAlAVDataset

AlpamayoDeployedonCar

nVIDIA

Pavone,Liu,FromResearchtoProduction:HowAlpamayoAcceleratesAutonomousVehicleDevelopment,GTCTalk2026

/en-us/on-demand/session/gtc26-s81779/

NVIDIAAlpamayo

AnOpenEcosystemDesignedtoAccelerateReasoning-BasedAutonomousVehicleDevelopment

NVIDIAAlpamayoOpenPlatform

t=0.5s

Reasoning:

Slowdownduetotheleadvehicleahead

↑ContinuerightontoSjosavägenin10m

AlpamayoModels

Open10B-parameterReasoningVLAs

·Steerablebehaviorvianavigationandtextprompts

·Flexiblemulti-camerasupport

·Post-trainingscripts

AlpaSim

Opensimulationframeworkforclosed-looptestingandvalidation

·Extensiblepluginsystem

·LaunchinganAlpaSim-basedClosed-LoopE2EDrivingBenchmark

Entrypoints:

16

·

https://huggingface.co/blog/drmapavone/nvidia-alpamayo

·

https://huggingface.co/blog/drmapavone/nvidia-alpamayo-1-5

PhysicalAIDataset

MostdiverseopenAVdataset

·~1700hoursofmulti-sensordatafrom2500citiesin25countries

·HumanQA'dreasoninglabels

·CoCauto-labelingpipeline

·LaunchingaReasoningBenchmarkfocusedonchallengingscenarios

NVIDIA

EcosystemRecognition

ThankYOUforengagingwithAlpamayo!

Computex2026BestChoiceAward

AlpamayoOpenPlatformrecognizedfor:

·EmpoweringmanufacturerstobuildAVswithhuman-likeperception,reasoning,anddecision-makingcapabilities.

·Enablingclosed-looptestingacrossdiversetrafficscenarios,weatherconditions,andedgecases.

·Fosteringarobustopen-sourceAlecosystem.

17Zero-shotapplicationtoJapanesedata!

(TierIV)

400,000+downloadsofAlpamayoModels

2,300,000+downloadsofthePhysicalAlAVDataset

3,000+starsacrossallAlpamayoGitHubrepos

2026ChallengewInnERSLeaderboardSubmitAbout

Method

Team

MMS

MMS(selected)

MMS(heavyrai.

MMS(constructio...

TTVLM-BaseQwen3.52BMD+BV0.25

STLA-MINES

5.15

4.52

6.44

5.89

4.45

4.84

KinematicsV2

A1.5

Rangers

KE:SAI

4.31

4.31

4.18

4.21

4.39

4.50

Zero-shot3rdplaceonKITScenesLongTailChallenge!②nVIDIA(KE:SAITeam)

NVIDIAAlpamayoOpenPlatform

Chain-of-CausationReasoningLabels

100%manually-verifiedlabelstobootstrapreasoningmodelR&D,evaluation,andbenchmarks

·Explorereasoningimmediately,fromtrainingtoevaluation.

ienieiaotheefesie.snteointhe

center.Unlesstheegoischanginglane8,butthemissionis"KeepLane7;sovahice1mugntnothecritical

eieceiforhnaedsteceteoisaeceleratin,but

vehicle3;

Thisvehicleisontherightside.Unlesstheegowas

chngnglanesbutthemisionisTKepLane,somaybe

sehiaveticte4unhevonue4sthntnegi

vehicle4

egosfane,Lhascuudbeapoterislconfict.

byect

ReasoningAuto-LabelingPipeline

Toolstogeneratereasoninglabelsonyourowndata

·Generatemeta-actions,identifykeyframes,and/orproducechain-of-causationreasoninglabels.

·ExtensibleAPls;useyourfavoriteserverorlocalVLM/NVlabs/alpamavo-coc-autolabeler

Recipes

EachrecipefoldercontainsitsownREADMEwithinstallationandtraininginstructions.

Recipe

Description

recipes/alpamayo1sft/

Alpamayo1supervisedfine-tuning(HuggingFaceTrainer+DeepSpeed)

recipes/alpamayo1_5_sft/

Alpamayo1.5SFT(HuggingFaceTrainer+DeepSpeed)

recipes/alpamayo1xrl/

Alpamayo1and1.5RLpost-training(Cosmos-RL/GRPO)

UtilityScripts

Script

Purpose

scripts/curate_p