驱动汽车科技创新发展演讲资料-理想自动驾驶-2024-04-自动驾驶

TheConvergenceofAutonomousDrivingandSystem1&2Thinking

PengJia

LiAuto,China

Contents

01

LiADOverview

02

LiADTechnologyHighlights

LiAuto'sViewonAutonomousDriving

Real

Rule-drivenL2:2D/Mono3DData-drivenL3:BEV/End2EndKnowledge-drivenL4:VLM/WorldModel

World

EveryDay

Driving

Scenarios

L4

UnknownScenarios

L3

ExpandedDriving

Scenarios

L2

Known

Scenarios

LiADFramework

SYSTEM1

Intuition&instinct

SYSTEM2

Rationalthinking

5%

Takeseffort

Slow

Logical

Lazy

Indecisive

95%

Unconscious

Fast

Associative

Automaticpilot

System1--End-to-EndModelforL3AD

Fastend-to-endresponsetothesurroundingenvironment.

System2--LargeMultimodal-ActionModel

Exploreandlogicallythinkunderunknownenvironments.Modalitiesincludelanguage,vision,pointclouds,canbusandnavigationtosolveL4unknownscenes.

System1System2TrainingLoop

Perception

Decision&Planning

Control

Short-termMemory

Vehicle

L3EndtoEndModel

Sensors

L4MultimodalLLM

Recognition

GeneralKnowledge

SimReinforcementLearningModel

EvaluationNetwork

GenerativeWorldModel

Cloud

Well-recognizedworksfromLiADteam

1

MUTR3D

2021World's1st

Incamera-based3Dtracking

FUTR3D

Industryleading

DenseTNR

1stPlaceSolution

InICCV2021INTERPRET

Challenge

DETR3D

2021World's1st

Incamera-based3Ddetection

HDMapNet

CVPR2021ADP3Workshop

Multi-sensor3Ddetectionmodel

(BestPaperNomination)

CORL2021DETR3D:

/pdf/2110.06922.pdf

CVPR2022MUTR3D:

/pdf/2205.00613.pdf

ICML2023VectorMapNet:

/pdf/2206.08920.pdf

CVPR2023NPN:

/pdf/2304.08481.pdf

CVPR2023FUTR3D:

/pdf/2203.10642.pdf

ICRA2022HDMapNet:

/pdf/2107.06307.pdf

CVPR2023VIP3D:

/pdf/2208.01582.pdf

ArchitectureofADMax3.0

SafetyPerception

SafetyPlanner

OneModel(Multi-taskPerception)

Prediction&PlanningNetwork

ShadowApp1

ShadowApp2

Shadow

StaticBEV

ObjectBEV

Occupancy

MPC

……

Spatio-TemporalPlanner

End2EndTrafficSignalNetwork

Camera×7

LiDAR

Radar

NavigationMap

NVIDIADRIVEOrin×2

InferencePerfOptimization

ForPerceptionPipeline111ms/9fps=>48ms/21fps

ItemOptimizationActionOptTypeLatency(percentage)

1

ApplyMPStoavoidCUDAcontextswitchoverhead(systemofmultipleprocesseswithGPUcalls).

Pipeline

-9.91%

2

RemoveunneededCUDAcalls(e.g.,cudaWaitExternalSemaphore).

Pipeline

-5.60%

3

EnlargeCUDA_DEVICE_MAX_CONNECTIONStoresolvefalsedependencyamongCUDAstreams.

Pipeline

-4.80%

4

Schedulemodelheadswithdifferentinferencefrequency.

Pipeline

-8.10%

5

ReplaceD2DwithH2DcopyinimagingstreamstoutilizeGPUCopyEngineinsteadofCUDAcore.

Pipeline

-3.25%

6

Optimizebevpoolv2pluginbyreducingwarpdivergence&fp16(from

Lidar_AI_Solution

).

Model

-10.80%

7

EliminatereformattingkernelsduetounfusedQATnodes.

Model

-3.60%

8

ApplyTRTMHAfastkerneltoacceleratetransformerblocks.

Model

-5.60%

9

UseSparsity.

Model

-3.60%

10

OffloadROI-aligntoPVA.

Model

-1.80%

Finalperfimprovement-57.06%

LiADL4Research——Drive-VLM

SequenceofImagesDriveVLMSceneDescriptionCriticalObjectsAnalysisHierarchicalPlanning

Input:

Basedonthesummarized

analysis,pleasedetermine

themetaactionsanddecisions.

Drive-VLM:

MetaActions:['Slowdown’,'Stop',‘Reverse’].

Decisions:Driveslowly,stopinfrontoftheconstructionworkers,andthenreverse.

Input:

Basedontheabovedecisionsandanalysis,pleaseindicatethe3sfuturetrajectoryoftheegovehicle.

Drive-VLM:

Futuretrajectory:[(x1,y1),(x2,y2),…,(xn,yn)]

Input:

Picture:<img>

Pleaseindicatethescenecondition.

Drive-VLM:

Weather:Sunny.

Time:Day.

RoadEnvironment:Urban.

AlternativeLane:Ownlaneaheadimpassable,occupiedbyparked

vehicles,leftlanepassable,oncominglane,proceedwithcaution.

EgoLanePosition:Donotapply.

VisionEncoder

FeatureAlignment

Input:

Pleaselistthekeyobjects.

Drive-VLM:

Criticalobjects:<Twoconstructionworkers>

Box:<(x1,y1),(x2,y2)>

Matching

3DPerceptionResults

Input:

Matchedobjects:

Twoconstructionworkers

Thepositionandhistory

trajectoryofmatchedobjectsinBEVregion:<…>

Unmatchedobjects:NoneDescribetheobjectand

indicateitsinfluence.

Drive-VLM:

Characteristic:Garbagecleaninginprogressontherightlaneahead.

Influence:Blockingtherouteofourvehicle.

Input:

Ego-statesandhistoricaltrajectory:<…>

Basedontheanalysisof

sceneandcriticalobjects,determinethedrivingmetaactionsanddecisions.

Collaboration

Dual

System

Slow-Fast

3DPerception

MotionPredictionTrajectoryPlanning

TraditionalAVPipeline

*SubmittedtoCVPR24

https://openreview.net/forum?id=jL4YMzXYII

LLMDeployedOnNVIDIADRIVEOrin

LLaMA2-3B(BS=1,Input_len=128,Output_len=128)

PlatformConfigContextLatency(ms)DecoderPerf(tokens/s)

DriveOSLinux,OrinINT4(GPTQ)52.565.6

LLaMA2-7B(BS=1,Input_len=128,Output_len=128)

PlatformConfigContextLatency(ms)DecoderPerf(tokens/s)

DriveOSLinux,OrinINT4(GPTQ)73.1541.8

LiADSimResearch——StreetGaussians

Originalscene

Street-gaussianswapping

Originalscene

Street-gaussianswapping

Reallog

Camerasimulation

Originscene

Linrescene

Unisimscene

RenderingImages

Decomposition

Semanticmaps

Geometrymodel

PositionμRotationαOpacityRScale$

Point-basedRendering

BackgroundmodelComposition

Dynamicappearancemodel

)⊗

TimebasisSHbasis

……⊗

∑(

OptimizableTrackedboxes

Objectmodel

Scenerepresentation

3DGS[16]NSG[31]MARS[51]Ours

PSNR↑

29.95

30.23

31.37

34.54

PSNR*↑

17.74

22.05

23.07

25.16

SSIM↑

0.907

0.866

0.904

0.936

LPIPS↓

0.140

0.331

0.246

0.091

FPS↑

277

0.47

0.68

133

Table1.QuantitativeresultsontheWaymo[40]dataset.

Therenderingimageresolutionis1066°ø1600.“PNSR*”denotesthePSNRofmovingobjects.

*SubmittedtoCVPR24

https://openreview.net/forum?id=jL4YMzXYII

LiADResearch——BEV-CLIP:MultimodalDataRetrieval

Weightmatrix

BEV

Encoder

Θ

SharedCross-madalPrompt

Pedscrossing

crosswalk,

manycars……

Language

Textembedding

Θ

Encoder

LoRA

Weightmatrix

KGEmbedding

(a)(b)

Knowledgegraph

BEVCaptionGenerationHead

Contrastiveloss

(c)

Figure2.OverallstructureofBEV-CLIP.

(a)ProcessingofBEVandtextfeatures.Theimagefrom6surroundingcamerasaregeneratedintoaBEVfeaturebytheBEVEncoderwithfrozenparameters.Atthesametime,theinputtextembeddingisconcatenatedwiththekeyword-matchedKnowledgeGraphnodeembedding,andfedintotheLanguageEncoder

withLoRAbranchforprocessing.(b)Sharedcross-modalprompt(SCP),whichalignstheBEVandlinguisticfeaturesinthesamehiddenspace.(c)Jointsupervisionofcaptiongenerationandretrievaltasks.⊙denotesdotproduct.

1.81