2026英伟达GTC大会：利用PhysicsNeMo加速机器人中的柔性触觉传感器仿真

利用

PhysicsNeMo

加速机器人中的柔性触觉传感器仿真Juana

Du,

Lia

LiangAgenda•WhyTactileSimulation?•What

is

PhysicsNeMo?•Contact

InteractionSimulation•Mesh

Rigidbody

Modeling•

PerformanceWhy

Tactile

Sensor

Simulation•

Provides

real-timefeedbackduringobject

interaction•

Enables

preciseforcecontroland

slip

detection•

Essentialfordexterous

manipulationtasks•

Complements

vision

in

occluded

environmentsTactilesensing

iscritialforcontact-rich

manipulationWhySimulatingTactile?•Real-worlddatacollection

isexpensive–

complex

hardware

setups,fragilesensors,anddifficult

large-scaleexperiments.•

Simulationaccelerates

research–

high-performance

GPUsenable

large-scale,physicallyaccuratevirtualtesting.Three

KeyChallenges•ModelingComplex

Physics–

Largedeformations,

nonlinear

materials,andcoupled

optical

processesarecomputationallyexpensive•

Speedvs.AccuracyTrade-off–

High-fidelity

FEM

istooslowfor

learning;simplified

modelssacrifice

precision•

Generating

RichSignals–

Producing

realistictactile

imagesandforcefieldsacrossdiversescenarios

remainsdifficultTactileiscritialforcontact-rich

manipualtionThree

key

challengesfortactilesimulationCoreChallengeTactilesensorcontact

involveselasticdeformation

and

large-scale

shape

changes,

making

it

difficultfortraditional

methodsto

balanceaccuracyandspeed.ExistingApproaches•Finite

Element

Method(FEM)–The

gold

standard

for

soft-body

deformation

simulation,

but

extremelycomputationallyexpensive.Scientific-grade

FEMsimulators

run

orders

of

magnitude

slowerthan

real-time,andevensimplifiedversionsfor

robotics

cannot

achieve

real-timeperformance.•

Rigid-BodyApproximation•Atwo-step

approach:firstsimulate

rigid-bodycontact

to

obtain

interaction

forces,

then

estimate

maximum

deformation

usingstiffnesscoefficients.•Some

research

adoptssoftcontact

models

to

improve

static/dynamic

modeling

accuracy,

but

existingsolutionsfailtofully

leverage

parallelcomputingcapabilities.•

PhysicsAI

Methods•Graph

neural

networks

learn

mesh-based

physicssimulationfrom

data.

Forexample,MeshGraphNets

can

predictdynamicsforvarious

physicalsystems(aerodynamics,

structural

mechanics,cloth)

and

run

1-2ordersof

magnitudefasterthantraditional

FEM.Contact

InteractionSimulationPhysicsNemoWhatcan

PhysicsAIenable?•

Near-realtimeemulation•

Highfidelity

sim•

Representativeofthe

highdimensionalgeometryandparameter

designspaceReservoirSimData

CenterCoolingExternalAeroDevelopingAI

modelsforengineeringandscienceapplications•

Toolstodevelopsolutionsthatobeyfirst

principles/

domain

knowledge•PerformantAIstackfor

real-world

problemscale•Modelarchitectures

andtraining

pipelinestunedforCAEto

accelerate

adoptionofAIUnlockingacceleratedsimulationswithAI•

AI

modelscan

runasimulation

1000xfasterthantraditional

numerical

solvers•Designcycles

reducedtosecs

from

hours•Enabling

moresimulationsfor

betterdesigns.Adopted

by

ISVsand

LHAs•Forsolutiondevelopersacross

Engineering

(CAE,

EDA)

and

Science

(Weather,

Nuclear

Physics)•Ex:

Luminary–Timeto

marketfromyearto

2

months•Ex:SimScale–Timeto

marketfrom

1yearto3

months*NVIDIA

PhysicsNeMoNVIDIA’sAIframeworkfor

developing

Physics-AI

modelsNIMs

Inference

pipelinesModel

architectures

Interfaces

Training

recipesDistributed

module

(Model,

DataandDomain

parallel)DALI

GPUOptimized

PyTorchWarpGeometric

modulePhysicsAI

modulePhysicsNeMo

Framework•Enterprise

grade

framework/toolkit

for

building

physics

AImodels•Built-in

physics

AI

training

pipelines

for

CAE•

Modelarchitecturestunedfor

PhysicsAI•

GPUaccelerateddata

pipelinesforengineering

data

formatsanddata

structures•

Utilitiesfor

injecting

Physics–

PDEs,

BCs,Geometry

constraints•Optimal

and

Scalable

training

pipelines•Memory

optimized

training

pipelines

and

modelarchitectures/layers•Scale

to

multi-node

systems

out

of

the

box–

dataand

model

parallel•Reference

AI

enhanced

sample

applicationsWhatis

PhysicsNeMo?Open-Source

Frameworkfor

ISVstodeveloping

PhysicsAIalgorithms/NVIDIA/PhysicsNeMoAIDEVELOPERSDistributedTrainingModel

ExplorationInference-AISurrogate

ModelMulti-domainsupportBuild

physics-ml

modelsfor

CFD,

HeatTransfer,Structural,

Electromagnetics,

MolecularDynamicsExplicitfirst

principlesPhysicsNeMo

FrameworkOptimizedTrainingAcceleratetrainingandthroughput

by

parallelizingthemodelandthetrainingdataacross

multi-node.SOTAModelArchitecturesEasily

explore

physics-ml

modelarchitectures–

NeuralOperators,

PINNs,GNNs,

Diffusion

Models.SupportNVIDIAAI

Enterpriseandexperts

byyoursideto

keepprojectsontrackEngineers&

ScientistsSimulationWorkflowModelDevelopmentTraining

DatasetPhysicsNeMo

FrameworkInductivebiasPhysicsconstrainedFully

physics

drivenFully

data

drivenDiverse

PhysicsAIapproaches:•fully

Physics

drivenAI

modeling•fully

data

driven

AI

modeling•

hybrid

(data+

Physics)AI

modelingFamiliesoftunedarchitectures•

NeuralOperators•

Transformers•

GNNs•

Unets•

Diffusion

Models•

PDE

informed

Neural

Networks

…

(List)Tailoredfordeveloping

PhysicsAI

modelsExploringdiverse

modelingapproaches•

Training

recipes

as

starting

reference

samples•ExternalAero,

Datacentercooling,

Chip

cooling,

…

…

.PhysicsDataTraininga

DiffusionTransformer

ModelDecrease

latencyat

highresolutionfor

bothtrainingand

inference.Trainand

evaluate

significantly

larger

resolutiondata•

`ShardTensor`

is

a

PhysicsNemo

utilityfor

buildingdomain-parallelapplications.•

ShardTensor:o

Combinesthedata,

metadata,and

deviceorchestration

concepts

intoone

object.o

Interoperateswith

Pytorch's

FSDPframeworktoenable

multilevel

parallelismo

Leverages

Pytorchsupportforavarietyof

operations(tensorops,

reshaping,

reductions)withextensions

in

PhysicsNeMoto

enablecriticaloperations

(ex:Convolutions,Attention,GroupNorm,etc).Multi-level

parallelismforenterpriseengineeringscalesolutionsPhysicsNeMoShardTensorContact

InteractionSimulation

DeepDiveContact

InteractionSimulation

DeepDiveHowtactilesensorsworkin

Realvs.

SimulationIn

RealStep1:

PhysicalContactStep2:SignalGenerationStep3:

ContactforcecalculationDeformation

generates

Signal

changes

Signal

mapped

to

Contact

forcesContact

interactiongenerates

DeformationContact

interactiongenerates

Deformationand

ForcesStep1:

PhysicalContactHowtactilesensorsworkin

Realvs.

SimulationThis

Talk

focus

on

Step

1In

SimulationApproach

1Approach2Step2:TactileSignalGenerationDeformation/3D

info

mapped

to

signals

(usually

required

for

VBTS)Direct

readingsfromStep1withContactSensor

APITactileSimulation:

NumericalSolversDiffTactileArchitecture

–

PaperTaccelArchitecture

–

PaperTacSLArchitecture

–

Paper●Mesh-basedsimulationsare

central

to

modeling

inmanydisciplinesacrossscienceandengineering.●Meshrepresentations

support

powerful

numericalintegrationmethods●GNNsare

a

natural

choice

for

data-driven

approaches

–itsmesh-basedandcancopewith

geometryirregularitiesandmulti-scale

physics.●Forwardmodel–

Predictvariablesofthe

mesh

at

timet+1givencurrentmeshat

t

and

history

of

previousmeshes.TactileSimulation:SensorandObject

are

modeledastetrahedra

meshes,

the

dynamics

can

be

accelerated

with

GNNGraphNeural

NetworksAI

for

acceleration

while

leveraging

the

meshRef:Pfaff,Tobias,et

al."Learning

mesh-based

simulation

with

graph

networks."

International

conference

on

learning

representations.

2020.Meshand

Rigidbody

ModelingDatasetsplitEpisodeFrametrain28925684val585072test676045Total41436801WholeScene(Liberoasset)Contact-rich

PartGelsightSensor

Pad613vertices,2205tetrahedraTask

SetupBowl502vertices,Attachment208

pointsSimulator:

libuipc860surfacetrianglesGraph

Neural

NetworksNetwork

DefinitionNodetypeNode

numberSensor

mesh

node613x2Bowl

affine

rigid

body(mass,

inertia,

affinetransformation)1Bowlcontact

nodevariousStaticcontact

node

(ground)variousControl

node

(attachment)208x2EdgetypeEdge

numberSensor

mesh

edge12988Sparse

bowledge(contact

node,com)variousContact

edge(pad/

bowl/ground)variousControledge416Graph

Neural

NetworksNetwork

DefinitionPerformanceTrainingandAccuracy•

Loss:sum

of

MSE

ofsensor

node

lossand

object

node

loss•

Trained

onasingle

L20,with

batch_size

128o

Thegraph-parallel

MeshGraphNet

implementation

in

PhysicsNeMoscalesto

multiple

nodes.Thedistributed

message

passing

isoptimizedfor

memoryand

performance.•

L2

error

metricsfor

accuracy

measurementPredict

movement

ErrorPredictvs.GT

(node)Predictvs.GT

(object

pos)Train

lossvs.Val

lossInferenceSpeed

benchmark•F