基于物理条件约束的可信视觉生成大模型-培训课件

基于物理条件约束的可信视觉生成大模型Visual

generative

modelInputOutputVAE:

maximize

variationallowerboundVideo

generative

methods•

Thefieldofvideo

generationhasseenrapiddevelopment,

reachingseveralmilestones...VAE:

maximize

variationallowerboundGAN:

AdversarialtrainingFlow-based

models:

Invertible

transform

ofDiffusionmodels:

GraduallyaddGaussian

noisedistributionsandthenreverseDiffusion

for

visual

generation

(1)•

DenoisingDiffusion

Probabilistic

Models

(DDPMs)Diffusion

for

visual

generation

(2)•

Stochastic

Differential

Equations

(Score

SDEs)Key

Elements

of

visual

Diffusion

Models•

Pixel

diffusion

(originalinput)•

Latent

spacediffusion•

Unet•

TransformerSora,

breakthrough•

Consistency:consistencyin3Drendering,long-rangecoherence,

andobjectpermanence.•

Highfidelity.•

Surprisinglength:extended

videolength

capability(Sora:

1

minutevs.previous

systems:

seconds).•

Flexible

resolution:generation

ofvideosacross

various

durations,aspectratios,

andresolutions.Sora,

key

technologies•

TheDiTframework

by

Meta

(2022.12)is

designedfor

videoprocessing.•

Google's

MAGViT

(2022.12)focuses

onVideoTokenization.•

GoogleDeepMindintroduced

NaViT(2023.07)to

supportvariousresolutions

andaspectratios.•

OpenAI's

DALL-E

3

(2023.09)enhancesVideoCaptiongeneration

forimproved

conditioned

videocreation.Modeling

the

physical

world•

We

knowthat

itis

verycomplicated

real

physical

model.probabilistic•

bayesian

inference;•

probabilisticgraphical

models.deterministic•

mathematicalequations;•

physics

basedsimulation;•

control

theory.Modeling

the

physical

world•

We

knowthatitisverycomplicatedrealphysicalmodel.probabilistic•

bayesian

inference;•

probabilisticgraphical

models.deterministic•

mathematicalequations;•

physics

basedsimulation;•

control

theory.Key

elements

of

a

physical

world•

GivenaSora

demo(thewalkingwomanintheTokyo

street),thekey

elementsofaphysicalworld,inthegraphicalway...•

Appearance•

Geometry•

Lighting•

Motion&Animation•

AudioModeling

the

physical

world•

[CVPR]Gaussian-Flow:4DReconstructionwithDynamic3DGaussianParticleEspressoChick-ChickenSplit-CookieFlame-SteakModeling

the

physical

world•

[CVPR]Gaussian-Flow:4DReconstructionwithDynamic3DGaussianParticleIt

is

hard

to

model

the

physical

world•

In

fact,

theworld

ishard

to

modelina

probablistic

way.•

Sora

resource

consumption...–

1billionsofimages;–

1millionsofhoursofvideo

data;–

10trillionstokens

aftertokenizingimagesandvideos–

Training

with~5,000A100sinparallel.It

is

hard

to

model

the

physical

world•

Sora

failure

casein

geometryandappearance.It

is

hard

to

model

the

physical

world•

Sora

failure

case

inlighting.It

is

hard

to

model

the

physical

world•

Sora

failure

case

inmotionandanimation.It

is

hard

to

model

the

physical

world•

VideoMV:ConsistentMulti-ViewGenerationBasedonLarge

VideoGenerativeModel•

Geometricenhancementisstillneededfor

multi-viewimages.It

is

hard

to

model

the

physical

world•

VideoMV:ConsistentMulti-ViewGenerationBasedonLarge

VideoGenerativeModel•

Fromastatic

aspects,SVDisabletomodelmulti-viewimages.It

is

hard

to

model

the

physical

world•

Stag4D:Spatial-Temporal

AnchoredGenerative4DGaussians•

From

atemporalaspects...It

is

hard

to

model

the

physical

world•

STAG4D:

Spatial-Temporal

AnchoredGenerative4DGaussians•

Fromatemporal

aspects...It

is

hard

to

model

the

physical

world•

Ilya

Sutskever:

compression

is

generalization.•

Thebest

losslesscompression

for

adataset

is

thebestgeneralization

for

data

outsidethedataset.Apply

the

deterministic

conditions•

Different

representationsof

deterministicconditionsinthephysicalworld.•

Muchlessdata

andparameters!GeometryLightingMotion&AnimationApply

the

deterministic

conditions•

Thereare

two

ways

to

injectdeterministicinformation.deterministic#1deterministic#2Image

Human

Animation•

Champ:

Controllable

andConsistent

HumanImage

Animation

with

3D

Parametric

GuidanceImage

Human

Animation•

Champ:

Controllable

andConsistent

HumanImage

Animation

with

3D

Parametric

GuidanceImage

Human

Animation•

Champ:

Controllable

andConsistent

HumanImage

Animation

with

3D

Parametric

GuidanceImage

Por

trait

Animation•

Hallo:

Hierarchical

Audio-Driven

VisualSynthesisfor

Portrait

Image

AnimationImage

Por

trait

Animation•

Hallo:

Hierarchical

Audio-Driven

VisualSynthesisfor

Portrait

Image

AnimationImage

Por

trait

Animation•

Hallo:

Hierarchical

Audio-Driven

VisualSynthesisfor

Portrait

Image

AnimationDynamic

Protein

Structure

Prediction•

4D

Diffusion

for

DynamicProtein

Structure

Prediction

with

Reference

GuidedTemporal

AlignmentDynamic

Protei