MLPerf基准套件技术概述

Demystifying

MLPerf

Benchmark

SuiteMLPerf基准套件技术概述AgendaTarget

AudienceAbout

usMLPerf

introductionHowto

startTarget

AudienceAnyonewhohasinterestonDeepLearning

codingAnyonewhowanttohaveareferenceorbaselineforsystem

purchaseGPUCPU/MemoryFPGAand

othersCloudSolution

providersEX.Dell,publishingbenchmarkson

server/storageAnswering

RFPsAnyonewholiketocontributeinMLPerf

communityWorld-classinfrastructureintheInnovation

LabZenithTOP500-classsystembasedonIntelScalableSystemsFramework(OPA,KNL,Xeon,

OpenHPC)424nodesdualIntelXeonGoldprocessors,

Omni-Path

fabric.•+160IntelXeonPhi(KNL)

servers.Over1PFcombined

performance!#265onTop500June2018,1.86PFtheoretical

peakLustre,IsilonH600,IsilonF800andNSS

storageLiquidcooledandair

cooledRattlerResearch/developmentsystemwithMellanox,NVIDIAandBright

Computing88nodeswithEDRInfiniBandandIntelXeon

Goldprocessors32xPowerEdgeC4140nodeswith4xNVIDIA

GPUsOther

systems32nodeAMDcluster,storagesolutions,

etc.13Kft.2lab,1,300+servers,~10PBstoragededicatedtoHPCincollaborationwiththe

communityHPCandDLEngineering-whatwe

doDesignandbuildsystemsforHPC

andDeepLearning

workloads.Systemsincludecompute,

storage,network,software,services,

support.Integrationwithfactory,software,

services.Powerandperformanceanalysis,

tuning,bestpractices,

trade-offs.Focusonapplication

performance.Vertical

solutions.Researchandproofofconcept

studies.Publishwhitepapers,blogs,

conferencepapers

()Accesstothesystemsinthe

labMLPerf

introductionWhatisMLPerfA

open

sourced

ML

benchmark

suite

for

measuring

performance

of

ML

frameworks,

ML

hardwareaccelerators,andMLcloud

platforms.CoverdifferentDL

domainsPropermetrics(trainingtime,accuracy,

latency)Real

datasetsWhatisthegoalof

MLPerfFairandusefulbenchmarksformeasuringtrainingandinferenceperformanceofML

hardware,software,and

services.WhatisthetypesofMLPerf

runsSubmission–submitter

onlyRe-run–Anyonecan

runWhat’sincludedin

MLPerfTrainingand

InferenceMLPerf

introduction(cont.)4Categories:AvailableOn-premisePreviewRDI2

Divisions:Closed

:Intendedtocomparehardwareplatformsorsoftwareframeworks“apples-to-apples”andrequiresusingthesamemodelandoptimizerasthereference

implementationOpen:Intended

to

foster

faster

models

and

optimizers

and

allows

anyML

approach

that

can

reach

the

target

quality.2Benchmark

suites:DatacenterEdgeExample:

Inference->Available->closed->DC->DLRM->Offline->99%MLPerfTrainingAvailableOpenClosedOn-premiseOpenClosedPreviewOpenClosedRDIOpenClosedInferenceAvailableOpenClosedDatacenterRESNET50ServerOfflineDLRMServerOffline99%99.9%BERT99%99.9%SSD-RESNET343D-UNETRNN-TEdgeOn-premiseOpenClosePreviewOpenClosedRDIOpenClosedMLPerf

TrainingTheMLPerftrainingbenchmarksuitemeasureshowfastasystemcantrain

MLmodels.Latestversionis

v0.7More

details:/mlperf/training_policies/blob/master/training_rules.adocAreaBenchmarkDatasetQuality

TargetReferenceImplementationModelVisionImage

classificationImageNet75.90%classificationResNet-50

v1.5VisionObject

detection(light

weight)COCO23.0%

mAPSSDVisionObject

detection(heavy

weight)COCO0.377Boxmin

APand0.339Maskmin

APMaskR-CNNLanguageTranslation(recurrent)WMT

English-German24.0Sacre

BLEUNMTLanguageTranslation

(non-recurrent)WMT

English-German25.00

BLEUTransformerLanguageNLPWikipedia2020/01/010.712

Mask-LMaccuracyBERTCommerceRecommendation1TBClick

Logs0.8025

AUCDLRMResearchReinforcementlearningGo50%winrate

vs.checkpointMiniGo(based

onAlphaGo

paper)MLPerf

InferenceTheMLPerfinferencebenchmarkmeasureshowfastasystemcanperform

MLinferenceusingatrained

model.Latestversionis

v0.7Benchmark

suitesDatacenterServer/OfflineEdgeSingleStream/Multiple

streamMore

details:/mlperf/inference_policies/blob/master/inference_rules.adocAreaTaskModelDatasetQSL

SizeQualityServer

latencyconstraintVisionImageclassificationResnet50-v1.5ImageNet(224x224)102499%ofFP32(76.46%)15msVisionObject

detection(large)SSD-ResNet34COCO(1200x1200)6499%ofFP32(0.20

mAP)100msVisionMedical

imagesegmentation3D

UNETBraTS2019(224x224x160)1699%of

FP32and99.9%

ofFP32

(0.85300mean

DICEscore)N/ASpeechSpeech-to-textRNNTLibrispeechdev-clean(samples<

15seconds)251399%ofFP32

(1-WER,whereWER=7.452253714852645%)1000

msLanguageLanguageprocessingBERTSQuAD

v1.1(max_seq_len=384)1083399%ofFP32and99.9%ofFP32(f1_score=90.874%)130msCommerceRecommendationDLRM1TBClick

Logs20480099%of

FP32and99.9%ofFP32(AUC=80.25%)30

msStarting

pointJointheMLPerf

community(/get-involved/)Emaildistribution

listJointhe

forumJointheworking

groupWeekly

meetingsAttendcommunity

meetingsMLCommonsSuggestionsonrunthebenchmarkFor

re-run:

Use

final

code

from

specific

submitter,

not

the

reference

codeEx.

/mlperf/training_results_v0.7/tree/master/DellEMC/benchmarks/resnet/implementations/mxnetUsethereferencecodeonlyifyouare

submittingStepsfora

runPrepareHWanddriverGettheraw

datasetsPreprocessthedatasetsSetuprunning

environment–Docker

imagesRunrequiredtimesandparsing

resultsCompliancetestandsubmission

checkerEx./mlperf/training_results_v0.7/tree/master/NVIDIA