DeepSeek弹性计算（DSec）：面向大规模高效智能体训练的沙箱基础设施（英文原文）

deepsee《

DeepSeekElasticCompute(DSec):

ASandboxInfrastructureforEffectiveAgenticTrainingatScale

JialiangHuang†‡,HongxuanTang†,JingchangChen†,YuxuanLiu†,YixiaoChen†,YuanCheng†,YiTao†,Jingli

Zhou†,YupengChen†,HaoyuChen†,JiaruiWang†,ShengkaiLin†,ChuqiZhang†,BryanLeeTeng†,LianGuo†,

ZheFu,WenjunGao,YisongWang,LiangZhao,ZehaoWang,ZiweiXie,YongqiangGuo,PeixinCong,Ziyi

Gao,ShuipingYu,HanweiXu,ZuofanWu,ZhizhouRen,YuyangZhou,BoweiZhang,ZhihuanHuang,Qihao

Zhu,LeiWang,TianleLin,HanYu,JiewenHu,DejianYang,ShuoYang,ShanghaoLu,ShaoyuanChen,Junjie

arXiv:2609.22978v1[cs.DC]19Sep2026

Qiu,ZhangliSha,YinminZhong,YongtongWu,ShiyuWang,WeiLiu,BingzhengXu,LonghaoChen,Qiushi

Du,YuzhenHuang,ShirongMa,YaohuiWang,MingshuChen,TongruiXiong,Y.C.Yan,HaowenLuo,Haofen

Liang,XiaokangZhang,WeihaoZeng,RunxinXu,PeiyiWang,JinhuaZhu,RuoyuZhang,WenkaiYang,Qi

Tang,JipingYu,TianYe,RuizhePan,HonghuiDing,XiaodongLiu,LingxiaoLuo,ZhihongShao,YuhanWu,

JibaiLu,WenLiu,HaolingZhang,JingchengHu,YaoyangYe,ChaofanLin,ZhaochenZhang,JiananTong,

HengxuWu,ZhihaoLi,YichengWang,LuyaoWang,YuzhuoBai,LingyueFu,RuifanXu,Y.Z.Wang,Zonglin

Li,MingqiWei,HaiyangShen,ChengyuanZhang,ChaoJin,ZiliZhang,R.H.Yang,XinboXu,JianZhou,

RuidongZhu,YuzheGuo,ZelunPan,ShaohengNie,ErhangLi,ShuhanLin,ZhengLiu,AnshuoChen,Zilong

Lyu,SinuoCao,RuiYu,ChuhaoWang,JunyiGuo,JunxiaoSong,KaifengChen,MenghaoYe,JunxianLi,Di

Wu,HaiyangMa,YilunWang,HaoranYang,YizaiCai,ShichunLiu,YipingWang,JunboSun,ShichengXu,

XiaoBi,YingHe,YichaoZhang,MingxingZhang‡,LiyueZhang*†,PanpanHuang,WenfengLiang

DeepSeek-AI‡TsinghuaUniversity

research@

Abstract

Large-scaleagentictrainingandevaluationwithlargelanguagemodels(LLMs)relyonisolated,statefulexecutionenvironmentsinwhichmodelsinspectrepositories,invoketools,executecommands,andinteractwithtask-specificservices.Theseworkloadscreatesandboxesinlargebursts,spanheterogeneousfunctionalityandisolationrequirements,retainstateacrosslonginteractions,anddrawfromlargeimagecorporawithlimitedreuse.Supportingthemthereforerequiresanelasticexecutionplatformratherthanasinglesandboxruntime.

ThisreportpresentsDeepSeekElasticCompute(DSec),aproductionsandboxplatformthatexposesFnCall,container,microVM,andfull-VMsandboxbackendsthroughaunifiedSDK.DSeccoordinatesplacementandlifecyclemanagementacrossthecluster,composesenviron-mentsfromindependentlyversionedlayers,combinesmemorysharing,reclamation,andCPUschedulingforhigh-densityexecution,andloadsimagedataondemandfromFire-FlyerFileSystem(3FS),acluster-widedistributedfilesystem.DSecisco-designedwiththereinforcementlearning(RL)framework,decouplesstatefulrolloutexecutionfrompreemptibleGPUtrain-ing,coordinatessandboxlifecyclewithtrainingtopreserverolloutstatewhilereclaimingidleresources,andmitigatesagentmisbehaviorsuchasrewardhacking.

Asingleproduction-scaleunitofDSecspansaround160nodes,servingabout3millionsand-boxesperday;inproduction,itsupportsover380,000concurrentsandboxesandsustainsover5,000sandboxcreationspersecond.Ourevaluationanddeploymentexperienceshowthatthesemechanismsreduceenvironmentsetupandimage-distributionoverhead,improvememory

efficiency,andpreservelatency-sensitiveperformanceunderhigh-densityovercommit.

*Correspondingauthor.†DSecprojectdevelopers.‡TsinghuaUniversity.

JialiangHuangisaPh.D.studentadvisedbyMingxingZhang.HecontributedtothisworkduringaninternshipatDeepSeek-AIunderthementorshipofLiyueZhang.

2

1.Introduction

RecentadvancesinfrontierLLMshavemadeagenticworkflowspracticalandwidelyadopted(

Guoetal.

,

2025

;

Jimenezetal.

,

2024

;

OpenAIetal.

,

2024

).Insteadofproducingasingletextanswer,anagenticmodelinteractswithanexecutionenvironment:itmaynavigatecodebases,calltools,executecommands,inspectfailures,andmodifyfiles,oroperatebrowsersanddesktopapplicationsthroughgraphicalinterfacesincomputer-usetasks(

Xieetal.

,

2024

;

Zhouetal.

,

2024

).Acrosstheseworkloads,themodeliteratesbasedonfeedbackuntilataskissolved.Thisexecutionmodelhasledtoagrowingecosystemofagenttoolsandorchestrationharnesses,suchasDeepSeekHarness(DSH)(

Shietal.

,

2026

),OpenCode(

Anomaly

,

2025

),andmulti-agenttrainingharnesses.Trainingreliableagentsrequiresreinforcementlearning(RL)atscale,inwhichmodelslearnthroughinteractionwithreal,isolatedexecutionenvironmentsratherthansolelyfromstaticinput-outputexamples.

Theagentictrainingpipelineencompassesenvironmentanddataconstruction,RLrollouts,rewardcomputation,policyupdates,andperiodicevaluation.Amongthesestages,RLrolloutandevaluationimposethehighestpressureonthesandboxplatformbecausetheyarelarge-scale,concurrent,andtightlycoupledwiththetrainingloop.InRL(

Guoetal.

,

2025

;

Ouyang

etal.

,

2022

),trainingproceedsasafeedbackloopwiththreestages.First,duringrollout,thecurrentmodelinteractswiththesandboxedenvironment:itreadsfiles,issuestoolcalls,executescommands,observesoutputs,andproducesatrajectoryforeachtask.Second,duringrewardcomputation,theframeworkscoresthetrajectoryusingnativeexecutionsignalssuchasexitcodes,stdout,testpassrates,ortask-specificverifiers.Third,duringpolicyupdate,theRLalgorithmupdatesthemodelparametersfromthecollectedtrajectoriesandrewards.Periodicevaluationfollowsasimilarexecutionpath,exceptthattheresultingtrajectoriesareusedtomeasuremodelcapabilityratherthantoupdateparameters.Recentsystemsfurtherpipelinegenerationandpolicyoptimizationthroughasynchronousrollouts,continuouslyreplenishingcompletedsamplestomaintainhighconcurrencyandmitigatelong-tailstragglers(

DeepSeek-AI

,

2026

).Foragenticworkloads,thisdesignkeepsmanystatefulsandboxsessionsinflightandmayinterruptandresumetheirassociatedrolloutsacrosspolicyupdatesorschedulerpreemptions,furtherincreasingtheplatform’sconcurrency,lifecycle-management,andstate-consistencyrequirements.

Foreachrolloutorevaluationtask,theplatformmustmaterializeanisolatedtask-specificenvironment,includingitsrepositories,dependencies,services,evaluationscripts,andcodingharnesses.Theenvironmentmustbecloseenoughtoarealmachinetorununmodifiedsoftwarestacks,packagemanagers,buildtools,browsers,emulators,andtask-specificservices.Arobust,high-throughputsandboxruntimeisthereforefoundationalforobtainingaccurateandverifiableRLandevaluationresults.

Agenticsandboxworkloadshaveseveralpropertiesthatshapetheplatformdesign:

(1)Rolloutandevaluationjobscreatesandboxesinaburstymanner.Asinglejobmayrequestupto32Ksandboxinstances,sotheplatformmustacceptandplacemanysandboxesconcurrently.Suchburstsmakehorizontalscalabilityasystem-widerequirementandrequiresharedservices,suchasschedulingandimagedistribution,toavoidcentralizedbottlenecks.

(2)Sandboxesmustrunathighdensity.Duringagentinteraction,asandboxoftenwaitsfortheLLMtogeneratethenextaction,soCPUusageissparseandnaturallysuitableforovercommit.Forinstance,inproduction,thisallowsasinglenodetohostupto800microVMsor3,200containers,butonlyiftheplatformcansafelyovercommitresourcesandmanagelifecyclepressureatnodescale.

3

(3)Agentsandboxesarestatefulandlong-lived.Themodelmaymodifyfiles,installdependencies,andstartservices,andlatertoolcallsdependonthisaccumulatedstate.SinceasandboxcanstayaliveacrossmanyLLMinteractionturns,memoryfootprint,guestpagecache,hostpagecache,andwritablestatemayremainpinnedlongaftertheCPUbecomesidle.Underhigh-densityovercommit,theseresidentcostsdirectlylimitclustercapacity,somemorysharingandreclamationbecomeimportantplatformrequirements.

(4)Agentworkloadsarehighlyheterogeneous.TheplatformmustcoverOJ-likescriptexecution,software-engineeringtasksoverfullrepositories,securitytasks,computer-usework-loads,mobiledevelopmentenvironments(e.g.,Android),andotherfull-systemenvironments.TheseworkloadsdiffersubstantiallyinCPUandmemorydemand,dependencyfootprint,re-quiredsystemfunctionality,andisolationstrength.Asinglesandboxabstractioncannotcoverallofthemefficiently.Forexample,lightweightfunctioncallsarepreferableforshortstatelesstasks,whereasvirtualmachines(VMs)arebettersuitedtoworkloadsthatrequireacompletecommercialoff-the-shelfoperatingsystem.

(5)Environmentdiversityishighevenwithinthesameworkloadclass.Trainingandevaluationcorporacontainmanytasks,andeachtaskmayrequireitsownrepository,depen-dencyversions,services,toolkits,evaluationscripts,orVMsnapshots.Asaresult,theplatformmustservealargenumberofdistinctimagesandenvironmentartifacts,withlimitedreuseformanyofthem.Underburstystartup,fetchingthesediversetaskimagesfromaregistrywouldconcentrateloadonthedistributionpath,inflatestartuplatency,andintroduceextraI/Othatinterfereswithalready-runningsandboxes.Inourablation,eagerimagepullingstretchescompletiontimeby1.7×,whileon-demandloadingreducescumulativediskwritesby57%.

(6)Agentexecutionisuntrustworthy.Agentsmaycorruptfilesystems,exhaustresources,orinterferewithsystemcomponents,potentiallydisruptingrolloutsorotherco-locatedwork-loads.Theplatformthereforerequiresfine-grainedaccesscontrolandmisbehavioranalysistocontainanddiagnoseagent-inducedfailures.

(7)Agentexecutionisinterruptible.GPUtrainingjobsmaybepreemptedwhilelong-runningrolloutsarestillinprogress.Theplatformmustthereforepreserveexecutionstateandsupportefficientrecoveryacrossinterruptions.

Thesepropertiesdefinetheroleofanagentsandboxplatform.DSecprovideselasticservicescaling,high-densityresourcemanagement,memorysharingandreclamation,multipleisolationmechanismsfordifferentworkloadclasses,scalableimagedistribution,andexplicitintegrationwiththetrainingframeworkforpreemption-saferesumption,task-specificnetworkpolicy,andagentmisbehavingmitigation.

TherestofthisreportpresentsDSecfromplatformabstractiontoimplementationandevaluation.

§2

introducesDSecfromtheuserperspective,includingsupportedworkloads,sandboxbackends,andoperatingscale.

§3

describestheend-to-endplatformarchitecture.

§4

characterizestheproductionworkloadandtheplatformchallengesitcreates.

§5

presentsthecoresystemmechanismsforenvironmentcomposition,imagedistribution,andhigh-densityresourcemanagement.

§6

describesco-designwiththeRLframeworkforenvironmentconstruction,statepreservation,resourcereclamationacrosspreemption,andtheanalysisofagentmisbehaviorwithtargetedaccess-controlmitigations.

§7

summarizesadditionalimplementationdetails.

§8

evaluatestheeffectivenessofthedesign,and

§9

discussesrelatedworks.

2.OverviewofDSec

Thischapterpresentstheuser-facingviewofDSec.Fromtheplatform’sperspective,usersarethetrainingframeworks,evaluationframeworks,anddata-constructionpipelinesthatcall

4

thesoftwaredevelopmentkit(SDK)onbehalfofresearchers;werefertothemcollectivelyasusersthroughoutthereport.ItcoverstheSDKentrypoint,thesandboxbackendsexposedbytheplatformandtheworkloadclassestheyserve,thelifecycleofasandboxsession,andtheoperatingscaleoftheproductiondeployment.

2.1.SDKEntryPoint

UsersaccessDSecthroughlibdsec,aPythonclientlibraryforthesandboxservice.libdsecgivesusersaunifiedSDKentrypointforcreatingandoperatingsandboxes,whilestillrequiringthemtochoosethesandboxbackendappropriateforthetask.Atypicalrequestspecifiesthesandboxtype,imageorenvironmentidentifier,CPUandmemorylimits,lifetimesettings,networkrules,andinitialusercontext.Aftercreation,theusercanexecuteshellcommandsortoolcallsandcollectcommandoutputsandreturnstatus.

List.1

showsaminimalcontainersession:theclientconnectstotheserviceendpoint,requestsasandboxwiththedesiredresourceandnetworkpolicy,runsacommand,andreleasesit.

Listing1|Aminimalsandboxsessionthroughlibdsec.

client=DSecClient()

await

client.open

()

args=DSecContainerRunArgs(

container_image="registry.../sphinx-9658:official",

memory_limit_mb=4096,cpu_cores_limit=4,

ttl_running_stop=300,#idletimeout

network_rules={"npm":False,"pypi":True},

init_user="root",

)

sandbox=await

client.run_container

(args,timeout=120)

result=await

sandbox.run_shell

("echohelloworld")

awaitsandbox.stop()

Inthisexample,thenetworkrulesallowaccesstoPyPI(pypi=True)butdenyaccesstoNPM(npm=False).Thisfine-grainednetworkcontrolisdiscussedindetailin

§6

.Thisinterfaceisintentionallynotafullsemanticabstractionoverallbackends.Functioncalls,containers,microVMs,andfullVMshavedifferentstartupcosts,isolationboundaries,filesystemsemantics,andoperating-systemcapabilities.libdsecprovidesaunifiedaccesspathandasimilaroperationalmodel,butthecallerremainsresponsibleforselectingabackendthatmatchestheworkload.

2.2.SandboxBackends

Sandboxruntimesfaceafundamentaltension:strongerisolationandmorecompletesystemfunctionalityusuallycomewithhigherstartuplatencyandresourceoverhead.Sincenosinglesandboxabstractionfitsallagentictasks,DSecsupportsmultiplebackendsspanningthistradeoffspace.

Tab.1

summarizestheirtypicalfit.

FnCalltargetsshort,statelesstaskssuchasOJworkloads,codecompilation,serverlessprograms,GPUkernels,andutilitycode.FnCalltasksruninreusableprecreatedCPUorGPUcontainers,avoidingper-invocationprovisioningoverhead.ForGPUworkloads,FnCallsupports(i)sharedmode,wheremultiplecontainersshareaGPUinstance,maximizingutilizationforlightweightworkloads,and(ii)exclusivemode,whereonecontainerreservesaGPUinstanceduringitslifecycleforperformance-sensitivetasks(e.g.,operatorevaluation).Containersarethemainbackendforsoftware-engineeringandgeneraltool-useworkloads.Theyprovidefaststartupandhighpackingdensity,andtheyruntheLinuxsoftwarestacksusedbymostrepository-leveltasks.Theirmainlimitationisthattheysharethehostkernel,whichisnot

5

Table1|TypicalworkloadfitacrossDSecsandboxbackends.

Characteristic

FnCall

Container

MicroVM

FullVM

RuntimePerformance

●●●

●●O

●亻OO

●OO

Dependencyfootprint

OOO

●●●

●●●

●●O

Isolationlevel

OOO

●●O

●●●

●●●

FullOSfunctionality

OOO

●OO

●●O

●●●

Resourceoverhead

OOO

●OO

●●O

●●●

Scenarios

OJ-liketasks

SWE

Security

COTSOS

GPUkernelexec

Tooluse

Computeruse

Graphics

More●=higherdemand.

alwaysappropriateforsecurity-sensitivetasks.FirecrackermicroVMs(

Agacheetal.

,

2020

)provideastrongerisolationboundarywhileretainingLinuxcompatibility.Theyareusefulforsecurity-sensitivetasks,strongertenantisolation,andworkloadsthatneedaVMboundarywithLinuxcompatibility.Thiscomesathighermemoryoverheadandslowerstartupthancontainers.FullVMbackendscoverworkloadsthatrequireacompletecommercialoff-the-shelfoperatingsystemenvironment,suchasAndroidVMsthroughQEMU(

Bellard

,

2005

),aswellasthosethatrequireaGUIorgraphicsrendering.Thesebackendshavethehighestresourceoverhead,buttheyarenecessaryfortasksthatdependonOS-specificAPIs,mobileruntimebehavior,orfull-systemexecution.

Inproduction,containersandmicroVMsdominatebothinstancecountandresourcecon-sumption.FnCallservesalargenumberoflightweightinvocationswithasmallsetofresidentenvironments,whilefullVMbackendscoverspecializedbutimportantworkloadclasses.

2.3.User-VisibleLifecycle

Althoughthesupportedbackendsdifferinternally,usersseeaunifiedhigh-levellifecycle.First,thecallercreatesasandboxbyselectingabackendandspecifyingtheenvironmentartifact,resourcelimits,lifetimepolicy,andnetworkpolicy.Theenvironmentartifactvariesbybackendandworkload.ForcontainersandmicroVMs,itisabaseimagetogetherwithtask-specificworkspaceandtoolkitlayers,whichtheplatformcomposesintotherunningenvironment.ForfullVMworkloads,itisapreparedVMimageorsnapshot.ForFnCall,itisataskspecificationcontainingthetasktype,dependencyfiles,andthecodeorscripttorun.Theseartifactsbecomethebasisfortheenvironmentcompositionandimage-distributionmechanismsdiscussedlaterinthereport.Second,theplatformpreparestheenvironmentandmakesitreadyforinteraction.Third,theuserissuescommandsortoolcalls,observesoutputs,andrunstask-specificchecksortests.Asandboxisstatefulthroughoutitslifetime:fileedits,installeddependencies,andstartedservicespersistacrosscalls,solatercommandsobservetheeffectsofearlierones.BecauseasandboxstaysaliveacrossmanyinteractionturnswhileitsCPUisoftenidlebetweenthem,itsresidentstateremainspinnedlongafterthelastcommand,oneofthehigh-densitychallengescharacterizedin

§4

.Finally,thesandboxisstoppedexplicitlyorreclaimedonceitstime-to-liveelapses,sothatidleorabandonedsessionsdonotholdresourcesindefinitely.

2.4.DeploymentScale

DSecisdeployedacrossmultiplescaleunitsthatsharea3FS(

DeepSeek-AI

)distributedfilesystemdeploymentforbaseimagesandworkspacestorage.Withinonescaleunit,theplatformspansnearly160CPUnodeswith30Kcoresand~250TBofDRAM.Itmanagespetabytesof

6

layersandimages.Onatypicalday,asinglescaleunitservesabout3Msandboxinstances,withpeakconcurrencyreaching~380Kandacreationrateexceeding5,000instancespersecond.

Thesenumbersareimportantforunderstandingtherestofthereport.DSecisnotasinglesandboxruntimeorathinwrapperaroundcontainers.Itisaproductionexecutionplatformthatmustcombineuser-facingsandboxabstractions,backend-specificruntimes,scalableimagestorage,high-densityresourcemanagement,andtraining-frameworkintegration.

3.PlatformArchitecture

§2

presentedDSecasusersseeit:anSDK,asetofsandboxbackends,andasessionlifecycle.ThischapterturnstotheplatformbehindthatinterfaceanddescribeshowarequesttravelsfromtheSDKtoarunningsandboxandwhichcomponentsitpassesthrough.Wedescribethearchitectureintermsofcluster-levelservicesandthesandboxruntime.Cluster-levelservicesproviderequestingress,identityandaccessmanagement,sandboxplacement,andaviewofclusterhealthandload.Thesandboxruntimehandlesnode-localadmission,sandboxcreation,execution,andresourcereclamation,relyingon3FSforimagedata.

3.1.Overview

Watcher

WorkerMonitor

EROFSImagesOverlayBDImages

Cluster-levelservices3FS

IAM

IdentityandAccessManagement

APIServer

IngressProxy

PlacementEngine

SandboxScheduler

libdsec

UnifiedPythonSDKforSandboxTrainingCluster

Proxy

MicroVM

Firecracker

Isolation:AppArmor+eBPF

Edge

LifecycleManagementCreate/Delete

Sandboxbackends

Proxy

Container

DockerQEMUVM

FnCall

Pre-created

Containers

GPU

QEMUVM

Aether

SessionManagement

Chronus

Exec/File/

HTTP

ManagementRequestDataRequest

Proxy

FullVM

GPU-PV

Per-noderuntime

Figure1|DSecarchitecture.EachproxymediatescommunicationbetweenacontainerorVMsandboxandtherestoftheplatform.FnCallfollowsaseparateexecutionpathanddoesnotusethisproxy.

Atahighlevel,asandboxcreationrequestisfirstsenttoIAMforauthenticationandau-thorization.Onceauthorized,therequestproceedstotheplacementengine,whichselectsatargetnodeusinghealthandloadinformationcollectedbythewatcher.Afterplacement,theapiserverforwardstherequesttotheedgeonthatnode.Theedgethencheckslocalcapacity,creatingthesandboxwiththerequestedbackendifcapacitypermitsandrejectingtherequest

7

otherwise.Imagedataneededbythesandboxisstoredin3FSandfetchedondemandduringstartupandexecution.Container,microVM,andfullVMsandboxesrunaper-sandboxproxy(aether)andoneormorechronusinstancesforcommandexecution,filesystemaccess,andotherruntimeoperations.Afteroneofthesesandboxesisrunning,itsoperationsareroutedthroughtheapiserver,edge,aether,andchronus.FnCall,bycontrast,usesneitheraethernorchronusandfollowsaseparaterequestpath:thesubmittedtaskisexecuteddirectlyinaprecreatedcontainer,followedbybest-effortcleanupoftaskstate.

3.2.Cluster-LevelServices

Cluster-levelservicesmanageaccesstotheplatformandcoordinatesandboxrequestsacrosscomputenodes.TheycompriseIAM,theapiserver,theplacementengine,andthewatcher.

IAM.IdentityandAccessManagement(IAM)authenticatescallersandauthorizesallmanage-mentrequeststoDSec.Forexample,requeststocreateordeletesandboxesorchangeauser’sresourceorconcurrencylimitsmustpassIAMchecksbeforeexecution.Aprincipalistheuserorserviceidentityassociatedwithamanagementrequest.IAMusesprojectstodefinescopesforresourcemanagementandaccesscontrol.Withinaproject,accesspoliciesspecifywhichprincipalsmayperformwhichmanagementoperationsonitsresources,whileresourcequotaslimitresourceconsumption.

Wesupportmulti-levelprojectnestingratherthantheflatortwo-levelhierarchiescommonincloudplatforms.Authorizedprincipals,includingagentsandharnesses,cancreatesubprojects,delegatepartoftheparentquota,andgrantmanagementpermissionswithinthem.Delegationisboundedbytheparent:aprincipalcannotgrantpermissionsitdoesnothold,andsubprojectpoliciesandquotascannotexceedtheparent’saccess-controlorresourcelimits.HumansandagentsusethesamemanagementAPIandauthorizationmodel.

APIServer.Theapiserverservesastheingressproxyforthesandboxcluster.TrainingandevaluationcodeinvokeslibdsecfromtrustedGPUservers,whilesandboxesexecuteuntrustedmodel-generatedcodeandmayaccessexternalnetworks.Thetwosidesarethereforenetwork-isolated,withtheapiserverastheonlypermittedcommunicationpath.Allsandboxrequests,includingcreation,commandexecution,andstreamingI/O,passthroughthisingress.Theapiservermaintainsnoper-sandboxstate.Itperiodicallyrefreshesthesetofedgenodesfromthewatcher,whileeachsandboxIDencodesitsowningedge.Anyapiserverinstancecanthereforeresolveandforwardarequestdirectlytothetargetedge,enablingtheingresstiertoscalehorizontally.

PlacementEngine.Theplacementengineselectsahostnodeforeachnewsandbox.Placementproceedsintwostages:filteringandranking.Thefilteringstageretainsonlyhealthynodesthatprovidethebackendandhardwarecapabilitiesrequiredbytherequest.Forexample,arequestforaGPU-enabledsandboxisrestrictedtonodesequippedwiththerequiredGPUs.Therankingstagerandomlysamplesafeweligiblenodesandselectstheleastloadedamongthem.

Watcher.Theplacementengine’sdecisionsareonlyasgoodasitsviewofthefleet,whichthewatcherprovides.Thewatcherperiodicallyprobesthehealthofeachedgeandhostandcollectsscheduling-relevantstate,suchasthenumberofrunningsandboxesacrossbackendtypes,brokendownperedge,peruser,andpertask.Theplacementengineperiodicallypullsthisstatefromthewatcherandusesthelatestviewwhenevaluatingnewcreationrequests.

Pleasenotethatneithertheplacementenginenorthewatcherrequiresdurablestate.Theplacementenginekeepsnosandboxexecutionstate,andthewatchercanrebuilditsfleetview

8

afterarestartbypollingtheedgesagain.Thismakesplacementengineandwatcherinstanceseasytoaddorreplacewithoutacostlyrecoverystep.

3.3.SandboxRuntime

Thesandboxruntimecreatesandoperatesindividualsandboxesandmanagestheirresources.Itincludesedge,aether,andchronus,andrelieson3FSforsharedimagestorage.

Edge.Eachnoderunsanedge,aper-machinecomponentthathandlescreationrequestsfromtheapiserverforcontainer,microVM,QEMU-basedfullVM,andFnCallbackends.Beforeacceptingacreationrequest,theedgechecksthenode’scurrentcapacityandrejectstherequestifcapacityisinsufficient.Thisnode-localadmissioncheckcomplementstheplacementengine’splacementdecision,whichisbasedonperiodicallyrefreshedclusterstate.Duringcreation,theedgeprovisionsstorage,appliestheeBPF-basednetworkpolicy,andlaunchestheruntime.

FnCallandcontainersruninsideQEMU/libvirtVMsratherthandirectlyonthehost.TheVMprovidesanisolatedkernelandnetworkstackandservesasanadditionalsecuritybound-arybetweenuntrustedcontainersandthebaremetal.Tobettersupportgraphics-intensiveworkloads,suchascomputer-useGUIapplications,browsers,videogames,and3Drendering,weleveragepara-virtualizedGPUinterfacesofthehosthypervisor(e.g.,virtio-gpu).WithinthefullVM,wesupportbothworkloadswhosegraphicsAPIsarenativelycompatiblewiththehostOS,aswellasthosewhoserenderingstackscanbetranslatedintohost-nativeAPIsthroughcompatibilitylayerssuchasDXVK(

DXVK

,

2018

).

Besidestrackingthesandboxlifecycle,edgecoordinatesdiskandmemorysnapshotsandreleasesnode-localresourceswhenthesandboxstopsoritsTTLexpires.

Aether.ContainerandVMsandboxesrunaether,across-platformproxythatestablishesacommunicationchannelwiththeedge.Thisedge-to-aetherchannelusesaplatform