2026KimiK3技术报告英文版

KIMIK3:OPENFRONTIERINTELLIGENCE

TECHNICALREPORTOFKIMIK3

ABSTRACT

WeintroduceKimiK3,a2.8TparameterMixture-of-Expertsmodelwith104billionactivatedparameters,nativevisioncapabilities,anda1-million-tokencontextwindow.KimiK3isbuiltonKimiDeltaAttention[64]andAttentionResiduals[58],whichimproveinformationflowacrosssequencelengthandmodeldepth.TogetherwithStableLatentMoE,whicheffectivelyactivates16of896routedexpertspertoken,andrefinedtraininganddatarecipes,theseadvancesyieldanapproximately2.5×improvementinoverallscalingefficiencyoverKimiK2[59].Post-traininghighlightsreinforcementlearningacrossgeneral,agentic,andcodingdomainsandmultiplereasoning-effortlevels,enablingcompositionalgeneralizationandrobustlong-horizonexecution.At2.8Tscale,KimiK3issupportedbyinfrastructureadvancesinmultipleareas:algorithm–systemco-designforKDA,perfectlybalancedexpert-paralleltrainingwithefficientmemorymanagement,million-tokenagenticRLwithpersistentrolloutandsandboxstates,anddeploymentinnovations.

ExtensiveevaluationsshowthatKimiK3achievesfrontier-levelperformanceacrosslong-horizoncoding,agentic,knowledge,reasoning,andvisiontasks.Whileitsoverallperformancestilltrailsthemostpowerfulproprietarymodels,namelyClaudeFable5andGPT-5.6Sol,KimiK3consistentlyoutperformsotheropenandproprietarymodelsevaluatedinoursuite.WereleasethefullKimiK3modelweightstofacilitatefutureresearchandacceleratethebroaderdeploymentandadoptionoffrontierintelligence.1

Figure1:KimiK3mainresults.

1

https://huggingface.co/moonshotai/Kimi-K3

2

KimiK3:OpenFrontierIntelligenceTECHNICALREPORT

1Introduction

FormuchofthedevelopmentofLargeLanguageModels(LLMs),scalingmeantinvestingmorecomputationbeforedeploymentbytraininglargermodelsonmoredata[55,46].Theriseofreasoningmodelshasestablishedtest-timecomputationasasecondaxisofscaling:OpenAI’so-seriesscalesreinforcementlearningandtest-timereasoning[85,84];Anthropic’sextended-thinkingmodelsallocateadaptivethinkingbudgetsandinterleavereasoningwithtooluse[6,7];DeepSeek-R1[41]andKimiK1.5[119]showthatlarge-scalereinforcementlearningcanelicitsophisticatedreasoningbehaviorsfromstrongpre-trainedmodels;andKimiK2.5AgentSwarm[60]furtherextendstest-timescalingfromsequentialreasoningtoparallelagentcoordination.Theseadvanceshavemadetest-timescalingacentralfocusoffrontierresearch.However,whiletheopen-sourcemodelecosystemhasadvancedrapidlyonthesecondaxis,ithasprogressedslowlyonthefirst:manyrecentmodelsremainwithinorslightlyabovethe1T-classparameterregime[146,29,136,121].Asincreasinglysophisticatedreasoningandagenticreinforcementlearningmethodsareappliedtopre-trainedfoundationsofsimilarscale,open-sourceprogressrisksconvergingwhilethegaptothestrongestproprietarysystemswidens.WithKimiK3,wepursuebothscalingaxestogethertothefrontier:scalingthepre-trainedfoundationtounprecedented3T-classparameterswhilescalingreinforcementlearning,reasoningeffort,andlong-horizoninteractionat1Mcontextlength.

WeintroduceKimiK3,anativemultimodalMixture-of-Expertsmodelwith2.8trilliontotalparameters,104billionactivatedparameters,andacontextwindowofuptoonemilliontokens.Itsarchitecturescalesinformationflowacrosssequencelength,networkdepth,andmodelwidth.KimiDeltaAttention(KDA)[64]providesefficientlong-sequencemixing,withperiodicallyinterleavedGatedMLAlayerspreservingglobalinteraction.AttentionResiduals(AttnRes)[58]allowseachlayertoselectivelyattendtorepresentationsfromallprecedinglayers.StableLatentMoEexpandstheroutedexpertspaceto896experts,with16activatedpertoken,whilenormalization,SiTU-GLU,andQuantileBalancingstabilizeoptimizationatextremesparsity.Thesearchitecturaladvances,combinedwithrefineddataandtrainingrecipes,yieldanapproximately2.5×improvementinoverallscalingefficiencyoverKimiK2[59].

Wepairthispre-trainingfoundationwithpost-trainingdesignedexplicitlyfor1Mcontexttest-timescaling.KimiK3undergoesreinforcementlearningacrosslong-horizoncoding,generalagents,generalreasoningandknowledgetasks,eachspanningmultiplereasoning-effortlevels.Trainingenvironmentsincludeverifiablesearchandprofessionalknowledgework,softwareengineeringandkerneloptimization,multimodalreasoningwithvision-in-the-looptooluse,persistentassistantworkflows,webdevelopment,andautonomousexecutiontasks.Theseenvironmentstrainageneralloopofreasoning,acting,observing,verifying,andadapting,oftenoverhundredsorthousandsoftoolcallsandmillionsofaccumulatedcontexttokens.Domain-andeffort-specializedpoliciesareconsolidatedintoaunifiedmodelthroughmulti-teacheron-policydistillation[76,135,29].

Realizingthisregimerequiresinfrastructurethatscaleswitharchitecturecomplexity,modelsize,andtrajectorylength.Forsystemsco-designforKDA,wedevelopfusedkernels,KDAContextParallelism,andstate-awareprefixcachingtomakeKDAefficientwithindevices,acrossdevices,andacrossrequests.For2.8T-parameterMoEpre-training,MoonEPprovidesperfectlybalancedexpertexecutionwithstaticcomputationshapesandzero-copycommunication,whilememoryefficienttrainingandmultimodalencoderoptimizationssustainutilizationwithinboundedmemory.Formillion-tokenagenticRL,ourco-locatedsystemcombinespartialrollouts,externalKV-cacheretention,adaptivethrottlingandresumablemicroVMsandboxestopreservelong-livedmodelandenvironmentstate.Finally,specializedkernels,andcache-andbudget-awarefleetschedulingtranslatetheseinnovationsintopredictableproductionserving.Theresultingmodelestablishesanewopenfrontier.Onbenchmarksspanninglong-horizoncoding,agentic,knowledge,reasoning,andvisiontasks,KimiK3trailsthestrongestproprietarysystemsoverall—ClaudeFable5andGPT-5.6Sol—andisconsistentlyaheadoftheotheropenandproprietarymodelsevaluatedinoursuite,asshowninFig.1.

Ourcontributionsaresummarizedasfollows:

•Pre-trainingattheopenfrontier.Wetraina2.8T-parameternativemultimodalMoEmodelwith104Bactivatedparametersanda1M-tokencontextwindow.KDA,AttnRes,StableLatentMoE,refineddataandtrainingrecipescollectivelyimproveoverallscalingefficiencybyapproximately2.5×overKimiK2.

•Reinforcementlearningformulti-efforttest-timescaling.WeconductRLacrossgeneral,agentic,andcodingdomainsandmultiplereasoning-effortlevels,thenconsolidatetheresultingcapabilitiesintoaunifiedmodel.

•Infrastructureformulti-trillion-parameter,million-tokenintelligence.WeintroduceKDAsystemsco-designs;MoonEPandmemory-efficientinfrastructurefor2.8T-parameterMoEpre-training;aco-locatedRLsystemwithresumablesandboxesformillion-tokenagentictrajectories;andmoreinfrastructureinnovations.

•Anopenfrontiermodel.WereleasethefullKimiK3modelweights,makingfrontierintelligenceavailableforresearch,deployment,andfurtherinnovation.

3

KimiK3:OpenFrontierIntelligenceTECHNICALREPORT

Output

wα

w

Linear

SharedExpertRoutedExpert

StableLatentMoE

α

Norm

1×

w

1212

α

3N

GatedMLA

Router

Linear

w

α

StableLatentMoE

3×

w

α

KDA

Linear

Norm

Blockn-1

KimiDeltaAttention

qk

vαβ

Blockn-2

L2

L2

Embedding

σ

CoonnvvLineear

σ

Conv

Linear

σ

MLP

Linear

MoonViT-V2

Figure2:TheKimiK3architecture,organizedaroundtoken,channel,andlayermixing,withanativevisionpathwayattheinput.EachblockcontainsthreeKimiDeltaAttention(KDA)layersfollowedbyoneGatedMLAlayer,witheachattentionlayerpairedwithaStableLatentMoEfeed-forwardnetwork.AttentionResiduals(AttnRes)uselearnedpseudo-queries(w)toderiveattentionweights(α)overtheembeddingandprecedingblockoutputs,enablingselectiveinformationflowacrossdepth.Topleft:theStableLatentMoEmodulewithsharedandroutedexperts.Bottomleft:theKDAmodule.Bottomright:thenativevisionpathway.

2ModelArchitecture

TheKimiK3architectureisdesignedtoscaleinformationflowalongthreecomplementarydimensions:sequencelength,networkdepth,andmodelwidth.Alongthesequencedimension,HybridAttentioncombinesthreeKimiDeltaAttention(KDA)[64]layerswithoneGatedMLAlayerineachblock,providinganefficientmechanismforlong-contexttokenmixingwhileretainingselectivehigh-capacityattention(§2.1).Alongthedepthdimension,AttentionResiduals(AttnRes)[58]enableeachmoduletoselectivelyretrieverepresentationsfromtheembedding,thecurrentblock,andprecedingblocks,extendinginformationaccessbeyondconventionalsequentialresidualaccumulation(§2.2).Alongthewidthdimension,eachattentionlayerisfollowedbyaStableLatentMoElayerthatperformssparsechannelmixing,effectivelyactivating16of896routedexpertsforeachtoken(§2.3).Fornativevision,MoonViT-V2encodesimagesandvideos,andalightweightprojectormapstheresultingvisualfeaturesintothesharedembeddingspacebeforebackboneprocessing(§2.4).TogetherwithPer-HeadMuon(§2.5),thesecomponentsprovideaunifiedarchitectureforscalinginformationflowacrosstokens,layers,andchannels.Combinedwithrefinedtraininganddatarecipes,theyyieldanapproximately2.5×improvementinoverallscalingefficiencyoverKimiK2.Figure2providesanoverviewofthearchitecture.

4

KimiK3:OpenFrontierIntelligenceTECHNICALREPORT

2.1HybridAttention

KimiK3usesalayerwisehybridoflinearandglobalattention,combiningKDA[64]withGatedMLA.Eachblockcontains3KDAlayersfollowedby1GatedMLAlayer,givinga3:1mixingratio.Thispatternisrepeatedthroughoutthebackbone.Thetwoattentionmechanismsaredescribedseparatelybelow.AnadditionalGatedMLAlayerisplacedattheendofthebackbone,ensuringthatthefinallayeralwaysperformsglobalattention.

2.1.1KimiDeltaAttention

KDAextendsthedelta-rulerecurrence[106,139]withachannel-wiseforgetgate[64].Considerasequenceofhiddenstatesxt∈Rd,wheretindexesthetokenpositionanddisthemodelhiddendimension.Forclarity,wefirstdescribeasingleattentionhead,withqueryandkeyvectorsqt,kt∈Rdk,valuevectorvt∈Rdv,andrecurrentstateSt∈Rdk×dv.KDAapplieschannel-wisedecaybeforethedelta-ruleupdate:

St=(I−βtktk)Diag(αt)St−1+βtktv,t=Sqt.(1)

Here,αt∈(0,1)dkisthechannel-wiseone-stepretentionfactor,andβt∈(0,1)controlsthedelta-rulewritestrength.FollowingKimiLinear[64],KDAparameterizestheper-headquantitiesas

v=Swish(ShortConv(Wxt))∈Rdv,

β=Sigmoid(Wxt)∈(0,1),

z=WWxt+b∈Rdk.(2)

Thequery,key,andvalueprojectionsapplyShortConvfollowedbySwish[139],andthequeryandkeyarefurther

normalizedwithL2Norm[142].Thelow-rankprojectionandhead-specificbiasb∈Rdkproduceafine-grained

decaylogitzforeachkeychannel.Thelower-boundedmappingfromztoαisintroducedafterthechunkwise

formulationbelow.

ChunkwiseparallelformFollowingKimiLinear[64],KDAisrecurrentacrosschunksandparallelwithineachchunk.ForachunksizeC,X[t]stacksthetokenvectorsinthet-thchunkforX∈{Q,K,V,O,U,W}.ThematrixS[t]∈Rdk×dvdenotestherecurrentstateenteringchunkt.Forpositions1≤i≤j≤C,definethechannel-wisecumulativedecay

γ]:=γ[→]r.(3)

AsinKimiLinear,Γ1[t→]C∈RC×dkstacksγ[],...,γ[]row-wise.TheUTtransformproducesU[t]andW[t],from

whichwedefinethepseudo-valueterm[t]:=U[t]−W[t]S[t].GiventheincomingstateS[t],alloutputsinchunktarecomputedinparallelas

A[t]=Tril[(Q[t]⊙Γ1[t→]C)(K[t]/Γ1[t→]C)⊤],

ForamatrixM,Tril(M)setsallstrictlyupper-triangularentriestozeroandretainsthelower-triangularentries,includingthediagonal.Thismaskenforcescausalinteractionswithinthechunk,andthediagonalisretainedbecauseeachoutputreadsthestateafterthecurrent-tokenupdate.ThefirstterminO[t]carriesinformationfromprecedingchunks,whereasthesecondtermaccountsforinteractionswithinthecurrentchunk.WereferreaderstoKimiLinear[64]fortheUTtransformandthefullderivationofthechunkwiseform.

Lower-boundeddecayEq.4rescalesthekeysineachchunkbythereciprocalcumulativedecay1/Γ1[t→]C.BecauseΓ1[t→]Cisaproductofretentionfactorsin(0,1),thisreciprocalcangrowwithoutboundandoverflowinfiniteprecision[141,64].KimiLinearcontrolsthisnumericalrangebycomputingrelativedecayinlogspaceanddividingeachchunkintosecondary16-tokentiles[141,64].Theoff-diagonaltilescanthenbecomputedwithdensematrixmultiplicationsonTensorCoresdirectly.Thediagonaltiles,incontrast,stillrequireexplicitposition-paircomputations,whichremainthemainintra-chunkbottleneck.

KimiK3:OpenFrontierIntelligenceTECHNICALREPORT

0

gmin=-5

g

KimiLinear:g=-eASoftplus(z)

KimiK3:g=gminSigmoid(eAz)

z

-∞

A=0

(a)Log-decayparameterization.

lower-bounded

KimiLinearKimiK3

Position-pairDiagonalTensorCore

(b)Diagonal-tilecomputation.

Figure3:Lower-boundeddecayanditseffectonchunkwiseKDAcomputation.(a)KimiLinearusesanunboundednegative-Softplusmapping,whereasKimiK3boundsthelog-decaywithascaledsigmoid;thecurvesshowA=0andgmin=-5.(b)KimiLinearevaluateseachdiagonaltilewithanexplicitposition-paircomputation,whiletheboundedrangeinKimiK3allowsallcausaltilestousedenseTensorCorematrixmultiplications.

KimiK3addressesthisbottleneckbychangingthemappingfromthedecaylogitsztotheper-steplog-decay

g.FollowingGDNandMamba-2,KimiLinearusesthenegative-Softplusmappingg=—eAhSoftplus(z)∈(—∞,0)dk[139,24,64].KimiK3insteadusesascaledsigmoidtoboundthelog-decayfrombelow:

g=gminSigmoid(eAhz)∈(gmin,0)dk,

α=exp(g)∈(egmin,1)dk,(5)

whereAhisalearnableper-headlog-scaleandgmin=—5isfixed.WeinitializeAh=0,andeachbiasbisinitialized

following[64,24,139].Withgmin=—5,everyretentionfactorsatisfiesα,j>e-5≈6.7×10-3,andthecumulative

log-decayovera16-tokentileliesin(—80,0).Thecorrespondingreciprocalrescalingfactoristhereforesmallerthane80andremainswithintheBF16dynamicrange.Thisfiniterangeallowsbothdiagonalandoff-diagonaltilestousedenseTensorCorematrixmultiplications,eliminatingtheposition-pairdiagonalpath.Thisparameterizationiscloselyrelatedtothelower-boundedrecurrencegatesinpriorwork[98,27,92].Fig.3illustratesthechangeindecayparameterizationanditscomputationalconsequence.

Full-rankgateFinally,KimiK3changesKDA’soutputgatefromthelow-rankparameterizationusedbyKimiLinear[64]toaninput-dependentfull-rankprojection.Afterapplyinghead-wiseRMSNorm[147]totherecurrentoutput,KDAappliesdata-dependentoutputgating[100]:

yt=Wo[Sigmoid(Wgxt

2.1.2GatedMLA

Multi-headLatentAttention(MLA),introducedinDeepSeek-V2[28],compressesthekey–valuerepresentationofeachtokenintoalow-dimensionallatentvectorct=Wcxt.Insteadofcachingfullhead-specifickeysandvalues,MLAcachesctandreconstructsthecontentkeysandvaluesthroughlearnedup-projectionsduringattentioncomputation.ThisfactorizationreducestheKV-cachefootprintwhileretainingglobaltoken-to-tokenattention.MLAwassubsequentlyadoptedbyKimiK2andKimiK2.5[59,60],andKimiK3retainsitintheperiodicglobal-attentionlayers.

UnlikeKimiK2andKimiK2.5,KimiK3followsthehybriddesignofKimiLinear[64]andappliesNoPositionEncoding(NoPE)toallMLAlayers.Consequently,noexplicitpositionalencodingisappliedtotheirqueriesorkeys.TheinterveningKDAlayersprovideposition-sensitiveandrecency-awaresequencemixing,whiletheMLAlayersprovideunrestrictedglobalcontentinteraction.Thisseparationalsoavoidsmodifyingpositional-encodingparameterswhenextendingthecontextlength,suchasretuningaRoPEfrequencybaseorapplyingYaRN[93].

Inaddition,KimiK3augmentsMLAwithaninput-dependent,channel-wisefull-rankoutputgate.LettdenotetheungatedMLAoutputatpositiont;thegatedoutputis

6

KimiK3:OpenFrontierIntelligenceTECHNICALREPORT

ThegateprojectionWgisfullrank,matchingthenewparameterizationusedbyKDAinKimiK3.Thisgateallowseachtokentomodulatethechannelsreadfromglobalattention[100].

Tocorrectthebiasedroundingerrorthatarisesinflashattention,weadoptthemethodof[99]andkeeptheattentionoutputinFP32duringtraining.Thischoicedoublestheon-chipfootprintoftheoutputtile;wethereforeredesignthetrainingkerneltooverlapitwiththeKVstagingbuffersinsteadofthequerytile,freeingsharedmemoryforadeeperKVpipelineandhighertrainingthroughput.

2.2AttentionResiduals

Standardresidualconnections[44]compressallpriorinformationintoasinglestatehloverdepth—abottleneckreminiscentofRNNsovertime.Forsequencemodeling,theTransformerreplacedrecurrencewithattention[10,126],allowingeachpositiontoselectivelyaccessallpreviouspositionswithdata-dependentweights.AttentionResiduals(AttnRes)[58]appliesthesamemethodologytodepth:eachlayerselectivelyretrievesrepresentationsfromallprecedinglayersratherthanaccumulatingthemuniformly.

FullAttentionResidualsForeachlayerl,wedefinealayer-specificlearnablepseudo-queryql=wl∈Rdandkeysandvalues

wherefi(hi)istheoutputoflayeriandh1isthetokenembedding.Theattentionweightsfollowasoftmaxkernelϕ(q,k)=exp(q⊤RMSNorm(k))[56,147],wheretheRMSNormpreventslayerswithlarge-magnitudeoutputsfromdominatingtheweights:

l-1

(9)

Sincenetworkdepthismodest(L<100),theO(L2d)arithmeticofthisfullformisaffordable;thepracticaloverheadistheO(Ld)memory(andcross-stagecommunicationunderpipelineparallelism)forkeepingalllayeroutputsalive.

BlockAttentionResidualsToreducethisoverhead,wepartitiontheLlayersintoNblocksofS=L/Nlayerseach.Withinblockn(layerindicesBn),layeroutputsarereducedtoasinglerepresentationbysummation,bn=

Σj∈Bnfj(hj),withbdenotingthepartialsumoverthefirstilayersoftheblock;wesetb0=h1sothetoken

embeddingisalwaysincludedasasource.Acrossblocks,fullattentionisappliedoveronlytheNblock-levelrepresentations:forthei-thlayerinblockn,thevaluematrixis

ifi=1(firstlayerofblockn)ifi≥2(subsequentlayers)

(10)

withkeysandattentionweightsfollowingEq.8andEq.9.ThefinaloutputlayerthenaggregatesallNblockrepresentations.UnderBlockAttnRes,memoryandcommunicationoverheaddropfromO(Ld)toO(Nd),whilethisblockstructurealsoboundstheinference-timestate,enablingtheparallelinter-blockresultstobebettermergedwiththesequentialintra-blockpartialsumsviaonlinesoftmax[80],significantlyreducinginferencetimecost.

Empirically,N≈8recoversmostofthebenefitacrossmodelscales[58];forKimiK3,wepartitionitslayersinto8blockswith12-layersize,givingapartialfinalblockand9totalblockswhencountingtheembeddinglayer.

2.3StableLatentMoE

Increasingboththeexpertpoolandthenumberofactiveexpertsexpandsthespaceofexpertspecializations,butinaconventionalMoEeachselectedexpertreceivesthefulld-dimensionaltokenrepresentation,socommunicationandexpert-weighttrafficgrowwiththeroutingmultiplicity.LatentMoE[32]makesthisexpansionaffordablebyseparatingthefullmodelwidthfromtherouted-expertwidth:sharedexpertsretainafull-widthpathforcommontransformations,whereasspecializedroutedexpertsoperateinacompactlatentspaceofwidthℓ.ThisenablesKimiK3toscalechannelmixingto896routedexpertswith16activeexpertspertoken,correspondingtoasparsityof56.

Thisextremesparsityamplifiestwofailuremodesofthevanilladesign.First,theroutedpathcomposesW↓,agatedmulti-branchexpertfeed-forwardnetwork,andW↑intoachainofnearlyfourconsecutivematrixmultiplications.Thisill-conditionedstructure,combinedwiththe2.8-trillion-parameterscale,producesexplodinginternalactivationsintheroutedbranch.Second,balancingtheloadofnearly103expertsexceedstheregimeinwhichexistingauxiliary-loss-free

KimiK3:OpenFrontierIntelligenceTECHNICALREPORT

GatebranchUpbranchCurve

GLU[26]σ(x)x

SwiGLU[108]

x·σ(x)

x

SiTU-GLU

f(x)100

|f(x)|≤β1β2=100

SwiGLU

SiTU-GLUGLU

x

-100

50100

Figure4:GateandupbranchesofGLU,SwiGLU,andSiTU-GLU,togetherwiththeirscalarresponses,whereσdenotesthesigmoidfunction.Bothbranchesreceivethescalarinputx,andallcurvessharethedomainx∈[-10,100];theinsetmagnifiesthenear-originregion.SiTU-GLU,showninredwithβ1=4andβ2=25,closelyfollowsSwiGLUneartheoriginandapproachesthebound|f(x)|≤β1β2=100forlargepositiveinputs,whereasSwiGLUremainsunbounded.

biasupdatesremainwellbehaved.StableLatentMoEaddressesthesetwofailuremodeswiththreecomponents:anRMSNormbeforetheup-projectionandSigmoidTanhUnitGLU(SiTU-GLU)tosuppressactivationexplosion,andQuantileBalancing(QB)forloadbalancing.

AsillustratedinFig.2,thelayerfollowstheshared-androuted-expertorganizationofDeepSeekMoE[23].Forx∈Rd,thesharedexpertsprocessxdirectly,whiletheroutedpathprojectsittoz=W↓x∈Rℓ,dispatchesztotheselectedexperts,andmapstheirweightedaggregatebacktoRdthroughW↑:

Here,u∈Rℓistheaggregatedroutedrepresentation,E:Rd→RdandE:Rℓ→Rℓarethesharedandroutedexpertfeed-forwardnetworks,andpiistherouterweightdefinedbytheQuantileBalancingrulebelow.KimiK3fixesthenumberoffull-widthsharedexpertstoNs=2ineverylayer.

2.3.1NormalizedLatentMoE

TheoriginalLatentMoEdirectlyappliesW↑totheaggregatedroutedrepresentationu,whosescalecanvarywiththeselectedexpertsandtheirroutingweights.AsshowninEq.11,KimiK3insteadinsertsRMSNorm[147]betweenexpertaggregationandtheup-projection.Thisnormalizationreducesthesensitivityoftheroutedbranchtoscalevariationbeforeitiscombinedwiththefull-widthsharedbranch.Beyondstabilizingtraining,theadditionalRMSNormconsistentlyimprovesvalidationlossanddownstreambenchmarks.

2.3.2SigmoidTanhUnitGLU

GatedLinearUnits(GLUs)modulatealinearvaluebranchwithasigmoid-activatedgate,computingSigmoid(Wgx)⊙Wux[26].SwiGLUreplacesthesigmoidgatewithSwish(x)=xSigmoid(x)andyieldsstrongempiricalperformanceinTransformers[108].SwiGLUhassubsequentlybecomeawidelyadoptedFFNdesigninlargelanguagemodels,whileacompleteaccountofitsempiricaleffectivenessremainsopen.

However,bothmultiplicativefactorsinSwiGLUareunbounded,socoincidentlargecoordinatescanproduceactivationoutliersandincreaseoverflowriskinlow-precisionarithmetic.ThesigmoidgateoftheoriginalGLUavoidsunboundedgategrowth,butitdoesnotretaintheapproximatelylinearpositiveregimeofSwish.Thismotivatesanactivationthatcontrolslarge-valuegrowthwhilepreservingthecharacteristiclocalandpositive-sideresponseofSwiGLU.Otherrecenteffortshaveexploredalternativeparameterizationsofthistrade-off[52].

Tosatisfytheserequirements,weproposeSigmoidTanhUnitGLU(SiTU-GLU).SiTU-GLUappliesthesmoothcapsoftcap(x,β)=βtanh(x/β)tothelinearfactoroftheSwishgateandindependentlytotheupbranch:

SiTU-GLU=[β1tanhSigmoid(Wgx(12)

7

KimiK3:OpenFrontierIntelligenceTECHNICALREPORT

(a)Imbalancedrouting(b)QuantileBalancing(c)Balancedrouting

E1E2E3E4

t1

t2

t3

t4

t5

t6

t7

t8

E1

E2

E3

E4

t1

t2

t3

t4

t5

t6

t7

t8

t1

t2

t3

t4

t5

t6

t7

t8

E1

E2

E3

E4

Figure5:IllustrationofQuantileBalancingwithm=8tokens,n=4routedexperts,andk=1selectedexpertpertoken.(a)Token-wiseTop-krouting(tokensontheleft,expertsontheright)producesloads(4,3,1,0);darkercirclesindicateoverheatedexperts,whereasfadedanddashedcirclesindicateunderutilizedanddyingexperts,respectively.(b)Eachgraybaristhemarginofthecurrentlybiasedscore,si,j+bj(t)-αi(t),sotherow-wisemaximareproducetheroutingin(a).Thedashedredlineineach

columnisthebiasadjustmentbj(t)-j(t+1),placedatthe(q+1)-thlargestmarginsothatexactlyq=2marginsexceedit.The

marker⋆denotestherow-wiseTop-kchoiceaftersubtractingthecolumnadjustments,i.e.,theroutingin(c).(c)Theretainedchoicesyieldthebalancedload(2,2,2,2);rededgesdenoteassignmentschangedbyQB.

ForKimiK3,wesetthesoft-caphyperparameterstoβ1=4forthegatebranchandβ2=25fortheupbranch.Thescaledtanhisapproximatelylinearneartheoriginandboundedatlargemagnitude,allowingSiTU-GLUtopreservethelocalresponseofSwiGLUwhilecontrollingbothfactorsintheproduct.Fig.4comparesthebranchdefiniti