月之暗面2026KimiK3技术报告 KIMI K3 OPEN FRONTIER INTELLIGENCE

KIMIK3:OPENFRONTIERINTELLIGENCETECHNICALREPORTOFKIMIK3KimiTeamABSTRACTWeintroduceKimiK3,a2.8TparameterMixture-of-Expertsmodelwith104billionactivatedparameters,nativevisioncapabilities,anda1-million-tokencontextwindow.KimiK3isbuiltonKimiDeltaAttention[64]andAttentionResiduals[58],whichimproveinformationflowacrosssequencelengthandmodeldepth.TogetherwithStableLatentMoE,whicheffectivelyactivates16of896routedexpertspertoken,andrefinedtraininganddatarecipes,theseadvancesyieldanapproximately2.5×improvementinoverallscalingefficiencyoverKimiK2[59].Post-traininghighlightsreinforcementlearningacrossgeneral,agentic,andcodingdomainsandmultiplereasoning-effortlevels,enablingcompositionalgeneralizationandrobustlong-horizonexecution.At2.8Tscale,KimiK3issupportedbyinfrastructureadvancesinmultipleareas:algorithm–systemco-designforKDA,perfectlybalancedexpert-paralleltrainingwithefficientmemorymanagement,million-tokenagenticRLwithpersistentrolloutandsandboxstates,anddeploymentinnovations.ExtensiveevaluationsshowthatKimiK3achievesfrontier-levelperformanceacrosslong-horizoncoding,agentic,knowledge,reasoning,andvisiontasks.Whileitsoverallperformancestilltrailsthemostpowerfulproprietarymodels,namelyClaudeFable5andGPT-5.6Sol,KimiK3consistentlyoutperformsotheropenandproprietarymodelsevaluatedinoursuite.WereleasethefullKimiK3modelweightstofacilitatefutureresearchandacceleratethebroaderdeploymentandadoptionoffrontierintelligence.1Figure1:KimiK3mainresults.1https://huggingface.co/moonshotai/Kimi-K3KimiK3:OpenFrontierIntelligenceTECHNICALREPORT21IntroductionFormuchofthedevelopmentofLargeLanguageModels(LLMs),scalingmeantinvestingmorecomputationbeforedeploymentbytraininglargermodelsonmoredata[55,46].Theriseofreasoningmodelshasestablishedtest-timecomputationasasecondaxisofscaling:OpenAI’so-seriesscalesreinforcementlearningandtest-timereasoning[85,84];Anthropic’sextended-thinkingmodelsallocateadaptivethinkingbudgetsandinterleavereasoningwithtooluse[6,7];DeepSeek-R1[41]andKimiK1.5[119]showthatlarge-scalereinforcementlearningcanelicitsophisticatedreasoningbehaviorsfromstrongpre-trainedmodels;andKimiK2.5AgentSwarm[60]furtherextendstest-timescalingfromsequentialreasoningtoparallelagentcoordination.Theseadvanceshavemadetest-timescalingacentralfocusoffrontierresearch.However,whiletheopen-sourcemodelecosystemhasadvancedrapidlyonthesecondaxis,ithasprogressedslowlyonthefirst:manyrecentmodelsremainwithinorslightlyabovethe1T-classparameterregime[146,29,136,121].Asincreasinglysophisticatedreasoningandagenticreinforcementlearningmethodsareappliedtopre-trainedfoundationsofsimilarscale,open-sourceprogressrisksconvergingwhilethegaptothestrongestproprietarysystemswidens.WithKimiK3,wepursuebothscalingaxestogethertothefrontier:scalingthepre-trainedfoundationtounprecedented3T-classparameterswhilescalingreinforcementlearning,reasoningeffort,andlong-horizoninteractionat1Mcontextlength.WeintroduceKimiK3,anativemultimodalMixture-of-Expertsmodelwith2.8trilliontotalparameters,104billionactivatedparameters,andacontextwindowofuptoonemilliontokens.Itsarchitecturescalesinformationflowacrosssequencelength,networkdepth,andmodelwidth.KimiDeltaAttention(KDA)[64]providesefficientlong-sequencemixing,withperiodicallyinterleavedGatedMLAlayerspreservingglobalinteraction.AttentionResiduals(AttnRes)[58]allowseachlayertoselectivelyattendtorepresentationsfromallprecedinglayers.StableLatentMoEexpandstheroutedexpertspaceto896experts,with16activatedpertoken,whilenormalization,SiTU-GLU,andQuantileBalancingstabilizeoptimizationatextremesparsity.Thesearchitecturaladvances,combinedwithrefineddataandtrainingrecipes,yieldanapproximately2.5×improvementinoverallscalingefficiencyoverKimiK2[59].Wepairthispre-trainingfoundationwithpost-trainingdesignedexplicitlyfor1Mcontexttest-timescaling.KimiK3undergoesreinforcementlearningacrosslong-horizoncoding,generalagents,generalreasoningandknowledgetasks,eachspanningmultiplereasoning-effortlevels.Trainingenvironmentsincludeverifiablesearchandprofessionalknowledgework,softwareengineeringandkerneloptimization,multimodalreasoningwithvision-in-the-looptooluse,persistentassistantworkflows,webdevelopment,andautonomousexecutiontasks.Theseenvironmentstrainageneralloopofreasoning,acting,observing,verifying,andadapting,oftenoverhundredsorthousandsoftoolcallsandmillionsofaccumulatedcontexttokens.Domain-andeffort-specializedpoliciesareconsolidatedintoaunifiedmodelthroughmulti-teacheron-policydistillation[76,135,29].Realizingthisregimerequiresinfrastructurethatscaleswitharchitecturecomplexity,modelsize,andtrajectorylength.Forsystemsco-designforKDA,wedevelopfusedkernels,KDAContextParallelism,andstate-awareprefixcachingtomakeKDAefficientwithindevices,acrossdevices,andacrossrequests.For2.8T-parameterMoEpre-training,MoonEPprovidesperfectlybalancedexpertexecutionwithstaticcomputationshapesandzero-copycommunication,whilememoryefficienttrainingandmultimodalencoderoptimizationssustainutilizationwithinboundedmemory.Formillion-tokenagenticRL,ourco-locatedsystemcombinespartialrollouts,externalKV-cacheretention,adaptivethrottlingandresumablemicroVMsandboxestopreservelong-livedmodelandenvironmentstate.Finally,specializedkernels,andcache-andbudget-awarefleetschedulingtranslatetheseinnovationsintopredictableproductionserving.Theresultingmodelestablishesanewopenfrontier.Onbenchmarksspanninglong-horizoncoding,agentic,knowledge,reasoning,andvisiontasks,KimiK3trailsthestrongestproprietarysystemsoverall—ClaudeFable5andGPT-5.6Sol—andisconsistentlyaheadoftheotheropenandproprietarymodelsevaluatedinoursuite,asshowninFig.1.Ourcontributionsaresummarizedasfollows:•Pre-trainingattheopenfrontier.Wetraina2.8T-parameternativemultimodalMoEmodelwith104Bactivatedparametersanda1M-tokencontextwindow.KDA,AttnRes,StableLatentMoE,refineddataandtrainingrecipescollectivelyimproveoverallscalingefficiencybyapproximately2.5×overKimiK2.•Reinforcementlearningformulti-efforttest-timescaling.WeconductRLacrossgeneral,agentic,andcodingdomainsandmultiplereasoning-effortlevels,thenconsolidatetheresultingcapabilitiesintoaunifiedmodel.•Infrastructureformulti-trillion-parameter,million-tokenintelligence.WeintroduceKDAsystemsco-designs;MoonEPandmemory-efficientinfrastructurefor2.8T-parameterMoEpre-training;aco-locatedRLsystemwithresumablesandboxesformillion-tokenagentictrajectories;andmoreinfrastructureinnovations.•Anopenfrontiermodel.WereleasethefullKimiK3modelweights,makingfrontierintelligenceavailableforresearch,deployment,andfurtherinnovation.KimiK3:OpenFrontierIntelligenceTECHNICALREPORT3GatedMLALinearStableLatentMoE3×KDABlockn_1Blockn_2EmbeddingσMLPLinearMoonViT-V2RouterLinearNormKimiDeltaAttentionvαβSharedExpertRoutedExpertqkL22wαwαwαwαLinearNormConvLinearStableLatentMoE3NOutputwασσFigure2:TheKimiK3architecture,organizedaroundtoken,channel,andlayermixing,withanativevisionpathwayattheinput.EachblockcontainsthreeKimiDeltaAttention(KDA)layersfollowedbyoneGatedMLAlayer,witheachattentionlayerpairedwithaStableLatentMoEfeed-forwardnetwork.AttentionResiduals(AttnRes)uselearnedpseudo-queries(w)toderiveattentionweights(α)overtheembeddingandprecedingblockoutputs,enablingselectiveinformationflowacrossdepth.Topleft:theStableLatentMoEmodulewithsharedandroutedexperts.Bottomleft:theKDAmodule.Bottomright:thenativevisionpathway.2ModelArchitectureTheKimiK3architectureisdesignedtoscaleinformationflowalongthreecomplementarydimensions:sequencelength,networkdepth,andmodelwidth.Alongthesequencedimension,HybridAttentioncombinesthreeKimiDeltaAttention(KDA)[64]layerswithoneGatedMLAlayerineachblock,providinganefficientmechanismforlong-contexttokenmixingwhileretainingselectivehigh-capacityattention(§2.1).Alongthedepthdimension,AttentionResiduals(AttnRes)[58]enableeachmoduletoselectivelyretrieverepresentationsfromtheembedding,thecurrentblock,andprecedingblocks,extendinginformationaccessbeyondconventionalsequentialresidualaccumulation(§2.2).Alongthewidthdimension,eachattentionlayerisfollowedbyaStableLatentMoElayerthatperformssparsechannelmixing,effectivelyactivating16of896routedexpertsforeachtoken(§2.3).Fornativevision,MoonViT-V2encodesimagesandvideos,andalightweightprojectormapstheresultingvisualfeaturesintothesharedembeddingspacebeforebackboneprocessing(§2.4).TogetherwithPer-HeadMuon(§2.5),thesecomponentsprovideaunifiedarchitectureforscalinginformationflowacrosstokens,layers,andchannels.Combinedwithrefinedtraininganddatarecipes,theyyieldanapproximately2.5×improvementinoverallscalingefficiencyoverKimiK2.Figure2providesanoverviewofthearchitecture.KimiK3:OpenFrontierIntelligenceTECHNICALREPORT42.1HybridAttentionKimiK3usesalayerwisehybridoflinearandglobalattention,combiningKDA[64]withGatedMLA.Eachblockcontains3KDAlayersfollowedby1GatedMLAlayer,givinga3:1mixingratio.Thispatternisrepeatedthroughoutthebackbone.Thetwoattentionmechanismsaredescribedseparatelybelow.AnadditionalGatedMLAlayerisplacedattheendofthebackbone,ensuringthatthefinallayeralwaysperformsglobalattention.2.1.1KimiDeltaAttentionKDAextendsthedelta-rulerecurrence[106,139]withachannel-wiseforgetgate[64].Considerasequenceofhiddend,wheretindexesthetokenpositionanddisthemodelhiddendimension.Forclarity,wefirstdescribeasingleattentionhead,withqueryandkeyvectorsqt,kt∈耽dk,valuevKDAapplieschannel-wisedecaybeforethedelta-ruleupdate:St=(I−βtktk)Diag(αt)St−1+βtktv,t=Sqt.(1)Here,αt∈(0,1)dkisthechannel-wiseone-stepretentionfactor,andβt∈(0,1)controlsthedelta-rulewritestrength.FollowingKimiLinear[64],KDAparameterizestheper-headquantitiesasq,k=L2Norm(Swish(ShortConv(W/kxt)))∈耽dk,v=Swish(ShortConv(Wxt))∈耽dv,β=Sigmoid(Wxt)∈(0,1),z=WWxt+b∈耽dk.(2)Thequery,key,andvalueprojectionsapplyShortConvfollowedbySwish[139],andthequeryandkeyarefurthernormalizedwithL2Norm[142].Thelow-rankprojectionandhead-specificbiasb∈耽dkproduceafine-graineddecaylogitzforeachkeychannel.Thelower-boundedmappingfromztoαisintroducedafterthechunkwiseformulationbelow.ChunkwiseparallelformFollowingKimiLinear[64],KDAisrecurrentacrosschunksandparallelwithineachchunk.ForachunksizeC,X[t]stacksthetokenvectorsinthet-thchunkforX∈{Q,K,V,O,U,W}.ThematrixS[t]dk×dvdenotestherecurrentstateenteringchunkt.Forpositions1≤i≤j≤C,definethechannel-wisecumulativedecayAsinKimiLinear,Γ1[t→]C∈耽C×dkstacksγ[],...,γ[]row-wise.TheUTtransformproducesU[t]andW[t],fromwhichwedefinethepseudo-valueterm[t]:=U[t]−W[t]S[t].GiventheincomingstateS[t],alloutputsinchunktarecomputedinparallelasA[t]=Tril[(Q[t]⊙Γ1[t→]C)(K[t]/Γ1[t→]C)⊤],ForamatrixM,Tril(M)setsallstrictlyupper-triangularentriestozeroandretainsthelower-triangularentries,includingthediagonal.Thismaskenforcescausalinteractionswithinthechunk,andthediagonalisretainedbecauseeachoutputreadsthestateafterthecurrent-tokenupdate.ThefirstterminO[t]carriesinformationfromprecedingchunks,whereasthesecondtermaccountsforinteractionswithinthecurrentchunk.WereferreaderstoKimiLinear[64]fortheUTtransformandthefullderivationofthechunkwiseform.Lower-boundeddecayEq.4rescalesthekeysineachchunkbythereciprocalcumulativedecay1/Γ1[t→]C.BecauseΓ1[t→]Cisaproductofretentionfactorsin(0,1),thisreciprocalcangrowwithoutboundandoverflowinfiniteprecision[141,64].KimiLinearcontrolsthisnumericalrangebycomputingrelativedecayinlogspaceanddividingeachchunkintosecondary16-tokentiles[141,64].Theoff-diagonaltilescanthenbecomputedwithdensematrixmultiplicationsonTensorCoresdirectly.Thediagonaltiles,incontrast,stillrequireexplicitposition-paircomputations,whichremainthemainintra-chunkbottleneck.KimiK3:OpenFrontierIntelligenceTECHNICALREPORTKimiKimiLinearKimiK3Position-pairDiagonalTensorCorePosition-pairDiagonalTensorCorelower-bounded__∞A=0KimiLinear:g=__eSoftplus(z)KimiK3:g=gSigmoid(ez)g0g=__5≈(a)Log-decayparameterization.(b)Diagonal-tilecomputation.Figure3:Lower-boundeddecayanditseffectonchunkwiseKDAcomputation.(a)KimiLinearusesanunboundednegative-Softplusmapping,whereasKimiK3boundsthelog-decaywithascaledsigmoid;thecurvesshowA=0andgmin=—5.(b)KimiLinearevaluateseachdiagonaltilewithanexplicitposition-paircomputation,whiletheboundedrangeinKimiK3allowsallcausaltilestousedenseTensorCorematrixmultiplications.KimiK3addressesthisbottleneckbychangingthemappingfromthedecaylogitsztotheper-steplog-decayg.FollowingGDNandMamba-2,KimiLinearusesthenegative-Softplusmappingg=−eAhSoftplus(z)∈(−∞,0)dk[139,24,64].KimiK3insteadusesascaledsigmoidtoboundthelog-decayfrombelow:g=gminSigmoid(eAhz)∈(gmin,0)dk,α=exp(g)∈(egmin,1)dk,(5)whereAhisalearnableper-headlog-scaleandgmin=−5isfixed.WeinitializeAh=0,andeachbiasbisinitializedfollowing[64,24,139].Withgmin=−5,everyretentionfactorsatisfiesα,j>e__5≈6.7×10__3,andthecumulativelog-decayovera16-tokentileliesin(−80,0).Thecorrespondingreciprocalrescalingfactoristhereforesmallerthane80andremainswithintheBF16dynamicrange.Thisfiniterangeallowsbothdiagonalandoff-diagonaltilestousedenseTensorCorematrixmultiplications,eliminatingtheposition-pairdiagonalpath.Thisparameterizationiscloselyrelatedtothelower-boundedrecurrencegatesinpriorwork[98,27,92].Fig.3illustratesthechangeindecayparameterizationanditscomputationalconsequence.Full-rankgateFinally,KimiK3changesKDA’soutputgatefromthelow-rankparameterizationusedbyKimiLinear[64]toaninput-dependentfull-rankprojection.Afterapplyinghead-wiseRMSNorm[147]totherecurrentoutput,KDAappliesdata-dependentoutputgating[100]:yt=Wo[Sigmoid(Wgxt)⊙RMSNorm(t)].(6)2.1.2GatedMLAMulti-headLatentAttention(MLA),introducedinDeepSeek-V2[28],compressesthekey–valuerepresentationofeachtokenintoalow-dimensionallatentvectorct=Wcxt.Insteadofcachingfullhead-specifickeysandvalues,MLAcachesctandreconstructsthecontentkeysandvaluesthroughlearnedup-projectionsduringattentioncomputation.ThisfactorizationreducestheKV-cachefootprintwhileretainingglobaltoken-to-tokenattention.MLAwassubsequentlyadoptedbyKimiK2andKimiK2.5[59,60],andKimiK3retainsitintheperiodicglobal-attentionlayers.UnlikeKimiK2andKimiK2.5,KimiK3followsthehybriddesignofKimiLinear[64]andappliesNoPositionEncoding(NoPE)toallMLAlayers.Consequently,noexplicitpositionalencodingisappliedtotheirqueriesorkeys.TheinterveningKDAlayersprovideposition-sensitiveandrecency-awaresequencemixing,whiletheMLAlayersprovideunrestrictedglobalcontentinteraction.Thisseparationalsoavoidsmodifyingpositional-encodingparameterswhenextendingthecontextlength,suchasretuningaRoPEfrequencybaseorapplyingYaRN[93].Inaddition,KimiK3augmentsMLAwithaninput-dependent,channel-wisefull-rankoutputgate.LettdenotetheungatedMLAoutputatpositiont;thegatedoutputisyt=Wo[Sigmoid(Wgxt)⊙t].(7)5KimiK3:OpenFrontierIntelligenceTECHNICALREPORTThegateprojectionWgisfullrank,matchingthenewparameterizationusedbyKDAinKimiK3.Thisgateallowseachtokentomodulatethechannelsreadfromglobalattention[100].Tocorrectthebiasedroundingerrorthatarisesinflashattention,weadoptthemethodof[99]andkeeptheattentionoutputinFP32duringtraining.Thischoicedoublestheon-chipfootprintoftheoutputtile;wethereforeredesignthetrainingkerneltooverlapitwiththeKVstagingbuffersinsteadofthequerytile,freeingsharedmemoryforadeeperKVpipelineandhighertrainingthroughput.2.2AttentionResidualsStandardresidualconnections[44]compressallpriorinformationintoasinglestatehloverdepth—abottleneckreminiscentofRNNsovertime.Forsequencemodeling,theTransformerreplacedrecurrencewithattention[10,126],allowingeachpositiontoselectivelyaccessallpreviouspositionswithdata-dependentweights.AttentionResiduals(AttnRes)[58]appliesthesamemethodologytodepth:eachlayerselectivelyretrievesrepresentationsfromallprecedinglayersratherthanaccumulatingthemuniformly.FullAttentionResidualsForeachlayerl,wedefinealayer-specificlearnablepseudo-queryql=wl∈耽dandkeysandvalueswherefi(hi)istheoutputoflayeriandh1isthetokenembedding.Theattentionweightsfollowasoftmaxkernelϕ(q,k)=exp(q⊤RMSNorm(k))[56,147],wheretheRMSNormpreventslayerswithlarge-magnitudeoutputsfromdominatingtheweights:Sincenetworkdepthismodest(L<100),theO(L2d)arithmeticofthisfullformisaffordable;thepracticaloverheadistheO(Ld)memory(andcross-stagecommunicationunderpipelineparallelism)forkeepingalllayeroutputsalive.BlockAttentionResidualsToreducethisoverhead,wepartitiontheLlayersintoNblocksofS=L/Nlayerseach.Withinblockn(layerindicesBn),layeroutputsarereducedtoasinglerepresentationbysummation,bn=Σj∈Bnfj(hj),withbdenotingthepartialsumoverthefirstilayersoftheblock;wesetb0=h1sothetokenembeddingisalwaysincludedasasource.Acrossblocks,fullattentionisappliedoveronlytheNblock-levelrepresentations:forthei-thlayerinblockn,thevaluematrixiswithkeysandattentionweightsfollowingEq.8andEq.9.ThefinaloutputlayerthenaggregatesallNblockrepresentations.UnderBlockAttnRes,memoryandcommunicationoverheaddropfromO(Ld)toO(Nd),whilethisblockstructurealsoboundstheinference-timestate,enablingtheparallelinter-blockresultstobebettermergedwiththesequentialintra-blockpartialsumsviaonlinesoftmax[80],significantlyreducinginferencetimecost.Empirically,N≈8recoversmostofthebenefitacrossmodelscales[58];forKimiK3,wepartitionitslayersinto8blockswith12-layersize,givingapartialfinalblockand9totalblockswhencountingtheembeddinglayer.2.3StableLatentMoEIncreasingboththeexpertpoolandthenumberofactiveexpertsexpandsthespaceofexpertspecializations,butinaconventionalMoEeachselectedexpertreceivesthefulld-dimensionaltokenrepresentation,socommunicationandexpert-weighttrafficgrowwiththeroutingmultiplicity.LatentMoE[32]makesthisexpansionaffordablebyseparatingthefullmodelwidthfromtherouted-expertwidth:sharedexpertsretainafull-widthpathforcommontransformations,whereasspecializedroutedexpertsoperateinacompactlatentspaceofwidthℓ.ThisenablesKimiK3toscalechannelmixingto896routedexpertswith16activeexpertspertoken,correspondingtoasparsityof56.Thisextremesparsityamplifiestwofailuremodesofthevanilladesign.First,theroutedpathcomposesW↓,agatedmulti-branchexpertfeed-forwardnetwork,andW↑intoachainofnearlyfourconsecutivematrixmultiplications.Thisill-conditionedstructure,combinedwiththe2.8-trillion-parameterscale,producesexplodinginternalactivationsintheroutedbranch.Second,balancingtheloadofnearly103expertsexceedstheregimeinwhichexistingauxiliary-loss-free6KimiK3:OpenFrontierIntelligenceTECHNICALREPORT||f(x)|≤β1β2=100SwiGLUSiTU-GLUGLUx50100f(x)100__100CurveGatebranchUpbranchGLU[26]σ(x)xSwiGLU[108]x·σ(x)xSiTU-GLUβ1tanhβ2tanhFigure4:GateandupbranchesofGLU,SwiGLU,andSiTU-GLU,togetherwiththeirscalarresponses,whereσdenotesthesigmoidfunction.Bothbranchesreceivethescalarinputx,andallcurvessharethedomainx∈[—10,100];theinsetmagnifiesthenear-originregion.SiTU-GLU,showninredwithβ1=4andβ2=25,closelyfollowsSwiGLUneartheoriginandapproachesthebound|f(x)|≤β1β2=100forlargepositiveinputs,whereasSwiGLUremainsunbounded.biasupdatesremainwellbehaved.StableLatentMoEaddressesthesetwofailuremodeswiththreecomponents:anRMSNormbeforetheup-projectionandSigmoidTanhUnitGLU(SiTU-GLU)tosuppressactivationexplosion,andQuantileBalancing(QB)forloadbalancing.AsillustratedinFig.2,thelayerfollowstheshared-androuted-expertorganizationofDeepSeekMoE[23].Forx∈耽d,thesharedexpertsprocessxdirectly,whiletheroutedpathprojectsitto≈=W↓x∈耽ℓ,dispatches≈totheselectedexperts,andmapstheirweightedaggregatebackto耽dthroughW↑:ℓistheaggregatedroutedrepresentation,E:耽d→耽dandE:耽ℓ→耽ℓarethesharedandroutedexpertfeed-forwardnetworks,andpiistherouterweightdefinedbytheQuantileBalancingrulebelow.KimiK3fixesthenumberoffull-widthsharedexpertstoNs=2ineverylayer.2.3.1NormalizedLatentMoETheoriginalLatentMoEdirectlyappliesW↑totheaggregatedroutedrepresentationu,whosescalecanvarywiththeselectedexpertsandtheirroutingweights.AsshowninEq.11,KimiK3insteadinsertsRMSNorm[147]betweenexpertaggregationandtheup-projection.Thisnormalizationreducesthesensitivityoftheroutedbranchtoscalevariationbeforeitiscombinedwiththefull-widthsharedbranch.Beyondstabilizingtraining,theadditionalRMSNormconsistentlyimprovesvalidationlossanddownstreambenchmarks.2.3.2SigmoidTanhUnitGLUGatedLinearUnits(GLUs)modulatealinearvaluebranchwithasigmoid-activatedgate,computingSigmoid(Wgx)⊙Wux[26].SwiGLUreplacesthesigmoidgatewithSwish(x)=xSigmoid(x)andyieldsstrongempiricalperformanceinTransformers[108].SwiGLUhassubsequentlybecomeawidelyadoptedFFNdesigninlargelanguagemodels,whileacompleteaccountofitsempiricaleffectivenessremainsopen.However,bothmultiplicativefactorsinSwiGLUareunbounded,socoincidentlargecoordinatescanproduceactivationoutliersandincreaseoverflowriskinlow-precisionarithmetic.ThesigmoidgateoftheoriginalGLUavoidsunboundedgategrowth,butitdoesnotretaintheapproximatelylinearpositiveregimeofSwish.Thismotivatesanactivationthatcontrolslarge-valuegrowthwhilepreservingthecharacteristiclocalandpositive-sideresponseofSwiGLU.Otherrecenteffortshaveexploredalternativeparameterizationsofthistrade-off[52].Tosatisfytheserequirements,weproposeSigmoidTanhUnitGLU(SiTU-GLU).SiTU-GLUappliesthesmoothcapsoftcap(x,β)=βtanh(x/β)tothelinearfactoroftheSwishgateandindependentlytotheupbranch:KimiK3:OpenFrontierIntelligenceTECHNICALREPORT(a)Imbalancedrouting(b)QuantileBalancing(c)BalancedroutingEEEEttttttttEEEEEtttttttttttttttt EEEEFigure5:IllustrationofQuantileBalancingwithm=8tokens,n=4routedexperts,andk=1selectedexpertpertoken.(a)Token-wiseTop-krouting(tokensontheleft,expertsontheright)producesloads(4,3,1,0);darkercirclesindicateoverheatedexperts,whereasfadedanddashedcirclesindicateunderutilizedanddyingexperts,respectively.(b)Eachgraybaristhemarginofthecurrentlybiasedscore,si,j+bj(t)—αi(t),sotherow-wisemaximareproducetheroutingin(a).Thedashedredlineineachj,()ForKimiK3,wesetthesoft-caphyperparameterstoβ1=4forthegatebranchandβ2=25fortheupbranch.Thescaledtanhisapproximatelylinearneartheoriginandboundedatlargemagnitude,allowingSiTU-GLUtopreservethelocalresponseofSwiGLUwhilecontrollingbothfactorsintheproduct.Fig.4comparesthebranchdefinitionsandscalarresponsesofGLU,SwiGLU,andSiTU-GLUonacommonslice.§Bgivesthelocalexpansion,limitingcase,formaloutputbound,andcomparisonwithhardclamping.2.3.3QuantileBalancingUnlikeauxiliary-loss-basedrouting[33],KimiK3adoptsauxiliary-loss-freerouting[30].Loadbalancingisimple-mentedbyaddinganexpert-specificbiasbjtotherouterscoreusedforTop-kselection.Fortokenxi,theroutercomputessi=Sigmoid(Wrxi)andappliesBecausebisomittedfrompi,j,itregulatesdispatchwithoutalteringthemixtureweightsorthegradient-basedoptimizationoftherouter.Theorig