腾讯云ES千亿级混合检索架构演进

传统搜索返回文档，SearchAI提供可直接消费的Context。SearchAIEngineText·Vector·Structured·MultimodalAIAI&ContextContextAssembly·InferenceChunking·Embedding·RerankingTextSearch·SemanticSearch·HybridSearch*EngineEngineFoundationIndex·Execution·ResourceManagementStorage·SchemaEvolution·Security0102030405RetrievalAlgorithm检索算法IndexStructure索引结构StorageLayout存储布局ExecutionEngine执行引擎SchemaEvolution在线演化规模可由内存承载Beyond-MemoryRetrievalScaleDrivesSystem-wideEvolution.规模突破内存边界规模驱动系统级演进。LOADLESS·SCANLESS·STORELESS·EVOLVEONLINECommunityInnovation社区创新→ProductionChallenges生产挑战→ArchitectureEvolution架构演进→TencentCloudPractice腾讯云实践ChapterII·ScalingHybridSearchtoHundred-BillionScale面向磁盘的大规模向量检索架构ChapterII·ScalingHybridSearch空间裁剪+极致压缩+高精重排QueryBillionsofvectorsQueryBillionsofvectorsParentCentroidsChildCentroidsLeafVectorClustersTraversethecentroidhierarchytolocSOARSOARAssignment 1-bit1-bitVectorFloat32VectorOversamplingOversamplingTop-10Results(ExactDistanceComputation)++ChapterII·ScalingHybridSearchtoHundred-BillionScaleProductionChallengesMEMORYFOOTPRINTBBQ·10B×1536DRETRIEVALOVERHEADMEMORYREDUNDANCY余SCHEMAEVOLUTIONRebuild·ExtraCapacity·Serviceoffline●LOADLESSSCANLESSSTORELESS●EVOLVEONLINEChapterII·ScalingHybridSearchtoHundred-BillionScaleTENCENTINNOVATION·01LOADLESSGlobalIVFESESDATASHARDAqueryforoneuseridstilltouchesamulti-userworkingset.LOGICALFILTER≠PHYSICALLOCALITYESESDATASHARDuserid-awarephysicalslicing·inACTIVEWORKINGSETSLICEUSERIDAlignphysicalorganizationwithaccesslocality.AFTERSlice-DiskBBQSliceBuserid=BOooooDISK-BACKEDSliceCuserid=COooooDISK-BACKEDSliceDuserid=DDISK-BACKED···moreuseridsOSPAGECACHEA·HOTBDCBDCBBCCDDPRODUCTIONVAMeasuredinProduction-71%ResidentMemoryGlobaldatasetstayslarge·woChapterII·ScalingHybridSearchtoHundred-BillionScalePRODUCTIONPRODUCTIONVALIDATIONQueryPerformanceFilteredhybridqueries·benefitvarieswithfilterQueryPerformanceTENCENTINNOVATION·02SCANLESS01·BEFORE两次独立遍历TEXTPOSTINGBUILD…BITSETBUILD…000100010000001DISKBBQIVFCLUSTERVECTORPOSTING…BuildBitSet→CandidateCheck02·AFTER一次协同遍历CONJUNCTIONITERATORadvance·skip·intersectTEXTPOSTINGTEXTPOSTINGKEYINSIGHTBothareorderedDocIDstreams.VECTORPOSTINGMATCHEDCANDIDATESCo-Traverse→SkipSameresults·LessscanningChapterII·ScalingHybridSearchtoHundred-BillionScaleTENCENTINNOVATION·03STORELESSSOAR以冗余副本换取边界召回；超大规模下，去冗01·NATIVEPrimary-aa-SOARcopyCentroidCentroid2Centroid1CentroidCentroid4Centroid3Centroid3+BoundaryRecallQUANTIZEDREPRESENTATIONSQUANTIZEDREPRESENTATIONSATSCALERedundancycompounds.02·OPTIMIZEDCentroidCentroid1Centroid3EMPTY×REPRESENTATIONCentroid2Centroid4PRODUCTIONVALIDATION5Mvectors·1536DChapterII·ScalingHybridSearchtoHundred-BillionScaleTENCENTINNOVATION·04SCOREFASTERqcDistHEAVYCOMPUTEAlreadyfastqcDistQuantizedCodesSCORENativeSIMDDotQuantizedCodesSCORENativeSIMDDotProductSCALARCORRECTIONCorrectionFactorsRuntimeValueLayoutSharedLayoutLookupCorrectionMemorySegment.get()ConcurrentHashMapMemorySegment.get()HOTHOT-PATHconcurrentscoringAFTER·NATIVEUNLOCKEDqcDistHEAVYCOMPUTEUnchangedqcDistQuantizedCodesSCORENativeSIMDDotQuantizedCodesSCORENativeSIMDDotProductSCALARCORRECTIONCorrectionFactorsCorrectionStaticCorrectionFactorsCorrectionStaticFinalLayoutLOCK-FREEHOTPATHNATIVEvsJAVASIMDTHEBARTOBEAT1536DHybridSearch·QPSbeforeJAVASIMDafterLOSESWINSSAMESAMERECALLUPSTREAMIMPACTContributedupstream·Elasticsearch#154373ChapterII·ScalingHybridSearchtoHundred-BillionScaleTENCENTINNOVATION·05EVOLVEONLINESUPPORTEDEVOLUTIONPATHSTEXT→TEXT→TEXTTEXT→VECTORVECTOR→VECTOR01FIELD-LEVEL只转换变化的字段INDEX·MANYFIELDSTEXTInvertedIndexNUMERICDocValues/BKDVECTORHNSWOTHER···DiskBBQDiskBBQ1FIELDCHANGED→1FIELDCONVERTED02STRUCTURE-LEVEL只重建变化的数据结构REUSEVECTORFIELD·HNSWREUSERawVectorData.vecHNSW-specificStructure.vex/.vemREBUILDONLYCHANGEDSTRUCTUREDiskBBQ-specificStructure.cenivf/.clivf/.mivf03ONLINE在线完成转换LIVETRAFFICCONVERSIONBACKGROUNDCONVERSIONSWITCHHNSWDiskBBQHNSWDiskBBQ×FASTER10Bdocs6.55h→22.6minDATAUNTOUCHEDSegmentfilesreused·Large-scalevalidationEVOLUTIONEvolvewhileservingChapterII·ScalingHybridSearchtoHundred-BillionScaleINNOVATEDATINNOVATEDATSCALELOADLESSCONTRIBUTEDUPSTREAMPROVENINPRODUCTIONTenant-awareLocalityTECHNICALPRsElasticsearch/Elasticsearch/LuceneCONTRIBUTIONRANKING(PRCREATORS)CONTRIBUTIONRANKING(PRCREATORS)KNOWLEDGEBASE大规模知识库ADVERTISING广告推荐Text×VectorCo-traversalSTORELESS·SCOREFASTERMemory×Hot-pathEfficiencyEVOLVEONLINERECOGNIZEDBYELASTICCOMMUNITYLEADERSHIPRISKCONTROLFine-grainedEvolution安全风控ChapterIII·TheNextFrontierAUTONOMOUSIAUTONOMOUSINTELLIGENCEORGANIZELEARNROUTEEVOLVE