打造LLMOps时代Prompt数据驱动引擎

数据驱动引擎刘逸伦-华为-2012文本机器翻译实验室团队介绍：华为文本机器翻译实验室从AIOps到LLMOps：大模型强大的泛化能力和语言理解能力推动AIOps发展语言上。框架。人的认知世界与大模型数字世界的桥梁，解决模型意图更清晰的推理路径，提升人与模型的交互效率。题-答案对齐依赖。效的交互策略，生成符合人意图效的交互策略，生成符合人意图和需要的内容。-答案上训练模型，帮助模型理解人的意图和需要。痛点1：传统智能运维算法依赖于任务数据，专家标注耗时耗力痛点2：传统运维系统可解释性差、可交互性弱痛点3：Prompt训练数据质量不稳定，导致模型性能下降痛点4：Prompt训练数据全面性不足,损害AI能力的全面性(ICSE2024/ICPC2024接收)型打造LLMOps数据飞轮(ICDE2024接收)•Logsaresemi-structuredteNewfeaturesNewfeatures01Motivation:Existingapproachreliesonmassivetrainingdataandlacksinterpretabilityanalysistaskswhentr(testedontheremaining20%or10%oflog•Performancedrasticallydeclineswhentrlimitedto10%orlo•Therelyingonmassivemethodsineffectiveandinflexiblein•Existingmethodsonlyoffe•Practitionersoftenneedtospendextresults)andact(identifyrootcauses,composereport,etc...).valueswithoutrationunderstandingofincidentMotivation:Largelanguagemodelshasthepotentialtoaddressthechallenges,andprompteLargelanguagemodels(LLMs)havepowerfulgenabilitytounseenuserinstralsobeabletohandleunseenlogsintheonlinesituationoflogInourpreliminaryexperChatGPTwiththesimpachievedanF1-scoreofonly0.189inanomalydetection.However,ourbestpromptstrategyoutperformedthesimplepromptby0.195Sinceloganalysisisadomain-specificandapplyingasimpleprompttoLLMscanresultinpoorUnlikeexistingdeeplearnreport,etc.Loginterpretawritingtask.TherearemanypromptphiloproposedinNLPtasks,suchasCoTheprimaryobjectiveofLogPromptinterpretabilityofloganalysisintheonlinescenario,throughproperstrategyofpromptingLLMs.Approach:Thechain-of-thought(CoT)promptsTheCoTPromptintheputanexamplewithintbeforeaninputmathpmodelisencouragedtofollowthethinkTheconceptofchainofthought(CoT),aseriesofintermediatereasoningsteps,isintroducedbyWeietal.[1].TheCoTpromptemulatesthehumanthoughtprocessbyrequiringthemodeltoincludethinkingstepswhenaddressingcomplexproblemsandcanenhancetheperformanceofLLMsinchallengingtasks,suchassolvingmathematicalproblems.AdvantagesofCoTpr•Breakdownunseenproblemsintomanageable•EnhanceinterpretabilityandtransparencyofLLMoutput•Unleashthelearnedabilitiesinthepre-trainingpha[1]J.Wei,X.Wang,D.Schuurmans,M.Bosma,F.Xia,E.Chi,Q.V.Le,D.Zhouetal.,“Chain-of-thoughtpromptingelicitsreasoninginlargelanguagemodels,”AdvancesinNeuralInformationProcessingSystems,vol.35,pp.24824–24837,2022.Approach:AdaptingtheCoTpromptintothefieldofloganalysisInmanualloganalysis,practitionersalsoengageinaseriesofreasoningstepstoreachanormallogandanabnormallogisunclear.Toemulatethethinki•ImplicitCoT:Humanmostlyhasreasonsbeforeconclusion.Thus,ijustifyingitsdecisions.•ExplicitCoT:Wefurtherexplicitlydefineintermediatestepstoregulatethethidefinitionofanomalytobeonly“alertsexplicStandardPromptPerforminganalysisbasedonTaskdescriptionandInputlogsStandardPromptLogPrompt(CoT)Approach:OtherpromptstrateIn-contextPrompt:ThisapproachussamplesoflabeledlogstosetthecontextfortheTheLLMthenpredictsonnewlogs,usingFormatFormatControl:Weemploytwofunctions,fx([X])andfZ([Z],toestarange,like“abinarychoicebetweenabnormnormal”,or“aparsedlogtemplSelf-prompt:thisstrategyinvolvestheLLownprompts.Ameta-promptdescribingthetaLLMtogeneratepromptprefixcaeffectivepromptchosenbased•TheeffectivenessofLogPromptisevaluatedmainlyusingthelogHubdatasets,whichcontainsreal-worldlogsfromninedifferentdomains,includingsupercomputers,distributedsystems,operatingsystems,mobilesystems,andserverapplications.•Twoofthedatasets(BGLandSpirit)wereannotatedbydomainexpertstoidentifyanomalouseventsforthepurposeofanomalydetection.•Toevaluatethelogparsingperformance,eightofthedatasetshavelogtemplatesmanuallyextractedfromasubsetof2000logmessagesineachdomain.•Alllogdataweretimestamped,enablingthedatasetstobesplitintotrain/testsetschronologically.•Inourprimaryexperiments,theunderlyingLLMisaccessedviaAPIsprovidedbyexternalservices.•Theinitialtemperaturecoefficientissetto0.5,maintainingabalancebyincreasingthemodel'sreasoningcapabilitiesthroughdiversetokenexplorationwhilelimitingdetrimentalrandomness.•Iftheresponseformatisinvalid,thequeryisresubmittedwithanincreasedtemperaturecoefficientof0.4untiltheresponseformatiscorrect.Theformatfailurerateislessthan1%,whichisconsistentwithexistingliterature.•Thetrain/testdatasetsaresplitchronologicallytosimulateonlinescenarios.Experiment:LogPrompthasastrongabilityofhandlingscarcityintrainingdata•TaskofLogParsing•Foreachdataset,mostbaselinemethodsaretrainedonthefirst10%logsandevaluatedontheremaining90%logs,whileLogPromptisdirectlytestedontheremaining90%logswithoutin-domaintrainingda•WeadopttheF1-scoreasthemetric.Tocalculateit,wetokenizethepredictedlogtemplateintoalistoftokens,thenwetreatthetokensasresultsfromaclassificationtaskof{template,variable}.•LogPromptachievedthebestF1-scoreonsixoftheeightdatasets,andoutperformedexistingmethodswhichrequireresourcesforin-domain•TaskofAnomalyDetection•Forbaselines,thefirst4000logsineachdatasetareusedfortraining.BothLogPromptandthetrainedbaselinesarethentestedonthe•Wereportthesession-levelF1-scoreofanomaly.Asessionisformedusingfixed-windowgroupingwithalengthof100logtemplate•Despitetheexistingmethodsbeingtrainedonthousandsoflogs,LogPromptstillachievedstrongperformancesinbothdatasetswithoututilizinganyin-domaintrainingdata,withanaverageimprovementof55.9%intermsofF1-score.TheadvantageofTheadvantageofLogPromptmakesitasuitablechoiceforloganalysisinonlinescenariosExperiment:LogPromptyieldhelpfulandcomprehensiblecontentforpractitionersduringreal-worldloganalysis•ANovelEvaluatingTaskofLogInterpretation•Atotalof200logswererandomlysampledforthehumanevaluation,accompaniedbyLogPrompt'sactualoutputs,with100logsrelatedtologparsingand100relatedtoanomaly(evenlydistributeda•Incorrectlypredictedlogs(FPsandFNs)werenotincludedinthisevaluation.Anequalnumberofnormalandabnormalsampleswereincludedforanomalydetection,andeachselectedlogforlogparsingwasrequiredtocontainatleasaccordingtothecriteria,independently•Wereportedtwometforbothtasksintermsofusefulnessandreadabilityconsistentlyexceededfour,andaverageHIPwasconsistentlyabove80%,indicatinganoverallhelpfulandreadablecontentconsideredbyexperiencedloganExperiment:MoreanalysisofLogPrompt’sinterpretability•FeedbacksfromPractitioners•BadCaseAnalysis•“IappreciatetheabilityofLogPromptinsupportingtheinterpretationoflogsfromvariousdomains.Asoursystemcontinuouslyincorporatingthird-partyservices,sometimesIhavetoreadthroughthemanualstodecipherlogsfromunfamiliardomains.TheexplanationsgeneratedbyLogPromptcanprovideaswiftgraspofthelogsbeforeIfoundtheofficialdefinitions.”•“LogPromptcandefinitelyhelpinthecomposeofashortlyaftersystemcrashes,whereIoftenneedtonon-technicalcolleaguesinmeetings.”•“IntherealmofsoftwareO&M,falsealarmsareaninescapablereality,withfalsepositivesimposingsubstantialtimecostsonengineecausingsevereramifications.Accompanyingexplanationswithautomaticanalysisoutcomesenablesengineerstomorepromptlyascertainthecredibilityofthepurportedanomaly,therebyreducingthetimespentonsubsequentactions.”•AmajorfactorofbadcasesistheLLM’slackofdomainknowledge,whichleadstooverlygeneralinterpretationsofsomedomain-specificterms.Forexample,itmayrefspecificparametersassiAnothercauseisthelackofsemanticconttlogs,whichcanbeattributedtotheirbrevityorrichnessofnon-NLPpatterns(i.e.,digits,codesandaddresses).Experiment:AblationstudyonthethreepromptstrategiesComparedtoprompt5,prompt2providesmoreformalandaccuratewords(suchas“standardized”and“convert”)andclearlyoutlinestheintermediatestepsforthetask(identifyingandreplacingvariables,thenconvertingtoatemplate).Interestingly,onlyutilizingtheimplicitCoT(requiringgeneratingreasons)canstillimprovethemodelperformance,likelyduetothereasonthatwithmoreNLPexplanations,thedistributionofthegeneratedanswersismoreclosetothatinthepre-trainingphasesofmodels.AnoverlylongcontextprefixedtothepromptmaycauseLLMstopaylessattentiothenewinputlogs,therebydeterioratingthetaskperformance,whichiswhythepeakisFuturework:Domainadaptingsmaller-scaledLLMsforcompatibilitywithadvancedpromptstrategies•ApplyingLogPrompttoasmaller-scaledLLM:Vicuna13B•Thedeploymentoflarge,proprietary,andAPI-dependentLLMsinlocalenvironmentscanbeachallengingtask.•ThereliabilityofservicesmaybecompromisediftheAPIservicesofLLMsbecomeunavailable.•Therefore,forindustrialusage,itiscrucialforLogPrompttohavecompatibilitywithalternativeopen-source,privacy-protected,andsmaller-scaleLLMs.OnlineLogParsingwithsmaller-scaledLLMsusin•AlthoughVicunahasonly13Bparameters,whenequippedwithLogPrompt,itachievesacomparableperformanceoflogparsingwiththe175BGPTmodelinthedatasetsofHDFS,LinuxandProxifier.•Additionally,whenthepromptstrategytransitionsfromasimpleprompttoLogPrompt,Vicunaexhibitssignificantimprovementsonperformance,withanaverageincreaseof380.7%inF1-score.•AsVicunaisopen-sourceandrequiresfewerresourcesfortraininganddeployment,thesuccessofLogPromptonVicunaholdspromisingimplicationsforbuildingin-domainindustrialapplications.•Theperformanceofsmaller-scaledLLMslikeVicunastillhasaroomforimprovement.Sincethebasemodel