OWASP：人工智能安全解决方案概览指南2025（英文）

updatedQuarterly

|GenAI,LLMSecOpsandSecuritySolutionLandscape

RevisionHistory

Revision

Date

Authors

Description

.01

6/4/2024

ScottClinton

InitialDraftICharter

.05

8/10/2024

ScottClintonI

ContributorsInputs

Updatedwithinitialfeedback

.06

10/15/2024

ScottClintonI

ContributorsIReviewerInputs

Re-factorSolutions

LandscapecategoriesI

1.0

10/15/2024

ContributorsIReviewers

FinalReleaseCandidate

TheinformationprovidedinthisdocumentdoesnotIandisnotintendedtoIconstitutelegaladvice.Allinformationisforgeneralinformationalpurposesonly.Thisdocumentcontainslinkstootherthird-partywebsites.SuchlinksareonlyforconvenienceIandOWASPdoesnotrecommend

orendorsethecontentsofthethird-partysites.

LicenseandUsage

ThisdocumentisIicensedunderCreativeCommons,CCBY-SA4.0Youarefreeto:

●Share—copyandredistributethematerialinanymediumorformat

●Adapt—remixItransformIandbuilduponthematerialforanypurposeIevencommercially.

●Underthefollowingterms:

oAttribution—YoumustgiveappropriatecreditIprovidealinktothelicenseIandindicateifchangesweremade.Youmaydosoinanyreasonablemannerbutnotinanywaythatsuggeststhelicensorendorsesyouoryouruse.

oAttributionGuidelines-mustincludetheprojectnameaswellasthenameoftheassetReferenced

■OWASPTop10forLLMs-LLMSecOpsSolutionsLandscape

■OWASPTop10forLLMs-CyberSecuritySolutionandLLMSecOpsLandscapeGuide

●ShareAlike—IfyouremixItransformIorbuilduponthematerialIyoumustdistributeyourcontributionsunderthesamelicenseastheoriginal.

Linktofulllicensetext:

/licenses/by–sa/4.0/legalcode

TheinformationprovidedinthisdocumentdoesnotIandisnotintendedtoIconstitutelegaladvice.Allinformationisfor

generalinformationalpurposesonly.Thisdocumentcontainslinkstootherthird-partywebsites.Suchlinksareonlyfor

convenienceandOWASPdoesnotrecommendorendorsethecontentsofthethird-partysites.

Version1.01of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

Contents

WhoIsThisDocumentFor? 3

Objectives 3

Scope 3

Introduction 4

DefiningtheSecuritySolutionsLandscape 4

LandscapeConsiderations 4

LLMApplicationCategories,SecurityChallenges 5

StaticPromptAugmentationAppIications 6

AgenticAppIications 7

LLMPIug-ins,Extensions 8

CompIexAppIications 9

LLMDevelopmentandConsumptionModels 10

LLMOpsandLLMSecOpsDefined 11

AQuickOpsPrimer-FoundationforLLMOps 11

LLMOpsLifeCYcIeStages-FoundationforLLMDevSecOps 12

Scoping/PIanning 13

DataAugmentationandFine-Tuning 14

AppIicationDeveIopmentandExperimentation 14

TestandEvaIuation 15

ReIease 15

DepIoY 16

Operate 16

Monitor 17

Govern 18

MappingtotheOWASPTop10forLLMThreatModeI 18

AppIicationServices 19

ProductionServices 19

OWASPTop10forLLMsSolutionsLandscape 20

EmergingGenAI/LLM-SpecificSecuritYSoIutions 21

LLM&GenerativeAISecuritYSoIutions 22

SoIutionLandscapeMatrixDefinitions 22

LandscapeSoIutionMatrix 23

Acknowledgements 29

OWASPTop10forLLMProjectSponsors 30

References 31

ProjectSupporters 32

Version1.02of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

WhoIsThisDocumentFor?

ThisdocumentistailoredforadiverseaudiencecomprisingdevelopersIAppSecprofessionalsIDevSecOpsandMLSecOpsteamsIdataengineersIdatascientistsICISOsIandsecurityleaderswhoarefocusedondevelopingstrategiestosecureLargeLanguageModels(LLMs)andGenerativeAIapplications.ItprovidesareferenceguideofthesolutionsavailabletoaidinsecuringLLMapplicationsIequippingthemwiththeknowledgeandtoolsnecessarytobuildrobustIsecureAIapplications.

Objectives

ThisdocumentisintendedtobeacompaniontotheOWASPTop10forLargeLanguageModel(LLM)ApplicationsListandtheCISOCybersecurity&GovernanceChecklist.Itsprimaryobjectiveistoprovideareferenceresourcefororganizationsseekingtoaddresstheidentifiedrisksandenhancetheirsecurityprograms.Whilenotdesignedtobeanall-inclusiveresourceIthisdocumentoffersaresearchedpointofviewbasedonthetopsecuritycategoriesandemergingthreatareas.Itcapturesthemostimpactfulexistingandemergingcategories.BycategorizingIdefiningIandaligningapplicabletechnologysolutionareaswiththeemergingLLMandgenerativeAIthreatlandscapeIthisdocumentaimstosimplifyresearcheffortsandserveasasolutionsreferenceguide.

Scope

ThescopeofthisdocumentistocreateashareddefinitionofsolutioncategoryareasthataddressthesecurityoftheLLMandgenerativeAIlifecycleIfromdevelopmenttodeploymentandusage.ThisalignmentsupportstheOWASPTop10ListForLLMsoutcomesandtheCISOCybersecurityandGovernanceChecklist.ToachievethisIthedocumentwillcreateaninitialframeworkandcategorydescriptorsIutilizingbothopen-sourcesolutionsandprovidingmechanismsforsolutionproviderstoaligntheirofferingswithspecificcoverageareasasexamplestosupporteachcategory.

Thedocumentadherestoseveralkeyrulestomaintainitsintegrityandusefulness:

●Vendor-AgnosticandOpenApproach:ItmaintainsaneutralstanceIavoidingrecommendationsofonetechnologyoveranotherIinsteadprovidingcategoryguidancewithchoicesandoptions.

●Straightforward,ActionableGuidance:ThedocumentoffersclearIactionableadvicethatorganizationscanreadilyimplement.

●CoordinatedKnowledgeGraph:ItincludescoordinatedtermsIdefinitionsIanddescriptionsforkeyconcepts.

●PointtoExistingStandards:WhereexistingstandardsorsourcesoftruthareavailableIthedocumentreferencestheseinsteadofcreatingnewsourcesIensuringconsistencyandreliability.

Version1.03of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

Introduction

WiththegrowthofGenerativeAIadoption,usage,andappIicationdeveIopmentcomesnewrisksthataffecthoworganizationsstrategizeandinvest.AstheserisksevoIve,sodoriskmitigationsoIutions,technoIogies,frameworks,andtaxonomies.ToaidsecurityIeadersinprioritization,conversationsaboutemergingtechnoIogyandsoIutionareasmustbeaIignedappropriateIytocIearIyunderstoodbusinessoutcomesforAIsecuritysoIutions.ThebusinessoutcomesofAIsecuritysolutionsmustbeproperlydefinedtoaidsecurityleadersinbudgeting

ManyorganizationshavealreadyinvestedheavilyinvarioussecuritytoolsIsuchasvulnerabilitymanagementsystemsIidentityandaccessmanagement(IAM)solutionsIendpointsecurityIDynamicApplicationSecurityTesting(DAST)IobservabilityplatformsIandsecureCI/CD(ContinuousIntegration/ContinuousDeployment)toolsItonameafew.HoweverIthesetraditionalsecuritytoolsmaynotbesu代cienttofullyaddressthecomplexitiesofAIapplicationsIleadingtogapsinprotectionthatmaliciousactorscanexploit.ForexampleItraditionalsecuritytoolsmaynotsu代cientlyaddresstheuniquedatasecurityandsensitiveinformationdisclosureprotectioninthecontextofLLMandGenAIapplications.ThisincludesbutisnotlimitedtothechallengesofsecuringsensitivedatawithinpromptsIoutputsIandmodeltrainingdataIandthespecificmitigationstrategiessuchasencryptionIredactionIandaccesscontrolmechanisms.

EmergentsolutionslikeLLMFirewallsIAI-specificthreatdetectionsystemsIsecuremodeldeploymentplatformsIandAIgovernanceframeworksattempttoaddresstheuniquesecurityneedsofAI/MLapplications.HoweverItherapidevolutionofAI/MLtechnologyanditsapplicationshasdrivenanexplosionofsolutionapproachesIwhichhasonlyaddedtotheconfusionfacedbyorganizationsindeterminingwheretoallocatetheirsecuritybudgets.

DefiningtheSecuritySolutionsLandscape

TherehavebeenmanyapproachestocharacterizingthesolutionslandscapeforLargeLanguageModeltoolsandinfrastructure.InordertodevelopasolutionslandscapethatfocusesonthesecurityofLLMapplicationsacrossthelifecyclefromplanningIdevelopmentIdeploymentIandoperationItherearefourkeyareasofinputwehavefocusedontodevelopbothadefinitionforLargeLanguageModelDevSecOPsandrelatedsolutionslandscapecategories.

LandscapeConsiderations

ApplicationTypesandScope-whichimpactsthepeopleIprocessesIandtoolsneededbasedonthecomplexityoftheapplicationandtheLLMenvironmentIas-a-serviceIself-hostedIorcustom-built.

EmergingLLMSecOpsProcess-whilethisisaworkinprogressImanyarelookingtoadaptandadoptexistingDevOpsandMLOpsandassociatedsecuritypractices.WeexpectourdefinitiontoevolveasthedevelopmentprocessesforLLMapplicationsbegintomature.

Version1.04of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

ThreatandRiskModeling-understandingtherisksposedbyLLMsystemsIapplicationusageIormisuselikethoseoutlinedintheOWASPTop10forLLMsandGenerativeAIApplicationsIarekeytounderstandingwhichsolutionsarebestsuitedtoimprovethesecuritypostureandcombatarangeofattacks.

TrackingEmergingSolutions-manyexistingsecuritysolutionsareadaptingtosupportLLMdevelopmentworkflowsandusecaseshowevergiventhenatureofnewthreatsandevolvingtechnologyandarchitecturesnewtypesofLLM-specificsecuritysolutionswillbenecessary.

LLMApplicationCategories,SecurityChallenges

OrganizationshavebeenleveragingMachineLearninginapplicationsfordecades.ThisoftenrequireddetailedexpertiseinDataScienceandextensivemodeltraining.GenerativeAIhaschangedthis.SpecificallyILargeLanguageModels(LLMs)havemademachinelearningtechnologywidelyaccessible.Theabilitytodynamicallyinteractinplainlanguagehasopenedthedoorforthecreationofanewclassofdata-drivenapplicationsandapplicationintegrations.FurthermoreIusageisnolongerlimitedtothehighlyskilledeffortsoftraditionaldevelopersanddatascientists.Pre-trainedmodelsenablenearlyanyonetoperformcomplexcomputationaltasksIregardlessofpriorexposuretoprogrammingorsecurity.OrganizationshavebeenleveragingMachineLearninginapplicationsfordecadesincludingNaturalLanguageProcessing(NLP)modelsthatoftenrequiredetailedexpertiseinDataScienceandextensivemodeltraining.

Withtheadventoftransformerstechnologyenablinggenerativecapabilitiescombinedwiththeeaseofaccessforpre-trainedas-a-servicemodelslikeChatGPTandotheras-a-serviceIFourmajorcategoriesofLLMApplicationArchitectureemerged;Prompt-centricIAIAgentsIPlug-ins/extensionsIandcomplexgenerativeAIapplicationwheretheLLMplaysakeyroleinalargerapplicationusecase.

(figure:ApplicationCategories&SummaryAttributes)

HavingacommonviewoftypicalLLMapplicationarchitecturesIincludingagentsImodelsILLMsIandtheMLapplicationstackIiscrucialfordefiningandaligningtheapplicationstackIsecuritymodelIandapplicationofferings.BelowIwehaveprovidedashortdescriptionofkeycharacteristicsIusecasesIandsecuritychallengesforeachapplicationcategory.

Version1.05of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

StaticPromptAugmentationApplications

TheseapplicationsinvolvespecificstaticnaturaIIanguageinputstoguidethebehaviorofa

largelanguagemodel(LLM)towardgeneratingthedesiredoutput.Thistechniqueoptimizestheinteractionbetweentheuserandthemodelbyfine-tuningthephrasingIcontextIandinstructionsgiventotheLLM.Theseapplicationsallowuserstoaccomplishawiderangeoftasksbysimply

refininghowtheyaskquestionsorprovideinstructions.

KeyCharacteristics

oHumantomodel/modeltohumaninteractionandresponse

oStaticpromptaugmentation

oFlexibilityandCreativity

oSimplicityandAccessibility

oRapidPrototypingandExperimentation

UseCaseExamples

oExperimentation/RapidPrototyping

oContentGenerationTools

oTextSummarizationApplications

oQuestion-AnsweringSystems

oLanguageTranslationTools

oChatbotsandVirtualAssistants

SecurityChallenges

oPrompt-basedapplicationsfacesecurityriskslikepromptinjectionattacksand

dataleakagefrompoorlycraftedprompts.Lackofcontextorstatemanagement

canleadtounintendedoutputsIincreasingmisusevulnerability.User-generated

promptsmaycauseinconsistentorbiasedresponsesIriskingcomplianceorethicalviolations.EnsuringpromptintegrityIrobustinputvalidationIandsecuringtheLLMenvironmentarecrucialtomitigatetheserisks.

Version1.06of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

AgenticApplications

TheseapplicationsleverageLargeLanguageModels(LLMs)toautonomouslyorsemi-autonomouslyperformtasksImakedecisionsIandinteractwithusersorothersystems.TheseagentsaredesignedtoactonbehalfofusersIhandlingcomplexprocessesthatofteninvolvemultiplestepsIintegrationsIandreal-timedecision-making.TheyoperatewithalevelofautonomyIallowingthemtocompletetaskswithoutconstanthumanintervention.

KeyCharacteristics

oAutonomyandDecision-Making

oInteractionwithExternalSystems

oStateManagementandMemory

oComplexWorkflowAutomation

oHuman-AgentCollaboration

UseCaseExamples

oVirtualAssistants

oCustomerSupportBots

oProcessAutomationAgents

oDataAnalysisandReportingAgents

oIntelligentPersonalizationAgents

oSecurityandComplianceAgents

SecurityChallenges

oAgentapplicationsIwiththeirautonomyandaccesstovarioussystemsImustbecarefullysecuredtopreventmisuse.Theyfacesecuritychallengeslike

unauthorizedaccessIincreasedexploitationrisksduetointeractionwithmultiplesystemsIandvulnerabilitiesindecision-makingprocesses.Ifsomeonegains

controlofanautonomousagent,theconsequencescouldbesevere,especiallyincriticalsystems.Ensuringrobustaccesscontrolsandencryptionmethodsto

protectagainstthisisessential.Ensuringdataintegrityandconfidentialityis

criticalIasagentsoftenhandlesensitiveinformationitisimportanttosecuredataatallstagesIincludingat-restIinmotionIandaccessthroughsecuredAPIs.Theirautonomyalsoposesrisksofunintendedorharmfuldecisionswithoutoversight.RobustauthenticationIencryptionImonitoringIandfail-safemechanismsare

essentialtomitigatethesesecurityrisks.ObservabilityandTraceabilitysolutionsthatmonitortheentirelifecycleoftheAgents(DesignIDevelopmentIDeploymentIandVisibilityondecision-making)mustbeconsideredtoensurereal-time

correctionsusingahumans-in-the-loopprocesscanbeenforced.

Version1.07of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

LLMPlug-ins,Extensions

Plug-insareextensionsoradd-onsthatintegrateLLMsintoexistingapplicationsorplatformsIenablingthemtoprovideenhancedornewfunctionalities.Plug-instypicallyserveasabridgebetweentheLLMandtheapplicationIfacilitatingseamlessintegrationIsuchasaddingalanguagemodeltoawordprocessorforgrammarcorrectionorintegratingwithcustomerrelationshipmanagement(CRM)systemsforautomatedemailresponses.

Whileitcanbesometimesdi代culttodrawthelinebetweenAgentsandplug-insorextensionswhichareoftencomponentsoflargerapplicationsIonemeasureisthewayitisdeployedandused.ForexampleIaplug-inwouldbeapre-builtagendesignedforreusethatyoucallexplicitlyIthroughanAPIIoraspartofanLLMspluginorextensionframeworkvs.customcoderunninginthebackgroundonaperiodicbasis.

KeyCharacteristics

oModularityandFlexibility

oSeamlessIntegration

oTaskSpecificFocus

oEaseofDeploymentandUse

oRapidUpdatesandMaintenance

UseCaseExamples

oContentGenerationTools

oTextSummarizationApplications

SecurityChallenges

oPluginsinteractingwithsensitivedataorcriticalsystemsmustbecarefullyvettedforsecurityvulnerabilities.Poorlydesignedormaliciouspluginscancausedatabreachesorunauthorizedaccess.LLMpluginsfacechallengeslikecompatibilityissuesIwhereupdatescanintroducevulnerabilitiesIandintegrationwithsensitivesystemsincreasestheriskofdataleaks.EnsuringsecureAPIinteractionsIregularupdatesIandrobustaccesscontrolsiscrucial.Resource-intensivepluginsmaydegradeperformanceIriskingexploitation.

o

Version1.08of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

ComplexApplications

ComplexapplicationsaresophisticatedsoftwaresystemsthatdeeplyintegrateLargeLanguageModels(LLMs)asacentralcomponenttoprovideadvancedfunctionalitiesandsolutions.TheseapplicationsarecharacterizedbytheircomprehensivescopeIscalabilityIandtheintegrationofmultipletechnologiesandcomponents.TheyaretypicallydesignedtosolveintricateproblemsIofteninenterpriseenvironmentsIandrequireextensivedevelopmentIengineeringIandongoingmaintenanceefforts.

KeyCharacteristics

oMulti-componentarchitecturesaredesignedtoprocesspromptsfromothernon-humansystems.

oOftenusemultipleintegrationsIincludingothermodels.

oMulti-ComponentArchitecture

oScalabilityandPerformance

oAdvancedFeaturesandCustomization

oEnd-to-EndWorkflowAutomation

UseCaseExamples

oLegalDocumentAnalysisPlatforms

oAutomatedFinancialReportingSystems

oCustomerServicePlatforms

oHealthcareDiagnostics

SecurityChallenges

oComplexLLMapplicationsfacemajorsecuritychallengesduetotheirintegrationwithmultiplesystemsandextensivedatahandling.TheseincludeAPIvulnerabilitiesIdatabreachesIandadversarialattacks.ThecomplexityincreasestheriskofmisconfigurationsIleadingtounauthorizedaccessordataleaks.Managingcomplianceacrosscomponentsisalsodi代cult.RobustencryptionIaccesscontrolsIregularsecurityauditsIandcomprehensivemonitoringareessentialtoprotecttheseapplicationsfromsophisticatedthreatsandensuredatasecurity.

Version1.09of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

LLMDevelopmentandConsumptionModels

OneofthefirstconsiderationsforanorganizationisdecidingupontheapproachtoleveragingLLMcapabilitiesbasedonthetypeofapplicationandgoalsfortheproject.TodayIdevelopershaveachoiceoftwoprimarydeploymentmodelswhenimplementingLLM-basedapplicationsandsystems.

CreateaNewModel:ThetrainingprocessforcustomLLMsisintensiveIofteninvolvingdomain-specificdatasetsandextensivefine-tuningtoachievedesiredperformancelevels.ThisapproachismoreakintoMLOpsbuildingMLmodelsfromthegroundupIwithdetaileddataanalysisIcollectionformattingIcleaningIandlabeling.Oneofthebenefitsofthisapproachisthatyouknowthelineageandsourceofthedatathemodelisbuiltonandcanattestdirectlytoitsvalidityandfit.HoweverIamajordownsideistheresourcesIcostIandexpertisenecessarytobuildItrainIandverifyamodelthatmeetstheprojectobjectives.CustomLLMsprovidetailoredsolutionsoptimizedforspecifictasksanddomainsIofferinghigheraccuracyandalignmentwithanorganization'sspecificneeds.

ConsumeandCustomizeExistingModels:Pre-trained(foundation)modelsIwhetherself-hostedorofferedasaserviceIsuchaswithChatGPTIBertandothersontheotherhandprovideamoreaccessibleentrypointfororganizations.ThesemodelscanbequicklydeployedviaAPIsIallowingforrapidsolutionvalidationandintegrationintoexistingsystems.TheLLMOpsprocessinthisscenarioemphasizescustomizationthroughfine-tuningwithspecificdatasetsIensuringthemodelmeetstheapplication'suniquerequirementsIfollowedbyrobustdeploymentandmonitoringtomaintainperformanceandsecurity.

Version1.010of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

LLMOpsandLLMSecOpsDefined

HavingacommonviewoftypicalLLMapplicationarchitecturesIincludingagentsImodelsILLMsIandtheMLapplicationstackIiscrucialfordefiningandaligningtheapplicationstackandsecuritymodel.

(figure:LLMOpsrelatedOperationsProcessforDataIMachineLearningandDevOps)

AQuickOPsPrimer-FoundationforLLMOPs

DevOpsIwhichemphasizescollaborationIautomationIandcontinuousintegrationanddeployment(CI/CD)Ihaslaidthegroundworkfore代cientsoftwaredevelopmentandoperations.BystreamliningthesoftwaredevelopmentlifecycleIDevOpsenablesrapidandreliabledeliveryofapplicationsIfosteringacultureofcollaborationbetweendevelopmentandoperationsteams.

DataOpsbuildsonDevOpsIwheredatapipelinesaremanagedwithsimilarautomationIversioncontrolIandcontinuousmonitoringIensuringdataqualityandcomplianceacrossthedatalifecycle.MLOpsalsoextendstheDevOpsprinciplestomachinelearningIfocusingontheuniquechallengesofmodeldevelopmentItrainingIdeploymentIandmonitoring.UtilizingDevOpsasafoundationensuresthatbothDataOpsandMLOpsinheritarobustinfrastructurethatprioritizese代ciencyIscalabilityIsecurityIandfasterinnovationindata-drivenandmachinelearningapplications.

MLOpsandDataOpsarefoundationaltoLLMOpsbecausetheyestablishthecriticalprocessesandinfrastructureneededformanagingthelifecycleoflargelanguagemodels(LLMs).DataOpsensuresthatdatapipelinesaree代cientlymanagedIfromdatacollectionandpreparationtostorageandretrievalIprovidinghigh-qualityIconsistentIandsecuredatathatLLMsrelyonfortrainingandinference.MLOpsextendstheseprinciplesbyautomatingandorchestratingthemachinelearninglifecycleIincludingmodeldevelopmentItrainingIdeploymentIandmonitoring.

Version1.011of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

LLMOpsandMLOpsIwhilerootedinthesamefoundationalprinciplesoflifecyclemanagementIdivergesignificantlyintheirfocusandrequirementsduetothespecificdemandsoflargelanguagemodels(LLMs).LLMOpsencompassesthecomplexitiesoftrainingIdeployingIandmanagingLLMsIwhichrequiresubstantialcomputationalresourcesandsophisticatedhandling.LLMOpsensurethatLLMsaree代cientlyintegratedintoproductionenvironmentsImonitoredforperformanceandbiasesIandupdatedasneededtomaintaintheireffectiveness.ThisholisticapproachensuresthatthedeploymentandoperationofLLMsarestreamlinedIscalableIandsecureIincludingconsiderationsfordatavalidationandprovenancetoensurethatthedatausedfortrainingandfine-tuningLLMsistrustworthyandfreefromtampering.Thiscanincludetechniquesfordataauditingandverification.

LLMOPsLifeCycleStages-FoundationforLLMDevSecOPs

AsmentionedearlierinthisdocumentItoalignsecuritysolutionsforLLMapplicationsforoursolutionguideweareusingtheLLMOpsprocesstodefinethesolutioncategoriessothattheyalignwiththechallengesdevelopersarefacingindevelopinganddeployingLLM-basedapplications.

(figure:CombinedLLMCustomandLLMPre-TrainedImage)

TheLLMOpsprocessesdiffersignificantlybetweenusingpre-trainedLLMmodelsforapplicationdevelopmentandcreatingcustomLLMmodelsfromscratchusingopen-sourceandcustomdatasetsIwhichinheritmorefromMLOpspracticeswithsomeadditions.WefirstneedtodefinethestagesIthetypicaldevelopertasksIandthesecuritystepsateachstageofthelifecycle.

Version1.012of34

|GenAI,LLMSecOpsandSecuritySolutionLandscape

(figure:LLMopsPre-TrainedProcessandSteps)

Thesephaseswehavedefinedinclude:Scope/PlanIModelFine-Tuning/DataAugmentationITest/EvaluateIReleaseIDeployIOperateIMonitorIandGovern.OfcourseIthisisaniterativeapproachIwhetheryouarepracticingwaterfallIagileIorahybridapproacheachofthesestepscanbeleveraged.

Scoping/Planning

Thefocusisondefiningtheapplication'sgoalsIunderstandingthespecificneedstheLLMwilladdressIanddetermininghowthepre-trainedmodelwillbeintegratedintothelargersystem.ThisstageinvolvesgatheringrequirementsIassessingpotentialethicalandcomplianceconsiderationsIandsettingclearobjectivesforperformanceIscalabilityIanduserinteraction.TheoutcomeisadetailedprojectplanthatoutlinesthescopeIresourcesIandtimelinesneededtoimplementtheLLM-poweredapplicationsuccessfully.

TypicalActivities:

LLMOps

LLMSecOps

●

DataSuitability

●

AccessControlandAuthentication

●

ModelSelection

Planning

●

Requirem