OWASP：人工智能安全卓越中心指南（英文）

secureAlAdoptionInitiative

AICybersecurityCenterofExcellenceGuide

RevisionHistory

Revision

Date

Authors

Description

.1

5/15/2024

ScottClinton

InitialOutlineDraft

.2

7/2/2024

ScottClinton,

SandyDunn,Team

Updatedwithinitialcomments

.5

7/10/2024

OpenFeedbackandComment

Earlydraft,openforcommentandinput

1.0rc

10/1/2024

ScottClinton,ContributingandReviewTeams-See

Acknowledgements

FirstReleaseincorporatingallfeedbackasof9/27/24

Theinformationprovidedinthisdocumentdoesnot,andisnotintendedto,constitutelegaladvice.Allinformationisforgeneralinformationalpurposesonly.Thisdocumentcontainslinkstootherthird-partywebsites.SuchlinksareonlyforconvenienceandOWASPdoesnotrecommendorendorsethecontentsofthethird-partysites.

LicenseandUsage

ThisdocumentislicensedunderCreativeCommons,CCBY-SA4.0Youarefreeto:

●Share—copyandredistributethematerialinanymediumorformat

●Adapt—remix,transform,andbuilduponthematerialforanypurpose,evencommercially.Underthefollowingterms:

●Attribution—Youmustgiveappropriatecredit,providealinktothelicense,andindicateifchangesweremade.Youmaydosoreasonably,butnotinanywaythatsuggeststhelicensorendorsesyouoryouruse.

○AttributionGuidelines-mustincludetheprojectnameandthenameoftheassetReferenced.

■OWASPTop10forLLMs-LLMAISecurityCenterofExcellence(CoE)Guide

■OWASPTop10forLLMs-LLMAISecurityCenterofExcellenceGuide

■OWASPTop10forLLMs-LLMAISecurityCoEGuide

●ShareAlike—Ifyouremix,transform,orbuilduponthematerial,youmustdistributeyourcontributionsunderthesamelicenseastheoriginal.

Linktofulllicensetext:/licenses/by-sa/4.0/legalcode

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

2

AICybersecurityCenterofExcellenceGuide

Contents

Introduction5

Whoisthisfor?5

CreatingtheCOEStructure6

COEConsiderations7

SomeExamplesofInternalChallenges7

UtilizingExternalExpertise7

SettingOverallObjectivesandKPIs-Top58

Objective1:EnhanceSecurityFrameworksforGenerativeAI8

Objective2:FosterCollaborationandKnowledgeTransfer8

Objective3:BuildTrustandTransparencywithStakeholders9

Objective4:AdvanceEthicalAIandSecurityPractices9

Objective5:OptimizeAIPerformanceandReliability10

WorkingGroupRolesandResponsibilities11

BuildersandOperatingGroups11

AIandMLDevelopers11

CybersecurityTeam12

ITandOperations14

LegalandCompliance15

EthicsandGovernance16

HumanResources17

RiskManagement18

DataScienceTeam19

UserandConsumerGroups20

MarketingandCommunications20

CustomerSupport22

LineofBusinessRepresentation24

ExampleImplementationPhases&Timeline25

Phase1:PlanningandSetup(Months1-3)25

Phase2:IntegrationandDevelopment(Months4-6)26

Phase3:Operationalization(Months7-9)26

Phase4:EvaluationandExpansion(Months10-12)26

EmergingTrendsinAISecurity27

Summary28

Glossary29

Acknowledgements30

OWASPTop10forLLMProjectSponsors31

SilverSponsors31

References32

ProjectSupporters33

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

3

AICybersecurityCenterofExcellenceGuide

Introduction

AsgenerativeAItechnologiesevolveandintegrateintovariousaspectsofbusinessandsociety,theneedforrobustgovernance,security,andpolicymanagementbecomesparamount.EstablishingaCenterofExcellence(COE)forGenerativeAISecurityaimstobringtogetherdiversegroupssuchassecurity,legal,datascience,operations,andend-userstofostercollaboration,developbestpractices,andensuresafe,e代cientdeploymentofAIcapabilities.

Whoisthisfor?

ThisdocumentisforCISOsecurityteamsandcross-functionalleadershiptogainanunderstandingandbestpracticesframeworktoassistineducatingteamsonimplementingtheircenterofexcellenceforLLMandGenerativeAIApplicationsecurityandadoption.

Objective

TheprimaryobjectiveoftheCOEistodevelopandenforcesecuritypoliciesandprotocolsforgenerativeAIapplications,facilitatecross-departmentalcollaborationtoharnessexpertisefromvariousfields,educateandtrainteamsontheethicalandsecureuseofgenerativeAItechnologies,andserveasanadvisorybodyforAI-relatedprojectsandinitiativeswithintheorganization.

CreatingaCOEforGenerativeAISecurityisacriticalsteptowardensuringthatgenerativeAItechnologiesaredeveloped,deployed,andmaintainedsecurelyandethically.Througheffectivecollaborationandgovernance,theCOEwillplayapivotalroleinshapingthefutureofAIwithintheorganization.

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

4

AICybersecurityCenterofExcellenceGuide

CreatingtheCOEStructure

CreatinganAISecuritycenterofexcellence(COE)formanagingandsecuringgenerativeAIapplicationsinvolvesstrategicplanningandcollaborationacrossmultipledepartments.

Theleadershipteamiscomprisedofheadsfromsecurity,legal,datascience,andoperations.ThisteamisresponsibleformakingstrategicdecisionsanddirectingtheCOE.SpecificworkinggroupsfocusedonkeyareassuchasSecurityandCompliance,LegalandRegulatoryAffairs,DataManagementandAnalytics,OperationalIntegration,andEnd-UserEngagement.

TheCOE’sstructureshouldbeflexibleandadaptable,evolvingalongsidethefast-pacedadvancementsinAItechnologies.Regularreviewsofroles,responsibilities,andprocessesareessentialtoensuretheCOEremainseffectiveandalignedwiththeorganization’sstrategicobjectives.

Theoverallgroupistaskedwithaddressingspecificchallengesanddeliveringsolutionsrelevanttotheirexpertise,whichinclude:

●PolicyDevelopment:CreatingsecuritypoliciestailoredtogenerativeAI.

●RiskAssessmentandManagement:Identify,evaluateandmonitorpotentialrisksanddevelopmitigationstrategies.

●TrainingandAwareness:Conductregulartrainingsessionsandworkshopstoinformallstakeholdersaboutbestpracticesandemergingthreats.

●ResearchandDevelopment:StayabreastofthelatestdevelopmentsinAIandsecuritytocontinuouslyrefinestrategiesandtools.

●StakeholderEngagement:Regularlyinvolveend-usersandotherstakeholdersin

decision-makingtoensuretheCOE'sinitiativesalignwithuserneedsandorganizationalgoals.

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

5

AICybersecurityCenterofExcellenceGuide

COEConsiderations

Leveragingamultidisciplinaryteambringstogetherdiverseskillsandperspectives,whicharevitalforaddressingthecomplexsecuritychallengesofAI.ButestablishingamultidisciplinaryteamwithinaCOEpresentsseveralchallengesthatneedcarefulmanagement.InadditiontoaddressingthesechallengesandenhancingtheCOE'scapabilities,leveragingexternalexpertisecanbehighlybeneficial.Withdiverseexpertiseandbackgrounds,integratingprofessionalsfromsecurity,legal,datascience,andoperationscanleadtoconflictsduetodifferingprioritiesandperspectives.Aligningthesediverseviewpointstowardscommonobjectivesiscritical.

SomeExamplesofInternalChallenges

●CommunicationBarriers:Effectivecommunicationamongteammemberswithvaried

professionallanguagesandmethodologiescanbedi代cult.Establishingacommonlanguageorsetofterminologiesisessentialforseamlesscollaboration.

●ResistancetoChange:Individualsfromdifferentdepartmentsmayresistnewworkflowsor

changesthatdisrupttraditionalprocesses.Managingchangeeffectivelyandensuringbuy-infromallstakeholdersiscrucial.

●ResourceAllocation:Competingforresourcesamongdifferentdepartmentscancreatefriction.Transparentandequitableresourcedistributionpoliciesneedtobeestablished.

●SkillGaps:AsgenerativeAIisarelativelynewandrapidlyevolvingarea,theremaybesignificantgapsinthenecessaryskillsamongexistingstaff,whichcanhindertheCOE'seffectiveness.

UtilizingExternalExpertise

LeveragingexternalexpertisecanprovecrucialtoenhancingthecapabilitiesoftheCenterofExcellence(COE)andaddressingitschallenges.EngagingconsultantsandadvisorsspecializinginAIsecurity,legalregulationsrelatedtoAI,anddataethicscanprovidetheCOEwithcriticalinsightsandguidance,helpingshapeeffectivepoliciesandprocedures.Additionally,partneringwithexternaltrainingproviderscanaddressskillgapswithintheteambyofferingspecializedtraininginAIandcybersecurity,ensuringthatallmembersareproficientinthelatesttechnologiesandindustrybestpractices.

Collaboratingwithtechnologypartners,suchastechcompaniesandvendors,offersCOEaccesstoadvancedtoolsandplatformsthatboostoperationalcapabilities.Furthermore,formingpartnershipswithacademicandresearchinstitutionsfacilitatescontinuouslearningandhelpstheCOEstayabreastofnewdevelopmentsinthefieldofgenerativeAI.

Engagingwithindustrygroupsandnetworksalsoaidsinunderstandingbroadertrends,gatheringinsightsfromsimilarinitiatives,andadoptingindustry-widebestpractices,allofwhichcontributetothestrategicgrowthandeffectivenessoftheCOE.

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

6

AICybersecurityCenterofExcellenceGuide

SettingOverallObjectivesandKPIs-Top5

Establishingamulti-disciplinaryCenterofExcellence(COE)fortrustworthyandsecuregenerativeAIadoptionrequiresclearobjectivesandmeasurableKeyPerformanceIndicators(KPIs).HerearetopfiveexamplesofObjectivesandKeyResults(OKRs),alongwiththeircorrespondingKPIsthatcanserveasastartingpointforbuildinganoperatingplanforyourCOE.Eachobjectiveshouldbesupportedbywell-definedKPIsthatassesscompliance,securityperformance,andtheirimpactonoverallbusinessoperations.ThisensuresthatAIsecurityinitiativesarenotonlytechnicallysoundbutalsoalignedwiththeorganization’sstrategicpriorities.

Objective1:EnhanceSecurityFrameworksforGenerativeAI

ThisobjectivefocusesondevelopingandimplementingcomprehensivesecuritypoliciestailoredtotheuniqueneedsofgenerativeAIapplications.Fullcompliancewithnationalandinternationalregulationsandreducingtheincidenceofsecuritybreachesarecritical.Bystrengtheningthesecurityframeworks,theCOEaimstoprotectAIsystemsagainstevolvingthreatsandvulnerabilities,ensuringrobustdefensemechanismsareinplace.

ExampleOKRs

ExampleKPIs

Developnew(orrefineexisting)andimplementcomprehensivesecuritypoliciesspecificto

generativeAIapplications.

Thenumberofsecuritypoliciesdeveloped(orrefined)andimplemented.

Achievefullcompliancewithnationaland

internationalregulationsconcerningAIsecurityanddataprivacy

Thecomplianceratewithrelevantregulations.

ReducetheincidenceofsecuritybreachesrelatedtoAIbyX%withinthenextyear.

FrequencyandseverityofsecuritybreachesinvolvingAItechnologies.

Objective2:FosterCollaborationandKnowledgeTransfer

ThegoalhereistoenhancesynergyamongvariousdepartmentswithintheCOEthroughregularworkshops,trainingsessions,andacentralizedknowledgebase.Thisobjectiveseekstoimproveteamcommunicationandcollaboration,fosteringanenvironmentwheresharedresourcesandcollectiveexpertisecontributetoinnovativesecuritysolutionsande代cientproblem-solving.

ExampleOKRs

ExampleKPIs

EstablishregularworkshopsandtrainingsessionsforalldepartmentsinvolvedintheCOE.

Thenumberofinterdepartmentalseminarsandtrainingsessionsconducted.

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

7

AICybersecurityCenterofExcellenceGuide

Createacentralizedknowledgebaseaccessibleto

allCOEmembers,containingup-to-dateinformationonAIsecuritytrendsandbestpractices.

UtilizationrateoftheknowledgebasebyCOEmembers.

AchieveanX%increaseincross-departmental

projectsfocusedonenhancingAIsecuritywithinthenextyear.

Thenumberofcross-departmentalinitiativeslaunched.

Objective3:BuildTrustandTransparencywithStakeholders

Buildingtrustandmaintainingtransparencyarepivotalinthisobjective.Bydevelopingandreleasingquarterlyreportsandconductingbi-annualstakeholdermeetings,theCOEaimstoengageopenlywithallstakeholders,ensuringtheyarewellinformedabouttheorganization'sAIsecurityinitiatives.ThisapproachhelpstocultivatearelationshipoftrustandfostersgreateracceptanceandsupportforAItechnologies.

ExampleOKRs

ExampleKPIs

Developandreleasequarterlyreportsonthe

organization’sAIsecuritystatusandinitiatives.

Thenumberofquarterlyreportspublished.

ThetotalcountofnewAIsecurityinitiativesthataredevelopedandimplementedeachquarter.

Conductbi-annualstakeholdermeetingstogatherfeedbackanddiscussconcernsregardingAI

security.

Thenumberofstakeholdermeetingsheld.

Thenumberofconcernsgatheredanddocumentedduringheldmeetings.

Implementandtrackastakeholderengagementprogramaimedatimprovingtransparencyandtrust.

Engagementlevelsandfeedbackscoresfromstakeholdersandpublicsurveys.

Percentageofinvitedstakeholderswhoattendthemeetings.

Objective4:AdvanceEthicalAIandSecurityPractices

ThisobjectiveaimstoembedethicalconsiderationsdeeplywithinAIoperations,ensuringasolidethicalframeworkguidesallAIdeployments.TheCOEcommitstopromotingresponsibleAIusagethatalignswithsecurityguidelinesandorganizationalethicsbydevelopingethicalguidelines,conductingannualaudits,andincreasingawarenessofthesestandards.

ExampleOKRs

ExampleKPIs

DevelopandenforceasetofethicalandsecurityguidelinestailoredforgenerativeAIusewithinthe

Thenumberofethicalguidelinesdevelopedandimplemented.

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

8

AICybersecurityCenterofExcellenceGuide

organization.

Thepercentageofemployeeswhohavecompletedtrainingprogramsonthe

proposed/developedethicalandsecurityguidelines.

ConductanannualaudittoensureadherencetoethicalAIandsecuritystandardsandpractices.

Resultsoftheethicsandsecurityaudit,detailingcomplianceandareasfor

improvement.

AchieveanX%increaseinawarenessand

understandingofethicalAIprinciplesamongemployeesbytheendoftheyear.

IncreaseinemployeescoresonAIsecurityandethicalAIawarenessassessments.

Objective5:OptimizeAIPerformanceandReliability

OptimizingtheperformanceandreliabilityofAIsystemsiscrucialtothisobjective.ItinvolvesenhancingAIsystemaccuracy,reducingerrorrates,andensuringsystemscanoperateeffectivelyundervariousconditions.TheCOEaimstodeliverhigh-performingAIapplicationsthatstakeholderscanconsistentlyrelyonbydeployingtoolsandupdatesthatimprovesysteme代ciency.

ExampleOKRs

ExampleKPIs

ReduceerrorratesinAIsystemsby25%overthenext12months.

ReductionpercentageinerrorratesofAIandAI-basedapplications.

Implementasystemforcontinuousmonitoringandreal-timeanalysisofAIsystemperformance.

Implementationstatusofthecontinuousmonitoringsystem.

Developanddeployatleasttwonewtoolsor

updatesthatimprovethee代ciencyandreliabilityofAIoperations.

NumberandimpactoftoolsorupdatesdeployedforimprovingAIperformance.

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

9

AICybersecurityCenterofExcellenceGuide

WorkingGroupRolesandResponsibilities

BuildersandOperatingGroups

AIandMLDevelopers

IncludingAIandMLDevelopersintheCOEiscrucialforbridgingthegapbetweentheoreticalsecuritymeasuresandpractical,actionableAIimplementations.Theirexpertiseensuresthatsecurityisembeddedinthedevelopmentphase,enhancingtherobustnessofAIapplicationsfromthegroundup.

Withtheadoptionofshift-leftstrategiesforapplicationsecurityandownershipappliestoAIandMLdevelopersaswell.InvolvingAIandMLdeveloperrepresentationhelpstoensurethatsecurityisafoundationalelementofAIapplications,leadingtomoresecuresolutions.Proactiveriskidentificationandmitigationbydevelopersreducethelikelihoodofsecuritybreachesanddataleaks,therebyprotectingtheorganizationanditscustomers.

DevelopersalsocontributetechnicalinnovationsthatenhanceAIsecuritymeasures,resultinginmorerobustsystems.Ensuringthatdevelopmentpracticescomplywithregulatoryrequirementspreventslegalissuesandbuildstrustwithregulatorsandstakeholders.Additionally,developersintegratesecuritypracticesacrossdifferentfunctionalareasofAIprojects,aligningandinformingallteammembers.DevelopersshouldhaveaccesstothelatestsecuritytoolsandtrainingtoseamlesslyintegratesecurityfeaturesintotheAIdevelopmentprocess.Thisproactivestrategyallowsfortheearlydetectionandmitigationofpotentialvulnerabilitiesthroughoutthedevelopmentlifecycle.

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

10

AICybersecurityCenterofExcellenceGuide

COEResponsibilities-AIandMLDevelopers

SecureDevelopmentPractices

Implementandadheretosecurecodingpracticestailored

explicitlytoAIandMLprojects.Thisincludesusingsecure

frameworks,regularcodereviews,andintegratingsecurityateachstageofthedevelopmentlifecycle.

VulnerabilityAssessment

Proactivelyidentifyandaddresspotentialvulnerabilities

withinAIalgorithmsanddataprocessingmethods.This

includesperformingthreatmodelingandriskassessmentsduringtheearlystagesofdevelopment.

SecurityTesting

RegularlyconductsecuritytestsonAImodelsand

applications,includingunittesting,toensuretheycanwithstandattacksandperformreliablyunderadverse

conditions.Thismayincludepenetrationtesting,stresstesting,andscenario-basedtesting.

CollaborationandKnowledgeSharing

Sharetechnicalknowledgeandinsightsthatcanaid

non-technicalteammembers.Workcloselywithother

departmentswithintheCOE,suchasRiskManagementandDataScience,toensureaholisticapproachtoAIsecurity

InnovationandResearch

Participateinresearcheffortstodevelopnewsecurityfeaturesorenhanceexistingones.

ComplianceandEthics

EnsurethatallAIdevelopmentiscompliantwithrelevantlawsandethicalguidelines.Thisincludestheresponsibleuseof

data,transparencyinAIoperations,andthemitigationofbiasesinAImodels.

DocumentationandReporting

MaintaincomprehensivedocumentationofallAIdevelopment

processes,securitymeasures,andtestingresults.This

documentationiscrucialforauditpurposesandsharingCOEbestpractices.

CybersecurityTeam

IncorporatingtheCybersecurityteamintheCOEisessentialtoensurethatAItechnologiesareprotectedfromemergingthreatsandvulnerabilities.Theirexpertiseindigitalsecurityformsacriticalbackbonefordeveloping,deploying,andmaintainingAIsystemsthataresecureandresilientagainstcyberthreatsincludingAi-aidedphishing,deepfake,andemergingAI-aidedthreats.Strongcybersecuritypracticesareessentialforbuildingtrustamongusersandstakeholders,makingAItechnologiesreliableandsecure.

Thecybersecurityteam'sexpertiseisvitalinprotectingAIsystemsfromsophisticatedattacksandpotentialexploits,ensuringcompliancewithrelevantlawsandregulations,safeguardingtheorganizationfromlegalissuesandenhancingitsreputation.

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

11

AICybersecurityCenterofExcellenceGuide

Additionally,effectiveincidentresponseandmanagementmaintaintheoperationalintegrityofAIsystems,enablingthemtofunctioneffectivelyevenduringattacks.

Thecybersecurityteamshouldfocusonimplementingautomatedmonitoringsystemsthatprovidereal-timealertsforpotentialthreats,enablingrapidresponseandminimizingtheriskofincidentescalation.ThecybersecurityteamalsointegratessecuritypracticesacrossallaspectsofAIdevelopmentanddeployment,creatingaunifiedsecurityframeworkwithintheorganization.

COEResponsibilities-CybersecurityTeams

ThreatIdentificationandAnalysis

ContinuouslyidentifyandanalyzepotentialcyberthreatstoAIsystems.Thisincludesmonitoringfornewvulnerabilities,

predictingpossibleattackvectors,andunderstandingtheimplicationsofthesethreatsonAIoperations.

SecurityProtocolDevelopment

DeveloprobustsecurityprotocolstailoredexplicitlytoAIandMLtechnologies.Thisinvolvescraftingcustomizedsolutionstoprotectdataintegrity,ensureprivacy,andsafeguardAI

systemsagainstunauthorizedaccess.

IncidentResponseandManagement

Establishandmanagearapidresponseframeworkforany

securityincidentsinvolvingAIsystems.Thisframeworkshouldincludereal-timethreatdetection,containmentstrategies,

andrecoveryplanstominimizedisruptionanddamage.

SecurityTestingandAudits

ConductcomprehensivesecuritytestingandauditsofAI

systemstovalidatetheeffectivenessofsecuritymeasures.Thisinvolvespenetrationtesting,securityassessments,andcompliancechecksagainstindustrystandards.

TrainingandAwarenessPrograms

Developandconductcomprehensivesecuritytestingand

auditsofAIsystemstovalidatetheeffectivenessofsecurity

measures.Thisinvolvespenetrationtesting,securityassessments,andcompliancechecksagainstindustrystandards.

BugBountyProgramManagement

Register,implement,andmanageabugbountyprogramonvulnerabilityreportingplatformstoproactivelyidentifyandremediatevulnerabilitiesinAIsystemswiththesupportofexternalsecurityresearchers.

CollaborationwithRegulatoryBodies

Engagewithregulatorybodiestoensurecompliancewith

nationalandinternationalcybersecurityregulations.This

includesadaptingAIsecuritypracticestomeetevolvinglegalandregulatoryrequirements.

InnovationinSecurityTechnologies

Stayabreastofthelatestdevelopmentsincybersecuritytechnologyandintegratecutting-edgesolutionsintotheorganization’sAIsystemstoenhanceitssecurityposture.

Version:1.0rcFromtheOWASPTop10forLLM,GenAITeam-

12

AICybersecurityCenterofExcellenceGuide

ITandOperations

TheITandOperationsteamiscrucialforensuringthetechnicalinfrastructureandoperationalproceduressupport