Capgemini-生成式人工智能与网络安全：经典之作（英）

GenerativeAIandCybersecurity:

Arevisitedclassic

ThegreatestriskswhenincorporatinggenerativeAIintoabusinessstructureare:

Misleadingoutcomesduetomodelhallucination

Dataleakageandcopyrightissuesduetounintentionaldisseminationorinclusionofregulatedor

company-confidentialdata

Trainingdata–subjects’privacyandconsentviolationswith

Modelcorruptionandabusewhenretrainingisbasedoncustomerresponsedata

AI

inadequateneed-to-knowandneed-to-useintrainingdataanddataoutputsmanagement

MeetingregulatoryandethicalresponsibilitiesinGenAIuse

Ethicalissuesorbiased

conclusionsbecauseofinaccurate,

incomplete,ortamperedtrainingdata

Thebiggest

risksaretodata

WhendesigningforsecuregenerativeAI,datariskstakepriority.Broadlyspeaking,theserisksoriginatefromthreeactivities:

Theexposureofconfidentialand/orregulatedinformation

Inaccurateinformationdisruptsprocesses,whetherdecisionaloroperational

GenAIfollowsafamiliarpatternforadoptionandcybersecurity,

promptingquestionsreminiscentofthosethataccompaniedthe

earlydaysofcloudcomputing.TherapidriseofgenerativeAI

presentsorganizationswiththeusualinnovationdilemma:isit

bettertoadoptacautiousandrestrictiveapproach,riskingmissingoutonopportunities,ortograntmorefreedom,attheriskof

exposingthemselvestonewrisks?

PotentialreputationaldamageiscausedwhenGenAItoolsareusedaschatbotsservingasinterfacesbetweencustomersandanorganization

Theseriskshavecommonthemesofidentifying,scrubbing,andprotectingtherightdataatthe

righttimeandputtingtherightguardrailsinplacearoundaGenAIsolution.Despiteitspotentialandtheexcitementsurroundingit,GenAIisultimatelyanotherenterprisetool:itrequirestheapplicationandadaptationofpolicies,controlsandmeasures

implementedatenterpriselevelandwithintheAIecosystem.Itbringschallengesofoperatingmodelsinternallyandmonitoringtheirinputandoutputcompliantly.

InaGenAIsystem,foundationalsecuritymustbedoneacrossfourdimensions:

.Framework,governance,andriskmanagement

.Dataandidentitysecurity

.TrustedGenAImodelsandtheiroutcomes

.Infrastructureandapplicationmonitoringanddelivery

ThreatmodelsareavailablefromNIST,MITRE,

Microsoft,Google,andothersintheindustrytobuildfasterandbereadyfornewrisks.

AGenAIsystemcanhavedifferentsecurityscopes.Usingcloudserviceproviders(CSP)asexamples,eachCSP(alsoknownashyperscalers)offersgenerative

AIsystemswithverydifferentsecurityscopes,

andeachproviderdefinesthisscopedifferently.

ConsidersharedresponsibilityaroundthereferencearchitecturefoundinFigure1.

2|GenerativeAI&Cybersecurity

GenerativeAI&Cybersecurity|3

Data

Datacollection,datapreparationandtransformation

Varioususecasesthatmatterstotheendusersandarerelevantbusinesscases

modelsthataretailoredtoagivenindustryorusecaseToolstooperationalizeGen-AImodels

Gen-AIapplicationssuchascompute,networkandstorage

Applications

SoftwareapplicationsthatprimarilyuseGen-AImodelstoperformatask

Monitoring&Maintain

Monitorperformance,userexperienceandoutcomequality

Models&Tools

Gen-AIfoundationmodels&domainspecific

Infrastructure

Infrastructurecomponentsusedtobuildout

Network

Communication

Storage

Compute

Figure1:ConceptualreferencearchitectureforGenAIsharedresponsibility.

AmazonWebServicesfocusesonprovidingthe

infrastructureforgenerativeAImodels,aswith

AmazonBedrock.Variousdegreesofcustomizationandownershiparepossible.Theclient’ssystemis

definedastheprovidedinfrastructure,andtheirpartofsharedresponsibilityincludesthesecurityofthemodels,data,andapplications.

GoogleCloudPlatform’s(GCP)approachfocusesontheinfrastructureandmodels,offeringVertexAIandtheModelGardentoempowercustomers.Customers

focusontheapplicationlayer,monitoring,andtheGenAIinterface,whileGCPhassharedresponsibilityfromthemodeldowntodataandinfrastructure.

WithMicrosoftAzure’sCo-pilot,theCSPtakes

ownershipofinfrastructure,model,application,and

everythinginbetween..Thecustomerfocusesondatasecurityandbusinesspurposes.Datainterfacesdefinetheirsystem,whilethemodels,infrastructure,and

applicationinterfacearetreatedmoreasblackboxes.

4|GenerativeAI&CybersecurityGenerativeAI&Cybersecurity|5

Establishing

asecurity

frameworkwithgovernance

PositionsonhowtoregulateGenAIvarywidely,

fromoutrightprohibitiontocompletelaissez-faire.Nosinglegovernmentorsupranationalpolitical

entitywillbeabletodictatehowGenAIproliferates.Nevertheless,enterprisesmustworkwithinlegal

andregulatorystructuresbasedontheirclients,geographies,andethics.

Toanticipatewhat’sexpectedingenerativeAIgovernance,enterprisesshouldconsiderthefollowing:

.ExistingandupcomingregulationsthatwillinfluenceAIuse

.Anenterprise’suniquerisktolerancesfortechnologyandregulations

.TeammembereducationonhowGenAIworks,itsinherentproblems,andriskssuchasdataleaksandtheorganization’sownpolicies

.AsecureGenAIreferencearchitecturedescribinghowtomanagerisks

Thereferencearchitecturemustaddresstherisksofvariousmodelsindiverseways.Afullproprietarysolution,includingGenAImodeldevelopmentandpre-training,meansanorganizationwillhavethe

abilityandobligationtoaddressitsspecificrisksend-to-end.

InthecaseofSoftware-as-a-ServicegenerativeAI,manyrisksneedtobeaddressedthroughcontractandthird-andfourth-partyriskmanagement.

OrganizationscanalsodeploymorethanoneGenAIsolutionwithdifferentarchitecturemodels,andhybridmodels.

Governancebodies-suchasaGenerativeAICenterofExcellence-areneededinenterprisestohelpshape

thesecureadoptionofGenAI.Theyhelpaccelerate

low-risk,high-impactbusinessexperimentswhile

enforcingappropriateoversightofhigh-riskplans.Bydevelopingrepeatable,enforceable,anddisseminatedguidelines,enterprisescanleverageGenAIsolutionsmorequicklyandsecurely.

Providerassumedresponsibility

SaaS

ExternalModel

PaaS

IaaS

Applications

Monitoring&Maintain

Models&Tools

Data

Infrastructure

M

Network

Communication

Storage

Compute

Figure2:SharedresponsibilitymodelsforvariouscloudproviderGenAIdeliverymodels

6|GenerativeAI&CybersecurityGenerativeAI&Cybersecurity|7

SecuringData

GenAIlackshumanfilterswhenitproducesdata:themachinesearchesthrougheverythingitcanaccess

andthenreproducesthisknowledgewithcompletecandorregardlessofsensitivity.Itis,therefore,

imperativetosetlimits.Todothis,enterprisesmust

inventorytheirdata:classifyit;implementcontrolsforquality,representativeness,integrity,andaccess;andcreaterepositoriesofauthorizeddataforGenAIapplications.

GenAI’sconsumptionofdatamakesdata

classificationevenmoreessentialtoadequately

protectanenterpriseandcustomers.Classification

allowstightercontrolofdatausedtotrain,specialize,andrefinemodels.Accesstoitsoutputcanbe

restrictedanddataleakprotectiontoolscanbe

implemented;oraresponsecanbelimitedusingasubsetofdatabasedonaright-to-knowrule.

Withathird-partyLLM,thereislimitedabilityto

build“native”guardrailsaroundinputsandoutputs.Likewise,theabilitytoimplementguardrailsinsidethelearningphasesofaGenerativeAdversarial

Network1islimitedwhenusingclosedmodelsinan

Data

1.Training

Themodelisbuiltwhich

encodestherealtionships,

patternsandsequences

withintrainingdataand

modelvalidationdata.

●

TrainedModel

●

3.EnsuringCorrectness

Thereisnoguranteeofreal-worldcorrectnessfromagenerativeAImodels,anditsometimes

hallucinatesfictionalresponse

2.Generation

Thetrainedmodelcanthengeneratenewoutputsliketheoriginaldataitwastrainedon

(Optional)

FineTuning

Thegenericfoundation

modelmightbefine-tuned

togiveitexposuretoa

specialistarea.

(Optional)Alignment

Modelmightbetweaked

toaligninmorewith

expectedhumanresponse

Figure3:DatalifecycleinsideagenerativeAIapplication

application.Itiscriticaltoconsiderwhetherdata

canbeinspectedandvalidated,andwhetherits

inputsandoutputscanbeobservedwhenchoosingcomponentsofasystem.

Amodel’soutputmustbesubjecttoverification

todetecthallucinations,maliciousreinforcement,

ordriftsfromexpectedbehaviorovertime.When

usingreal-timemodeloutput,suchaswithachatbot,theobservabilityofpastperformancetopreempt

unacceptableresponsesisimportant.Akeypointis

tounderstandthedatalifecycleanditssensitivity,ascapturedinFigure3.Datasecurityrequirementscanchangeoveritslifecycle,dependingonitsproximityto,orcominglingwithotherdata.

SuccessfullysecuringGenAIsolutionsisamulti-

disciplineapproachthatrequirespartnerships

betweencybersecurity,datagovernance,data

science,andlegalandcompliance,sincedisciplineddatamanagementisattheheartofachievingGenAIdatasecurity.

Dependencies

Data

Governance

Data

Sciences

Security

Legal&Compliance

Figure4:Multi-disciplineinteractionsnecessaryforGenAIsuccess

8|GenerativeAI&CybersecurityGenerativeAI&Cybersecurity|9

TrustedGen

AImodelsandtheiroutcomes

Itmaynotbepossibletogainaccesstoandthen

validatealldatasetsusedduringthelifecycleof

aGenAIsolution.Amodelsuchasthecommonly

usedLargeLanguageModel(LLM),multi-model

models,andtransformer-basedmodelsgeneratingoutcomesthroughuserpromptorAPIrequestscanfallintooneofthefollowingmodelcategories:

.Developedandinitiallytrainedbyanexternal

party(OpenAI’sChatGPT,forinstance)andused“asis”bytheenterprise

.Developedandinitiallytrainedbyanexternal

party,thenspecializedbytheenterprisetoa

specificdomain(i.e.,specialism)withanewdatasettoaddressspecificusecases

.DevelopedandtrainedbytheenterpriseentirelySupplychainsecurityandthird/fourth-partyrisk

managementarecrucialforthefirsttwocategories.

Itisevenmoreimportanttointegratesecuritycontrolssuchasmodelauditability,dataleakageprevention,hallucinationandbiasdetection(i.e.guardrails)intotheapplicationdevelopment

pipeline.

Dataquality

Therecurrentuseandprovenanceoftrainingdataisafocalpointwhenusingexternallysourcedmodels.Itscomposition,howoftenitchanges,andhowrecursionbetweencustomerprompt/responsepairingsand

reinforcementtrainingofthemodeloccursshouldbeclear.

Whendevelopingandtrainingaproprietarymodel(thirdcategoryabove),someriskscanbeamplifiedwhileothersaremitigated.Theneedtounderstanddata’sprovenanceandclassificationoftrainingdatawhilealsotestingforbiasandderogatoryresponsesfallsontheenterprise,eventhoughthosecanbe

differentdisciplines.Atthesametime,therisks

ofrecursivetrainingfromprompt/responsepairsarereducedastheinformationdoesn’tleavethelocalmodel.

Forallmodels,organizationsmustapplytheirownadditional,adaptablecontrols,suchas:

.Specificsecuritymonitoringrules

.Completelyoriginalmeasures,suchascontrolstodetectspecificnewattacksoruserbehaviors.

.Formultiandhybridarchitectures,APIsecurityandCI/CDsecure-by-designdomains

Thekeytoassuranceofdata’sintegrityisdue

diligenceonaprovider’ssecurity,privacycontrols,andcompliance.Theircommitmentsandresponsibilities

shouldbeclearlydefinedinanycontract.

10|GenerativeAI&CybersecurityGenerativeAI&Cybersecurity|11

Application

and

infrastructuremonitoring

anddelivery

ThefinalaspectofsecurityforGenAIisprotecting

applicationsfrombeingrenderedinoperativeor

unavailable.Thisrequiresdeployingsecuritycontrolswithinapplicationsandinfrastructure,covering

compute,endpoint,network,andstorage.

Thesamesecurityandcompliancehygieneappliedtoclassicsecuritymustbeappliedhere,especiallythosehandlingsensitivedata.Corporatesecurity

policiesandmandatorysecuritycontrolsovertheselayersareasimportantasever.

GenAIapplicationswillrequiresomenewsecuritycontrols,suchaspromptanalysis,andadaptationto

existingsecuritycontrols,suchasedgeprotection,tobeeffective.Buildingadequate,automated

governancearounddataclassificationandusageshouldbepartofanysecurityroadmap.

SoftwaresupplychainmanagementismoreimportantingenerativeAIapplicationdevelopment,e.g.,for

pinningdependencyversionsinmodeldevelopmenttoensuretrainingrunsdonotbecomecorrupted.Thisisimportantformonitoringanddeliverysinceitisa

partofthesoftwaredeliverylifecycle.Continuous

Integration(CI)andcontinuousdelivery(CD)throughaDevSecOpspipelineforapplicationdevelopment

canbeusedtosecuremodeldevelopment.Red

teaming2,anapplicationtotestforvulnerabilities,shouldincludetestingofanyprompts.Thisaimstostopmalicioususersfrom:

.Corruptingorrecoveringtrainingdata

.Manipulatingresultsforotherusers

.Performingdenialofserviceattacks

.Exfiltratingdata

AsgenerativeAIevolves,securityfunctionsnativetoGenAIwilltoo,aswilltheircapabilitiestointegratewithexternalsecuritysolutions.

12|GenerativeA