2025年前AIGC安全风险报告（英文版）

1

SafetyandSecurityRisksofGenerativeArtificial

Intelligenceto2025

Summary

2

GenerativeAIdevelopmenthasthepotential

tobringsignificantglobalbenefits.Butit

willalsoincreaseriskstosafetyandsecuritybyenhancingthreatactorcapabilitiesandincreasingtheefectivenessofattacks.

•ThedevelopmentandadoptionofgenerativeAI

technologieshasthepotentialtobringsubstantialbenefitsifmanagedappropriately.Productivity

andinnovationacrossmanysectorsincluding

healthcare,financeandinformationtechnologywillaccelerate.

•GenerativeAIwillalsosignificantlyincreaseriskstosafetyandsecurity.By2025,generativeAIismorelikelytoamplifyexistingrisksthancreatewholly

newones,butitwillincreasesharplythespeedandscaleofsomethreats.Thedificultyofpredicting

technologicaladvancescreatessignificantpotentialfortechnologicalsurprise;additionalthreats

willalmostcertainlyemergethathavenotbeenanticipated.

•Therapidproliferationandincreasingaccessibilityofthesetechnologieswillalmostcertainlyenableless-sophisticatedthreatactorstoconduct

previouslyunattainableattacks.

•Risksinthedigitalsphere(e.g.cyber-attacks,fraud,scams,impersonation,childsexualabuseimages)aremostlikelytomanifestandtohavethehighestimpactto2025.

•Riskstopoliticalsystemsandsocietieswillincreaseinlikelihoodasthetechnologydevelopsand

adoptionwidens.Proliferationofsyntheticmediariskserodingdemocraticengagementandpublictrustintheinstitutionsofgovernment.

Ourdefinitionsandscope

SafetyandSecurity:Theprotection,wellbeingandautonomyofcivil

societyandthepopulation.

ArtificialIntelligence(AI):Machine-drivencapabilitytoachieveagoalbyperformingcognitivetasks.

FrontierAI:Highlycapablegeneral-purposeAImodelsthatcanperformawidevarietyoftasksandmatchorexceedthecapabilitiespresentin

today’smostadvancedmodels.

GenerativeAI(GenAI):AIsystems

thatcancreatenewcontent.Themostpopularmodelsgeneratetextand

imagesfromtextprompts,butsomeuseotherinputssuchasimagestocreateaudio,videoandimages.

Largelanguagemodel(LLM):Modelstrainedonlargevolumesoftext-baseddata,typicallyfromtheinternet.

Risk:Asituationinvolvingexposuretodetrimentalimpacts.

Threat:Amaliciousriskinvolvinganactorwithintent.

Thisassessmentdoesnotconsider

militaryrisksrelatingtoGenerativeAI.

•PhysicalsecurityriskswilllikelyriseasGenerativeAIbecomesembeddedinmorephysicalsystems,includingcriticalinfrastructure.

•Theaggregateriskissignificant.Thepreparednessofcountries,industriesandsocietyto

mitigatetheserisksvaries.Globallyregulationisincompleteandhighlylikelyfailingtoanticipatefuturedevelopments.

Thisassessmentdrawsonabroadrangeofsourcesincludingexistingandnovelresearch,intelligenceassessments,expertinsightsandopensource.

Detail

3

1.Thedevelopmentandapplicationofgenerative

AIintersectswithmanyothertechnologies.Its

developmentandusewillhavebroadimpacts

-positiveandnegative-internationally.

Therapidpaceoftechnologicalprogress,

lackofconsensusonhowtomeasureandcompareperformanceofAImodels,andthebroadcapabilitiesofthetechnologymeansthatthesafetyandsecurityimplicationsarechallengingtoassess.Wehavetherefore

limitedouranalysistothekeyrisksand

imposedalimitedtimehorizonto2025.WeexcludeconsiderationoftherisksresultingfrommilitaryapplicationsofgenerativeAI.

2.Theperceivedadvantagesfromfirst-moverstatusandwidespreadmediaattentionhaveacceleratedglobalinterestingenerative

AI.Since2020,progressingenerativeAI

hasgreatlyoutpacedexpertexpectations,

withmodelsoutperforminghumansinasmallnumberofspecifictasks.Progresscontinuestoberapidandto2025,itisunlikelythatthepaceoftechnologicaldevelopmentwillslow.

Higherperforming,largerLLMswillalmostcertainlybereleased,butitisunclearhowfarthiswilltranslateintosignificantlyimprovedpracticalapplicationsby2025.Globalregulationis

incomplete,fallingbehindcurrenttechnicaladvancesandhighlylikelyfailingtoanticipatefuturedevelopments.

TheGenerativeAIEcosystem

3.PrivatesectorAIfirmswillremainkeyactorsincutting-edgegenerativeAIresearchandfrontiermodelsto2025.Theresearchers,

funding,hardware,computeanddata

willcontinuetobeconcentratedinthese

commercialorganisations,enablingthemtoundertakethemostadvanceddevelopments.

4.Open-sourcegenerativeAIisfacilitatingrapidproliferationandincreasingdemocratisation

ofgenerativeAIbyreducingthebarriersto

entryfordevelopingmodels.Todatetheir

performancehasmostlylaggedbehindthatofthefrontiermodels;opensourcemodelswill

almostcertainlyimprove,buttheyarehighly

unlikelytobemorecapablethanleading

commercialfrontiermodelsby2025.The

proliferationofopen-sourcemodelsincreasesaccessibilityandthereforebringsglobalsafetyandsecurityimplications,especiallyformodelswhichhavethepotentialtoallowmalicioususethroughlackofefectivesafeguards.

4

Threatactors

5.Theincreasingperformance,availabilityandaccessibilityofgenerativeAItoolsallowspotentially

anyonetoposeathreatthroughmalicioususe,misuseormishap.GenerativeAIwillalmost

certainlycontinuetolowerthebarrierstoentryforlesssophisticatedthreatactorsseekingto

conductpreviouslyunattainableattacks.Aswellasorganisedgroups,politicalactivistsandloneactorswilllikelyusegenerativeAIforideological,politicalandpersonalpurposes.

6.CriminalsarehighlylikelytoadoptgenerativeAItechnologyatthesamerateandpaceasthe

generalpopulation,butsomeinnovativegroupsandindividualswillbeearlyadopters.Useofthetechnologybycriminalswillhighlylikelyacceleratethefrequencyandsophisticationofscams,fraud,impersonation,ransomware,currencytheft,dataharvesting,childsexualabuseimages

andvoicecloning.Butto2025,criminalswillbelesslikelytosuccessfullyexploitgenerativeAItocreatenovelmalware.

7.To2025,generativeAIhasthepotentialtoenhanceterroristcapabilitiesinpropaganda,

radicalisation,recruitment,fundingstreams,weaponsdevelopmentandattackplanning.But

dependenceonphysicalsupplychainswillalmostcertainlyremainanimpedimenttotheuseofgenerativeAIforsophisticatedphysicalattacks.

SafetyandSecurityRisksOverview

5

8.Overthenext18months,generativeAIismorelikelytoamplifyexistingrisksthancreatenew

ones.Butitwillincreasesharplythespeedandscaleofsomethreats,andintroducesome

vulnerabilities.Therisksfallintoatleastthreeoverlappingdomains:

•Digitalrisksareassessedtobethemostlikelyandhavethehighestimpactto2025.Threatsincludecybercrimeandhacking.GenerativeAIwillalsoimprovedigitaldefencestothese

threats.

•Riskstopoliticalsystemsandsocietieswillincreaseinlikelihoodto2025,becomingassignificantasdigitalrisksasgenerativeAIdevelopsandadoptionwidens.Threatsincludemanipulationanddeceptionofpopulations.

•PhysicalriskswilllikelyriseasgenerativeAIbecomesembeddedintomorephysicalsystems,includingcriticalinfrastructureandthebuiltenvironment.Ifimplementedwithoutadequatesafetyandsecuritycontrols,AImayintroducenewrisksoffailureandvulnerabilitiestoattack.

9.Theseriskswillnotoccurinisolation;theyarelikelytocompoundandinfluenceotherrisks.Therewillalsoalmostcertainlybeunanticipatedrisks,includingrisksthatresultfromlackofpredictabilityofAIsystems.

SafetyandSecurityRisks

Types

10.Themostsignificantrisksthatcouldmanifestby2025include:

Cyber-attacks:GenerativeAIcanbeusedtocreatefasterpaced,moreefectiveandlargerscalecyberintrusionviatailoredphishingmethodsorreplicatingmalware.Butexperimentsinvulnerabilitydiscoveryandevadingdetectionaresignificantlylessmatureatthisstage.WeassessthatgenerativeAIisunlikelytofullyautomatecomputerhackingby2025.

Increaseddigitalvulnerabilities:GenerativeAIintegrationintocriticalfunctionsand

infrastructurepresentsanewattacksurfacethroughcorruptingtrainingdata(‘data

poisoning’),hijackingmodeloutput(‘promptinjection’),extractingsensitivetrainingdata

(‘modelinversion’),misclassifyinginformation(‘perturbation’)andtargetingcomputingpower.

Erosionoftrustininformation:GenerativeAIcouldleadtoapollutionofthepublicinformationecosystemwithhyper-realisticbotsandsyntheticmedia(‘deepfakes’)

influencingsocietaldebateandreflectingpre-existingsocialbiases.Thisriskincludescreatingfakenews,personaliseddisinformation,manipulatingfinancialmarketsand

underminingthecriminaljusticesystem.By2026syntheticmediacouldcomprisealargeproportionofonlinecontent,andriskserodingpublictrustingovernment,whileincreasing

polarisationandextremism.Authenticationsolutions(e.g.‘watermarking’)areunderdevelopmentbutarecurrentlyunreliable,requiringupdatesasgenerativeAIevolves.

Politicalandsocietalinfluence:GenerativeAItoolshavealreadybeenshowncapable

ofpersuadinghumansonpoliticalissuesandcanbeusedtoincreasethescale,

persuasivenessandfrequencyofdisinformationandmisinformation.Moregenerally,generativeAIcangeneratehyper-targetedcontentwithunprecedentedscaleandsophistication.

6

Insecureuseandmisuse:GenerativeAIintegrationintocriticalsystemsandinfrastructure

risksdataleaks,biasedanddiscriminatorysystemsorcompromisedhumandecision-makingthroughpoorinformationsecurityandopaquealgorithmprocesses(e.g.

‘hallucinations’).Inappropriateusebyanylarge-scaleorganisationcouldhaveunintended

consequencesandresultincascadingfailures.GenerativeAIintegrationintocriticalfunctionsmayalsoresultinover-relianceonsupplychainsthatareopaque,potentiallyfragileandcontrolledbyasmallnumberoffirms.

Weaponinstruction:GenerativeAIcanbeusedtoassembleknowledgeonphysical

attacksbynon-stateviolentactors,includingforchemical,biologicalandradiological

weapons.LeadinggenerativeAIfirmsarebuildingsafeguardsagainstdangerousoutputs,buttheefectivenessofthesesafeguardsvary.Otherbarrierstoentrywillpersist(e.g.

acquiringcomponents,manufacturingequipment,tacitknowledge),butthesebarriershavebeenfallingandgenerativeAIcouldacceleratethistrend.

Conclusions

GenerativeAIhasthepotentialtobringsubstantialbenefitsifmanagedappropriately,acceleratingproductivityandinnovationacrossmanysectorsincludinghealthcare,financeandinformation

technology.Butthereisariskthatinadequateunderstandingofthetechnologyresultingin

disproportionatepublicanxietycouldresultinfailuretoadoptgenerativeAIandputsomeb