2025放眼算法之外：AI 真实风险报告

Beyondalgorithms:therealrisksofAI

S2GRUPO

S2GRUPO

Index

1Introduction4

2AttacksanddangersagainstAImodels8

2.1Promptinjection12

2.2Insecureoutputmanagement14

2.3Datapoisoning16

2.4Modelinformationleakage(dataleakage)18

2.5Denialofservice20

2.6Supplychainvulnerabilities22

2.7Disclosureofconfidentialinformation24

2.8Manipulationofinsecureplugins26

2.9Excessiveagency28

2.10Over-reliance30

2.11Modeltheft32

3Challenges34

3.1AIcodemaintainability36

3.2ComplexityoftheAIsupplychain39

3.3ReuseofAIcode42

3.4Optimisationofcybercriminals’capabilities44

3.4.1MalwareasaService49

3.4.2RansomwareasaService53

3.4.3AIattacks55

Beyondalgorithms:therealrisksofAI

S2GRUPO

4TypesofactorsseekingtoexploitAIsecurityrisks60

4.1Cybercrime62

4.1.1FunkSec64

4.1.2.GXCTeam65

4.1.3IndrikSpider66

4.1.4RenaissanceSpider67

4.2APTgroups68

4.2.1APT2873

4.2.2EmberBear74

4.2.3APT4175

4.2.4RedHotel76

4.2.5Sodium77

4.2.6Ta49978

4.2.7ImperialKitten79

4.2.8CharmingKitten80

4.2.9APT4281

4.2.10LazarusGroup82

4.2.11VoidArachne83

5Recommendations84

5.1AIsecurityAudit86

5.2Awarenessandcybersecuritytraining88

6AboutS2GRUPO90

1Introduction

Beyondalgorithms:therealrisksofAI

4

Artificialintelligence(AI)hasestablisheditselfasa

transformativeelementinorganisations,acceleratingprocesses,optimisingdecision-making,enablingnewdata-drivenbusinessmodelsandopeningupendlessopportunitiesforinnovation.Its

adoptionrangesfromdataanalysisandtheautomationofcriticalprocessestoearlythreatdetectionandincidentresponse.

However,thissameexpansionhassignificantlyincreasedthe

surfaceareaofexposureandintroducednewriskvectorsthataffecttechnology,processesandpeopleacrosstheboard.AIisadual-usetechnology:whileitenhancesdefenceandefficiency,italsolowersthecostandscalestheoffensivecapabilitiesof

cybercriminalsandstateactors.

Thepowerofadvancedalgorithmsandmachinelearning

hasstrengthenedcybersecurityprevention,detectionand

responsecapabilities.Butatthesametime,theseadvances

havegivenrisetonewthreatsthataremoresophisticatedanddifficulttocontrol.FromattacksthatmanipulatetrainingdatatoscenariosofexcessivedependenceonexternalprovidersorhostileuseofAI-poweredtools,thecurrentlandscaperequiresacomprehensivevisionthatcombinestechnologicalinnovationwithrobustriskmanagementandalgorithmicaccountability.

Inthiscontext,thisreportstructuresitsanalysisintothreecomplementarylevels:

First,itdescribesthemostrelevantattacksandvulnerabilities

affectingAIsystems,i.e.,thewaysinwhichmodelscanbe

manipulated,compromised,orexploited.Next,itaddressesthe

structuralchallenges(suchascodemaintainability,supplychaincomplexity,andreuseofexternalcomponents)thatactasfertilegroundfortheseattackstothriveorpersistovertime.Finally,

itidentifiestheactorswhoexploittheserisksandproposes

practicalrecommendationstostrengthendefenceandregulatorycompliance.

S2GRUPO

5

2

Attacksanddangers

againstAImodels

Beyondalgorithms:therealrisksofAI8

2.1Promptinjection

Theattackermanipulatesthemodel’sinputs(prompts,

documents,orexternaldata)toalteritsbehaviour,execute

unintendedactions,orextractconfidentialinformation.Althoughitmayinvolvesocialengineeringtechniques,itsprimarynature

isthatofaninjectionvulnerability,whereuntrusteddatais

interpretedasinstructions.Asitisanattackagainstthemodel’slogic,thistypeofactioncaneasilygounnoticedbytraditional

defences.

Itcanoccurdirectly(theattackermodifiestheuserprompt)

orindirectly,whenharmfulinstructionsarehiddenincontent

processedbythemodel(e.g.,documents,webpages,orexternaldatasources).Inbothcases,thesystemlosestheseparation

betweendataandinstructions,allowingtheattackertobypasscontrolsoraccesssensitiveinformation.

Examples

InanorganisationthatusesanAIassistanttoprocess

internaldocumentation,anattackerinsertshidden

instructionsintoanapparentlylegitimatefile.Themodelinterpretstheseinstructionsandreturnssensitive

information,suchaspersonalorfinancialdata,ignoringestablishedsecuritypolicies.

Ifthissamescenariooccursinapublicadministrationor

regulatedsector,theconsequencesareamplified:thebreachmayinvolveviolationsoftheGDPR,non-compliancewiththeENSandlossofinstitutionaltrust.

Beyondalgorithms:therealrisksofAI12

Beyondalgorithms:therealrisksofAI14

2.2

Insecureoutputmanagement

WhenAI-generatedresultsareconsumeddirectlyinbusiness

processes(transactionautomation,diagnosticsupport,devicemanagement,orcodegeneration),incorrectormanipulated

outputcanhavecriticaleffects.Thisriskisoftenthefinal

manifestationofothervulnerabilities,suchaspromptinjections,datapoisoning,orundetectedbiasesinmodeltraining.

Theriskisexacerbatedwhenmodeloutputsareexecutedwithouthumanvalidation,securitycontrols,orintermediateverificationenvironments.

Examples

Atabank,amodelgeneratesaninsecureSQLscriptthat

isautomaticallyexecutedinthebankingcore,openinga

breachinthesystem.Atahospital,adiagnosticsupport

modelissuesanerroneousrecommendationduetobiasesorcontaminateddata,triggeringanincorrectclinicaldecision.

Beyondalgorithms:therealrisksofAI16

2.3

Data

poisoning

Themodelcanbemanipulatedbyintroducingcontaminated

orharmfuldataduringitstrainingorupdating.Thisaltersits

behaviour,degradesitsperformanceorintroducesharmfulbiasesthatcompromiseitsintegrity,potentiallyturningitintoatool

forattack.Thistypeofsabotagecancomefromcompromised

externalsources,datasetprovidersorautomaticintegrationswithunverifiedAPIs.

Apoisonedmodelnotonlygenerateserroneousoutputs,butcanalsobecomeapointofriskpropagationtolaterstages,closely

linkingthisrisktovulnerabilitiesintheAIsupplychain(Section2.6),asmanymodelsaretrainedwithdatafromthirdpartiesoropen

repositories,andtoinsecureoutputmanagement(Section2.2),wherethesecontaminatedoutputstranslateintoautomatedactionsoroperationaldecisions.

Examples

Atanenergyutility,acontaminateddatasetcausesthe

demandpredictionmodeltoprioritiseerroneousdecisions,generatingoperationalcostsandexposuretofraudintheelectricitymarket.

Beyondalgorithms:therealrisksofAI18

2.4Modelinformationleakage(data

leakage)

Themodelmayexposesensitiveinformationlearnedduring

trainingorstoredininternalmemories(e.g.,embeddingsorvectorbases).Thisexposurecanoccurthroughmaliciousqueries,

modelinversionattacks(membershipinference)orconfigurationerrorsthatallowpersonal,confidentialorstrategicdatatobe

retrieved.Insensitiveenvironments,suchashealthcare,financeordefence,suchaleakcanhaveseverelegal,reputationaland

operationalimpacts.

Thisriskisdirectlyrelatedtothedisclosureofconfidential

information(Section2.7),asbothsharetheobjectiveofprotectingdataconfidentialityandcontrol,andtoover-relianceonthemodel(Section2.10),whichcanleadtoinadvertentexposureofsensitiveinformationwhensystemsblindlytrustthemodel’sresultsor

integrateitwithoutaccesslimitsorfiltering.

Examples

Alanguagemodeltrainedonmedicalrecordsrespondstoexternalqueriesbyrevealingactualfragmentsofclinical

information.Theincidentexposespatients’personaldata,violatingtheGDPRandcompromisinginstitutionaltrust.

Beyondalgorithms:therealrisksofAI20

2.5

Denial

ofservice

TheattackercansaturateanAImodelwithmassiverequests,

complexqueriesormanipulateddata,degradingitsperformance,blockingresourcesorartificiallyincreasingexecutioncosts

incloudenvironments.Althoughthisisaclassicattack,inthe

contextofAIittakesonanewdimension:modelsareexpensivetorun,dependonlargeamountsofmemoryandGPUs,andareoftenintegratedwithexternalservicesorpublicAPIsthatexpandtheirexposuresurface.

ThisriskiscloselyrelatedtotheAIsupplychain(Section2.6),asafailureorabuseinthird-partyservices(e.g.,endpoints,APIs,orinferenceintegrations)canaffecttheavailabilityoftheentire

system,andtoexcessiveagency(Section2.9),whenautonomousmodelsexecuterepetitiveactionswithouthumanvalidation,

unintentionallyamplifyingtheimpactoftheattack.

Examples

InacorporateSOC,anadversarylaunchesbruteforce

attacksagainsttheAIAPIthatassistsinalertclassification,causingcriticaldelaysinincidentdetectionandanincreaseincloudserviceoperatingcosts.Inanotherscenario,a

clinicalpredictionmodelissubjectedtoamassiveflowofautomatedrequests,saturatingthehospitalinfrastructureandtemporarilyblockingdiagnosticsupportsystems.

Beyondalgorithms:therealrisksofAI22

2.6Supplychainvulnerabilities

Relianceonlibraries,frameworks,pre-trainedmodels,and

externalservicescanintroduceinheritedvulnerabilitiesthat

cancompromisetheentireAIpipeline,evenifthemodelitselfisnotaffected.Afailureinacommondependency(asoccurredin2021withLog4j5)canenableremotecodeexecutionordatamanipulation,affectingthesystemswhereAIisintegrated.

Thisriskcombinesbothtechnicalexposure(vulnerableor

unmaintainedcomponents)andoperationaldependenceonexternalvendorsandrepositories.

Examples

Inthehealthcaresector,theintegrationofacontaminated

orpoorlypatchedthird-partymodelintoadiagnosticsystemcausesbiasesinresultsandsecurityfailuresthatallow

unauthorisedaccesstoclinicaldata.Similarly,acritical

vulnerabilityinanauxiliarylibraryintheAIenvironment(e.g.,Scikit-learnorTensorFlow)couldcompromisetheentire

infrastructure,evenwithoutdirectlyaffectingthemodel.

5WorstApacheLog4jRCEZerodayDroppedonInternet

/2021/12/worst-log4j-rce-zeroday-dropped-on.html

Beyondalgorithms:therealrisksofAI24

2.7Disclosureofconfidentialinformation

AImodelsmayrevealsensitiveorprivateinformationasaresult

ofmaliciousqueries,incorrectaccessconfigurations,orlackofoutputvalidations.Thismayincludepersonaldata,corporate

information,internalparameters,orfragmentsofthetraining

set.TheriskisamplifiedinenvironmentswhereAIinteractswithexternalusersorotherinformationsystemswithoutsupervision.

Thisattackiscloselyrelatedtoinformationleakageordata

leakage(Section2.4),asbothinvolveuncontrolledexposure

ofinformationlearnedbythemodel,andtoover-reliance

(Section2.10),whenmodeloutputsareconsumedwithouthumanrevieworaccesslimits.Itcanalsobeexacerbatedbyinsecure

outputmanagementifmodelresponsesareautomaticallyintegratedintoproductionprocessesorapplications.

Examples

Ataninsurancecompany,alanguagemodeltrainedon

medicalrecordsexposesrealpatientdatafragmentswhen

respondingtoqueriesdesignedbyanattacker.Inanother

case,acorporateAIassistantrevealsinternalcredentialsandcustomerrecordsduetoimproperconfigurationofrolesandpermissionsinitsdeploymentenvironment.

Beyondalgorithms:therealrisksofAI26

2.8

Manipulationofinsecureplugins

Pluginsandexternalintegrations(aswithanyothertraditional

software)extendthecapabilitiesofthesystem,butalso

introducenewattackvectorsiftheyarenotvalidatedorare

installedwithoutcontrol.InthecaseofAI,theriskisexacerbated:

manyadd-onsallowthemodeltodirectlyaccessdatabases,

executecommandsorcommunicatewithexternalservices,

multiplyingthepotentialimpactofmanipulation.Anattacker

canexploitavulnerableorpoorlydesignedplugintoinject

instructions,manipulatedataorredirectoutputs,compromisingtheintegrityofthesystemortheinformationprocessed.

Thisriskiscloselyrelatedtotheriskofinsecureoutput

(Section2.2),whenthepluginexecuteswithoutcontrol,tothe

supplychain(Section2.6),whenthereareunauditedexternal

components,andtoexcessiveagency/dependency(Sections2.9and2.10),incaseswheredelegationiscarriedoutwithout

verification.

Examples

Inafinancialenvironment,anERPpluginmanipulates

accountingqueries,alteringfinancialrecordsandgeneratinginternalfraud.Inanothercase,anAIassistantwith

accesstoemailandcloudstoragepluginsismanipulatedtoautomaticallysendsensitiveinformationtoexternal

addresses,followingapromptinjectioncamouflagedinaprocesseddocument.

Beyondalgorithms:therealrisksofAI28

2.9Excessiveagency

TheriskofexcessiveagencyariseswhenanAIagentisgiven

operationalautonomytoperformcriticalactions(suchas

modifyingsystems,authorisingtransactionsorinteractingwithproductionenvironments)withoutadequatehumansupervision.Thisuncontrolledautonomyturnserrorsormanipulationsinto

realandimmediateimpactsonbusinessprocesses.

Unlikeover-reliance(Section2.10),wheretheproblemliesin

excessivetrustinAIonthepartofindividualsororganisations,over-agencyoriginatesinthesystemitself,whenitactsonits

owninitiativeorwithtoomuchleeway.Inotherwords,inagency,theriskarises‘fromwithin’themodel;inover-reliance,itarises‘fromoutside’,duetoalackofhumancontrolorunderstanding.

Unlikeotherrisks,excessiveagencydoesnotcreatenew

vulnerabilities,butratherenhancesandchainsexistingones:

itamplifiestheeffectsofpromptinjection(Section2.1),turns

insecureoutputmanagement(Section2.2)intoanoperational

failure,andmultipliestheimpactofmanipulatinginsecureplugins(Section2.8).Itisalsolinkedtoover-reliance(Section2.10),whenorganisationsblindlydelegatedecisionstoAIthatshouldrequirehumanorcontextualvalidation.

Examples

Inatradingenvironment,anAIagentexecutesmanipulatedstockordersthroughacompromisedplugin,generating

millionsinlossesbeforehumanoperatorscanintervene.Inahealthcareenvironment,anAIsystemautonomouslyadjustsdosesortreatmentsbasedoncontaminateddata,causing

seriousclinicalerrors.

Beyondalgorithms:therealrisksofAI30

2.10Over-reliance

Theriskofexcessivedependencyariseswhenanorganisationover-delegatescriticaldecisionsorprocessestoAImodelsortheecosystemofservicesthatsupportthem.Itisnotlimitedtorelyingonaspecificproviderormodel,butrathertobuildingtheentireoperationonaninterdependentnetworkofexternalmodels,APIsandframeworks,whichincreasesthevulnerablesurfaceareaandhindersgovernance.

Whileexcessiveagency(Section2.9)focusesonAIthatactswithtoomuchautonomy,excessivedependencereflectstheoppositeeffect:humanteamsandorganisationsthatloseautonomyby

blindlyrelyingonautomatedsystems.Botharesidesofthesamecoin,andtheytendtofeedintoeachother:themoreautonomousAIis,themoredependentusersbecome;themoredependent

usersare,themoreautonomyisgrantedtoAI.

Thisriskisnotonlytechnological:withthemassadoptionofAI,

humanteamsprogressivelyloseunderstandingofhowandwhycertaindecisionsaremade.Whenmodelsbecome“operational

blackboxes,”theabilitytoreacttotheunexpected(preciselytheweakpointofmachinelearning)iseroded.Inthisway,excessivedependencereinforcesotherrisks:itamplifiestheeffectsof

excessiveagency(Section2.9),multipliestheconsequencesofunsafeexitmanagement(Section2.2)andcanexacerbatethe

disclosureofconfidentialinformation(Section2.7)ifblindtrustisplacedinthemodel’sresults.

Examples

Acriticalinfrastructureoperatordelegatescontrolofits

energysystemtoasetofexternalmodelshostedinthe

cloud.Achangeinusagepoliciesorafailureinoneoftheseservicesleavestheorganisationwithoutoversightor

decision-makingcapacity.Atthesametime,internalteamsnolongerhavesufficienttechnicalknowledgetomanuallyoperatethesystemorinterpretdetectedanomalies.

Beyondalgorithms:therealrisksofAI32

2.11Modeltheft

Modeltheftorextractioninvolvesanattackerreplicatingthe

behaviourofanAImodelthroughsystematicqueries(model

extraction),oraccessingitsparametersandweightsthrough

reverseengineeringtechniques.Thistypeofattackviolates

intellectualproperty,butalsoopensthedoortoevasion,

poisoningorindustrialespionageattacks.Theriskincreases

whenmodelsarepubliclyexposedthroughAPIs,unauthenticatedintegrations,oropencoderepositories.

Modeltheftisrelatedtodataleakage(Section2.4),asitinvolvesthelossofknowledgeincorporatedduringtraining,andto

datapoisoning(Section2.3),whenanadversaryredistributes

amanipulatedmodeltocompromisethirdparties.Itisalso

associatedwithover-reliance(Section2.10),asorganisationsthatdonotprotecttheirmodelsordiversifytheirinfrastructureare

moreexposedtolossorillicitcopying.

Examples

AstateactorclonesaEuropeanpharmaceuticalcompany,spredictivemodelbymakingautomatedqueriestoitspublicAPI,replicatingitsresultsandacceleratingitsownclinicaltrialswithoutincurringtheoriginalR&Dcosts.Inanother

scenario,acompetitorobtainsapartialcopyofafinancialfrauddetectionmodelandredistributesitafterintroducingdeliberatebiases,withtheaimofdiscreditingtheowner

organisation.

Beyondalgorithms:therealrisksofAI34

3Challenges

Theproblemisexacerbatedinenvironmentswherelegacy

modelscoexistwithexperimentaldevelopmentsintegrated

intocriticalprocesses.Lackofdocumentation,reliance

onunmaintainedcode,ortheabsenceofclearsupport

responsibilitiesincreasetheriskofsecuritybreaches,operationaldisruptions,andultimatelyregulatorynon-compliance.

Maintainability,therefore,transcendsthetechnical.The

EuropeanAIActrequires“high-risk”systemstohavetraceability,changelogs,andsufficientdocumentationtoensuretheir

oversight.Complementarily,theNIS2directiveandtheDORAregulationreinforcetheseobligationsintermsofsecurity,

resilience,andbusinesscontinuity.AnorganisationunabletokeepitsAIcodeuptodateandauditedisnotonlyexposedtotechnicalvulnerabilities,butalsotoregulatorysanctionsandlossoftrustfromcustomersandpartners.

Ultimately,alackofmaintainabilitylimitstheabilitytoinnovatesafely,underminesbusinesscontinuityandincreases

dependenceonthirdparties,evencompromisingthe

technologicalsovereigntyofstrategicsectors.Ensuring

maintainabilityinvolvesintegratingcontinuousupdate

processes,technicalanddataaudits,andacultureofrigorousdocumentationthatreinforcesbothoperationalsecurityandlegalcompliance.

Beyondalgorithms:therealrisksofAI38

3.2ComplexityoftheAIsupplychain

Theartificialintelligencesupplychainismuchmoreextensiveandcomplexthanthatoftraditionalsoftware.Itisnotlimitedtoframeworksandlibraries,butencompassestrainingdatasets,pre-trainedmodels,third-partyAPIs,cloudinfrastructures,externalrepositories,anddistributionservices.Eachof

theselinksaddspointsofvulnerabilitythatcancompromisethesecurityandreliabilityofsystems.

AccordingtoENISA’sThreatLandscape20249,attacksonthe

digitalsupplychainarealreadyamongthemainthreatsin

Europe,andAIisparticularlyexposedduetoitsheavyreliance

onexternalcomponentsthatrarelyundergothoroughauditing

processes.Similarly,IBM10hasdocumentedhowcybercriminals

areexploitingpoorlysecuredmodelrepositoriesandAPIsasentryvectorstocompromisecriticalinfrastructure.

Oneofthemostsignificantrisksistheincorporationofbiasesthroughexternaldatasetsandpre-trainedmodels.Many

organisationsintegratepublicdatasetsorthird-partymodelswithoutvalidatingtheirorigin,coverageorrepresentativeness.

Thesebiasescanbeinadvertentlyinheritedorintentionally

introducedthroughmanipulateddatainopenrepositories.AsENISApointedoutinits2020report11,biasesintrainingshouldbeconsideredasecurityrisk,aswellasanethicalproblem,astheycanleadtoinconsistent,discriminatoryordirectlyunsafedecisionsincriticalenvironments.

9ENISA.(2024).ENISAThreatLandscape2024.EuropeanUnionAgencyforCybersecurity.

https://www.enisa.europa.eu/publications/enisa-threat-landscape-2024

10IBM.HowcybercriminalsarecompromisingAIsoftwaresupplychains.6septiembre2024.

/think/insights/cyber-criminals-compromising-ai-software-supply-chains

11ENISA.

(2020).ArtificialIntelligence–ThreatLandscape.EuropeanUnionAgencyforCybersecurity.

https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges

39

S2GRUPO

Thelackoftransparencyinexternalcomponentsexacerbatesthechallenge.Opaquefoundationalmodels,poorlydocumentedlibrariesorcloudserviceswithoutcleartraceabilitylimitthe

abilityoforganisationstodetectvulnerabilitiesorassesstherobustnessofthesystemstheyintegrate.Thisopacityisin

directconflictwiththetraceability,documentationandriskmanagementrequirementssetoutintheAIAct.

Finally,theAIsupplychainposesanadditionalproblem:update

andcontinuitymanagement.Achangeinalow-levellibraryor

APIcanquicklypropagatethroughouttheentirepipeline,alteringperformanceorintroducingvulnerabilities.Manyorganisations

choosetodelaytheseupdatesforfearofbreakingcompatibility,whichincreasestheirexposuretocyberattacks.

Thecomplexityofthissupplychainisnotonlyatechnical

challenge,butalsoaregulatoryimperative.TheAIAct,