Abnormal+生成式 AI 攻击的 CISO 指南

CISOGuideto

GenerativeAIAttacks

Understandingand

DefendingAgainst

AI-Generated

EmailAttacks

10%

ofalldataproducedwillbegeneratedbyAIby2025.

Gartner

65%

ofexecutivesbelievegenerativeAIwillhaveahighimpactonbusiness.

53%

ofITprofessionalsbelievethatChatGPTwillbeusedthisyeartohelphackerscraftmorebelievableandlegitimate-soundingphishingemails.

CISOGuidetoGenerativeAIAttacks

2

TheRisingThreatof

GenerativeAI

Anyonewhohasspenttimeonlinein2023haslikelyheardaboutChatGPTandGoogleBard,twoofthemorepopularplatformsthatharnessgenerativeartificialintelligence(AI)tocompletewrittencommands.Andinthefewmonthssincetheirrelease,they’vehadaprofoundimpactonvariousaspectsofourdigitalworld.

Byleveragingadvancedmachinelearningtechniques,generativeAIenablescomputerstogenerateoriginalcontentincludingtext,images,music,andcodethatcloselyresembleswhatahumancouldcreate.Thetechnologyitselfhasfar-reachingimplications,manyofwhichcanbeusedforbothpersonalandprofessionalgood.Artistsandauthorscanuseittoexplorenewcreativedirections,pilotsanddoctorscanuseitfortrainingandreal-worldsimulation,andtravelagentscanhaveitcreatetripitineraries—amongthousandsofotherapplications.

Butlikeanythingelse,cybercriminalscantakeadvantageofthistechnologyaswell.Andunfortunately,theyalreadyhave.PlatformsincludingChatGPTcanbeusedtogeneraterealisticandconvincingphishingemailsandmoredangerousmalware,whiletoolslikeDeepFaceLabcan

createsophisticateddeepfakecontentincludingmanipulatedvideoandaudiorecordings.Andthisislikelyonlythebeginning.

TocombatthemaliciousapplicationsofgenerativeAI,it’scrucialfororganizationstocontinuallydevelopandimplementrobustdefenses,enhancedetectioncapabilities,andstayvigilanttoemergingthreats—

beforetheybecomethenextvictimofan

AI-generatedattack.

KPMG

BlackBerry

©2023AbnormalSecurityCorporation.Allrightsreserved.

CISOGuidetoGenerativeAIAttacks

HowGenerativeAICan

BeUsedinCyberAttacks

CredentialPhishing

CybercriminalsmayemploygenerativeAItechniquestoenhancethesophisticationandrealismofphishingemailsandtheircorrespondinglandingpages,increasingthechancesoftrickingusersintorevealingsensitiveinformationorinputtingcredentials.

EndpointExploitation

Ifanattackeridentifiesvulnerabilitiesinsoftwarerunningonendpoints,generativeAIcouldbeusedtocreateautomatedattackpayloads.ByleveragingAItocreatespecificcodeorcommands,attackerscouldautomatethedeliveryofmaliciouspayloadstovulnerablesystemsandendpoints.

BEC&SocialEngineering

Byinputtingspecificinformationaboutatargetand/orpreviousconversationhistory,generativeAIcanbeusedtoengageinconversationswithusers,attemptingtobuildtrustandmanipulatethemintotakingspecificactions.Themodelscangeneratepersuasivemessagesandbeusedthroughoutanentireconversationtoconvincethetargettopayafakeinvoice,changebankingdetails,orprovideaccesstosensitiveinformation.

MalwareCreation

GenerativeAIcanautomatetheprocessofgeneratingnewvariantsofmalware,makingitmorechallengingfortraditionalsignature-basedendpointprotectionsystemstodetectandblockthemeffectively.Byleveraginggenerative

AItechniques,attackerscancreatepolymorphicorself-mutatingmalwarethatchangesitscodeorbehavior,allowingittoevadedetectionandpersistoncompromisedendpoints.

It’sworthnotingthatcyberattackscreatedbygenerativeAItoolscantakemanyformsandoftendo.Theseattackscanbepartoflargerphishingoraccounttakeoverschemesandcanhavedireconsequencesforbothemployeesandtheirorganizations.

3

©2023AbnormalSecurityCorporation.Allrightsreserved.

CISOGuidetoGenerativeAIAttacks

ImpactofGenerativeAI

onCybercrime

WhilepopularusageofgenerativeAIisstillrelativelynewandtheextentoftheimpactcannotyetbedetermined,thereareearlysignalstoindicatethatthisisathreatwecannotsimplyignore.ChatGPTaloneexceeded1millionuserswithin5daysoflaunch,withindicationsthatatleastsomeofthosepeoplewerelookingtouseitfornefariouspurposes.

WeekstoReach1MillionUsers

ChatGPT

DALL-E

CoPilot

1,000,000

750,000

500,000

250,000

0

248101214161820222426

Weeks

Gartnerpredictsthatby2025,generativeAIwillaccountfor10%ofalldataproduced,upfromlessthan1%in2021andthegenerativeAImarketisprojectedtoreachover$14.7trillionby2030.AndwhilegenerativeAIwilllikelybemostlyusedforlegitimatepurposes,wewouldbenaivetobelievethatbadactorswillnotuseitfortheirownmaliciouspurposes.

InaJanuary2023studybyBlackBerry,78%ofITprofessionalspredictedthatasevereattackcreditedtoChatGPTwilloccurwithintwoyears,and71%believethatnation-statesarealreadyleveragingitformaliciouspurposes.OfthewaysthatthreatactorsmayharnessChatGPTspecifically,respondentsaremostconcernedabouttheabilitytocraftmorebelievableandlegitimate-soundingphishingemails,tocreatenewmalware,andtohelplessexperiencedhackersimprovetheirtechnicalknowledgeanddeveloptheirskills.

4

©2023AbnormalSecurityCorporation.Allrightsreserved.

CISOGuidetoGenerativeAIAttacks

WhyGenerativeAIAttacks

areanIncreasingIssue

GenerativeAItoolshaveenabledcybercriminalstoquicklyandeasilycreatevarioustypesofattacks.WithplatformslikeChatGPT,attackerscanautomateandscaletheirattackplaybooks.Businessemailcompromiseisalreadythemostfinancially-devastatingcybercrimeforbusinessesworldwide,resultinginmorethan$43billioninexposedlossessince2016andthisevolutioningenerativeAIisonlygoingtomaketheproblemworse.Here’swhy:

IncreasedEaseofAccess

EverypersonintheworldwhohasaccesstotheInternetcanaccessChatGPTandsimilartools,makingitpossiblefornewcybercriminalstostartsendingattacks,evenwithoutpreviousknowledge.TheproliferationofgenerativeAIenablesnearlyanyonetobecomeasophisticatedcybercriminalinamatterofseconds,providingnotonlytipsonhowtogetstarted,butalsotheexactelementsneededtoexecuteasuccessfulattack.

IncreasedVolume

WithanincreaseinthenumberofpeopleusinggenerativeAItocreateattacks,it’snaturalthatthevolumeofattackswillincreaseaswell.Butthisisnottheonlythingatplay.GenerativeAIenablescriminalstocreateemailsmuchfasterthaneverbefore—compilinginsecondswhatusedtotakehours,whichcreatesasuperweaponattheirdisposal.

IncreasedSophistication

Usershavelongbeentaughttolookfortyposandgrammaticalerrorsinemailstounderstandwhetheritisanattack,butgenerativeAIcancreateperfectly-craftedemailsthatlookcompletelylegitimate—makingitimpossibleforemployeestodecipheranattackfromarealemail.Andit’snotonlyEnglishanymoreeither.ChatGPTalonecangeneratetextinmultiplelanguages,includingSpanish,Russian,Arabic,German,andJapanese.

Furthermore,theabilityforattackerstousepreviousconversationhistory,oftenaccessedthroughacompromisedemailaccount,canenablegenerativeAItocontinueconversationsthatappeartocomefromarealuser.Havingtheabilitytoreferencepreviousinformationin-threadorsoundexactlyliketheimpersonateduseronlyincreasesthelikelihoodthatthetargetwillfallvictimtotheattack.

5

©2023AbnormalSecurityCorporation.Allrightsreserved.

CISOGuidetoGenerativeAIAttacks

AReal-WorldAttackExample

DespitethefactthatChatGPThasonlybeenavailableforafewmonths,cybercriminalsarealreadystartingtouseitforattacks.AbnormalSecurityrecentlydetectedanumberofattackscreatedbygenerativeAItools—mostlyusedincredentialphishingcampaigns.

Unfortunately,theseattacksarenearlyimpossibletodetectbytheaverageenduser.Asyoucansee,thereareanumberofthingsthatmakethisemaillookextremelylegitimate.

Subject:Importantnotice:YourFacebookPageisatriskofbeingdisabledSender:MetaforBusiness<facebookpagereport@>

To:MichaelBamen<mbamen@>RECIPIENTDate:May3,2023,1:10pmET

Hello,

PerfectGrammar

Unlikethephishingemails

ofthepast,thereisnota

singlemisspelledwordor

grammaticalerrorinthis

lengthyemail.

WeregrettoinformyouthatyourFacebookPagehasbeenfoundinviolationofourcommu-

nitystandardsandpolicies.OursystemhasdetectedcontentonyourPagethatviolatesour

guidelinesregardinghatespeech,harassment,orotherprohibitedcontent.

Asaresult,wehavetemporarilyunpublishedyourPageuntilfurthernotice.WetakethesemattersveryseriouslyandrequireallPagestocomplywithourpoliciestoensureasafeandrespectfulexperienceforallusersonourplatform.

"Confirm"buttonbelow.REQUEST/ENGAGE

Ifyoubelievethatthisactionwastakeninerror,youmaysubmitanappealbyclickingonthe

Oursupportteamwillreviewyourappealand

providefurtherguidanceonhowtoresolvethismatter.

RelevantTopic

Theemailhasbeensentto

theadminofthecompany’s

FacebookPage,stating

ConfirmSUSPICIOUSLINK

thatthePagehasbeentemporarilyunpublished.

Ourteamwillreviewyourappealassoonaspossibleandprovideyouwithanupdateonthestatusofyourpage.Pleasenotethatinsomecases,itmaytakeuptoseveraldaystocompleteourinvestigation.

Thankyouforyourunderstandingandcooperationinthismatter.Wetakeourcommunitystandardsveryseriouslyandappreciateyourcommitmenttofollowingthem.

Bestregards,

TheFacebookTeam

Thismessageisanautomatedemail.Pleasedonotreply.

MetaPlatforms.229ParkAveS,NewYork,NY10007

UrgentInstructions

Usingatoneexpectedofa

business,theemailstates

thattherecipientshould

clickontheincludedlinkto

fileanappeal.

Iftherecipientweretoreceivethisemail,theywouldbemuchmorelikelytoclickthelinkthanifthisemailhadnotbeengeneratedbyAI.Thefactthatthisemailissowell-craftedmakesitmoredifficulttodetectbyhumans,underscoringtheincreasedneedforemailsecuritythatcanuseothersignalstodetectandblockthesewell-writtenandconvincingattacks.

6

©2023AbnormalSecurityCorporation.Allrightsreserved.

Instructions

Attack-Specific

Information

EmailThreadHistory

CISOGuidetoGenerativeAIAttacks

ThePotentialforMore

ComplexAttacks

Advancedphishingattacksaren’tanythingnew,butwithgenerativeAI,thereisanevengreaterthreatpotential.Imaginehoweasyitcouldbetocreateanattack(atmassivescale)usingsimpleinstructions.Forexample,let’sassumethatanattackerhasgainedaccesstoanemailaccountatavendororganizationandhasaccesstoprioremailthreadhistory.Byknowingwhothecompromisedusertypicallycommunicateswith,andknowingwhatthatcommunicationtypicallylookslike,theattackercancreateaverysophisticatedemailinamatterofseconds.

ThecybercriminalonlyhastoprovideatoollikeChatGPTwiththreeelementstocreateapersonalizedemailattack:instructions,attack-specificinformation,andtheemailthreadhistory.

Belowthereareinstructionsforyouandanemailconversation.

-Pleasewriteanemailresposeforperson1toaskperson2toupdatetheiraccountspayablesystemwiththenewbankaccountandroutingnumber.

-Theemailshouldbebrief,shouldsoundcasual,sincere,andpersonal(refer-encingthelastcorrespondence)

-Makesuretotellperson2thatiftheyhaveanyquestionstotextyou

Informationforyou:

Person1:LuciaForeman,AccountsReceivableManageratProliaSystemsPerson2:ReneeWest,VPofTreasuryatC

BankAccount:123

RoutingNumber:456

Person1cellphonenumber:123-456-7890

PreviousEmailConversation:

OnFriday,Dec3at8:58AMESTReneeWest<renee.west@>wrote:

HiLucia,thanksforconfirming.Iapplogizeforthedelayedresponse,inSanFranciscoithasbeenrainingalotsoI’vebeenabitsick.

Haveagreatweekend!

Cordially,

Renee

OnFriday,Dec3at8:33AMESTLuciaForeman<luciaforeman@proliasys->wrote:

HiRenee,

Justwantedtoconfirmwereceivedyourlatestpayment.Sorryaboutthehiccupandasalways,don’thesitatetoletusknowifthere’sanythingelseyouneed.

Best,

Lucia

7

©2023AbnormalSecurityCorporation.Allrightsreserved.

Subject:Requesttoupdateaccountspayablesystem

DearRenee,

Ihopeyou’refeelingbetterandtherainhasstoppedinSanFrancisco.I’mreeachingouttoyoubecausewehaverecentlyupdatedourbankaccountinformation.Ournewaccountnumberis123androutingnumberis456.

Couldyoupleaseupdateyouraccountspayablesystemwiththisnewinforma-tiontoensureasmoothpaymentprocessmovingforward?Ifyouhaveanyquestions,pleasedon’thesitatetotextmeat123-456-7890.

Thankyouforyourpromptattentiontothismatter.Weappreciateyourpartner-shipandhopeyouhaveagreatweek.

Bestregards,

LuciaForeman

AccountsReceivableManager

ProliaSystems

Usingthismethod,it’spossibleforevennon-technicalattackerstoautomaticallygeneratethousandsofpersonalizedattacks,indistinguishablefromnormalbusinesscommunications.Andunfortunately,ifthisemailweretolandinthetarget’sinbox,itislikelytoworkduetothetrustedrelationshipandpersonalizedcontent.

8

CISOGuidetoGenerativeAIAttacks

Withthistypeofinput,atoollikeChatGPTcanprovidean

extremelyconvincingemailthatwouldcontainabsolutely

zerotraditionalindicatorsofcompromiselikemalicious

attachmentsorasuspiciouslink.Itevenreferencesthe

previousconversationhistorytoappearlegitimate.

©2023AbnormalSecurityCorporation.Allrightsreserved.

CISOGuidetoGenerativeAIAttacks

HowtoStopEmailAttacks

GeneratedbyAI

Tocounterthesehigh-volumeandhighly-sophisticatedemailattacks,organizationsneedtherightemailsecurityplatform.Thenext-generationplatformincludestheuseofgoodAItocombatbadAI,aswellasthefollowingelements:

BehavioralDataScienceApproach

ThesolutionshoulduseafundamentallydifferentapproachthatleveragesbehavioraldatascienceandAItoprofileandbaselinegoodbehavioranddetectanomalies.Itshoulduseidentitymodeling,behavioralandrelationshipgraphs,anddeepcontentanalysistoidentifyandstopemailsthatappearsuspicious,andincludetheabilitytodetectwhethertheemailwascreatedusinggenerativeAImodels.

APIArchitectureandIntegrations

AsolutionthatconnectstoMicrosoft365andGoogleWorkspaceviaanAPIandindoingso,providesaccesstothesignalsanddataneededtodetectsuspiciousactivity.Thisincludesunusualgeolocations,dangerousIPaddresses,changesinmailfilterrules,unusualdevicelogins,andmore.Moreadvancedsolutionscanalsoconnecttootherapplications,includingSlack,Okta,Zoom,andCrowdStrike,tounderstandidentityanddetectmulti-channelattacks.

OrganizationalandSupplyChainInsights

Asolutionthatunderstandsbothformalandinformalorganizationalhierarchyandmapsinternalandcross-organizationalrelationshipstounderstandtypicalcommunicationpatternsandbehavior.Itshouldincludeafocusonvendorrelationshipstoprotectagainstbusinessemailcompromise,accounttakeovers,andothertypesoffraudthroughoutthesupplychain.

Withthesecapabilities,thesolutioncanusethousandsofsignalstodetectanomalousbehaviorsothatattackscreatedbygenerativeAIwillbestoppedbeforetheyreachtheinbox.

9

©2023AbnormalSecurityCorporation.Allrightsreserved.

TheproliferationoftoolslikeChatGPTandGoogleBardhasmadeitpossibleforbadactorstoincreasethevolumeandsophisticationoftheirattacksseeminglyovernight.Withtheabilitytocreatewell-writtenan