2026 AI 信任红利报告：建立自主式 AI 智能体信任的三大治理转型

TheAItrust

dividend

ThreeGovernanceShiftstoBuildTrustinAgenticAIAutonomy

September2026

TheAItrustdividend02

Tableofcontents

Introduction03

What'sdifferentaboutagenticAI?04

Governanceimplications05

Threenecessaryshifts07

Risksandcontrolsinpractice09

Buildingtrustedautonomy11

Actionsforleaders12

Authorsandcontributors13

TheAItrustdividend03

Introduction

AIagents1requiregovernancethatcombinesexplicitaccountability,operationaloversight,andcontrolsthatscaleasagentusemultiplies.

Theseevolvingtoolsintroduceagencyandautonomy2

butlacktrueintelligenceoraccountability,whichraisesquestionsabouttrustandreliabilityincriticalactivities—andtrustamongstakeholdersiswhatturnsAIintoa

strategicadvantage.Trustgivesstakeholders

confidenceinbusinessdecisions,products,ethics,

behaviours,andgovernance.Incontrast,afailureto

delivertrustcanresultinlossofcompanyvalueand

brandloyalty,aswellasrisklegalorregulatoryaction.

TrustinautonomousAIagentswillarisefrom

governanceandcontrolsthatevolvebeyondtraditionalapproachesandmethodsusedinearlyAIusecase

experimentation.Oncetrustisestablished,itproducesadividend.PwC’s

AIPerformancestudy

showsthat

organisationscandeployAIfasterandcapturedramaticallyhigherreturnsthantheirpeers.

Inthisthirdarticleinour

AItrustdividend

series,weidentifythegovernancefocus,operatingmodel,andaccountabilityshiftsleadersneedtomaketoscaleagenticAIsafely,compliantly,andwithconfidence.

1AIAgents:AIsystemsthatcaninteractwiththeirenvironment,

receiveinformation,andundertakeself-directedactionsinserviceofalarger,externallyspecifiedgoal.

2AutonomousAI:Thecapabilityofasystemtoselectbetween

multiplepossibleactionsequencestoachievethesystem’sgoals,

basedonthecurrentsituationandinternallydefinedcriteria.CurrentAIsystemsdonotyetexhibitautonomyinthefullestsenseofthisterm.

Whileemergingagenticsystemscanindependentlyplan,selecttoolsandexecutesequencesofactions,theirgoals,permissions,operatingboundariesandavailableactionsremainexternallydefined.Their

apparentautonomyisthereforebetterunderstoodas"boundedor

delegatedautonomy"withinahuman-designedoperatingenvelope,ratherthanindependentdeterminationofobjectivesorunconstrainedaction.

TheAItrustdividend04

What'sdifferentaboutagenticAI?

FivecharacteristicsmakeagenticAIdifferentthanitspredecessors.

•Dependingontheusecasedesign,agenticAIcanreasonovergoalsandcontext,ratherthansimplyfollowingpredefinedrules.

•Itcanmakedecisionsbetweenpossiblecoursesofactionbasedonobjectives,context,andconstraints.

•Itcantakeactionsautonomouslyacrosssystems,tools,andprocesses.

•Itsbehaviourisnon-deterministic,drawingoninsightsfrompatterns,context,andpriorinteractions,and

producingoutcomesthatcannotalwaysbepredictedwithcertainty.

•Whencombinedwithmemoryandfeedback,itcanlearnandadapt,causingbehaviourtoevolveovertime.

Thiscombinationoftraitscreatessignificant

opportunitiesforinnovationandtransformation.AI

leadersareleaningintotheseopportunities.Our

AI

Performancestudy

showsthatAIleaders,definedasthe20%oforganisationsthatarecapturing74%ofAI’s

economicvalue,aretwiceaslikelytouseAIthat

operatesautonomouslythantheirpeers.However,

opportunityandriskgohandinhand.AIagentsdemandnewapproachestogovernance,riskmanagement,

oversight,andscalablecontrolstoensuresafety,compliance,andreliability.

TheAItrustdividend05

Governanceimplications

AstherateofAIadoptionacceleratesbeyondmany

organisations’governancematurity,theyarepushedtochoosebetweenslowingAIimplementationorimprovinggovernanceresponsiveness.Withcompetitivepressurebuilding,theraceisontomorecloselymatch

governancecapabilitytocompetitivepressuresinordertoaccelerateAIinnovationandenableAIadoptionat

scale.

GovernanceneedstoevolvefromafocusoncontrollingAItechnologydeploymentstoalsogoverningoperationalAIdecision-makingandactions.Inthisevolution,

governancearrangementsshouldbeinfluencedbytwofactors:theagent’slevelofautonomyandthecriticalityoftheroleanddecisionstheagentisperforming.

Criticalityoftenincreasesasagentsinteractdirectlyintheorganisation’svaluechain.

Thinkingofgovernanceinthecontextofautonomyandcriticalityhelpstocalibratetheproperapproach.For

example,thecontrolpostureforagentswithlow

criticalityandautonomycouldbetoretainevidence.

Thosewithhighercriticalitybutlowautonomymayneedapprovals.

Highautonomybutlowcriticalityagentscouldrequireamonitoringcadenceandagentswithhighautonomyandcriticalitymayneedtheabilitytointervenebasedona

continuousmonitoringcadence.Itisfortheseagents

operatingattheedgeofthetwospectrumswhere

continuous,technology-enabledoversightisessentialtomatchthepaceandscaleofagenticoperations.

Criticality

Abilitytointervene

+

Monitoringcadence

Monitoringcadence

Approval

Evidenceretained

Autonomy

TheAItrustdividend06

Thetruesourceofintentlieswithhumandesigners,

builders,andbusinessowners.Governancemust

thereforeexaminehowintentisencodedintothesystem

Inpractice,governanceofagenticAImustaddressboththeunderlyingAIsystemandthebusinessactivityit

performs.Traditionalcontrolsovermodels,data,testing,

anddeploymentremainimportant,buttheyarenolonger

sufficient.Foreachagent,governancemustnow

addresstwodistinctquestions.Gettingthesetwo

aspectsrightwillprovidemuchhigherconfidenceintheoutputsgeneratedbytheAIsystem.

1.Istheagentdoingthingstherightway(execution)inareassuchasperformance,dataprotection,security,monitoring,andoperationalcompliance?

2.Istheagentdoingtherightthings(intent),meaningisitpursuingtherightobjectiveswithinitsdelegated

authorityandriskappetite?Everyagentneedsclearlimits:whatitcandecidealone,andwhenitmust

escalatetoahuman.

andhowongoingevaluationofitsadherenceisperformed.

Inaddition,governingthecombinedsystemofagentsworkingtogetherbecomesmateriallymoreimportant,includingitsoperationalbehaviour,orchestration,

permissions,tools,memory,decisionrights,actions,escalation,andhumanoversight.

TheAItrustdividend07

Threenecessaryshifts

AgenticAIgovernancerequiresleveragingexistingandnewcapabilitiesinahighlycoordinated,automated,andcontinuousmanner.Asorganisationsrevisithow

decisionsaremadeandbywhom,orwhatinthecaseofagenticAI,therearethreeprimaryshiftstoconsider.

Shift1:Humanaccountabilityatscale

Human-in-the-loop(HITL)controlscanprovideanimportantearly-stagemechanismforlearninghowagentsbehaveinreal-worldenvironments,validateoutcomes,andretainaccountability.

However,human-in-the-loopsupervisionisnolonger

sustainablewhereagentsincreaseinnumber,pace,andcomplexity.Here,weseeanevolutiontowardsmore

scalablehumanaccountabilitywithsupervision

increasinglyembeddedandengineered.Whatdoes

agentsupervisionandaccountabilitylooklikeinpracticewhentheratioofhumantoagentcouldbeoneto

hundreds?AsAIagentsundertakemoremonitoringandoperationaloversight,humansremainaccountablefor

outcomes,decisions,andriskappetite.Everyagentneedsabusinessownerwhoisaccountablefortheagent’sdecisionsandactions.

TheBoardshouldoverseewhethertheorganisationhaseffectivegovernanceforagenticAI,whileexecutiveandfunctionalmanagementshouldestablishclearownership,escalationrights,anddecisionaccountability,ratherthandirectlysupervisingeveryagentactivity.

Asapracticalsteptofacilitatehumanaccountability,

someorganisationsarebuildingnewuserinterfacesto

presentinformation(inputs,outputs)tohumanssuchthattheycanassesswhetheranagenthasmadeappropriaterecommendationsanddecisions.Thisincludesworkingthroughhowtobestpresentthepathtakenbyanagentinawaywhichisconsumable,understandable,and

connectedtotheoverallpurposeofanagenticworkfloworagroupofagents—thusenablingacontroldecisiontobemade.Aminimumdecisionrecordoftenincludes

thedelegatedobjective,actiontaken,evidenceor

rationale,anyexceptionflags,theaccountableowner,andescalationoutcome.Overtime,trustinagentic

operationsgrowsandthisAItrustdividendallowsforscaledadoptionwithinriskappetite.

TheAItrustdividend08

Shift2:Operatingmodeladaptation

Governancemustkeeppacewiththerateofinnovationandthefast-growingvolumeofagenticactivity.Evenforthosewhodecidetogoslow,areassuchascyber

securityneedtoconsideradvancedagenticAIuseandfamiliaritytoremainadequatelyprotectedfrom

adversarialthreats.

Aneffectiveoperatingmodelensuresthatautonomoussystemsremainobservable,measurable,andauditablethroughouttheirlifecycle.Continuousobservability

enablesoversightindynamicAIenvironments,while

rapidsensing,decision-making,andintervention

capabilitieshelporganisationsmanagerisksinrealtime.

Inconcept,organisationsshouldestablishanoperatingmodelthatrecognisesagentsas

digitalworkforce

counterparts

toemployees,utilisingsimilargovernanceprinciples.Everyagentshouldhaveaverifiedidentity,aclearlydefinedrole,task-specificaccessrights,

auditableactivitylogs,andwell-definedboundariesforautonomousdecision-making.Asagentsgaingreater

autonomy,correspondinglevelsofhumanoversight

shouldbestrengthened,particularlyforactionsinvolvingcustomers,employees,sensitivedata,financial

outcomes,orlegalandregulatoryobligations.

Amaturemodelmirrorstheprinciplesusedtogovern

humanworkforces,grantingauthoritybasedonroleandresponsibility(inthecontextofprocesscriticalityand

autonomy),whilerecognisingthatagentsrequireahigherdegreeofcontrol.

Shift3:Ongoinggovernanceoversight

WithagenticAI,governanceemphasisshiftsfrom

predominantlypre-productioncontrolstoabalance

betweendesigncontrolsandcontinuousoperational

monitoring,evaluation,andoversight.Governancepostlaunchshouldretainsufficient,risk-proportionate

evidenceofmaterialactions,decisions,tooluse,andexceptionstoenableinvestigationandintervention.

Post-launchobservabilitycanencompassunderstandingtheagent’sfullruntimebehaviourincludingwhatinputsitreceived,howitreasonedororchestrated,whichtoolsitinvoked,whatactionsittook,whatpermissionsit

exercised,whatstatechanged,andwhatoutcome

resulted.Insomescenarios,fullactionloggingsuchasthisismostappropriate.However,inothersthismaybeunfeasible,privacy-sensitive,ordisproportionately

costly.

Controlmethodssuchas

LLM

-

as

-

a

-

judgetechnology

(usingAItotestandevaluateAI)canhelpinpre-

productiontestingorongoingobservabilitywith

sufficientcontrolsovertheirowncalibration,bias,reproducibility,andfalse-negativerisk.

Theshifttocontinuousoversightalsorequiresaparallelshiftinbudgetingfromproject-basedtobusiness-as-

usualfundingandresourcingforagenticgovernanceoversight.Considerationshouldbegiventothetotalcostofagents,inclusiveofdeveloping,testing,

managing,andoptimisingeachagentoveritslifetime.

Thetrustdividend09

Risksandcontrolsinpractice

Inapplication,theriskcategoriesagenticAIgovernancemustaddressarefamiliar—itisriskexposureand

controldesignthatchange.Forexample,data,legalandcompliance,andprocessrisksareoftenwell-understoodcategories,butagenticexecutionchangestheirscale,

speed,andcontrolpoints.Infrastructureriskrequires

newthought,particularlyasitrelatestousinga

foundationalmodelprovider,agenticharnesses,andtherun-timeenvironmentthatunderpinseveryagentic

operation.

Modelrisk(suchashallucinations)andmodel

orchestrationacrossagenticplatformsareoftennew

territoriesasisuserisk(suchashumanover-relianceonAIoutputs).Wefindthesearetypicallyareasofgreaterchallengefororganisationsgiventhetechnology’srapidevolution.

Inadditiontoriskexposureandcontrolsconsiderations,fourprocessprioritiesrepeatedlyemergefromourworkhelpingorganisationsconfidentlyscaleagenticAI:

•Observationandevaluationarecritical:Ifsuch

monitoringisnotpossibleonaprocessordataqualityoutputcontrolsbasis,agenttelemetryormodel

orchestrationisrequiredtomaintainconfidencethat

agentscontinuetooperatewithinriskand

performancetolerances.Perouroperatingmodel

analogy,deployingagentsisalittlelikehiringhumans,withongoingobjectivealignmentandperformance

evaluationkeytotheirongoingsuccess.

•Agentcontrolblueprintsquicklybecomea

necessityasagentusemultiples:Bycreating

controlblueprintsbasedonreusablepatternsof

agentssharingthesamecriticality,risk,andautonomylevels,andthenmakingthemaccessibleonthe

organisation’sagentplatforms,newagentscanbe

deployedfasterfrombothoperationalandcomplianceperspectives.Increasinglytheseblueprintsdon’tjustguidehowagentsoperate,butalsohowtheyare

supervised,governed,andoptimisedovertime.

TheAItrustdividend10

•Donewell,anagentinventoryexpandsto

performanceandvaluemonitoring:Maintaininganagentinventoryisnecessaryforcomplianceand

oversight.Italsolessenstherisksofagent

proliferationwhich,overtime,addscost,complexity,

andrisk.Toinventoryeffectively,informationflows

needtobeconsolidatedintoonecockpitwhichthenfacilitatestheintegrationofperformanceandvalue

monitoring.Asthisoccurs,theAIgovernancefunctioncanevolvefromacompliancefocusintoa

governanceandvaluecapability,helpingthe

organisationunderstandnotonlywhetheragentsareoperatingsafely,butwhethertheremitgrantedto

themisactuallydeliveringvalue.

•Businessriskbecomesthemostcriticallens:As

agenticAIuseacceleratesandAIgovernance

matures,risksandcontrolsdiscussionspreviously

focusedoncomplianceriskshifttobusinessrisk.

Businessriskisfirmlycentredinthebusinesswiththesupportofstronggovernance.Businessconfidenceinagentautonomyisdependentongovernance

maturity.

Ofutmostimportanceisthatorganisationshavetherightfoundationsinplacebutmaintainflexibilityforthefutureastheneedfornewcapabilitiesarises.

TheAItrustdividend11

Buildingtrustedautonomy

TheemergenceofautonomousAIagentsrequires

organisationstomovebeyondviewingAIasmerely

models,data,orcode.Agenticsystemsintroducenewformsofautonomyintoorganisations,challenging

existinggovernanceboundariesandassumptions.Theycantransformhowworkgetsdone,butgranting

autonomyrequiresconfidence.

Thatconfidencecomesfromorchestratedgovernanceandamoderncontrolenvironmentthatmovesbeyondhumanreviewandreliesoncontinuous,technology-

enabledoversightwithhumanaccountability.

TheAItrustdividend