军事环境中的机器人和自治系统

TableofContents

Summaries.RoboticandAutonomousSystemsinaMilitaryContext9

Introduction9

TheMilitaryApplicabilityofRAS10

SocietalconsiderationsforthemilitaryapplicabilityofRAS15

OperationalconsiderationsforthemilitaryapplicabilityofRAS19

Conclusion22

Chapter1.TheMilitaryApplicabilityofRoboticandAutonomousSystems24

1.Introduction24

2.CategorizationofRASinamilitarycontext26

3.AssessingthemilitaryvalueofRAS32

4.CurrentRASapplicationsinthelanddomain37

5.OpportunitiesandRisks44

6.NextSteps47

Chapter2.TheEthicsofRoboticandAutonomousSystems

inaMilitaryContext51

ExecutiveSummary51

1.Introduction54

2.BackgroundandRecentDevelopments58

3.HumanAgency60

4.HumanDignity81

5.ResponsibilityandAccountability94

6.ConclusionandRecommendations105

Annexes108

Chapter3.ManagingRAS:TheNeedforNewNormsandArmsControl117

ExecutiveSummary117

1.Introduction118

2.RegulatingRAS:whyandwhat?120

3.Thecurrentstateofregulation123

4.Challengestoregulatoryusefulnessandeffectiveness127

5.Potentialpathsforward132

6.Conclusion140

Chapter4.EffectiveStakeholderCooperationduringtheLifecycle

ofRoboticandAutonomousSystems143

1.Introduction143

2.DistinctcooperationchallengesforRAS145

3.Development146

4.Integration149

5.Use151

6.Lifecyclemanagement153

7.Recommendations155

Chapter5,TheImplementationofRoboticandAutonomousSystems:

TheFutureisNow,preparefor2045159

1.Introduction159

2.Methodology161

3.Scenario:"2045:theeraofrelentlesscompetition”163

4.TheUnitoftheFuture171

5.Backto2020:LinesofDevelopment175

6.Conclusion186

Bibliography187

CapstoneReportII'heHague

RoboticandAutonomousSystemsinRICentrefor

nAStrategic

aMilitaryContextPaStudies

Summaries

RoboticandAutonomousSystems

inaMilitaryContext

Introduction

InDecember2019duringaspeechataRussianDefenseMinistryboardmeeting,Putinstatedthat

“roboticsystemsandunmannedaerialvehiclesarebeingrigorouslyintroducedandusedincombat

training,whichdramaticallybooststhecapabilitiesofarmedunitsandsubunits?Afewmonths

later,theRussianDefenseMinistryannouncedaclosedtenderworthapproximately4.2million

eurosthatsought"Researchoncreatinganexperimentalmodelofneuralnetworkdevelopment,

training,andimplementationforthenewgenerationofartificialintelligencemilitarysystems".

WhileChinaisfarlessboastfulpublicly,theirstrategyformilitarysuperiorityisleadbyevolutions

inAIandautomation,causingsomeanalyststostipulatethatthePLAaimstodominatethrough

system-of-systemsconflictandhighlyintelligentizedwarfare.

Theseblipssignifyalargerphenomenon.Militariesaroundtheworldaredeveloping,integrating

andusingroboticandautonomoussystemsinlinewiththeforthevolutionofwarfareandfurther

thinkingneedstobedoneregardingtheconditionsunderwhichthisprocesstakesplacewithinthe

Netherlandsandwhatchallengesandimplicationsarelikelytoariseasaconsequence.

TheHCSSprojectrRASinaMilitaryContext*soughttocontributetothisdiscussion.Overatwo-

yearperiod,theprojectyieldedfivepublicresearchpaperscoveringarangeoftopicsrelevantto

theimplementationofRASinamilitarycontext.Thissynthesistiesthesetopicstogetherand

presentsthemostpertinentfindingsoftheproject.ObservationsfromHCSSresearchonethical

requirements,legaldiscourse,partnercooperation,implementationandconceptdevelopmentand

experimentationaresummarizedbelow,precededbyaprimersectiononthemilitaryapplicability

ofRAS.

TheMilitaryApplicabilityofRAS

RoboticandAutonomousSystems(RAS)presentnumerous,significantandfar-reaching

opportunitieswithinamilitarycontext.Inordertoobservethewaysinwhichthesesystems

areapplicableinthiscontextandevaluatetheirutility,somedefinitionsandconceptsneedto

beaddressed:

Autonomy:Thelevelofindependencethathumansgrantasystemtoexecuteagiventask.

Itistheconditionorqualityofbeingself-governingtoachieveanassignedtaskbasedon

thesystem'sownsituationalawareness(integratedsensing,perceiving,analyzing),planning,

anddecisionmaking.Autonomyreferstoaspectrumofautomationinwhichindependent

decisionmakingcanbetailoredforaspecificmission,levelofrisk,anddegreeofhuman­

machineteaming.Levelsofautonomycanrangefromremotelycontrolled(non-autonomous),

OperatorAssistance,PartialAutomation,ConditionalAutomation,HighAutomation,or

FullAutomation.

Robot:Apoweredmachinecapableofexecutingasetofactionsbydirecthumancontrol,

computercontrol,orboth.Itiscomposedminimallyofaplatform,software,anda

powersource.

RoboticandAutonomousSystems(RAS):RASisanacceptedterminacademiaandthe

scienceandtechnology(S&T)communityandhighlightsthephysical(robotic)andcognitive

(autonomous)aspectsofthesesystems.RASisaframeworktodescribesystemswithboth

aroboticelementandanautonomouselement.Itisimportanttonotethateachofthe

consecutivepartsofRAScoversabroadspectrum.The'systems'-partreferstoawidevariety

ofphysicalsystemsoverawiderangeof(inourcase:military)applicationareas.Automated

softwaresystemsrunningoncomputersornetworks,including(bots\piecesofsoftwarethat

canexecutecommandswithnohumanintervention,donotqualifyasRASbecausetheylack

aphysicalcomponent.The'robotic'part,whichreferstothephysicallayoutofthesystem,

holdsthatthesystemisunmannedoruninhabited.Allotherphysicalaspects(size,form,

whetheritflies,floatsorrolls,etc.)areleftopen.

LethalAutonomousWeaponSystem(LAWS):Aweaponthat,withouthumanintervention,

selectsandengagestargetsmatchingcertainpredefinedcriteria,followingahumandecision

todeploytheweaponontheunderstandingthatanattack,oncelaunched,cannotbestopped

byhumanintervention.

Meaningfulhumancontrol(MHC):MHCencompasses(atleast)thefollowingthreeelements:

(1)Peoplemakeinformed,consciousdecisionsconcerningtheuseofweapons;(2)Peopleare

adequatelyinformedinordertoensurethattheuseofforceconformstointernationallaw,

withinthescopeoftheknowledgethattheyhaveonthegoal,theweapon,andthecontextin

whichtheweaponisputtouse;(3)Theweaponinquestionhasbeendesignedandtestedina

realisticoperationalsettingandthepeopleinvolvedhavereceivedadequatetraining,inorder

tousetheweaponinaresponsiblemanner.MHCisacomplexconceptand,inmanycases,

theabovedescriptionisnotconclusive.TheofficialDutchstandpointisthat"allweapons,

includingautonomousweapons,mustremainundermeaningfulhumancontrol.”

Therhetoricof“killerrobots"hasnarrowedthepublic'sviewofroboticandautonomoussystems

inamilitarycontexttobeingexclusivelyaboutlethaluseofforcebyhighlyorfullyautonomous

systems.Inreality,RAScanbeappliedtonumerousmilitaryfunctionsandtasks,withvarious

levelsofautonomyineachfunction(SeeFigure1),Thebroadmilitaryapplicabilityofroboticand

autonomoussystemsyieldsnumerousandvastopportunities.Thechallengefortheyearsahead

istomakethemostoftheseopportunitiesandwieldthepotentialformilitaryadvantagewhilst

simultaneouslymitigatingtherisksposed.

£

ServiceandOffensiveUse

SupportofForce

Monitoring,Area,perimeterLethalUse

surveillanceandandborderofForce

reconnaissancedefense

II'Non-lethalUse

.ofForce

TargetacquisitionPoint/Object

anddamagedefense

assessment

Escort

Cyber/signal

intelligence

Figure1.TherangeofapplicationareasforRASinamilitarycontext

ImplementingRASintothesefunctionsbringssignificantchallenges,butalsoheraldsnew

opportunitiesformilitariestobemoreeffective,efficientandagile.ThepotentialofRAStocontinue

to(rjevolutionisethedefensearenacanbeevaluatedaccordingtothesecategories.

Speed.Withthehelpofartificialintelligence,whichstimulatesrapiddecision-makingand

prioritizationofthreats,RASarealreadycapableofsurpassinghumanreactiontimesandshortening

theOODA(Observe,Orient,Decide,Act)loop.

Reliability.DelegatingtaskstomachinesrequiresanimmensedegreeoftrustandasofyetRAS

cannotproveadequatereliabilityacrossallmilitaryapplicationareas.However,ourconfidence

inthesesystemswillincreaseastheyprovetheirreliabilityandeffectivenessinexecutingspecific

tasks.

Accuracy.AIsystemshavedevelopedfacialimagerecognitionandsensoryabilitiespastthelevel

ofhumanperformance,thoughtheclaimthatunmannedsystemsaremoreprecisethanhuman

operativesiswidelydisputed.

Mass.Owingtoincreasedrangeandendurance,RAShasthecapabilitytoenhancecoverageofthe

battlespaceandoverwhelmadversaries.Thebestexampleofthispotentialis'swarming：

Reach.RASgreatlyenhancetheavailablepointsofpresenceforsurveillance,intelligence,

reconnaissanceandweaponssystems.

Robustness.Intheshortterm,RASwillbemorevulnerablethanhumanstofailduetounanticipated

conditionsincludingpoorweatherandchangestothemission.Thisfrailtyextendstothevirtual

domain:aslossesinconnection,hackingandotherinterferencecanrenderasystemincapable.

Safety.RAScanperform'dull,dangerousanddirty1taskssothathumanscanfocusonthemore

specializedtasksandbekeptoutofthelineoffire.

Cost.Althoughexclusiveaccesstothemostcutting-edgetechnologywillbereservedforthe

wealthiestplayers,thecostofsystemsthatarenowconsideredhighly-advancedwillfallthroughout

thenexttwentyyears,thusbecomingmorewidelyattainable.

Maintenance.UpdatingandupgradingRASsoftwareandhardwaremayprovemoredifficultgiven

thecomplexityofthesystemsandthemultiple(external)partnersinvolved.

Timeefficiency.RAScanperformdullandrepetitivemonitoringtasksatahighstandard24/7

withouttheneedforrest,logisticalplanningcanbesolvedefficiently,andthelimitsofhuman

multitaskingcanbequicklysurpassed.

Flexibility.AlthoughRAScurrentlyexcelinexecutingspecifictasksandhumanswillremainthe

mostflexiblefortheforeseeablefuture.Thisdynamicislikelytochangeasdeveloperscontinueto

innovatecurrentsystems.

Adaptiveness.RASarehighlyadaptiveandbeeasilyreconfiguredduringthesystem'slifecycle

(scaled,extended,upgradedetc.)overtimesotokeepupwithnewrequirementsemergingina

dynamicenvironment.

Externallegitimacy.Themilitary'sengagementwithRASmustthusstrikeabalancebetweenthe

advancedcapabilitiesthey(potentially)provideandthevaluesandnormsofthesocietyitserves.

Internallegitimacy.TrustandorganizationalnormalizationofRASwillbestrengthenedovertime.

Asunderstandingofthesystems,theirpredictabilityandtheirfamiliaritygrow,theirlegitimacy

withintheorganisationwillsolidify.

Figure2.NumberofRASprojectspercountryofuse(2018)

Recognisingthispotentialandsensinganeedtobemilitarilycompetitiveinarapidlychanging

internationalarena,anumberofstatesuseRASaspartoftheirarmedforces(Figure2).However,

despitetheapparentopportunities,implementingRASinamilitarycontextisnosimpletask.

Numerouspracticalanddoctrinalchallengesmartheimplementationprocess,demanding

discussionamongpolicymakers,innovators,researchers,thedefencecommunity,andmembersof

civilsociety,andinmanycases,thesechallengestesttheverysystemswedependontoregulate,

develop,acquire,integrateanduseothermilitarytechnologies.Inordertoanalyzethesechallenges,

itisimportanttoidentifythreestagesoftheRASsystemlifecycle:development,integrationand

use(Figure3).Thenextsectionsummarisesthedoctrinalconsiderations(ethicalandlegal)andthe

practicalconsiderations(privatesectorcooperationandconceptexperimentationanddevelopment)

exploredandanalysedduringtheHCSSRASProject.1

工Development

ThedevelopmentofRASisadynamicprocessofhardwareandsoftware

designandproduction,whichatlaterstagesisconsistentlyrevisited

accordingtotheresultsofsystemtesting,integration,monitoringand

use.ThedesignanddevelopmentofRASrequiresdeeperinteraction

andcooperationbetweenthedefencesectorandtheprivatesector.

Asaconsequence,theprivatesectorhasakeyroleinshapingthe

developmentstageoftheRASlifecycleandaddressingthedoctrinal

andpracticalconsiderationsrelevanttothisstage.

Integration

ThisstageconcernstheorganizationalembeddingofRAS,whereby

therelationshipwiththedeveloper/producerofthesystemchanges

andnewactors,suchastheactualmilitaryend-users,emergeor

acquireamoredominantrole.Duringthisstage,thenatureof*hand-

ovefchangesraisesnewquestionsabouttheroleofdifferentactors.

TheuseofRASinoperationalenvironmentsinfluencesthewaysthe

militaryworks,withwhomandunderwhatconditions.Thisisdue

tothefactthatgreaterautonomyofsystemsinquestiondrivesthe

operatorsandcommanderstointeractwiththesystemat“higher

levelsofabstraction0.Besidesdeployment,thisstagealsoincludes

maintenanceandserviceofRAS.

Figure3.ExplanationoftheRASlife-cycle.2

1ThispaperservesasasynthesisfornumberofresearchpapersproducedbyHCSSonthetopicofRASinaMilitaryContext.The

aforementionedresearchpapersdelveintoeachtopicingreatdetailandshouldbeconsultedformoredetailedinformation.

2ThespecificnatureofRASoftenresultsinaspiraldevelopmentprocess,wherebystagesofthelifecyclereoccurand/oroccur

simultaneously.Thoughrecognizingthecomplexityandinterrelatednatureofthisprocess,thesomewhatlineardivisionofthe

RASlifecycleintoDevelopment,IntegrationandUseisemployedinthispaperforthesakeofsimplicity.

Societalconsiderationsforthemilitary

applicabilityofRAS

Ethicalandlegalconsiderationsonthedevelopment,integrationanduseofRASforamilitary

contextabound.Whilethecurrentethicaldebateonroboticandautonomoussystems(RAS)

isoftendominatedbyrelativelyextremenarrativessurroundingatotalbanon"killerrobots',

currentdiscussionsonRAShavesidelinednuancesthathavecriticalimplicationsfordeciding

howtointroduceRASinamilitarycontext.ThebrewingAIarmsraceandthediffusionofcheap,

technologicallyadvancedsystemsamongstateandnon-stateactorscompelscountriestoadopt

RAS.Howmilitariescandosowhilstalsokeepingin-tacthumanagency,humandignityand

responsibilityisofgreatimportance.

Ethicalconsiderations

Maintaininghumanagency,particularlyinthecontextofAWS,isoneofthemostcontentious

issuesofdebatewithrespecttotheintegrationofRASinthemilitarydomain.Humanagency

isaconceptthatencompasses“self-control,morality,memory,emotionrecognition,planning,

communicationandthought.03Itincludes°featuresofself-awareness,self-consciousnessandself­

authorship/andasaresultrelatestomoralagencyandaffectstheattributionofresponsibility.34

Humancontrol,alsoreferredtoas'meaningfulhumancontrol'(MHC),isanoperationalcomponent

ofhumanagency,whichdistinguishesbetweenhumanandartificialdecision-makingprocesses.5

AfundamentalaspectofmaintainingMHCistheoperator'sunderstandingofthealgorithmic

process'parameters,theoutcomespresentedasaresultofthecomputation,andtheabilityto

explainthemachine'spathtoconclusionafterthefact.Fromthispointofdeparturestemsan

importantethicalconcernofRAS,andAIinparticular:thelackofalgorithmictransparency.

Algorithmssuchasneuralnetworkssufferfromopacityastheyoperateas/blackboxes",whereby

thepathtakenbythealgorithmtoarriveattheconclusionisoftennottraceable.6Thediminished

understandinganoperatorhasofsuchsystemsreducestheirabilitytopredictand/orexplainthe

3Gray,Gray,andWegner,^DimensionsofMindPerception/

4EuropeanGrouponEthicsinScienceandNewTechnologies(EGE),StatementonArtificialIntelligence,Roboticsand

“autonomous"Systems.

5MHCinterrelateswith“effectivecontrol”,aprerequisiteinpublicinternationallawforlegalliabilityandunlawfulconduct.In

thecontextoftheuseofRAS/AWS,thetermisusedalongside“effectivecommand'/todeterminestateresponsibility.

6Preece,4,Asking'Why'inAI:ExplainabilityofIntelligentSystems-PerspectivesandChallenges^Matthias,**TheResponsibility

Gap:AscribingResponsibilityfortheActionsofLearningAutomata/178-79.

system'sreasoningprocess,underminingthecontrolthattheoperatorhasovertheoutcomesand

hence,theresponsibilityforits(mis)use.Furthermore,theevolutionarynatureofalgorithm-driven

systems,bothasaresultofself-learningpropertiesandsoftwareupdates,hasthepotentialto

considerablyaffectexplainabilityofsystems1actions.Self-learningAIthatindependentlydevelops

itsunderstandingofthesurroundingenvironment,automationbiasandexcessivetrustinsystem

outputsmaylimithumancontroloveraRASsystem'soperation.Asthedesignofasystemcan

incorporatevariousdegreesofautonomy(fromremotecontrolledtofullyautonomous)withinthe

multiplefunctionsofasystemacrosstheObserve,Orient,Decide,Act(00DA)loop,meaningful

humancontrolprinciplesshouldbeconsideredattheearlieststagesofdevelopment.

Thefundamentalguidingprincipleistoworkwith'ethicsbydesign',wherebyethicalconsiderations

areincorporatedintheusecaseidentification,systemdesign,validation,manufacturing,and

testingprocesses,ratherthansolelyinthe'use'stageofthesystemlifecycle.Thisentailsbuilding

anunderstandingofthesystemperformanceandbehaviorearlyinthedesignandtestingstages

byinvolvingend-usersearly,meaningthatoperators,supervisorsandcommanderswillbebetter

abletotrace,understandandpredictthesystem'sdecision-makingprocess.Bestpracticeguidelines

shouldbecreatedfbrtheoutsourcingofthedevelopmentprocesstoexternalcontractors.

Legalconsiderations

ThelackofmeaningfulhumancontrolinRASresultsinconsiderationsfbrlegaldiscoursetoo

anddebateongoverningautonomousweaponsisgainingmomentum.Internationalpositionsstill

differwidely,rangingfromproponentsandopponentsofabanonsuchweaponstoagroupof

countriesthatlieinbetweenandemphasizetheneedforfurtherclarificationandelaborationof

existingregimes.Itisclearhoweverthatcurrentrules,standards,andpracticesarerelevantbut,

mostprobably,insufficienttocoverdevelopmentswithregardtoautonomousweapons.Atthe

veryleast,RASwouldrequirerefinementsofexistingregulation.Whiletheconsensus-basedCCW/

GGE7stillcountsasanecessarytooltofurtherthisdebate,itisdoubtfulwhetherthiseffortaloneis

sufficient.DespitetheinclusionofNGOsandacademia,statepartiesaredominantinthisformat,

andindustryisonlypresentinabackbenchcapacity.Amidvariousapproachestodefinitions,

norms,andstandardsattheinternationallevel,theNetherlandsneedtodecideonthedirections

ofthemodernizationoftheirarmedforcesandtheirinternationalpostureamidanintensifying

publicdebate.

LegalapproachestoregulatingRASincludehardlaw,softlaw,andvoluntarymeasures.Hardlaw

concernsbindingtreatiesthatarenegotiatedandagreeduponbetweenstates.Softlawinvolves

quasi-legalinstrumentssuchaspoliticallybindingCodesofConduct(CoCs)orConfidenceand

SecurityBuildingMeasures(CSBMs),sometimesinvolvingmultiplestakeholdersotherthanstates.

7TheGroupofGovernmentalExperts(GGE)oftheHighContractingPartiestotheConventiononCertainConventionalWeapons.

Finally,voluntaryinstrumentsincludebehaviouralprinciplesornormsandexchangesofbest

practicesorotherinformation,withinoroutsidetraditionalarmscontrolcommunities(FigureX).

Figure4.ThethreeapproachestotheregulationofRAS.

NumerouscharacteristicsinherenttoRASmakeregulation(especiallyhardlawapproaches)

particularlychallenging.First,thesetechnologiesaredevelopingfastandoffersometimesspectacular

prospectsfortheiruseinbothmilitaryandcivilianapplications,hencethetemptationofsome

politiciansandNGOstohoneinonalarmistscenariosandframethediscussionexclusivelyinterms

of'killerrobots：Thisoversimplificationhindersnuancedlegaldebate.Thefactthatautonomyis

notastaticfunctionofweaponryfurthercomplicatesthematter,becausethiselusivenessmeans

thatthediscussionisnotalwaysaboutidentifiable(weapons)systems,asisthecasewithexisting

regimesthatmostlyregardspecificcategoriessuchaschemical,biological,ornuclearweapons,or

certaintypesofconventionalarmsordeliverysystems.Instead,regulationneedstograpplewith

algorithmsthataredual-useandthatmayormaynotbeharmfulwhenimplementedintoasystem

(forexample,harmmaydependonanactionthatthesystemlearns*afteryearsofuse).

Second,giventheearlyandcomplexphaseoftheRASdebate,andthelackofcommonlanguage

ondefinitionsandcategorizations,hardlawcouldbeoutofreach.Existingarmscontrolregimes

relatetoestablished,well-definedweaponcategories.Furthermore,thecurrentgeopoliticalclimate

seemshardlyconducivetonewmultilateralarmscontrolagreements.

Lastly,technologicalinnovationhastraditionallyemanatedfromthemilitary-industrialcomplex,

withinnovationsfindingcivilianapplicationslateron(knownasthespin-offeffect].Inthecase

ofRAS,thetrendappearstobegoinginthereversedirection(spin-in),anditisclaimedcivilian

innovationfacilitatesthedesignofLAWS.Intheframeworkofdesigningcontrolregimes,'spin-in'

requiresstronginteractionwiththeprivatesectorandwillleadtoformsof"sharedresponsibility

andaccountability",whicharenotentirelynewbutwillbemoredifficulttomanage.

Forthesereasons,softand/orvoluntaryinstrumentsappeartobethemorerealisticwayforward,

astheseareeasiertoreach,havelowerthresh