SoftwareX Update (1.1) to ANDURIL - A MATLAB toolbox for ANDURIL 的 SoftwareX 更新 (1.1) - 用于的 MATLAB 工具箱

Contentslistsavailableat

ScienceDirect

SoftwareX

journalhomepage:

/locate/softx

SoftwareX10(2019)100295

Softwareupdate

Update(1.1)toANDURIL—AMATLABtoolboxforANalysisandDecisionswithUnceRtaInty:Learningfromexpertjudgments

ANDURYL

CornelisMarcelPieter’tHart

a

,

b

,

∗

,GeorgiosLeontaris

a

,OswaldoMorales-Nápoles

a

aCivilEngineeringandGeosciences,DelftUniversityofTechnology,TheNetherlands

bTunnelEngineeringConsultants(TEC),Amersfoort,TheNetherlands

article info

Articlehistory:

Received9July2019

Receivedinrevisedform19July2019Accepted23July2019

Keywords:

StructuredexpertjudgmentCooke’sclassicalmodelExpertopinion

PythontoolboxEXCALIBURsoftwareANDURIL

abstract

ThisisanupdatetoPII:

S2352711018300608

Inthispaper,wediscussANDURYL,whichisaPython-basedopensourcesuccessoroftheMATLABtoolboxANDURIL.TheoutputofANDURYLisingoodagreementwiththeresultsobtainedfromANDURILandEXCALIBUR.AdditionalfeaturesavailableinANDURYL,andnotavailableinitspredecessors,arediscussed.

©2018TheAuthors.PublishedbyElsevierB.V.Allrightsreserved.

Codemetadata

Currentcodeversion Code:ANDURYLv1.0,Paperv1.1

Permanentlinktocode/repositoryusedforthiscodeversion

/ElsevierSoftwareX/SOFTX_2019_237

CodeOceancomputecapsule

/10.24433/CO.7459237.v1

LegalCodeLicense GNUGeneralPublicLicense

Codeversioningsystemused None

Softwarecodelanguages,tools,andservicesused Python,SCIPY,NUMPY,MATPLOTLIB

Compilationrequirements,operatingenvironments&dependencies PythonVERSION3.6

IfavailableLinktodeveloperdocumentation/manual

/10.24433/CO.7459237.v1

Supportemailforquestions

C.M.P.tHart@tudelft.nl

Softwaremetadata

Currentcodeversion ANDURYLv1.0

Permanentlinktocode/repositoryusedforthiscodeversion

CodeOcean

LegalCodeLicense GNUGeneralPublicLicense

Codeversioningsystemused CodeOcean

Softwarecodelanguages,tools,andservicesused Python,SCIPY,NUMPY,MATPLOTLIB

Compilationrequirements,operatingenvironments&dependencies PythonVERSION3.6

IfavailableLinktodeveloperdocumentation/manual

/10.24433/CO.7459237.v1

Supportemailforquestions

C.M.P.tHart@tudelft.nl

DOIoforiginalarticle:

/10.1016/j.softx.2018.07.001

.

∗

Correspondingauthorat:CivilEngineeringandGeosciences,DelftUniversityofTechnology,TheNetherlands.

E-mailaddress:

c.m.p.thart@tudelft.nl

(C.M.P.’tHart).

/10.1016/j.softx.2019.100295

2352-7110/©2018TheAuthors.PublishedbyElsevierB.V.Allrightsreserved.

PAGE

2

C.M.P.’tHart,G.LeontarisandO.Morales-Nápoles/SoftwareX10(2019)100295

C.M.P.’tHart,G.LeontarisandO.Morales-Nápoles/SoftwareX10(2019)100295

PAGE

3

Table1

OverviewofresultcomparisonAIandAYagainstCC.

Software

Numberofstudiescompared

Numberofdifferentscoresin

Table

2

Numberofscoreswithapproximationdifferences

NumberofscoreswhereAI

=AYbutdifferenttoCC

RelativeagreementaftercorrectionforapproximationandAI=AY

AI

18(55%)

13(96%)

4

9

100%

AY

33(100%)

23(96%)

8

9

99%

Motivationandsignificance

AMATLABtoolbox,namedANDURIL,

1

(AI),implementingCooke’sclassicalmodel[

1

]forstructuredexpertjudgmentispresentedin[

2

].UntilrecentlyEXCALIBUR

2

(CC)wastheonlyavailablesoftwareimplementingCooke’sclassicalmethod.ThoughEggstaff’sstudieswerebasedonaMATLABimplemen-tation

3

[

3

,

4

],thedevelopedsourcecodeforthesestudiesisnotavailablefordistribution.

InthispaperwepresentANDURYL(AY),whichisaPython[

5

]implementationofCooke’sclassicalmodel[

1

].TheprogramnamereplacingtheIwithYindicatesthattheAYsourceisbasedonPythoninsteadofMATLAB.TheprogramstructureofAIhasbeenretainedinthisimplementation.ThemainobviousadvantageofAYisthattheMATLABlicenserequiredforAIisnotrequiredforAY.OtheraddedfeatureswithrespecttoAIwillbediscussedalongthispaper.

Softwaredescription

AYisrunfromthecommandlinewiththePythonfunctionmain.py,asitdoesnothaveagraphicaluserinterface.Userscanadaptthecodetoruntheirownstudiesinsequencesaspresentedinanduryl_example.py.TheprogramstructureissetupinsuchawaythatthereisonemainPythonfunctionandurylwhichisusedtorunthefullscopeofAY.Inthismainscript,thedataobtainedfromexpertjudgmentsmaybeenteredinordertoconductthedesiredanalysis.Theinputvariablesaresetasglobalvariablesandbackedup.With‘restore’statementsthevariablescanberesettotheoriginalinputvalues,whichcanbeusedinlatercalculations,butmightalsobeusefulinfurtherdevelopmentsofAY.Inthecurrentimplementation,thisisusedintheprocessforinvestigatingtherobustnessoftheobtainedDecisionMakers(DM).ThesupportedfunctionalitiesofCooke’sclassicalmodelinAYare:

CalculationofDMusingglobalweights;

CalculationofDMusingitemweights;

CalculationofDMusingequaloruserdefinedweights;

OptimizationofDM;

Robustnesscheckitemwise;

Robustnesscheckexpertwise;

Plottingassessmentsitemwise;

Plottingrobustnessresults.

ThefunctionsofAYaresimilartothefunctionspresentedforAI.AYkeepsitsarchitectureassimilaraspossibletothatofAI.Themaindifferencehoweverisinthefunctioncalcu-late_weights,whichmergesAI’sfunctionsglobal_weightsanditem_weights.AmoredetailedexplanationoftheprogramispresentedintheSupplement.TheremainingdifferenceswillbefurtherdiscussedinSection

4

.Nextwepresentresultsofcom-paringAY’soutputtobothCCandtheMATLABimplementationAI.

Freelyavailableat

/ElsevierSoftwareX/SOFTX_2018_39

.

Freelyavailableat

/wp/excalibur

.

ThisMATLABimplementationisnotEXCALIBUR.

ComparingoutputofANDURYLwithpreviousexpertjudg-mentstudies

In[

4

],33post-2006studiesusingCooke’sclassicalmethodarepresentedusingCC.WeusethesedatatocompareoutputfromAYtobothCCandtheMATLABimplementationAIofthepreviouspaper[

2

].

Table

2

presentstheresultsreportedinTable1of[

4

](thestudynamefollowedbyCC)extendedwithcalculationsfromAI(AI)andAY(AY).

Table

2

includesthestatisticalaccuracy(SA),information(In)andthecombinedscores(Co).

Equalweight,Globalweightswithoutoptimization(GlobalNoOp.),Globalweightsoptimized(PWGlobal),Itemweightsoptimized(PWItem)andtheexpertwithhighestcombinedscore(BestExpert)arepresented.Inthesupplement,anextendedtableincludingItemweightswithoutoptimization(ItemNoOp.)andtheexpertwiththelowestcombinedscoreispresented.

Fromthe33studiesreported[

4

],14wereperformedusing5quantiles,3withquantilesotherthanthe5th,50thand95thorcontainedmissingitemsforsomeexperts.TheseresultscannotbecomparedwithAIandaremarkedby(*).OntheEBPPstudy,asoftwareerrorappearedintheMATLABcode.ThiserrorwillberesolvedinafutureupdateofAI.Hence,atotal18studieswerecomparedwithAI.Eachstudyin

Table

2

presents17numbers.Differencesbetweenthecalculationsreportedin[

4

]andAIarehighlightedinblue.Thereareatotalof153bluenumbersin

306

Table

2

andhenceanagreementof(1−13)×100≈96%between

AIandthecalculationsreportedin[

4

]forthestudiesthatcanbecompared.Fromthe13numbers4areclearlyapproximationdifferences.NoticethatthoughthenumbersinCCareMATLAB-basedwecompareourresultstothepublishedresultsin[

4

]andnowaytoinvestigatefurthertheapproximationusedin[

4

]isavailabletotheauthors.Additionally,9numbersareequaltotheresultsobtainedwithAY.Thesetwoobservationswouldbringtheagreementto100%.

Differencesbetweenthecalculationsreportedin[

4

]andAYarehighlightedinredinthesametable.Thereareatotalof23

561

rednumbersin

Table

2

andhenceanagreementof(1−23)×

100≈96%betweenAYandthecalculationsreportedin[

4

].Fromthe23rednumbers8areclearlyapproximationdifferences.Additionally,9AYresultsareequaltothoseobtainedwithAIwhichwouldbringtheagreementto≈99%.ThisresultindicatethatbothAIandAYmaybeusedwithenoughconfidencebyinterestedusers.

Theresultsofthecomparisonaresummarizedin

1

.

In

Table

2

,9valuesareequalforAIandAYbutdifferentcom-paredtoCC.Theauthorscheckedtheinputfilesofthe‘‘Icesheets"study.Itwasfoundthattherealizationfile(*.rls)andthefilewithassessments(*.dtt)presentedinconsistenciesinthelabelingofassessmentquestions.WespeculatethatthiscouldbethesourceofthismisalignmentofbothAIandAYwithCC.

Thedifferencesfoundinthe‘‘Gerstenberger",‘‘Goodheart"and‘‘Hemopilia"studyarerelatedtotheoptimizationprocess.Forexample,theoptimizationprocessfor‘‘Goodheart"datashowsinCC1expertastheoptimalcombination.ForbothAIandAYtheoptimalcombinationconsistsof3experts.WithoutthesourcecodeofCCtheauthorscannotinvestigatefurtherthissourceofmisalignment.

Table2

ComparisonofresultspresentedinTable1of[

4

](CC)andcalculationswithAI(AI)andAY(AY).

aTheauthorsfoundasoftwareerrorinAI,thisparticularstudyhasnotbeenvalidatedtoAI.InafutureupdateofAIthesoftwareerrorwillbesolved.

Fig.1.Hypotheticalexampleof4expertsassessing10seedvariables.

Table3

StatisticalaccuracyandInformativenesscomputedwithAYandCCforthehypotheticalexamplepresentedin

Fig.

1

assumingexpertselicited10th,50thand90thpercentilesoftheiruncertaintydistribution.

ExpertID

Calibration

Calibration

Information

Information

(CC)

(AY)

(CC)

(AY)

ExpertA

5.529E−10

5.530E−10

1.371

1.371

ExpertB

5.529E−10

5.530E−10

0.571

0.571

ExpertC

0.371

0.371

0.039

0.039

ExpertD

0.526

0.526

0.629

0.629

Global

0.526

0.526

0.431

0.431

(non-opt.)

Impact

TheadvantagesofAI,discussedin[

2

],withrespecttoCCareinheritedbyAY.AnumberoflimitationsofAIwerediscussedinthesupplementof[

2

].BesidesthefullopensourcecharacterusingPythonasaprogramminglanguage,twootheradvantageswereimplementedincomparisonwithCCand/orAI.Theseareelaboratedfurthernext.

Userdefinedquantiles

From

Table

2

itmaybeobservedthatAYpresentsgoodagree-mentwiththe11studiesreportedin[

4

]where5quantiles(5th,25th50th,75thand95th)wereusedtoelicitexpertjudgments,hencewedonotelaboratefurtheronthisissue.

Asstatedearlier,AYprovidestheoptionofuserdefinedquan-tiles.CCallowsfortheuseof3,4or5userdefinedquantiles.

Fig.

1

presentsahypotheticalexampleof4experts:A,BCandD,assessing10calibrationorseedvariables.Therealization(R)isalsoshown.

Intuitively,thereadermayalreadyappreciatethatexpertAwillbeinformativebutwithlowSA.ExpertBwillbelessinfor-mativeandalsopresentlowSA.TheSAforCandDwillbeequal,however,DwillbemoreinformativethanC.

Table

3

presentsacomparisonofthecalculationsofSAandinformativenessbe-tweenAYandCCassumingexpertselicited10th,50thand90thpercentilesoftheiruncertaintydistribution.Thereadermayap-preciatethattheagreementbetweenthecalculationsperformedbyCCandAYisalmostexact.

BecausethesourcecodeofAYisavailableandextendedwithrespecttoCC,practitionersmayusemorethat3,4or5userdefinedquantilestoelicitexpertjudgments.Thesamehypothet-icalexamplewithfourexpertsasin

Table

3

isusedbutwithexpertsassessing7quantiles(10th,25th,35th,50th,65th,75th

Table4

StatisticalaccuracyandInformativenesscomputedwithAYwith7quantilesforthehypotheticalexamplepresentedinSection

4.1

assumingexpertselicited10th,25th,35th50th,65th,75thand90thpercentilesoftheiruncertaintydistribution.

ExpertID

Calibrationscore

Informationscore

Un-normalizedweights

Normalizedweights

ExpertA

8.542E−08

1.3738

1.173E−07

9.403E−07

ExpertB

8.542E−08

0.5710

4.877E−08

3.908E−07

ExpertC

0.0041

0.0393

0.0002

0.0013

ExpertD

0.1004

0.6302

0.0633

0.5069

Global

0.1004

0.6114

0.0614

0.4918

(non-opt.)

and90th)ispresentedin

Table

4

(intermediateassessmentshavebeenobtainedbyinterpolatinglinearlytheestimatessummarizedin

Fig.

1

).

ThoughthisoptionisavailableinAY,itisuncleartotheauthorsitsapplicabilityinpracticesincethecomplexityofelic-itingexpertjudgmentsgrowssignificantlywiththenumberofquantilestobeelicitedfromexperts.Itisalsouncleartotheauthorsifnostudyconsideredtheelicitationofmorethan5quantilesbecausethisfeaturewasnotavailableinanysoftwareimplementation.

Missingitemsforsomeexperts

In[

6

]twopanelsof9expertsweregatheredinordertoassessuncertaintyovereconomicgrowthandoilpricesforMexicoin2020and2030.Inthepanelcorrespondingtointernationalgasandoilprices,expertAdidnotanswer10of26calibrationvariables.NoanswerforexpertDwasrecordedfor5calibra-tionvariables.Similarly,noanswerto1calibrationvariablewasobservedforexpertG.TheresultsofcalculationsobtainedwithmissingitemsforbothAYandCCarepresentedin

Table

5

.Similarlyasin

Table

3

,theagreementbetweenthecalculationsobtainedwithCCandAYisalmostexact.

Conclusions

TheMATLABtoolboxnamedAIforcombiningexpertjudg-mentsapplyingCooke’sclassicalmodelforstructuredexpertjudgmenthasbeenextended.ThenewsoftwareiscalledAN-DURYL.ThemainpurposefordevelopingthesetoolboxesistocreateopensourcesolutionsthatcanbeusedbypractitionersandresearcherswhoareinterestedinapplyingordevelopingfurtherCooke’smethod.IncomparisonwithAIand/orCC,AYpresentsthefollowingnewfeatures:

AYhasinheritedalladvantagesofAIdiscussedin[

2

].Ad-ditionally,AYisfullyopensourceandallowsforuserdefinedquantiles(see

4.1

)andmissingitems(see

4.2

).

ThesoftwaretoolpresentedinthispapervalidatesCooke’sclassicalmodelsuccessfullywitharangeofstudiespresentedin[

4

].DespitethelimitationsofthecurrentversionofAY,itistotheauthorsbeliefthatsimilarlyasAIthedevelopedtoolboxwillbevaluabletothosewhoareinterestedindevelopingandfurtherapplyingthemethod.ItistheambitionoftheauthorstoextendAIandAYwithmorefeaturesthanthosecurrentlyavailableinCCandwiththemorerecenttechniquesofelicitationofmultivariatedependence[

7

].

Declarationofcompetinginterest

Wewishtoconfirmthattherearenoknownconflictsofinter-estassociatedwiththispublicationandtherehasbeennosignif-icantfinancialsupportforthisworkthatcouldhaveinfluenceditsoutcome.

Table5

ComparisonofcalculationsfromAYandCCfortheexpertpanelpresentedin[

6

].

ExpertID

Calibration(CC)

Calibration(AY)

Information(CC)

Information(AY)

Information(CC)

Information(AY)

ExpertA

1.634E−7

1.635E−7

1.347

1.347

1.235

1.235

ExpertD

0.07205

0.07209

1.045

1.045

1.004

1.004

ExpertG

0.0004775

0.0004774

1.075

1.0745

1.262

1.262

Global

0.1512

0.1512

0.8549

0.8549

0.8