HarnessDev：大模型能否自主构建并迭代其智能体框架

1

llByteDanceIseed

AYTokenwave

HarnessDev:CanLLMsCreateandEvolveTheirOwn

AgentHarness?

1ByteDanceSeed,2SingaporeUniversityofTechnologyandDesign,

3GeorgiaInstituteofTechnology,4M-A-P,5TokenWave.AI

FullauthorlistinContributions

arXiv:2609.01437v1[cs.SE]1Sep2026

Abstract

Asagentsmovefromresearchprototypestodeployedtools,theircapabilityincreasinglydependsonmodel-externalexecutioninfrastructure,commonlytermedtheagentharness.Changingthisharnesswhileholdingmodelweightsfixedcansubstantiallyaltertaskperformance.Currentagentevaluationstypicallyreportdownstreamperformanceunderachosenharness,leavingamodel’sabilitytodeveloptheharnessitselfcomparativelyunderexplored.WeintroduceHarnessDev,abenchmarkthatshiftstheunitofevaluationfromtaskoutputstorunnableinfrastructure.HarnessDevcoverstwostages.InCreation,theagentstartsfromaminimalseedandasmallnumberofcases,thenbuildsacompleteexecutionsystem.InEvolution,itstartsfromitsowncreatedharnessanditerativelyrevisesitusingdownstreamexecutionfeedback,withthegoalofimprovingbenchmarkperformance.Wethenevaluateeachconstructedharnessoncapability—tasksuccessonheld-outbenchmarks,andefficiency—execution-tokencost.ThereportedCreationresultscoversixcreatorLLMs,fourdomains,andfivedownstreambenchmarkstotaling2,207uniquedownstreaminstances,withhiddenevaluationtaskswithheldfromdevelopment.Wefindthatgeneratedharnessesremainsubstantiallybehindmaturehuman-engineeredreferencesoncodeandonsearchandresearch,whilematchingorexceedingtheselectedreferencesonwritingandmachine-learningexperimentation,withlargevariationinexecutioncost.Evolutionproducessomeperformancegains,buttheyareunstableandtransferonlypartiallytoheld-outtasks.Experimentswithafixedruntimemodelfurthershowthatthegainsdependstronglyonthemodelexecutingtheharness,indicatinglimitedtransferacrossmodels.

Date:September2,2026

ProjectPage:

https://self-developing-agents.github.io/

1Introduction

Asagentsmovefromresearchprototypestodeployedtoolssuchascodingassistants[

2

,

32

],data-analysiscopilots,browserworkers

[5

],andresearchpipelines,theircapabilityincreasinglydependsonsoftwareoutsidethemodel’sweights.Thissurroundingexecutioninfrastructure,commonlytermedtheagentharness[

31

,

39

],managestheexecutionloop,tooluse,context,failurerecovery,andresultverificationthatturnmodeloutputsintoactions

[3

].Itsimpactissubstantial:withidenticalweights,GPT-5solves35.2%ofTerminal-Bench2.1insideTerminus2but49.6%insideCodexCLI[

45

].Asagentsspecializetomoredomains,thedemandforpurpose-builtharnesseswillcontinuetogrow.Becausethesesystemsrequirecontinuousdevelopmentratherthanone-timeimplementation,apracticalquestioniswhetherLLMscanassistharnessengineers—oreventakeoversucharole—inbuildingandcontinuallyimprovingtheharness.

2

ExistingAgentBenchmarks

SWE-bench·GAIA·WebArena·τ-bench

FIXEDHARNESS

toolscontextmemoryloopverify

AGENT

ONETASK

Evaluatedartifact:answer

SCAFFOLDFIXED

Evolution

keepdevelopingitasrequirementsshift

Creation

buildafullharnessfromaweakseed

H1

updateshipped

+ctxcompaction

+retry(n=3)

✓oldtestspass

▫memory:none

Hn

+memorystore

+subagents

✓verifygate

hidden✓

regression✓

H0

fromCreation

✓loop·tools

▫ctx:overflows

▫memory:none

▫retry:none

WeakSeed

noloop

noplanner

noverifier

CLIfilesexec

HarnessDev

Notonlytasksolvers,butalsosystembuilders

devtasksvisible·hiddentaskswithheld

everyversionfrozen,runnable&tested

Runnable

H0

CREATORLLM

analyze→edit→solve

REQ1

DEVELOP

REQN

DEVELOP

DEVELOP

Figure1HarnessDevcoverstwostagesofharnessdevelopment.InCreation,acreatorbuildsacompleteharnessfromaweakbutrunnableseedandasmallnumberofdevelopmentcases.InEvolution,itcontinuestoimproveitsownpersistentharnessusingdownstreamexecutionfeedback.Bothstagesevaluaterunnableinfrastructurethatpersistsacrosstasksratherthanaone-oftaskoutput.

Despitethispracticalneed,mostagentevaluationsselectaharnessforagivencomparisonandreportmodelperformanceondownstreamtasks

[18

,

26

,

29

,

54

,

61

].Thissetupsupportscontrolledtask-levelcomparison,buttreatstheharnessaspartoftheexperimentalconfigurationratherthanasanartifacttobedeveloped.Recentworkhasbeguntostudyharnessrepresentations,automatedagentdesign,andagentsthatbuildorimproveagentsystems

[15

,

27

,

31

,

39

,

56

,

57

].However,itremainsunderexploredwhethermodelscanbothcreateandcontinuallyimproverunnable,persistentharnesses.Answeringthisquestionrequiresseparatingthemodelthatdevelopstheharnessfromthemodelthatexecutesdownstreamtasks,recordingthedevelopmentenvironment,andmeasuringdownstreamperformance,transferacrossexecutors,distancefromhuman-engineeredsystems,regression,andcost.

Thisevaluationgapisparticularlyconsequentialbecauseharnessengineeringisfundamentallydifferentfromordinarycodeediting.Whenamodelmodifiesastandaloneprogram,thetargetbehaviorisexternallyspecifiedandsuccessislocallyverifiable.Whenamodelmodifiesitsownharness,itiseditingtheexecutionsubstratethroughwhichitacts:thechangealtershowthemodelitselfobserves,plans,andrecoversinallfuturetasks.Effectiveharnessimprovementthereforedemandsthatthemodelrecognizeitsownbehaviorallimitationsfromexecutiontraces

[42

],diagnosestructuralbottlenecksinthesystemitrunsinside,andcommittargetedchangesthataccumulateintolasting,reusablecapabilitygainsratherthanone-offfixes[

49

,

53

].Asfrontiermodelsgrowcapableenoughtoeditmulti-filecodebasesandcloserealpullrequests,thisabilityisalreadylatent;whatismissingisabenchmarkthatmeasuresit.

WeintroduceHarnessDev(Figure

1

),abenchmarkthatfillsthisgapbyshiftingtheunitofevaluationfromtaskoutputstorunnableinfrastructure:measuringamodel’sabilitytoconstructandmaintainexecutionsystemsthataredurable,inspectable,andreusable.Thenamereflectsthesoftware-engineeringsenseofdevelop:developingaharnessincludesbothbuildingitfromscratch(i.e.,Creation)andimprovingitthroughcontinuediterationandmaintenance(i.e.,Evolution).Thebenchmarkcoverstwostagesofharnessdevelopment.InCreation,acreatorLLMstartsfromadeliberatelyweakbutrunnableseedandbuildsacompleteharnessforanewtaskfamily.InEvolution,itstartsfromanexistingharnessandcontinuestodevelopittowardbetterdownstreamtaskperformance.Together,thetwostagesevaluatewhethermodelscancompleteharnessdevelopmenttasksandcontinuouslyimprovetheresultingsystem.

Evaluatingageneratedharnessisharderthanevaluatingageneratedanswer.Aharnesscanoverfittothemodelthatwroteit,memorizedevelopmentexamples,improveonecapabilitywhilesilentlyregressinganother,orimprovethefeedback-setscorethroughbenchmark-specificchangesthatdonottransfertonewtasks.Wethereforeevaluatealongtwoaxes.Capabilitymeasureswhethertheharnessworks:werunitonheld-outdownstreamtasksandreporttask-levelsuccessrates.Efficiencymeasureshowmanyexecutor-modeltokensthefrozenharnessconsumeswhendeployedtosolvedownstreamtasks.

3

Ourfindingsfollowthetwostagesofharnessdevelopment:

HarnessCreation.Currentmodelscanconstructrunnableharnessesfromaweakseed,butthegapfrommaturehuman-engineeredsystemsvariessubstantiallybyharnesstype.Wheneachharnessrunswiththemodelthatbuiltit,model-builtharnessesmatchthereferenceonshort-formwritingandexceeditonmachine-learningexperimentation.Thegapislargestforsearchandresearchharnesses,whichrequirelong-horizoninformationseeking,andremainssubstantialforcodeharnesses,whichmustcoordinaterepositoryinspection,editing,andverificationovermanyturns.Harnessesbuiltbydifferentcreatormodelsalsodiffersubstantiallynotonlyindownstreamtaskperformance,butalsointhenumberofexecutortokenstheyconsume.Higherexecutioncostdoesnotreliablyproducebetterresults,soharnessqualitymustbeassessedthroughbothcapabilityandefficiency.

HarnessEvolution.Currentmodelscanusedownstreamexecutionfeedbacktoimprovetheirownharnesses,butreliableevolutionremainsdifficult.Performanceoftenrisesandfallsacrosssuccessiverevisions,andgainsobservedduringdevelopmentbecomesmallerandlessconsistentonunseentasks.Theoutcomealsodependsstronglyonthemodelthatrunstheharness:changingthisruntimemodelaltersboththestartingperformanceandwhethersubsequentrevisionshelp.Theseresultsshowthatmodelscanmakeusefullocalimprovements,whilerobustevolutionacrossunseentasksandruntimemodelsremainsanopenchallenge.

2Background

Mostagentbenchmarksbeginaftertheproblemhasalreadybeenmadeexecutable:thetaskisspecified,therewardorjudgeisdefined,andtheexecutionscaffoldisfixed.Thissettingisnecessaryforcontrolledcomparison,butithidestheworkthatdominatesrealdeployment.Inindustrythatworkisspreadacrossseveralroles—solutionsarchitects,appliedandplatformengineers—butitsmostvisiblerecentcrystallizationistheforward-deployedengineer(FDE),atitlepopularizedbyPalantirandsinceadoptedbyfrontier-modelcompanies

[36

,

38

].AnFDEisembeddedatthecustomersiteafterasystemisadoptedandturnsageneral-purposemodelintosomethingthatrunsagainstthatcustomer’sdataformats,workflows,andcomplianceconstraints—forexample,rewritinganingestionpathbecauselogsmayonlyberetainedforafixedperiod,orlocalizingafailureinacross-jurisdictioncontractpipeline.Therole’ssuccesscriterionisnotademoorabenchmarkscorebutwhetherthedeployedsystemisgenuinelyused,keepsworking,andimproves;itsfailuresarefoldedbackintotheproductasfixesandfeaturerequests[

36

].TherapidgrowthofFDEhiringacrossfrontier-modelanddata-platformcompanies[

44

]reflectsasimplefact:acapablemodelisnotyetaworkingsystem

[30

],andtodaythegapisclosedbyhumanengineers.

Viewedfromthemodel’sside,FDEworksuppliesthreepiecesofstructurethatbenchmarkdesignersnormallypresuppose.First,thetargetisvague:aninformalbusinessintentmustbetranslatedintoconcreteobjectives,constraints,andsuccesscriteria.Second,thefeedbacksignalisabsentorunreliable:tests,judges,traces,orotherself-evaluationmustbeconstructedbeforeanyonecantellwhetherthesystemisimproving—“compliant”onlybecomescheckableoncesomeoneencodeswhatcompliancemeanshere.Third,theexecutionsystemdoesnotexistinausableform:thetools,contextmanagement,state,lifecyclelogic,andverificationinterfacethroughwhichfuturetaskswillrunmustbebuilt,adapted,andthenmaintained—anFDEstayswiththesystemasrequirementsshift,ratherthandeliveringonceandleaving.

Thispaperfocusesonthethirdlayer.Inatypicalcontrolledagentevaluation,researchersselectanagentconfigurationandreporttaskcompletionunderthatconfiguration.Theharnessisthereforeusuallypartoftheevaluationsetupratherthantheobjectbeingdeveloped.HARNEssDEvinsteadaskswhetherlanguagemodelscancreatethisexecutionscaffoldfromaweakstartingpointandthenimproveitusingfeedbackwhilepreservingconstraintcomplianceandheld-outperformance.WhereasAspIREstudieshowbroaddeploymentneedsbecomecapabilitygrowthandS3Gymstudieswhetherinteractionexperiencecanbejudgedandreused,HARNEssDEvisolateshowmodelsbuildandmaintainthesystemsthatcarrythem.

4

3Benchmark

3.1Overview

HarnessDevevaluatestheexecutionsystemthatamodeldevelops,ratherthantheansweritproducesforasingletask.Thesubmittedartifactisarunnableharnessthatisfrozenandthenreusedacrossdownstreamtasks.AcreatorLLMLCworksinsideadevelopmentenvironmentDtoproducearunnableharnessH.ThedevelopmentsignaldiffersbysettingandisdefinedinTable

1

.Afterdevelopment,Hisfrozen.AnexecutorLLMLEthenrunsinsideitonadownstreamtaskx,andevaluatorJscorestheresultingoutputy:

(LC,D)→H,(H,LE,x)→score.(1)

Thus,DisusedtobuildH,whereasLEisusedonlyafterHisfrozen.Inimplementationterms,Hcontainstheexecutionloop,tools,contextmanagement,persistentstate,lifecyclecontrol,andverification;wedescribethesecomponentsinwordsratherthanassigningeachanothersymbol.

Thebenchmarkstudiestwostagesofharnessdevelopment.

RQ1—Creation.Canamodelbuildaneffectiveharnessfromaweakbutrunnableseed?Thecreatormustturnataskspecificationandafewdevelopmentcasesintoinfrastructurethatgeneralizestounseentasks.

RQ2—Evolution.Canamodelimproveanexistingharnesswhilepreservingbehaviorthatalreadyworks?ThecreatorevolvesitsownCreationharnessfromdownstreamexecutionfeedback.Weadditionallyanalyzetheresultingartifactsandtrajectories,includingeditstatistics,feedbackresponse,held-outgeneralization,andtransferacrossexecutors.

3.2Developmentsettings

Allsettingsprovideamutabledevelopmentworkspace,buttheydifferinthestartingharnessandthesignalavailabletothecreator.Table

1

givesthecentraldistinction.

Setting

Startsfrom

Developmentsignal

Output

Creation

WeakseedHseed

Specificationand1–3developmentcases

FinalharnessH

Evolution

Creator’sRQ1H0

Resultsfromadesignatedfeedbackset

Frozenpairedcandidatesandacreator-declaredfinalharness

Table1Harness-developmentsettings.Creationbuildsanewharness;Evolutionimprovesthatharnessfromdownstreamexecutionfeedback.

WeakseedHseed.Creationshouldmeasurewhetheramodelcandesignanexecutionsystem,notwhetheritcanreproducebenchmarkboilerplate.EverycreatorthereforereceivesthesameHseed:arunnablecompatibilitylayer,notatask-solvingagent.Itparsestaskandmodelconfiguration,exposespermittedlow-leveltools,andwritestherequiredresults,trajectories,logs,andtaskartifacts.Itstoolsarepassiveandactonlywhentheharnesscallsthem.

Theseedhasnoagentloop,taskdecomposition,toolpolicy,contextmanagement,persistenttaskstate,verifier,retryorrecoverylogic,orstoppingrule.Itmayissueoneconnectivityprobe,butitdoesnotattemptthetask.Unmodified,itproducesanemptyorpartialartifactandscoreszerooneverydownstreambenchmark.AnynonzeroCreationscoremustthereforecomefromexecutionlogicaddedbythecreator.Thisboundarymattersbecauserealscaffoldscombinemanycontrolprimitives,andtheircompositionaffectstaskperformance

[31

,

39

,

40

].

Thisdesignavoidstwoextremes.Anemptyrepositorywouldmixharnessdesignwithcommand-lineandfile-formatsetup;amatureagentwouldgiveawaytheplanningandverificationstructurebeingtested.Hseedremovesthesetupburdenwithoutprovidingasolutionpolicy.Figure

2

showstheseedanditsdevelopment

5

Publicdevelopmentloop

creatormayreviseHusingvisiblefeedback

EditH

creatoragent

Rundevcases

publictasksonly

Inspectfeedback

logs+scores

HIDDENEVALUATIONWITHHELDtasks•answers•officialscores

Unifiedinputcontract

identicalinterfaceacrosscreators

Task

prompt+paths

LLMconfig

model+endpoint

Workspace

files+environment

Output

designateddirectory

Stableinterface;notask-solvingpolicy.

PASSIVEPRIMITIVESAVAILABLE,NOTORCHESTRATED

PathsFilesSearch

ProcessLLMgatewayArtifactI/O

CONTROLLAYERNOTPROVIDED

WeakseedHseedrunnable•auditable•policy-freeSINGLENON-ACTINGPASS

2

ParseOptionalWrite

taskprobeaudit

FIXEDCONTRACTWEAKSEEDPUBLICDEVELOPMENT

NoexecutionlooportoolpolicyNomanagedcontext

NodurablestateorrecoveryNoverifierorstoprule

1

Readinputs

partial

Figure2TheweakSeedHarnessanditsdevelopmentenvironment.Theseedfixesonlytheinputandauditenvelopesandexposespassive,unorchestratedprimitives;unmodified,itperformsnotaskwork.Developmentfeedbackispublic;hiddentasks,answers,andofficialscoresremaininaccessible.

environment;Figure

3

showsthecontrollayerthecreatormustimplementandthescorer-readableartifactsafinishedharnessdelivers.Appendix

C.1

givesitsimplementationskeleton.

Creation(RQ1).AlongwithHseed,thecreatorreceivesatask-familyspecification,toolandpermissionconstraints,ashortdesigntutorial,andonetothreedevelopmentcases.Itmayrevisetheharnessusingfeedbackfromthosecases,butitneverseesthehumanimplementationorthehiddenevaluationset.TheresultingharnessHisfrozenbeforeevaluation.

Evolution(RQ2).ThecreatorstartsfromitsownfrozenRQ1codeharnessH0.Duringdevelopment,itreceivesresultsfromafixed100-taskSWE-Profeedbacksetandall89Terminal-Benchtasks.The100SWE-Protasksareasubsetofthe731-instancepublicsplitusedinCreation,andthe630-instanceheld-outsplitofSection

4.3

isdrawnfromthesamesplit.Inthereportedprotocol,thecontrollerfirstevaluatesH0onbothbenchmarks.Eachofficialpost-H0candidateisthenfrozenandsubmittedasapair:onecomplete100-taskSWE-Proevaluationandonecomplete89-taskTerminal-Benchevaluationofthesamecommit.Acandidateenterstheofficialtrajectoryonlyafterbothlegssettle.Same-commitinfrastructurerepairsaremerged;probes,partiallegs,stoppedruns,andinvalidinstancesareexcluded.

Thecontrollerprovidesabudgetoftenpost-H0full-evaluationpairs.Betweentwochargedpairs,thecreator

CreatedrunnableharnessH

CONTROLPLANEIMPLEMENTEDBYTHECREATOR

ETC

LoopToolsContext

Unifiedauditcontract

result.jsontrajectory.jsonl

response.md

runtimelogs

sharedstatus,metrics,actions,andobservations

oneintegratedexecutablesystem

SLV

StateLifecycleVerify

Domainfinalizer

mapstherunintothedomain’s

authoritativescorer-readableartifact

thesharedcontractdoesnotforceonefileformat

Scorer-readableartifacts

authoritativeoutputvariesbydomain

Code

repositorystate+patch

Data/MLE

submission+metrics

Writing

finaluser-facingprose

Search

answer+citedevidence

C

CreatorLLM

implementsthehar-nesscontrollayer

notthedown-

streamtaskanswer

oneoutputbus;nocross-domainfile-formatassumption

Figure3Fromcontrollayertoscorableartifacts.Thecreatormustimplementthesixcontrolmodules(E/T/C/S/L/V,coloredasinFigure

1

).Afinishedharnessreportsthroughunifiedauditableoutputsanddeliversadomain-specific,scorer-readablefinalartifact.

6

mayuseatmosttwofixed-subsetprobes,eachcoveringthesamefirstfivetasksfrombothbenchmarks.Proberesultsarediagnosticandneverbecomeofficialscores.Thecreatorterminatesbydeclaringanon-H0committhathasacompleteofficialpair.

BothbenchmarksshownduringEvolutionarefeedback-bearingdevelopmentsets,soin-trajectoryscoresmeasureonlineadaptationandversionselection.Generalizationismeasuredseparatelyafterfreezing:everyofficialversionisadditionallyevaluatedon630SWE-Proinstancesdisjointfromthefeedbackset,andthesescoresarenevershowntothecreator.Throughoutthispaper,held-outmeanswithheldfromthecreator’sdevelopmentloop;Section

4.3

givesthefullsetting.

3.3Domainsanddownstreambenchmarks

Creationcoversfourdomainsandfivedownstreambenchmarks(Table

2

);Evolutioncurrentlyfocusesoncodeharnesses.Together,thesuitescontain2,207uniquedownstreaminstances.TheEvolutionfeedbacktaskscomefromthesamebenchmarksuitesandarethereforenotcountedagain.

Domain

Benchmark

Tasks

Primarymetric

Code

SWE-benchPro

[12

],publicsplit(abbreviatedSWE-Probelow)

731

Tasksuccess

Code

Terminal-Bench2.1

[45

]

89

Tasksuccess

Dataanalysis

MLE-bench

[7

]

75

Medalscore

Writing

EQ-Bench3

[37

]

46

Rubricscore

Research

BrowseComp

[52

]

1,266

Accuracy

Table2DownstreamevaluationcoverageforCreation.Creationusesall731instancesoftheSWE-benchPropublicsplit.Evolutiondrawsits100-taskSWE-Profeedbacksetandits630-instanceheld-outsplitfromthatsamepublicsplit,andusesall89Terminal-Benchtasksasfeedback.

Matureopen-sourcesystemsdefinethecapabilitysurfaceineachdomain.Dependingonavailability,asystemmayserveasahuman-engineeredreferenceorprovideadevelopmentenvironment.Theserolesareassignedseparately;inclusiondoesnotimplythatasystemservesboth.Appendix

A

liststhecandidatesystems,whilethebenchmarkreleasefixestheirroles,versions,andlicenses.

3.4Evaluationprotocol

Everyscoreisproducedbyafrozenharnessinastandardizedruntime.TheexecutorLLMLEandevaluatorJremainfixedwithineachcomparison,soscorechangesreflectchangestotheharness.Developmentandevaluationarealsoseparated:hiddenscoresarenotreturnedinCreation,andEvolutionexposesonlyitsdesignatedfeedbacksetduringdevelopment.

Creation.Wecompareacreatedharnesswiththecommonseedand,whereavailable,amaturehuman-engineeredharness.SELF-EvALsetsLE=LCandmeasuresthecompletecreator–harnesssystem.UNIFIED-EvALrunseverygeneratedharnesswiththesamefixedLE,makingharnessesdirectlycomparable.

Evolution.Evolutioncandidatesareevaluatedonthedesignatedfeedbackbenchmarksduringdevelopment.Onlycompletetwo-benchmarkpairsentertheofficialtrajectory,andthecreatorselectsafinalpairedcandidate.Afteralltrajectoriesend,everyofficialversionisadditionallyevaluatedonadisjointheld-outsetthatisnevershowntothecreator,soadaptationtoobservedfeedbackandheld-outgeneralizationarereportedseparately.Appendix

D

detailstheevaluationsettingsandmodelroles.

Constraintcompliance.Thecreator-visiblespecificationstateswhatasubmittedharnessmaynotdo:hard-codeinstance-specificsolutions,derivepatchesfromtaskidentifiers,file-nameallowlists,orknownanswers,consulthiddentests,hiddenanswers,hiddenpatches,privatescorerinternals,orofficialevaluationfeedback,orreplacetheprovidedprovider-neutralruntimeinterfacewithitsownLLMaccesspath.Twoproperties

7

maketheseconstraintscheckableratherthanadvisory.First,thescorepathisisolatedfromtheharness:aharness’sself-reportedstatusisneverascoringinput,SWE-Procreditcomesonlyfromtherealrepositorydiffleftinthetaskworkdir,andTerminal-Benchcreditonlyfromfinalenvironmentstate,sonoharnesscanearnscorebyassertingsuccess.Second,everyrunretainsitstrajectory,result,andmetricartifactsalongsidethefrozenharnesssource,whichsupportsapost-hocauditofthedeliveredcodeandofwhatthatcodeactuallyexecuted.Weauditedthedeliveredharnesssourceandtherecordedexecutionartifactsofeveryrunreportedinthispaper,andreporttheoutcomeasanullresult:noharnessobtainedscorethroughaprohibitedroute,andnorunisexcludedonthesegrounds.

3.5Metrics

Foreachfrozenharness,wereporttwoquantities:downstreamtaskperformanceunderthebenchmark’snativemetricandtheexecutor-modeltokensconsumedduringevaluation.Executioncostisreportedasboththetotalandthemeanpertask;tokensusedbythecreatortobuildormodifytheharnessareexcluded.Creationcompareseachharnesswiththeweakseedandavailablehuman-engineeredreferences.EvolutionreportstheperformancechangefromitsinitialharnessH0underthesameexecutorandscorer.

4Experiments

WeinstantiatethebenchmarkdefinedinSection

3

andreportresultsforCreation,Evolution,andcross-modelbehavior.Thebehavioralanalysiscomparesresourceuse,responsetofeedback,andtransferacrossexecutors.

4.1Experimentalsetup

WeevaluatesixcreatorLLMsLC:Opus4.8

[4

],GPT-5.5

[33

],Gemini3.1Pro

[13

],DeepSeekV4Pro

[11

],Qwen3.7Max

[1

],andSeed2.0Pro

[6

].ModelsrunthroughtheirofficialAPIsorOpenRouter[

35

].WeuseClaudeCode2.1.177asthedevelopmentenvironmentD,exceptthatGPT-5.5usesCodex0.144.3.Appendix

B

givesthemodel,decoding,anddevelopment-environmentconfiguration,thedownstreamexecutionresourcesandtimelimits,andthehuman-referencesources.

WefollowthedevelopmentandevaluationprotocolsinSections

3.2

–

3.4

.ForRQ1,weindependentlycreateandevaluatethreeharnessesforeachcreator–benchmarkpairandreportavg@3.UnderSELF-EvAL,thecreatoralsoservesasexecutor;underUNIFIED-EvAL,everyharnessusesGemini3.1Pro,whichisolatesexecutorcompatibility.Human-referenceresultsareverifiedpublicsystemresultsratherthanpairedcontrolsunderoneexecutor;theirsourcesarelistedinAppendix

B.2

.

4.2HarnessCreation

RQ1askswhetheramodelcanturntherunnableweakseedintoaneffectiveharnessforataskfamily.Wecomparethegeneratedharnesseswiththezero-scoringseedandwithverifiedhuman-engineeredsystems,reportingbothdownstreamperformanceandexecutiontokens.

Overallfindings.Creationqualityvariessubstantiallyacrosstaskfamilies(Tables

3

–

4

andFigure

4

).UnderSELF-EvAL,Opus4.8hasthehighestoverallscore(67.8),butremainsbelowthehuman-engineeredreference(86.2).