2025QECon全球软件质量效能大会：测试效能革新之路 - ebay支付风控新一代低成本行为驱动测试平台演进

The

EvolutionofeBay

Payment

Riskteam’s

Next-Generation,

Low-Cost

Behavior-Driven

Testing

PlatformShi,

Kai｜eBay

MemberofTechnicalStaffJoined

eBay

‘

s

Payment

Risk

team

in

2019

,

currently

responsible

for

development

work

in

the

risk

domain

.

Possesses

over

10years

of

experience

inthefields

of

quality

and

efficiency,with

extensive

design

and

practical

experiencein

automationtesting

and

efficiency

improvementtools

and

platforms.Shi

，

KaieBay

MemberofTechnical

StaffAgenda01

Behavior-DrivenTestAutomationwithCucumber02

Reducecosts

byaddressing

pain

points.03

Test

reportautomaticanalysisAgentPART

01Behavior-DrivenTestAutomationwithCucumber01Low

pass

rateThetestsuite

comprisesnumerouscases

，yetthepass

rate

remainssuboptimal.Thesechallengescreateaviciouscycle,

leading

to

lower

pass

rates

andlimitedeffectivenessofautomatedtesting

projects.02Challengingto

Diagnose

and

ResolveTestcode

is

oftenwritten

inascripted

manner,

resultingin

longand

chaoticstructuresthatare

difficultto

maintain.

Challengesin

LegacyTestAutomationThetest

intent

is

unclear,andoutputs

are

poorlystructured,

lacking

keyinformation

like

requests,responses,andserver

logs.03TestCode

Lacks

MaintainabilityAsset

LiabilityInthe

longterm,thedecisivefactor

indeterminingwhether

thetest

automation

isanassetora

liability

is

cost,

which

includes

the

cost

ofwriting,

execution,

result

analysis,and

maintenance.What

is

more

important

isthe

balanceofthesefourcosts.Highfalse

negative

rateHighfalse

positive

rateFailedcases

aredifficulttoattributeDisorganizedcodehinders

maintenance.Slowexecution

speedLowfalse

negative

rateLowfalse

positive

rateFailedcases

areeasyto

attributeEasy

maintenanceHighexecution

speed Is

the

test

automation

an

asset

or

a

liability?•

Clear

projectstructure•

Clear

test

intent•

Modularizationoftestingsteps•

Reusabletest

stepUser-FriendlyModularization•

High

pass

rate•

Low

false

positive

rate•High

coverage

rate•

Ready

out

of

the

box•

Easilyextensible•

Fastexecution

speed•

Sufficientandeffective

informationfor

rapidattributionandfixing

Thales

Project–Objectivesfor

RefactoringtheTest

ProjectStableand

ReliableTeststep

definitionTestcasesoutlinespecific

behaviors

and

areeasily

understood

byall,enablingevery

project

memberto

participate

in

reviews.

The

Beginningof

Refactoring:Structuringwith

Behavior-DrivenTest01A

BDDTestCaseComprises

SeveralTestSteps,

EachStep

NaturallyServingas

a

Separate

ModuleTestCasesAre

Readable

Like

Documents,FacilitatingCommunicationAcrossTimeand

Positions02

Facilitates

Communication

Testcase

definitionImproveCoverage

RateNatural

Modularization03Cucumber

FrameworkThecomplexityof

secondary

development

iscomparatively

higher.

However,

itsflexibility

allows

ittoadapt

to

eBay'stechnologystackand

offers

a

higher

potentialceiling.

ConsiderationsforChoosingCucumberKarate

FrameworkProvides

a

DSL

but

is

notpure

BDD,withshortcomings

in

large-scale

use.Behave

FrameworkPython-Based,

Incompatible

witheBay'sJava-BasedSDKs2.The

initialversiononlysupportsgeneral

RESTfulAPItesting,

and

it

primarily

reliesonthefollowingthreecomponents:•Cucumber

is

usedtoorganizeautomatedtestcases.•RestAssuredservesasthecomponent

for

sending

and

validatingtest

requests.•

Junit5actsastheorchestrator

for

test

execution.1.The

issueswithautomatedtestingarewidespread;

we

need

tocreatea

newgenerationof

universaltesting

framework

to

helpteams

improveoveralldevelopmentquality

andefficiency.

Thales

Project：Agenericautomationtestingframeworkbased

on

Cucumber3.Our

principle

indevelopingtheframework

isto

connect

excellentframeworks

ratherthanencapsulatethem.InitialVersionofThales

CorePART

02Reduce

Costs

by

Addressing

Pain

Points.

TheCommon

Pain

Points

InTestAutomation

Pain

points:•Excessivetestfailures

inthetest

report

result

insignificant

timeconsumption.•

Thestructureof

the

testreport

isdisorganized,

making

itdifficultto

understand.•Insufficient

information

makes

itdifficultto

identifythe

root

cause.Pain

points:•Insufficienttest

datafor

specificscenarios.•

Significanttime

requiredfor

business

understandingand

datacreation.•

Testdata

may

become

outdated.Pain

points:•

Testingenvironment

instability

leadstovolatile

results.•Executingthefulltest

suite

is

time-consumingand

mayrequireseveral

hours.1.eBay'sdatacreation

platform

providesfoundational

datacreationacrossvariousdomains,

making

it

easy

tocreatetestdatathrough

APIs

and

SDKs2.

However,generaldatacreation

cannot

meettherequirementsofsomespecialtest

cases.The

built-intestdata

preparationstepscombine

DCP

and

domain

knowledgetoenablethecreationof

complex

test

data

witha

single

declaration.3.We

providethesesteps

intheformof

aJAR

package,

whichcan

be

integrated

intoyourowntesting

projects

to

usethesesteps.Additionally,when

usingthem,the

IDEoffersauto-suggestionand

step

completionfeatures. The

built-in

steps

of

test

data

creationGiven

A

user

has

been

blocked

more

than

10times

in

the

past

30

daysxProblems•

Dependentservicecould

be

unstable

whentest

isexecuting,

leadingtounreliabletest

results.•Mockupsarea

common

solution,

buttheycanobscurecode

issues,

especially

ifthemockserverfailsto

detect

bad

requestsand

returnsa

correct

response.•

Fixed

mockupapiscan

not

handle

diversitytestscenarios.Solutions•

Dynamicallycreate

mockapiwith

natural

languageto

mock

unstable

dependencies.•

Mock

requestcheck.Check

request

in

mockserver,

returnerrorresponse

ifrequest

is

notcorrect,

preventing

hiding

issue.•

Testcase

level

mockup

isolation,eachtestcase

has

its

own

mock

api,

theywon’t

inferenceeachother.

Dynamic

Mockingon

DemandTospeed

upthe

process

oftestexecution,Thalesexecutetest

casesin

parallel.•Defaultthreadcount

is

10,which

is

configurable.•

Forsafeexecution,Thales

provides

atag“@sync”that

makes

particulartestcasesto

executesynchronously.•

Onaverage,

parallelexecution

in

safe

modecouldsave

50%

ofthe

execution

time

ConcurrentexecutionofTest

cases•

Whenateststep

isdesignated

as

an

asynchronousstep,

theframeworkwillexecute

it

ina

non-blocking

manner•

Oncetheasynchronousstep

iscompleted,the

result

ofthe

correspondingtestcasewillbe

updated

accordingly.•Inthecontextofa

large-scaletest

suite,this

feature

has

the

potentialto

reduceexecutiontime

byapproximately60%.•

Thetest

reportwill

reflecttheasynchronousstep

as

though

itwereexecutedsynchronously.•Insituationswherethesafety

of

atest

case

for

parallel

execution

is

uncertain,

it

isadvisableto

implementasynchronousteststepstoensureefficient

execution.•

Evenwhentestcasesare

executed

concurrently,theincorporationofasynchronousstepscanfurther

expedite

the

execution

process.•

Somestepsaretimeconsuming,

such

asverify

the

result

from

a

DB.

Insuch

instances,employingasynchronoustest

steps

caneffectivelyenhancethespeed

of

execution.

Asynchronousexecutionoftest

stepsWhyHow•

Thefailedtestcases

are

notexecuted

immediatelyfollowing

theirfailure.•Uponcompletion

ofthe

previous

testcycle,thefailedtest

cases

are

compiled

intoa

newtemporarytest

suite.•

The

retry

mechanism

istheninitiated,andthetestingconcludeseitherwhen

alltestcasesaresuccessful

or

whenthe

maximum

numberof

retryiterations

is

reached.Whysteplevel

retry

is

not

enoughConsiderascenariowhere

a

retry

mechanism

isemployedto

retrievedata.

Ifthedataabsence

occurs

at

an

indeterminate

internal

node,

particularlyanasynchronousone,step-level

retries

may

proveineffective

(as

illustrated

intheexample

below).Suchissuesareoftendifficultto

diagnose,

and

manualtestexecution

may

be

requiredtoachievesuccess.Overtime,

users

may

becomeaccustomedto

perceivingtestfailures

asa

normaloccurrence,

leadingtoagradual

erosion

ofthe

reliabilityoftest

results.

Retryoffailedtest

casesHowto

retryfailedtest

casesAcomprehensivetest

report

isthe

keytothe

successofa

testing

project.1.In

the

realm

of

automated

testing,

it

is

inevitable

thatcertaintest

caseswill

fail.2.Thesefailures

necessitateathorough

investigation

toascertaintheir

root

causes3.However,

whentest

reports

are

deficient

in

detailed

information,the

processofdiagnosingthese

issuesbecomesexceedinglydifficultandtime-consuming4.Atestingframeworkthatdemands

excessive

timefortroubleshootingwillwitnessagradual

decline

in

itssuccess

rate,

ultimately

rendering

it

ineffectiveandobsolete.5.It

automatically

gathers

code

logs

andframework

inputs/outputs

intostructuredtest

reports,showcasingtestobjectivesandexecution

details.

Informativetest

reportAcomprehensivetest

report

isthe

keytothe

successofa

testing

project.1.In

the

realm

of

automated

testing,

it

is

inevitable

thatcertaintest

caseswill

fail.2.Thesefailures

necessitateathorough

investigation

toascertaintheir

root

causes3.However,

whentest

reports

are

deficient

in

detailed

information,the

processofdiagnosingthese

issuesbecomesexceedinglydifficultandtime-consuming4.Atestingframeworkthatdemands

excessive

timefortroubleshootingwillwitnessagradual

decline

in

itssuccess

rate,

ultimately

rendering

it

ineffectiveandobsolete.5.It

automatically

gathers

code

logs

andframework

inputs/outputs

intostructuredtest

reports,

Informativetest

reportshowcasingtestobjectivesandexecution

details.Acomprehensivetest

report

isthe

keytothe

successofa

testing

project.1.In

the

realm

of

automated

testing,

it

is

inevitable

thatcertaintest

caseswill

fail.2.Thesefailures

necessitateathorough

investigation

toascertaintheir

root

causes3.However,

whentest

reports

are

deficient

in

detailed

information,the

processofdiagnosingthese

issuesbecomesexceedinglydifficultandtime-consuming4.Atestingframeworkthatdemands

excessive

timefortroubleshootingwillwitnessagradual

decline

in

itssuccess

rate,

ultimately

rendering

it

ineffectiveandobsolete.5.It

automatically

gathers

code

logs

andframework

inputs/outputs

intostructuredtest

reports,

Informativetest

reportshowcasingtestobjectivesandexecution

details.Acomprehensivetest

report

isthe

keytothe

successofa

testing

project.1.In

the

realm

of

automated

testing,

it

is

inevitable

thatcertaintest

caseswill

fail.2.Thesefailures

necessitateathorough

investigation

toascertaintheir

root

causes3.However,

whentest

reports

are

deficient

in

detailed

information,the

processofdiagnosingthese

issuesbecomesexceedinglydifficultandtime-consuming4.Atestingframeworkthatdemands

excessive

timefortroubleshootingwillwitnessagradual

decline

in

itssuccess

rate,

ultimately

rendering

it

ineffectiveandobsolete.5.It

automatically

gathers

code

logs

andframework

inputs/outputs

intostructuredtest

reports,

Informativetest

reportshowcasingtestobjectivesandexecution

details.•Thales

core

exists

in

theform

ofajarpackage.Thisjarpackageisavailableout

ofthe

box,andeachbusinessteamcaneasily

createtheir

owntest

projectsbasedonThales

core.•

Userscanusethe

business-relatedsteps

providedbyThales

core•

Userscanusethe

basiccomponents

providedbyThalescoretoimplementtheirown

testing

steps•

Userscanalsousethe

capabilities

ofthe

open-sourceframeworktobuildnewfeatures,

andthesefeaturescanbecontributedtoThales

core

TheoverviewofThales

Core1.The

platformcollectsdatafromtheexecution

oftest

cases•Contentsofthetest

cases•Execution

duration•

Pass

rate•

Server-sideerror

logsofthetest

casesChoosing

nottoact

is

moreimportantthanchoosingto

act.2.Collecting

key

metricscan

help

us

understandthe

performanceofThalesCoreand

promptly

identify

userswho

needassistance.

Thales

platform3.The

platformdoes

not

provideagraphicaltestcase

authoring

page.•It

isprudenttoacknowledgethat

the

test

cases

we

developmaynot

be

exhaustive

and

might

notidentifyallpotential

code

defects.•

TheThalesplatformcollectsallerror

logs

thrown

bybackendservicesduringtheexecutionoftestcases.•

Theanomalydetectionreportanalyzes

the

errorlogsofthecurrenttestexecution

basedonhistoricalerrortypesanderrorquantities,

providinganalysisandalerts.•It

helpsfindpotentialcode

issues

before

rollout.

Anomalydetectionreportofserver-sideerrorsWhy•It

isprudenttoacknowledgethat

the

test

cases

we

developmaynot

be

exhaustive

and

might

notidentifyallpotential

code

defects.•

TheThalesplatformcollectsallerror

logs

thrown

bybackendservicesduringtheexecutionoftestcases.•

Theanomalydetectionreportanalyzes

the

errorlogsofthecurrenttestexecutionbased

on

historicalerrortypesanderrorquantities,providing

analysis

andalerts.•It

helpsfindpotentialcode

issues

before

rollout.

Anomalydetectionreportofserver-sideerrors•It

isprudenttoacknowledgethat

the

test

cases

we

developmaynot

be

exhaustive

and

might

notidentifyallpotential

code

defects.•

TheThalesplatformcollectsallerror

logs

thrown

bybackendservicesduringtheexecutionoftestcases.•

Theanomalydetectionreportanalyzes

the

errorlogsofthecurrenttestexecutionbased

on

historicalerrortypesanderrorquantities,providing

analysis

andalerts.•It

helpsfindpotentialcode

issues

before

rollout.

Anomalydetectionreportofserver-sideerrorsPART

03AIagentforanalyzing

test

reportsAdvantages:•

Large

language

modelsandtestcaseswritten

in

natural

languageare

inherentlycompatible,complementingeachothereffectively.•

Implementingend-to-endautomationfrom

PRDtotestcaseexecution

streamlinesthe

processandsignificantly

reducesthetime

requiredfortestdevelopment.Disadvantages:•The

hallucinations

of

large

language

models

can

rendertestcases

unusable.•

Highlydependentonthe

PRD

(Product

Requirements

Document).-Objective•Uncontrollable

dependencies

-

SubjectiveAdvantages:•

Large

language

modelsexcelatattribution

basedonexisting

information.TheThales

platform

hasalreadygathered

mostofthe

information

relatedtotestcaseexecution.•

Automatedtesting

is

performedwitheach

deployment

release,andeach

releaserequiresanexaminationofthe

test

report,

resulting

ina

highfrequencyof

use.Disadvantages:The

hallucinationsof

large

language

models

result

in

inaccurateanalysis.ChooseTestcaseautomaticgenerationorTest

report

automatic

analysis?

TwoAI-poweredfeaturesweaimto

develop.Test

reportautomaticanalysisTestcaseautomatic

generationTheOverview:1.The

automatedtesting

project

utilizingThales

core

is

executedonJenkins,duringwhichfailedtest

cases

are

senttotheThales

platform.2.TheThales

platform

is

usedto

further

enrich

the

context

informationofthefailedtestcases.3.TheThales

agent

analyzesthetest

results

based

on

the

LLM

platform(Phoenix)and

RAG.

HowTest

ReportAutomaticAnalysisAgentworksThe

process:1.Duringexecution,Thalestransmitsallcase

information

totheThales

platfor