【世界银行】验证教学观察工具：基于内容的小学游戏与教学方法

DisclosureAuthorizedPublicDisclosureAuthorizedPublicDisclosureAuthorized

PublicDisclosureAuthorizedPublic

EDUCATION

WORKINGPAPER

No.10|February2024

ValidatingTeachingObservationTools:

AContent-BasedApproachforPLAYandTeachPrimary

DiegoLunaBazalduaandEmmaCarter

Educationoperations

SUPPORTHUB

THEWORLDBANK

IBRD.lDAIWORLDBANKGROUP

©2025InternationalBankforReconstructionandDevelopment/TheWorldBank

1818HStreetNW

WashingtonDC20433

Telephone:202-473-1000

Email:AskEd@

Internet:

/en/topic/education

ThisworkisaproductofthestaffofTheWorldBankwithexternalcontributions.Thefindings,interpretations,andconclusionsexpressedinthisworkdonotnecessarilyreflecttheviewsofTheWorldBank,itsBoardof

ExecutiveDirectors,orthegovernmentstheyrepresent.

TheWorldBankdoesnotguaranteetheaccuracy,completeness,orcurrencyofthedataincludedinthisworkanddoesnotassumeresponsibilityforanyerrors,omissions,ordiscrepanciesintheinformation,orliabilitywith

respecttotheuseoforfailuretousetheinformation,methods,processes,orconclusionssetforth.The

boundaries,colors,denominations,links/footnotesandotherinformationshowninthisworkdonotimplyanyjudgmentonthepartofTheWorldBankconcerningthelegalstatusofanyterritoryortheendorsementor

acceptanceofsuchboundaries.ThecitationofworksauthoredbyothersdoesnotmeantheWorldBankendorsestheviewsexpressedbythoseauthorsorthecontentoftheirworks.

NothinghereinshallconstituteorbeconstruedorconsideredtobealimitationuponorwaiveroftheprivilegesandimmunitiesofTheWorldBank,allofwhicharespecificallyreserved.

RightsandPermissions

Thematerialinthisworkissubjecttocopyright.BecauseTheWorldBankencouragesdisseminationofitsknowledge,thisworkmaybereproduced,inwholeorinpart,fornoncommercialpurposesaslongasfullattributiontothisworkisgiven.

Anyqueriesonrightsandlicenses,includingsubsidiaryrights,shouldbeaddressedtoWorldBankPublications,TheWorldBankGroup,1818HStreetNW,Washington,DC20433,USA;fax:202-522-2625;e-mail:

pubrights@.

Coverdesign:MarianneSiblini

ValidatingTeachingObservationTools:AContent-BasedApproachforPLAYandTeachPrimary

DiegoLunaBazalduaandEmmaCarter

TableofContents

Abstract 3

Keywords 3

Acknowledgements 4

ExecutiveSummary 5

1.Studyobjective 6

2.Descriptionofobservationtools 6

3.Method 10

4.Results 12

a.Contentvalidityresults:ContentmatchingofTeachPrimarybehaviorsandPLAYitems 12

b.Concurrentvalidityresults:RelationshipamongTeachPrimaryandPLAYscores 19

c.ConstructvalidityresultsforPLAY:InternalstructureforthePLAYitems 19

5.Conclusion 22

References 23

Annexes 25

AnnexA:PLAYConstructsofsupportforchildren’sengagementinlearning 25

AnnexB:TablecomparingthemainfeaturesoftheTeachPrimaryclassroomobservationtooland

PLAYtoolkit 30

AnnexC:Overviewofexperts 31

AnnexD:FullmappingresultsforPhase1 32

AnnexE:MappingresultsforPhase2 47

AnnexF:CorrelationofTeachPrimaryandPLAYscores 52

AnnexG:CorrelationofPLAYitems,item-totalcorrelationsandvisualrepresentationsofinternal

structure 53

AnnexH:ExploratoryFactorAnalysissolution 58

Abstract

Thisstudycomparestwotoolsusedtoobserveandassesshowteacherssupportstudent

learning:theLEGOFoundation’sEngagetool,formerlyknownasPLAY,andtheWorldBank’s

TeachPrimarytool.Thegoalwastoevaluatetheirsimilaritiesandoverlapintheirmeasurementofteachingpractices.Agroupofnineexpertsreviewedbothtoolsindetail,carefullymatching

itemsfromPLAYtothoseinTeachPrimary.Resultsindicatethataboutone-thirdoftheitems

closelymatched,showingthatPLAYisbuiltonasolidframeworktocaptureengagingteaching

practices.Anotherthirdoftheitemsdidnothaveaclearmatch,meaningthateachtool

separatelycapturesuniqueaspectsofteachingthattheotherdoesnot.Thestudyalsolookedathowwellthetwotools’scoresalignusingrealclassroomdatafromEthiopia,Peru,andSierra

Leone.Theresultsshowedasmallpositiverelationshipbetweenthetwotools,meaningtheymeasurerelatedbutdifferentaspectsofteaching.ThestudyalsofoundthatthewayPLAYis

structuredmayneedsomeadjustmentstobetterreflecttheskillsitisdesignedtomeasure.Bycombiningexpertopinionswithdataanalysis,thisstudyprovidesstrongevidencethatboth

toolsarevaluablebutservedifferentpurposes.Thefindingshighlighttheimportanceofusingmultipleapproachestoassessteachingquality.Thestudyalsooffersrecommendationsfor

improvingtoolimprovementandsuggestsnextstepsonhowclassroomobservationtoolscanhelpsupportimprovedteachingandlearningworldwide.

JEL:I25-EducationandEconomicDevelopment

Keywords

Classroomobservations,teachingquality,engagingteaching,socio-emotionallearning,teachingpractices,contentvalidity,constructvalidity.

Acknowledgements

ThisdocumentwasdevelopedbyDiegoLunaBazaldua(SeniorEducationSpecialist;HEDGE),

EmmaCarter(Consultant;HEDGE).TheteamworkedundertheoverallguidanceofHalilDundar(PracticeManager,HEDGE).LauraGregory(SeniorEducationSpecialist;HEDGE),SergioVenegas(Economist,HEDGE)andMaryamAkmal(Economist,HEDGE)providedoverallsupportinthe

differentstagesofthisstudy.

PeerreviewersincludeAliHasanAnsari(SeniorEconomist,HAWE3),HumaKidwai(Senior

EducationSpecialist,HAEE1),EzequielMolina(SeniorEconomist,HLCED),andLauriPynnonen(SeniorEducationSpecialist,HEAED).AdditionalvaluableinputswerereceivedfromAmer

HasanandHarshilKumarSahai,andothermembersoftheEducationGlobalPracticewho

participatedindiscussionsonthepaper.Thenextlistofinternationalexpertscontributedtocontentreviewsofthetools:AbbieRaikes,CarolinaMeloHurtado,CarolinaMoreiraVasquez,DawitTiruneh,ElizabethHentschel,JonathanSeiden,LenkaJanikBlaskova,RabeaMalik,andTadesseTeferi.

ThisworkissponsoredbytheFoundationalLearningCompact(FLC)TrustFund.

ExecutiveSummary

Engagingteachinghasemergedasakeyapproachtofosteringchildren’ssocio–emotionaland

cognitiveskills.Becauseoftheimportanceofembeddedengagingactivitiesintheteachingandlearningprocess,LEGOFoundationandtheWorldBankhavecollaboratedtoadvanceamoreevidence–basedapproachtoenhancingholisticteachingandlearning.Theseeffortsemphasizepracticesthatfocusonengagement–basedandsocio–emotionallearning.Thisreportpresentsfindingsfromasignificant

componentofthepartnership,focusingoncomparingtheLEGOFoundation'sEngage,formerlyknownPlayfulLearningAcrosstheYears(PLAY)

1

,classroomobservationtoolwiththeWorldBank’sTeach

Primaryclassroomobservationinstrument.Thegoalwastoassesstheextentofoverlapbetweenthesetools,bothqualitativelyandquantitatively,providingcontentandconcurrentvalidityevidenceforthenewlydesignedPLAYinstrument.Thisstudyalsoaimedtoenhanceunderstandingofthetechnical

propertiesofbothtools,emphasizingsynergiesanddistinctions.

Methodologically,thequalitativeaspectofthestudyfollowedtheDelphimethod,involvingnineexpertsfromdiverseinternationalbackgrounds.Theyconductedtwophasesofcontentreview.Phase1includedadetailedcontentcomparisonanditem–to–itemmatching,withrationalesprovidedformatchingdecisionsmade.Phase2focusedonreviewingambiguouscases—itemsthatdidnotclearlymatchonPhase1.Inaddition,quantitativeanalysisassessedthecorrelationbetweenTeachPrimaryandPLAYaggregatescorestoevaluateconcurrentvalidity,usingdatafromEthiopia,Peru,andSierraLeone.

KeyqualitativeresultsshowedthatoverathirdoftheitemsinthePLAYtoolstronglycorrespondedwithbehavioralitemsfromtheTeachPrimaryinstrument,providingevidenceoftheempiricalbasisoftheformer.Additionally,thestudyidentifiedasimilarproportionofitemswithminimalornocorrespondencetoTeachPrimarybehaviors,highlightingthePLAYtool'suniqueabilitytocaptureotheraspectsofteachingconducivetoqualityengagement–basedinstructionandlearning.Quantitativefindingsincludedatotalscorecorrelationof0.15betweentheTeachPrimaryandPLAYinstruments,supportingconcurrentvalidityandreinforcingthetools’measurementofseparatebutrelatedconstructsofqualityteacherpracticeandengagement–basedteachingandlearning,respectively.Takentogether,thesefindingsrevealedthatcombiningqualitativelearningsfromtheDelphimethodwithrobustquantitativevalidationresultscreatesacomprehensivevalidationframework,strengtheningconfidenceintheseclassroomobservationtoolstomeasurequalityengagement–basedandsocio–emotionallearning.

Inadditiontotheanalyticalfindings,valuablelessonswerelearnedfromtheapplicationoftheDelphitechnique,whichcouldimproveitseffectivenessforconsensus–buildinganddecision–makinginfutureresearch.Amongtheseweretheprovisionofsufficienttimefortaskstofacilitatedataaccuracyandtheimportanceofconsistentcommunicationwithexperts.Finally,clearincentivesalongwithstreamlinedprocessesforlabor–intensiveexerciseswererecommendedashelpfulstrategiesforenhancingparticipants’engagement.

1ThenamePLAYisusedthroughoutthispaperasitcorrespondstothenameusedinpubliclyavailablematerialsatthetimeofwriting.TheauthorsacknowledgethatthetoolisnowcalledEngage.

1.Studyobjective

TheLEGOFoundationandtheWorldBankhavecollaboratedtosupporteducationsystemsinadoptingamoreevidence-informedapproachtoimproveholisticteachingandlearning,particularlythroughengagement-basedandsocio-emotionalteachingpractices.Acrucialaspectofthisobjectiveisenhancingthemeasurementofthesepractices,especiallywithinlow-andmiddle-incomecountries(LMICs).OneactivityincludedinthiscollaborationinvolvedadetailedcontentcomparisonoftheLEGOFoundation'sPlayfulLearningAcrosstheYears(PLAY)andtheWorldBank'sTeachPrimaryclassroomobservationtools.Thiscomparisonaimedtounderstandtheextentofcontentandscoreoverlap,aswellastheirconnectionwithstudents'learningexperiencesandoutcomes.Additionally,theexercisefocusedonidentifyingimportantcontentdistinctionsbetweentheinstrumentsandhowtheymayprovidecomplementaryinformationtousers.

Thiscomparisonhasalsoservedasaformofcontentvalidityevidenceandfacilitatessubsequentquantitativeanalysisaimedatunderstandingandcomparingthetechnicalpropertiesofthetools.Forinstance,thiscontentmatchinganalysishighlightsthedegreeofconcordancebetweenitems,subscales,andunderlyinglatentconstructsmeasuredbyPLAYandTeachPrimary.Italsoenhancesourunderstandingoftheassociationbetweenconstructsindicatingthequalityofengagement-basedteachingpracticesandthosereflectinggeneralteachingpracticesandteacher-studentinteractions.ThiscanbeachievedbyimplementingasimilarmethodologytothatusedbytheWorldBank’sTeachteaminrelationtoinclusiveteachingclassroompractices(see

Molinaetal.,2024)

,examiningtherelationshipbetweentheunderlyingconstruct—engagingteachingpractices—andTeachPrimarybehaviors.ThisapproachenablesmutuallearningandincreasesawarenessofthesynergiesbetweenthePLAYandTeachPrimarymeasurementapproaches,facilitatingmeasurementscale-upandpromotingengagement-basedandsocio-emotionallearningintheclassroom.

Thispaperisstructuredasfollows:Section1beginswithadescriptionoftheLEGOFoundation’sPLAYandtheWorldBank’sTeachPrimaryclassroomobservationtools.Section2detailsthestudy’smethodologicalapproach,includingthestudy’sdesign,profilesofexpertsinvolved,andanalyticalmethods.Section3providesanoverviewoftheresultsfromthetwokeyphasesofthisstudy.Finally,Section4presentskeyconclusionsanddiscussesthelimitationsrelatedtothemethodused.

2.Descriptionofobservationtools

PLAYClassroomObservationTool

TheLEGOFoundationhascollaboratedwithRTIInternationalandNewYorkUniversity(NYU)GlobalTIESforChildrentodevelopaculturallyresponsivetoolkitthatincludesclassroomobservationandsurveymeasurementtoolsfocusedoncapturingthequalityofengagingteachingandlearninginclassrooms.ThePLAYtoolkitaimstomeasurehowadultssupportchildren'sengagementinlearning,whichleadstoabroadrangeoflearningoutcomes.

PLAYincludestoolsformultipleagegroupsacrossdifferentsettings.Forstudentsaged6to12,themeasurementtoolsaredesignedspecificallyforclassroomuse.Thesetoolsincludeaclassroominventory,teacher-childobservation,ateachersurvey,andachildsurvey.Theteacher-childclassroomobservation

tool,whichisthefocusofthecomparisontaskdescribedinthenextsectionofthisdocument,encompassesfourkeydimensionsessentialforsupportingchildren'sengagementinlearning.Thesedimensions,alsoreferredtoasconstructs,aresummarizedbelowanddetailedfurtherinAnnexA.

•Supportforexploration:Adultsupportforchildren'sinteractionwiththingsandideastoexpandthought.

•Supportforagency:Adultsupportforchildren'sabilitytoinfluencehowandwhattheylearn.

•Supportforpersonalandsocialconnection:Adultsupportfor(1)relatinglearningtochildren'spersonalexperiences,(2)childrenlearningthroughsocialinteraction,and/or(3)childrenfeelingasenseofbelongingandbeingsociallyconnected.

•Supportforemotionalclimate:Adultsfosteranenvironmentwhereinteractionsbetweenadult(s),child(ren),andpeersarewarm,respectful,andpositive.

Thesefourconstructsarecapturedin25itemswithinthetool.Eachitemisscoredona3-pointscale,measuringthefrequencyandqualityofteacherandstudentbehaviors,aswellasparticipationintheclassroom.

TheWorldBank’sTeachPrimary(2ndEdition)ClassroomObservationTool

TheTeachPrimarytoolcapturesteachingpracticesthatsupportqualitylearningandnurturechildren'scognitiveandsocio-emotionalskills.Initiallylaunchedin2019afterarigorousdevelopmentandvalidationprocess,TeachPrimaryunderwentarevisionin2020-2021toenhanceitsmeasurementofinclusiveteachingpracticesthatareresponsiveandfacilitatewhole-childdevelopment.SimilartoPLAY,theTeachPrimarytoolisdesignedforclassroomswithstudentsaged6to12.

TheTeachPrimaryobservationtoolassessestwomainaspects:(i)thetimeteachersspendonlearningandtheextenttowhichstudentsremainontask,and(ii)thequalityofteachingpracticesthatdevelopstudents'socio-emotionalandcognitiveskills.TheTimeonTaskcomponentusesthree"snapshots"of1–10secondstorecordtheteacher'sactionsandthenumberofstudentswhoareontask.Observersnotewhethertheteacherprovidesalearningactivityformoststudents.Ifso,theyscantheclassroomtodetermineifstudentsareontask.Off-taskbehaviorisratedasfollows:0-1studentsbeingoff-task(High);2-5studentsbeingoff-task(Medium);6ormorestudentsbeingoff-task(Low).

TheQualityofTeachingPracticescomponentisorganizedintothreeprimaryareas:

1.ClassroomCulture:Theteachercreatesacultureconducivetolearning.Thisfocusesontheextenttowhichtheteacher:

(i)createsasupportivelearningenvironmentbytreatingallstudentsrespectfully,consistentlyusingpositivelanguage,respondingtostudents'needs,andbothchallengingstereotypesandnotexhibitingbiasintheclassroom.

(ii)establishespositivebehavioralexpectationsbysettingclearbehavioralexpectations,acknowledgingpositivestudentbehavior,andeffectivelyredirectingmisbehavior.

2.Instruction:Theteacherfacilitateslessonsthatdeepenunderstandingandencouragecriticalthinkingandanalysis.Thisfocusesontheextenttowhichtheteacher:

(i)facilitatesthelessonbyexplicitlyarticulatinglessonobjectivesthatarealignedtothelearningactivity,explainingcontentusingmultipleformsofrepresentation,andconnectingthelearningactivitytoothercontentknowledgeorstudents'dailylives,andbymodelingthelearningactivitythroughenactingorthinkingaloud.

(ii)doesnotsimplymovefromonetopictothenextbutchecksforunderstandingbyusingquestions,prompts,orotherstrategiestodeterminestudents'levelofunderstanding,bymonitoringstudentsduringgroupandindependentwork,andbyadjustinghis/herteachingtothelevelofstudents.

(iii)givesfeedbackbyprovidingspecificcommentsorpromptstohelpclarifystudents'misunderstandingsoridentifytheirsuccesses.

(iv)encouragesstudentstothinkcriticallybyaskingopen-endedquestionsandprovidingthemwiththinkingtasksthatrequirethemtoanalyzecontentactively.Studentsexhibitcriticalthinkingabilitybyaskingopen-endedquestionsorperformingthinkingtasks.

3.SocioemotionalSkills:Theteacherfosterssocio-emotionalskillstohelpstudentssucceedinsideandoutsidetheclassroom.Thisfocusesontheextenttowhichtheteacher:

(i)instillsautonomybyprovidingstudentswithopportunitiestomakechoicesandtakeonmeaningfulrolesintheclassroom.Studentsexhibittheirautonomybyvolunteeringtoparticipateinclassroomactivities.

(ii)promotesperseverancebyacknowledgingstudents'effortsratherthanfocusingsolelyontheirintelligenceornaturalabilities,havingapositiveattitudetowardstudents'challenges,framingfailureandfrustrationsaspartofthelearningprocess,andencouragingstudentstosetshort-andlong-termgoals.

(iii)fosterssocialandcollaborativeskillsbyencouragingcollaborationthroughpeerinteractionandbypromotinginterpersonalskills,suchasperspective-taking,empathizing,emotionregulation,andsocialproblem-solving.Studentsexhibitsocialandcollaborativeskillsbycollaboratingwithoneanotherthroughpeerinteraction.

Thesethreeareasencompassnineelements,summarizedabove,capturedin28itemslinkedtospecificteachingbehaviors.Basedonthequalityofobservedteachingpracticescapturedinobservedbehaviors,theseitemsareratedonarubricaslow,medium,orhigh.Thescoresarethentranslatedintoa5-pointscalederivedfromtwo15-minutelessonobservationsofthesameteacher.FurtherdetailsabouteachbehaviorandelementcanbefoundintheTeachPrimary

manual

(Molinaetal.,2021).

Tooladaptation

TeachPrimaryObservationTool

TheTeachPrimarytoolwasoriginallydevelopedinEnglishbuthasbeenadaptedandimplemented

acrossvariouslinguisticandculturalcontextstoensureitsapplicability.ForSpanish-speakingcountries,thetoolhasbeentranslatedintoSpanishandusedextensively,requiringminimalcontentadaptation

duetoitsbroadalignmentwithteachingpracticesintheseregions.InAfricancountries,wheremultipleofficiallanguagesofinstructioncoexist,Teachhasbeentranslatedintoseverallocallanguagestoensureaccessibilityforenumeratorsandrelevancefordiverseclassroomsettings,butoverallthe

implementationreliedonEnglishinSierraLeoneandEthiopia.TheseadaptationsofTeachPrimaryhavefocusedprimarilyonlinguistictranslationwhilemaintainingfidelitytotheoriginalconstructs,ensuringthetool'svalidityacrossdifferentcontextsandisunderstoodbyenumerators.

PLAYClassroomObservationTool

ThePLAYtool,developedinEnglish,requiredtranslationintoSpanishforitsimplementationinPeru.ThistranslationfacilitatedthescoringofteachersinSpanish-speakingclassroomswhilemaintainingthe

theoreticalconstructsandscoringframeworkoftheoriginaltool.Whilenoadditionaladaptationsweremadeforothercountriesincludedinthestudy,theflexibilityofthePLAYtoolallowsforfurther

customizationshouldfuturecontextsrequireit.

Enumeratortraining

Trainingenumeratorsinclassroomobservationtoolsisessentialforensuringthatdatacollectedis

accurate,consistent,andreliable.Ithelpsenumeratorsunderstandhowtousethetoolcorrectly,

observeandrecordbehaviorsobjectively,andavoidpersonalbiases.Propertrainingalsoensuresethicalobservations,protectingtherightsofstudentsandteachers,andhelpsenumeratorsadapttodifferent

classroomsettings.Byaligningallenumeratorsonhowtoapplythetool,thequalityofthedataimproves,makingitmoreusefulfordecision-makingandcreatingbetterinsightsintoclassroompractices.

TeachPrimaryObservationTool

EnumeratortrainingfortheTeachPrimarytoolfollowsastructuredandrigorousprocessfacilitatedbycertifiedmastertrainers.Publiclyavailableresources,includingimplementationguidelinesandtrainingmaterials,provideacomprehensiveframeworkforpreparingenumeratorsintheTeachPrimarytool

administrationandscoring.Trainingtypicallyspansseveraldaysandinvolvesmultiplecomponents:

•ConceptualUnderstanding:TraineesareguidedthroughthetheoreticalunderpinningsoftheTeachtool,includingitsconstructsandscoringframework.

•PracticalExposure:Enumeratorsengagewithcontext-relevantvideosofteachersinclassrooms,gainingpracticalexperienceinscoringclassroombehaviors.

•SkillPractice:TraineespracticeadministeringandscoringTeachPrimarytobuildconfidenceandaccuracy.

•Certification:Enumeratorsmustpassacertificationexamtodemonstratemasteryofthetool.

Thisincludesachievingahighlevelofagreementintheirratingsofstandardizedvideos,ensuringreliabilityandreadinessforfielddatacollection.

Thisrigorousapproachensuresconsistencyandreliabilityindatacollectionacrossdiverseimplementationcontexts.

PLAYClassroomObservationTool

TrainingforthePLAYtoolwasdesignedtobuildcapacityamongenumeratorsthroughacascading

trainingmodel.Acertifiedmastertrainerconductedacomprehensivetrainingsessionforaselectgroupoftrainers,whothenreplicatedthetrainingwithenumerators.KeycomponentsofthePLAYtraining

included:

•Orientation:Enumeratorswereintroducedtothetool’sconstructsandthemethodsforobservingandscoringengagingteachingandlearning.

•PracticeOpportunities:Sessionsincludedopportunitiestopracticescoringbasedonobservationscenariosandexamples.

UnlikeTeach,thePLAYtrainingdidnotrequireenumeratorstoreachapredeterminedlevelofinter-rateragreementforcertification.However,thetrainingensuredthatenumeratorswerefamiliarwiththetoolandconfidentintheirabilitytoapplyitduringdatacollection.

3.Method

Studydescription

ThisstudycomparedtheitemcontentofthePLAYandTeachPrimaryclassroomobservationtools.Specifically,itinvolvedmappingeachofthe25itemsfromPLAYtooneormoreofthe30itemsfromTeachPrimarytodeterminetheextentofoverlapanddifferencebetweentheinstruments.AtableoutliningthemainfeaturesofthesetoolsisprovidedasAnnexB.

Participantsinthisstudycomprisedagroupofexpertswithbackgroundsinthedevelopmentand/oruseofclassroomobservationmeasures,particularlywithinlow-andmiddle-incomecountrycontextsandacrossmultipleworldregions.Initially,15expertswereidentifiedforpotentialinvolvement.Afterbeingcontactedviaemailandprovidedwithabriefoverviewofthestudy,nineconsentedtoparticipate.Followingthisconfirmation,participantsreceivedamoredetailedoutlineofthetaskandTermsofReferenceconcerningtheirconsultancyrole.Participantsengagedincontentreviewandcomparisonofthetwotools,item-to-itemmatchingtasks,andparticipatedinanyfollow-updiscussionsortasks.AnoverviewoftheexpertsisprovidedasAnnexC.

•TheDelphimethodforcontentvalidity

ThisstudystronglydrewupontheprinciplesoftheDelphimethod.TheDelphimethod,utilizedinmultiplefields,includingeducation,health,andpsychology,allowsexpertstoprovidejudgmentsonatopic,whichcanthenbemodifiedorrefinedbasedonreactionstothecollectiveviewsofthegroup

(Green,2014;

Hyatt

etal.,2024;

ManizadeandMason,2010)

.Unlikeface-to-facetechniques,theDelphimethodpromotesindependenceofthoughtandallowsresearcherstoleveragetheexpertiseofnumerousexpertsfromdifferentlocationswhilefosteringanenvironmentthatencouragesgroupacceptanceandsharedresponsibilityfortheoutcome.TheDelphimethodtypicallyinvolvesseveralkeycharacteristicsforobtaininggroupconsensusonatopic,including(1)participantanonymity,(2)anonymityofresponses,(3)multipleandstructurediterationsofconsultations,(4)groupconsensus,(5)controlledfeedbackofresponsestoparticipants,and(6)statisticalanalysisofdata

(GrantaandKinney,1992,pp.13-14)

.StudiesutilizingtheDelphimethodhavebeenconductedwithasfewassevenandasmanyas1,000participants,thoughsamplesbetween10and50aretypicallyadvisedduetotheextensivedatageneratedandtheanalysesrequiredforeachindividualresponse

(IqbalandPipon-Young,2009)

.

TheDelphimethodhasemergedasapopularapproachfordeterminingcontentvalidity

(Green,2014;

ManizadeandMason,2010)

.Contentvaliditydenotestheextenttowhichtheitemsofameasurementinstrumentarerelevanttoandrepresentativeofthetargetedconstructitisintendedtomeasure.Itconsiderstheimportance,relevance,andclarityofinstrumentitems,domains,anddefinitions,aswellastheappropriatenessandadequacyofitemresponsescales

(Roebiantoetal.,2023)

.Contentvalidityis

particularlyrelevantasitincreasestheaccuracyofmeasuringconstructs.Forexample,inclassroomobs