版权说明:本文档由用户提供并上传,收益归属内容提供方,若内容存在侵权,请进行举报或认领
文档简介
CONTENTVALIDITYJeffreyM.MillerNovember,2003CONTENTVALIDITYJeffreyM.Mil1OriginsContentvalidityreferstothedegreetowhichthecontentoftheitemsreflectsthecontentdomainofinterest(APA,1954)Isthecontentaboutwhatwesaythetestisabout?OriginsContentvalidityrefers2DistinctorSubsumed?Guion’s(1980)HolyTrinity 1.Criterion-related(Predictive/Concurrent) 2.Construct 3.CriterionCronbach(1984)/Messick(1989)–ThethreearedifferentmethodsofinquirysubsumedbytheoverarchingconstructvalidityDistinctorSubsumed?Guion’s(3CurrentDefinition“Validityreferstothedegreetowhichevidenceandtheorysupporttheinterpretationsoftestscoresentailedbyproposedusesoftests(AERA/APA/NCME,1999)CurrentDefinition“Validityre4SoDoesContentMatter???Contentisnotapartofthescoresoitisnotapartofvalidity(Messick,1975;Tenopyr,1977)Contentisaprecursortodrawingascore-basedinference.Itisevidence-in-waiting(Shepard,1993;Yalow&Popham,1983)Contentisafeatureofthetest,notthescoreSoDoesContentMatter???Conte5Precursorsto“SloppyValidation”?Theoverarchingconstructvalidityparadigmrelegatesthestatusofcontentvalidityandjustifiespoorimplementation
Thecurrentdefinitionofvalidityrelegatesthestatusofcontentvalidityandjustifiespoorimplementation
Intendedorunintended,whatthenhappenstothevalidationofcontent?Precursorsto“SloppyValidati6ProphecyFulfilled?“Wefearthateffortstowithdrawthelegitimacyofcontentrepresentativenessasaformofvaliditymay,intime,substantiallyreduceattentiontotheimportofcontentcoverage(Yalow&Popham,1983).”“Unfortunately,inmanytechnicalmanuals,contentrepresentationisdealtwithinaparagraph,indicatingthatselectedpanelsofsubjectmatterexperts(SMEs)reviewedthetestcontent,ormappedtheitemstothecontentstandards–andalliswell(Crocker,2003)”ProphecyFulfilled?“Wefearth7RecentArgument“Contentrepresentationistheonlyaspectofvalidationthatcanbecompletedpriortoadministeringthetestandreportingresults.Ifthisprocessyieldsdisappointingresults,thereisstilltimetorecoup”(Crocker,2003)RecentArgument“Contentrepres8TheStandardProcedureCrocker&Algina(1986)DefinetheperformancedomainofinterestSelectapanelofqualifiedexpertsinthecontentdomainProvideastructuredframeworkfortheprocessofmatchingitemstotheperformancedomainCollectandsummarizedatafromthematchingprocessTheStandardProcedureCrocker9Hambleton’s(1980)12StepsPrepareandselectobjectiveordomainspecificationsClarifytest’spurposes,desirableformats,numberofitems,instructionforwritingWriteitemstomeasuretheobjectivesItemwritersperformtheinitialeditSystematicallyassessitemmatchtoobjectivestodeterminerepresentativenessPerformadditionalitemeditingHambleton’s(1980)12StepsPre10Hambleton’s(1980)12StepsAssemblethetestSelectandimplementmethodforsettingstandardsforinterpretingperformanceAdministerthetestCollectdataaddressingreliability,validity,andnormsPrepareuser’smanual/technicalmanualConductongoingstudiesrelatingtesttodifferentsituationsandpopulationsHambleton’s(1980)12StepsAss11Beyond“TheExpertsAgreed”Althoughtheproceduresareexplicitanddetailed,ultimateassuranceofcontentvalidityisbasedonthemethodonauthority
Ourtrainingintheimportanceofthescientificmethodmayexplainwhy“Theexpertsagreed”doesn’tsettlewell.Wehavethequantitativeitemanalysis,factoranalysis,IRT,andCronbach’salphainthesamereportasthequalitativeexpertagreementBeyond“TheExpertsAgreed”Alt12Katz’sPercentage(1958)Usingthismethod,expertsratewhetherornottheitemtapstheobjectiveonayesornodichotomousscaleLetyes=1andno=0Thenletn=thenumberof1’sforaparticularraterTheproportionissimplythesumofthen’sacrossallratersdividedbytheproductofthetotalnumberofitems(N)andthetotalnumberofraters(J)P=sumofn/(N*J)Katz’sPercentage(1958)Using13Theobviouslimitationsare:Influencebythenumberofitemsand/orratersDichotomousdecision(hencenodegreeofcertainty/uncertainty)Inclusionofallitems(hencenoregardforindividualitemweighting)NoinclusionofobjectivesthatareNOTintendedtobemeasuredand/ormultipleobjectivesTheobviouslimitationsare:14Klein&Kosecoff’sCorrelation(1975)Expertsratetheimportanceoftheobjectiveona1to5pointLikertscaleThemeanormedianisusedasanindexofrelativeimportanceforanitemThen,judgesratehowwelltheitemmatcheseachobjectiveonayes(1)/no(0)scale.Letp=theproportionofjudgeswhoassigna1toanitemononeobjectiveLetP=thesumofthep’sforallitemsmeasuringaparticularobjectivePearson’sristhencomputedusingthePofobjectiveimportanceandthePofitemtoobjectivematchKlein&Kosecoff’sCorrelation15AlthoughthistechniquetriestocontroltheproblemofindividualitemweightingviarankingsofimportanceANDincludesthepossibilityofmultipleobjectives,thelimitationsareAgain,sensitivitytothenumberofitemsandthenumberofjudgesThepossibilityofahighrwhenitemsdonotmatchanyobjectiveAlthoughthistechniquetries16Aiken’sV(1985)content-validitycoefficientnexpertsratethedegreetowhichtheitemtapsanobjectiveona1tocLikert-scaleLetlo=thelowestpossiblevalidityrating(usually,thisis1ontheLikert-scale)Letr=theratingbyanexpertLets=r–loLetS=thesumofsforthenratersAiken’sVisthenV=S/[n*(c-1)]Therangewillbefrom0to1.0Ascoreof1.0isinterpretedasallratersgivingtheitemthehighestpossibleratingAiken’sV(1985)content-valid17Aiken’sVcanbeusedwitharight-tailedbinomialprobabilitytabletoobtainstatisticalsignificanceAiken’sVdoesnotinclude 1.ObjectivesthatareNOTintendedtobemeasured 2.MultipleobjectivesAiken’sVcanbeusedwithar18Rovinelli&Hambleton’sIndexofItem-ObjectiveCongruence(1977)
Contentexpertsrateitemsregardinghowwelltheydo(ordonot)taptheestablishedobjectivesTheratingsare:1:itemclearlytapsobjective0:unsure/unclear-1:itemclearlydoesnottapobjectiveSeveralcompetingobjectivesareprovidedforeachitemAstatisticalformula(orSASprogram)isthenappliedtotheratingsofeachitemacrossraters.Theresultisanindexrangingfrom–1to+1Rovinelli&Hambleton’sIndex19Anindexof–1canbeinterpretedascompleteagreementbyallexpertsthattheitemismeasuringallthewrongobjectivesAnindexof+1canbeinterpretedascompleteagreementbyallexpertsthattheitemisonlymeasuringthecorrectobjectiveAnindexof–1canbeinterpre20Theindexofitem-objectivecongruenceassumesthattheitemtapsoneandonlyoneobjectiveHowever,thereisaformula(andSAScode)forsituationswhenanitemtapsmorethanoneobjective.Theindexofitem-objectiveco21Penfield’s(2003)ScoreIntervalManyofthequantificationproceduresaddressthemeanratingforanitemAnimprovementwouldbetoconstructaconfidenceintervalforthemeanratingofanitem.Wecouldthensaythat,givenameanratingof3.42ona4-pointLikert-scale,weare95%certainthatthetruepopulationmeanratingisbetween1.2and3.5orthatitisbetween3.4and3.5anddeterminetheaccuracyofexpertagreement.Penfield’s(2003)ScoreInterv22Thetraditionalconfidenceintervalassumesanormaldistributionforthesamplemeanofaratingscale.However,theassumptionofpopulationnormalitycannotbejustifiedwhenanalyzingthemeanofanindividualscaleitembecause1.)theoutcomesoftheitemsarediscrete,and2.)theitemsareboundedbythelimitsoftheLikert-scale.Thetraditionalconfidenceint23TheScoreconfidenceintervaltreatsratingscalevariablesasoutcomesofabinomialdistribution.Thisasymmetricintervalwasshowntoberobusttoalackoffittoabinomialdistributionespeciallywhenthesamplesizeand/orthenumberofscalecategoriesissmall(e.g.,lessthanorequaltofive).TheScoreconfidenceinterval24ConclusionContentvalidityaddressestheadequacyandrepresentativenessoftheitemstothedomainoftestingpurposesContentvalidityisnotusuallyquantifiedpossiblydueto1.)subsumingitwithinconstructvalidity;2.)ignoringitasimportant;and/or3.)relyingonacceptedexpertagreementproceduresIndicesareavailable,andthereisapushtowardsimprovingthereportingofcontentvalidationproceduresConclusionContentvalidityadd25CONTENTVALIDITYJeffreyM.MillerNovember,2003CONTENTVALIDITYJeffreyM.Mil26OriginsContentvalidityreferstothedegreetowhichthecontentoftheitemsreflectsthecontentdomainofinterest(APA,1954)Isthecontentaboutwhatwesaythetestisabout?OriginsContentvalidityrefers27DistinctorSubsumed?Guion’s(1980)HolyTrinity 1.Criterion-related(Predictive/Concurrent) 2.Construct 3.CriterionCronbach(1984)/Messick(1989)–ThethreearedifferentmethodsofinquirysubsumedbytheoverarchingconstructvalidityDistinctorSubsumed?Guion’s(28CurrentDefinition“Validityreferstothedegreetowhichevidenceandtheorysupporttheinterpretationsoftestscoresentailedbyproposedusesoftests(AERA/APA/NCME,1999)CurrentDefinition“Validityre29SoDoesContentMatter???Contentisnotapartofthescoresoitisnotapartofvalidity(Messick,1975;Tenopyr,1977)Contentisaprecursortodrawingascore-basedinference.Itisevidence-in-waiting(Shepard,1993;Yalow&Popham,1983)Contentisafeatureofthetest,notthescoreSoDoesContentMatter???Conte30Precursorsto“SloppyValidation”?Theoverarchingconstructvalidityparadigmrelegatesthestatusofcontentvalidityandjustifiespoorimplementation
Thecurrentdefinitionofvalidityrelegatesthestatusofcontentvalidityandjustifiespoorimplementation
Intendedorunintended,whatthenhappenstothevalidationofcontent?Precursorsto“SloppyValidati31ProphecyFulfilled?“Wefearthateffortstowithdrawthelegitimacyofcontentrepresentativenessasaformofvaliditymay,intime,substantiallyreduceattentiontotheimportofcontentcoverage(Yalow&Popham,1983).”“Unfortunately,inmanytechnicalmanuals,contentrepresentationisdealtwithinaparagraph,indicatingthatselectedpanelsofsubjectmatterexperts(SMEs)reviewedthetestcontent,ormappedtheitemstothecontentstandards–andalliswell(Crocker,2003)”ProphecyFulfilled?“Wefearth32RecentArgument“Contentrepresentationistheonlyaspectofvalidationthatcanbecompletedpriortoadministeringthetestandreportingresults.Ifthisprocessyieldsdisappointingresults,thereisstilltimetorecoup”(Crocker,2003)RecentArgument“Contentrepres33TheStandardProcedureCrocker&Algina(1986)DefinetheperformancedomainofinterestSelectapanelofqualifiedexpertsinthecontentdomainProvideastructuredframeworkfortheprocessofmatchingitemstotheperformancedomainCollectandsummarizedatafromthematchingprocessTheStandardProcedureCrocker34Hambleton’s(1980)12StepsPrepareandselectobjectiveordomainspecificationsClarifytest’spurposes,desirableformats,numberofitems,instructionforwritingWriteitemstomeasuretheobjectivesItemwritersperformtheinitialeditSystematicallyassessitemmatchtoobjectivestodeterminerepresentativenessPerformadditionalitemeditingHambleton’s(1980)12StepsPre35Hambleton’s(1980)12StepsAssemblethetestSelectandimplementmethodforsettingstandardsforinterpretingperformanceAdministerthetestCollectdataaddressingreliability,validity,andnormsPrepareuser’smanual/technicalmanualConductongoingstudiesrelatingtesttodifferentsituationsandpopulationsHambleton’s(1980)12StepsAss36Beyond“TheExpertsAgreed”Althoughtheproceduresareexplicitanddetailed,ultimateassuranceofcontentvalidityisbasedonthemethodonauthority
Ourtrainingintheimportanceofthescientificmethodmayexplainwhy“Theexpertsagreed”doesn’tsettlewell.Wehavethequantitativeitemanalysis,factoranalysis,IRT,andCronbach’salphainthesamereportasthequalitativeexpertagreementBeyond“TheExpertsAgreed”Alt37Katz’sPercentage(1958)Usingthismethod,expertsratewhetherornottheitemtapstheobjectiveonayesornodichotomousscaleLetyes=1andno=0Thenletn=thenumberof1’sforaparticularraterTheproportionissimplythesumofthen’sacrossallratersdividedbytheproductofthetotalnumberofitems(N)andthetotalnumberofraters(J)P=sumofn/(N*J)Katz’sPercentage(1958)Using38Theobviouslimitationsare:Influencebythenumberofitemsand/orratersDichotomousdecision(hencenodegreeofcertainty/uncertainty)Inclusionofallitems(hencenoregardforindividualitemweighting)NoinclusionofobjectivesthatareNOTintendedtobemeasuredand/ormultipleobjectivesTheobviouslimitationsare:39Klein&Kosecoff’sCorrelation(1975)Expertsratetheimportanceoftheobjectiveona1to5pointLikertscaleThemeanormedianisusedasanindexofrelativeimportanceforanitemThen,judgesratehowwelltheitemmatcheseachobjectiveonayes(1)/no(0)scale.Letp=theproportionofjudgeswhoassigna1toanitemononeobjectiveLetP=thesumofthep’sforallitemsmeasuringaparticularobjectivePearson’sristhencomputedusingthePofobjectiveimportanceandthePofitemtoobjectivematchKlein&Kosecoff’sCorrelation40AlthoughthistechniquetriestocontroltheproblemofindividualitemweightingviarankingsofimportanceANDincludesthepossibilityofmultipleobjectives,thelimitationsareAgain,sensitivitytothenumberofitemsandthenumberofjudgesThepossibilityofahighrwhenitemsdonotmatchanyobjectiveAlthoughthistechniquetries41Aiken’sV(1985)content-validitycoefficientnexpertsratethedegreetowhichtheitemtapsanobjectiveona1tocLikert-scaleLetlo=thelowestpossiblevalidityrating(usually,thisis1ontheLikert-scale)Letr=theratingbyanexpertLets=r–loLetS=thesumofsforthenratersAiken’sVisthenV=S/[n*(c-1)]Therangewillbefrom0to1.0Ascoreof1.0isinterpretedasallratersgivingtheitemthehighestpossibleratingAiken’sV(1985)content-valid42Aiken’sVcanbeusedwitharight-tailedbinomialprobabilitytabletoobtainstatisticalsignificanceAiken’sVdoesnotinclude 1.ObjectivesthatareNOTintendedtobemeasured 2.MultipleobjectivesAiken’sVcanbeusedwithar43Rovinelli&Hambleton’sIndexofItem-ObjectiveCongruence(1977)
Contentexpertsrateitemsregardinghowwelltheydo(ordonot)taptheestablishedobjectivesTheratingsare:1:itemclearlytapsobjective0:unsure/unclear-1:itemclearlydoesnottapobjectiveSeveralcompetingobjectivesareprovidedforeachitemAstatisticalformula(orSASprogram)isthenappliedtotheratingsofeachitemacrossraters.Theresultisanindexrangingfrom–1to+1Rovinelli&Hambleton’sIndex44Anindexof–1canbeinterpretedascompleteagreementbyallexpertsthattheitemismeasuringallthewrongobjectivesAnindexof+1canbeinterpretedascompleteagreementbyallexpertsthattheitemisonlymeasuringthecorrectobjectiveAnindexof–1canbeinterpre45Theindexofitem-objectivecongruenceassumesthattheitemtapsoneandonlyoneobjectiveHowever,there
温馨提示
- 1. 本站所有资源如无特殊说明,都需要本地电脑安装OFFICE2007和PDF阅读器。图纸软件为CAD,CAXA,PROE,UG,SolidWorks等.压缩文件请下载最新的WinRAR软件解压。
- 2. 本站的文档不包含任何第三方提供的附件图纸等,如果需要附件,请联系上传者。文件的所有权益归上传用户所有。
- 3. 本站RAR压缩包中若带图纸,网页内容里面会有图纸预览,若没有图纸预览就没有图纸。
- 4. 未经权益所有人同意不得将文件中的内容挪作商业或盈利用途。
- 5. 人人文库网仅提供信息存储空间,仅对用户上传内容的表现方式做保护处理,对用户上传分享的文档内容本身不做任何修改或编辑,并不能对任何下载内容负责。
- 6. 下载文件中如有侵权或不适当内容,请与我们联系,我们立即纠正。
- 7. 本站不保证下载资源的准确性、安全性和完整性, 同时也不承担用户因使用这些下载资源对自己和他人造成任何形式的伤害或损失。
最新文档
- (2025年)医学法律法规考试题库及参考答案
- (2026)病案编码员资格证试题库(附含参考答案)
- (完整版)内科护理学试题及答案
- 《采购管理》期末试卷一(含答案)
- 2024年导游证考试题库及答案
- (2025)保密教育测试解析含答案
- (2025年)护理三基三严理论考试试题及答案
- (2025年)四川省成都市公开遴选公务员笔试题及答案解析(A类)
- 2025 年母婴保健培训试题及答案
- 2026年煤矿电工题库题及参考答案
- 家庭保洁作业安全方案
- 2025年人教版七年级下册历史全册知识点
- 学校后勤服务采购合同协议三篇
- DELE A1西班牙语全真模拟试题(含详细答案解析)
- 2026年新疆中考语文真题及答案解析
- IEC 62351-92023 电力系统管理和相关信息交换.数据和通信安全.第9部分电力系统设备的网络安全密钥管理标准立项发展报告
- 2026年《安全员》考试试题及答案
- 出土坡道施工方案
- 2026年湖南高职单招职业技能测试考试题库(附含答案)
- 物业应急指挥体系建设手册
- 关于手术室医疗废物泄漏的应急预案演练脚本
评论
0/150
提交评论