美联储-发送多个预付费激励是否有效？来自随机实验的证据（英）

FinanceandEconomicsDiscussionSeries

FederalReserveBoard,Washington,D.C.

ISSN1936-2854(Print)

ISSN2767-3898(Online)

DoesitPaytoSendMultiplePre-PaidIncentives?EvidencefromaRandomizedExperiment

AndrewC.Chang,JoanneW.Hsu,EvaMa,KateBachtell,andMicahSjoblom

2024-023

Pleasecitethispaperas:

Chang,AndrewC.,JoanneW.Hsu,EvaMa,KateBachtell,andMicahSjoblom(2024).“DoesitPaytoSendMultiplePre-PaidIncentives?EvidencefromaRandomizedExperi-ment,”FinanceandEconomicsDiscussionSeries2024-023.Washington:BoardofGovernorsoftheFederalReserveSystem,

/10.17016/FEDS.2024.023

.

NOTE:StafworkingpapersintheFinanceandEconomicsDiscussionSeries(FEDS)arepreliminarymaterialscirculatedtostimulatediscussionandcriticalcomment.TheanalysisandconclusionssetfortharethoseoftheauthorsanddonotindicateconcurrencebyothermembersoftheresearchstafortheBoardofGovernors.ReferencesinpublicationstotheFinanceandEconomicsDiscussionSeries(otherthanacknowledgement)shouldbeclearedwiththeauthor(s)toprotectthetentativecharacterofthesepapers.

Page1of22

DoesitPaytoSendMultiplePre-PaidIncentives?

EvidencefromaRandomizedExperiment

April15,2024

AndrewC.Chang,*BoardofGovernorsoftheFederalReserveSystem,

andrew.c.chang@

JoanneW.Hsu,†UniversityofMichigan,

jwhsu@

EvaMa,BoardofGovernorsoftheFederalReserveSystem,

eva.ma@

KateBachtell,NORCattheUniversityofChicago,

bachtell-kate@

MicahSjoblom,NORCattheUniversityofChicago,

sjoblom-micah@

Abstract

Toencouragesurveyparticipationandimprovesamplerepresentativeness,theSurveyof

ConsumerFinances(SCF)offersanunconditionalpre-paidmonetaryincentiveandseparatepost-paidincentiveuponsurveycompletion.Weconductedapre-registeredbetween-subject

randomizedcontrolexperimentwithinthe2022SCF,withatleast1,200householdsper

experimentalgroup,toexaminewhetherchangingthepre-paidincentivestructureaffectssurveyoutcomes.Weassesstheeffectsof:(1)alteringthetotaldollarvalueofthepre-paidincentive

(“incentiveeffect”),(2)givingtwoidenticalpre-paidincentivesholdingthetotaldollarvalue

fixed(“remindereffect”),and(3)offeringmultiplepre-paidincentivesofdifferentamounts

holdingthetotaldollarvaluefixed(“slopeeffect”)onsurveyresponserates,interviewerburden,anddataquality.Ourevidenceindicatesthatasingle$15pre-paidincentiveincreasesresponseratesandmaintainssimilarlevelsofinterviewerburdenanddataquality,relativetoasingle$5pre-paidincentive.Splittingthe$15intotwopre-paidincentivesofdifferentamountsincreasesinterviewerburdenthoughlengtheningtimeinthefieldwithoutimprovingresponserates,

reducingthenumberofcontactattemptsneededforaresponse,orimprovingdataquality,regardlessofwhetherthefirstpre-paidislargerorsmallerthanthesecond.

Keywords:pre-paidincentives;unconditionalincentives;sequentialincentives;responserates;surveys;dataquality;householdfinance

JELCodes:C83;C93;G5

*:ORCID0000-0002-9769-789X.†:ORCID0000-0002-0715-6230.Theanalysisandconclusionssetforthare

thoseoftheauthorsanddonotindicateconcurrencebyothermembersoftheBoardofGovernorsoftheFederal

ReserveSystem,itsresearchstaff,ortheNORCattheUniversityofChicago.WethankCathyHaggerty,

MichaelKalmar,KatherineMcGonagle,KevinB.Moore,HeatherSawyer,AliceHenriquesVolz,andconference

participantsatthe2023JointStatisticalMeetingsforcommentsandhelpwiththisproject.Ourpre-registrationplan

isontheOpenScienceFrameworkunder“2022SCFPre-paidIncentivesExperiment”at

/10.17605/OSF.IO/BXJNE.

ThisexperimentwasapprovedbytheInternalReviewBoardoftheNORCattheUniversityofChicagounderprotocolID#21-08-433.

Page2of22

Introduction

Anincreasinglychallengingenvironmentforrecruitingparticipantsforsurveysposesmanyrisks

forresearcherswhoneedtobalanceachievingrepresentativesamplesandmaintainingdata

qualitywithcontrollingfinancialcostsandtimeinthefield.Incentives,includingpre-paid

incentives,whichareunconditionalonsurveycompletion,areoneimportanttooltoencouragesurveyresponse.Whilesurveystypicallyemployasinglepre-paidincentive,duringdata

collectionpre-paidincentivescanbestructuredinavarietyofways.

TheSurveyofConsumerFinances(SCF),

1

sponsoredbytheFederalReserveBoard(FRB),

beganemployingpre-notificationpostcardsfollowedbyapre-paidincentiveoffivedollarscashwithinvitationlettersforthe2016wave,basedonfindingsfromHsu,Schmeiser,Haggerty,andNelson(2017).Toexplorehowchangesinthestructureofrespondentincentivescouldimproveresponserates,durationinthefield,anddataqualityweembeddedan8weekfieldexperimentwithinthe2022SCF.Werandomlyassignedrespondentsacross6groupswithvaryingpre-paidincentivestructuresandamounts,includingonegroupwiththepreviouslyusedsinglepre-paidincentiveof$5.Inourexperiment,wetestedthreedifferentconditions:(1)alteringthetotal

dollarvalueofthepre-paidincentive(“incentiveeffect”),(2)givingtwoidenticalpre-paid

incentivesholdingthetotaldollarvaluefixed(“remindereffect”),and(3)offeringmultiplepre-paidincentivesofdifferentamountsholdingthetotaldollarvaluefixed(“slopeeffect”).

Weanalyzetheexperimentalresultstodeterminetheeffectsofdifferentpre-paidincentive

structuresonsurveyresponserates,interviewerburden,anddataqualityusingamixed-mode

(phoneandface-to-face)surveyonhouseholdfinances.Theresultsprovideinsightsintothe

costsandbenefitsofdifferentdesignsandvaluesofpre-paidincentivesforcompletinganin-

depthinterviewer-administeredsurveyonasensitivetopic.Moreover,byofferingincentive

amountsinavarietyofvalues,weinvestigatewhethertherelationshipbetweenhigherincentiveamountsandcompletionratesismonotonicordeterioratesatlargervalues.

Wefindthatlargerpre-paidincentivesyieldedhigherresponseratesofabout2or3percentagepoints(aroundathirdofthebaselineresponserate)withnodeleteriouseffectonothersurveyoutcomes,andthatitisbettertosendtheincentiveasasinglepayment,ratherthansplittingitintomultiplepayments.Splittingtheincentiveintopaymentsofdifferentamountsincreased

fieldinterviewerburdenthroughincreasingtimeinthefieldwithoutimprovingresponserates,reducingthenumberofcontactattemptsneededforaresponse,orimprovingdataquality.

TheoreticalBackgroundontheResponsetoPre-paidMonetaryIncentives

Anextensivebodyofempiricalandtheoreticalresearchsupportstheuseofincentivesaspartofabroaderstrategytopromotesurveycompletion.Anabundanceofresearchhasshownthat

unconditionalpre-paidincentivesareparticularlyeffectiverelativetoconditionalpost-paid

1BoardofGovernorsoftheFederalReserveSystem(2023b).TheSCFcollectsinformationonUShousehold

income,wealth,debts,andotherfinancialoutcomes.SeeAladangady,Bricker,Chang,Goodman,Krimmel,Moore,Reber,HenriquesVolz,andWindle(2023)foradescriptionoftheSCF.

Page3of22

incentives(BlohmandKoch2021).Inaccordancewiththeoriesofsocialexchange,

noncontingentpre-paidincentivescanencouragesurveyresponsebyprovidingapsychologicalsenseofobligationtoreturnthefavoroftheincentive(Gouldner1960;Dillman1978).

Leverage-saliencytheoryprovidesanothermechanism:apre-paidincentivecouldhelpestablishtrustthatarespondentwillhonortherequestandcompletethesurvey(Groves,Singer,and

Corning2000).Thatsaid,incentivescouldappealtoexternalmotivations,whicharetypicallylesseffectivethanintrinsicoraltruisticmotivationsatgeneratingcompliancewithasurvey

request(Hansen1980).

Theeffectivenessofincentivesvariesbydesignandsize.Differentincentiveamountsmay

generateavaryingleverageorstrongerfeelingsofobligation.Accordingtotheoriesofeconomicexchange,respondentsrespondtosurveyswhentheoverallbenefitsoutweighthecosts,andthuslargerincentivesshouldyieldhigherresponserates(BinerandKidd1994).However,response

ratesmaynotincreasemonotonicallywiththesizeoftheincentive.Somestudieshavefoundthattherelationshipbetweenthesizeofpre-paidincentivesandresponseratesisnonlinear(Warriner,Goyder,Gjertsen,Hohner,andMcSpurren1996;TrussellandLavrakas2004).

2

Onepossible

explanationisthatthesizeoftheincentivemayappealdifferentiallytofactorsofloworhigh

leverage,feelingsofreciprocity,orperceptionsofsurveylegitimacy.Inaddition,excessively

largeincentivescouldleadrespondentstodistrustthesurveyorappealtooheavilytoextrinsic

motivatorstobeeffective.Thatsaid,ananalysisofanincreaseintheSCF’spost-paidincentivefrom2007to2010foundthattheincreasereducedthecontactattemptsandtimeinthefield

neededforaresponsewhilemaintainingdataquality(Bricker2014).Anda2014experimentthatimitatedtheinstrumentandfieldstrategyoftheSCFfoundnonegativeeffectsforverylarge

conditionalpost-paidincentives(Hsu,Schmeiser,Haggerty,andNelson2017).

Mostsurveysemploypre-paidincentivesonceduringdatacollection.

3

Butgiventhe

proliferationofjunkmail,individualsmaypaylessattentiontotheirpostalmailormaymissaninitialmailerorincentiveentirely.Consequently,somestudieshaveinvestigatedtheuseof

repeated(sequential)pre-paidincentiveswiththehopethatarepeatincentivewillelicitmore

careful(orany)readingandconsiderationofthesurveymaterialsbytherespondent.MesserandDillman(2011)foundthatfollowingupaninitialrequestthatincludeda$5pre-paidincentive

withasecond$5pre-paidincentiveviaprioritymailtononrespondersincreasedtheresponse

rateofastate-widewebsurveyfrom59%to68%.Inthecontextofadifferentmail-websurvey,Wagner,West,Couper,Zhang,Gatward,Nishimura,andSaw(2023)foundthatfollowingupa$2pre-paidincentiveincludedintheinitialmailingwithanadditional$5sentviaprioritymailconsiderablyincreasedresponserates.However,inbothofthesestudiestheeffectofthesecondincentivecannotbedisentangledfromtheeffectoftheprioritymailing.

2Relatedly,Han,Montaquila,andBrick(2013,Table2)findthatthesizeofapre-paidincentiveaffectsresponseratesconditionalonhowquicklyeligiblerespondentscompleteascreener.

3Correspondingly,therearealargenumberofstudiesthatevaluatewhetherasinglepre-paidincentiveaffectsresponserates.Forexample:Hsu,Schmeiser,Haggerty,andNelson(2017);Frederiks,Romanach,Berry,andToscas(2020);Jackson,McPhee,andLavrakas(2020);Powell,Geronimo-Hara,Tobin,Donoho,Sheppard,

Walstrom,Rull,andFaix(2023).

Page4of22

Dykema,Stevenson,Assad,Kniss,andTaylor(2021)conductedanexperimentinamailsurvey

ofphysiciansandfoundthat“secondincentives[sentamonthafterthefirstincentives]were

associatedwithhigherresponseratesandlowercostspercompletedsurvey”butnomeasurableeffectonitemnonresponse.Dillman,Smyth,andChristian(2014,p.424)nowrecommend

researchersinmixed-modestudiestoincludeasecondcashincentivewiththeirfollow-up

communicationstoprovideopportunitiesfor“latercommunications[to]beread,andhopefullyactedupon,therebyincreasingoverallresponse.”Thatsaid,Dykema,Jaques,Cyffka,Assad,

Hammers,Elver,Malecki,andStevenson(2015)foundthatasecondpre-paidincentive,againtargetingnonresponders,didnotincreaseresponserates.

Fewstudieshaveinvestigatedthedifferentialeffectofrepeatedincentivesthatareincreasingordecreasinginsize,whatwecallthe“slopeeffect”.OneexampleisDykema,Stevenson,Assad,Kniss,andTaylor(2021),whofoundfewmeasurabledifferencesinresponseratesoritem

responseforsecondincentivesthatarelargerthanthefirstincentive,relativetosecond

incentivesthataresmallerthanthefirst.Similarly,theyfoundtherepresentativityofrespondersforeitheroftheseconditionswasnotsignificantlydifferentfrombenchmarkadministrativedata,sotheslopeeffectintheirstudyappearstohavemadenodifferencesinsurveyoutcomes.

Inourstudy,weanalyzepre-paidincentivesinthecontextofanationallyrepresentativemixed

mode(face-to-faceandphone)surveyinwhichrespondentsareinitiallycontactedviamail.

Whilemostexistingstudiesinvolvedsendingsecondincentivesonlytononresponders,ourstudydeliveredsecondmailingsandincentivestopotentialrespondentstwoweeksafterthefirst,

regardlessofresponse.

Methods

OverviewoftheSurveyofConsumerFinances

TheSCFisanationallyrepresentativesurveyonthefinancesofUShouseholds,conductedonadifferentcross-sectionofUSfamiliestriennially(BoardofGovernorsoftheFederalReserve

System,2023b).Topicscoveredincludeincome,assets,debts,otherfinancialcharacteristics,andeconomicbehavior.Thesurveyisadministeredbyfieldinterviewers(FIs)and,historically,isprimarilyconductedface-to-face.

GiventhehighconcentrationofwealthintheUnitedStates,theSCFusesadual-framesampletoensurecoverageacrossthefulldistributionofwealth.TheSCFemploysbothanaddress-based

multistagenationallyrepresentativearea-probability(AP)samplecomplementedbyastratifiedlistsamplespecificallydesignedtooversamplewealthyAmericans.

4

Inthefaceofan

increasinglychallengingenvironmentforsurveyresponseratesoverthepasttwodecades,theSCFhasrepeatedlyextendedtimeandaddedexpensesinthefield.Eachwavesince2004

requiredanaverageextensionof2.5additionalmonthsbeyondthetargetfieldperiodofeightmonths,withthe2019and2022wavesneedingextensionsofabout4months.

5

4SeeKennickell(2005)foradiscussionofthesamplingprocedure.

5In2022theAPsampleresponseratewasabout42percent,andthelistsampleresponseratewasabout27percent,usingRR1fromAAPOR(2015).SeetheappendixestoBhutta,Bricker,Chang,Dettling,Goodman,Hsu,Moore,Reber,HenriquesVolz,andWindle(2020);Aladangady,Bricker,Chang,Goodman,Krimmel,Moore,Reber,

Page5of22

StudyDesign

The2022SCF,whichwassponsoredbytheFRBwithcooperationfromtheStatisticsofIncomeDivision(SOI)attheInternalRevenueService,andconductedbytheNationalOpinionResearch

Center(NORC)attheUniversityofChicago,includedapre-registeredbetween-subject

randomizedcontrolexperimentwithintheAPsamplewithsixdifferentpre-paidincentive

groups(a1x6celldesign).

6

Werefertoparticulargroupsusingtheformat$[First]/$[SecondPre-PaidAmount].Onegroup($5/$0),structuredtobeidenticaltothepre-paidincentivefromthe

2016and2019SCFs,receivedasingle$5pre-paidincentive,followedbyasecondmailerwithnomonetaryincentive.Theotherfivetreatmentgroups,showninTable1,facilitatedtestingtheeffectsof:(1)largertotalincentivepayments(theincentiveeffect),(2)twoincentivepaymentsperrespondentagainstasinglepre-paidincentivewiththesametotalvalue(theremindereffect),and(3)secondpre-paidincentivesthatarelargerorsmallerinsizethanthefirstincentive,

controllingfortotalvalue(theslopeeffects).

Table1:TreatmentGroupDefinitions

Group

First

Pre-paidincentiveamount

SecondPre-paidincentiveamount

Sample

Size

Notes

$5/$0

$5

-

2,152

The2016/2019SCFincentive,controlfortotal$amountforsinglepre-paidincentives

$5/$5

$5

$5

1,292

Baselinemultiplemailertreatment

$10/$0

$10

-

1,291

Total$amountcontrolforbaselinemultiplemailers($5/$5group)

$5/$10

$5

$10

1,293

Testsupwardslopeofincentive

$10/$5

$10

$5

1,291

Testdownwardslopeofincentive

$15/$0

$15

-

1,297

Total$amountcontrolforslopeconditions($5/$10and$10/$5groups)

WestratifiedourrandomizationbyNationalFrameArea(NFA,aprimarysamplingunitofgeographyusedbyNORCtocreatetheAPframe)acrossallNFAsintheAPsample.

RandomizationwasconductedbyNORCwithaquasi-randomnumbergeneratorwithno

rerandomization.Duetolimitsondrawingthesamplingframeforthe2022SCFandthe

simultaneousnatureofthetreatmentacrossallsampleunits,householdswerenotreallocatedacrosstreatmentgroupstobalancetreatmentgroupsizeswhenahouseholdwasoutofscope.

WecalibratedexperimentalgroupsamplesizesbasedonthetreatmenteffectsizesfromHsu,Schmeiser,Haggerty,andNelson(2017),whichistheclosestpapertooursintermsof

institutionalsetting.Weselectedsamplesizestogive80%powerforourexpectedtreatment

HenriquesVolz,andWindle(2023)forinformationontheresponseratesandfieldperiodduration.IntheSCF,householdsmustanswerallcriticalquestionswithinthesurveyinstrumentforaninterviewtobecomplete.The2022SCFcodebookdenoteswhichquestionsarecritical(BoardofGovernorsoftheFederalReserveSystem

2023a).ThereisnoSCFstandardforpartialcompletes.

6Ourpre-registrationplanisontheOpenScienceFrameworkunder“2022SCFPre-paidIncentivesExperiment”at

/10.17605/OSF.IO/BXJNE

(Chang2023).Thepre-registrationplanalsoincludesaplanfor

analysisoftheexperimentaldata.

Page6of22

effectonresponserates.

7

Theresultingsamplesizes,showninTable1,areatleast1,200

householdspertreatmentgroupandareovereighttimeslargerthanthosefeaturedinHsu,

Schmeiser,Haggerty,andNelson’swork.Similarly,thetreatmentgroup-levelsamplesizesaretwotofourtimeslargerthanMesserandDillman(2011)andDykemaetal.(2015,2021),whoalsolookattheeffectsofsecondpre-paidincentivesonresponserates.Ourlargersamplesizesgiveusmuchmorestatisticalpowertodetecttreatmenteffects.

FieldingofourexperimentbeganbymailingallhouseholdsintheAPsampleapostcard

introducingtheSCFbetweenlateMarchandearlyApril2022.Thefirstenvelopemailing,sentonApril6th,2022withtheUSPS,includedaninvitationletteraskingtheheadofhouseholdto

providetheircontactinformationthroughasecurewebsiteandwasaccompaniedbythefirstpre-paidcashincentive—$5,$10,or$15,dependingonthegroup.

8

Asecondenvelope,mailedtwoweekslaterwiththeUSPS,includedapre-contactletterindicatingthatafieldinterviewerwouldbereachingouttodescribethestudyfurtherandscheduleaninterview.Thesecondenvelope

wasaccompaniedbyasecondpre-paidcashincentiveequalto$5or$10forthreeofthesix

experimentalgroups.Householdsintheremainingthreegroupsdidnotreceivecashwiththesecondenvelope.Allhouseholdsreceivedasecondenvelope,sotheamountofthesecondpre-

paidcashincentivewasnotdependentonsurveycompletion.Theinvitationletterandall

additionalmaterialsleftbytheinterviewerincludedatoll-freenumberthatarespondentcouldcallwithquestionsatanytimeduringtheexperimentortocompleteaninterview.

Envelopesincludedtransparentwindows,sohouseholdscouldseefromanexaminationoftheexteriorthattherewascashinside.SeeFigure1foramock-upofthemailingenvelope.

Envelopeswith$5hadasingle$5billinthewindow.Envelopeswith$10or$15hada$10billplacedinthewindow,andthosethatreceived$15alsohada$5billinsidetheenvelope,placeddirectlybehindthe$10bill,sothe$5billwasnotvisiblefromtheexterior.Therefore,thetotalamountofthepre-paidcashincentivewasonlydiscernablefromopeningtheenvelope.Other

visualelementswereminimizedtodistinguishtheenvelopefromcommercialsalesmaterialsandavoiddetractingfromthecashenclosure.

7Seeourpre-registrationplanforadditionaldetailsonthepowercalculations(Chang2023).

8Householdswhoprovidedtheircontactinformationthroughthesecurewebsitewerepaidanadditional$10,anamountthatwasnotdependentonthehousehold’sexperimentalgroup.

Page7of22

Figure1:MailingEnvelopeMock-Up

Front:

Back:

Intervieweroutreachtosampledhouseholdsbeganinmid-April2022.Initialcontactattemptsfocusedonhouseholdsthathadprovidedcontactinformationthroughthesecurewebsite.Toavoidexperimentereffects,fieldstaffwerenotawareofahousehold’spre-paidincentive

amount(s),thoughthisinformationcouldbevoluntarilygiventofieldstaffbythehousehold

afterinitialcontact.ThefieldperiodforourexperimentendedonJune1st,2022,8weeksafterwemailedthefirstenvelope.Completedinterviewsfollowedthesamepost-paidincentive

structureregardlessofpre-paidtreatmentassignment.

Page8of22

Model

Weusedpairwisecomparisonswithregressions/linearprobabilitymodelsoftheform:

yi,s=Σ∀sas,y+βytTeatmenti,s+Ei,s,y(1)

Weestimatedthemodelswithordinaryleastsquareswiththeomittedcategoryasthecontrol

group.Theoutcomeyforindividualiinstratum(NFA)sisafunctionofafullvectorofstratumdummies(as,y),followingtherecommendationofBruhnandMcKenzie(2009),andatreatmentgroupindicator.Thetreatmentindicatordependedonthehypothesisbeingtested.WeweightedanalysesbytheNFA’sinverseprobabilityofselectionintothe2022SCF.StandarderrorswerecalculatedusingHuber-Whiteheteroskedasticityconsistentstandarderrors(White,1980).

ResearchQuestionsandAnalysisMethods

Ourexperimentaldesignfacilitatedtestingthefollowinghypotheses:

Hypothesis1:Largervaluesoftotalcashpre-paidincentivesaffectsurveyoutcomes(theincentiveeffect).

.Hypothesis1a:$5vs.$10.Include$10/$0and$5/$0groupsinequation(1),testβy(treatmentgroupis$10/$0).

.Hypothesis1b:$10vs.$15.Includegroups$15/$0and$10/$0inequation(1),testβy(treatmentgroupis$15/$0).

.Hypothesis1c:$5vs.$15.Includegroups$15/$0and$5/$0inequation(1),testβy(treatmentgroupis$15/$0).

Hypothesis2:Asecondpre-paidincentive(througharemindereffect),controllingforthetotal

pre-paidincentivecashamountacrossenvelopes(theincentiveeffect),affectssurvey

outcomesundernochangeintheincentiveamountbetweenincentives(noslopeeffect).

.Include$5/$5and$10/$0groupsinequation(1),testβy(treatmentgroupis$5/$5).

Hypothesis3:Theincreasingordecreasingtheamountofpre-paidincentivecashbetweenmultipleenvelopes(theslopeeffect),controllingforthetotalcashamount(theincentive

effect)affectssurveyoutcomes.

.Hypothesis3a:Increasingslopeofincentive.Include$5/$10and$15/$0groupsinequation(1),testβy(treatmentgroupis$5/$10).

.Hypothesis3b:Decreasingslopeofincentive.Includegroups$10/$5and$15/$0inequation(1),testβy(treatmentgroupis$10/$5).

Wetestedsurveyoutcomesrelatedtoresponserates,interviewerburden,anddataquality.Forresponserates,weemployedAAPOR(2015)’sRR1withtheSCFstandardforcompleted

interviews,andalsolookedattheshareofinviteeswhoscheduledinterviewappointments.Forinterviewerburden,weanalyzed:thenumberofcontactattemptsneededtocompletean

interview;thenumberofcontactattemptsneededtoelicitawebsitevisitorascheduled

Page9of22

appointment;

9

andthedurationoftimebetweenwhenthefirstpre-paidwasmailedanda

completedinterview.Finally,fordataquality,welookedattwooutcomes.Thefirstisitem

responserates:theshareofquestionsthattherespondentcompletedfromtheinstrumentofthosethattherespondentwaseligiblefor.Theseconddataqualityoutcomeistheitemresponserateforquestionsthatrequiredadollarvalueasaresponsethatwereansweredwithanexactdollar

value,amongthoseinterviewscompletedwithatleast50%ofeligibledollarvaluequestionsbeingnon-missing.

10

Table2displayssamplesizesandaverageoutcomesforeachgroup.

11

9Wehadinitiallyspecifiedaddingcontactattemptsbeforeacallbacktothismeasure,butforoperationalreasonswewereunabletoeffectivelytrackcallbacks.

10Affectingthesurveyresponseratethroughourpre-paidincentivetreatmentmayalsohaveaffectedthe

participationofrespondentswhoaredifferentiallylikelytocompletethesurveywithmissinganswers.SeetheAppendixfordetailsonhowweusedSCFparadatatoconstructthesemeasures.SeeMeyer,Mok,andSullivan(2015)forareviewofsurveydataquality.

11OurresponseratesarenotcomparabletotheoverallSCFresponseratesbecauseallexperimentaloutcomesweremeasuredwithin8weeksofthestartoftheexperiment,incontrasttoanapproximatelyyear-longfieldperiodfortheentireSCF.

Page10of22

Table2:SampleSizes,ResponseRates,InterviewerBurden,andDataQualityby

ExperimentalGroup

ExperimentalGroup:

AmountofFirst/SecondPre-paidIncentive

$5/$0$5/$5$10/$0$5/$10$10/$5$15/$0

Total

N

2,1521,2921,2911,2931,2911,297

8,616

Worked

1,411865849844873870

5,712

%Worked

65.667.065.865.367.667.1

66.3

ResponseRates

Appointments(count)

270186200193201193

1,243

%Appointments

12.514.415.514.915.614.9

14.4

Completes(count)

140102111115110121

699

%Completes

6.57.98.68.98.59.3

8.1

InterviewerBurden

AvgAttemptsBeforeAppt

1.61.81.51.51.61.4

1.6

StandardDeviation

1.61.91.61.51.71.9

1.7

AvgAttemptsBeforeComplete

3.23.73.43.53.63.3

3.4

StandardDeviation

1.82.11.91.72.21.8

1.9

AvgDaysBeforeComplete

29.226.329.431.030.126.5

28.8

StandardDeviation

13.813.214.013.813.413.6

13.7

DataQuality

Avg%QuestionsAnswered

97.997.698.497.897.598.0

97.9

StandardDeviation

2.22.61.92.62.82.0

2.4