用户偏爱LLM生成内容-直到发现是AI所为

UsersFavorLLM-GeneratedContent—UntilTheyKnowIt’sAI

PetrParshakov

HSEUniversityandMoscowSchoolof

IuliiaNaidenova

HSEUniversity

SofiaPaklina

HSEUniversity

NikitaMatkin

HSEUniversity

CornelNesseler

UniversityofStavanger

ManagementSKOLKOVO

Abstract

arXiv:2503.16458v1[cs.HC]23Feb2025

Inthispaper,weinvestigatehowindividualsevaluatehumanandlargelanguemodelsgen-eratedresponsestopopularquestionswhenthesourceofthecontentiseitherconcealedordisclosed.Throughacontrolledfieldex-periment,participantswerepresentedwithasetofquestions,eachaccompaniedbyare-sponsegeneratedbyeitherahumanoranAI.Inarandomizeddesign,halfoftheparticipantswereinformedoftheresponse’soriginwhiletheotherhalfremainedunaware.Ourfindingsindicatethat,overall,participantstendtopre-ferAI-generatedresponses.However,whentheAIoriginisrevealed,thispreferencedimin-ishessignificantly,suggestingthatevaluativejudgmentsareinfluencedbythedisclosureoftheresponse’sprovenanceratherthansolelybyitsquality.TheseresultsunderscoreabiasagainstAI-generatedcontent,highlightingthesocietalchallengeofimprovingtheperceptionofAIworkincontextswherequalityassess-mentsshouldbeparamount.

1Introduction

Therapidevolutionoflargelanguagemodels(LLMs)overrecentyearshasfundamentallytrans-formedthelandscapeoftextgenerationandhuman-computerinteraction.Thesemodelsarenowin-tegraltovariousapplications,rangingfromcus-tomerserviceandcontentcreationtopersuasivemessagingandpersonalizedcommunication.Re-centadvancementsinLLMshavedemonstratedsignificantpotentialtoenhancelaborproductiv-ityacrossvariousbusinessapplications,includingcustomercommunicationandcontentcreation(

Ay-

ersetal.,

2023;

Brynjolfssonetal.,

2025;

Zhang

andGosline,

2023

).LLMsarealsousefulincreat-ingpersuasivepublicmessages(

Karinshaketal.,

2023

)ormaintainingpersonalizedin-depthcon-versationstochangeindividual’sbeliefs(

Costello

etal.,

2024

).Understandinghowindividualsper-ceiveAI-generatedresponsesisessentialforen-

suringtheeffectivenessandacceptanceofthesetechnologiesinbusinessandsocialsettings.

AsLLM-generatedcontentincreasinglymirrorshuman-authoredtextinfluencyandcoherence,thechallengeofdistinguishingbetweenthetwosourceshasbecomemorecomplex.Priorresearchhashighlightedmeasurabledifferencesinlinguis-ticfeaturesandsentimentexpressionbetweenhu-manandAItexts,whilealsonotingthattheper-ceivedqualityofresponsesmayshiftwhentheoriginofthecontentisdisclosed.Thisdynamicisparticularlyimportantinsettingswheretrustandcredibilityareparamount,suchasincustomerin-teractionsorpubliccommunications.Manystud-iesfocusonspecificareas(e.g.,healthorpub-licannouncements)andneglecttheimportanceofgeneral-interesttexts.

OurstudyinvestigateshowresponsestopopularquestionsonplatformssuchasQuoraandStackOverflowareperceivediftheresponsecomesfromahumanorfromanLLM.Weexaminetheseper-ceptionsacrossavarietyofdomains,includingPhysicalSciences,LifeSciences,HealthSciences,SocialSciences,andHumanities,usingadiversesetofpopularquestions.Incorporatingrespon-dentcharacteristicssuchasgender,age,educa-tionalbackground,andprogrammingskills,ourresearchaimstoofferanuancedunderstandingofthefactorsthatdrivetrustandpreferenceincontentgeneration.

2Literaturereview

2.1HumanandLLMgeneratedtexts

Recentresearchhassystematicallyexaminedthedistinctionsbetweenhuman-generatedandAI-generatedtexts,revealingmeasurabledifferencesinsentencestructure,emotionexpression,andotherlinguisticfeatures(

Muñoz-Ortizetal.,

2024;

NituandDascalu,

2024

).Earlystudiesdemon-stratedthattraditionalmachinelearningclassifiers

couldeffectivelydifferentiatebetweenhumanandAI-generatedcontent;however,theadventofad-vancedlargelanguagemodels(LLMs)hassignifi-cantlycomplicatedthistask(

Hayawietal.,

2024

).Infact,AI-generatedtextshaveattimesmatchedorevenexceededhuman-writtentextsinspecificapplications,suchaspersuasivemessaging(

Karin-

shaketal.,

2023

)andprovidingwritingfeedbackineducation(

Escalanteetal.,

2023

).Theincreas-ingsophisticationofLLMshasledtoasignifi-cantconvergencebetweenAI-generatedtextandhuman-writtencontent,renderingthedistinctionbetweenthetwoincreasinglychallenging(

Ollivier

etal.,

2023;

Hayawietal.,

2024

).Asthesemodelsevolve,theyproducetextthatnotonlymimicshu-manwritingstylesbutalsoadherestothenuancesoflanguage,context,andcoherencethatcharacter-izeauthentichumancommunication.EvaluationsofexistingLLM-generatedtextdetectorshavere-portedinconsistencies(

Weber-Wulffetal.,

2023

)andhighfalsepositiverateswhenthesesystemsareappliedtohuman-authoredtexts(

Elkhatatetal.,

2023

).Inaddition,theobjectivityofAI-generatedcontentisalsoquestionable.TheliteraturehasrevealedinherentbiaseswithinoutputsproducedbyLLMs.Studieshavedocumentedsignificantgenderandracialbiases,notablyindepictionsofhealthcareprofessionalsandsurgeons,wheremalerepresentationsarefrequentlyfavored(

Menz

etal.,

2024;

Ceviketal.,

2024

).Politicalbiashasalsobeenobserved,withcertainplatformssuchasChatGPTexhibitingatendencytowardleft-leaningperspectives(

Motokietal.,

2024;

Rozado,

2024

).Moreover,LLMstendtomanifesthuman-likecon-tentbiases,asdemonstratedbytransmissionex-periments(

AcerbiandStubbersfield,

2023

)andlinguisticanalyses(

Fangetal.,

2024

).

2.2PerceptionofLLM-generatedcontent

ComparativestudiesrevealthathumanevaluatorsoftenstruggletoreliablydifferentiatebetweenAI-generatedandhuman-authoredcontent(

Boutadjine

etal.,

2024

).

ZhangandGosline

(2023

)demon-stratedthatgenerativeandaugmentedAIcontentisfrequentlyperceivedassuperiortothatproducedbyhumanexperts,evenwhenhumansutilizeAItools.However,disclosingthesourceofcontentnarrowstheperceivedqualitygap,suggestingabiasfavor-inghumancontributionsoverAI.Participantsratedcontentmorefavorablywhenattributedtohumanexperts,whereasawarenessofAIinvolvementhad

minimalimpactonperceptions.

Ayersetal.

(2023

)examinedtheabilityofanAIchatbot(ChatGPT)todeliverqualityandempatheticresponsestopatientquestionscomparedtophysicians.Theirfindingsrevealedthatchatbotresponseswerepreferredinthemajorityofevaluations,ratedhigherinqual-ity,anddeemedmoreempatheticthanthoseofphysicians.Notably,chatbot-generatedtextswerealsosignificantlylongerthanphysicianresponses.

Karinshaketal.

(2023

)highlightedthatlargelan-guagemodels(LLMs),particularlyGPT-3,canproducehigh-qualitypersuasivecontent;however,individualstendtopreferpublichealthmessagesoriginatingfromhumaninstitutionsratherthanAIsources.Somestudieshighlightthenuancedper-ceptionsandpreferencessurroundingAI-generatedcontentacrossvariousdomains.Forexample,

Es-

calanteetal.

(2023

)comparedhumantutorfeed-backwithAI-generatedfeedbackineducationalsettings,revealingmixedresults.Whileface-to-faceinteractionswithtutorsenhancedstudenten-gagement,AI-generatedfeedbackwasfavoredforitsclarityandspecificity.Theresearchby

Chen

etal.

(2024

)demonstratesthatconsumerspreferAI-generatedadswithagenticappeals,whilefa-voringhuman-createdadswithcommunalappeals.Thesefindingsunderscorethatcontextualfactorsandtheawarenessoftextoriginplayacriticalroleinshapinguserpreferences.

3Methodologyanddesign

3.1Datacollection

Thisstudyaimstoevaluateresponsesthatareofinteresttoabroadaudience.Toachievethis,theanalysisfocusesonpopularquestionsfromQuora,aplatformwhereuserscanpostquestionsandpro-videanswers,withthemostpopularresponsesprominentlydisplayed.Additionally,forthedo-mainofPhysicalSciencesandEngineering,ques-tionsfromStackOverflow,awidelyusedplatformforprogramming-relatedqueries,areincorporated.Theselectedquestionsarebroadlycategorizedintofivescientificareas:PhysicalSciencesandEngi-neering,LifeSciences,HealthSciences,SocialSciences,andHumanities.Thiscategorizationen-suresbalancedrepresentationacrossdomainsandpreventsthedominanceofanysinglearea.Whilethequestionsarenotstrictlyscientific,theyarede-signedtoappealtoageneralaudiencewithdiversebackgrounds.Examplesofsuchquestionsinclude,"WhatstartedWWII?","Whatwasthebestteamin

thehistoryofsports?",and"Whatroledoestherapyplayintreatinganxiety?".

FromQuoraandStackOverflow,fivequestionsareselectedforeachofthefivescientificareas,re-sultinginatotalof25questions.Thesequestionsarethenposedtofourprominentlargelanguagemodels(LLMs)—ChatGPT,Claude,Gemini,andLlama—selectedfortheirsuperiorperformanceintextgenerationatthetimeofthestudy.Eachques-tiongeneratesfiveresponses:fourfromtheLLMsandonefromahumanrespondent.Thecompletelistofquestions,organizedbyfield,ispresentedintheAppendix

A.1.

Theaveragelengthofhumanresponsesis1,515charactersor265words,whileresponsesfromtheLLMsvarybetween1,854and2,265characters.Detailedsummarystatisticsre-gardingresponselengths,measuredinbothchar-actersandwords,areprovidedintheAppendix

A.2.

Wecreatedasurveyinwhichparticipantsarerandomlypresentedwith5questions,eachaccom-paniedbytworesponses:onegeneratedbyanLLMandonebyahuman.Participantsarealsoaskedtoprovidedemographicinformation,includingage,gender,andeducationalbackground.Priortocom-mencingthesurvey,participantsareinformedthattheywillberequiredtochoosebetweentwore-sponsesforeachquestion.Importantly,arandompartoftheparticipantisnotinformedabouttheoriginoftheresponses(i.e.,whethertheyaregen-eratedbyanLLMorahuman)tomitigatepotentialbiases,suchastheHawthorneeffect(

Sedgwickand

Greenwood,

2015

).TheexperimentisregisteredontheSocialScienceRegistry

1.

Allparticipantswereinformedabouthowtheirdatawouldbeusedinthisstudy,andexplicitconsentwasobtainedpriortoparticipation.Wecontactedourinstitu-tion’sEthicalCommitteeandwereinformedthat,aswedonotstoreanypersonaldata,additionalethicalapprovalwasnotrequired.

Thesurveyisimplementedintwoformats:aTelegrambotandawebapplicationdevelopedus-ingStreamlit.Bothplatformsutilizethesamealgo-rithmicstructureandshareanidenticaldatabaseofquestions.Intotal,thestudyinvolves993partici-pants,with507respondentsusingtheTelegrambotand486thewebapplication.Fromthisinitialpool,130respondentsareexcludedfromthefinalsampleduetotheirfailuretoprovideanyresponses.

ThesurveyisdistributedthroughP,a

1SocialScienceRegistry

platformdesignedtomatchsurveyswithappropri-aterespondents.Participantsarecompensatedatarateof9GBP(approximately11USD)perhour.Thefinalsampleconsistsof846participants,whotakeanaverageof6.6minutestocompletethesur-vey.Thedemographiccompositionofthesampleis35%maleand45%female,withanaverageageof30years.Additionaldetailedsummarystatistics,includingbreakdownsbyawarenessgroups,areprovidedintheAppendix

B.1

andtheAppendix

B.2.

3.2Estimationstrategy

Toexaminewhetherindividualspreferhuman-orAI-generatedanswersbasedontheirawarenessofthesource,weemployedalogisticregressionanal-ysis.Specifically,werestructuredourdatasetsuchthattheunitofobservationisarespondent-answerpair.Wethenintroducedadummyvariable,hu-man,whichtakesthevalueof1iftherespondentselectedthehuman-generatedanswer.Thisvari-ableservesasthedependentvariable.

Amongtheindependentvariables,theprimaryvari-ableofinterestisaware,whichindicateswhetherarespondentcanseethesourceoftheanswer(hu-manorAImodel).Thecoefficientassociatedwithawarereflectswhetherknowledgeoftheanswer’ssourceinfluencestheprobabilityofselectingahuman-generatedresponse.Additionally,wein-cludedrespondentcharacteristicssuchasgender,age,levelofeducation,fieldofstudy,andprogram-mingskills.Wealsoaccountedforthedurationofthesurveyandthefieldofthequestion.Further-more,thebasicmodelspecificationispresentedbelow:

Pr(humani=1)=β0+β1awarei+β2femalei

+β3agei+β4durationi

+β5education_leveli

+β6education_fieldi+

+β7programming_skillsi

+β8modeli+β9question_fieldi

(1)

Toensurerobustness,weclusteredthestandarderrorsbygender.Toassessthevaryingeffectsofawarevariableacrossdifferentrespondentandquestioncharacteristics,wealsointroducedinter-actiontermswithgender,programmingskills,andquestionfield.

Figure1:Distributionofchosenanswersbygenderandsourceawareness

4Empiricalresults

4.1Exploratoryanalysis

Toillustratethepossiblefactorsthatmightinflu-encethechoicesbetweenanswersgeneratedbyhumansandAImodels,weperformedagraphi-calanalysis.Themostnoticeableandinterestingresultswerefoundinthedifferencesinchoicesacrossgenderandprogrammingskills.

Figure

1

presentsthedistributionofchosenan-swersbasedonwhetherparticipantswereawareofthesourceoftheresponseandtheirgender.Acrossallconditions,aconsistentpatternemerges,indicat-ingahigherpreferenceforAI-generatedresponsescomparedtohuman-generatedones.Aninterestingresultisthatwhenrespondentsareawareofthesourceoftheanswers,theyslightlytendtochoosehuman-generatedanswers,especiallyamongmalerespondents.

Figure

2

illustratesthedistributionofselectedanswersbasedonparticipants’programmingskillsandtheirawarenessofthesourceoftheresponses.TheresultsalsoindicateaconsistentpreferenceforAI-generatedresponsesacrossallconditions.Whenparticipantswereinformedabouttheoriginoftheanswers,thepreferenceforAI-generatedresponsesremaineddominant,althoughaslightincreaseintheselectionofhuman-generatedre-sponsescanbeobserved,particularlyamongthosewithprogrammingskills.However,thisdifferenceappearsmarginalandwouldrequiremorein-depth

Figure2:Distributionofchosenanswersbyprogram-mingskillsandsourceawareness

hypothesistesting.

Takentogether,thesefindingsindicatethatpriortechnicalknowledgedoesnotsignificantlyimpactthelikelihoodofselectingAI-generatedresponses.Evenwhenparticipantsaremadeawareofthesource,thepreferenceforAI-generatedanswersremainsstrong,reinforcingthenotionthatsuchresponsesareperceivedasequallyormorereliablecomparedtohuman-generatedalternatives.

4.2Regressionanalysis

Table

1

presentstheregressionresultsexamin-ingindividuals’preferencesforhuman-generatedanswersbasedontheirawarenessoftheanswersource.Thedependentvariableinallmodelsistheselectionofahuman-generatedresponse.Controlvariablesforeducationlevel,fieldofstudy,pro-grammingskills,AImodel,andquestionfieldareincludedinallmodels.

Acrossallspecifications,awarenessofthean-swersourcehasasignificantimpactonindividuals’preferences.InModels(1),(2),and(4),thein-dividualcoefficientforawarenessispositiveandstatisticallysignificant,suggestingthatwhenre-spondentsknowwhetherananswerisgeneratedbyahumanoranAI,theyaremorelikelytopreferhuman-generatedresponses.

Thevariablefemaleisconsistentlypositiveandsignificantacrossallmodels,indicatingthatfe-malerespondentsaremorelikelytopreferhuman-generatedresponsescomparedtomalerespondents.

Dependentvariable:

Aigeneratedanswer(=0),humangeneratedanswer(=1)

(1)(2)(3)(4)

AwareofAI-answer

0.109**

0.146***

—0.112

0.206***

(0.051)

(0.017)

(0.071)

(0.024)

Female

0.047***

0.090***

0.046***

0.047***

(0.015)

(0.024)

(0.013)

(0.015)

Age

—0.012**

—0.012**

—0.012**

—0.012**

(0.005)

(0.005)

(0.005)

(0.005)

Duration

0.0005

0.001

0.002

0.0004

(0.003)

(0.003)

(0.004)

(0.003)

AwareofAI-answer·Humanities

—0.111***

(0.017)

AwareofAI-answer·LifeSciences

—0.130**

(0.061)

AwareofAI-answer·PhysicalSciencesandEngineering

—0.068

(0.244)

AwareofAI-answer·SocialSciences

—0.184

(0.125)

AwareofAI-answer·Female

—0.089***

(0.023)

AwareofAI-answer·Programmingskills

0.322**

(0.126)

EducationLevel

Yes

Yes

Yes

Yes

EducationField

Yes

Yes

Yes

Yes

ProgrammingSkills

Yes

Yes

Yes

Yes

AIModel

Yes

Yes

Yes

Yes

QuestionField

Yes

Yes

Yes

Yes

Observations

3,206

3,206

3,206

3,206

LogLikelihood

—1,951.209

—1,951.050

—1,949.411

—1,950.902

Note:*p<0.1;**p<0.05;***p<0.01

Table1:Logisticregressionresults

Agehasasmallbutsignificantnegativeeffect,im-plyingthatolderindividualsareslightlylesslikelytopreferhuman-generatedanswers.

TheinteractioneffectsinModels(2)–(4)pro-videfurtherinsights.InModel(2),theinteractionbetweenawarenessofthesourceandfemaleisneg-ativeandsignificant,indicatingthattheeffectofknowingthesourcediffersbygender.Sincethejointeffectoffemaleanditsinteractionwithaware-nessofthesourceisnotstatisticallysignificant,weobservethepositiveeffectofknowingthesourceonlyformalerespondents.Specifically,formalerespondents,knowingwhetherananswerisgener-atedbyahumanorAIincreasestheprobabilityofselectingahuman-generatedresponse.

Model(3)includesaninteractionbetweenaware-nessofthesourceandprogrammingskills,whichispositiveandstatisticallysignificant,indicatingthatindividualswithprogrammingexpertisearemorelikelytopreferhuman-generatedresponseswhentheyareawareofthesource.

Model(4)explorestheroleofthefieldofthequestion.Theinteractiontermsshowthatforques-tionsinHumanitiesandLifeSciences,respon-dentsexhibitasignificantnegativepreferenceforhuman-generatedanswerswhentheyareawareofthesource.However,nosignificanteffectsareobservedforquestionsinPhysicalSciencesandEngineeringorSocialSciences.

Overall,thesefindingshighlightthatknowingthesourceofananswerinfluencesindividuals’pref-erences,withvariationsbygender,programmingexpertise,andthefieldofthequestion.

5Conclusion

Thisstudyhasinvestigatedtheinfluenceofsourceawarenessonindividuals’preferencesbetweenhuman-generatedandAI-generatedresponses.Ourempiricalresultsindicatethat,althoughrespon-dentsgenerallyshowatendencytofavorAI-generatedcontent,thedisclosureofananswer’sorigininducesameasurableshifttowardhuman-generatedresponses.Moreover,theextentofthisshiftiscontingentuponrespondentcharacteristicssuchasgenderandprogrammingexpertise,aswellasthecontextualdomainofthequestion.Thesefindingsnotonlyextendtheliteratureontheper-ceptualdifferencesbetweenhumanandAItexts

(Boutadjineetal.,

2024;

ZhangandGosline,

2023

)butalsocomplementpreviousstudiesthathavenotedtheimpactofsourcedisclosureoncontent

perception(

Ayersetal.,

2023;

Karinshaketal.,

2023

).

Interestingly,womendemonstrateastrongerpreferenceforhumanresponsescomparedtomen,potentiallyduetodifferencesinstyleortextlengthpreferences,asevidencedinAppendix

A.2.

Whilethesepreferencesremainconsistentregardlessofknowledgeabouttheanswer’sorigin,menexhibitasignificantresponsetothedisclosureofwhetherahumanoranLLMauthoredthetext.Thefind-ingsontext-lengthdifferencesbetweenhumanandAI-generatedtextsalignwiththosereportedby

Karinshaketal.

(2023

).

Incontrasttoearlierresearchthatreportedaper-vasivedifficultyamongevaluatorsindistinguishingbetweenAIandhuman-authoredtexts,ouranal-ysisrevealsthatsourceawarenesscanleadtoanuancedreallocationoftrustdependingonbothindividualanddisciplinaryfactors.WhilepreviousstudieshavelargelyemphasizedthechallengesindetectingAI-generatedcontent(

Weber-Wulffetal.,

2023;

Elkhatatetal.,

2023

),ourfindingssuggestthattransparencyregardingcontentoriginplaysacriticalroleinshapingevaluativejudgments,par-ticularlyamongsubgroupswithspecifictechnicalproficienciesanddemographicprofiles.

Theimplicationsoftheseresultsaresignificantforthedeploymentoflargelanguagemodelsinvarioussectors.AsAI-generatedcontentbecomesincreasinglyprevalentinbusinesscommunications,publicmessaging,andeducationalsettings,ensur-ingthatendusersunderstandtheoriginsofthecontentmayenhanceitsperceivedcredibilityandeffectiveness.Futureresearchshouldfurtherex-aminethelong-termeffectsofsourcetransparency,extendtheinquirytoencompassnon-textualmedia,andexplorethedynamicsofaugmentedhuman-AIcollaboration.Suchinvestigationswillbeessentialinrefiningourunderstandingofhowbesttointe-grateAItechnologiesintocontextswheretrustandauthenticityremainparamount.

6Limitations

Itisimportanttonotethatourresearchislimitedtotextualcontentanddoesnotencompassnon-textualformssuchasgraphicsoraudio,whicharewidelyutilizedincommunication.Additionally,maintain-inghumanoversightremainscrucialtoensurethatgenerativeAI-producedcontentisappropriateforsensitivetopicsandtopreventthedisseminationofunsuitablematerial.Furthermore,ourstudydoes

notexploreperceptionsofaugmentedhumanoraugmentedAI-generatedcontent,anemergingandpromisingareaofresearch(

Vaccaroetal.,

2024;

ZhangandGosline,

2023

).

Thisstudyfacesseverallimitationsthatwar-rantcarefulconsideration.First,ouranalysisisconfinedtotextualcontentanddoesnotencom-passnon-textualmediasuchasgraphics,audio,orvideo,whichareincreasinglyintegraltomoderncommunication.Thisfocusontextmaylimitthegeneralizabilityofourfindingsincontextswheremultimodalcontentplaysacriticalrole.

Anotherlimitationrelatestoourdatacollectionmethods.AlthoughweutilizedplatformssuchasQuoraandStackOverflowtoensureadiversesetofquestionsandrespondentbackgrounds,theon-linenatureofthesesourcesmayintroduceselectionbiases.Thesample,whilesubstantial,mightnotfullycapturethebroaderpopulation’scultural,de-mographic,ortechnologicaldiversity,whichcouldinfluencetheobservedpreferencesandperceptions.

Moreover,ourinvestigationintoevaluativejudg-mentsofcontentoriginreliesonself-reportedre-sponsesandobservablechoiceswithinanexperi-mentalframework.Despiteeffortstomitigatepo-tentialbiases,suchastheHawthorneeffect,thereremainsariskofdemandeffectsorsocialdesir-abilityinfluencingparticipants’selections.Thismethodologicalconstraintsuggeststhatfurtherstudiesemployingalternativedesignsoradditionalqualitativemeasuresmaybeneededtovalidateourconclusions.

Additionally,whileourworkcontributestounderstandingthedistinctionsbetweenhuman-generatedandAI-generatedtext,itdoesnotex-ploretheemergingdomainofaugmentedhumanoraugmentedAI-generatedcontent(

Vaccaroetal.,

2024;

ZhangandGosline,

2023

).Thedynamicsofcollaborativecontentcreation,wherehumancre-ativityisintertwinedwithAIassistance,presentapromisingavenueforfutureresearch.Similarly,theinfluenceofnon-textualelementsandthein-tegrationofmultimodalcommunicationonuserperceptionsremainopenquestions.

Finally,itisimportanttoacknowledgethattheplatformsandcontextsinwhichthedatawerecol-lectedmayimposetheirownnormsandbiasesonbothhumanandAI-generatedresponses.Thesecontextualfactorscouldaffectthestyle,substance,andevaluativejudgmentsofthecontent.Futureresearchshouldaimtoreplicateandextendthese

findingsacrossdifferentmediaandculturalsettingstoenhanceourunderstandingofhowsourceaware-nessinfluencescontentperceptioninabroaderspectrumofcommunicationenvironments.

7EthicsStatement

ThisworkadherestotheACLCodeofEthicsandcomplieswiththeethicalguidelinesestablishedforACL2023.Inconductingthisresearch,weensuredthatalldatacollectionprocesseswereperformedinaccordancewithethicalstandards,includingob-taininginformedconsentfromallparticipantsandanonymizingthecollecteddatatoprotectprivacy.Werecognizethatadvancesinlargelanguagemod-elsandtheincreasingprevalenceofAI-generatedcontenthavesignificantsocietalimplications.Ac-cordingly,ourstudyhasbeendesignedwithacom-mitmenttotransparencyandaccountability.Wehavecarefullyconsideredpotentialethicalrisks,includingthepropagationofbiasesandthemis-useofAI-generatedcontent,andhaveincorporatedmeasurestomitigatetheseconcerns.Ouraimistocontributetoabetterunderstandingofhowsourceawarenessinfluencescontentperception,whileun-derscoringtheimportanceofhumanoversightinthedeploymentofAItechnologies.

7.1PotentialRisks

Potentialrisksassociatedwiththisresearchpri-marilyarisefromthemisinterpretationandmis-useofourempiricalfindings.Inparticular,theobservedshiftinuserpreferences