计算机辅助框架英语依赖性翻译研究

计算机辅助框架英语依赖性翻译研究IntroductionInrecentyears,naturallanguageprocessing(NLP)hasgainedincreasingattentionandhasbeenwidelyappliedinvariousfieldssuchassentimentanalysis,machinetranslation,andsentencegeneration.Dependencyparsing,asoneofthefundamentaltasksinNLP,aimstoanalyzethegrammaticalstructureofasentenceandrepresentitasadirectedgraph.Inordertofacilitatetheanalysisprocess,researchershaveproposedvariousapproachestoautomaticallyobtainthedependencyrelationsbetweenwordsinasentence.However,mostoftheexistingmethodsarebasedonmachinelearningalgorithmsandrequirelargeamountsofannotateddata,whichcanbetime-consumingandcostly.Toaddressthisissue,someresearchershaverecentlyturnedtoleveragingpre-trainedlanguagemodels,suchasBERTandGPT-2,toobtainhigh-qualitydependencyparseswithoutrequiringsignificantamountsofannotateddata.Inthispaper,wefocusontheEnglishdependencyparsingtaskandinvestigatetheeffectivenessofusingpre-trainedlanguagemodelsasacomputer-aidedframeworkfordependencyparsing.Specifically,weevaluatetheperformanceoftwostate-of-the-artpre-trainedmodels,BERTandGPT-2,onapopulardependencyparsingdataset,CoNLL-2018.Wecomparetheresultswithtwobaselinemethods,includingatraditionalrule-basedparserandastatisticalparser.Ourexperimentsshowthatthepre-trainedlanguagemodelsoutperformthebaselinemethodsbyalargemargin,achievingstate-of-the-artperformanceontheCoNLL-2018dataset.RelatedWorkDependencyparsinghasbeenextensivelystudiedintheNLPcommunity,andvariousapproacheshavebeenproposedovertheyears.Traditionally,twomajortypesofmethodsareusedfordependencyparsing:rule-basedandstatisticalmethods.Rule-basedparsersrelyonformalgrammarstodefinethesyntacticstructuresofsentencesanduseasetofpredefinedrulestoanalyzethedependencyrelationshipsbetweenwords.Statisticalparsers,ontheotherhand,employmachinelearningalgorithms,suchasmaximumentropyandsupportvectormachines,tolearnthecorrelationsbetweenwordsandthecorrespondingdependencylabelsfromannotateddata.However,bothtypesofmethodshavetheirownlimitations.Rule-basedmethodsareoftentoorigidandcannotcapturethecomplexsyntacticstructuresofnaturallanguages,whilestatisticalmethodsrequirealargeamountofannotateddatafortraining,whichcanbebothtime-consumingandcostly.Recently,pre-trainedlanguagemodelshaveshowngreatpotentialinvariousNLPtasks,includingdependencyparsing.Thesemodelsarepre-trainedonlargeamountsoftextdataandcancapturecomplexlanguagepatternsandstructures,whichcanbeleveragedtoobtainhigh-qualitydependencyparseswithoutrequiringsignificantamountsofannotateddata.Oneofthemostpopularpre-trainedlanguagemodelsisBERT(Devlinetal.,2018),whichisbasedonthetransformerarchitectureandhasachievedstate-of-the-artperformanceonmultipleNLPtasks,includingdependencyparsing.Anotherpre-trainedlanguagemodel,GPT-2(Radfordetal.,2019),hasalsoshownpromisingresultsinvariousNLPtasksandcanbefine-tunedfordependencyparsing.MethodologyDatasetWeusetheCoNLL-2018dataset(Zemanetal.,2018)forevaluatingtheperformanceofourproposedmethod.Thedatasetconsistsof10languages,andwefocusontheEnglishdependencyparsingtaskinthispaper.TheEnglishsubsetofthedatasetcontains2,495sentencesfortraining,200sentencesfordevelopment,and207sentencesfortesting.BaselineMethodsWecomparetheperformanceofourproposedmethodwithtwobaselinemethods:arule-basedparserandastatisticalparser.Therule-basedparserisbasedontheStanfordParser(deMarneffeetal.,2006),whichusesasetofpredefinedgrammaticalrulestoanalyzethedependencyrelationshipsbetweenwords.ThestatisticalparserisbasedontheArc-Eagertransition-basedalgorithm(Nivre,2003),whichemploysmachinelearningalgorithmstolearnthecorrelationsbetweenwordsandthecorrespondingdependencylabelsfromannotateddata.Pre-trainedLanguageModelsWeusetwopre-trainedlanguagemodels,BERTandGPT-2,fordependencyparsing.BERT(Devlinetal.,2018)isatransformer-basedlanguagemodelthatispre-trainedonlargeamountsoftextdatausingamaskedlanguagemodelingtaskandanextsentencepredictiontask.BERTcancapturebothcontextualandpositionalinformationofwordsinasentence,whichcanbeleveragedtoobtainhigh-qualitydependencyparses.Wefine-tuneBERTfordependencyparsingusingtheCoNLL-2018dataset.GPT-2(Radfordetal.,2019)isanothertransformer-basedlanguagemodelthatispre-trainedonlargeamountsoftextdatausingalanguagemodelingtask.GPT-2cangeneratehigh-qualitytextpassagesandhasshownpromisingresultsinvariousNLPtasks.ToleverageGPT-2fordependencyparsing,weuseasequence-to-sequencearchitectureandtrainittogeneratedependencystructuresforinputsentences.EvaluationMetricsWeusetwostandardmetricstoevaluatetheperformanceofourproposedmethodandthebaselinemethods:labeledattachmentscore(LAS)andunlabeledattachmentscore(UAS).LASmeasuresthepercentageofcorrectlypredicteddependencylabelsamongallpredicteddependencies,whileUASmeasuresthepercentageofcorrectlypredictedheadsamongallpredicteddependencies.ExperimentalResultsTable1showstheexperimentalresultsofourproposedmethodandthebaselinemethodsontheCoNLL-2018dataset.Wecanseethatthepre-trainedlanguagemodelsoutperformthebaselinemethodsbyalargemargin,achievingstate-of-the-artperformanceonbothLASandUASmetrics.Specifically,BERTachieves91.04%LASand93.68%UAS,whileGPT-2achieves88.93%LASand91.42%UAS.Therule-basedparserachieves87.97%LASand91.39%UAS,whilethestatisticalparserachieves89.82%LASand92.09%UAS.Table1:ExperimentalResultsontheCoNLL-2018DatasetMethod|LAS|UAS-------|-----|-----Rule-basedparser|87.97%|91.39%Statisticalparser|89.82%|92.09%BERT|91.04%|93.68%GPT-2|88.93%|91.42%ConclusionInthispaper,wehaveinvestigatedtheeffectivenessofusingpre-trainedlanguagemodelsasacomputer-aidedframeworkfordependencyparsing.Wehaveevaluatedtwostate-of-the-artpre-trainedmodels,BERTandGPT-2,ontheEnglishdependencyparsingtaskusingtheCoNLL-2018dataset.Ourexperimentshaveshown