新编文档-AutomaticEvaluationofRobustnessandDegradationin：在鲁棒性和降解的自动评价-精品文档

1、Automatic Evaluation of Robustness and Degradation in lagging and ParsingJohnny Bigert, Ola Knutssor Jonas SjoberghRoyal In stitute of lech no logy,Stockholm, SwedenContact： j ohnnykth seProblemNLP systems are often faced with noisy and ill-formed in put: How do we reliably evaluate the performanee

2、of NLP systems? Which methods of tagging and parsing are robust?Problem The performanee of a NLP system is sensitive to noisy and ill-formed input Manual evaluations of robustness is tedious and time-consuming Manual evaluation is difficult to compare and reproduce Resources with noisy data is rareO

3、utline Introduce artificial spelling errors using software (Missplel) Increasing error levels will affect the NLP system performanee Evaluation of degradation of tagging and parsing performanee (AutoEval)Introducing spelling errors Missplel (Bigert et al) Gen eric tool to in troduce huma like spelli

4、ng errors Highly con figurable Language and tag set independent Freeware, open sourcenada kth se/theory/humanlang/tools htmlIntroducing spelling errors Start with correct text(Swedish, the SUC corpus, Ejerhed et al) Introduce errors in, say, 10% of the words Spelling errors resulting in non-existing

5、 words only No change in parse treeIntroducing spelling errors 10 misspelled texts for each error level Elimi nate the in flue nee of cha nee Six error levels:0%, 1%, 2%, 5%, 10%, 20% 15 000 words with parse infoMissplel exampleLetters would be welcomeNN2VMOVBIAJO-NN1Litters would bee welcmoeNN2 dam

6、erau/wordexis 卜 no tagcha nge VMO okNN1 soun d/wordexist-tagcha ngeERR damerau/nowordexist-tagcha ngeTagging The texts were tagged using HMM tagger (Tnl； Brants) Brill tagger (fnTBL, Ngai & Florian) Baseline tagger (unigram)Parsi ng The tagged texts were parsed using GIA parser (Knutsson et al)

7、Baseline parser (unigram, CoNLL) GTA Granska text analyzer Rule-based Hand-crafted rules Con text-free formalismParsi ngParser output in IOB format (Ramshaw & Marcus):Viktigaste (the most important)APB|NPBredskapen (tools)NPIvid (in)PPBympning (grafting)NPB|PPIar (is)VCBannars (normally)ADVPBpap

8、per (paper)NPB|NPBoch (and)NPIpenna (pen)NPB|NPI/0menade (meant)VCBhan (he)NPBB 11111111 nD T11 nLLLLLLLLLLLLL ccccccccccccc0Evaluati onEvaluation was carried out using AutoEval (Bigert et al): Automated handling of pldin-text and XML input/output and data storage Script Ianguage Highly con figurabl

9、e and exte ndible (C+) Freeware, open sourcenada kth se/theory/humanlang/tools htmlEvaluati on Taggi ng: Accuracy, correct tag if exact match Parsi ng: Accuracy, correct row if exact match Precision and recall per phrase category correct if exact match after removing all other phrase types Clause bo

10、undary identification Precision and recall for CLBResultsResults of the tagging task (accuracy):Tagger0%1%2%5%10%20%Base85.284.4 (0-9)83.5 (1-9)81.2 (46)77.1(9.5)69.0(19.0)Brill94.593.8(0-7)93.0 d-5)90.9 (3-8)87.4(7.5)80.1(15.2)TnT95.595.0(0-5)94.3 (12)92.4 (3-2)89.5(6.2)833(12-7)ResultsResults of t

11、he parsing task (accuracy):Tagger0%1%2%5%10%20%Base81.080.2 (0-9)79.1(2.3)76.5 (5-5)72.4(10.6)64.5(20.3)Brill86.285.4 (0-9)84.5 (19)82.0 (48)78.0(9.5)70.3(18.4)TnT88.788.0 (0-7)87.2 (16)85.2 (3-9)817(78)75.1(15.3)Baseline parser: 59.2% at the 0% error level, using TnTCon clusi ons Automated method to determine the robustness of tagging and parsing un der the in flue nee of noisy in put No manual intervention Greatly simplifies repeated