参考文稿标记化

•Text•EveryNLPtaskneedstodotextSegmen-ng/tokenizingwordsinrunningNormalizingwordSegmen-ngsentencesinrunningHowmanyFragments,filledSeuss’scatinthehatisdifferentfromotherLemma:samestem,partofspeech,roughwordcatandcats=sameWordform:thefullinflectedsurfacecatandcats=differentHowmanytheylaybackontheSanFranciscograssandlookedatthestarsandType:anelementoftheToken:aninstanceofthattypeinrunningHow15tokens(or13types(or12)(orHowmanyN=numberofV=vocabulary=setof|V|isthesizeofthe

ChurchandGale(1990):|V|>Tokens=Types=2.42031113SimpleTokeniza4onin(InspiredbyKenChurch’sUNIXforGivenatextfile,outputthewordtokensandtheirtr-scSortinSortinalphabeticalMergeandcountMergeandcounteach194572195...

2561563....Thefirststep:tr-sc’A-Za-z’’\n’<shakes.txt|Thesecondstep:tr-sc’A-Za-z’’\n’<shakes.txt|sort|WhatWhathappenedMoreMergingupperandlowertr‘A-Z’‘a-z’<shakes.txt|tr–sc‘A-Za-z’‘\n’|sort|uniqSor-ngthetr‘A-Z’‘a-z’<shakes.txt|tr–sc‘A-Za-z’‘\n’|sort|uniq–c|sort–niadIssuesinFinland’s FinlandFinlandsFinland’swhat’re,I’m,isn’t Whatare,Iam,is HewlettPackard stateoftheart lower-caselowercaselowercaseSan onetokenorm.p.h., Tokeniza4on:languageL'ensembleonetokenorL?L’?LeWantl’ensembletomatchwithunGermannouncompoundsarenot‘lifeinsurancecompanyGermaninforma-onretrievalneedscompoundDan

Tokeniza4on:languageChineseandJapanesenospacesbetween莎拉波娃现在居住在美国东南 的佛罗里 lives FurthercomplicatedinJapanese,withmul-plealphabetsDates/amountsinmul-pleWordTokeniza4oninAlsocalledWordChinesewordsarecomposedofCharactersaregenerally1syllableand1Averagewordis2.4charactersStandardbaselinesegmenta-onMaximumMatching(alsocalledWordSegmenta4onGivenawordlistofChinese,andaStartapointeratthebeginningoftheFindthelongestwordindic-onarythatmatchesthestringstar-ngatpointerMovethepointeroverthewordinGotoDan

thecatinthehatthetabledowntherethetabledownDoesn’tgenerallyworkinButworksastoni