Business models for Dictionaries and NLP.pdf

BusinessmodelsforDictionariesandNLP AdamKilgarri November Abstract NLPneedsdictionaries anddictionary makerscanuseNLPtomakebetterdictio naries sothereisgreatpotentialforsynergybetweenthetwoactivities Todate there hasbeenonlyverylimitedcollaboration Thetworeasonsforthisare a dictionary publishers concernsregardingintellectualproperty and b thedi erentlanguagesthat lexicographersandNLPresearchersspeak InthispaperIpresentamodelforovercoming the rstandsuggestssomestrategiesforthesecond Introduction NLPneedsdictionaries anddictionary makerscanuseNLPtomakebetterdictionaries so thereisgreatpotentialforsynergybetweenthetwoactivities Thereisamplemotivation forNLPtocourtdictionarypublishers andviceversa Todate NLPresearchhasuseddictionariesanddictionarieshaveusedNLP butthetwo processeshavenotbeenbroughttogether TheNLPthathasgoneintomakingdictionaries hasnotbeentheNLPthatwasbasedonanearlierversionofthesamedictionary orindeedof anypublisheddictionaries TheNLPhaseitherbeendictionary independentifbroughtin fromoutside ordevelopedinhouse WhileNLPgroupshavemadeinnumerablecorrections improvements additionsandextensionstothedictionarydatabasestheyhavelicensedfrom publishers thesechangeshaveneverbeenusedbythepublishertoimprovethenextprinting orthenexteditionofthedictionary Therehavebeentworeasonsforthisfailureofsynergy Firstly dictionarypublishersare concernedaboutanumberofquestionsrelatingtointellectualproperty Secondly lexicog raphersandNLPresearchersspeakverydi erentlanguages soitisnotstraightforwardto developtheatmosphereoftrustinwhichthelexicographersfullyunderstandwhattheNLP researchhasdoneorbelievethattheNLP enhancements totheirdictionaryareusefulto them NLP NaturalLanguageProcessing isheretakentodescribeallthosetechnologiesthatmanipulate humanlanguageinputsbycomputer examplesbeingautomaticpart of speechtagging parsing concordancing machinetranslation informationextractionandtextgeneration Languageengineering and computational linguistics arenear synonyms SophisticatedNLPsoftwarethathasbeenusedforEnglishincludestaggers egCOBUILDusedENGCG fromHelsinki Longman OUPandChambers HarrapusedtheBNC whichwasPOS taggedbyCLAWS from Lancaster andparsers theexperimentalHECTORprojectusedtheFidditchparser Hindle These programsdonotusepublishers dictionaries Nordothestatistics basedcollocation ndersthathavebeen quitewidelyused Anexception amongEnglishdictionaries is CIDE wherethedevelopmentof taggingsoftwarewascloselyintegratedwiththedictionaryproductionprocess IntheremainderofthepaperI rstdescribewhatthetwosideshavetogainfromeach other thenthehistoryoftheissue andthenpresentamutuallybene cialbusinessmodel I thencommentonthelanguageproblem Thepapermayseemtoreplicatethe SpecialIssueofthisjournaleditedbyBran Boguraev Boguraev Indeed theoverallmotivationisthesame topromotemutual understandingandcollaborationbetweenthetwo elds ButwhereastheSpecialIssueset outtodescribewhatbene tstherecouldbe inthispaper thisisallbutassumed withthe followingtwosectionssummarisingthemainthemes Rather welookattheinstitutional andbusinessreasonswhymorecollaborationhasnothappenedandconsiderhowtheycan beovercome DictionariesforNLP NLPhascausetocourtthepublishersbecauseitneedslexicalinformationforalmostevery thingitdoes Lexicalinformationtendstobeexpensiveanddi culttoproduce solicensing itfromthosewhohavealreadyinvestedinit thedictionarypublishers makesgood sense Thisappliestothefullrangeoflexicalinformation orthography phonology mor phology syntax semantics pragmatics translations domain thesauri collocation Allof thesehave overthelasttwenty veyears beenextractedfromelectronicversionsofdictio naries onatypesetter stape orCD ROM orotherelectronicmedium andusedinNLP forresearchinallcases forproductsinmost NLPhasfounddictionariesvaluableforall thesepurposesinspiteoftheoverheadsof ndingtherelevantinformationinwhatisusu allyapoorly structuredinput andtheinevitableerrors inconsistenciesandomissions They wouldbemuchmoreenthusiasticifitwerenotforthesefailings NLPforbetterdictionaries DictionarypublishershavecausetocourtNLPforanumberofreasons Themostobviousis thatthereismoneytobemadefromlicensingarrangements Largesumsfrequentlychange hands ThishasbeenthedominantmotivationfordictionarypublishersapproachingNLP groups Adistinctionmustbemadeatthispointbetweendictionarypublishersandlexicog raphers Thismotivationhasbeenvividinthemindsofthepublishers butoflittleinterest tothelexicographers ThemoreintriguingreasonsrelatetothepotentialNLPhastoimprovedictionaryquality Ofthese twovarietiescanbedistinguished thosethatactonadictionarydatabase and thosethatactontextcorpora Bene tsofNLPappliedtodictionarydatabases Thebene tsarecloselyrelatedtothebene tstothedictionaryproductionprocessof simply havingthedictionaryinadatabaseatall Manyerrorscanbedetectedautomatically and consistencyofallmanneroffeatureschecked Super cially thesemayseemtobesimple databasefunctions However manyfeaturescanonlybecheckedforconsistencyifthedata modelunderlyingthedatabaseisbasedonanunderstandingoflexicography Theskills requiredtomodelthedatalieattheintersectionoflinguistics lexicographyandcomputation viz inNLP Moreover therearealwaysfurtherchecksandanalyseswhichgobeyondanythingthe databasewillprovidedirectly Forexample onemightwanttochecktowhatextentthewords inthesamethesauralcategorygetcorrespondingde nitions AnsweringthiswillrequireNLP expertise aswellasbothawell structureddatabase sotherelevantdataisaccessible and lexicographicexpertise toworkoutwherethenon correspondencesarejusti ed Acommonsituationisthatagooddatabasemodelisavailablebutthedictionarydatabase doesnot tit Whatisthenrequiredisaprocesssometimescalled up translation of carryingthedataoverfromtheexistingformatintothenewone Standardly muchofthis willbedoablebysimplytranslatingmark up butinmanycases themappingwillbeone to many orwillbecontext dependent ordistinctionswillberequiredforthenewdatamodel whichareimplicitinthetextorfont changesoftheexistingdictionary Inmostsuchcases a veryhighpercentageoftheup translationcanbedoneautomatically butitisanNLPtask todoit NLPappliedtotextcorpora Thearrivalofelectronictextcorporaiscausingarevolutioninlexicography Previously the primarysourceofevidenceforaword sbehaviourwasthelexicographer sintuition Now whereverpublishershavebeenabletoassemblealargetextcorpus andhaveconcordancing technology itisthecorpus Thereiswidespreadagreementthatthisisahugeboonfor lexicography Yetitclearlyintroducesmanynewchallenges Organisingthelexicographer sworking environmentsothats hehasinstantaccesstoaconcordanceforthewords heisworkingonis noteasy particularlywherelexicographersworkathome Eventhesimplestconcordancing programrequiressomeminimalNLPinput Minimally itneedstoknowwhichcharactersare punctuationcharacters soshouldbeignoredwhende ningwhatawordis ifaspaceistreated astheonlyword delimiter thesearchterm butter willnotmatchthecorpusobject butter InEnglish thecharacters and presentproblemsatthislevel Morphological analysiswillmakethecorpusmoreuseful asthenallformsofawordcanbefoundwitha simplesearch ForEnglish withitssimplemorphology itiseasyenoughtosearchforall fourformsseparately orwitharegularexpression ForFinnish itisnot Thecorpuswill bemoreusefulagainifitispart of speechtagged word sensedisambiguatedorparsed see eg TapanainenandJ arvinen Eachlevelofannotationallowsthelexicographermore controlinsearchingthecorpusforthelinguisticallyinterestingphenomena sothat when lookingthroughtheconcordance noiseisreducedandduplicatingpatternsdonotneedto bescannedexhaustively Beyondtheconcordance therearecorpusstatisticsandmachinelearning Statistically organisedcollocationlistshaveprovedtheirworthforallthedictionaryprojectsthathave hadaccesstoverylargecorpora ChurchandHanks thepaperthatopenedthe currentdebateinNLPaboutcollocationstatistics isanoteworthyexampleoflexicogra phy NLPcollaboration Acquiringlexicalinformationfromcorporaiscurrentlyoneofthe mostdynamicareasinNLP Mypurposeinsayingthisisnottoputfearofredundancyinthe heartsoflexicographersbuttoindicatehowmuchmoresatisfactorytheirworkistobecome whenthetoolsattheirdisposalaresomuchmorepowerful Thetechniquestendto nd manyplausiblehypothesesinacorpus butareunabletosortthewheatfromthecha or evidently toassignmeaningstothepatternsthey nd Thelexicographer staskremainsthe sameatheartbutwithlessdrudge History Therecenthistoryofdictionary NLPinteractionbeginswith Amsler and Michiels Eachtookatypesetterstapeforadictionary Amsler sagendawastoseewhetherthe dictionarycouldbeusedasasourceofgeneralknowledgeoftheeverydayworld knowledge that forexamplesalsationsaredogs forusewitharti cialintelligenceprograms Michiels exploredhowthemorespeci callylinguisticinformationmightbeusedforNLP Bothofthese agendaswerepursuedatlengthinthecourseofthe s notablyintheEUACQUILEX project attheComputerResearchLaboratoryatNewMexicoStateUniversity andatIBM inYorktownHeights BoguraevandBriscoe representstheactivityatitshighwater mark with Byrdetal demonstratingtheascent Wilks Slator andGuthrie reviewingthewhole and IdeandVeronis o eringapostmortem theirsubtitleis Havewewastedourtime Asaresearchtopic theuseofdictionarydatabaseshasnowrunitscourse Thishas anumberofinterpretations ThemostpositiveisthattheNLPworldhasnowdeveloped afairunderstandingofwhatinformationdictionarydatabasescontainandwhatisinvolved inextractingitforNLPuse sothatthetopichasshiftedfrom research to development Thiswouldo eraperspectiveonwhyalmostalltheresearchwasonEnglishdictionaries theywereusedfortestingoutmethods andthemethodscouldthenbeusedfordictionaries forotherlanguages Thenextinterpretationissimplythattheresearchfashionhaschanged particularlyto languagecorpora andmethodsforextractinglexicalinformationfromthem Thenthereistheinterpretationthatmotivatesthispaper Intheacademicsector work donetoenhanceadictionaryisusuallyworkwastedifothersarenotpermittedtousethe enhancedresource Publishers anxiousaboutintellectualproperty havefrequentlynotper mittedit AnumberofNLPworkershaveexplicitlychosennottodoanyfurtherworkon dictionaries exceptthoseinthepublicdomain forthisreason theysuspectthatanysuch work howevergood willbedestinedforoblivion Intheevent theacademicworld sownproductarrived Asof WordNethasbeen availablefree overthewebandwithoutconstraint WordNethasbeenthelexicalresourceof the s andvariouspeoplehavearguedthatWordNetsensesarethedefactostandardfor NLP WordNetwasproducedbylinguistsandpsychologists accordingtoapsycholinguistic agenda anditssuitabilityforNLPresearchremainsalivelytopicofdebate But asagainst thepracticalitiesofgettingholdofit thesepuristconcernshavecarriedlittleweightand everyoneusesWordNetnow WordNetaddressedsemantics forsyntax theNLPcommunityinvestedinCOMLEX and morerecently NOMLEX Grishman MacLeod andMeyers MacLeodetal COMLEXandNOMLEXaredesignedfromtheoutsettomeettheneedsoftheNLP communityandthereiscurrentlyworkinprogressonlinkingtheCOMLEXsyntaxpatterns totheWordNetwordsenses The EuralexconferenceinLi egemayhaveinauguratedanewphaseinthedebate withtheimpetusthistimecomingfromlexicography WhereasearlierEURALEXeshave beenresistanttocomputationorvieweditwarily atLi ege thisauthor sperceptionwas TherearealternativestoCOMLEX OtherlexiconssuchasXTAG Group andANLT Carrolland Grover havealsobeendevelopedbytheNLPcommunity TheANLTlexiconwasdevelopedinthe mid eighties itincludedmaterialfromadictionaryandcopyrightissuespreventeditbeingusedwidelyfor NLPuntilthemid nineties thatitwasuniversallyacceptedthatNLPhadaroletoplayindictionaryproduction The questionwasnolongerwhethertouseit buthowtouseitwell Exceptinthe dictionary use sessions itwashardto ndpaperswhichdidnotassumetheavailabilityofcorporaand concordancingsoftwareormore Onepaperofparticularsaliencetotheargumentwasjointly presentedbyUlrichHeid anNLPacademicandVincentDocherty adictionarypublisher DochertyandHeid here atlast thestateoftheartinNLPwasbeingusedto provideinputstolexicographersforcompilingabetterdictionaryforpeople Publishers anxieties Theovertreasonwhypublishers dictionarieshavenotbeenmorewidelyusediscopyright Thepublishers argumentissimpleanddirect thepublisher stradeisinintellectualproperty soitisnotreasonabletoexpectthemtogivethedictionaryawayorriskitfallingintothe publicdomain Thereareseveralthreadstothecase The rstis simply piracy Ifadictionarystarts beingcopiedfreely or worsestill startsbeingcopiedforafeebutwherethefeedoesnot gotothelegalcopyrightowner thepublisheristheloser Thepublisherwantstoavoid thishappeningaboveallelse AlicenceagreementwithanNLPresearchgroupprovidesan avenuebywhichadictionarymay nditswaytobeingillegallycopiedandre copied Theotherthreadsrelatetothepossiblere useorre salebythepublisherofversionsof theirdictionarywhichhavebeenupgradedinsomewaybyanNLPresearchgroup Theissueshereconcerncontamination Whereadictionarypublisheristhesoleowner ofthecopyrightforadictionary itwouldliketokeepitthatway Therewillbenoother partiestoconsiderinfuturenegotiations andallthepro tswillcometothepublisher Ifa dictionarydatabasehasbeenenhancedbyanNLPgroup then primafacie ashareofthe intellectualpropertybelongstotheNLPgroup Itisnotstraightforwardtoarriveatamodel forhowsharesoftheintellectualpropertyshouldbeallocated Thestartingpointwould usuallybethequantityoflabourthateachpartyhadputin orthefractionofthetextthat eachproduced Thelattercannotbeappliedatall whatevertheNLPenhancementsmay be itcertainlywillnotmakesensetoquantifythemasafractionoftotaltext length The formerisalsooflittleuse theNLPlaboratorywillprobablyhavebeenusingvariouspieces ofsoftwareandexpertisetoaidwiththeenhancement whichwillhavebeenundertakenfor internaluse inotherNLPapplicationsorresearchprojects sothere saleorre licensingof theupgradeddictionarywouldbeaside e ect Thepublishermaywellhavedoubtsabouttheaccuracyandconsistencyoftheenhanced dictionary butmaynotbewell placedtoevaluate asthisinitselfmayrequireNLPtechniques andanunderstandingoftheissueswhicharelikelytobecriticalforNLPapplications Theenhanceddictionarymayhavepotentialusesforprintproducts forelectronicprod uctsfortheconsumermarket orforlicensingonforNLPuse Aquestioninrelationto the rsttwoisaddressedinsection Regardingthethird thepotentialcomplexityofthe arrangementsisforbidding WhereadictionaryislicensedforNLPuse thelicencemay be broadly forresearchuseorforproductdevelopment Researchuseisstraightforward providedthedemarcationintheNLPgroupbetweenresearchanddevelopmentisclearly drawn Whereitisforproductdevelopment thepublishermayreceivealicencefeeora royaltyforeachproduct containing thedictionary orsomeofeach Containing ininverted Ourhosts longstandingengagementwithNLPmaywellhaveplayedarole commas becausethedictionarywillnotgenerallyexistintheproductinanyrecognisable form butasoneofanumberofinputstothesystem slexicon whichmaywellbecompiled andunreadable Thatproductmayitselfbeaconsumerproduct ormaybeacomponentof somelargerapplication sotheremaybemanysteps eachpotentiallyinvolvinglicencesand royalties betweenthedictionarypublisherandtheend product Onehassomesympathy withthepublishernotwishingtohaveitsnegotiatinghandconstrainedbyundertakingsto NLPgroups Responses Piracyisnotanstrongargument Therearemanywaysinwhichadictionarymaybe piratedormayenterthepublicdomain Itmaybere keyed itmaybeOCR dfromthe printedversion thecontentsofCD ROMsmaybeunscrambled hackersmayhackintothe publisher ssystem tapesmaybestolen Itwillofcoursebethedutyoflicence holdersto ensurethatpeoplewhoshouldnothaveaccesstotheirversionofthedictionary donot and thatitdoesnotgetcopiedoutsidethecon nesofthelaboratory andiftheyfailinthisduty theywillbeculpable Buttheyarenomorelikelytofailinthedutythanassortedotherof thepublisher semployeesandagentswhohaveaccesstothedata Regardingcopyrightonanenhanceddictionary rstly manyresearchgroupsinUniversi tieswillnotwanttoclaimashareofcopyright Theoriginallicenceallowingthemtousethe dictionarycouldthencontainaclausestipulatingthatlexiconsforwhichthedictionaryhas beenaninputcanonlybepassedontothirdpartiesbythepublisher andallthecopyright insuchlexiconsshallbevestedinthepublisher WhereNLPgroupsarenothappytohandovercopyrightinthisway therewillbepo tentiallycomplexnegotiationsrequired astherewillwhenalexiconisacomponentofsome otherapplicationatoneormorefurtherremovesfromtheendproduct Thesesortsofcon siderationsare however becomingcommonplaceinthesoftwareandmultimediaindustries sothepublisher sconcernsreducetothefollowing arethegainstobehadfromsynergywith NLPsu cienttomeritthee ort Isthepublisherwillingtotakeonthechallengeofthe complexnegotiations TheModel Theprimaryconcernofthepublisheristoretaincontroloftheresourceandtomaximise the owofincomeitgivesriseto TheprimaryconcernofmanyNLPresearchersisthat theirworkisavailableforotherstouseandextend Reputationandstatusisdependenton beingcited intheacademicworld Ifpeopleuseyourresourcetheywillciteyou Licence incomeisalesserconsideration rstly becauseititisnotexpected secondly becauseit wouldprobablygototheinstitutionratherthantheindividual andthirdly simplybecause itisnotthecurrencyofacademicstatus TheappropriatemodelisforthepublishertoencourageNLPresearcherstousethedictio nary onthebasisthatanyenhancementstothedictionarywillbereturnedtothepublisher forittouseitselfifitsowishes fordictionaryrevisionandotherimprovedconsumerprod ucts andtomarketandgenerallymakeavailabletootherNLPgroups Theagreementwould putthepublisherunderanobligationtomaketheenhanceddictionaryavailable undersim ilartermsagain andforafeethatwouldnotbeprohibitive andthiswouldbethebene t totheNLPgrouptocounterbalancethefactthattheywould inthesimplestcase relinquish claimstoashareinthecopyright Limitationsarerequiredonthisobligationofthepublishertopublish Firstly theen hanceddictionarywouldhavetobeofadequatequality andtothisend thepublishermust acquiretheexpertisetoassessthequality ThiswilldemandsomeinvestmentinNLP expertiseonthepartofthepublisher butthen thepublishershouldnotexpecttoreapthe fruitsofthenewmarketwithoutanyinvestment Secondly thepublisherwillwanttostrikedi erentdealsforthedictionarywithdi erent customers onewouldnotexpecttermsforMicrosofttobeasforaUniversitygrouporstartup company Thetermsoftheagree