python 基础练习题

1、Advanced computation linguistics 1. Collect the most frequent words in 5 genres of Brown Corpus: news, adventure, hobbies, science_fiction, romanceTo collect most frequent words from the given genres we can follow the following steps: import nltk from nltk.corpus import brown brown.categories()adven

2、ture, belles_lettres, editorial, fiction, government, hobbies, humor, learned, lore, mystery, news, religion, reviews, romance, science_fiction news_text = brown.words(categories=news,adventure,hobbies,science_fiction,romance) from bability import FreqDist fdist=FreqDist(w.lower() for w in n

3、ews_text) voca=fdist.keys() voca:50the, , ., and, of, to, a, in, he, , , was, for,that, it, his, on, with, i, is, at, had, ?, as, be, you, ;, her, but, she, this, from, by, -, have, they, said, not, are, him, or, an, one, all, were, would, there, !, out, will voca1=fdist.items() voca1:50(the, 18635)

4、, (, 17215), (., 16062), (and, 8269), (of, 8131), (to, 7125), (a, 7039), (in, 5549), (he, 3380), (, 3237), (, 3237), (was, 3100), (for, 2725), (that, 2631), (it, 2595), (his, 2237), (on, 2162), (with, 2157), (i, 2034), (is, 2014), (at, 1817), (had, 1797), (?, 1776), (as, 1725), (be, 1610), (you, 160

5、0), (;, 1394), (her, 1368), (but, 1296), (she, 1270), (this, 1248), (from, 1174), (by, 1157), (-, 1151), (have, 1099), (they, 1093), (said, 1081), (not, 1051), (are, 1019), (him, 955), (or, 950), (an, 911), (one, 903), (all, 894), (were, 882), (would, 850), (there, 807), (!, 802), (out, 781), (will,

6、775)This means that the frequency of word “the” is more than others.2. Exclude or filter out all words that have a frequency lower than 15 occurrencies. (hint using conditional frequency distribution)By adding functionalities on the first task of collecting words based on their frequency of occurren

7、ces, we can filter words which has frequency occurrence of =15. filteredText= filter(lambda word: fdistword=15,fdist.keys() voca=fdist.keys() filteredText:50 /*first 50 words*/the, , ., and, of, to, a, in, he, , , was, for,that, it, his, on, with, i, is, at, had, ?, as, be, you, ;, her, but, she, th

8、is, from, by, -, have, they, said, not, are, him, or, an, one, all, were, would, there, !, out, will filteredText-50: /*last 50 words*/musical, naked, names, oct., offers, orders, organizations, parade, permit, pittsburgh, prison, professor, properly, regarded, release, republicans, responsible, ret

9、irement, sake, secrets, senior,sharply, shipping, sir, sister, sit, sought, stairs, starts, style, surely, symphony, tappet, theyd, tied, tommy, tournament, understanding, urged, vice, views, village, vital, waddell, wagner, walter, waste, wed, wearing, winning3. Then exclude or filter out all stopw

10、ords from the lists you have created.(hint using conditional frequency distribution)To filter the stop words we have to define tiny function using the word net library for english language. from nltk.corpus import stopwords stopwords.words(english)i, me, my, myself, we, our, ours, ourselves, you, yo

11、ur, yours, yourself, yourselves, he, him, his, himself, she, her, hers, herself, it, its, itself, they, them, their, theirs, themselves, what, which, who, whom, this, that, these, those, am,is, are, was, were, be, been, being, have, has, had, having, do, does, did, doing, a, an, the, and, but, if, o

12、r, because, as, until, while, of, at, by, for, with, about, against, between, into, through, during, before, after, above, below, to, from, up, down, in, out, on, off, over, under, again, further, then, once, here, there, when, where, why, how, all, any, both, each, few, more, most, other, some, suc

13、h, no, nor, not, only, own, same, so, than, too, very, s, t, can, will, just, don, should, now def content_fraction(text):. stopwords= nltk.corpus.stopwords.words(english). content = w for w in text if w.lower() not in stopwords . return len(content) / len(text) . content_fraction(nltk.corpus.reuter

14、s.words() 0. filterdText = filterStopword(freqDist) filterdText:50, ., , , ?, ;, -, said, would, one, !, could, (, ), :, time, like, back, two, first, man,made, Mrs., new, get, way, last, long, much, even, years, good, little, also, Mr., see,right, make, got, home, many, never, work, know, day , aro

15、und, year, may, came, still freqDist:50, the, ., of, and, to, a, in, , , was, for, that, he, on, with, his, I, it, is, The, had, ?,at, as, be, ;, you, her, He, -, from, by, said, h ave, not, are, this, him, or, were, an, but,would, she, they, one, !, all, out From the result in filterdText words lik

16、e the, it, is and so on does not exist compared to the same number of output with stop words. len(freqDist)2341 len(filterdText)2153We can further check that how many stop-words have been removed from the freqDist15 using len( ) function.4. Create a new list of lemmas or roots by normalizing all wor

17、ds by stemmingfor create the normalized list of lemmas we apply the Porter Stemmer nltk functionality. file = open(filterdText.txt) text = file.read() textTokens = nltk.word_tokenize(text)Now we do stemming p = nltk.PorterStemmer ( ) rootStemming = p.stem(t) for t in textTokens textTokens:100!, &, ,

18、 , em, (, ), , -, ., 1, 10, 100, 11, 12, 13, 14, 15, 16, 17, 18, 1958, 1959,1960, 1961, 2, 20, 200, 22, 25, 3, 30, 4, 5, 50, 6, 60 , 7, 8, 9, :, ;, ?, A., Actually,Af, Ah, Aj, Alexander, Also, Although, Americ a, American, Americans, Among, Angeles,Anne, Anniston, Another, April, Association, Augu s

19、t, Austin, Avenue, B, Bdikkat, B.,Barton, Beach, Belgians, Besides, Bill, Billy, B lue, Board, Bob, Bobbie, Boston, Brannon,British, C., Cady, California, Catholic, Cath y, Center, Central, Charles, Charlie, Chicago,Christian, Church, City, Class, Clayton, Club, Co., Coast, Cobb, CollegeThis functio

20、n can display sorted non normalized sample outputs for comparison rootStemming:100!, &, , , em, (, ), , -, ., 1, 10, 100, 11, 12, 13, 14, 15, 16, 17, 18, 1958, 1959,1960, 1961, 2, 20, 200, 22, 25, 3, 30, 4, 5, 50, 6, 60 , 7, 8, 9, :, ;, ?, A., Actual, Af,Ah, Aj, Alexand, Also, Although, America, Ame

21、rican, American, Among, Angel, Ann,Anniston, Anoth, April, Associ, August, Austin,Avenu, B, Bdikkat, B., Barton, Beach,Belgian, Besid, Bill, Billi, Blue, Board, Bob , Bobbi, Boston, Brannon, British, C., Cadi,California, Cathol, Cathi,Center,Central,Charl,Charli,Chicago,Christian, Church, Citi,Class

22、, Clayton, Club, Co., Coast, Cobb, Colleg This can sorted stemmed sample output for comparison 5. Create a new list of lemmas or roots by normalizing all words by lemmatizationAfter importing the file we need to lemmatize, which is the same step as the previous one:and using the same rawText wnl = n

23、ltk.WordNetLemmatizer() rootLemmatize = wnl.lemmatize(t) for t in textTokens rootLemmatize:100 / *the first 100 sorted lemmatized lemmas for comparison*/!, &, , , em, (, ), , -, ., 1, 10, 100, 11, 12, 13, 14, 15, 16, 17, 18, 1958, 1959,1960, 1961, 2, 20, 200, 22, 25, 3, 30, 4, 5, 50, 6, 60 , 7, 8, 9

24、, :, ;, ?, A., Actually,Af, Ah, Aj, Alexander, Also, Although, Americ a, American, Americans, Among, Angeles,Anne, Anniston, Another, April, Association, Augu st, Austin, Avenue, B, Bdikkat, B.,Barton, Beach, Belgians, Besides, Bill, Billy, B lue, Board, Bob, Bobbie, Boston, Brannon,British, C., Cad

25、y, California, Catholic, Cath y, Center, Central, Charles, Charlie, Chicago,Christian, Church, City, Class, Clayton, Club,Co.,Coast,Cobb, CollegeWe end this task by writing the out put of the lemmatized lemmas or rootwords to the file (rootLemmatize.txt).6. Use the most frequent lemmas to find seman

26、tic similarities using WordNet.To find synsets with related meanings we have to traverse the WordNet network. knowing which word is semantically related is useful for indexing a collection of texts. For example a search for a general term like England will match for specific terms like UK.Top 100 fr

27、equent lemmas: file = open(filterdText.txt) /*from filterdText we get the words*/ tmp = file.read() from nltk.tokenize import RegexpTokenizer /*remove punctuations*/ tokenizer = RegexpTokenizer(rw+) textSimilarity = tokenizer.tokenize(tmp) freqDistSimilarity = FreqDist(w.lower() for w in textSimilar

28、ity) /* extract the first 100 frequent lemmas to new list*/ for word in textSimilarity: . freqDistSimilarity.inc(word) tmpFDS = freqDistSimilarity.keys():100 /* first 100 most frequent lemmas*/ freqDistSimilarity.items():50(s, 64), (t, 60), (re, 24), (d, 22), (you, 20), (ll, 18), (m, 14), (he, 12),

29、(let, 12), (man, 10), (p,10), (we, 10), (I, 8), (i, 8), (ve, 8), (won, 8), (year, 8), (B, 6), (a, 6), (actually, 6), (also, 6),(although, 6), (among, 6), (another, 6), (association, 6), (b, 6), (beach, 6), (bill, 6),(blue, 6),(board, 6), (center, 6), (central, 6), (church, 6), ( city, 6), (class, 6)

30、, (club, 6), (college, 6),(come, 6), (committee,6) ,(council,6),(county,6),(court,6),(day,6),(department,6),(district,6), (don, 6), (earth, 6), (education, 6), (even, 6), (every, 6) def pathSimilarity(word1,word2, s=wnet.path_similarity): /*path similarity between two words*/. synSets1= wnet.synsets

31、(word1). synSets2= wnet.synsets(word2). pointSimilarity = . for synSet1 in synSets1:. for synSet2 in synSets2:. pointSimilarity.append(s(synSet1,synSet2). if len(pointSimilarity)=0:. return 0. else:. return max(pointSimilarity) tmpFDS30:35 /*arbitrary path similarity test for 5 lemmas*/center, central, church, city, class for wo