《Pandas数据处理》课件 4.1.1 去掉停用词（微课）

去掉停用词主讲人：刘学重庆市九龙坡职业教育中心问题描述现有某电视剧弹幕信息，请去掉弹幕信息里面的停用词，然后以列表的形式输出弹幕中词频最高的10个词。contentslikeCounttv_name0二刷的朋友有吗20111我希望一切能重来312这段眼神变化的太妙了913良心啊，一小时18414基本都好201............59995这个叶爸有点东西2271259996眼镜掉在案发现场了901259997俺的眼睛掉在厂里了101259998他不戴假发你更不习惯171259999那是什么药呀3312输出结果词语词频孩子2030爬山1913严良1511真的1407一个1305妈妈939演技902一起865普普846感觉782问题分析问题描述问题解答怎样将句子切割成为词语？

怎样把弹幕信息表和停用词表联合起来？怎样统计词频？cut()merge()value_counts()操作提示利用jieba库中的cut()函数对弹幕信息进行分词后转换为数据框，将之与停用词数据框进行合并，筛选出不在停用词表中的词语，统计这些词出现的词频，这样得到了题目要求的结果。程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码pandas提供了大量能使我们快速便捷地处理数据的函数和方法。程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码jieba是python的一个中文分词库，具有高性能、高准确率、可扩展等特点程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码contentslikeCounttv_name0二刷的朋友有吗20111我希望一切能重来312这段眼神变化的太妙了913良心啊，一小时18414基本都好201............59995这个叶爸有点东西2271259996眼镜掉在案发现场了901259997俺的眼睛掉在厂里了101259998他不戴假发你更不习惯171259999那是什么药呀3312data=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码切割后的列表为：['$','0','1','2','3','4','5','6','7','8','9......'非独','靠','顺','顺着','首先','！','，','：','；','？']程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码

stopword0$10213243.....741！742，743：744；745？生成的停用词表stop_word=pd.DataFrame(stop_word,columns=["stopword"])程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码word0二刷1的2朋友3有4吗......339467那339468是339469什么339470药339471呀word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])程序代码word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])contentslikeCounttv_name0二刷的朋友有吗20111我希望一切能重来312这段眼神变化的太妙了913良心啊，一小时18414基本都好201............59995这个叶爸有点东西2271259996眼镜掉在案发现场了901259997俺的眼睛掉在厂里了101259998他不戴假发你更不习惯171259999那是什么药呀3312“二刷的朋友有吗我希望一切能重来这段眼神变化的太妙了良心啊，一小时……好了警官你是下一个好一个不戴眼镜的斯文败类儿子你啥时候学习啊居然还不说实话？我不戴假发更厉害演完这部电影，伊能静开始怕了你看我还有机会吗这个叶爸有点东西眼镜掉在案发现场了俺的眼睛掉在厂里了他不戴假发你更不习惯那是什么药呀”程序代码word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])“二刷的朋友有吗我希望一切能重来这段眼神变化的太妙了良心啊，一小时……好了警官你是下一个好一个不戴眼镜的斯文败类儿子你啥时候学习啊居然还不说实话？我不戴假发更厉害演完这部电影，伊能静开始怕了你看我还有机会吗这个叶爸有点东西眼镜掉在案发现场了俺的眼睛掉在厂里了他不戴假发你更不习惯那是什么药呀”[‘二刷’,‘的’,‘朋友’,‘有’,‘吗’,‘我’,‘希望’,……‘他’，’不戴’，‘假发’，你，‘更不’，‘习惯’，‘那是’，‘什么’，‘药’，‘呀’]程序代码word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])[‘二刷’,‘的’,‘朋友’,‘有’,‘吗’,‘我’,‘希望’,……‘他’，’不戴’，‘假发’，你，‘更不’，‘习惯’，‘那是’，‘什么’，‘药’，‘呀’]word0二刷1的2朋友3有4吗......339467那339468是339469什么339470药339471呀程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")wordstopword0二刷NaN1的的2朋友NaN3有有4吗吗.........339467那那339468是是339469什么什么339470药NaN339471呀呀程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]word=word.value_counts()print(word.head(10))程序代码wordstopword0二刷NaN2朋友NaN6希望NaN9重来NaN10这段NaN.........339451案发现场NaN339455眼睛NaN339458厂里NaN339462戴假发NaN339466习惯NaNword=word.query("stopword.isnull()andword.str.len()>1",engine='python')["word"]程序代码importpandasaspdimportjiebadata=pd.read_csv(r"D:\pydata\项目四\某电视剧弹幕信息.csv")stop_word=open(r"D:\pydata\项目四\停用词.txt","r",encoding='utf-8')stop_word=stop_word.read().split()stop_word=pd.DataFrame(stop_word,columns=["stopword"])word=pd.DataFrame(jieba.cut("".join(data.iloc[:,1].astype(str))),columns=["word"])word=pd.merge(word,stop_word,left_on=["word"],right_on=["stopword"],how="left")word=word.query("stopword.isnull()andword.str.