文本挖掘简介

文本挖掘简介文本挖掘简介OutlineIntroductionTF-IDFSimilarityIntroductionWhy？Textmining≈WebminingHow？ClassificationorClusteringRetrieval文本分类一般过程预处理将文档集表示成易于计算机处理的形式

特征表示与选择、降维根据适宜的权重计算方法表示文档中各项的重要性

学习建模构建分类器文本分类预处理去标点、多余空格、数字（可选）大小写统一去停用词（stopwords）没有实际含义的词，比如and,you,have等等统一词根PorterStemmer分词英文？中文特征表示向量空间模型以词项为特征组成高维特征向量TF/IDF得到权值TF-IDFTF（TermFrequency）表示词项频率IDF（InverseDocumentFrequency）逆文档频率TF*IDF值9SimilarityApplications

ManyWeb-miningproblemscanbeexpressedasfinding“similar”sets:Plagiarism/MirrorPages/ArticlesfromtheSameSource/DuplicationRemoveCollaborativeFilteringasaSimilar-SetsProblemRecommendtousersitemsthatwerelikedbyotheruserswhohaveexhibitedsmilartastesMeasurementEditdistanceShorttext,wordsForpersonaltextJaccarddistanceLongtext,ignoringthewordsimilarityForgovernmenttextMicrosoftAcademicSearchPK://academic.research.microsoft/Author/2037349.aspx://academic.research.microsoft/Author/3054641.aspxReal-worldDataisRatherDirty！KennethDeJongKennethDejong2023/1/13Trie-Join@VLDB201011/38Typoin“author”Typoin“title”relaxedrelatedArgyriosZymnisArgyrisZymnisDBLPCompleteSearch2023/1/13Real-worldDataisRatherDirty！Trie-Join@VLDB201012/38ThesimilarityjoinisanessentialoperationfordataintegrationandcleaningPerformasimilarityjoinonNameattribute(findallrecordpairswhoseNameattributesaresimilar)Output:(2037349,3054641),…SimilarityJoinsRIdNameUniv.2037349KennethDeJongGeorge…………3054641KennethDejongGeorge…………2023/1/13Trie-Join@VLDB201013/38NearDuplicateDataOnoneend,awindedPeteSamprastriedtosummonenoughenergytogivetheNewYorkfansanothermemorablewintotalkaboutitonthesubwayridehome.Ontheotherside,RogerFedererworeaslygrinlikeheknewagewasabouttocatchuptotheformerworldNo.1-themanwhoownstherecordof14GrandSlamshewants.03/11/2008|11:28AMByJAYCOHEN,APSportsWriterMar11,4:23amEDTSimilarityJoinTokenize:Eachrecordisasetoftokensfromafiniteuniverse.Supposeeachrecordisasingletextdocumentx=“yesassoonaspossible”y=“assoonaspossibleplease”x={A,B,C,D,E}y={B,C,D,E,F}wordyesassoonas1possbilepleasetokenABCDEF参考文献ChuanXiao,WeiWang,XueminLin,JeffreyXuYu.EfficientSimilarityJoinsforNearDuplicateDetection.WWW200