scrapy+sphinx搭建搜索引擎

scrapy sphinx搭建搜索引擎 银平pkufranky 2010 06 07 Outline OverviewScrapy python爬虫框架Sphinx C 全文搜索引擎demo scrapy sphinx实现小说搜索引擎 Overview 搜索引擎 爬虫分类 搜索引擎通用搜索引擎垂直搜索引擎资源型垂直搜索引擎爬虫通用爬虫专用爬虫 Overview 搜索引擎 分词倒排索引http nlp stanford edu IR book html htmledition a first take at building an inverted index 1 html Scrapy python爬虫框架 ArchitectureBuilt inmiddlewaresExtensions从网页中提取数据 Architecture ComponentsScrapyEngineSchedulerDownloaderSpiderItemPipelineMiddlewaresEvent drivennetworking twisted Architecture Built inmiddlewares DownloadermiddlewaresDefaultHeadersMiddlewareHttpAuthMiddlewareHttpCacheMiddlewareRedirectMiddlewareRetryMiddlewareSpidermiddlewaresDepthMiddlewareRefererMiddlewareSchedulermiddlewaresDuplicatesFilterMiddleware Extensions 特性Scrapy启动时加载的普通class监听各种signal engine started item scraped item dropped Built inextensionsCoreStatsWebConsole 从网页中提取数据 CrawlSpider Rule Matcher callback使用XPath进行提取ScrapyshellParsley aselectorlanguage supersetofXPathandcss3 内存泄露 li main a href Sphinx C 全文搜索引擎 Sphinx特性Sphinx组件索引搜索SphinxSE mysql存储引擎 Sphinx特性 highindexingspeed upto10MB seconmodernCPUs highsearchspeed avgqueryisunder0 1secon2 4GBtextcollections highscalability upto100GBoftext upto100MdocumentsonasingleCPU providesgoodrelevancerankingthroughcombinationofphraseproximityrankingandstatistical BM25 ranking providesdistributedsearchingcapabilities providesdocumentexceprtsgeneration providessearchingfromwithinMySQLthroughpluggablestorageengine supportsboolean phrase andwordproximityqueries supportsmultiplefull textfieldsperdocument upto32bydefault supportsmultipleadditionalattributesperdocument ie groups timestamps etc supportsstopwords supportsbothsingle byteencodingsandUTF 8 supportsEnglishstemming Russianstemming andSoundexformorphology supportsMySQLnatively MyISAMandInnoDBtablesarebothsupported supportsPostgreSQLnatively Sphinx组件 indexer binary searchd binary search binary sphinxapi apilibrariesforPHP Python Perl Ruby spelldumpindextool 索引 数据源 数据库 xml 等等 表的每一行视为一篇文档 可在配置中指定哪些列需要进行索引属性 表的某些列可被指定为文档的属性 不被索引 但可用来做过滤和排序 索引 2 索引配置的片段sql query SELECTid title content author id forum id post dateFROMmy forum postssql attr uint author idsql attr uint forum idsql attr timestamp post date过滤和排序应用示例 onlysearchpostsbyauthorwhoseIDis123 cl SetFilter author id array 123 onlysearchpostsinsub forums1 3and7 cl SetFilter forum id array 1 3 7 sortfoundpostsbypostingdateindescendingorder cl SetSortMode SPH SORT ATTR DESC post date 搜索 匹配模式 匹配模式SPH MATCH ALLSPH MATCH ANYSPH MATCH PHRASESPH MATCH BOOLEANSPH MATCH EXTENDED2最灵活的SPH MATCH EXTENDED2hello worldhello world namehello introworld helloworld aaa bbb ccc helloworldfoo 10 theworldisawonderfulplace 3 helloworld title exampleprogram 5 bodypython php perl code 搜索 排序模式 SPH SORT RELEVANCESPH SORT EXTENDED weightDESC priceASC idDESCSPH SORT EXPR cl SetSortMode SPH SORT EXPR weight user karma ln pageviews 0 1 搜索 分布式搜索 横向划分数据 分别进行索引在主searchd上配置分布式索引主searchd发送请求到各个从searchd 合并返回的结果 并最终返回cluster中的每个searchd都可作为主searchd 进行负载均衡 搜索 SphinxQL 使用sql语法进行搜索 searchd实现了mysql的网络协议可将searchd当做mysql服务器使用 通过mysqlclient连接SELECT weight 10 docboostASskeyFROMexampleORDERBYskeSELECT FROMtest1WHEREMATCH testdoc 3 SELECT FROMtestWHEREMATCH titlehello bodyworld OPTIONranker bm25 max matches 3000 SphinxSE mysql存储引擎 特点类似InnoDB MyISAM 需要编译进mysql本身不存储数据 而是与searchd通信来获取数据优点任何语言都可使用 而naiveapi只支持几种语言当搜索结果需要在mysql端进一步处理时 效率更高 JOIN mysql likefiltering Sphinxvs xapian Sphinxsearchd提供搜索服务不用自己实现ind