scrapysphinx搭建搜索引擎.pptVIP

scrapy+sphinx搭建搜索引擎 银平 2010-06-07 Outline a.Overview b.Scrapy python爬虫框架 c. Sphinx C+全文搜索引擎 d.demo scrapy + sphinx实现 小说搜索引擎 Overview - 搜索引擎/爬虫分类 a.搜索引擎 a.通用搜索引擎 b.垂直搜索引擎 c.资源型垂直搜索引擎 b.爬虫 a.通用爬虫 b.专用爬虫 Overview - 搜索引擎 a.分词 b.倒排索引 /IR-book/html/htmledition/a-first-take-at- building-an-inverted-index-1.html Scrapy python爬虫框架 a.Architecture b.Built-in middlewares c. Extensions d.从网页中提取数据 Architecture a.Components a.Scrapy Engine b.Scheduler c.Downloader d.Spider e.Item Pipeline f.Middlewares b.Event-driven networking: twisted Architecture Built-in middlewares a.Downloader middlewares a.DefaultHeadersMiddleware b.HttpAuthMiddleware c.HttpCacheMiddleware d.RedirectMiddleware e.RetryMiddleware b.Spider middlewares a.DepthMiddleware b.RefererMiddleware c. Scheduler middlewares a.DuplicatesFilterMiddleware Extensions a.特性 a.Scrapy启动时 加载的普通class b.监听各种signal (engine_started, item_scraped, item_dropped) b.Built-in extensions a.CoreStats b.WebConsole c. 从网页中提取数据 a.CrawlSpider: Rule/Matcher/callback b.使用XPath进行提取 c. Scrapy shell d.Parsley: a selector language, superset of XPath and css3 ( 内存泄露) li.maina/href Sphinx C+全文搜索引擎 a.Sphinx特性 b.Sphinx组件 c. 索引 d.搜索 e.SphinxSE: mysql存储引擎 Sphinx特性 a. high indexing speed (upto 10 MB/sec on modern CPUs); b. high search speed (avg query is under 0.1 sec on 2-4 GB text collections); c. high scalability (upto 100 GB of text, upto 100 M documents on a single CPU); d. provides good relevance ranking through combination of phrase proximity ranking and statistical (BM25) ranking; e. provides distributed searching capabilities; f. provides document exceprts generation; g. provides searching from within MySQL through pluggable storage engine; h. supports boolean, phrase, and word proximity queries; i. supports multiple full-text fields per document (upto 32 by default); j. supports multiple additional attributes per document (ie. groups, timestamps, etc); k. supports stopwords; l. supports both single-byte encodings and UTF-8; m.supports English stemming, Russian stemming, and Soundex for morphology; n. supports MySQL natively (MyISAM and InnoDB tables are both supported); o. supports PostgreSQL natively. Sphinx组件 a.indexer (binary) b.searchd (binary) c. search (binary) d.sphinxapi (api libraries for PHP, Python, Perl, Ruby) e.spelldump f. indextool 索引 a.数据源: 数据库, xml, 等等。 a.表的每一行视为 一篇文档, b.可在配置中指定哪些列需要进行索引 b.属性：表的某些列可被指定为文档的属性，不被索引，但可 用来做过滤 和排序 索引(2) 索引配置的片段 sql_query = SELECT id, title, content, author_id, forum_id, post_date FROM my_forum_posts sql_attr_uint = author_id sql_attr_uint = forum_id sql_attr_timestamp = post_date 过滤 和排序应用示例 / only search posts by author whose ID is 123 $cl-SetFilter ( “author_id“, array ( 123 ) ); / only search posts in sub-forums 1, 3 and 7 $cl-SetFilter ( “forum_id“, array ( 1,3,7 ) ); / sort found posts by posting date in descending order $cl-SetSortMode ( SPH_SORT_ATTR_DESC, “post_date“ ); 搜索 匹配模式 匹配模式 a.SPH_MATCH_ALL oSPH_MATCH_ANY oSPH_MATCH_PHRASE oSPH_MATCH_BOOLEAN oSPH_MATCH_EXTENDED2 最灵活的SPH_MATCH_EXTENDED2 hello | world hello | -world name hello intro world “hello world“ aaa SetSortMode ( SPH_SORT_EXPR, “weight + ( user_karma + ln(pageviews) )*0.1“ ); 搜索 分布式搜索 a.横向划分数据，分别进 行索引 b.在主searchd上配置分布式索引 c. 主searchd发送请求到各个从searchd，合并返回的结果，并 最终返回 d.cluster中的每个searchd都可作为主searchd, 进行负载 均衡 搜索 SphinxQL: 使用sql语法进行搜索 a.searchd实现 了mysql的网络协议 b.可将searchd当做mysql服务器使用，通过mysql client连接 SELECT *, weight*10+docboost AS skey FROM example ORDER BY ske SELECT * FROM test1 WHERE MATCH(“test doc“/3) SELECT * FROM test WHERE MATCH(title hello body world) OPTION ranker=bm25, max_matches=3000 SphinxSE: mysql存储引擎 特点 a.类似InnoDB, MyISAM, 需要编译进 mysql b.本身不存储数据，而是与searchd通信来获取数据 优优点 a.任何语言都可使用，而naive api只支持几种语言 当搜索结果需要在mysql端进一步处理时，效率更高 (JOIN, mysql-like filtering) Sphinx vs. xapian Sphinx a.searchd提供搜索服务 b.不用自己实现 indexer，不用写C+代码，仅通过配置就能实 现索引和搜索 c. 分布式搜索 xapian a.类似lucene，api直接访问 索引文件进行搜索 得自己实现 indexer 可定制性强 (豆瓣从sphinx切到xapian) demo scrapy + sphinx实现搜索引擎 以爬取，索引，搜索起点小说为 例，实现 一个小说搜索引擎. demo的代码可从git