开源网络爬虫程序一览

1、开源网络爬虫程序(spider) 一览2007.10.15来自：OliverSegals blog共有评论(17)条发表评论 收藏spider是搜索引擎的必须模块.spider数据的结果直接影响到搜索引擎的评价指标. 第一个spider程序由MIT的Matthew K Gray操刀该程序的目的是为了统计互联网中 主机的数目。WebLech URL SpiderPySolitaire (Python Solita % spider是搜索引擎的必须模块.spider数据的结果直接影响到搜索引擎的评 价指标.第一个spider程序由MIT的Matthew K Gray操刀该程序的目的是为了统计互联

2、网中主机的数目Spier定义(关于Spider的定义，有广义和狭义两种).狭义:利用标准的http协议根据超链和web文档检索的方法遍历万维网信息空间的 软件程序.广义:所有能利用http协议检索web文档的软件都称之为spider.其中 Protocol Gives Sites Way To Keep Out The Bots Jeremy Carl, Web Week, Volume 1, Issue 7, November 1995是和spider息息相关的协议，大家有兴趣参考 .HeritrixHeritrix is the Internet Archives open-source,

3、 extensible, web-scale, archival-quality web crawler project.Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture

4、for the benefit of future researchers and generations, this name seemed apt.语言:JAVAWebLech URL SpiderWebLech is a fully featured web site download/mirror tool in Java, which supports many features required to download websites and emulate standard web-browser behaviour as much as possible. WebLech i

5、s multithreaded and comes with a GUI console.语言:JAVAJSpiderA Java implementation of a flexible and extensible web spider engine. Optional modules allow functionality to be added (searching dead links, testing the performance and scalability of a site, creating a sitemap, etc .语言:JAVAWebSPHINXWebSPHI

6、NX is a web crawler (robot, spider) Java class library, originally developed by Robert Miller of Carnegie Mellon University. Multithreaded, tollerant HTML parsing, URL filtering and page classification, pattern matching, mirroring, and more.语言:JAVAPySolitairePySolitaire is a fork of PySol Solitaire

7、that runs correctly on Windows and has a nice clean installer. PySolitaire (Python Solitaire) is a collection of more than 300 solitaire and Mahjongg games like Klondike and Spider语言：PythonThe Spider Web Network Xoops Mod TeamThe Spider Web Network Xoops Module Team provides modules for the Xoops co

8、mmunity written in the PHP coding language. We develop mods and or take existing php script and port it into the Xoops format. High quality mods is our goal.语言:phpFetchgalsA multi-threaded web spider that finds free porn thumbnail galleries by visiting a list of known TGPs (Thumbnail Gallery Posts).

9、 It optionally downloads the located pictures and movies. TGP list is included. Public domain perl script running on Linux.语言:perlWhere SpiderThe purpose of the Where Spider software is to provide a database system for storing URL addresses. The software is used for both ripping links and browsing t

10、hem offline. The software uses a pure XML database which is easy to export and import.语言:XMLSperowiderSperowider Website Archiving Suite is a set of Java applications, the primary purpose of which is to spider dynamic websites, and to create static distributable archives with a full text search inde

11、x usable by an associated Java applet.语言:JavaSpiderPySpiderPy is a web crawling spider program written in Python that allows users to collect files and search web sites through a configurable interface.语言：PythonSpidered Data RetrievalSpider is a complete standalone Java application designed to easil

12、y integrate varied datasources. * XML driven framework * Scheduled pulling * Highly extensible * Provides hooks for custom post-processing and configuration语言:JavawebloupeWebLoupe is a java-based tool for analysis, interactive visualization (sitemap), and exploration of the information architecture

13、and specific properties of local or publicly accessible websites. Based on web spider (or web crawler) technology.语言:javaASpiderRobust featureful multi-threaded CLI web spider using apache commons httpclient v3.0 written in java. ASpider downloads any files matching your given mime-types from a website. Tries to reg.exp. match emails by default, logg