NET抓取和分析网页的类

主要功能有 1 提取网页的纯文本 去所有 html 标签和 javascript 代码 2 提取网页的链接 包括 href 和 frame 及 iframe 3 提取网页的 title 等 其它的标签可依此类推 正则是一样的 4 可以实现简单的表单提交及 cookie 保存 第一部分 using System using System Data using System Configuration using System Net using System IO using System Text using System Collections Generic using System Text RegularExpressions using System Threading using System Web 网页类 public class WebPage region 私有成员 private Uri m uri 网址 private List m links 此网页上的链接 private string m title 此网页的标题 private string m html 此网页的 HTML 代码 private string m outstr 此网页可输出的纯文本 private bool m good 此网页是否可用 private int m pagesize 此网页的大小 private static Dictionary webcookies new Diction ary 存放所有网页的 Cookie private string m post 此网页的登陆页需要的 POST 数据 private string m loginurl 此网页的登陆页 endregion region 私有方法 这私有方法从网页的 HTML 代码中分析出链接信息 List private List getLinks if m links Count 0 Regex regex new Regex 2 regex 0 new Regex m href s w W RegexOptions Multiline RegexOptions Igno reCase regex 1 new Regex src s RegexOptions Multiline RegexOptions IgnoreCase for int i 0 i 2 i Match match regex i Match m html while match Success try string url new Uri m uri match Groups url Value AbsoluteUri string text if i 0 text new Regex s Link link new Link url text m links Add link catch Exception ex Console WriteLine ex Message match match NextMatch return m links 此私有方法从一段 HTML 文本中提取出一定字数的纯文本 HTML 代码 提取从头数多少个字 是否要链接里面的字 纯文本 private string getFirstNchar string instr int firstN bool withLink if m outstr m outstr instr Clone as string m outstr new Regex m w W R egexOptions Multiline RegexOptions IgnoreCase Replace m outstr m outstr new Regex m w W Reg exOptions Multiline RegexOptions IgnoreCase Replace m outstr m outstr new Regex m w W R egexOptions Multiline RegexOptions IgnoreCase Replace m outstr if withLink m outstr new Regex m w W RegexOptions Multiline RegexOptions IgnoreCase Replace m outstr Regex objReg new System Text RegularExpressions Regex RegexOptions Multiline RegexOptions IgnoreCase m outstr objReg Replace m outstr Regex objReg2 new System Text RegularExpressions Regex s RegexOptions Multiline RegexOptions IgnoreCase m outstr objReg2 Replace m outstr return m outstr Length firstN m outstr Substring 0 firstN m out str 此私有方法返回一个 IP 地址对应的无符号整数 IP 地址 private uint getuintFromIP IPAddress x Byte bt x GetAddressBytes uint i uint bt 0 256 256 256 i uint bt 1 256 256 i uint bt 2 256 i uint bt 3 return i endregion region 公有文法 此公有方法提取网页中一定字数的纯文本 包括链接文字 字数 public string getContext int firstN return getFirstNchar m html firstN true 此公有方法提取网页中一定字数的纯文本 不包括链接文字 public string getContextWithOutLink int firstN return getFirstNchar m html firstN false 此公有方法从本网页的链接中提取一定数量的链接 该链接的 URL 满足某正则式 正则式 返回的链接的个数 List public List getSpecialLinksByUrl string pattern int count if m links Count 0 getLinks List SpecialLinks new List List Enumerator i i m links GetEnumerator int cnt 0 while i MoveNext cnt return SpecialLinks 此公有方法从本网页的链接中提取一定数量的链接 该链接的文字满足某正则式 正则式 返回的链接的个数 List public List getSpecialLinksByText string pattern int count if m links Count 0 getLinks List SpecialLinks new List List Enumerator i i m links GetEnumerator int cnt 0 while i MoveNext cnt return SpecialLinks 此公有方法获得所有链接中在一定 IP 范围的链接 起始 IP 终止 IP public List getSpecialLinksByIP string ip start string ip end IPAddress ip start IPAddress Parse ip start IPAddress ip end IPAddress Parse ip end if m links Count 0 getLinks List SpecialLinks new List List Enumerator i i m links GetEnumerator while i MoveNext IPAddress ip try ip Dns GetHostEntry new Uri i Current url Host AddressList 0 catch continue if getuintFromIP ip getuintFromIP ip start return SpecialLinks 这公有方法提取本网页的纯文本中满足某正则式的文字 正则式 返回文字 public string getSpecialWords string pattern if m outstr getContext Int16 MaxValue Regex regex new Regex pattern RegexOptions Multiline RegexOptions IgnoreCase Match mc regex Match m outstr if mc Success return mc Groups 1 Value return string Empty endregion region 构造函数 private void Init string url try m uri new Uri url m links new List m html m outstr m title m good true if url EndsWith rar url EndsWith dat url EndsWith msi m good false return HttpWebRequest rqst HttpWebRequest WebRequest Create m uri rqst AllowAutoRedirect true rqst MaximumAutomaticRedirections 3 rqst UserAgent Mozilla 4 0 compatible MSIE 5 01 Windows NT 5 0 rqst KeepAlive true rqst Timeout 30000 lock WebPage webcookies if WebPage webcookies ContainsKey m uri Host rqst CookieContainer WebPage webcookies m uri Host else CookieContainer cc new CookieContainer WebPage webcookies m uri Host cc rqst CookieContainer cc HttpWebResponse rsps HttpWebResponse rqst GetResponse Stream sm rsps GetResponseStream if rsps ContentType ToLower StartsWith text rsps Content Length 1 22 rsps Close m good false return Encoding cding System Text Encoding Default string contenttype rsps ContentType ToLower int ix contenttype IndexOf charset if ix 1 try cding System Text Encoding GetEncoding rsps ContentType S ubstring ix charset Length 1 catch cding Encoding Default m html new StreamReader sm cding ReadToEnd else m html new StreamReader sm cding ReadToEnd Regex regex new Regex charset RegexOptio ns IgnoreCase string strcding regex Match m html Groups cding Value try cding Encoding GetEncoding strcding catch cding Encoding Default byte bytes Encoding Default GetBytes m html ToCharArray m html cding GetString bytes if m html Split Length 100 m html Encoding Default GetString bytes m pagesize m html Length m uri rsps ResponseUri rsps Close catch Exception ex Console WriteLine ex Message m uri ToString m good false 第二部分 public WebPage string url string uurl try uurl Uri UnescapeDataString url url uurl catch Regex re new Regex x00 xff Match mc re Match url if mc Success string han mc Groups h Value url url Replace han System Web HttpUtility UrlEncode han Enco ding GetEncoding GB2312 Init url public WebPage string url string loginurl string post string uurl try uurl Uri UnescapeDataString url url uurl catch Regex re new Regex x00 xff Match mc re Match url if mc Success string han mc Groups h Value url url Replace han System Web HttpUtility UrlEncode han Enco ding GetEncoding GB2312 if loginurl Trim post Trim WebPage webcookies ContainsKey new Uri url Host Init url else region 登陆 string indata post m post post m loginurl loginurl byte bytes Encoding Default GetBytes post CookieContainer myCookieContainer new CookieContainer try 新建一个 CookieContainer 来存放 Cookie 集合 HttpWebRequest myHttpWebRequest HttpWebRequest WebRequest Cr eate loginurl 新建一个 HttpWebRequest myHttpWebRequest ContentType application x www form urlencoded myHttpWebRequest AllowAutoRedirect false myHttpWebRequest UserAgent Mozilla 4 0 compatible MSIE 5 0 1 Windows NT 5 0 myHttpWebRequest Timeout 60000 myHttpWebRequest KeepAlive true myHttpWebRequest ContentLength bytes Length myHttpWebRequest Method POST myHttpWebRequest CookieContainer myCookieContainer 设置 HttpWebRequest 的 CookieContainer 为刚才建立的那个 myCookieContainer Stream myRequestStream myHttpWebRequest GetRequestStream myRequestStream Write bytes 0 bytes Length myRequestStream Close HttpWebResponse myHttpWebResponse HttpWebResponse myHttpWebR equest GetResponse foreach Cookie ck in myHttpWebResponse Cookies myCookieContainer Add ck myHttpWebResponse Close catch Init url return endregion region 登陆后再访问页面 try m uri new Uri url m links new List m html m outstr m title m good true if url EndsWith rar url EndsWith dat url Ends With msi m good false return HttpWebRequest rqst HttpWebRequest WebRequest Create m uri rqst AllowAutoRedirect true rqst MaximumAutomaticRedirections 3 rqst UserAgent Mozilla 4 0 compatible MSIE 5 01 Windows N T 5 0 rqst KeepAlive true rqst Timeout 30000 rqst CookieContainer myCookieContainer lock WebPage webcookies WebPage webcookies m uri Host myCookieContainer HttpWebResponse rsps HttpWebResponse rqst GetResponse Stream sm rsps GetResponseStream if rsps ContentType ToLower StartsWith text rsps Con tentLength 1 22 rsps Close m good false return Encoding cding System Text Encoding Default int ix rsps ContentType ToLower IndexOf charset if ix 1 try cding System Text Encoding GetEncoding rsps ContentTy pe Substring ix charset Length 1 catch cding Encoding Default m html new StreamReader sm cding ReadToEnd m pagesize m html Length m uri rsps ResponseUri rsps Close catch Exception ex Console WriteLine ex Message m uri ToString m good false endregion endregion region 属性 通过此属性可获得本网页的网址 只读 public string URL ge