Python使用BeautifulSoup包编写爬虫时的一些关键点

比如对于已经得到了如下html代码：

  November 
 
  
   2012 
   
 
 

的soup变量eachMonthHeader了。

想要提取其中的

Month的label的值：November

最简单，也是最省事的办法是，直接搜两个label，然后肯定会找到这两个label，然后分别对应着Month和Year的label，然后获得对应的string即可：

foundTwoLabel = eachMonthHeader.findAll("label");
print "foundTwoLabel=",foundTwoLabel;
monthLabel = foundTwoLabel[0];
yearLabel = foundTwoLabel[1];
 
monthStr = monthLabel.string;
yearStr = yearLabel.string;
 
print "monthStr=",monthStr; # monthStr= November
print "yearStr=",yearStr; # yearStr= 2012
 

但是很明显，这样的逻辑性很不好，而且万一处理多个这样的soup变量，而且两者的顺便颠倒了，那么结果也就错误了。

此时，可以考虑利用soup变量的parent属性，从一个soup变量本身，获得其上一级的soup变量。
示例代码如下：

#   November 
#  
#   
  #  2012 
  #  
#  
foundCboMonth = eachMonthHeader.find("input", {"id":re测试数据pile("cboMonth\d+")});
#print "foundCboMonth=",foundCboMonth;
tdMonth = foundCboMonth.parent;
#print "tdMonth=",tdMonth;
tdMonthLabel = tdMonth.label;
#print "tdMonthLabel=",tdMonthLabel;
monthStr = tdMonthLabel.string;
print "monthStr=",monthStr;
 
foundCboYear = eachMonthHeader.find("input", {"id":re测试数据pile("cboYear\d+")});
#print "foundCboYear=",foundCboYear;
tdYear = foundCboYear.parent;
#print "tdYear=",tdYear;
tdYearLabel = tdYear.label;
#print "tdYearLabel=",tdYearLabel;
yearStr = tdYearLabel.string;
print "yearStr=",yearStr;

 

我们再来看一个例子：

from BeautifulSoup import BeautifulSoup 
doc = ['   Page title  ',
    '
This is paragraph one .',
    '
This is paragraph two .',
    ' ']
soup = BeautifulSoup(''.join(doc))

print soup.prettify()
#  
#  
#   
#  Page title
#   
#  
# 
#  

#  This is paragraph
#  
#   one
#   
#  .
#  

#  

#  This is paragraph
#  
#   two
#   
#  .
#  

#  
#  

 

soup.head.parent.name
# u'html'
soup.head.parent.parent.__class__.__name__
# 'BeautifulSoup'
soup.parent == None
# True

 

#这里respHtml是ASCII或UTF-8编码，此时可以不指定编码类型，即可正确解析出对应的soup
soup = BeautifulSoup(respHtml);
 

#此处respHtml是GB2312编码的，所以要指定该编码类型，BeautifulSoup才能解析出对应的soup
htmlCharset = "GB2312";
soup = BeautifulSoup(respHtml, fromEncoding=htmlCharset);