I am transferring some parsers from BeautifulSoup3 to BeautifulSoup4, and I thought it would be nice to work on how fast it would work, considering that lxml is very fast and this is the parser that I use with BS4, here are the profile results:
For BS3:
43208 function calls (42654 primitive calls) in 0.103 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:2(<module>)
18 0.000 0.000 0.000 0.000 <string>:8(__new__)
1 0.000 0.000 0.072 0.072 <string>:9(parser)
32 0.000 0.000 0.000 0.000 BeautifulSoup.py:1012(__init__)
1 0.000 0.000 0.000 0.000 BeautifulSoup.py:1018(buildTagMap)
...
For BS4 using lxml:
164440 function calls (163947 primitive calls) in 0.244 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.040 0.040 0.069 0.069 <string>:2(<module>)
18 0.000 0.000 0.000 0.000 <string>:8(__new__)
1 0.000 0.000 0.158 0.158 <string>:9(parser)
1 0.000 0.000 0.008 0.008 HTMLParser.py:1(<module>)
1 0.000 0.000 0.000 0.000 HTMLParser.py:54(HTMLParseError)
...
why it BS4causes 4 times more functions? why does it use HTMLParserat all if i installed it for use lxml?
The most notable things that I changed from BS3 to BS4 were as follows:
BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES) --->
BeautifulSoup(html, 'lxml')
[x.getText('**SEP**') for x in i.findChildren('font')[:2]] --->
[x.getText('**SEP**', strip=True) for x in i.findChildren('font')[:2]]
everything else is just name changes (e.g. findParent -> find_parent)
EDIT:
my environment:
python 2.7.3
beautifulsoup4==4.1.0
lxml==2.3.4
EDIT 2:
Here is a small sample code to try:
from cProfile import Profile
from BeautifulSoup import BeautifulSoup
from bs4 import BeautifulSoup as BS4
import urllib2
def parse(html):
soup = BS4(html, 'lxml')
hl = soup.find_all('span', {'class': 'mw-headline'})
return [x.get_text(strip=True) for x in hl]
def parse3(html):
soup = BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES)
hl = soup.findAll('span', {'class': 'mw-headline'})
return [x.getText() for x in hl]
if __name__ == "__main__":
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
html = ''.join(opener.open('http://en.wikipedia.org/wiki/Price').readlines())
profiler = Profile()
print profiler.runcall(parse, html)
profiler.print_stats()
profiler2 = Profile()
print profiler2.runcall(parse3, html)
profiler2.print_stats()