newbie programmer and lurker here, hoping for reasonable advice. :)
Using a combination of Python, BeautifulSoup, and the Bing API, I was able to find what I wanted with the following code:
import urllib2
from BeautifulSoup import BeautifulStoneSoup
Appid =
query =
soup = BeautifulStoneSoup(urllib2.urlopen("http://api.search.live.net/xml.aspx?Appid=" + Appid + "&query=" + query + "&sources=web"))
totalResults = soup.find('web:total').text
Therefore, I would like to do this through several thousand search queries, and I was wondering if
- execution of this request a thousand times will be interpreted as clogging the server,
- What steps should I take to avoid clogging the servers (what are the best practices?) And
- Is there a cheaper (data) way to do this using any of the main search engine APIs?
It seems unnecessarily expensive to capture all this data to capture one number per keyword, and I was wondering if I had missed anything.
FWIW, API Google () API Yahoo BOSS ( ), Bing API. , , .