Geek asks and answers

Programmatically loading a large number of types <insert file here here>

I am wondering if there is an easy way to download a large number of files of one arbitrary type, for example, upload 10,000 XML files. I used to use the Bing API. It is free and offers unlimited inquiries. However, it does not index as many file types as Google. Google indexes XML files, CSV files, and KML files. (You can find all this by doing a search like "filetype: XML".) As far as I know, Bing does not index them in such a way as to easily find information. Is there any other API that has these features?

+3
source share
1 answer

How about using wget? You can specify a wgetURL (for example, a Google search result) and tell him to follow all the links on this page and load them (I'm sure you could also give him a filter).

Just tried and received. ERROR 403: Forbidden.Obviously, Google blocks requests from Wget. You will need to provide another user agent. A quick search provided this example:

http://www.mail-archive.com/ wget@sunsite.dk /msg06564.html

Then he worked with the given example.

0
source

More articles:


All Articles