Geek asks and answers

Python urllib2 errors

I have a long list of URLs that I need to open for my service. Whenever I try to open the whole list, I get a lot of errors, such as when the program starts: [Errno 24] Too many open files. I am using urllib2 and gevent.

Does anyone have any solutions?

Thank.

+5
source share
2 answers

Use Gevent.pool to open a certain number of URLs at a time, and then reuse these green dots when you open the URL.

+2
source

, , : URL-, . / -- , .

concurrency URL-: URL- . . 5 10 , .

0

More articles:


All Articles