Geek asks and answers

Scrapy0.22: An error occurred while connecting: <class 'twisted.internet.error.ConnectionLost'>

Good morning,

I get a connection error while executing one of my spiders:

2014-02-28 10:21:00+0400 [butik] DEBUG: Retrying <GET http://www.butik.ru/> (failed 1 times): An error occurred while connecting: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.].

Then the spider closes.

All other spiders with a familiar structure work smoothly, but this one:

class butik(Spider):
    name = "butik"
    allowed_domains = ['butik.ru']
    start_urls      = ['http://www.butik.ru/']

    def parse(self, response): 
        sel = Selector(response)
        print response.url
        maincats = sel.xpath('//div[@id="main_menu"]//a/@href').extract()
        for maincat in maincats:
            maincat = 'http://www.butik.ru'+ maincat 
            request = Request(maincat, callback=self.categories)
            yield request

I absolutely do not know what steps should be taken to fix this problem, and I am happy for any tips and answers. If you need more information, I would be happy to provide the necessary code.

Thanks in advance

J

0
source share
1 answer

urllib2. , scrapy , , urllib2 parse:

import urllib2

def parse(self,response):
    # ...
    url = 'www.example.com'
    req = urllib2.Request(url,data)
    response = urllib2.urlopen(req)
    the_page = response.read()
    # ...
0

More articles:


All Articles