Styjun

In Scrapy how can I use different callback functions for allowed domains, and denied domains.

I'm using the following rules:

rules = [Rule(LinkExtractor(allow=(), deny_domains = allowed_domains), callback='parse_denied_item', follow=True),
Rule(LinkExtractor(allow_domains = allowed_domains), callback='parse_item', follow=True)]

Basically I want parse_item to be called whenever there is a request from an allowed_domain (or sub-domain of one of those domains). Then I want parse_denied_item to be called for all requests that are not whitelisted by allowed_domains.

How can I do this?

I believe the best approach is not to use allowed_domains on LinkExtractor, and instead parse the domain out of response.url in your parse_* method and perform a different logic depending on the domain.

You can keep separate parse_* methods and a triaging method that, depending on the domains, calls yield from self.parse_*(response) (Python 3) with the corresponding parse_* method:

rules = [Rule(LinkExtractor(), callback='parse_all', follow=True)]

def parse_all(self, response):
 # [Get domain out of response.url]
 if domain in allowed_domains:
 yield from self.parse_item(response)
 else:
 yield from self.parse_denied_item(response)

Based on Gallaecio's answer. An alternate option is to use process_request of Rule. process_request will capture the request before it is sent.

From my understanding (which could be wrong) Scrapy will only crawl domains listed in self.allowed_domains (assuming its used). However, if an offsite link is encountered on a scraped page, Scrapy will send a single request to this offsite link in some cases [1]. I'm not sure why this happens. I think this is possibly occurring because the target site is performing a 301, or 302 redirect and the crawler is automatically following that URL. Otherwise, it's probably a bug.

process_request can be used be used to perform processing on a request before it is executed. In my case, I wanted to log all links that aren't being crawled. So I'm verifying an allowed domain is in request.url before proceeding, and logging any of those that aren't.

Here is an example:

rules = [Rule(LinkExtractor(), callback='parse_item', process_request='process_item', follow=True)]

def process_item(self, request):
 found = False
 for url in self.allowed_domains:
 if url in request.url:
 #an allowed domain is in the request.url, proceed
 found = True

 if found == False: #otherwise log it
 self.logDeniedDomain(urlparse(request.url).netloc)

 # according to: https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Rule
 # setting request to None should prevent this call from being executed (which is not the case for all)
 # middleware is used to catch these few requests
 request = None

 return request

[1]: If you're encountering this problem, using process_request in Downloader middleware appears to solve it though.

My Downloader middleware:

def process_request(self, request, spider):
 #catch any requests that should be filtered, and ignore them
 found = False
 for url in spider.allowed_domains:
 if url in request.url:
 #an allowed domain is in the request.url, proceed
 found = True

 if found == False:
 print("[ignored] "+request.url)
 raise IgnoreRequest('Offsite link, ignore')

 return None

Make sure you import IgnoreRequest as well:

from scrapy.exceptions import IgnoreRequest

and enable the Downloader middleware in settings.py.

To verify this, you can add some verification code in process_item of your crawler to ensure no requests to out of scope sites have been made.