Styjun

I am using a Scrapy CrawlSpider to crawl websites and process on their page content. For this I am using the Scrapy Docs Crawlspider Example.

A particular page on the links takes in a parameter target via GET request (say http://www.example.com?target=x) and redirects (302) if the value is erroneous. On receiving this 302 HTTP response, scrapy follows the redirect, but doesn't processes the response in parse_item method, as intended by me.

I came across a few solutions suggesting meta/dont_redirect/http_status_list but none seem to be taking effect.

Please suggest how can I parse the response of 302 redirection, without/before following on the 302 redirected location.

Scrapy version: 0.24.6

To disable redirects you should add meta={'dont_redirect': True) to your yielded scrapy.Requests.

so your spider should look something like this:

import scrapy
class MySpider(scrapy.Spider):
 name = 'myspider'
 start_urls = ['http://example.com',]
 def start_requests(self):
 for url in self.start_urls:
 yield scrapy.Request(url, meta='dont_redirect':True)

What happens here is that scrapy has a default downloader middleware called RedirectMiddleware which is enabled by default and handles all redirections, by supplying this meta argument you are telling this middleware to not do it's job for this particular request.

if you want to disable redirects for every request(which usually is not the best idea) you can just add

REDIRECTS_ENABLED = False

to your settings.py in scrapy project.

There is a brilliant illustration on scrapy docs on how all of the scrapy pieces, like middlewares and spiders, work together:
http://doc.scrapy.org/en/latest/topics/architecture.html

class LagouSpider(CrawlSpider):
 handle_httpstatus_list = [302]
 meta = 'dont_redirect': True, "handle_httpstatus_list": [302]
 name = 'lagou'
 allowed_domains = ['www.lagou.com']
 start_urls = ['https://www.lagou.com']
 login_url = "https://passport.lagou.com/login/login.html"
 custom_settings = 'REDIRECT_ENABLED': False
 rules = (
 Rule(LinkExtractor(allow=("zhaopin/.*",)), follow=True),
 Rule(LinkExtractor(allow=("gongsi/jd+.html",)), follow=True),
 Rule(LinkExtractor(allow=r'jobs/d+.html'), callback='parse_job', follow=True),
 )
 headers = 
 'Accept': 'application/json, text/javascript, */*; q=0.01',
 'Accept-Language': 'zh-CN,zh;q=0.9',
 'Connection': 'keep-alive',
 'Host': 'www.lagou.com',
 'Referer': 'https://www.lagou.com/',
 'X-Anit-Forge-Code': '0',
 'X-Anit-Forge-Token': 'None',
 'Accept-Encoding': 'gzip, deflate, br',
 'X-Requested-With': 'XMLHttpRequest'
 


 def start_requests(self):
 global rc, im
 browser = webdriver.Chrome(executable_path="/home/wqh/下载/chromedriver")
 browser.get(self.login_url)
 # ··········(some code)

 return [scrapy.Request(self.start_urls[0], cookies=cookie_dict, 
 meta=self.meta)]
 def parse_job(self, response):
 if response.status == 302:
 print("302")
 time.sleep(100)