Styjun

Is it a problem that pull requests are approved without any comments

Will TSA allow me to carry a Continuous Positive Airway Pressure (CPAP) device?

What does War Machine's "Canopy! Canopy!" line mean in "Avengers: Endgame"?

I wrote a scene that the majority of my readers loved. How do I get back to that place while writing my new book?

Java 8: How to convert String to Map<String,List<String>>?

Align text within align

Do I include animal companions when calculating difficulty of an encounter?

How to connect an offset point symbol to its original position in QGIS?

Short story written from alien perspective with this line: "It's too bright to look at, so they don't"

California: "For quality assurance, this phone call is being recorded"

3 as a Sum of 3 Pan Digital Expressions

Explain Ant-Man's "not it" scene from Avengers: Endgame

Is there any rule preventing me from starting multiple bardic performances in a single round?

Who operates delivery flights for commercial airlines?

Pronoun introduced before its antecedent

Is there any word or phrase for negative bearing?

Credit card offering 0.5 miles for every cent rounded up. Too good to be true?

Does any lore text explain why the planes of Acheron, Gehenna, and Carceri are the alignment they are?

Diet Coke or water?

Past participle agreement with the subject in the case of pronominal verbs

Incremental Ranges!

Is it legal in the UK for politicians to lie to the public for political gain?

Does the growth of home value benefit from compound interest?

Personalization conditions switching doesn`t work in Experience Editor (9.1.0, Initial Release)

Scrapy from outside of project - empty settings

Django add management command without installing as appHow do I check if a list is empty?How to add timeout to Deferred from Twisted's deferToThread API?building scrapy spiders into my own program, i don't want to call scrapy from command line)Why my Scrapy project stop scraping, but still scrawling website wellCan't run Scrapy project in DjangoGetting scrapy project settings when script is outside of root directoryScrapy Shell and Scrapy SplashScrapy: Multiple crawler in one projectCalling Scrapy Spider from DjangoUsing parsel in Scrapy project

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

0

1

I'm trying to run spiders from Django management command.

It works but it doesn't use settings from scrapy project.

django_project/
 django_project/
 app1/
 scraping/ # This is app but it has scrapy project inside too
 scrapy_spider/
 settings.py
 spiders/

When I try to specify settings inside the command, it returns:

ModuleNotFoundError: No module named 'scrapy_spider'

COMMAND

import os
from django.core.management.base import BaseCommand
from scrapy.utils.project import get_project_settings

from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner

from scraping.scrapy_spider.spiders.autoscrape_index_spider import AutoScrapeIndexSpider
from scraping.scrapy_spider.spiders.autoscrape_spider import AutoScrapeSpider


class Command(BaseCommand):

 def handle(self, *args, **options):
 os.environ['SCRAPY_SETTINGS_MODULE'] = 'scraping.scrapy_spider.settings'
 runner = CrawlerRunner(settings=get_project_settings())

 @defer.inlineCallbacks
 def crawl():
 yield runner.crawl(AutoScrapeIndexSpider)
 yield runner.crawl(AutoScrapeSpider)
 reactor.stop()

 crawl()
 reactor.run()

Do you know how to make it work?

python scrapy

share|improve this question

edited Mar 24 at 14:21

Milano

asked Mar 24 at 13:24

MilanoMilano

4,7531445135

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Maybe it is related to this question: stackoverflow.com/questions/40333400/…
  
  – zeevb
  Mar 24 at 14:13
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Do you have __init__.py in proper places?
  
  – vezunchik
  Mar 24 at 16:31
  

  
  
  
  

add a comment | 

0

1

I'm trying to run spiders from Django management command.

It works but it doesn't use settings from scrapy project.

django_project/
 django_project/
 app1/
 scraping/ # This is app but it has scrapy project inside too
 scrapy_spider/
 settings.py
 spiders/

When I try to specify settings inside the command, it returns:

ModuleNotFoundError: No module named 'scrapy_spider'

COMMAND

import os
from django.core.management.base import BaseCommand
from scrapy.utils.project import get_project_settings

from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner

from scraping.scrapy_spider.spiders.autoscrape_index_spider import AutoScrapeIndexSpider
from scraping.scrapy_spider.spiders.autoscrape_spider import AutoScrapeSpider


class Command(BaseCommand):

 def handle(self, *args, **options):
 os.environ['SCRAPY_SETTINGS_MODULE'] = 'scraping.scrapy_spider.settings'
 runner = CrawlerRunner(settings=get_project_settings())

 @defer.inlineCallbacks
 def crawl():
 yield runner.crawl(AutoScrapeIndexSpider)
 yield runner.crawl(AutoScrapeSpider)
 reactor.stop()

 crawl()
 reactor.run()

Do you know how to make it work?

python scrapy

share|improve this question

edited Mar 24 at 14:21

Milano

asked Mar 24 at 13:24

MilanoMilano

4,7531445135

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Maybe it is related to this question: stackoverflow.com/questions/40333400/…
  
  – zeevb
  Mar 24 at 14:13
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Do you have __init__.py in proper places?
  
  – vezunchik
  Mar 24 at 16:31
  

  
  
  
  

add a comment | 

I'm trying to run spiders from Django management command.

It works but it doesn't use settings from scrapy project.

django_project/
 django_project/
 app1/
 scraping/ # This is app but it has scrapy project inside too
 scrapy_spider/
 settings.py
 spiders/

When I try to specify settings inside the command, it returns:

ModuleNotFoundError: No module named 'scrapy_spider'

COMMAND

import os
from django.core.management.base import BaseCommand
from scrapy.utils.project import get_project_settings

from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner

from scraping.scrapy_spider.spiders.autoscrape_index_spider import AutoScrapeIndexSpider
from scraping.scrapy_spider.spiders.autoscrape_spider import AutoScrapeSpider


class Command(BaseCommand):

 def handle(self, *args, **options):
 os.environ['SCRAPY_SETTINGS_MODULE'] = 'scraping.scrapy_spider.settings'
 runner = CrawlerRunner(settings=get_project_settings())

 @defer.inlineCallbacks
 def crawl():
 yield runner.crawl(AutoScrapeIndexSpider)
 yield runner.crawl(AutoScrapeSpider)
 reactor.stop()

 crawl()
 reactor.run()

Do you know how to make it work?

python scrapy

share|improve this question

edited Mar 24 at 14:21

Milano

asked Mar 24 at 13:24

MilanoMilano

4,7531445135

django_project/
 django_project/
 app1/
 scraping/ # This is app but it has scrapy project inside too
 scrapy_spider/
 settings.py
 spiders/

ModuleNotFoundError: No module named 'scrapy_spider'

import os
from django.core.management.base import BaseCommand
from scrapy.utils.project import get_project_settings

from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner

from scraping.scrapy_spider.spiders.autoscrape_index_spider import AutoScrapeIndexSpider
from scraping.scrapy_spider.spiders.autoscrape_spider import AutoScrapeSpider


class Command(BaseCommand):

 def handle(self, *args, **options):
 os.environ['SCRAPY_SETTINGS_MODULE'] = 'scraping.scrapy_spider.settings'
 runner = CrawlerRunner(settings=get_project_settings())

 @defer.inlineCallbacks
 def crawl():
 yield runner.crawl(AutoScrapeIndexSpider)
 yield runner.crawl(AutoScrapeSpider)
 reactor.stop()

 crawl()
 reactor.run()

share|improve this question

edited Mar 24 at 14:21

Milano

asked Mar 24 at 13:24

MilanoMilano

4,7531445135

share|improve this question

edited Mar 24 at 14:21

Milano

asked Mar 24 at 13:24

MilanoMilano

4,7531445135

asked Mar 24 at 13:24

MilanoMilano

4,7531445135

asked Mar 24 at 13:24

MilanoMilano

4,7531445135