How to create a single executable file in windows 10 with scrapy and pyinstaller?python scrapy conversion to exe file using pyinstallerHow to merge two dictionaries in a single expression?How do I check whether a file exists without exceptions?How do I copy a file in Python?How can I safely create a nested directory?How do I list all files of a directory?How to read a file line-by-line into a list?How do I install pip on Windows?how to build my scrapy spider to an exe file using py2exe?How to build a stand alone Scrapy Spider?Creating Python Tkinter exe file with Pyinstaller problem
How can I ping multiple IP addresses at the same time?
Co-worker is now managing my team. Does this mean that I'm being demoted?
Do my partner and son need an SSN to be dependents on my taxes?
Is this broken pipe the reason my freezer is not working? Can it be fixed?
Is there a risk to write an invitation letter for a stranger to obtain a Czech (Schengen) visa?
How to ask if I can mow my neighbor's lawn
Are there any individual aliens that have gained superpowers in the Marvel universe?
How would Japanese people react to someone refusing to say “itadakimasu” for religious reasons?
How could I create a situation in which a PC has to make a saving throw or be forced to pet a dog?
In windows systems, is renaming files functionally similar to deleting them?
How to sort human readable size
Fibonacci sequence and other metallic sequences emerged in the form of fractions
Fantasy game inventory — Ch. 5 Automate the Boring Stuff
Having some issue with notation in a Hilbert space
Does cooling a potato change the nature of its carbohydrates?
Can you place a web spell on a surface you cannot see?
Fill the maze with a wall-following Snake until it gets stuck
What is this plant I saw for sale at a Romanian farmer's market?
How to make all magic-casting innate, but still rare?
First occurrence in the Sixers sequence
How "fast" do astronomical events occur?
How to address players struggling with simple controls?
Digital signature that is only verifiable by one specific person
Got a new frameset, don't know why I need this split ring collar?
How to create a single executable file in windows 10 with scrapy and pyinstaller?
python scrapy conversion to exe file using pyinstallerHow to merge two dictionaries in a single expression?How do I check whether a file exists without exceptions?How do I copy a file in Python?How can I safely create a nested directory?How do I list all files of a directory?How to read a file line-by-line into a list?How do I install pip on Windows?how to build my scrapy spider to an exe file using py2exe?How to build a stand alone Scrapy Spider?Creating Python Tkinter exe file with Pyinstaller problem
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I have created a scrapy spider and successfully converted to windows executable using pyinstaller with disc folder.
In order to do that, I have to make some slight changes in the scrapy site-packages and add those packages in the windows disc folder, it works perfectly,
How can I make this into a single exe with the commented scrapy packages from the disc folder?
I have already tried with --OneFile command in pyinstaller, but it shows the scrapy error?
python scrapy pyinstaller
edited Mar 25 at 5:26
Paradox
564724
asked Mar 25 at 4:58
Arun AugustineArun Augustine
304110
add a comment |
I have created a scrapy spider and successfully converted to windows executable using pyinstaller with disc folder.
In order to do that, I have to make some slight changes in the scrapy site-packages and add those packages in the windows disc folder, it works perfectly,
How can I make this into a single exe with the commented scrapy packages from the disc folder?
I have already tried with --OneFile command in pyinstaller, but it shows the scrapy error?
python scrapy pyinstaller
edited Mar 25 at 5:26
Paradox
564724
asked Mar 25 at 4:58
Arun AugustineArun Augustine
304110
What is the error ?
– error404
Mar 25 at 5:28
ImportError: No module named 'scrapy.spiderloader', I am calling scrapy from a script and spider details are passing inside crawl process() function. Any idea about how to tell the pyinstaller to fetch my updated scrapy package from a location?
– Arun Augustine
Mar 25 at 5:43
have you tried pyinstaller's hiddenimport ? For example: --hidden-import=saidmodule.py
– RockAndRoleCoder
Mar 25 at 6:09
I have tried with hidden imports for package scrapy, from the command line as well as the .spec file. both show the same error.
– Arun Augustine
Mar 25 at 8:39
add a comment |
I have created a scrapy spider and successfully converted to windows executable using pyinstaller with disc folder.
In order to do that, I have to make some slight changes in the scrapy site-packages and add those packages in the windows disc folder, it works perfectly,
How can I make this into a single exe with the commented scrapy packages from the disc folder?
I have already tried with --OneFile command in pyinstaller, but it shows the scrapy error?
python scrapy pyinstaller
edited Mar 25 at 5:26
Paradox
564724
asked Mar 25 at 4:58
Arun AugustineArun Augustine
304110
I have created a scrapy spider and successfully converted to windows executable using pyinstaller with disc folder.
In order to do that, I have to make some slight changes in the scrapy site-packages and add those packages in the windows disc folder, it works perfectly,
How can I make this into a single exe with the commented scrapy packages from the disc folder?
I have already tried with --OneFile command in pyinstaller, but it shows the scrapy error?
python scrapy pyinstaller
python scrapy pyinstaller
edited Mar 25 at 5:26
Paradox
564724
asked Mar 25 at 4:58
Arun AugustineArun Augustine
304110
edited Mar 25 at 5:26
Paradox
564724
asked Mar 25 at 4:58
Arun AugustineArun Augustine
304110
edited Mar 25 at 5:26
Paradox
564724
edited Mar 25 at 5:26
Paradox
564724
edited Mar 25 at 5:26
Paradox
564724
564724
asked Mar 25 at 4:58
Arun AugustineArun Augustine
304110
asked Mar 25 at 4:58
Arun AugustineArun Augustine
304110
asked Mar 25 at 4:58
Arun AugustineArun Augustine
304110
304110
What is the error ?
– error404
Mar 25 at 5:28
ImportError: No module named 'scrapy.spiderloader', I am calling scrapy from a script and spider details are passing inside crawl process() function. Any idea about how to tell the pyinstaller to fetch my updated scrapy package from a location?
– Arun Augustine
Mar 25 at 5:43
have you tried pyinstaller's hiddenimport ? For example: --hidden-import=saidmodule.py
– RockAndRoleCoder
Mar 25 at 6:09
I have tried with hidden imports for package scrapy, from the command line as well as the .spec file. both show the same error.
– Arun Augustine
Mar 25 at 8:39
add a comment |
What is the error ?
– error404
Mar 25 at 5:28
ImportError: No module named 'scrapy.spiderloader', I am calling scrapy from a script and spider details are passing inside crawl process() function. Any idea about how to tell the pyinstaller to fetch my updated scrapy package from a location?
– Arun Augustine
Mar 25 at 5:43
have you tried pyinstaller's hiddenimport ? For example: --hidden-import=saidmodule.py
– RockAndRoleCoder
Mar 25 at 6:09
I have tried with hidden imports for package scrapy, from the command line as well as the .spec file. both show the same error.
– Arun Augustine
Mar 25 at 8:39
What is the error ?
– error404
Mar 25 at 5:28
What is the error ?
– error404
Mar 25 at 5:28
ImportError: No module named 'scrapy.spiderloader', I am calling scrapy from a script and spider details are passing inside crawl process() function. Any idea about how to tell the pyinstaller to fetch my updated scrapy package from a location?
– Arun Augustine
Mar 25 at 5:43
ImportError: No module named 'scrapy.spiderloader', I am calling scrapy from a script and spider details are passing inside crawl process() function. Any idea about how to tell the pyinstaller to fetch my updated scrapy package from a location?
– Arun Augustine
Mar 25 at 5:43
have you tried pyinstaller's hiddenimport ? For example: --hidden-import=saidmodule.py
– RockAndRoleCoder
Mar 25 at 6:09
have you tried pyinstaller's hiddenimport ? For example: --hidden-import=saidmodule.py
– RockAndRoleCoder
Mar 25 at 6:09
I have tried with hidden imports for package scrapy, from the command line as well as the .spec file. both show the same error.
– Arun Augustine
Mar 25 at 8:39
I have tried with hidden imports for package scrapy, from the command line as well as the .spec file. both show the same error.
– Arun Augustine
Mar 25 at 8:39
add a comment |
2 Answers
2
active
oldest
votes
Very similar issue discussed here: python scrapy conversion to exe file using pyinstaller
Initially I used auto-py-to-exe package (which is actually GUI for pyinstaller)
I added following line to auto-py-to-exe -> advanced settings -> hidden import:
scrapy.spiderloader,scrapy.statscollectors,scrapy.logformatter,scrapy.extensions,scrapy.extensions.corestats,scrapy.extensions.corestats,scrapy.extensions.telnet,scrapy.extensions.memusage,scrapy.extensions.memdebug,scrapy.extensions.closespider,scrapy.extensions.feedexport,scrapy.extensions.logstats,scrapy.extensions.spiderstate,scrapy.extensions.throttle,scrapy.core.scheduler,scrapy.squeues,queuelib,scrapy.core.downloader,scrapy.downloadermiddlewares,scrapy.downloadermiddlewares.robotstxt,scrapy.downloadermiddlewares.httpauth,scrapy.downloadermiddlewares.downloadtimeout,scrapy.downloadermiddlewares.defaultheaders,scrapy.downloadermiddlewares.useragent,scrapy.downloadermiddlewares.retry,scrapy.downloadermiddlewares.ajaxcrawl,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.httpcompression,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.cookies,scrapy.downloadermiddlewares.httpproxy,scrapy.downloadermiddlewares.stats,scrapy.downloadermiddlewares.httpcache,scrapy.spidermiddlewares,scrapy.spidermiddlewares.httperror,scrapy.spidermiddlewares.offsite,scrapy.spidermiddlewares.referer,scrapy.spidermiddlewares.urllength,scrapy.spidermiddlewares.depth,scrapy.pipelines,scrapy.dupefilters,scrapy.core.downloader.handlers.datauri,scrapy.core.downloader.handlers.file,scrapy.core.downloader.handlers.http,scrapy.core.downloader.handlers.s3,scrapy.core.downloader.handlers.ftp,scrapy.core.downloader.webclient,scrapy.core.downloader.contextfactory
After that following command appeared in last text box (don't forget to change path to your script) :
pyinstaller -y -F --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.telnet --hidden-import scrapy.extensions.memusage --hidden-import scrapy.extensions.memdebug --hidden-import scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy.extensions.logstats --hidden-import scrapy.extensions.spiderstate --hidden-import scrapy.extensions.throttle --hidden-import scrapy.core.scheduler --hidden-import scrapy.squeues --hidden-import queuelib --hidden-import scrapy.core.downloader --hidden-import scrapy.downloadermiddlewares --hidden-import scrapy.downloadermiddlewares.robotstxt --hidden-import scrapy.downloadermiddlewares.httpauth --hidden-import scrapy.downloadermiddlewares.downloadtimeout --hidden-import scrapy.downloadermiddlewares.defaultheaders --hidden-import scrapy.downloadermiddlewares.useragent --hidden-import scrapy.downloadermiddlewares.retry --hidden-import scrapy.downloadermiddlewares.ajaxcrawl --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.httpcompression --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.cookies --hidden-import scrapy.downloadermiddlewares.httpproxy --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy.downloadermiddlewares.httpcache --hidden-import scrapy.spidermiddlewares --hidden-import scrapy.spidermiddlewares.httperror --hidden-import scrapy.spidermiddlewares.offsite --hidden-import scrapy.spidermiddlewares.referer --hidden-import scrapy.spidermiddlewares.urllength --hidden-import scrapy.spidermiddlewares.depth --hidden-import scrapy.pipelines --hidden-import scrapy.dupefilters --hidden-import scrapy.core.downloader.handlers.datauri --hidden-import scrapy.core.downloader.handlers.file --hidden-import scrapy.core.downloader.handlers.http --hidden-import scrapy.core.downloader.handlers.s3 --hidden-import scrapy.core.downloader.handlers.ftp --hidden-import scrapy.core.downloader.webclient --hidden-import scrapy.core.downloader.contextfactory "C:/path/script.py"
If after this your command return: ImportError: No module named 'modulename' - add missing module to hidden imports and repeat this process with new extended hidden imports.
(I repeated this procedure 48 times in order to receive working exe file (and receive list of sumbodules)!!)
answered Mar 25 at 13:41
GeorgiyGeorgiy
72919
add a comment |
Fixed it by using --hidden imports in the spec file.
Pyinstaller doesn't support all second level module imports in scrapy.
Run the pyinstaller command, Just update the spec file with below hidden import changes,
hiddenimports=['scrapy.spiderloader','scrapy.statscollectors','scrapy.logformatter','scrapy.extensions','scrapy.extensions.logstats', 'scrapy.extensions.corestats','scrapy.extensions.memusage','scrapy.extensions.feedexport','scrapy.extensions.memdebug', 'scrapy.extensions.closespider','scrapy.extensions.throttle','scrapy.extensions.telnet','scrapy.extensions.spiderstate', 'scrapy.core.scheduler','scrapy.core.downloader','scrapy.downloadermiddlewares','scrapy.downloadermiddlewares.robotstxt', 'scrapy.downloadermiddlewares.httpauth','scrapy.downloadermiddlewares.downloadtimeout','scrapy.downloadermiddlewares.defaultheaders', 'scrapy.downloadermiddlewares.useragent','scrapy.downloadermiddlewares.retry','scrapy.core.downloader.handlers.http', 'scrapy.core.downloader.handlers.s3','scrapy.core.downloader.handlers.ftp','scrapy.core.downloader.handlers.datauri', 'scrapy.core.downloader.handlers.file','scrapy.downloadermiddlewares.ajaxcrawl','scrapy.core.downloader.contextfactory', 'scrapy.downloadermiddlewares.redirect','scrapy.downloadermiddlewares.httpcompression','scrapy.downloadermiddlewares.cookies', 'scrapy.downloadermiddlewares.httpproxy','scrapy.downloadermiddlewares.stats','scrapy.downloadermiddlewares.httpcache', 'scrapy.spidermiddlewares','scrapy.spidermiddlewares.httperror','scrapy.spidermiddlewares.offsite','scrapy.spidermiddlewares.referer', 'scrapy.spidermiddlewares.urllength','scrapy.spidermiddlewares.depth','scrapy.pipelines','scrapy.dupefilters','queuelib', 'scrapy.squeues',]
Fixed with 45 module import issues. Using --onefile helps to run the scrapy project in a single executable. Hope anyone finds it useful.
answered Mar 29 at 3:26
Arun AugustineArun Augustine
304110
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55331478%2fhow-to-create-a-single-executable-file-in-windows-10-with-scrapy-and-pyinstaller%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Very similar issue discussed here: python scrapy conversion to exe file using pyinstaller
Initially I used auto-py-to-exe package (which is actually GUI for pyinstaller)
I added following line to auto-py-to-exe -> advanced settings -> hidden import:
scrapy.spiderloader,scrapy.statscollectors,scrapy.logformatter,scrapy.extensions,scrapy.extensions.corestats,scrapy.extensions.corestats,scrapy.extensions.telnet,scrapy.extensions.memusage,scrapy.extensions.memdebug,scrapy.extensions.closespider,scrapy.extensions.feedexport,scrapy.extensions.logstats,scrapy.extensions.spiderstate,scrapy.extensions.throttle,scrapy.core.scheduler,scrapy.squeues,queuelib,scrapy.core.downloader,scrapy.downloadermiddlewares,scrapy.downloadermiddlewares.robotstxt,scrapy.downloadermiddlewares.httpauth,scrapy.downloadermiddlewares.downloadtimeout,scrapy.downloadermiddlewares.defaultheaders,scrapy.downloadermiddlewares.useragent,scrapy.downloadermiddlewares.retry,scrapy.downloadermiddlewares.ajaxcrawl,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.httpcompression,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.cookies,scrapy.downloadermiddlewares.httpproxy,scrapy.downloadermiddlewares.stats,scrapy.downloadermiddlewares.httpcache,scrapy.spidermiddlewares,scrapy.spidermiddlewares.httperror,scrapy.spidermiddlewares.offsite,scrapy.spidermiddlewares.referer,scrapy.spidermiddlewares.urllength,scrapy.spidermiddlewares.depth,scrapy.pipelines,scrapy.dupefilters,scrapy.core.downloader.handlers.datauri,scrapy.core.downloader.handlers.file,scrapy.core.downloader.handlers.http,scrapy.core.downloader.handlers.s3,scrapy.core.downloader.handlers.ftp,scrapy.core.downloader.webclient,scrapy.core.downloader.contextfactory
After that following command appeared in last text box (don't forget to change path to your script) :
pyinstaller -y -F --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.telnet --hidden-import scrapy.extensions.memusage --hidden-import scrapy.extensions.memdebug --hidden-import scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy.extensions.logstats --hidden-import scrapy.extensions.spiderstate --hidden-import scrapy.extensions.throttle --hidden-import scrapy.core.scheduler --hidden-import scrapy.squeues --hidden-import queuelib --hidden-import scrapy.core.downloader --hidden-import scrapy.downloadermiddlewares --hidden-import scrapy.downloadermiddlewares.robotstxt --hidden-import scrapy.downloadermiddlewares.httpauth --hidden-import scrapy.downloadermiddlewares.downloadtimeout --hidden-import scrapy.downloadermiddlewares.defaultheaders --hidden-import scrapy.downloadermiddlewares.useragent --hidden-import scrapy.downloadermiddlewares.retry --hidden-import scrapy.downloadermiddlewares.ajaxcrawl --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.httpcompression --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.cookies --hidden-import scrapy.downloadermiddlewares.httpproxy --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy.downloadermiddlewares.httpcache --hidden-import scrapy.spidermiddlewares --hidden-import scrapy.spidermiddlewares.httperror --hidden-import scrapy.spidermiddlewares.offsite --hidden-import scrapy.spidermiddlewares.referer --hidden-import scrapy.spidermiddlewares.urllength --hidden-import scrapy.spidermiddlewares.depth --hidden-import scrapy.pipelines --hidden-import scrapy.dupefilters --hidden-import scrapy.core.downloader.handlers.datauri --hidden-import scrapy.core.downloader.handlers.file --hidden-import scrapy.core.downloader.handlers.http --hidden-import scrapy.core.downloader.handlers.s3 --hidden-import scrapy.core.downloader.handlers.ftp --hidden-import scrapy.core.downloader.webclient --hidden-import scrapy.core.downloader.contextfactory "C:/path/script.py"
If after this your command return: ImportError: No module named 'modulename' - add missing module to hidden imports and repeat this process with new extended hidden imports.
(I repeated this procedure 48 times in order to receive working exe file (and receive list of sumbodules)!!)
answered Mar 25 at 13:41
GeorgiyGeorgiy
72919
add a comment |
Very similar issue discussed here: python scrapy conversion to exe file using pyinstaller
Initially I used auto-py-to-exe package (which is actually GUI for pyinstaller)
I added following line to auto-py-to-exe -> advanced settings -> hidden import:
scrapy.spiderloader,scrapy.statscollectors,scrapy.logformatter,scrapy.extensions,scrapy.extensions.corestats,scrapy.extensions.corestats,scrapy.extensions.telnet,scrapy.extensions.memusage,scrapy.extensions.memdebug,scrapy.extensions.closespider,scrapy.extensions.feedexport,scrapy.extensions.logstats,scrapy.extensions.spiderstate,scrapy.extensions.throttle,scrapy.core.scheduler,scrapy.squeues,queuelib,scrapy.core.downloader,scrapy.downloadermiddlewares,scrapy.downloadermiddlewares.robotstxt,scrapy.downloadermiddlewares.httpauth,scrapy.downloadermiddlewares.downloadtimeout,scrapy.downloadermiddlewares.defaultheaders,scrapy.downloadermiddlewares.useragent,scrapy.downloadermiddlewares.retry,scrapy.downloadermiddlewares.ajaxcrawl,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.httpcompression,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.cookies,scrapy.downloadermiddlewares.httpproxy,scrapy.downloadermiddlewares.stats,scrapy.downloadermiddlewares.httpcache,scrapy.spidermiddlewares,scrapy.spidermiddlewares.httperror,scrapy.spidermiddlewares.offsite,scrapy.spidermiddlewares.referer,scrapy.spidermiddlewares.urllength,scrapy.spidermiddlewares.depth,scrapy.pipelines,scrapy.dupefilters,scrapy.core.downloader.handlers.datauri,scrapy.core.downloader.handlers.file,scrapy.core.downloader.handlers.http,scrapy.core.downloader.handlers.s3,scrapy.core.downloader.handlers.ftp,scrapy.core.downloader.webclient,scrapy.core.downloader.contextfactory
After that following command appeared in last text box (don't forget to change path to your script) :
pyinstaller -y -F --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.telnet --hidden-import scrapy.extensions.memusage --hidden-import scrapy.extensions.memdebug --hidden-import scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy.extensions.logstats --hidden-import scrapy.extensions.spiderstate --hidden-import scrapy.extensions.throttle --hidden-import scrapy.core.scheduler --hidden-import scrapy.squeues --hidden-import queuelib --hidden-import scrapy.core.downloader --hidden-import scrapy.downloadermiddlewares --hidden-import scrapy.downloadermiddlewares.robotstxt --hidden-import scrapy.downloadermiddlewares.httpauth --hidden-import scrapy.downloadermiddlewares.downloadtimeout --hidden-import scrapy.downloadermiddlewares.defaultheaders --hidden-import scrapy.downloadermiddlewares.useragent --hidden-import scrapy.downloadermiddlewares.retry --hidden-import scrapy.downloadermiddlewares.ajaxcrawl --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.httpcompression --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.cookies --hidden-import scrapy.downloadermiddlewares.httpproxy --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy.downloadermiddlewares.httpcache --hidden-import scrapy.spidermiddlewares --hidden-import scrapy.spidermiddlewares.httperror --hidden-import scrapy.spidermiddlewares.offsite --hidden-import scrapy.spidermiddlewares.referer --hidden-import scrapy.spidermiddlewares.urllength --hidden-import scrapy.spidermiddlewares.depth --hidden-import scrapy.pipelines --hidden-import scrapy.dupefilters --hidden-import scrapy.core.downloader.handlers.datauri --hidden-import scrapy.core.downloader.handlers.file --hidden-import scrapy.core.downloader.handlers.http --hidden-import scrapy.core.downloader.handlers.s3 --hidden-import scrapy.core.downloader.handlers.ftp --hidden-import scrapy.core.downloader.webclient --hidden-import scrapy.core.downloader.contextfactory "C:/path/script.py"
If after this your command return: ImportError: No module named 'modulename' - add missing module to hidden imports and repeat this process with new extended hidden imports.
(I repeated this procedure 48 times in order to receive working exe file (and receive list of sumbodules)!!)
answered Mar 25 at 13:41
GeorgiyGeorgiy
72919
add a comment |
Very similar issue discussed here: python scrapy conversion to exe file using pyinstaller
Initially I used auto-py-to-exe package (which is actually GUI for pyinstaller)
I added following line to auto-py-to-exe -> advanced settings -> hidden import:
scrapy.spiderloader,scrapy.statscollectors,scrapy.logformatter,scrapy.extensions,scrapy.extensions.corestats,scrapy.extensions.corestats,scrapy.extensions.telnet,scrapy.extensions.memusage,scrapy.extensions.memdebug,scrapy.extensions.closespider,scrapy.extensions.feedexport,scrapy.extensions.logstats,scrapy.extensions.spiderstate,scrapy.extensions.throttle,scrapy.core.scheduler,scrapy.squeues,queuelib,scrapy.core.downloader,scrapy.downloadermiddlewares,scrapy.downloadermiddlewares.robotstxt,scrapy.downloadermiddlewares.httpauth,scrapy.downloadermiddlewares.downloadtimeout,scrapy.downloadermiddlewares.defaultheaders,scrapy.downloadermiddlewares.useragent,scrapy.downloadermiddlewares.retry,scrapy.downloadermiddlewares.ajaxcrawl,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.httpcompression,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.cookies,scrapy.downloadermiddlewares.httpproxy,scrapy.downloadermiddlewares.stats,scrapy.downloadermiddlewares.httpcache,scrapy.spidermiddlewares,scrapy.spidermiddlewares.httperror,scrapy.spidermiddlewares.offsite,scrapy.spidermiddlewares.referer,scrapy.spidermiddlewares.urllength,scrapy.spidermiddlewares.depth,scrapy.pipelines,scrapy.dupefilters,scrapy.core.downloader.handlers.datauri,scrapy.core.downloader.handlers.file,scrapy.core.downloader.handlers.http,scrapy.core.downloader.handlers.s3,scrapy.core.downloader.handlers.ftp,scrapy.core.downloader.webclient,scrapy.core.downloader.contextfactory
After that following command appeared in last text box (don't forget to change path to your script) :
pyinstaller -y -F --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.telnet --hidden-import scrapy.extensions.memusage --hidden-import scrapy.extensions.memdebug --hidden-import scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy.extensions.logstats --hidden-import scrapy.extensions.spiderstate --hidden-import scrapy.extensions.throttle --hidden-import scrapy.core.scheduler --hidden-import scrapy.squeues --hidden-import queuelib --hidden-import scrapy.core.downloader --hidden-import scrapy.downloadermiddlewares --hidden-import scrapy.downloadermiddlewares.robotstxt --hidden-import scrapy.downloadermiddlewares.httpauth --hidden-import scrapy.downloadermiddlewares.downloadtimeout --hidden-import scrapy.downloadermiddlewares.defaultheaders --hidden-import scrapy.downloadermiddlewares.useragent --hidden-import scrapy.downloadermiddlewares.retry --hidden-import scrapy.downloadermiddlewares.ajaxcrawl --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.httpcompression --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.cookies --hidden-import scrapy.downloadermiddlewares.httpproxy --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy.downloadermiddlewares.httpcache --hidden-import scrapy.spidermiddlewares --hidden-import scrapy.spidermiddlewares.httperror --hidden-import scrapy.spidermiddlewares.offsite --hidden-import scrapy.spidermiddlewares.referer --hidden-import scrapy.spidermiddlewares.urllength --hidden-import scrapy.spidermiddlewares.depth --hidden-import scrapy.pipelines --hidden-import scrapy.dupefilters --hidden-import scrapy.core.downloader.handlers.datauri --hidden-import scrapy.core.downloader.handlers.file --hidden-import scrapy.core.downloader.handlers.http --hidden-import scrapy.core.downloader.handlers.s3 --hidden-import scrapy.core.downloader.handlers.ftp --hidden-import scrapy.core.downloader.webclient --hidden-import scrapy.core.downloader.contextfactory "C:/path/script.py"
If after this your command return: ImportError: No module named 'modulename' - add missing module to hidden imports and repeat this process with new extended hidden imports.
(I repeated this procedure 48 times in order to receive working exe file (and receive list of sumbodules)!!)
answered Mar 25 at 13:41
GeorgiyGeorgiy
72919
Very similar issue discussed here: python scrapy conversion to exe file using pyinstaller
Initially I used auto-py-to-exe package (which is actually GUI for pyinstaller)
I added following line to auto-py-to-exe -> advanced settings -> hidden import:
scrapy.spiderloader,scrapy.statscollectors,scrapy.logformatter,scrapy.extensions,scrapy.extensions.corestats,scrapy.extensions.corestats,scrapy.extensions.telnet,scrapy.extensions.memusage,scrapy.extensions.memdebug,scrapy.extensions.closespider,scrapy.extensions.feedexport,scrapy.extensions.logstats,scrapy.extensions.spiderstate,scrapy.extensions.throttle,scrapy.core.scheduler,scrapy.squeues,queuelib,scrapy.core.downloader,scrapy.downloadermiddlewares,scrapy.downloadermiddlewares.robotstxt,scrapy.downloadermiddlewares.httpauth,scrapy.downloadermiddlewares.downloadtimeout,scrapy.downloadermiddlewares.defaultheaders,scrapy.downloadermiddlewares.useragent,scrapy.downloadermiddlewares.retry,scrapy.downloadermiddlewares.ajaxcrawl,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.httpcompression,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.cookies,scrapy.downloadermiddlewares.httpproxy,scrapy.downloadermiddlewares.stats,scrapy.downloadermiddlewares.httpcache,scrapy.spidermiddlewares,scrapy.spidermiddlewares.httperror,scrapy.spidermiddlewares.offsite,scrapy.spidermiddlewares.referer,scrapy.spidermiddlewares.urllength,scrapy.spidermiddlewares.depth,scrapy.pipelines,scrapy.dupefilters,scrapy.core.downloader.handlers.datauri,scrapy.core.downloader.handlers.file,scrapy.core.downloader.handlers.http,scrapy.core.downloader.handlers.s3,scrapy.core.downloader.handlers.ftp,scrapy.core.downloader.webclient,scrapy.core.downloader.contextfactory
After that following command appeared in last text box (don't forget to change path to your script) :
pyinstaller -y -F --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.telnet --hidden-import scrapy.extensions.memusage --hidden-import scrapy.extensions.memdebug --hidden-import scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy.extensions.logstats --hidden-import scrapy.extensions.spiderstate --hidden-import scrapy.extensions.throttle --hidden-import scrapy.core.scheduler --hidden-import scrapy.squeues --hidden-import queuelib --hidden-import scrapy.core.downloader --hidden-import scrapy.downloadermiddlewares --hidden-import scrapy.downloadermiddlewares.robotstxt --hidden-import scrapy.downloadermiddlewares.httpauth --hidden-import scrapy.downloadermiddlewares.downloadtimeout --hidden-import scrapy.downloadermiddlewares.defaultheaders --hidden-import scrapy.downloadermiddlewares.useragent --hidden-import scrapy.downloadermiddlewares.retry --hidden-import scrapy.downloadermiddlewares.ajaxcrawl --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.httpcompression --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.cookies --hidden-import scrapy.downloadermiddlewares.httpproxy --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy.downloadermiddlewares.httpcache --hidden-import scrapy.spidermiddlewares --hidden-import scrapy.spidermiddlewares.httperror --hidden-import scrapy.spidermiddlewares.offsite --hidden-import scrapy.spidermiddlewares.referer --hidden-import scrapy.spidermiddlewares.urllength --hidden-import scrapy.spidermiddlewares.depth --hidden-import scrapy.pipelines --hidden-import scrapy.dupefilters --hidden-import scrapy.core.downloader.handlers.datauri --hidden-import scrapy.core.downloader.handlers.file --hidden-import scrapy.core.downloader.handlers.http --hidden-import scrapy.core.downloader.handlers.s3 --hidden-import scrapy.core.downloader.handlers.ftp --hidden-import scrapy.core.downloader.webclient --hidden-import scrapy.core.downloader.contextfactory "C:/path/script.py"
If after this your command return: ImportError: No module named 'modulename' - add missing module to hidden imports and repeat this process with new extended hidden imports.
(I repeated this procedure 48 times in order to receive working exe file (and receive list of sumbodules)!!)
answered Mar 25 at 13:41
GeorgiyGeorgiy
72919
answered Mar 25 at 13:41
GeorgiyGeorgiy
72919
answered Mar 25 at 13:41
GeorgiyGeorgiy
72919
answered Mar 25 at 13:41
GeorgiyGeorgiy
72919
72919
add a comment |
add a comment |
Fixed it by using --hidden imports in the spec file.
Pyinstaller doesn't support all second level module imports in scrapy.
Run the pyinstaller command, Just update the spec file with below hidden import changes,
hiddenimports=['scrapy.spiderloader','scrapy.statscollectors','scrapy.logformatter','scrapy.extensions','scrapy.extensions.logstats', 'scrapy.extensions.corestats','scrapy.extensions.memusage','scrapy.extensions.feedexport','scrapy.extensions.memdebug', 'scrapy.extensions.closespider','scrapy.extensions.throttle','scrapy.extensions.telnet','scrapy.extensions.spiderstate', 'scrapy.core.scheduler','scrapy.core.downloader','scrapy.downloadermiddlewares','scrapy.downloadermiddlewares.robotstxt', 'scrapy.downloadermiddlewares.httpauth','scrapy.downloadermiddlewares.downloadtimeout','scrapy.downloadermiddlewares.defaultheaders', 'scrapy.downloadermiddlewares.useragent','scrapy.downloadermiddlewares.retry','scrapy.core.downloader.handlers.http', 'scrapy.core.downloader.handlers.s3','scrapy.core.downloader.handlers.ftp','scrapy.core.downloader.handlers.datauri', 'scrapy.core.downloader.handlers.file','scrapy.downloadermiddlewares.ajaxcrawl','scrapy.core.downloader.contextfactory', 'scrapy.downloadermiddlewares.redirect','scrapy.downloadermiddlewares.httpcompression','scrapy.downloadermiddlewares.cookies', 'scrapy.downloadermiddlewares.httpproxy','scrapy.downloadermiddlewares.stats','scrapy.downloadermiddlewares.httpcache', 'scrapy.spidermiddlewares','scrapy.spidermiddlewares.httperror','scrapy.spidermiddlewares.offsite','scrapy.spidermiddlewares.referer', 'scrapy.spidermiddlewares.urllength','scrapy.spidermiddlewares.depth','scrapy.pipelines','scrapy.dupefilters','queuelib', 'scrapy.squeues',]
Fixed with 45 module import issues. Using --onefile helps to run the scrapy project in a single executable. Hope anyone finds it useful.
answered Mar 29 at 3:26
Arun AugustineArun Augustine
304110
add a comment |
Fixed it by using --hidden imports in the spec file.
Pyinstaller doesn't support all second level module imports in scrapy.
Run the pyinstaller command, Just update the spec file with below hidden import changes,
hiddenimports=['scrapy.spiderloader','scrapy.statscollectors','scrapy.logformatter','scrapy.extensions','scrapy.extensions.logstats', 'scrapy.extensions.corestats','scrapy.extensions.memusage','scrapy.extensions.feedexport','scrapy.extensions.memdebug', 'scrapy.extensions.closespider','scrapy.extensions.throttle','scrapy.extensions.telnet','scrapy.extensions.spiderstate', 'scrapy.core.scheduler','scrapy.core.downloader','scrapy.downloadermiddlewares','scrapy.downloadermiddlewares.robotstxt', 'scrapy.downloadermiddlewares.httpauth','scrapy.downloadermiddlewares.downloadtimeout','scrapy.downloadermiddlewares.defaultheaders', 'scrapy.downloadermiddlewares.useragent','scrapy.downloadermiddlewares.retry','scrapy.core.downloader.handlers.http', 'scrapy.core.downloader.handlers.s3','scrapy.core.downloader.handlers.ftp','scrapy.core.downloader.handlers.datauri', 'scrapy.core.downloader.handlers.file','scrapy.downloadermiddlewares.ajaxcrawl','scrapy.core.downloader.contextfactory', 'scrapy.downloadermiddlewares.redirect','scrapy.downloadermiddlewares.httpcompression','scrapy.downloadermiddlewares.cookies', 'scrapy.downloadermiddlewares.httpproxy','scrapy.downloadermiddlewares.stats','scrapy.downloadermiddlewares.httpcache', 'scrapy.spidermiddlewares','scrapy.spidermiddlewares.httperror','scrapy.spidermiddlewares.offsite','scrapy.spidermiddlewares.referer', 'scrapy.spidermiddlewares.urllength','scrapy.spidermiddlewares.depth','scrapy.pipelines','scrapy.dupefilters','queuelib', 'scrapy.squeues',]
Fixed with 45 module import issues. Using --onefile helps to run the scrapy project in a single executable. Hope anyone finds it useful.
answered Mar 29 at 3:26
Arun AugustineArun Augustine
304110
add a comment |
Fixed it by using --hidden imports in the spec file.
Pyinstaller doesn't support all second level module imports in scrapy.
Run the pyinstaller command, Just update the spec file with below hidden import changes,
hiddenimports=['scrapy.spiderloader','scrapy.statscollectors','scrapy.logformatter','scrapy.extensions','scrapy.extensions.logstats', 'scrapy.extensions.corestats','scrapy.extensions.memusage','scrapy.extensions.feedexport','scrapy.extensions.memdebug', 'scrapy.extensions.closespider','scrapy.extensions.throttle','scrapy.extensions.telnet','scrapy.extensions.spiderstate', 'scrapy.core.scheduler','scrapy.core.downloader','scrapy.downloadermiddlewares','scrapy.downloadermiddlewares.robotstxt', 'scrapy.downloadermiddlewares.httpauth','scrapy.downloadermiddlewares.downloadtimeout','scrapy.downloadermiddlewares.defaultheaders', 'scrapy.downloadermiddlewares.useragent','scrapy.downloadermiddlewares.retry','scrapy.core.downloader.handlers.http', 'scrapy.core.downloader.handlers.s3','scrapy.core.downloader.handlers.ftp','scrapy.core.downloader.handlers.datauri', 'scrapy.core.downloader.handlers.file','scrapy.downloadermiddlewares.ajaxcrawl','scrapy.core.downloader.contextfactory', 'scrapy.downloadermiddlewares.redirect','scrapy.downloadermiddlewares.httpcompression','scrapy.downloadermiddlewares.cookies', 'scrapy.downloadermiddlewares.httpproxy','scrapy.downloadermiddlewares.stats','scrapy.downloadermiddlewares.httpcache', 'scrapy.spidermiddlewares','scrapy.spidermiddlewares.httperror','scrapy.spidermiddlewares.offsite','scrapy.spidermiddlewares.referer', 'scrapy.spidermiddlewares.urllength','scrapy.spidermiddlewares.depth','scrapy.pipelines','scrapy.dupefilters','queuelib', 'scrapy.squeues',]
Fixed with 45 module import issues. Using --onefile helps to run the scrapy project in a single executable. Hope anyone finds it useful.
answered Mar 29 at 3:26
Arun AugustineArun Augustine
304110
Fixed it by using --hidden imports in the spec file.
Pyinstaller doesn't support all second level module imports in scrapy.
Run the pyinstaller command, Just update the spec file with below hidden import changes,
hiddenimports=['scrapy.spiderloader','scrapy.statscollectors','scrapy.logformatter','scrapy.extensions','scrapy.extensions.logstats', 'scrapy.extensions.corestats','scrapy.extensions.memusage','scrapy.extensions.feedexport','scrapy.extensions.memdebug', 'scrapy.extensions.closespider','scrapy.extensions.throttle','scrapy.extensions.telnet','scrapy.extensions.spiderstate', 'scrapy.core.scheduler','scrapy.core.downloader','scrapy.downloadermiddlewares','scrapy.downloadermiddlewares.robotstxt', 'scrapy.downloadermiddlewares.httpauth','scrapy.downloadermiddlewares.downloadtimeout','scrapy.downloadermiddlewares.defaultheaders', 'scrapy.downloadermiddlewares.useragent','scrapy.downloadermiddlewares.retry','scrapy.core.downloader.handlers.http', 'scrapy.core.downloader.handlers.s3','scrapy.core.downloader.handlers.ftp','scrapy.core.downloader.handlers.datauri', 'scrapy.core.downloader.handlers.file','scrapy.downloadermiddlewares.ajaxcrawl','scrapy.core.downloader.contextfactory', 'scrapy.downloadermiddlewares.redirect','scrapy.downloadermiddlewares.httpcompression','scrapy.downloadermiddlewares.cookies', 'scrapy.downloadermiddlewares.httpproxy','scrapy.downloadermiddlewares.stats','scrapy.downloadermiddlewares.httpcache', 'scrapy.spidermiddlewares','scrapy.spidermiddlewares.httperror','scrapy.spidermiddlewares.offsite','scrapy.spidermiddlewares.referer', 'scrapy.spidermiddlewares.urllength','scrapy.spidermiddlewares.depth','scrapy.pipelines','scrapy.dupefilters','queuelib', 'scrapy.squeues',]
Fixed with 45 module import issues. Using --onefile helps to run the scrapy project in a single executable. Hope anyone finds it useful.
answered Mar 29 at 3:26
Arun AugustineArun Augustine
304110
answered Mar 29 at 3:26
Arun AugustineArun Augustine
304110
answered Mar 29 at 3:26
Arun AugustineArun Augustine
304110
answered Mar 29 at 3:26
Arun AugustineArun Augustine
304110
304110
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55331478%2fhow-to-create-a-single-executable-file-in-windows-10-with-scrapy-and-pyinstaller%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What is the error ?
– error404
Mar 25 at 5:28
ImportError: No module named 'scrapy.spiderloader', I am calling scrapy from a script and spider details are passing inside crawl process() function. Any idea about how to tell the pyinstaller to fetch my updated scrapy package from a location?
– Arun Augustine
Mar 25 at 5:43
have you tried pyinstaller's hiddenimport ? For example: --hidden-import=saidmodule.py
– RockAndRoleCoder
Mar 25 at 6:09
I have tried with hidden imports for package scrapy, from the command line as well as the .spec file. both show the same error.
– Arun Augustine
Mar 25 at 8:39