Styjun

Reference request: quantifier elimination test

First Entry Member State schengen visa

Publishing papers seem natural to many, while I find it really hard to think novel stuff to pursue till publication. How to cope up with this?

Can the word "desk" be used as a verb?

Writing an ace/aro character?

Is it better in terms of durability to remove card+battery or to connect to charger/computer via USB-C?

Found and corrected a mistake on someone's else paper -- praxis?

Optimization models for portfolio optimization

How do I explain that I don't want to maintain old projects?

Number of states in taxi environment (Dietterich 2000)

Why is the Cauchy Distribution is so useful?

Four ships at the ocean with the same distance

Conditions for Roots of a quadratic equation at infinity

What happens to unproductive professors?

This LM317 diagram doesn't make any sense to me

What factors could lead to bishops establishing monastic armies?

What is the problem here?(all integers are irrational proof...i think so)

Users forgetting to regenerate PDF before sending it

US citizen traveling with Peruvian passport

How do you move up one folder in Finder?

Why did Dumbledore ignore this line?

What was this character's plan?

Why do airports remove/realign runways?

Distinguish the explanations of Galadriel's test in LotR

Using multiprocessing.pool.Pool to initialize parallel processing freezes indefinitely

What is the difference between concurrency and parallelism?Parallel in DelphiCelery parallel distributed task with multiprocessingSystem time for parallel and serial processingPython - Parallelism not workingHow to run different methods in parallel and obtain the outputs for further processing?How, in python, to split up lage result of itertools.product into groups and iterate in parallelDifferent responses from Spyder's Ipython and standard anaconda consolePrints of multiprocessing workers don't show in Spyder on WindowsCode works line-by-line in Spyder, but not when the whole script is run

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

1

I am currently trying to run parallelized code from a spyder console in anaconda. I believe the issue may be with my computer not allowing anaconda to control CPU cores, but I am not sure how to correct this issue.

Another interesting point is that when I run an async example, but when I try to produce the results I receive the same issue.

I have tried multiple simple examples that should be working. There are no package loading errors

from multiprocessing.pool import ThreadPool, Pool

def square_it(x):
 return x*x

# On Windows, make sure that multiprocessing doesn't start
# until after "if __name__ == '__main__'" 

with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

print(results) 

I expect for my code to complete all code.

python parallel-processing anaconda spyder

share|improve this question

edited Mar 25 at 23:25

zvone

10.5k11 gold badge2525 silver badges4848 bronze badges

asked Mar 25 at 20:45

Brendan NewellBrendan Newell

2788 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  What is the error you are getting?
  
  – zvone
  Mar 25 at 21:06
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I am not getting an error, my console shows it is just running that line of code and will not stop. The code should take less than 1 second to run, with one core, so there is not an issue.
  
  – Brendan Newell
  Mar 25 at 21:56
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Are you on Windows?
  
  – juanpa.arrivillaga
  Mar 25 at 23:30
  

  
  
  
  

add a comment | 

1

I am currently trying to run parallelized code from a spyder console in anaconda. I believe the issue may be with my computer not allowing anaconda to control CPU cores, but I am not sure how to correct this issue.

Another interesting point is that when I run an async example, but when I try to produce the results I receive the same issue.

I have tried multiple simple examples that should be working. There are no package loading errors

from multiprocessing.pool import ThreadPool, Pool

def square_it(x):
 return x*x

# On Windows, make sure that multiprocessing doesn't start
# until after "if __name__ == '__main__'" 

with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

print(results) 

I expect for my code to complete all code.

python parallel-processing anaconda spyder

share|improve this question

edited Mar 25 at 23:25

zvone

10.5k11 gold badge2525 silver badges4848 bronze badges

asked Mar 25 at 20:45

Brendan NewellBrendan Newell

2788 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  What is the error you are getting?
  
  – zvone
  Mar 25 at 21:06
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I am not getting an error, my console shows it is just running that line of code and will not stop. The code should take less than 1 second to run, with one core, so there is not an issue.
  
  – Brendan Newell
  Mar 25 at 21:56
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Are you on Windows?
  
  – juanpa.arrivillaga
  Mar 25 at 23:30
  

  
  
  
  

add a comment | 

I am currently trying to run parallelized code from a spyder console in anaconda. I believe the issue may be with my computer not allowing anaconda to control CPU cores, but I am not sure how to correct this issue.

Another interesting point is that when I run an async example, but when I try to produce the results I receive the same issue.

I have tried multiple simple examples that should be working. There are no package loading errors

from multiprocessing.pool import ThreadPool, Pool

def square_it(x):
 return x*x

# On Windows, make sure that multiprocessing doesn't start
# until after "if __name__ == '__main__'" 

with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

print(results) 

I expect for my code to complete all code.

python parallel-processing anaconda spyder

share|improve this question

edited Mar 25 at 23:25

zvone

10.5k11 gold badge2525 silver badges4848 bronze badges

asked Mar 25 at 20:45

Brendan NewellBrendan Newell

2788 bronze badges

from multiprocessing.pool import ThreadPool, Pool

def square_it(x):
 return x*x

# On Windows, make sure that multiprocessing doesn't start
# until after "if __name__ == '__main__'" 

with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

print(results) 

share|improve this question

edited Mar 25 at 23:25

zvone

10.5k11 gold badge2525 silver badges4848 bronze badges

asked Mar 25 at 20:45

Brendan NewellBrendan Newell

2788 bronze badges

share|improve this question

edited Mar 25 at 23:25

zvone

10.5k11 gold badge2525 silver badges4848 bronze badges

asked Mar 25 at 20:45

Brendan NewellBrendan Newell

2788 bronze badges

edited Mar 25 at 23:25

zvone

10.5k11 gold badge2525 silver badges4848 bronze badges

edited Mar 25 at 23:25

zvone

10.5k11 gold badge2525 silver badges4848 bronze badges

asked Mar 25 at 20:45

Brendan NewellBrendan Newell

2788 bronze badges

asked Mar 25 at 20:45

Brendan NewellBrendan Newell

2788 bronze badges

1 Answer
1

active

oldest

votes

2

This code is meant to run square_it in parallel, in 5 different processes

def square_it(x):
 return x*x

with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

print(results)

The it does that is that 5 new processes are created, then in each of them, the same python module is loaded and function square_it is called.

What happens when the module is imported in one of the 5 subprocesses is the same thing which happens when it is initially loaded in the main process: it creates another Pool of 5 subprocesses, which do that indefinitely.

To avoid that, you have to make sure that the subprocesses do not recursively create more and more subprocesses. You do that by creating the subprocesses only in the "main" module, aka "__main__":

def square_it(x):
 return x*x

if __name__ == "__main__":
 with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

 print(results)

share|improve this answer

answered Mar 25 at 23:21

zvonezvone

10.5k11 gold badge2525 silver badges4848 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This worked perfectly thanks!
  
  – Brendan Newell
  Mar 26 at 12:03
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55346144%2fusing-multiprocessing-pool-pool-to-initialize-parallel-processing-freezes-indefi%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

2

This code is meant to run square_it in parallel, in 5 different processes

def square_it(x):
 return x*x

with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

print(results)

The it does that is that 5 new processes are created, then in each of them, the same python module is loaded and function square_it is called.

What happens when the module is imported in one of the 5 subprocesses is the same thing which happens when it is initially loaded in the main process: it creates another Pool of 5 subprocesses, which do that indefinitely.

To avoid that, you have to make sure that the subprocesses do not recursively create more and more subprocesses. You do that by creating the subprocesses only in the "main" module, aka "__main__":

def square_it(x):
 return x*x

if __name__ == "__main__":
 with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

 print(results)

share|improve this answer

answered Mar 25 at 23:21

zvonezvone

10.5k11 gold badge2525 silver badges4848 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This worked perfectly thanks!
  
  – Brendan Newell
  Mar 26 at 12:03
  

  
  
  
  

add a comment | 

2

This code is meant to run square_it in parallel, in 5 different processes

def square_it(x):
 return x*x

with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

print(results)

The it does that is that 5 new processes are created, then in each of them, the same python module is loaded and function square_it is called.

What happens when the module is imported in one of the 5 subprocesses is the same thing which happens when it is initially loaded in the main process: it creates another Pool of 5 subprocesses, which do that indefinitely.

To avoid that, you have to make sure that the subprocesses do not recursively create more and more subprocesses. You do that by creating the subprocesses only in the "main" module, aka "__main__":

def square_it(x):
 return x*x

if __name__ == "__main__":
 with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

 print(results)

share|improve this answer

answered Mar 25 at 23:21

zvonezvone

10.5k11 gold badge2525 silver badges4848 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This worked perfectly thanks!
  
  – Brendan Newell
  Mar 26 at 12:03
  

  
  
  
  

add a comment | 

This code is meant to run square_it in parallel, in 5 different processes

def square_it(x):
 return x*x

with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

print(results)

The it does that is that 5 new processes are created, then in each of them, the same python module is loaded and function square_it is called.

What happens when the module is imported in one of the 5 subprocesses is the same thing which happens when it is initially loaded in the main process: it creates another Pool of 5 subprocesses, which do that indefinitely.

To avoid that, you have to make sure that the subprocesses do not recursively create more and more subprocesses. You do that by creating the subprocesses only in the "main" module, aka "__main__":

def square_it(x):
 return x*x

if __name__ == "__main__":
 with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

 print(results)

share|improve this answer

answered Mar 25 at 23:21

zvonezvone

10.5k11 gold badge2525 silver badges4848 bronze badges

def square_it(x):
 return x*x

with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

print(results)

def square_it(x):
 return x*x

if __name__ == "__main__":
 with Pool(processes=5) as pool:
 results = pool.map(square_it, [5, 4, 3, 2 ,1])

 print(results)

share|improve this answer

answered Mar 25 at 23:21

zvonezvone

10.5k11 gold badge2525 silver badges4848 bronze badges

answered Mar 25 at 23:21

zvonezvone

10.5k11 gold badge2525 silver badges4848 bronze badges

answered Mar 25 at 23:21

zvonezvone

10.5k11 gold badge2525 silver badges4848 bronze badges