Styjun

Closed Loop System

Two people from small group of friends want to have a "meeting" with me. The circumstances are strange and give me a bad feeling

Did Ohio pass a law granting students the right to give scientifically wrong answers consistent with their religious beliefs?

How can I find out where to buy uncommon (for the location) items while traveling?

Can a verb be modified too many times?

"Indexed" version of compactness and Axiom of Choice

Locked folder with obscure app from Sourceforge, now cannot unlock folder

SSD not reaching advertised speed

Hot Rim Looking for Suggestions

Prevent function taking const std::string& from accepting 0

SSH host identification changes on one wireless network

Are conjugate vectors unique?

Beyond breadboards? Good ways to package simple GPIO-using project?

Hand propping Cessna 172N, with a pilot and a non intuned passenger

Is there a preferred time in their presidency when US presidents pardon the most people?

What is :>filename.txt Doing?

How to handle a colleague who appears helpful in front of manager but doesn't help in private?

What does "parallel in-register" mean for a protein structure in a fibril?

Is Earth's Surface "In orbit"?

N-Dimensional Cartesian Product

Ethics: Is it ethical for a professor to conduct research using a student's ideas without giving them credit?

Why does this process map every fraction to the golden ratio?

What's the best way for guitar and piano to play together in a jazz quartet?

How do you get the Super Rod in Pokémon Sword and Shield?

How to call multiprocessing code in Python instance method without deadlocking

Python multiprocessing: object identifier unique across processesPython multiprocessing queues slower than pool.mapMultiprocess tasks in python that can't be pickled?Multiprocess in python uses only one processHow to maintain class state when executing class methods for multiple classes in parallel?Calling a normal function after multiprocessingpython multiprocessing .join() deadlock depends on worker functionPython Multiprocessing Pool Worker Processes Failing to TerminatePython multiprocessing and too many open filesPython Multiprocessing Not Speeding Up

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;

0

0

I want to use the multiprocessing library in Python to parallelize reading files and storing loaded information to a list. I also want this method for loading files in parallel to be an instance method that can also be called by other instance methods for some class.

When I call the parallelized loading function (ie. load_multiple_files) the first time, the data is loaded in parallel and returned as expected. However, the second time the function is called, the processes deadlock.

I tried reordering the two calls (i.e. results_B before results_A), and the processes deadlock on the second call. I also tried removing pool.join(), but that was unsuccessful.

Here is some current pseudocode:

class Foo():
 def __init__(self, filepaths_A: List[str], filepaths_B: List[str]):
 results_A = load_multiple_files(filepaths_A) # works as expected, with files loaded in parallel
 results_B = load_multiple_files(filepaths_B) # processes deadlock and program hangs

 def load_file(self, filepath: str):
 # load file and return a numpy array
 with File(filepath, 'r') as f:
 result = l['data'][:]
 return result

 def load_multiple_files(self, filepaths: List[str]):
 """ Wrapper for loading multiple files in parallel
 pool = mp.Pool()
 results = pool.map(self.load_file, filepaths)
 pool.close()
 pool.join()

 return results

I expect that regardless of the number of times the load_multiple_files method is called, the loading process will be parallelized and will return the loaded data.

Any potential solutions to this problem using multiprocessing.Pool?

python-multiprocessing

share|improve this question

asked Mar 28 at 22:08

aadaad

3866 bronze badges

add a comment
 | 

0

0

I want to use the multiprocessing library in Python to parallelize reading files and storing loaded information to a list. I also want this method for loading files in parallel to be an instance method that can also be called by other instance methods for some class.

When I call the parallelized loading function (ie. load_multiple_files) the first time, the data is loaded in parallel and returned as expected. However, the second time the function is called, the processes deadlock.

I tried reordering the two calls (i.e. results_B before results_A), and the processes deadlock on the second call. I also tried removing pool.join(), but that was unsuccessful.

Here is some current pseudocode:

class Foo():
 def __init__(self, filepaths_A: List[str], filepaths_B: List[str]):
 results_A = load_multiple_files(filepaths_A) # works as expected, with files loaded in parallel
 results_B = load_multiple_files(filepaths_B) # processes deadlock and program hangs

 def load_file(self, filepath: str):
 # load file and return a numpy array
 with File(filepath, 'r') as f:
 result = l['data'][:]
 return result

 def load_multiple_files(self, filepaths: List[str]):
 """ Wrapper for loading multiple files in parallel
 pool = mp.Pool()
 results = pool.map(self.load_file, filepaths)
 pool.close()
 pool.join()

 return results

I expect that regardless of the number of times the load_multiple_files method is called, the loading process will be parallelized and will return the loaded data.

Any potential solutions to this problem using multiprocessing.Pool?

python-multiprocessing

share|improve this question

asked Mar 28 at 22:08

aadaad

3866 bronze badges

add a comment
 | 

I want to use the multiprocessing library in Python to parallelize reading files and storing loaded information to a list. I also want this method for loading files in parallel to be an instance method that can also be called by other instance methods for some class.

When I call the parallelized loading function (ie. load_multiple_files) the first time, the data is loaded in parallel and returned as expected. However, the second time the function is called, the processes deadlock.

I tried reordering the two calls (i.e. results_B before results_A), and the processes deadlock on the second call. I also tried removing pool.join(), but that was unsuccessful.

Here is some current pseudocode:

class Foo():
 def __init__(self, filepaths_A: List[str], filepaths_B: List[str]):
 results_A = load_multiple_files(filepaths_A) # works as expected, with files loaded in parallel
 results_B = load_multiple_files(filepaths_B) # processes deadlock and program hangs

 def load_file(self, filepath: str):
 # load file and return a numpy array
 with File(filepath, 'r') as f:
 result = l['data'][:]
 return result

 def load_multiple_files(self, filepaths: List[str]):
 """ Wrapper for loading multiple files in parallel
 pool = mp.Pool()
 results = pool.map(self.load_file, filepaths)
 pool.close()
 pool.join()

 return results

I expect that regardless of the number of times the load_multiple_files method is called, the loading process will be parallelized and will return the loaded data.

Any potential solutions to this problem using multiprocessing.Pool?

python-multiprocessing

share|improve this question

asked Mar 28 at 22:08

aadaad

3866 bronze badges

class Foo():
 def __init__(self, filepaths_A: List[str], filepaths_B: List[str]):
 results_A = load_multiple_files(filepaths_A) # works as expected, with files loaded in parallel
 results_B = load_multiple_files(filepaths_B) # processes deadlock and program hangs

 def load_file(self, filepath: str):
 # load file and return a numpy array
 with File(filepath, 'r') as f:
 result = l['data'][:]
 return result

 def load_multiple_files(self, filepaths: List[str]):
 """ Wrapper for loading multiple files in parallel
 pool = mp.Pool()
 results = pool.map(self.load_file, filepaths)
 pool.close()
 pool.join()

 return results

share|improve this question

asked Mar 28 at 22:08

aadaad

3866 bronze badges

share|improve this question

asked Mar 28 at 22:08

aadaad

3866 bronze badges

asked Mar 28 at 22:08

aadaad

3866 bronze badges

asked Mar 28 at 22:08

aadaad

3866 bronze badges