Styjun

A continuous water "planet" ring around a star

Heating Margarine in Pan = loss of calories?

TEMPO: play a sound in animated GIF/PDF/SVG

Why is there a large performance impact when looping over an array over 240 elements?

80's/90's superhero cartoon with a man on fire and a man who made ice runways like Frozone

How can this older-style irrigation tee be replaced?

How to assign many blockers at the same time?

Why aren't rainbows blurred-out into nothing after they are produced?

Lengthened voiced stops and the airstream through the nose

How exactly are corporate bonds priced at issue

The cat ate your input again!

Is there a Morita cocycle for the mapping class group Mod(g,n) when n > 1?

Is there a standardised way to check fake news?

Safest way to store environment variable value in a file

What is this "Table of astronomy" about?

Super Duper Vdd stiffening required on 555 timer, what is the best way?

A Non Math Puzzle. What is the middle number?

Understanding the point of a kölsche Witz

Bitcoin successfully deducted on sender wallet but did not reach receiver wallet

How are you supposed to know the strumming pattern for a song from the "chord sheet music"?

Does Molecular Weight of a Gas affect its lifting properties at the same velocity over the same wing?

Is there a SQL/english like language that lets you define formulations given some data?

What are these funnel-looking green things in my yard?

Are employers legally allowed to pay employees in goods and services equal to or greater than the minimum wage?

Parallelizing a matrix-blockvector product using python's module multiprocessing?

What is the difference between concurrency and parallelism?Comparison of the multiprocessing module and pyro?Multiprocessing vs Threading PythonUsing the multiprocessing module for cluster computingPython multiprocessing pool.map for multiple argumentsParallel construction of a distance matrixComputing eigenvalues in parallel for a large matrixadapt multiprocessing Pool to mpi4pypython, multiprocessing and dmtcp: checkpointing one process in Pool?Easy way to use parallel options of scikit-learn functions on HPC

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

Here is my problem :

I have a code that makes the multiplication between a very very big matrix and several vectors at each iteration (kind of a block vector that can be think as a tall and skinny matrix). I'm using python mulprocessing module to parallelize this product and I'm launching those jobs on a cluster. The problem is that the cluster-hours bought depends on the memory your jobs are launching. And as such, the data of the problem, essentially the big matrix, is send to the memory of all the processors used. Therefore multiplying the memory used by the number of processors and costing a tremendous amount of cluster-hours. I'd like to know, if there is a way that the memory used is stored only in one place and used over several processors with this python's module ? Basically I'd like to do distributed memory on a cluster with python with a simple module as multiprocessing..

I'm launching those kind of jobs on several cluster, depending on the queue at the time, but I'm mostly interested on getting this done on Cori, the nersc super-computer.

I couldn't find anything matching this problem in particular certainly because one easy answer is : Do parallel well and use MPI you b**** ! But the code is in python, and I'm not really looking for optimal performance at the moment, just reduced computational time, keeping the code simple and easly changeable.

Thank you for any of your answer !

python-3.x parallel-processing multiprocessing shared-memory

share|improve this question

asked Mar 27 at 9:21

Thibault CimicThibault Cimic

111 bronze badge

add a comment | 

0

Here is my problem :

I have a code that makes the multiplication between a very very big matrix and several vectors at each iteration (kind of a block vector that can be think as a tall and skinny matrix). I'm using python mulprocessing module to parallelize this product and I'm launching those jobs on a cluster. The problem is that the cluster-hours bought depends on the memory your jobs are launching. And as such, the data of the problem, essentially the big matrix, is send to the memory of all the processors used. Therefore multiplying the memory used by the number of processors and costing a tremendous amount of cluster-hours. I'd like to know, if there is a way that the memory used is stored only in one place and used over several processors with this python's module ? Basically I'd like to do distributed memory on a cluster with python with a simple module as multiprocessing..

I'm launching those kind of jobs on several cluster, depending on the queue at the time, but I'm mostly interested on getting this done on Cori, the nersc super-computer.

I couldn't find anything matching this problem in particular certainly because one easy answer is : Do parallel well and use MPI you b**** ! But the code is in python, and I'm not really looking for optimal performance at the moment, just reduced computational time, keeping the code simple and easly changeable.

Thank you for any of your answer !

python-3.x parallel-processing multiprocessing shared-memory

share|improve this question

asked Mar 27 at 9:21

Thibault CimicThibault Cimic

111 bronze badge

add a comment | 

Here is my problem :

I have a code that makes the multiplication between a very very big matrix and several vectors at each iteration (kind of a block vector that can be think as a tall and skinny matrix). I'm using python mulprocessing module to parallelize this product and I'm launching those jobs on a cluster. The problem is that the cluster-hours bought depends on the memory your jobs are launching. And as such, the data of the problem, essentially the big matrix, is send to the memory of all the processors used. Therefore multiplying the memory used by the number of processors and costing a tremendous amount of cluster-hours. I'd like to know, if there is a way that the memory used is stored only in one place and used over several processors with this python's module ? Basically I'd like to do distributed memory on a cluster with python with a simple module as multiprocessing..

I'm launching those kind of jobs on several cluster, depending on the queue at the time, but I'm mostly interested on getting this done on Cori, the nersc super-computer.

I couldn't find anything matching this problem in particular certainly because one easy answer is : Do parallel well and use MPI you b**** ! But the code is in python, and I'm not really looking for optimal performance at the moment, just reduced computational time, keeping the code simple and easly changeable.

Thank you for any of your answer !

python-3.x parallel-processing multiprocessing shared-memory

share|improve this question

asked Mar 27 at 9:21

Thibault CimicThibault Cimic

111 bronze badge

share|improve this question

asked Mar 27 at 9:21

Thibault CimicThibault Cimic

111 bronze badge

share|improve this question

asked Mar 27 at 9:21

Thibault CimicThibault Cimic

111 bronze badge

asked Mar 27 at 9:21

Thibault CimicThibault Cimic

111 bronze badge

asked Mar 27 at 9:21

Thibault CimicThibault Cimic

111 bronze badge