Styjun

I need to run multiple python files simultaneously, but when one finishes, kill all of the rest. I only need to know the output of the python file that finishes first. After much trial and error, I have decided to use a bash file (I am running Ubuntu) to start all of the files simultaneously, however, I cannot stop all of them once the first one is done.Please help, thanks!!! I have tried many answers I have found online with no success. I have even written a python file to kill other python files which works when started from the terminal but not when called from the bash file or another python file. I feel like I am over complicating the problem. Any Help is greatly appreciated.

Assuming I take in a variable named pass and I pass that to each file here are the commands I want to run in parallel:

 read pass
 python3 1.py "$pass"
 python3 2.py "$pass"
 python3 3.py "$pass"
 python3 4.py "$pass"
 python3 5.py "$pass"
 python3 6.py "$pass"
 python3 7.py "$pass"
 python3 8.py "$pass"
 python3 9.py "$pass"

Here is a little example of how you can start multible python scripts in parallel. After the first script print the output result, all other where be killed.

python script

import sys
import time
seconds = sys.argv[1]
time.sleep(int(seconds))
print("runner sleep seconds.".format(seconds))

bash script

start() # function for starting python scripts
 echo "start $@" # print which runner is started
 "$@" & # start python script in background
 pids+=("$!") # save pids in array


start ./runner.py 5 # start python scripts
start ./runner.py 7
start ./runner.py 1
start ./runner.py 2
start ./runner.py 3
start ./runner.py 9
start ./runner.py 8
start ./runner.py 3
start ./runner.py 4
start ./runner.py 7

wait -n "$pids[@]" # wait for first finished process
kill "$pids[@]" 2>/dev/null # kill all/other process

output

start ./runner.py 5
start ./runner.py 7
start ./runner.py 1
start ./runner.py 2
start ./runner.py 3
start ./runner.py 9
start ./runner.py 8
start ./runner.py 3
start ./runner.py 4
start ./runner.py 7
runner sleep 1 seconds.

Updated Answer

Thank you for your inputs. In the light of those, I would suggest you use:

parallel --dry-run -k --halt now,success=1 python3 .py "$pass" ::: 1..9

Sample Output

python3 1.py "$pass"
python3 2.py "$pass"
python3 3.py "$pass"
python3 4.py "$pass"
python3 5.py "$pass"
python3 6.py "$pass"
python3 7.py "$pass"
python3 8.py "$pass"
python3 9.py "$pass"

If that looks correct, you can remove --dry-run and -k and simply run:

parallel --halt now,success=1 python3 .py "$pass" ::: 1..9

Original Answer

I think you can leverage GNU Parallel for this, specifically its halt policy. So, as an example, start 10 jobs, that sleep 10, 20, 30, 40... 100 seconds each and kill the others as soon as any one succeeds:

parallel --halt now,success=1 sleep ::: $(seq 10 10 100)
parallel: This job succeeded:
sleep 10

If you add -k and --dry-run after parallel, you will see that it starts the following jobs:

parallel -k --dry-run --halt now,success=1 sleep ::: $(seq 10 10 100)

sleep 10
sleep 20
sleep 30
sleep 40
sleep 50
sleep 60
sleep 70
sleep 80
sleep 90
sleep 100

Another example which kills outstanding jobs once 20% of the jobs have succeeded:

parallel --halt now,success=20% sleep ::: $(seq 10 10 100)
parallel: This job succeeded:
sleep 10
parallel: This job succeeded:
sleep 20