Styjun

I am using aria2 to download some data with the option --on-download-complete to run a bash script automatically to process the data.

aria2c --http-user='***' --http-passwd='***' --check-certificate=false --max-concurrent-downloads=2 -M products.meta4 --on-download-complete=/my/path/script_gpt.sh

Focusing on my bash script,

#!/bin/bash

oldEnd=.zip
newEnd=_processed.dim

for i in $(ls -d -1 /my/path/S1*.zip)
do
if [ -f $i%$oldEnd$newEnd ]; then 
 echo "Already processed"
else
 gpt /my/path/graph.xml -Pinput1=$i -Poutput1=$i%$oldEnd$newEnd
fi
done 

Basically, everytime a download is finished, a for loop starts. First it checks if the downloaded product has been already processed and if not it runs a specific task.

My issue is that everytime a download is completed, the bash script is run. This means that if the analysis is not finished from the previous time the bash script was run, both tasks will overlap and eat all my memory resources.

Ideally, I would like to:

• Each time the bash script is run, check if there is still and ongoing process.




• If so, wait until it is finished and then run

Its like creating a queu of task (like in a for loop where each iteration waits until the previous one is finished).

I have tried to implement the solutin with wait or identifying the PID but nothing succesfull.

Maybe changing the approach and instead of using aria2 to process the data that is just donwloaded, implemente another solution?

You can try to acquire an exclusive file lock and only run when it lock is released. Your code could be like

#!/bin/bash

oldEnd=.zip
newEnd=_processed.dim


 flock -e 200

 while IFS= read -r -d'' i
 do
 if [ -f "$i%$oldEnd$newEnd" ];
 then 
 echo "Already processed"
 else
 gpt /my/path/graph.xml -Pinput1="$i" -Poutput1="$i%$oldEnd$newEnd"
 fi
 done < <(find /my/path -maxdepth 1 -name "S1*.zip" -print0)
 200> /tmp/aria.lock

This code opens an exclusive lock against file descriptor 200 (the one we told bash to open to redirect output to the lock file, and prevents other scripts to execute the code block until the file is closed. The file is closed as soon as the code block is finished, allowing other waiting processes to continue the execution.

BTW, you should always quote your variables and you should avoid parsing the ls output. Also, to avoid problems with whitespaces and unexpected globbing, outputting the file list separated by zeros and reading it with read is a way to avoid those problems.