Geek asks and answers

Bash: running the same program across multiple cores

I have access to a machine on which I have access to 10 cores, and I would like to use them. What I used to do on my machine would be something like this:

for f in *.fa; do
  myProgram (options) "./$f" "./$f.tmp"
done

I have 10 files that I would like to do this - call them blah00.fa, blah01.fa, ... blah09.fa.

The problem with this approach is that myProgram uses only 1 core at a time, and by doing so on a multi-core computer, I will use 1 core at a time 10 times, so I would not use my mahcine to the maximum possible .

How can I change my script so that it runs all 10 of my .fa files at once? I looked at Running a loop in bash through several kernels , but I could not get a command from this to do what I wanted for sure.

+5
source share
4 answers

you can use

for f in *.fa; do
    myProgram (options) "./$f" "./$f.tmp" &
done
wait

which will run all your tasks in parallel, then wait until they all end before moving on. In the event that you have more jobs than cores, you should run all of them and let the OS scheduler worry about exchanging processes in out.

One of the modifications is the launch of 10 tasks at a time

count=0
for f in *.fa; do
    myProgram (options) "./$f" "./$f.tmp" &
    (( count ++ ))        
    if (( count = 10 )); then
        wait
        count=0
    fi
done

, parallel, , , , 10 . wait , , - .

+8

GNU Parallel :

parallel myProgram (options) {} {.}.tmp ::: *.fa

: http://git.savannah.gnu.org/cgit/parallel.git/tree/README

= =

GNU Parallel :

./configure && make && make install

root, ~/bin ~/bin ~/share:

./configure --prefix=$HOME && make && make install

, "make", src/parallel src/sem src/niceload src/sql .

= =

"make" (, Microsoft Windows):

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem
mv parallel sem dir-in-your-$PATH/bin/

, : https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

+4

Do you consider using Parallel ?

+3
source 
# Wait while instance count less than $3, run additional instance and exit
function runParallel () {
    cmd=$1
    args=$2
    number=$3
    currNumber="1024"
    while true ; do
        currNumber=`ps -e | grep -v "grep" | grep " $1$" | wc -l`
        if [ $currNumber -lt $number ] ; then
            break
        fi
        sleep 1
    done
    echo "run: $cmd $args"
    $cmd $args &
}

loop=0
# We will run 12 sleep commands for 10 seconds each 
# and only five of them will work simultaneously
while [ $loop -ne 12 ] ; do
    runParallel "sleep" 10 5
    loop=`expr $loop + 1`
done
0
source

More articles:


All Articles