Geek asks and answers

Running jobs in a cluster submitted via qsub from Python. Does this make sense?

I have a situation when I do calculations in Python, and based on the results, I have a list of target files that are candidates that will be transferred to the second program.

For example, I have 50,000 files that contain ~ 2000 elements each. I want to filter certain elements and invoke a command line program to do some calculations for some of them.

This program # 2 can be used through the shell command line, but also requires a long set of arguments. Due to performance considerations, I would have to run program # 2 in the cluster.

Now I run program number 2 through    subprocess.call("...", shell=True) But I would like to launch it through qsub in the future. I have little experience in how exactly this could be done reasonably enough.

Does it make sense to write temporary “qsub” files and run them through subprocess () directly from a Python script? Is there a better, perhaps more pythonic solution?

Any ideas and suggestions are very welcome!

+3
source share
2 answers

That makes sense, although I would go for another solution.

As I understand it, you have program number 1, which determines which of your 50,000 files should be calculated by program number 2. Both programs # 1 and # 2 are written in Python. Great choice.

, Python, : https://gist.github.com/stefanedwards/8841307

qsub-, ( , ), . -v, , :

[me@local ~] $ python isprime.py 1
1: True 
[me@local ~] $ head -n 5 isprime.py
#!/usr/bin/python
### This is a python script ...
import os
os.chdir(os.environ.get('PBS_O_WORKDIR','.'))

[me@local ~] $ qsub -v isprime='1 2 3' isprime.py
123456.cluster.control.com
[me@local ~]

isprime.py , argparse. , script , (os.environ).

№2 , №1 subprocess.call(['qsub','-v options=...','programme2.py'], shell=FALSE)

(, SQLite). № 1, , (, , ). №2 , .

: №2

python script bash script, :

#!/bin/bash
cd .
# put options into context/flags etc.
if [ -n $option1 ]; then _opt1="--opt1 $option1"; fi
# we can even define our own defaults
_opt2='--no-verbose'
if [ -n $opt2 ]; then _opt2="-o $opt2"; fi
/path/to/exe $_opt1 $opt2

, python script, , ( ), , subprocess, , , ..

+2

, , cmd, , 2- . subprocess.call(cmd, shell=True) 2- Python script ( , script).

, , , , . , , Python, "" , :-): cmd , .

, qsub ( SGE LSF - , ) , . , qsub ...args... cmd, cmd cmd. , qsub, qsubcmd ( , ). , Python script,

subprocess.call(qsubcmd, shell=True)

subprocess.call(cmd, shell=True)

, qsub , node (s) . , Python script, , ( , ssh , ).

, , subprocess subprocess. shell=True, . .

+2

More articles:


All Articles