Geek asks and answers

Reliable non-blocking reads from the stdout subprocess

Note: I have a process that writes a single line to stdout ("print (" hello ")) and waits for raw_input. I start this process with p = subprocess.Popen, then call p.stdout.readline () .... it blocks indefinitely. I set shell = False ... Why can't I read this first output?

p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin = subprocess.PIPE)
    print(p.stdout.readline())

I saw thisthread about not blocking io for subprocess. I am trying to create an interactive subprocess object that reads and writes to a process. I found that stdout is blocking, although there is output sent to stdout. To test this, I wrote a program that spits integers into stdout and calls it through a subprocess. Then I used a non-blocking method that included a queue for reading stdout. I found that even though the process spills out 1000 lines in stdout per second, Stdout blocks for long periods of time, returning the line completely randomly ... one out of every thousand, then one out of 10, etc. thoughts on why this might happen? If I slow down the printer to print every 0.5 seconds, I never read stdout. She likes 1000 entries in sdtout before she answers in one line.

Note: I slept so that there was a time between the start of listening and reading. Now I always return from stdout, but I get a random single line. I guess I don’t understand what is happening. Is this not a buffer? I mean, I want to get everything that spits out my process, but it looks like it only saves the last line. I can not find the documentation for these PIPE things.

  def getStdOut(self):
        '''
            reads standard out
        '''
        return self.nonBlockingRead()


    def enqueue_output(self, out, queue, kill):
        line = ""
        kill = True
        while kill:
            line = out.readline()
            queue.put(line)

#             for line in iter(out.readline, b''):
#                 queue.put(line)
#             out.close()

    def nonBlockingRead(self):
        '''
             taken from the internet
        '''
        import sys
        from subprocess import PIPE, Popen
        from threading  import Thread
        sleep(0.5) #I inserted this later
        try:
            from Queue import Queue, Empty
        except ImportError:
            from queue import Queue, Empty  # python 3.x

        ON_POSIX = 'posix' in sys.builtin_module_names

        killThread = False                
        self.q = Queue()
        t = Thread(target=self.enqueue_output, args=(self.theProcess.stdout, self.q, killThread))
        t.daemon = True # thread dies with the program
        t.start()

        line = ''
        try:  
            line = self.q.get_nowait() # or q.get(timeout=.1)
        except Exception as e:
            print(e)
            print('no output yet')
        killThread = True
        return line




if __name__ == "__main__" :
    '''
        unit tests
    '''
    import time
    cmd = 'python "C:\InteractiveTest.py"'
    aSubProc = subProcessWStdIn(cmd, 10)
    while True :    
#        print(aSubProc.writeToProcess('y'))
        print(aSubProc.getStdOut())
+3
source share
1 answer

readlinereads one line from a file object PIPEto read it all, just wrap it in a while loop. You should also call sleepafter each read to save processor cycles.

Here is a simple example:

import subprocess

p = subprocess.Popen(
    ['ls', '-lat'],
    shell=False,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    stdin=subprocess.PIPE
)
while True:
    line = p.stdout.readline()
    if line == '':
        break
    print(line.strip())  # remove extra ws between lines

EDIT:

woah, sorry, I completely missed the part you tried to read in this other process ...

, fn , :

print('Hello')
in = raw_input()

PIPE, , . print() docs

, sys.stdout.flush() print raw_input:

print('Hello')
sys.stdout.flush()  # "flush" the output to our PIPE
in = raw_input()
+1

More articles:


All Articles