Geek asks and answers

How to write a python program that will process a text stream?

I apologize if this is a repeated question. How to write a python script to process data as a string stream? I need to do this because the files I process are huge, and I prefer not to read the file in memory.

I know that you can read one line of the file at a time, but I want something that will handle the text stream.

+3
source share
3 answers

You can simply read the data from stdinas described in this. It will look like this:

for line in sys.stdin:
    # do suff

If you want to process the file, just call the script as follows (on Unix platforms):

cat file.txt | python script.py

You can, of course, also connect the output of any other program.

+14

, . :

python script.py file1.txt file2.txt file3.txt file4.txt

script.py

import fileinput
for line in fileinput.input():
    # do stuff here

fileinput - , . Space_C0wb0y :

python script.py - < file.txt


cat file.txt | python script.py -

fileinput , Space_C0wb0y, , , .

+7
f = open('somefile.txt')
for line in f:
    process(line)

In fact, it fcan be anything that is iterable, for example, a list of lines or even sys.stdinif you want to read from standard input.

0
source

More articles:


All Articles