Geek asks and answers

How to pre-process an endless stream of text before writing it to a file?

I use tcpflow to register network traffic on the server. I want to write this data to a file, but not all. The monitoring process will run in the background as a daemon endlessly.

Some lines of the stream include the number of bytes, and if I see that the number of bytes (say 800 bytes), then I need to write the next 800 bytes to a file. If not, I do not want to write to the file.

What is the best way for me to do such “preprocessing” on a thread to decide what to redirect to a log file? Is there some second daemon script that listens on the thread that connects to this script?

Example:

I see the following line in the stream:

1343932842: 010.079.091.189.35856-010.104.001.199.11211: set i:1:20163484235 0 0 1429

First, I need to check that it has a “set”. Then I look at the last fragment of the line (1429), then read the next 1429 bytes and write them to a file.

+5
source share
3 answers

Yes, use a daemon program that takes a stream as input and does only what you described. I would recommend C instead of a script, as it has very simple I / O and very low overhead.

Assuming you have an executable file called “capture” and a filter program called “filter”, you can link them together with the bash shell using

bash-prompt$ capture capture-params | filter

, capture stdout, filter stdin. , , ..., ( ). stdout, ,

bash-prompt$ capture capture-params | filter > output-file.txt
+1

awk. , . tail -f file.log | awk -f myscript.awk

awk script, , if-then-else , , awk- - .

0

To date, the most elegant application for describing what you are describing is to use a database with limited coverage. RRDtool is an industry standard OpenSource, high-performance data logging and graphics.

Using the bash command, you can enter your data into the database, and if you want, graphical display is also very simple.

SEE: http://oss.oetiker.ch/rrdtool/gallery/index.en.html

0
source

More articles:


All Articles