Unix file sharing

Just wondering if there is a faster way to split a file into N pieces other than unix "split".

Basically, I have large files that I would like to split into smaller pieces and work on each of them in parallel.

+3
source share
2 answers

I assume that you are using split -bone that will be more efficient in terms of CPU than line splitting, but still reads the entire input file and writes it to each file. If the sequential nature of the execution of this part splitis your bottleneck, you can use ddto extract fragments of the file in parallel. For each parallel process you will need a separate command dd. Here's one example command line (assuming it the_input_fileis a large file that extracts a bit from the middle):

dd skip=400 count=1 if=the_input_file bs=512 of=_output

, count bs (, , ). skip, . ; dd skip .

, , , . , , .

+1

, , , .

( man -k split man split), , split.

, C, / , .

: , , , , ( split)

0

All Articles