Geek asks and answers

A fast way to transfer a large number of small files to the network using PHP

I have 2 Linux servers on the same LAN.
Using PHP, I need to copy 100,000 small (10 KB) files from server A to server B.

Now I am using ssh2_scp_send and its very slow (10K files in 20 minutes).

How to make it faster?

+5
source share
4 answers

The use of gzipped TAR through the SSH tunnel is growing rapidly. Values ​​are faster than clean scp, especially for many small files. The following is an example linux command line:

user @local # cd / source /; tar czf - * | ssh user @remote "cd / target /; tar xzf -"

: , PHP- - .

. PHP libssh extension. , STDIN, -, SSH.

, , , , , , .

-z . ( , )

:

<?php
$local_cmd = "cd /tmp/source && tar -czf - *";
$remote_cmd = "tar -C /tmp/target -xzf -";

$ssh = new SSH_Connection('localhost');
$auth = $ssh->auth_password('gast', 'gast');
$bytes = $ssh->command_pipe($local_cmd, $remote_cmd);
echo "finished: $bytes bytes of data transfered\n";

class SSH_Connection {
    private $link;
    private $auth;

    function __construct ($host, $port=22) {
        $this->link = @ssh2_connect('localhost', 22);
    }  

    function auth_password ($username, $password) {
        if (!is_resource($this->link))
            return false;
        $this->auth = @ssh2_auth_password($this->link, $username, $password); 
        return $this->auth;
    }   

    function command_pipe ($localcmd, $remotecmd) {
        if (!is_resource($this->link) || !$this->auth)
            return false;
        // open remote command stream (STDIN)
        $remote_stream = fopen("ssh2.exec://{$this->link}/$remotecmd", 'rw');
        if (!is_resource($remote_stream))
            return false;
        // open local command stream (STDOUT)
        $local_stream = popen($localcmd, 'r');
        if (!is_resource($local_stream))
            return false;
        // connect both, pipe data from local to remote
        $bytes = 0;
        while (!feof($local_stream)) 
            $bytes += fwrite($remote_stream,fread($local_stream,8192));
        // close streams
        fclose($local_stream);
        fclose($remote_stream);
        return $bytes; 
    }   

    function is_connected () { return is_resource($this->link); }
    function is_authenticated () { return $this->auth; }
}
+4

PHP? , rsync -

0

Zip:

<?php
$zip = new ZipArchive();

if ($zip->open('TempFileToTransfer.zip', ZIPARCHIVE::CREATE)!==TRUE) {
    exit("cannot open <$filename>\n");
}

$zip->addFile('File 1');
$zip->addFile('File 2');
$zip->addFile('File 3');
$zip->close();
?>

zip_entry_open zip_entry_read .

1000 ?

0

The overhead of sending a large number of small files is probably a slowdown here.

You can archive it on a local server, shell_exec ()

send it to a remote server, ssh2_scp_send ()

then expand it at the other end. ssh2_exec ()

I feel that the overhead of archiving / extension will be less, but I have not tested this.

0
source

More articles:


All Articles