How to increase the performance of a file upload using native sftp functions and fwrite in PHP

Hi I am using the following code to upload a huge file (500MB) to a sftp server.

<?php

$connection = ssh2_connect($this->host, $this->port, null);
$sftp = ssh2_sftp($connection);

$connection_string = ((int) $sftp) . $remotePath . $remoteFilename;
$stream = fopen('ssh2.sftp://' . $connection_string, 'w');
$source = fopen($localFilepath, 'r');

if (!$stream) {
    throw new Exception('Could not create file: ' . $connection_string);
}

while (!feof($source)) {
    // Chunk size 32 MB
    if (fwrite($stream, fread($source, 33554432)) === false) {
        throw new Exception('Could not send data: ' . $connection_string);
    }
}

fclose($source);
fclose($stream);

But the upload is very slow. The code is running on Google Cloud Run. The upload speed is around 8 MiB/s.

I also tried to use lftp via shell_exec but this lead to even more issues due to Cloud Run.

The uplink can’t be the problem as I can send files via CURL post without any issues.

Anyone able to help here?

Many thanks and best, intxcc

Answer

The issue is that even though 32MB are read and then written to the sftp stream, fwrite will chunk at a different size. I think just a few KB.

For filesystems (which is the usual case with fwrite) this is fine, but not with high latency due to fwriting to a remote server.

So the solution is to increase the chunk size of the sftp stream with

stream_set_chunk_size($stream, 1024 * 1024);

So the final working code is:

<?php

$connection = ssh2_connect($this->host, $this->port, null);
$sftp = ssh2_sftp($connection);

$connection_string = ((int) $sftp) . $remotePath . $remoteFilename;
$stream = fopen('ssh2.sftp://' . $connection_string, 'w');
$source = fopen($localFilepath, 'r');

// Stream chunk size 1 MB
stream_set_chunk_size($stream, 1024 * 1024);

if (!$stream) {
    throw new Exception('Could not create file: ' . $connection_string);
}

while (!feof($source)) {
    // Chunk size 32 MB
    if (fwrite($stream, fread($source, 33554432)) === false) {
        throw new Exception('Could not send data: ' . $connection_string);
    }
}

fclose($source);
fclose($stream);

Hope this helps the next person that is getting gray hair trying to figure that out 😉