Stream Filter memory usage

PHP v7.4.16

I have a fairly basic stream filter (which extends php_user_filter), which I’m using to normalise CSV files as they’re transferred to another destination (s3 bucket using the stream wrapper in this case).

Contents of filter() function:

 while ($bucket = stream_bucket_make_writeable($in)) {

            // Replace any CRLF Windows line-endings with *nix n
            $bucket->data = preg_replace('~(*BSR_ANYCRLF)R~', "n", $bucket->data);

            $encoding = mb_detect_encoding($bucket->data, mb_detect_order(), true);
            $bucket->data = strtolower($encoding) != "utf-8"
                ? iconv($encoding, "utf-8//IGNORE", $bucket->data)
                : $bucket->data;

            $consumed += $bucket->datalen;

            // Send bucket to downstream brigade
            stream_bucket_append($out, $bucket);
        }

        // Return the code that indicates that the userspace filter returned buckets in $out
        return PSFS_PASS_ON;

I can successfully use the filter like this:

 private function streamFileToS3($file_entity) {
        $source = fopen($file_entity->local_path, 'r');
        $dest = fopen($file_entity->s3_path, 'w');
        @stream_filter_prepend($dest, 'csv', STREAM_FILTER_WRITE);
        $chunk_size = (10 * 1024 * 1024);
        while (!feof($source)) {
          
            $chunk = fread($source, $chunk_size);
            fwrite($dest, $chunk);
        }
        fclose($dest);
        fclose($source);
        return true;
    }

Now here’s the important bit.

When appending the filter the to the file handle I’m writing to, the script uses the memory I’m expecting.

 @stream_filter_prepend($dest, 'csv', STREAM_FILTER_WRITE);

But when I try to use the filter on the file handle I’m reading from:

 @stream_filter_prepend($source, 'csv', STREAM_FILTER_READ);

The output of memory_get_usage() increments by the size of the fread() chunk each iteration.

Does anybody know why this is, and if there’s any way around it?

Thanks in advance

Answer

After some more time spent investigating, I came across a couple of instances of other people running into similar problems:

https://bugs.php.net/bug.php?id=78902 https://bugs.php.net/bug.php?id=79143

It seems there have been a couple of memory-leak issues present throughout PHP V7. Since my environment is using Docker, it was very straight forward to test some other versions to see if the issue persisted.

In my case, the issue was present when using:

php:7.4.2-apache
php:7.4.18-apache

But the problem appears to be solved when using

php:8.0.5-apache

Hope this helps somebody else that might be banging their head against a wall with this one.

Advertisement

Answer