Improve `ZSTD_compressStream2` from bucket read to stream

[nono303]

Actually, compression is done by ZSTD_compressStream2
looping on apr_bucket_read

This requires processing the compression block by block (8KiB from what I have seen) and not as a stream.

Once the block processed, it must be flush with ZSTD_e_flush while on "real streamed mode" we could use ZSTD_e_continue :

to let "encoder decides when to output compressed result, for optimal compression ratio"

I’m not an expert on APR so my question is, could we get the bucket brigades data as a single stream?

I had a look on existing filter module src and bucket brigades documentation but didn’t find something relevant

[nono303]

After some research, it turns out that the most efficient way to implement ZSTD_compressStream2 is with ZSTD_e_continue mode (or ZSTD_compressStream as an alias) which is non-blocking and therefore answers question #10.

I try to do as it based on https://github.com/tokers/zstd-nginx-module/blob/master/filter/ngx_http_zstd_filter_module.c , https://github.com/facebook/zstd/blob/dev/examples/streaming_compression.c and https://facebook.github.io/zstd/zstd_manual.html#Chapter8 to implement ZSTD_e_continue but I quickly get lost in the definition of the buffers:

• Positioning (in or outside the brigade bucket loop)
• Size: 1 input buffer per bucket of the size read apr_bucket_read or only 1 of the total size apr_brigade_length
• Same for out_buffer should it be bucket size ZSTD_compressBound(apr_bucket_read) or total size ZSTD_compressBound(apr_brigade_length) or maybe ZSTD_CStreamOutSize()
• In case of change with single output buffer, how to ensure ordering in APR_BRIGADE_INSERT_TAIL

So, if you are fluent with apr_brigade to make non-blocking compression effective, you can make a PR on the cstream branch.
Thx!

[foglede]

If I understand correctly, I think you can use eg：
https://github.com/foglede/mod_zstd/blob/main/mod_zstd.c#L545
in input_filter processing new ZSTD_compressBound，input_filter can choose the processing method chunt or steam
This will speed up the whole process by a few milliseconds.
and input_filter can also determine the content of the response
if response < 64kb the no zstd eg:
https://github.com/tokers/zstd-nginx-module/blob/f4ba115e0b0eaecde545e5f37db6aa18917d8f4b/filter/ngx_http_zstd_filter_module.c#L144

same thread-safe work can be done，not out_filter