Roundup doesn't compress (Content-Encoding) assets when returning
them to the client.

This leads to poor scores on google core web vitals.

Roundup behind nginx or apache can compress using gzip. Also
running with uwsgi can do the same. Neither seems to compress
with brotli. Using gnunicorn as a wsgi server won't compress
data.

This patch adds compression with:

  gzip, brotli, zstd

to assets served by roundup. Note that 404 and other error pages are
sent without compression. Also the response to a range request by the
client is not compressed.

Still to be implemented:

  Implement priority order for Accept-Encoding. Right now the
  first matching entry in the list: zstd, br, gzip is used
  (1). So an

     Accept-Encoding: br;q=0.5, gzip;q=0.75, zstd;q=0.25

  will return zstd encoded data but it should return gzip.

  Allow admin to enable/disable on the fly compression and choose
  preferred ordering of compression algorithms if client doesn't
  specify a preference. E.G. brotli may be too expensive on a small
  system and the admin should be able to disable it. (Precompressed
  assets will still be delivered with compression.)

  Consider adding an option to not compress content smaller
  that a certain size. E.G. under 100 bytes is uncompressed.

  The default compression settings should be sufficient, so
  I do not plan on exposing quality/compression levels in the
  config.

The code compresses dynamic output (expansion of web page
templates) on the fly.  It will also compress attachments
accessed via /fileN, /msgN on the fly.

It will compress files accessed via /@@files on the fly. However
for these files we can improve compression and reduce cpu load
and response time by precompressing the files. If you have
style.css and generate:

   style.css.zstd
   style.css.br
   style.css.gzip

it will return the file that matches acceptable compression using the
order in (1) above. So if zstd is in Accept-Encoding it will read and
return style.css.zstd if the client requests the url:

  https://example.com/@@file/style.css

Note the file will always be compressed if the client supports it
since roundup will compress on the fly. Using pre-compressed assets
just reduces server computation time and improves response time. Also
compression uses a middle of the road setting for all compression
methods. E.G. for br it doesn't use quality of 11 (the default) but 4
which gets ~70% of the compression in a fraction of the time. By
precompressing the assets you can squeeze the most space saving out of
the file without having to wait a second for a response.

Client.precompressed_mime_types[] is a list of mime types that should
not be compressed. It includes image/png and image/jpeg. So these
files will not be compressed when set to the client. I believe this
can be changed using interfaces.py in the tracker to add/remove mime
types.

The python standard library provides gzip. To get brotli or zstd
support, you have to install brotli or zstd via pip.

Plan is to apply this post 2.1.0 release in mid July.

First pass in changeset:   6458:8f1b91756457

Had a couple of fixes to the tests afterward.

Changes from original description:

 404 pages may be compressed if > 100 bytes in size. No option to
   set size limit. Will see if anybody requests it.

 No option to choose which methods can be used for on the fly
   compression. Use of virtualenv and installing just the compression
   modules you want to use is suggested. Preferred order is hardcoded
   (first to last) zstd, brotli gzip.

 Admin/config settings are limited:
    turn on/off dynamic compression. Default on.
    enable/disable looking for precompressed static
        assets. Default off since admin needs to compress the
        assets. Default of off means there is no search penalty
        by default.

Any precompressed asset that can be handled by the client will
prevent on the fly compression of that asset. This happens even if
the on-the-fly compression is a better choice.

E.G. if the file foo.css.gzip exists, a request for @@file/foo.css
with brotli or gzip encoding will send the gzip encoding. It will
not upgrade to brotli encoding because a pre-compressed foo.css.br
does not exist.

If foo.css.gzip did not exist, foo.css would be served and brotli 
encoding would be done on the fly.

If the client can't handle gzip, foo.css would be served with brotli.

This way pure dynamic content can use brotli, but static assets can
use only the acceptable provided pre-compressed assets.

Marking as pending and if no updates in a couple of weeks will close.

Closing. Not going to implement the other ideas as YAGNI.