Roundup Tracker - Issues

Message7298

Author rouilj
Recipients rouilj
Date 2021-07-05.02:23:16
Message-id <1625451796.83.0.129581430598.issue2551147@roundup.psfhosted.org>
In-reply-to
Roundup doesn't compress (Content-Encoding) assets when returning
them to the client.

This leads to poor scores on google core web vitals.

Roundup behind nginx or apache can compress using gzip. Also
running with uwsgi can do the same. Neither seems to compress
with brotli. Using gnunicorn as a wsgi server won't compress
data.

This patch adds compression with:

  gzip, brotli, zstd

to assets served by roundup. Note that 404 and other error pages are
sent without compression. Also the response to a range request by the
client is not compressed.

Still to be implemented:

  Implement priority order for Accept-Encoding. Right now the
  first matching entry in the list: zstd, br, gzip is used
  (1). So an

     Accept-Encoding: br;q=0.5, gzip;q=0.75, zstd;q=0.25

  will return zstd encoded data but it should return gzip.

  Allow admin to enable/disable on the fly compression and choose
  preferred ordering of compression algorithms if client doesn't
  specify a preference. E.G. brotli may be too expensive on a small
  system and the admin should be able to disable it. (Precompressed
  assets will still be delivered with compression.)

  Consider adding an option to not compress content smaller
  that a certain size. E.G. under 100 bytes is uncompressed.

  The default compression settings should be sufficient, so
  I do not plan on exposing quality/compression levels in the
  config.

The code compresses dynamic output (expansion of web page
templates) on the fly.  It will also compress attachments
accessed via /fileN, /msgN on the fly.

It will compress files accessed via /@@files on the fly. However
for these files we can improve compression and reduce cpu load
and response time by precompressing the files. If you have
style.css and generate:

   style.css.zstd
   style.css.br
   style.css.gzip

it will return the file that matches acceptable compression using the
order in (1) above. So if zstd is in Accept-Encoding it will read and
return style.css.zstd if the client requests the url:

  https://example.com/@@file/style.css

Note the file will always be compressed if the client supports it
since roundup will compress on the fly. Using pre-compressed assets
just reduces server computation time and improves response time. Also
compression uses a middle of the road setting for all compression
methods. E.G. for br it doesn't use quality of 11 (the default) but 4
which gets ~70% of the compression in a fraction of the time. By
precompressing the assets you can squeeze the most space saving out of
the file without having to wait a second for a response.

Client.precompressed_mime_types[] is a list of mime types that should
not be compressed. It includes image/png and image/jpeg. So these
files will not be compressed when set to the client. I believe this
can be changed using interfaces.py in the tracker to add/remove mime
types.

The python standard library provides gzip. To get brotli or zstd
support, you have to install brotli or zstd via pip.

Plan is to apply this post 2.1.0 release in mid July.
History
Date User Action Args
2021-07-05 02:23:16rouiljsetrecipients: + rouilj
2021-07-05 02:23:16rouiljsetmessageid: <1625451796.83.0.129581430598.issue2551147@roundup.psfhosted.org>
2021-07-05 02:23:16rouiljlinkissue2551147 messages
2021-07-05 02:23:16rouiljcreate