Roundup Tracker - Issues

Issue 2551140

Use redis as caching layer or session db implementation.
Type: rfe Severity: normal
Components: Versions:
Status: open
: : rouilj
Priority: normal : Effort-Medium, GSOC

Created on 2021-05-28 02:20 by rouilj, last changed 2022-08-04 22:05 by rouilj.

msg7255 Author: [hidden] (rouilj) Date: 2021-05-28 02:20
This ticket is for using a redis instance for caching for rdbms and 
anydbm backends as well as session database.

issue2550721 has a patch for using memcached to provide extended
caching of database lookups (for rdbms only). This improves
performance and the lack of persistence is not an issue.

Something similar could be done with redis. An added advantage of redis 
is that it has persistence. So it could be used not only for primary
database caching (where we don't care about persistence) but also
as a session db. For the session db we need things like session tokens,
registration tokens and json one time use keys to be persistent
across restarts.

Deploying redis for both of these roles would allow a performance 
increase using only one extra redis process rather than having
both memcached and redis.

Session db performance is an issue reported in msg6486 of issue2551036
when recording rate limiting parameters for the REST interface.
msg7630 Author: [hidden] (rouilj) Date: 2022-08-04 22:05
Sessiondb implemented with redis for anydbm and sqlite primary databases in:

changeset:   6814:3f60a71b0812

Test by running 300 gets in parallel 20 at a time (on 4 core system) against the
rest endpoint /rest/data/issue/4 and watching what the remaining rate limit number is
from the header X-RateLimit-Remaining.

I started with rate limit of 360 with one hit being restored every 10 seconds.

With 300 requests, it take 30 seconds to complete on my small system. I expected
my last count of hits to be 63 (360 - 300 + 30/10), and it was. 0 dropped items.
This is best case, I have seen 1-2 drops in some test cases for .6% loss rate.

sqlite 45 seconds to complete, target 64, last count 107 -> 43 dropped, 14% rate.

anydbm 51 seconds to complete, target 65, last count 67, 2 dropped, 1% rate.

I am not happy about the drops for sqlite or anydbm under high transaction rates.
Makes me wonder what is happening there. But at least we have something that
can address high tx rates.

TODO: cache rdbms/anydbm lookups.
Date User Action Args
2022-08-04 22:05:37rouiljsetstatus: new -> open
messages: + msg7630
2021-05-28 02:20:55rouiljcreate