Created on 2021-05-28 02:20 by rouilj, last changed 2022-08-04 22:05 by rouilj.
|msg7255||Author: [hidden] (rouilj)||Date: 2021-05-28 02:20|
This ticket is for using a redis instance for caching for rdbms and anydbm backends as well as session database. issue2550721 has a patch for using memcached to provide extended caching of database lookups (for rdbms only). This improves performance and the lack of persistence is not an issue. Something similar could be done with redis. An added advantage of redis is that it has persistence. So it could be used not only for primary database caching (where we don't care about persistence) but also as a session db. For the session db we need things like session tokens, registration tokens and json one time use keys to be persistent across restarts. Deploying redis for both of these roles would allow a performance increase using only one extra redis process rather than having both memcached and redis. Session db performance is an issue reported in msg6486 of issue2551036 when recording rate limiting parameters for the REST interface.
|msg7630||Author: [hidden] (rouilj)||Date: 2022-08-04 22:05|
Sessiondb implemented with redis for anydbm and sqlite primary databases in: changeset: 6814:3f60a71b0812 Test by running 300 gets in parallel 20 at a time (on 4 core system) against the rest endpoint /rest/data/issue/4 and watching what the remaining rate limit number is from the header X-RateLimit-Remaining. I started with rate limit of 360 with one hit being restored every 10 seconds. With 300 requests, it take 30 seconds to complete on my small system. I expected my last count of hits to be 63 (360 - 300 + 30/10), and it was. 0 dropped items. This is best case, I have seen 1-2 drops in some test cases for .6% loss rate. sqlite 45 seconds to complete, target 64, last count 107 -> 43 dropped, 14% rate. anydbm 51 seconds to complete, target 65, last count 67, 2 dropped, 1% rate. I am not happy about the drops for sqlite or anydbm under high transaction rates. Makes me wonder what is happening there. But at least we have something that can address high tx rates. TODO: cache rdbms/anydbm lookups.
|2022-08-04 22:05:37||rouilj||set||status: new -> open|
messages: + msg7630