Roundup Tracker - Issues

Issue 2551274

classification
Better logging in REST api
Type: behavior Severity: normal
Components: API Versions: devel
process
Status: open
:
: schlatterbeck : Heiko, rouilj, schlatterbeck
Priority: normal : rest

Created on 2023-04-17 09:42 by schlatterbeck, last changed 2024-01-17 16:57 by rouilj.

Messages
msg7758 Author: [hidden] (schlatterbeck) Date: 2023-04-17 09:42
Currently the REST-API doesn't log anything when an error occurs. I'm currently dealing with a problem where the lag between retries and me receiving the problem report is on the order of weeks. From the apache log (roundup runs as WSGI under apache in that application) I see that they receive a 400 from apache. Now I'd really like to know what they sent.

I propose to add logging to failed requests: Currently all errors are handled in the function _data_decorator in rest.py. So we could add a log there centrally and log the parameters of the request and the error message they receive.

I'm unsure about the severity of the log. Since this is the normal operation (and errors are not errors in roundup) I'd suggest to log this as INFO or lower. Or should this be a DEBUG log? I would run this always-on in production as it is normal operation that I have to debug a client web service on my end :-) So I tend to make this INFO.

What do you think?
msg7761 Author: [hidden] (rouilj) Date: 2023-04-17 22:58
Info sounds reasonable.
msg7817 Author: [hidden] (rouilj) Date: 2023-07-25 20:40
Ralf, do you have patches for this?
msg7824 Author: [hidden] (rouilj) Date: 2023-08-09 05:22
Ralf:

What do you think about a config.ini setting:


  rest_logging = [none|debug|info|...]

so you can enable/disable rest logging and set the log level?

As an aside, should we have more section in config.ini, e.g. [rest]?

I would need to figure out how aliases work and move some items around if
we were to add a new section.
msg7825 Author: [hidden] (schlatterbeck) Date: 2023-08-09 07:48
Hi John, thanks for looking into this!

On Wed, Aug 09, 2023 at 05:22:49AM +0000, John Rouillard wrote:
> 
> Ralf:
> 
> What do you think about a config.ini setting:
> 
> 
>   rest_logging = [none|debug|info|...]
> 
> so you can enable/disable rest logging and set the log level?

Good idea! This would cover my needs to run this continuously and I
could make my log-level higher (to hide other things). Better than
having a hard-coded log-level.

> As an aside, should we have more section in config.ini, e.g. [rest]?
Hmm, for now I wouldn't do this we currently don't have too many
rest-specific items I think.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com
msg7867 Author: [hidden] (rouilj) Date: 2023-12-11 00:17
Ralf did you do anything for this ticket?
msg7935 Author: [hidden] (schlatterbeck) Date: 2024-01-17 16:16
I've just pushed a first stab at this.

I would really like to also log the parameters of the REST call (maybe also configurable with an option). For a GET request this is typically a FieldStorage object while for POST or PUT it typically is a dictionary (serialized from JSON). For logging this it would require some serialization. I could live with a multi-line JSON object (serializing the dictionary for POST or PUT) but FieldStorage cannot currently be easily serialized (although serializing this to a nested dictionary should be doable).

Now since FieldStorage is already deprecated in python (going away in 3.13) we might introduce some replacement in the future? It might be nice if everything was a nested dictionary...

Oh and for file uploads and other huge objects we probably want to limit the amount being logged.
msg7936 Author: [hidden] (rouilj) Date: 2024-01-17 16:57
> Now since FieldStorage is already deprecated in python (going away in 3.13) 

It's not just FieldStorage that is deprecated but all of cgi.py.

I have already vendored cgi.py under roundup/anypy/vendored/cgi.py so Roundup can
run under 3.13. This works (tested using a 3.13 alpha docker image).

Alternatives are discussed at:

  https://discuss.python.org/t/alternative-function-for-deprecated-cgi/21960/11

which addresses mostly wsgi (not CGI) use a, vendored cgi.py works.

When we drop python2 support, replacing cgi.py would be a more tractable project,
but the glib solutions to use the email module for parsing put/post bodies etc. are not
trivial replacements.
History
Date User Action Args
2024-01-17 16:57:03rouiljsetmessages: + msg7936
2024-01-17 16:16:03schlatterbecksetstatus: new -> open
messages: + msg7935
2023-12-11 00:17:06rouiljsetmessages: + msg7867
2023-08-09 07:48:13schlatterbecksetmessages: + msg7825
2023-08-09 05:22:48rouiljsetmessages: + msg7824
2023-07-25 20:40:07rouiljsetmessages: + msg7817
2023-04-17 22:58:22rouiljsetmessages: + msg7761
2023-04-17 09:42:44schlatterbeckcreate