Roundup Tracker - Issues

Issue 2551209

mailgw submission fails, nosy mail gets sent, message not added to database
Type: crash Severity: normal
Components: Database, Mail interface Versions: 1.6.1
Status: fixed fixed
: rouilj : ber, marcus.priesch, rouilj, schlatterbeck
Priority: :

Created on 2022-06-07 16:20 by marcus.priesch, last changed 2023-02-20 00:19 by rouilj.

File name Uploaded Description Edit Remove
failed_submission_email.txt marcus.priesch, 2022-06-07 16:20
message_to_mailgw.txt marcus.priesch, 2022-06-07 16:20
msg7563 Author: [hidden] (marcus.priesch) Date: 2022-06-07 16:20
as requested per issue2551203, here are the requested email messages

the problematic characters dropped into the mail by copy/paste from the devtools browser window.

however, maybe the header line:

    Content-Type: text/plain; charset=UTF-8; format=flowed

should have triggered conversion to utf-8 instead of ascii.

when opening message_to_mailgw.txt in python with ascii codec, you get exactly the same error message:

>>> open ("message_to_mailgw.txt","r",encoding="ascii").read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python-exec/python3.9/../../../lib/python3.9/encodings/", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 1817: ordinal not in range(128)

msg7564 Author: [hidden] (marcus.priesch) Date: 2022-06-07 16:25
maybe next bug ... ;)

there is only one attachment sent with the initial email although i
added two attachments in the web-interface.

msg7565 Author: [hidden] (rouilj) Date: 2022-06-07 20:24
This is an interesting issue. Bern, Ralf this ticket has a message that when process by 
the mail gateway resulted in a UunicodeDecodeError with python 2.7 and roundup 1.6.1
(run as

I think this bug is fixed in the current development release as I don't get an error 
message when I run a current dev tracker under python 2 or 3.

Does this look like a reasonable way to test. I generated the test message using:

  (echo From; curl https://issues.roundup- ) | sed -e 's/\[issue[0-9]*]/[issue1]/' -e 
's/' > unencode.msg

where is the address for my admin user.

This creates an mbox style mail spool file that I used as:

  python3 roundup/scripts/ trackers/issue.tracker mailbox unencode.msg

I did the same with python2 and a python2 tracker homedir.

I verified that the characters that should cause the error were stored in db/msg/msgXX
by running 'od -c' on the file and verifying the 3 digit non-ascii character codes
on the line before the answer: header.

It might be worth looking at

for background and

for other still open encoding issues that sadly do not have reproduction steps.

Do you think I should try to set up a 1.6.1 install to test with to verify that
my test method can cause the issue?

If you guys think this is a valid test, I'll close this ticket out as not-reproducable.
msg7566 Author: [hidden] (ber) Date: 2022-06-08 06:34
> Does this look like a reasonable way to test.

I don't know just by reading. Thus

> Do you think I should try to set up a 1.6.1 install to test with
> to verify that my test method can cause the issue?

Seems the prudent approach which also is the fastest to produce a reliable result.
msg7575 Author: [hidden] (rouilj) Date: 2022-06-08 18:35
Well this is interesting.

Bern, I spun up 1.6.1 under python2 on my test machine and was unable to produce
the error. I did need to add a date to the "From " mbox header line
(From Wed Jun 5 01:06:54 2022).
The message was properly added each time (although I did make the message
id unique otherwise the message was ignored).

So I logged into and running the same test (using my email
and a date stamp) did produce the error, however I also saw:

   sudo -u roundup /srv/roundup/env/bin/roundup-mailgw /srv/roundup/trackers/roundup 
mailbox unencode.msg
     UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in
     order to keep installing from binary please use "pip install psycopg2-binary"
     For details see: <
  No handlers could be found for logger "detector"

the handler issue was unexpected. The detector module to generate an rss
feed from the tracker was one that I copied from my sysadmin tracker. It opened
a logger for "detector" which wasn't defined. So I changed the code to open the 
"roundup" logger instead (and also changed an extension to do the same).

Now when running the same command I can successful submit the same message (after
changing the message id) multiple times without an error being emailed and the
message shows up in the web interface (with the non-ascii chars represented by a glyph).

So this looks like an exception or other error was generated in a reactor
and it triggered the error email but also aborted the finalization of the
message in the database somehow. I am not quite sure how an error in a
detector can do that.

Sadly trying that same thing on my test instances isn't producing the problem.
I have a different version of python2 so that may be an issue.
msg7576 Author: [hidden] (ber) Date: 2022-06-09 06:44
Hi Marcus,
so far we cannot reproduce the problem.

There must be something different in your setup than in John's.
(Could be some different in Python version or version settings inherited from the general
operating system or something done by a set of reactors or other modules.)

Can you reproduce this in a plain, fresh minimal setup?
(Then you and John could compare.)

msg7577 Author: [hidden] (marcus.priesch) Date: 2022-06-09 08:29
Hi Bernhard,

it happended when submitting to - so, sorry, i 
cannot ...

msg7578 Author: [hidden] (rouilj) Date: 2022-06-09 13:31
Hi Bern:

There appears to be a misunderstanding.

In message <>,
Bernhard Reiter writes:

>so far we cannot reproduce the problem.

This is true for my test instance.

However the problem Marcus' describes is on the tracker when he submitted the
attached message. It's not one of Marcus' trackers.

I was able to reproduce the problem on the
tracker. My "fix" to the logging.getLogger() calls to eliminate the
error seems to have fixed it on Howvwer I
can't explain why that made the error go away.

Also I can't reproduce that failure in my test code to understand why
that failure caused the behavior Marcus described.

>There must be something different [...]  (Could be some different in
>Python version or version settings inherited from the general
>operating system or something done by a set of reactors or other

That's what I am assuming. However I am using the tracker from
mercurial that should be the same as what is deployed on Work continues to verify that assumption.

-- rouilj
msg7720 Author: [hidden] (rouilj) Date: 2023-02-20 00:19
I had no luck verifying this.

Since I can't reproduce the error after the fix on (IRO),
I am closing this.

Also I don't get error emails from IRO and since BPO is in read-only mode, they
are not monitoring it as much anymore.

I do have access to the error log, but I don't check it on a regular basis.
Date User Action Args
2023-02-20 00:19:16rouiljsetstatus: open -> fixed
resolution: fixed
messages: + msg7720
2022-06-09 13:31:49rouiljsetmessages: + msg7578
2022-06-09 08:29:07marcus.prieschsetmessages: + msg7577
2022-06-09 06:44:14bersetmessages: + msg7576
2022-06-08 18:35:25rouiljsetmessages: + msg7575
2022-06-08 06:34:23bersetmessages: + msg7566
2022-06-07 20:24:30rouiljsetstatus: new -> open
assignee: rouilj
messages: + msg7565
nosy: + schlatterbeck, ber, rouilj
2022-06-07 16:25:31marcus.prieschsetmessages: + msg7564
2022-06-07 16:20:11marcus.prieschcreate