as requested per issue2551203, here are the requested email messages

the problematic characters dropped into the mail by copy/paste from the devtools browser window.

however, maybe the header line:

    Content-Type: text/plain; charset=UTF-8; format=flowed

should have triggered conversion to utf-8 instead of ascii.

when opening message_to_mailgw.txt in python with ascii codec, you get exactly the same error message:

>>> open ("message_to_mailgw.txt","r",encoding="ascii").read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python-exec/python3.9/../../../lib/python3.9/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 1817: ordinal not in range(128)


thanks,
marcus.

maybe next bug ... ;)

there is only one attachment sent with the initial email although i
added two attachments in the web-interface.

regards,
marcus.

This is an interesting issue. Bern, Ralf this ticket has a message that when process by 
the mail gateway resulted in a UunicodeDecodeError with python 2.7 and roundup 1.6.1
(run as issues.roundup-tracker.org).

I think this bug is fixed in the current development release as I don't get an error 
message when I run a current dev tracker under python 2 or 3.

Does this look like a reasonable way to test. I generated the test message using:

  (echo From roundup-admin@localhost.example.com; curl https://issues.roundup-
tracker.org/file1789/message_to_mailgw.txt ) | sed -e 's/\[issue[0-9]*]/[issue1]/' -e 
's/marcus@priesch.co.at/roundup-admin@localhost.example.com/' > unencode.msg

where roundup-admin@localhost.example.com is the address for my admin user.

This creates an mbox style mail spool file that I used as:

  python3 roundup/scripts/roundup_mailgw.py trackers/issue.tracker mailbox unencode.msg

I did the same with python2 and a python2 tracker homedir.

I verified that the characters that should cause the error were stored in db/msg/msgXX
by running 'od -c' on the file and verifying the 3 digit non-ascii character codes
on the line before the answer: header.

It might be worth looking at

  https://issues.roundup-tracker.org/msg7562

for background and

  https://issues.roundup-tracker.org/issue2550740 
  https://issues.roundup-tracker.org/issue2550540

for other still open encoding issues that sadly do not have reproduction steps.

Do you think I should try to set up a 1.6.1 install to test with to verify that
my test method can cause the issue?

If you guys think this is a valid test, I'll close this ticket out as not-reproducable.

> Does this look like a reasonable way to test.

I don't know just by reading. Thus

> Do you think I should try to set up a 1.6.1 install to test with
> to verify that my test method can cause the issue?

Seems the prudent approach which also is the fastest to produce a reliable result.

Well this is interesting.

Bern, I spun up 1.6.1 under python2 on my test machine and was unable to produce
the error. I did need to add a date to the "From " mbox header line
(From roundup-admin@localhost.example.org Wed Jun 5 01:06:54 2022).
The message was properly added each time (although I did make the message
id unique otherwise the message was ignored).

So I logged into issues.roundup-tracker.org and running the same test (using my email
and a date stamp) did produce the error, however I also saw:

   sudo -u roundup /srv/roundup/env/bin/roundup-mailgw /srv/roundup/trackers/roundup 
mailbox unencode.msg
  /srv/roundup/env/local/lib/python2.7/site-packages/psycopg2/__init__.py:144:
     UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in
     order to keep installing from binary please use "pip install psycopg2-binary"
     instead.
     For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-
pypi>.
  """)
  No handlers could be found for logger "detector"

the handler issue was unexpected. The rsswriter.py detector module to generate an rss
feed from the tracker was one that I copied from my sysadmin tracker. It opened
a logger for "detector" which wasn't defined. So I changed the code to open the 
"roundup" logger instead (and also changed an extension to do the same).

Now when running the same command I can successful submit the same message (after
changing the message id) multiple times without an error being emailed and the
message shows up in the web interface (with the non-ascii chars represented by a glyph).

So this looks like an exception or other error was generated in a reactor
and it triggered the error email but also aborted the finalization of the
message in the database somehow. I am not quite sure how an error in a
detector can do that.

Sadly trying that same thing on my test instances isn't producing the problem.
I have a different version of python2 so that may be an issue.

Hi Marcus,
so far we cannot reproduce the problem.

There must be something different in your setup than in John's.
(Could be some different in Python version or version settings inherited from the general
operating system or something done by a set of reactors or other modules.)

Can you reproduce this in a plain, fresh minimal setup?
(Then you and John could compare.)

Regards
Bernhard

Hi Bernhard,

it happended when submitting to roundup-tracker.org - so, sorry, i 
cannot ...

regards,
marcus.

Hi Bern:

There appears to be a misunderstanding.

In message <1654757054.12.0.23438825126.issue2551209@roundup.psfhosted.org>,
Bernhard Reiter writes:

>so far we cannot reproduce the problem.

This is true for my test instance.

However the problem Marcus' describes is on the
https://issues.roundup-tracker.org tracker when he submitted the
attached message. It's not one of Marcus' trackers.

I was able to reproduce the problem on the issues.roundup-tracker.org
tracker. My "fix" to the logging.getLogger() calls to eliminate the
error seems to have fixed it on issues.roundup-tracker.org. Howvwer I
can't explain why that made the error go away.

Also I can't reproduce that failure in my test code to understand why
that failure caused the behavior Marcus described.

>There must be something different [...]  (Could be some different in
>Python version or version settings inherited from the general
>operating system or something done by a set of reactors or other
>modules.)

That's what I am assuming. However I am using the tracker from
mercurial that should be the same as what is deployed on
issues.roundup-tracker.org. Work continues to verify that assumption.

-- rouilj

I had no luck verifying this.

Since I can't reproduce the error after the fix on issues.roundup-tracker.org (IRO),
I am closing this.

Also I don't get error emails from IRO and since BPO is in read-only mode, they
are not monitoring it as much anymore.

I do have access to the error log, but I don't check it on a regular basis.