Since migrating to roundup2 incoming mail messages generally were rendered with additional line breaks, i.e. each line break was doubled. This was with default setting leave_body_unchanged = no.

Using setting leave_body_unchanged = yes doesn't show this behaviour.

I think it is due to line 1919 in mailgw.py, with

        content = '\n\n'.join(l)

I don't know if there was a specific reason to add two line breaks or if it was by accident, please have a look at the patch attached.

On Sat, Dec 05, 2020 at 05:55:06PM +0000, Mathias Behrle wrote:
> 
> Since migrating to roundup2 incoming mail messages generally were
> rendered with additional line breaks, i.e. each line break was
> doubled. This was with default setting leave_body_unchanged = no.
> 
> Using setting leave_body_unchanged = yes doesn't show this behaviour.
> 
> I think it is due to line 1919 in mailgw.py, with
> 
>         content = '\n\n'.join(l)

Thanks Mathias for the bug-report
Hmm, I'm always using 'leave_body_unchanged = yes' in all my setups.
Can anyone comment who uses leave_body_unchanged = no?

Note that this code is *old* (commit 2002 according to hg blame).

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com

Hi Mathias:

I can't manage to reproduce your problem. It looks like the regression 
tests do test this code path and your patch breaks all these tests.

The \n\n is correct as 'l' is a set of message sections separated by
blank lines, so \n\n terminates the last line in the previous section 
and adds a blank line.

What are the settings for these options in the mailgw section of your 
config.ini:

  keep_quoted_text
  leave_body_unchanged
  eol_re
  blankline_re
  sign_re

Thanks.

-- rouilj

Hi John,

> I can't manage to reproduce your problem. It looks like the regression 
> tests do test this code path and your patch breaks all these tests.
> 
> The \n\n is correct as 'l' is a set of message sections separated by
> blank lines, so \n\n terminates the last line in the previous section 
> and adds a blank line.

Ok, than this should not be the reason.
> 
> What are the settings for these options in the mailgw section of your 
> config.ini:
> 
>   keep_quoted_text
>   leave_body_unchanged
>   eol_re
>   blankline_re
>   sign_re

Settings of the mailgw section:

[mailgw]
keep_quoted_text = yes
leave_body_unchanged = yes/no -> this one makes the difference
default_class = issue
language = 
subject_prefix_parsing = strict
subject_suffix_parsing = strict
subject_suffix_delimiters = []
subject_content_match = always
subject_updates_title = no
refwd_re = (\s*\W?\s*(fw|fwd|re|aw|sv|ang)\W)+
origmsg_re = ^[>|\s]*-----\s?Original Message\s?-----$
sign_re = ^[>|\s]*-- ?$
eol_re = [\r\n]+
blankline_re = [\r\n]+\s*[\r\n]+
unpack_rfc822 = no
ignore_alternatives = no
convert_htmltotext = none
keep_real_from = no


Thank you for looking into this!

Hi Mathias,

I am having no luck seeing this bug.

Do you have or can you generate an email that I can test with
that will cause the failure.

I have been testing in the current head release, but you can test
with an unpacked distribution tarball without your patch applied (pip download 
-d . roundup). Or you can get the current tip from https://sourceforge.net/p/roundup/code/ci/default/tarball?path=.

Untar/unzip and in the top level directory:

  Run python3 demo.py with any backend database (sqlite is probably
  best as I assume you are running an rdbms in production).

This will set up the demo test directory. Then you should be able to test 
with:

    python3 roundup/scripts/roundup_mailgw.py demo mailbox tracker.test.mbox

where tracker.test.mbox is an mbox formatted file with one message in it. You 
can create an mbox formatted file by saving a raw email
to the file then edit it and add:

  From user@domain Mon Sep 09 01:56:35 2020

at the top of the file where user@domain matches the email address
in the From: email header. I recommend using the mailbox as it allows
using the debugger by adding '-m pdb'.

I could see one of the following differences causing my failure to
reproduce:

  python version:  can you supply the version (pythonX -V) that you
                   are using for roundup?
  locale settings: Are you using a German locale?

I have applied other patches where I have been unable to reproduce
the issue, but they didn't break the regression tests. So I won't
apply your patch, but I would like to find out what is going wrong
for you.

Have a great 2021.

-- rouilj

Hi Mathias:

I still need to be able to reproduce this issue.
Do you have a sample email that exhibits this problem?

-- rouilj

Hi John,

thanks again for looking into the issue.

> I still need to be able to reproduce this issue.
> Do you have a sample email that exhibits this problem?

I now made some further tests and I confirm that the proposed patch has nothing
to do with the problem.

I am joining several mails to show the results that depend only the different 
setting of leave_body_unchanged.

root@roundup:~# locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

My roundup instance
- is almost trunk
- is running in an LXC container
- in a virtualenv
- on Debian buster

(roundup) venv@roundup:~/projects/roundup$ python
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0] on linux

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://www.m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6
    AC29 7E5C 46B9 D0B6 1C71  7681 D6D0 9BE4 8405 BBF6

Hi Mathias:

I took a quick look.

One thing I noticed is that you have pgp signatures on the emails.
Do you have pgp enabled? if so what is required_incoming set to?

We change the PGP processing library from pyme to gpg. I wonder if
the extra lines are coming from there.

-- rouilj

* John Rouillard: " [issue2551106] Additional line breaks with setting
  leave_body_unchanged = no" (Mon, 15 Mar 2021 02:28:59 +0000):

Hi John,

> I took a quick look.
> 
> One thing I noticed is that you have pgp signatures on the emails.

That was also one of my thoughts.

> Do you have pgp enabled? if so what is required_incoming set to?

No pgp enabled.

require_incoming = signed

> We change the PGP processing library from pyme to gpg. I wonder if
> the extra lines are coming from there.

Cheers

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://www.m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6
    AC29 7E5C 46B9 D0B6 1C71  7681 D6D0 9BE4 8405 BBF6

Hi Mathias:

It looks like your result file is a email nosy message.

Please view the message file in the roundup tracker home directory
under the db/files/msg/<prefix>/msg<msgid>.
Prefix the the first or first two digits of the message id.
Does the extra line spacing show up there as well?

I submitted file1746 to my running test tracker (modified with email
address of a tracker user and a different subject issue#) at:

  https://rouilj dot dynamic-dns.net slash sysadmin/issue1

(url mangled to prevent it from being hit by bots/indexers).

I didn't get any extra whitespace in the msg file, nosy email or web
interface. (If you view the url above, it uses a line height that adds
whitespace between lines. Try highlighting lines to verify that there
are no additional blank line.)

The mailgw settings are:

    [mailgw]
    keep_quoted_text = yes
    leave_body_unchanged = no
    default_class = issue
    language =
    subject_prefix_parsing = strict
    subject_suffix_parsing = strict
    subject_suffix_delimiters = []
    subject_content_match = always
    subject_updates_title = no
    refwd_re = (\s*\W?\s*(fw|fwd|re|aw|sv|ang)\W)+
    origmsg_re = ^[>|\s]*-----\s?Original Message\s?-----$
    sign_re = ^[>|\s]*-- ?$
    eol_re = [\r\n]+
    blankline_re = [\r\n]+\s*[\r\n]+
    unpack_rfc822 = no
    ignore_alternatives = no
    convert_htmltotext = none
    keep_real_from = no

that looks exactly like yours.

If I get a chance later today I'll spin up and configure a demo
tracker and see if I can reproduce with that. However my tracker
doesn't do anything special with message formatting so ...

I wonder if it has something to do with system locale or some such.
All my boxes are configured with en_US. Ralf, do you have a test
tracker using DE that you could test file1746 with
leave_body_unchanged=no.

Ping Mathias:

Did you verify the extra newlines are in the stored message file?

Any other thoughts on my last update?

Have a great day.

-- rouilj

* John Rouillard: " [issue2551106] Additional line breaks with setting
  leave_body_unchanged = no" (Mon, 05 Apr 2021 15:57:51 +0000):

Hi John,

> Did you verify the extra newlines are in the stored message file?

Yes, the additional line breaks are in the db files, it is not a font problem
of the MUA or similar. I am joining the relevant db files as tar.gz.

> Any other thoughts on my last update?

Not really. 

Since I seem to be the only one hitting that problem probability is
high that the issue is related to my local installation. Since I am well served
with my current (and anyway preferred) settings I think it is adequate to
postpone this issue and to not invest too much energy from your side on it until
someone else hits the problem or I can give more input. Big thanks goes to you
for the effort already done!

> Have a great day.

To you as well!

Mathias

Hi Mathias:

I will close this ticket out as unfixed.

Hope you enjoy using roundup.

-- rouilj