Roundup Tracker - Issues

Issue 2551106

classification
Title: Additional line breaks with setting leave_body_unchanged = no
Type: behavior Severity: normal
Components: Mail interface Versions: devel
process
Status: closed Resolution: later
Dependencies: Superseder:
Assigned To: rouilj Nosy List: mbehrle, rouilj, schlatterbeck
Priority: normal Keywords: patch

Created on 2020-12-05 17:55 by mbehrle, last changed 2021-04-12 18:00 by rouilj.

Files
File name Uploaded Description Edit Remove
mailgw.patch mbehrle, 2020-12-05 17:55
issue2551106_test_additional_line_breaks_leave_body_unchanged_yes_result.txt mbehrle, 2021-03-13 09:33
issue2551106_test_additional_line_breaks_leave_body_unchanged_yes_sent.txt mbehrle, 2021-03-13 09:33
issue2551106_test_additional_line_breaks_leave_body_unchanged_no_result.txt mbehrle, 2021-03-13 09:33
issue2551106_test_additional_line_breaks_leave_body_unchanged_no_sent.txt mbehrle, 2021-03-13 09:33
db_files.tar.gz mbehrle, 2021-04-05 19:27
Messages
msg7039 Author: [hidden] (mbehrle) Date: 2020-12-05 17:55
Since migrating to roundup2 incoming mail messages generally were rendered with additional line breaks, i.e. each line break was doubled. This was with default setting leave_body_unchanged = no.

Using setting leave_body_unchanged = yes doesn't show this behaviour.

I think it is due to line 1919 in mailgw.py, with

        content = '\n\n'.join(l)

I don't know if there was a specific reason to add two line breaks or if it was by accident, please have a look at the patch attached.
msg7040 Author: [hidden] (schlatterbeck) Date: 2020-12-05 18:17
On Sat, Dec 05, 2020 at 05:55:06PM +0000, Mathias Behrle wrote:
> 
> Since migrating to roundup2 incoming mail messages generally were
> rendered with additional line breaks, i.e. each line break was
> doubled. This was with default setting leave_body_unchanged = no.
> 
> Using setting leave_body_unchanged = yes doesn't show this behaviour.
> 
> I think it is due to line 1919 in mailgw.py, with
> 
>         content = '\n\n'.join(l)

Thanks Mathias for the bug-report
Hmm, I'm always using 'leave_body_unchanged = yes' in all my setups.
Can anyone comment who uses leave_body_unchanged = no?

Note that this code is *old* (commit 2002 according to hg blame).

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com
msg7041 Author: [hidden] (rouilj) Date: 2020-12-20 05:52
Hi Mathias:

I can't manage to reproduce your problem. It looks like the regression 
tests do test this code path and your patch breaks all these tests.

The \n\n is correct as 'l' is a set of message sections separated by
blank lines, so \n\n terminates the last line in the previous section 
and adds a blank line.

What are the settings for these options in the mailgw section of your 
config.ini:

  keep_quoted_text
  leave_body_unchanged
  eol_re
  blankline_re
  sign_re

Thanks.

-- rouilj
msg7042 Author: [hidden] (mbehrle) Date: 2020-12-20 11:46
Hi John,

> I can't manage to reproduce your problem. It looks like the regression 
> tests do test this code path and your patch breaks all these tests.
> 
> The \n\n is correct as 'l' is a set of message sections separated by
> blank lines, so \n\n terminates the last line in the previous section 
> and adds a blank line.

Ok, than this should not be the reason.
> 
> What are the settings for these options in the mailgw section of your 
> config.ini:
> 
>   keep_quoted_text
>   leave_body_unchanged
>   eol_re
>   blankline_re
>   sign_re

Settings of the mailgw section:

[mailgw]
keep_quoted_text = yes
leave_body_unchanged = yes/no -> this one makes the difference
default_class = issue
language = 
subject_prefix_parsing = strict
subject_suffix_parsing = strict
subject_suffix_delimiters = []
subject_content_match = always
subject_updates_title = no
refwd_re = (\s*\W?\s*(fw|fwd|re|aw|sv|ang)\W)+
origmsg_re = ^[>|\s]*-----\s?Original Message\s?-----$
sign_re = ^[>|\s]*-- ?$
eol_re = [\r\n]+
blankline_re = [\r\n]+\s*[\r\n]+
unpack_rfc822 = no
ignore_alternatives = no
convert_htmltotext = none
keep_real_from = no


Thank you for looking into this!
msg7045 Author: [hidden] (rouilj) Date: 2021-01-03 18:06
Hi Mathias,

I am having no luck seeing this bug.

Do you have or can you generate an email that I can test with
that will cause the failure.

I have been testing in the current head release, but you can test
with an unpacked distribution tarball without your patch applied (pip download 
-d . roundup). Or you can get the current tip from https://sourceforge.net/p/roundup/code/ci/default/tarball?path=.

Untar/unzip and in the top level directory:

  Run python3 demo.py with any backend database (sqlite is probably
  best as I assume you are running an rdbms in production).

This will set up the demo test directory. Then you should be able to test 
with:

    python3 roundup/scripts/roundup_mailgw.py demo mailbox tracker.test.mbox

where tracker.test.mbox is an mbox formatted file with one message in it. You 
can create an mbox formatted file by saving a raw email
to the file then edit it and add:

  From user@domain Mon Sep 09 01:56:35 2020

at the top of the file where user@domain matches the email address
in the From: email header. I recommend using the mailbox as it allows
using the debugger by adding '-m pdb'.

I could see one of the following differences causing my failure to
reproduce:

  python version:  can you supply the version (pythonX -V) that you
                   are using for roundup?
  locale settings: Are you using a German locale?

I have applied other patches where I have been unable to reproduce
the issue, but they didn't break the regression tests. So I won't
apply your patch, but I would like to find out what is going wrong
for you.

Have a great 2021.

-- rouilj
msg7099 Author: [hidden] (rouilj) Date: 2021-03-12 22:45
Hi Mathias:

I still need to be able to reproduce this issue.
Do you have a sample email that exhibits this problem?

-- rouilj
msg7102 Author: [hidden] (mbehrle) Date: 2021-03-13 09:33
Hi John,

thanks again for looking into the issue.

> I still need to be able to reproduce this issue.
> Do you have a sample email that exhibits this problem?

I now made some further tests and I confirm that the proposed patch has nothing
to do with the problem.

I am joining several mails to show the results that depend only the different 
setting of leave_body_unchanged.

root@roundup:~# locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

My roundup instance
- is almost trunk
- is running in an LXC container
- in a virtualenv
- on Debian buster

(roundup) venv@roundup:~/projects/roundup$ python
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0] on linux

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://www.m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6
    AC29 7E5C 46B9 D0B6 1C71  7681 D6D0 9BE4 8405 BBF6
msg7112 Author: [hidden] (rouilj) Date: 2021-03-15 02:28
Hi Mathias:

I took a quick look.

One thing I noticed is that you have pgp signatures on the emails.
Do you have pgp enabled? if so what is required_incoming set to?

We change the PGP processing library from pyme to gpg. I wonder if
the extra lines are coming from there.

-- rouilj
msg7113 Author: [hidden] (mbehrle) Date: 2021-03-15 08:12
* John Rouillard: " [issue2551106] Additional line breaks with setting
  leave_body_unchanged = no" (Mon, 15 Mar 2021 02:28:59 +0000):

Hi John,

> I took a quick look.
> 
> One thing I noticed is that you have pgp signatures on the emails.

That was also one of my thoughts.

> Do you have pgp enabled? if so what is required_incoming set to?

No pgp enabled.

require_incoming = signed

> We change the PGP processing library from pyme to gpg. I wonder if
> the extra lines are coming from there.

Cheers

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://www.m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6
    AC29 7E5C 46B9 D0B6 1C71  7681 D6D0 9BE4 8405 BBF6
msg7147 Author: [hidden] (rouilj) Date: 2021-03-28 18:43
Hi Mathias:

It looks like your result file is a email nosy message.

Please view the message file in the roundup tracker home directory
under the db/files/msg/<prefix>/msg<msgid>.
Prefix the the first or first two digits of the message id.
Does the extra line spacing show up there as well?

I submitted file1746 to my running test tracker (modified with email
address of a tracker user and a different subject issue#) at:

  https://rouilj dot dynamic-dns.net slash sysadmin/issue1

(url mangled to prevent it from being hit by bots/indexers).

I didn't get any extra whitespace in the msg file, nosy email or web
interface. (If you view the url above, it uses a line height that adds
whitespace between lines. Try highlighting lines to verify that there
are no additional blank line.)

The mailgw settings are:

    [mailgw]
    keep_quoted_text = yes
    leave_body_unchanged = no
    default_class = issue
    language =
    subject_prefix_parsing = strict
    subject_suffix_parsing = strict
    subject_suffix_delimiters = []
    subject_content_match = always
    subject_updates_title = no
    refwd_re = (\s*\W?\s*(fw|fwd|re|aw|sv|ang)\W)+
    origmsg_re = ^[>|\s]*-----\s?Original Message\s?-----$
    sign_re = ^[>|\s]*-- ?$
    eol_re = [\r\n]+
    blankline_re = [\r\n]+\s*[\r\n]+
    unpack_rfc822 = no
    ignore_alternatives = no
    convert_htmltotext = none
    keep_real_from = no

that looks exactly like yours.

If I get a chance later today I'll spin up and configure a demo
tracker and see if I can reproduce with that. However my tracker
doesn't do anything special with message formatting so ...

I wonder if it has something to do with system locale or some such.
All my boxes are configured with en_US. Ralf, do you have a test
tracker using DE that you could test file1746 with
leave_body_unchanged=no.
msg7166 Author: [hidden] (rouilj) Date: 2021-04-05 15:57
Ping Mathias:

Did you verify the extra newlines are in the stored message file?

Any other thoughts on my last update?

Have a great day.

-- rouilj
msg7168 Author: [hidden] (mbehrle) Date: 2021-04-05 19:27
* John Rouillard: " [issue2551106] Additional line breaks with setting
  leave_body_unchanged = no" (Mon, 05 Apr 2021 15:57:51 +0000):

Hi John,

> Did you verify the extra newlines are in the stored message file?

Yes, the additional line breaks are in the db files, it is not a font problem
of the MUA or similar. I am joining the relevant db files as tar.gz.

> Any other thoughts on my last update?

Not really. 

Since I seem to be the only one hitting that problem probability is
high that the issue is related to my local installation. Since I am well served
with my current (and anyway preferred) settings I think it is adequate to
postpone this issue and to not invest too much energy from your side on it until
someone else hits the problem or I can give more input. Big thanks goes to you
for the effort already done!

> Have a great day.

To you as well!

Mathias
msg7189 Author: [hidden] (rouilj) Date: 2021-04-12 18:00
Hi Mathias:

I will close this ticket out as unfixed.

Hope you enjoy using roundup.

-- rouilj
History
Date User Action Args
2021-04-12 18:00:55rouiljsetstatus: open -> closed
resolution: later
messages: + msg7189
2021-04-05 19:27:32mbehrlesetfiles: + db_files.tar.gz
messages: + msg7168
2021-04-05 15:57:51rouiljsetmessages: + msg7166
2021-03-28 18:43:57rouiljsetpriority: normal
messages: + msg7147
2021-03-15 08:12:36mbehrlesetmessages: + msg7113
2021-03-15 02:28:59rouiljsetmessages: + msg7112
2021-03-13 09:33:22mbehrlesetfiles: + issue2551106_test_additional_line_breaks_leave_body_unchanged_yes_result.txt, issue2551106_test_additional_line_breaks_leave_body_unchanged_yes_sent.txt, issue2551106_test_additional_line_breaks_leave_body_unchanged_no_result.txt, issue2551106_test_additional_line_breaks_leave_body_unchanged_no_sent.txt
messages: + msg7102
2021-03-12 22:45:31rouiljsetmessages: + msg7099
2021-01-03 18:06:47rouiljsetmessages: + msg7045
2020-12-20 11:46:07mbehrlesetmessages: + msg7042
2020-12-20 05:52:17rouiljsetstatus: new -> open
assignee: rouilj
messages: + msg7041
nosy: + rouilj
2020-12-05 18:17:53schlatterbecksetnosy: + schlatterbeck
messages: + msg7040
2020-12-05 17:55:06mbehrlecreate