Roundup Tracker - Issues

Issue 2551106

classification
Title: Additional line breaks with setting leave_body_unchanged = no
Type: behavior Severity: normal
Components: Mail interface Versions: devel
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: rouilj Nosy List: mbehrle, rouilj, schlatterbeck
Priority: Keywords: patch

Created on 2020-12-05 17:55 by mbehrle, last changed 2021-01-03 18:06 by rouilj.

Files
File name Uploaded Description Edit Remove
mailgw.patch mbehrle, 2020-12-05 17:55
Messages
msg7039 Author: [hidden] (mbehrle) Date: 2020-12-05 17:55
Since migrating to roundup2 incoming mail messages generally were rendered with additional line breaks, i.e. each line break was doubled. This was with default setting leave_body_unchanged = no.

Using setting leave_body_unchanged = yes doesn't show this behaviour.

I think it is due to line 1919 in mailgw.py, with

        content = '\n\n'.join(l)

I don't know if there was a specific reason to add two line breaks or if it was by accident, please have a look at the patch attached.
msg7040 Author: [hidden] (schlatterbeck) Date: 2020-12-05 18:17
On Sat, Dec 05, 2020 at 05:55:06PM +0000, Mathias Behrle wrote:
> 
> Since migrating to roundup2 incoming mail messages generally were
> rendered with additional line breaks, i.e. each line break was
> doubled. This was with default setting leave_body_unchanged = no.
> 
> Using setting leave_body_unchanged = yes doesn't show this behaviour.
> 
> I think it is due to line 1919 in mailgw.py, with
> 
>         content = '\n\n'.join(l)

Thanks Mathias for the bug-report
Hmm, I'm always using 'leave_body_unchanged = yes' in all my setups.
Can anyone comment who uses leave_body_unchanged = no?

Note that this code is *old* (commit 2002 according to hg blame).

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com
msg7041 Author: [hidden] (rouilj) Date: 2020-12-20 05:52
Hi Mathias:

I can't manage to reproduce your problem. It looks like the regression 
tests do test this code path and your patch breaks all these tests.

The \n\n is correct as 'l' is a set of message sections separated by
blank lines, so \n\n terminates the last line in the previous section 
and adds a blank line.

What are the settings for these options in the mailgw section of your 
config.ini:

  keep_quoted_text
  leave_body_unchanged
  eol_re
  blankline_re
  sign_re

Thanks.

-- rouilj
msg7042 Author: [hidden] (mbehrle) Date: 2020-12-20 11:46
Hi John,

> I can't manage to reproduce your problem. It looks like the regression 
> tests do test this code path and your patch breaks all these tests.
> 
> The \n\n is correct as 'l' is a set of message sections separated by
> blank lines, so \n\n terminates the last line in the previous section 
> and adds a blank line.

Ok, than this should not be the reason.
> 
> What are the settings for these options in the mailgw section of your 
> config.ini:
> 
>   keep_quoted_text
>   leave_body_unchanged
>   eol_re
>   blankline_re
>   sign_re

Settings of the mailgw section:

[mailgw]
keep_quoted_text = yes
leave_body_unchanged = yes/no -> this one makes the difference
default_class = issue
language = 
subject_prefix_parsing = strict
subject_suffix_parsing = strict
subject_suffix_delimiters = []
subject_content_match = always
subject_updates_title = no
refwd_re = (\s*\W?\s*(fw|fwd|re|aw|sv|ang)\W)+
origmsg_re = ^[>|\s]*-----\s?Original Message\s?-----$
sign_re = ^[>|\s]*-- ?$
eol_re = [\r\n]+
blankline_re = [\r\n]+\s*[\r\n]+
unpack_rfc822 = no
ignore_alternatives = no
convert_htmltotext = none
keep_real_from = no


Thank you for looking into this!
msg7045 Author: [hidden] (rouilj) Date: 2021-01-03 18:06
Hi Mathias,

I am having no luck seeing this bug.

Do you have or can you generate an email that I can test with
that will cause the failure.

I have been testing in the current head release, but you can test
with an unpacked distribution tarball without your patch applied (pip download 
-d . roundup). Or you can get the current tip from https://sourceforge.net/p/roundup/code/ci/default/tarball?path=.

Untar/unzip and in the top level directory:

  Run python3 demo.py with any backend database (sqlite is probably
  best as I assume you are running an rdbms in production).

This will set up the demo test directory. Then you should be able to test 
with:

    python3 roundup/scripts/roundup_mailgw.py demo mailbox tracker.test.mbox

where tracker.test.mbox is an mbox formatted file with one message in it. You 
can create an mbox formatted file by saving a raw email
to the file then edit it and add:

  From user@domain Mon Sep 09 01:56:35 2020

at the top of the file where user@domain matches the email address
in the From: email header. I recommend using the mailbox as it allows
using the debugger by adding '-m pdb'.

I could see one of the following differences causing my failure to
reproduce:

  python version:  can you supply the version (pythonX -V) that you
                   are using for roundup?
  locale settings: Are you using a German locale?

I have applied other patches where I have been unable to reproduce
the issue, but they didn't break the regression tests. So I won't
apply your patch, but I would like to find out what is going wrong
for you.

Have a great 2021.

-- rouilj
History
Date User Action Args
2021-01-03 18:06:47rouiljsetmessages: + msg7045
2020-12-20 11:46:07mbehrlesetmessages: + msg7042
2020-12-20 05:52:17rouiljsetstatus: new -> open
assignee: rouilj
messages: + msg7041
nosy: + rouilj
2020-12-05 18:17:53schlatterbecksetnosy: + schlatterbeck
messages: + msg7040
2020-12-05 17:55:06mbehrlecreate