Since rev4965:a850f8bae536 the test/test_mailgw.py
the emails will be compared sharper, especially missmatches in the number of the same 
header fields will be found.

After fixing issue2550869 five failures remain in other headers, here is a list:
======================================================================
FAIL: testMultipartRFC822 (test.test_mailgw.MailgwTestCase)

AssertionError: Generated message not correct (diff follows, expected vs. actual):
  FROM: ['"Contrary, Mary" <issue_tracker@your.tracker.email.domain.example>'] != 
['roundup-admin@your.tracker.email.domain.example', '"Contrary, Mary" 
<issue_tracker@your.tracker.email.domain.example>']
  From: ['"Contrary, Mary" <issue_tracker@your.tracker.email.domain.example>'] != 
['roundup-admin@your.tracker.email.domain.example', '"Contrary, Mary" 
<issue_tracker@your.tracker.email.domain.example>']

======================================================================
FAIL: testNewIssueNoAuthorEmail (test.test_mailgw.MailgwTestCase)

  TO: ['chef@bork.bork.bork, mary@test.test, richard@test.test', 'mary@test.test, 
richard@test.test'] != ['mary@test.test, richard@test.test', 'mary@test.test, 
richard@test.test']
  To: ['chef@bork.bork.bork, mary@test.test, richard@test.test', 'mary@test.test, 
richard@test.test'] != ['mary@test.test, richard@test.test', 'mary@test.test, 
richard@test.test']

======================================================================
FAIL: testNewIssueNoAuthorInfo (test.test_mailgw.MailgwTestCase)

  TO: ['chef@bork.bork.bork, mary@test.test, richard@test.test', 'mary@test.test, 
richard@test.test'] != ['mary@test.test, richard@test.test', 'mary@test.test, 
richard@test.test']
  To: ['chef@bork.bork.bork, mary@test.test, richard@test.test', 'mary@test.test, 
richard@test.test'] != ['mary@test.test, richard@test.test', 'mary@test.test, 
richard@test.test']

======================================================================
FAIL: testSecurityMessagePermissionContent (test.test_mailgw.MailgwTestCase)

  TO: ['chef@bork.bork.bork, richard@test.test', 'richard@test.test'] != ['richard@test.test', 
'richard@test.test']
  To: ['chef@bork.bork.bork, richard@test.test', 'richard@test.test'] != ['richard@test.test', 
'richard@test.test']
======================================================================
FAIL: testUnknownUser (test.test_mailgw.MailgwTestCase)

  FROM: ['Roundup issue tracker <roundup-admin@your.tracker.email.domain.example>', 
'Roundup issue tracker <roundup-admin@your.tracker.email.domain.example>'] != 
['roundup-admin@your.tracker.email.domain.example', 'Roundup issue tracker <roundup-
admin@your.tracker.email.domain.example>']
  MIME-Version: ['1.0', '1.0'] != ['1.0']
  From: ['Roundup issue tracker <roundup-admin@your.tracker.email.domain.example>', 
'Roundup issue tracker <roundup-admin@your.tracker.email.domain.example>'] != 
['roundup-admin@your.tracker.email.domain.example', 'Roundup issue tracker <roundup-
admin@your.tracker.email.domain.example>']
  Precedence: ['bulk', 'bulk'] != ['bulk']

----------------------------------------------------------------------
Ran 119 tests in 32.850s
FAILED (failures=5)

changeset:   4970:e2793024bd44
user:        Bernhard Reiter <bernhard@intevation.de>
date:        Tue Mar 10 22:04:29 2015 +0100
files:       test/test_mailgw.py
description:
test_mailgw.py fixing the MIME-Version and Precedence duplicate headers of 
testUnknownUser (partly issue2550877)

More thoughts:
Hypothesis a): the headers "to:" and "from:" are not case sensitive, so 
"FROM:" and "From:" are equivalent.
Hypothesis b) therefore there should be only one of "to:" and "from:"

They they hold, the tests would need to be fixed first, then the question 
is: What is the right version of from: and to:? (Which one would the smtp 
protocol actually take?)

The next steps:
How can I verify the hypothesis?
idea i) look up SMTP standard
idea ii) write/use/adapt an example code and try it with a real smtp 
server/client

The mail header

  Precedence: bulk

causes trouble with some of our 'user's. We have defined a small number of 
user objects for our trackers which have as their email address a mailing 
list. For example, we have a user 'A Lead' in one of our trackers that has 
the leads' mailing list as its email address (the 'address' column). This 
is used in the nosy and assignedto attributes. Emails for these users get 
directed to our mailman server which silently deletes them if they have 
this mail header.

Is there a way a configuration option could be used to decide where or not 
to generate this mail header? Something like 'mail.bulk_email = yes' in 
config.ini.

On Wednesday 11 March 2015 at 17:37:34, Tom Ekberg wrote:
> Is there a way a configuration option could be used to decide where or not
> to generate this mail header? Something like 'mail.bulk_email = yes' in
> config.ini.

Not that I know of.
Best would be to split out a new issue, because I first want to get
the behaviour as intended and sending a Precendence:bulk is intended
so far. Maybe you can describe the problems in greater detail.

It looks like open issue 2550564 discusses this problem. I'll use that 
issue instead of this one.

Checked hypothesis a) by idea i) looking up the standards. a) holds.
Details:
The current standard for email is rfc5322 (Internet Message Format).
Section 3.6 of rfc5322 states that from: and to: headers can only appear once.

The rules are given as 3.6.2.  Originator Fields
        from = "From:" mailbox-list CRLF
and 3.6.3.  Destination Address Fields
        to = "To:" address-list CRLF
in ABNF (rfc5234) which says: 
        Characters will be specified either by a decimal value
        (e.g., the value %d65 for uppercase A and %d97 for lowercase A) 
        or by a case-insensitive literal value enclosed in quotation marks 
        (e.g., "A" for either uppercase or lowercase A).
so the headers  fieldnames are case-insensitive.

Okay the "FROM:" and "TO:" lines are not really send, it is just debug output from 
Mailer.smtp_send(). And the case insensitivity explains why we get two comparisons,
"From:" and "FROM:" are found as two different keys, but they both match both values.

Reading the tests, testMultipartRFC822 was missing the FROM:
and testUnknownUser just had the wrong line in there with the display. This is easy to fix.

changeset:   4974:65a76a7f3e01
user:        Bernhard Reiter <bernhard@intevation.de>
date:        Wed Mar 11 21:47:41 2015 +0100
files:       test/test_mailgw.py
description:
issue2550877 partly fixed.
  * Added docstring hint to _get_mail().
  * Fixed FROM: lines in testMultipartRFC822() and testUnknownUser()

So we are down to 3 tests failing because of to: missmatches, have to find out
what the correct behaviour is, here.

In all three tests still failing check is in the envelope "TO:", but because the default
of messages_to_author is no, the setting is not changed for these tests, chef should
not get an email. So I've fixed the tests by removing chef from the TO:.

changeset:   rev4976:b198e50dc7dc
user:        Bernhard Reiter <bernhard@intevation.de>
date:        Thu Mar 12 22:11:20 2015 +0100
files:       test/test_mailgw.py
description:
issued2550877 fixed, the tests adapted to the more precise header comparison. Added a 
hint to the code how headers are handled.

So the test run fine again.
I have a nosy behaviour question in testSecurityMessagePermissionContent()
which I am going to ask on -devel.

----------Original Message----------
From: John Kristensen <john@jerrykan.com>

On 13/03/15 08:33, Bernhard Reiter wrote:
> with 4977:1417dbdb8dbf all tests run fine again here.

The tests seem to pass with python v2.7, but are still broken with v2.5/v2.6

The 28 failues with python2.6
(email.__version__ 4.0.2) are all with a difference in whitespace.
My idea is to just make the whitespace equal. Or to explicitely parse
all headers)

FAIL: testEnc01 (test.test_mailgw.MailgwTestCase)
From: ['roundup-admin@your.tracker.email.domain.example',
'=?utf-8?b?w6TDtsO8w4TDlsOcw58sIE1hcnk=?=\n
<issue_tracker@your.tracker.email.domain.example>'] !=
['roundup-admin@your.tracker.email.domain.example',
'=?utf-8?b?w6TDtsO8w4TDlsOcw58sIE1hcnk=?=\n\t<issue_tracker@your.tracker.email.domain.example>']
  Reply-To: ['Roundup issue tracker\n
<issue_tracker@your.tracker.email.domain.example>'] != ['Roundup issue
tracker\n\t<issue_tracker@your.tracker.email.domain.example>']

FAIL: testEncNonUTF8 (test.test_mailgw.MailgwTestCase)
Reply-To: ['Roundup issue tracker\n
<issue_tracker@your.tracker.email.domain.example>'] != ['Roundup issue
tracker\n\t<issue_tracker@your.tracker.email.domain.example>']

FAIL: testFollowup (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupExplicitSubjectChange (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupNoNosyAuthor (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupNoNosyAuthorButCopy (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupNoNosyAuthorNoCopy (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupNoNosyRecipients (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupNoSubjectChange (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupNosyAuthor (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupNosyAuthorAndCopy (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupNosyAuthorNosyCopy (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupNosyRecipients (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupStupidQuoting (test.test_mailgw.MailgwTestCase)
FAIL: testFollowupTitleMatch (test.test_mailgw.MailgwTestCase)
FAIL: testMultipartCharsetLatin1AttachFile (test.test_mailgw.MailgwTestCase)
FAIL: testMultipartCharsetLatin1NoAttach (test.test_mailgw.MailgwTestCase)
FAIL: testMultipartCharsetUTF8AttachFile (test.test_mailgw.MailgwTestCase)
FAIL: testMultipartCharsetUTF8NoAttach (test.test_mailgw.MailgwTestCase)
FAIL: testMultipartEnc01 (test.test_mailgw.MailgwTestCase)
FAIL: testMultipartRFC822 (test.test_mailgw.MailgwTestCase)
FAIL: testNewIssueAuthMsg (test.test_mailgw.MailgwTestCase)
FAIL: testNewIssueNoAuthorEmail (test.test_mailgw.MailgwTestCase)
FAIL: testNewIssueNoAuthorInfo (test.test_mailgw.MailgwTestCase)
FAIL: testNosyGeneration (test.test_mailgw.MailgwTestCase)
FAIL: testPropertyChangeOnly (test.test_mailgw.MailgwTestCase)
FAIL: testSecurityMessagePermissionContent (test.test_mailgw.MailgwTestCase)
FAIL: testSimpleFollowup (test.test_mailgw.MailgwTestCase)
FAILED (failures=28)

The remaining problem is not easy to fix because
python2.5/2.6 and 2.7 behave differently. :/
One creates continuation lines with ' ' (2.7.) and the other with '\t'.

See http://bugs.python.org/issue1974
and https://hg.python.org/cpython/rev/5deb27042e5a 

This means we also get differnent behaviour when writing emails
with roundup running on either 2.5/2.6 or 2.7. :/

Behaviour fixed for python 2.5 2.6 2.7.

changeset:   rev4979:f1a2bd1dea77
tag:         tip
user:        Bernhard Reiter <bernhard@intevation.de>
date:        Mon Mar 16 16:16:02 2015 +0100
files:       CHANGES.txt roundup/anypy/email_.py roundup/cgi/client.py
roundup/mailer.py test/test_mailgw.py
description:
issue2550877: Writing headers with the email module will use
continuation_ws = ' ' now for python 2.5 and 2.6 when importing
anypy.email_.