Roundup Tracker - Issues

Issue 2550564

classification
Title: Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong.
Type: behavior Severity: normal
Components: Mail interface Versions: devel
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ivan Pozdeev, ber, pefu, richard, stefan, tekberg
Priority: Keywords:

Created on 2009-07-13 22:14 by stefan, last changed 2016-04-20 16:29 by Ivan Pozdeev.

Files
File name Uploaded Description Edit Remove
precedence-bulk.txt tekberg, 2015-03-12 15:35 Detail regarding (a) and (b) of Bernhard's msg5271
unnamed richard, 2015-03-13 22:54
Messages
msg3780 Author: [hidden] (stefan) Date: 2009-07-13 22:14
Roundup 1.4.8 wants to set a "Precedence: bulk" header on all email it
sends.  That causes EZMLM to drop the message on the floor.

I think that this is a Roundup bug; there's no reason to think that
*all* mail sent by Roundup should be bulk.  The default nosy reactor
could do this, but the Roundup *core* doesn't have to.  And, not being
able to have a popular mailing list manager on the nosy list seems
like a bug.
msg3789 Author: [hidden] (richard) Date: 2009-07-15 06:24
On 14/07/2009, at 8:14 AM, Stefan Seefeld wrote:
> Roundup 1.4.8 wants to set a "Precedence: bulk" header on all email it
> sends.  That causes EZMLM to drop the message on the floor.
>
> I think that this is a Roundup bug; there's no reason to think that
> *all* mail sent by Roundup should be bulk.  The default nosy reactor
> could do this, but the Roundup *core* doesn't have to.  And, not being
> able to have a popular mailing list manager on the nosy list seems
> like a bug.

That seems reasonable.
msg3806 Author: [hidden] (pefu) Date: 2009-07-16 17:57
Hi Stefan,

please come up with a real world use case for a mailing list
as user on a roundup nosy list.  

IMHO the nosy list is a very fine mailing list manager by itself, 
which is far more powerful than any other mailing list manager from
the users point of view.

But BTW:
I also never understood why mailman has the concept of so called
umbrella lists, which was also build for nesting of mailing lists.

May be I'm too simpled minded?  
But I can hardly imagine the breed of users who want such 
nesting of mail delivery.

Regards, Peter Funk, Germany.
msg3807 Author: [hidden] (stefan) Date: 2009-07-16 18:56
On 07/16/2009 01:57 PM, Peter Funk wrote:

> please come up with a real world use case for a mailing list
> as user on a roundup nosy list.

I don't need to come up with it, I'm *using* it ! :-)

> IMHO the nosy list is a very fine mailing list manager by itself,
> which is far more powerful than any other mailing list manager from
> the users point of view.

I don't want to argue with you about whether and how the nosy list 
feature of Roundup compares to mailman. That's not my point.

My point is that there are people who set up Roundup to send mail to 
lists, and there is no reason to set such a policy inside the Roundup 
core that breaks that.

Regards,
		Stefan
msg3809 Author: [hidden] (pefu) Date: 2009-07-17 12:11
Stefan Seefeld <stefan@codesourcery.com> wrote:
> On 07/16/2009 01:57 PM, Peter Funk wrote:
>
> > please come up with a real world use case for a mailing list
> > as user on a roundup nosy list.
>
> I don't need to come up with it, I'm *using* it ! :-)
>
> > IMHO the nosy list is a very fine mailing list manager by itself,
> > which is far more powerful than any other mailing list manager from
> > the users point of view.
>
> I don't want to argue with you about whether and how the nosy list
> feature of Roundup compares to mailman. That's not my point.
>
> My point is that there are people who set up Roundup to send mail to
> lists, and there is no reason to set such a policy inside the Roundup
> core that breaks that.

Understood.  But IMHO the default behaviour should remain as is.
Suppressing this header should be a configurable option.

Two more thoughts:

Adding a global config option to the mailer class would be simple. But
the inclusion of the bulk-header should depend on the recipient of
the message: If you have a mailing list as nosy recipient, other
nosy users might have auto responders installed, that suppress their
action only if they see a Bulk header.  Unfortunately currently the
mailer class sends a message to all recipients at once.  Making the
presence of the header dependent on the recipient would require to
split the nosy list in two parts. :-(

What about mailgw.py?  If you use a mailing list as recipient, then
other paople might want to do the opposite and want to put a roundup
tracker instance as recipient on a mailing list.  Currently the
ignoring of bulk-messages in mailgw.py is also hard coded:

        ...
        # detect Precedence: Bulk, or Microsoft Outlook autoreplies
        if (message.getheader('precedence', '') == 'bulk'
                or subject.lower().find("autoreply") > 0):
            raise IgnoreBulk
        ...

Regards, Peter
P.S.: I tried to send this reply through email first in the morning,
but it came back from psf.upfronthosting.co.za due to a exception
in mailgw.py similar to one in issue2550534 . I filed a comment
there, because I suspect it is the same problem.
-- 
Peter Funk, ✉Oldenburger Str.86, D-27777 Ganderkesee
office: ArtCom GmbH, ✉Haferwende 2, D-28357 Bremen, Germany
tel:+49-421-20419-0 cell:+49-179-640-8878 <http://www.artcom-gmbh.de/>
msg3810 Author: [hidden] (stefan) Date: 2009-07-17 12:29
On 07/17/2009 08:11 AM, Peter Funk wrote:

>> My point is that there are people who set up Roundup to send mail to
>> lists, and there is no reason to set such a policy inside the Roundup
>> core that breaks that.
>
> Understood.  But IMHO the default behaviour should remain as is.
> Suppressing this header should be a configurable option.

That's my point exactly: Right now this is not only 'default behavior', 
it is not configurable at all ! Taking it out of Roundup's own code, 
putting it into detectors (nosyreactor, say), is precisely what I'm 
proposing.

> Adding a global config option to the mailer class would be simple. But
> the inclusion of the bulk-header should depend on the recipient of
> the message: If you have a mailing list as nosy recipient, other
> nosy users might have auto responders installed, that suppress their
> action only if they see a Bulk header.

Those users may change their filter, as a mail coming from a Roundup 
tracker can easily be detected otherwise.

>  Unfortunately currently the
> mailer class sends a message to all recipients at once.  Making the
> presence of the header dependent on the recipient would require to
> split the nosy list in two parts. :-(

That's a good point, and one for which we have multiple open issues. It 
would be good to be able to refactor the mail handling so it becomes 
configurable. Nosy mails may be sent individually, or grouped, or...

> What about mailgw.py?  If you use a mailing list as recipient, then
> other paople might want to do the opposite and want to put a roundup
> tracker instance as recipient on a mailing list.  Currently the
> ignoring of bulk-messages in mailgw.py is also hard coded:
>
>          ...
>          # detect Precedence: Bulk, or Microsoft Outlook autoreplies
>          if (message.getheader('precedence', '') == 'bulk'
>                  or subject.lower().find("autoreply")>  0):
>              raise IgnoreBulk
>          ...

Hmm, that looks a lot more reasonable to me. But again, this may 
arguably belong into a configurable policy. But this is a different 
issue. :-)

Thanks,
		Stefan
msg3812 Author: [hidden] (richard) Date: 2009-07-19 22:01
On 17/07/2009, at 10:11 PM, Peter Funk wrote:
> Unfortunately currently the
> mailer class sends a message to all recipients at once.  Making the
> presence of the header dependent on the recipient would require to
> split the nosy list in two parts. :-(

The config "email_sending" variable already exists to control this  
behaviour. There's actually a bug filed about attachment handling in  
the split case.
msg3814 Author: [hidden] (stefan) Date: 2009-07-19 22:50
On 07/19/2009 06:01 PM, Richard Jones wrote:

> There's actually a bug filed about attachment handling in
> the split case.

You may be thinking of issue2550568, which is fixed on trunk.

Regards,
		Stefan
msg3819 Author: [hidden] (richard) Date: 2009-07-20 06:16
You really don't want to go messing with the Precedence: bulk stuff. Mail loops with 
autoresponders can get out of hand really, really quickly. I would strongly argue against making 
incoming bulk protection code optional.
msg3824 Author: [hidden] (stefan) Date: 2009-07-21 00:30
I'd be happy if this was configurable, with the default value be set to
what is currently used. However, there are cases such as mine where we
do want Roundup to be able to send to a list, and thus would appreciate
a means to tune the behavior accordingly.
msg3835 Author: [hidden] (richard) Date: 2009-08-10 00:39
Just to be clear: I'm perfectly happy to allow OUTGOING "bulk" header configurable, but I would 
strongly argue against making the INBOUND "bulk" header detection configurable. It can, in 
only average conditions, kill a mail server or two with a single loop.
msg5270 Author: [hidden] (tekberg) Date: 2015-03-11 18:00
Is there any progress on this issue?

Our group has several trackers, some of which have fake users whose email 
address is a mailing list. When one of those fake users (e.g. Leads) is in 
the nosy list, email gets send to our university-wide mailman server for 
delivery. When mailman sees 'Precedence: bulk' it silently drops it. What 
I have to do in each case I see it (Ubuntu 14 logwatch does this too) is 
to either comment out the setting of this mail header, or make it 
configurable. Having a config.ini option to specify this would be great 
for me.
msg5271 Author: [hidden] (ber) Date: 2015-03-11 19:48
Hi there, to make some progress I propose
to keep this issues about sending Precedence:bulk.

a) point to some documents that describe precedence header intended and real behaviour
b) try to find the point (commit) where this was introduced and the reason for it.
c) Check out some examples. The nosy detector for issues.roundup-tracker.org itself
is send to the mailinglist and it seems to work (I don't know precisely why).

I agree that mail loop prevention is important, I have also seen email loop clutter up a full 
company internet connection within a short timeframe.
msg5274 Author: [hidden] (tekberg) Date: 2015-03-12 15:35
I did some investigation on items (a) and (b) of Bernhard's msg5271. It's 
pretty long so I put it in the file precedence-bulk.txt. Reading it may 
help your understanding of this header field.

Regarding (c) the problem I had was when the mailing list was managed by 
mailman. Perhaps other mailing list managers don't look at Precedence: 
bulk.
msg5275 Author: [hidden] (ber) Date: 2015-03-12 20:21
Tom,
thanks for doing the research, so we have some indication that Precedence: bulk can
indicate other email systems to not send an automated reply. But also rfc2076 (an 
informational rfc) that it is "discouraged".  So my next question is: Which software
actually will filter out Precedence:bulk?

I think that roundup's message do not qualify as "bulk" in the sense that a lot of them are 
send right way. Usually nosy lists are quite short I presume.

We have a number of mailman mailinglist and I think roundup nosy messages go right through 
to them, even with the roundup devel mailinglist, I think this is also mailman. 

Tom, are you positive that Mailman is dropping it by default? Or is there something special
about your mailman installation? Which version of mailman anyway

I think the we could drop the header or make it an option, it does not seem to be necessary 
today.
msg5279 Author: [hidden] (tekberg) Date: 2015-03-13 20:26
I have some answers to Bernhard's questions about UW's mailman. We are 
running a locally modified 2.1.19 which is the latest version. I asked about 
the modifications and was told they are not related to Precedence: bulk.

The email I received on 12 Apr 2011 when I first encountered this problem has 
the following in the body:
-----
Thank you for the additional information. That was useful in figuring out 
what the problem was. I went to the sendmail log for Mailman and this is what 
I found:

Apr 12 10:18:22 mailman2 sendmail[30654]: p3CHIMsU030654: from=<roundup-
admin@web.labmed.washington.edu>, auth=no, size=2004, class=-60, nrcpts=1, 
msgid=<1302628686.17.0.575133366818.issue2384@web.labmed.washington.edu>, 
proto=ESMTP, daemon=MTA, relay=mx2.cac.washington.edu [140.142.33.18]
bash-3.2$

The message is listed as class=-60 which is Precedence: Bulk and Mailman 
discards those. Unfortunately, there isn't any way for us to change that on 
this end so the sender will need to configure their mail program such that 
this gets changed. It looks like this sender sent several messages to Mailman 
lists today and they all have that Precedence: Bulk setting and so were 
discarded.
-----
I was told that there is a global configuration option to specify where or 
not to drop messages containing Precedence: bulk. Obviously for us, this flag 
is set. I couldn't determine where that is located.
msg5280 Author: [hidden] (tekberg) Date: 2015-03-13 21:15
Another mailman expert dug deeper to find out what is going on. Here is 
what he said:

  I need to correct myself. The bulk detection feature is a local
  modification to the UW Mailman service.

So that is why our mailman drops 'Precedence: bulk' emails while the un-
modified mailman doesn't.
msg5281 Author: [hidden] (richard) Date: 2015-03-13 22:54
Again folks, I would caution messing with the "Precedence: bulk" header
stuff for the mail-server killing reasons noted previously in this issue.

On Sat, 14 Mar 2015 at 08:21 Tom Ekberg <issues@roundup-tracker.org> wrote:

>
> Tom Ekberg added the comment:
>
> Another mailman expert dug deeper to find out what is going on. Here is
> what he said:
>
>   I need to correct myself. The bulk detection feature is a local
>   modification to the UW Mailman service.
>
> So that is why our mailman drops 'Precedence: bulk' emails while the un-
> modified mailman doesn't.
>
> ________________________________________________
> Roundup tracker <issues@roundup-tracker.org>
> <http://issues.roundup-tracker.org/issue2550564>
> ________________________________________________
>
msg5282 Author: [hidden] (ber) Date: 2015-03-16 08:35
Tom, Richard,

thanks for your remarks!

What I learn from it is:
a) Dropping emails with "Precedence: bulk" does not seem to be a general
problem with mailman, but with a local modification within the
University of Washington. So they could add a customer to strip this
header from emails that have the roundup header set and would be done.
So we probably look for other real world examples where setting
"Precedence: bulk" is a problem.

b) On the other hand, rfc2046 discourages setting "Precedence: bulk",
so it should not at all be necessary. Setting no return-path seems a
regular message, but I don't think we should use it outself. If roundup 
does not react to automatic emails we would be fine to prevent loops.
So here also the question is: In which real world situations would 
roundup emails without this header be a problem?
msg5532 Author: [hidden] (Ivan Pozdeev) Date: 2016-04-18 21:05
1) As per https://tools.ietf.org/html/rfc3834, the "Precedence: bulk"
header's SOLE purpose is to signal mailers that this is autogenerated
mail, so they shall not auto-reply to it. Full stop.

2) The same RFC recommends setting "Auto-Submitted: <anything but `no'>"
instead and doesn't recommend to use "Precedence:";
https://tools.ietf.org/html/rfc5436 updates the RFC, suggesting some
more specific values for the header.

3) Setting "Reply-To:<>" instead is a "hard" solution - no one will _be
able_ to reply (though Outlook/Exchange (2003) is reported to use
"From:" instead sometimes
(https://social.technet.microsoft.com/Forums/exchange/en-US/72e7608a-9ff1-43c3-b6dd-454dafc67672/replyto-address-is-not-used-bei-out-of-office-replies?forum=exchangesvrclientslegacy)),
another one is to set it (and possibly "From:") to a "black hole"
address like "noreply@<domain>".

4) But 3) doesn't look like an option for Roundup because one can reply
to threads by replying to notifications AFAIK.

5) Mailer support for "Auto-Submitted:": supported by Lotus Domino
(https://jira.atlassian.com/browse/JRA-15325), unsupported by Exchange
2007 (message 2011, linked blog post 2006) but "Precedence:" supported
(http://stackoverflow.com/a/7316397/648265), supported by GMail
(https://productforums.google.com/forum/#!msg/apps/INFoYJFBDrI/F9YCFjOrYB0J).
Other mailers I know of do not have auto-reply functionality out of the box.

---

C1) As per 1), the university server admins are completely wrong to drop
these unconditionally and shall be directed to the RFCs, this discussion
and other related references here.
C2) As per 5), we cannot drop "Precedence:" quite yet (unless someone
digs up evidence that the bulk of Exchange servers on duty - used by
tech people, at least - now support the new header)
C3) but as per 2), it may be good to complement it with
"Auto-submitted:" in accordance with the newer RFC.
msg5536 Author: [hidden] (pefu) Date: 2016-04-19 05:55
Hello Ivan,

Stefan brought this issue up in 2009 and has probably found a way to
solve his problem outside Roundup. After reading your msg5532 carefully
Is still don't know what you are proposing:  Is it simply an addendum to
the research done by Tom last year and posted as:
http://issues.roundup-tracker.org/file1568/precedence-bulk.txt ?
If yes then I propose we should close this issue with 
Resolution "wont fix".

Best Regards, Peter Funk
msg5538 Author: [hidden] (Ivan Pozdeev) Date: 2016-04-19 07:34
> After reading your msg5532 carefully
> Is still don't know what you are proposing:  Is it simply an addendum to
> the research done by Tom last year and posted as:
> http://issues.roundup-tracker.org/file1568/precedence-bulk.txt ?
> If yes then I propose we should close this issue with
> Resolution "wont fix".
And I thought I was clear...
It is a more thorough research, and the conclusions are marked "C<n>)".

And they are that we cannot drop the header now and shouldn't be doing 
that to solve the original problem;
neither letting a user set it makes much sense.
So it's indeed WONTFIX.
msg5541 Author: [hidden] (tekberg) Date: 2016-04-19 14:03
So, by saying WONTFIX you are saying one of 2 things to me:
  (1) use some other tracker that doesn't put Precedence: bulk
      on email headers or has an option to remove it, or
  (2) continue to locally patch roundup to remove it so that
      our mailman (managed by another group) will pass our
      mail instead of silently dropping it.

I don't understand why there is resistance to adding a config option 
to not allowing this mail header attribute to be optional. Nobody is 
suggesting to drop Precedence entirely, only to make it a config 
option.
msg5542 Author: [hidden] (pefu) Date: 2016-04-19 14:30
Hello Tom,

Tom Ekberg added the comment:
> So, by saying WONTFIX you are saying one of 2 things to me:
>   (1) use some other tracker that doesn't put Precedence: bulk
>       on email headers or has an option to remove it, or
>   (2) continue to locally patch roundup to remove it so that
>       our mailman (managed by another group) will pass our
>       mail instead of silently dropping it.
> 
> I don't understand why there is resistance to adding a config option 
> to not allowing this mail header attribute to be optional. Nobody is 
> suggesting to drop Precedence entirely, only to make it a config 
> option.

I understand that having to locally patch an upstream version of some 
free software package is a hassle which should be avoided.

So I'm not against such a configuration option.  I've only asked
whether I understood the points made by Ivan in msg5532 this morning.

Tom: Would you care to provide such a patch?

Best Regards,
Peter Funk
-- 
Peter Funk, home: ✉Oldenburger Str.86, D-27777 Ganderkesee
mobile:+49-179-640-8878 phone:+49-421-20419-0 <http://www.artcom-gmbh.de/>
office: ArtCom GmbH, ✉Haferwende 2, D-28357 Bremen, Germany
msg5543 Author: [hidden] (tekberg) Date: 2016-04-19 18:58
I sent an email to the group here at UW that manages mailman. This is the response 
I got:

The RFCs don't actually mandate the usage, value or actions for precedence ("the 
use of Precedence is not specifically recommended by this specification", or 
"Precedence: Non-standard, controversial, discouraged"), so we stick to what our 
experience tells us. Most emails with this tag are things such as "Out of office" 
or vacation messages. Accordingly we drop emails that have this tag to save our 
users from unwanted emails.
msg5544 Author: [hidden] (ber) Date: 2016-04-20 06:46
Am Dienstag, 19. April 2016 16:03:16 schrieb Tom Ekberg:
> I don't understand why there is resistance to adding a config option
> to not allowing this mail header attribute to be optional.

Just in general a config options should only be added if there is
a usecase for several people. Each config option will have to be maintained
in the future and thus multiplies complexity of the product.

I have not looked into the use case for "Precedence: bulk",
for me an important argument would be, if you can make a case about
other people having a need for this.
msg5545 Author: [hidden] (Ivan Pozdeev) Date: 2016-04-20 16:29
> C2) As per 5), we cannot drop "Precedence:" quite yet (unless someone
> digs up evidence that the bulk of Exchange servers on duty - used by
> tech people, at least - now support the new header)

Exchange 2010 is claimed to support RFC 3834 (i.e. likely honors the new
header):
https://technet.microsoft.com/en-US/library/ff848256(v=exchg.141).aspx .

According to
http://www.securityspace.com/s_survey/data/man.201603/mxsurvey.html ,
however, the vast majority of installations still use Exchange 2003 (see
http://capitalhead.com/articles/microsoft-exchange-server-build-numbers-and-release-dates.aspx
for build number reference. Note that the builds in the analysis are
smtpsvc.dll builds which still roughly correspond to those of Exchange).
History
Date User Action Args
2016-04-20 16:29:44Ivan Pozdeevsetmessages: + msg5545
2016-04-20 06:46:09bersetmessages: + msg5544
2016-04-19 18:58:49tekbergsetmessages: + msg5543
2016-04-19 14:30:34pefusetmessages: + msg5542
2016-04-19 14:03:16tekbergsetmessages: + msg5541
2016-04-19 07:34:10Ivan Pozdeevsetmessages: + msg5538
2016-04-19 05:55:25pefusetmessages: + msg5536
2016-04-18 21:05:51Ivan Pozdeevsetnosy: + Ivan Pozdeev
messages: + msg5532
2015-03-16 08:35:49bersetmessages: + msg5282
2015-03-13 22:54:08richardsetfiles: + unnamed
messages: + msg5281
title: Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong. -> Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong.
2015-03-13 21:15:28tekbergsetmessages: + msg5280
2015-03-13 20:26:52tekbergsetmessages: + msg5279
2015-03-12 20:21:53bersetassignee: stefan ->
messages: + msg5275
2015-03-12 15:35:10tekbergsetfiles: + precedence-bulk.txt
messages: + msg5274
2015-03-11 19:48:11bersetnosy: + ber
messages: + msg5271
severity: urgent -> normal
2015-03-11 18:00:54tekbergsetnosy: + tekberg
messages: + msg5270
2009-08-10 00:39:50richardsetmessages: + msg3835
2009-07-21 00:30:27stefansetmessages: + msg3824
2009-07-20 06:16:22richardsetmessages: + msg3819
2009-07-19 22:50:31stefansetmessages: + msg3814
title: Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong. -> Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong.
2009-07-19 22:01:36richardsetmessages: + msg3812
title: Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong. -> Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong.
2009-07-17 12:29:40stefansetmessages: + msg3810
2009-07-17 12:11:03pefusetmessages: + msg3809
2009-07-16 18:56:40stefansetmessages: + msg3807
title: Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong. -> Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong.
2009-07-16 17:57:22pefusetnosy: + pefu
messages: + msg3806
2009-07-15 16:50:50stefansetstatus: new -> open
assignee: stefan
2009-07-15 06:24:43richardsetmessages: + msg3789
title: Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong. -> Roundup sets "Precedence: bulk" on all outgoing mail, which seems wrong.
2009-07-13 22:14:51stefancreate