Roundup 1.4.8 wants to set a "Precedence: bulk" header on all email it
sends.  That causes EZMLM to drop the message on the floor.

I think that this is a Roundup bug; there's no reason to think that
*all* mail sent by Roundup should be bulk.  The default nosy reactor
could do this, but the Roundup *core* doesn't have to.  And, not being
able to have a popular mailing list manager on the nosy list seems
like a bug.

On 14/07/2009, at 8:14 AM, Stefan Seefeld wrote:
> Roundup 1.4.8 wants to set a "Precedence: bulk" header on all email it
> sends.  That causes EZMLM to drop the message on the floor.
>
> I think that this is a Roundup bug; there's no reason to think that
> *all* mail sent by Roundup should be bulk.  The default nosy reactor
> could do this, but the Roundup *core* doesn't have to.  And, not being
> able to have a popular mailing list manager on the nosy list seems
> like a bug.

That seems reasonable.

Hi Stefan,

please come up with a real world use case for a mailing list
as user on a roundup nosy list.  

IMHO the nosy list is a very fine mailing list manager by itself, 
which is far more powerful than any other mailing list manager from
the users point of view.

But BTW:
I also never understood why mailman has the concept of so called
umbrella lists, which was also build for nesting of mailing lists.

May be I'm too simpled minded?  
But I can hardly imagine the breed of users who want such 
nesting of mail delivery.

Regards, Peter Funk, Germany.

On 07/16/2009 01:57 PM, Peter Funk wrote:

> please come up with a real world use case for a mailing list
> as user on a roundup nosy list.

I don't need to come up with it, I'm *using* it ! :-)

> IMHO the nosy list is a very fine mailing list manager by itself,
> which is far more powerful than any other mailing list manager from
> the users point of view.

I don't want to argue with you about whether and how the nosy list 
feature of Roundup compares to mailman. That's not my point.

My point is that there are people who set up Roundup to send mail to 
lists, and there is no reason to set such a policy inside the Roundup 
core that breaks that.

Regards,
		Stefan

Stefan Seefeld <stefan@codesourcery.com> wrote:
> On 07/16/2009 01:57 PM, Peter Funk wrote:
>
> > please come up with a real world use case for a mailing list
> > as user on a roundup nosy list.
>
> I don't need to come up with it, I'm *using* it ! :-)
>
> > IMHO the nosy list is a very fine mailing list manager by itself,
> > which is far more powerful than any other mailing list manager from
> > the users point of view.
>
> I don't want to argue with you about whether and how the nosy list
> feature of Roundup compares to mailman. That's not my point.
>
> My point is that there are people who set up Roundup to send mail to
> lists, and there is no reason to set such a policy inside the Roundup
> core that breaks that.

Understood.  But IMHO the default behaviour should remain as is.
Suppressing this header should be a configurable option.

Two more thoughts:

Adding a global config option to the mailer class would be simple. But
the inclusion of the bulk-header should depend on the recipient of
the message: If you have a mailing list as nosy recipient, other
nosy users might have auto responders installed, that suppress their
action only if they see a Bulk header.  Unfortunately currently the
mailer class sends a message to all recipients at once.  Making the
presence of the header dependent on the recipient would require to
split the nosy list in two parts. :-(

What about mailgw.py?  If you use a mailing list as recipient, then
other paople might want to do the opposite and want to put a roundup
tracker instance as recipient on a mailing list.  Currently the
ignoring of bulk-messages in mailgw.py is also hard coded:

        ...
        # detect Precedence: Bulk, or Microsoft Outlook autoreplies
        if (message.getheader('precedence', '') == 'bulk'
                or subject.lower().find("autoreply") > 0):
            raise IgnoreBulk
        ...

Regards, Peter
P.S.: I tried to send this reply through email first in the morning,
but it came back from psf.upfronthosting.co.za due to a exception
in mailgw.py similar to one in issue2550534 . I filed a comment
there, because I suspect it is the same problem.
-- 
Peter Funk, ✉Oldenburger Str.86, D-27777 Ganderkesee
office: ArtCom GmbH, ✉Haferwende 2, D-28357 Bremen, Germany
tel:+49-421-20419-0 cell:+49-179-640-8878 <http://www.artcom-gmbh.de/>

On 07/17/2009 08:11 AM, Peter Funk wrote:

>> My point is that there are people who set up Roundup to send mail to
>> lists, and there is no reason to set such a policy inside the Roundup
>> core that breaks that.
>
> Understood.  But IMHO the default behaviour should remain as is.
> Suppressing this header should be a configurable option.

That's my point exactly: Right now this is not only 'default behavior', 
it is not configurable at all ! Taking it out of Roundup's own code, 
putting it into detectors (nosyreactor, say), is precisely what I'm 
proposing.

> Adding a global config option to the mailer class would be simple. But
> the inclusion of the bulk-header should depend on the recipient of
> the message: If you have a mailing list as nosy recipient, other
> nosy users might have auto responders installed, that suppress their
> action only if they see a Bulk header.

Those users may change their filter, as a mail coming from a Roundup 
tracker can easily be detected otherwise.

>  Unfortunately currently the
> mailer class sends a message to all recipients at once.  Making the
> presence of the header dependent on the recipient would require to
> split the nosy list in two parts. :-(

That's a good point, and one for which we have multiple open issues. It 
would be good to be able to refactor the mail handling so it becomes 
configurable. Nosy mails may be sent individually, or grouped, or...

> What about mailgw.py?  If you use a mailing list as recipient, then
> other paople might want to do the opposite and want to put a roundup
> tracker instance as recipient on a mailing list.  Currently the
> ignoring of bulk-messages in mailgw.py is also hard coded:
>
>          ...
>          # detect Precedence: Bulk, or Microsoft Outlook autoreplies
>          if (message.getheader('precedence', '') == 'bulk'
>                  or subject.lower().find("autoreply")>  0):
>              raise IgnoreBulk
>          ...

Hmm, that looks a lot more reasonable to me. But again, this may 
arguably belong into a configurable policy. But this is a different 
issue. :-)

Thanks,
		Stefan

On 17/07/2009, at 10:11 PM, Peter Funk wrote:
> Unfortunately currently the
> mailer class sends a message to all recipients at once.  Making the
> presence of the header dependent on the recipient would require to
> split the nosy list in two parts. :-(

The config "email_sending" variable already exists to control this  
behaviour. There's actually a bug filed about attachment handling in  
the split case.

On 07/19/2009 06:01 PM, Richard Jones wrote:

> There's actually a bug filed about attachment handling in
> the split case.

You may be thinking of issue2550568, which is fixed on trunk.

Regards,
		Stefan

You really don't want to go messing with the Precedence: bulk stuff. Mail loops with 
autoresponders can get out of hand really, really quickly. I would strongly argue against making 
incoming bulk protection code optional.

I'd be happy if this was configurable, with the default value be set to
what is currently used. However, there are cases such as mine where we
do want Roundup to be able to send to a list, and thus would appreciate
a means to tune the behavior accordingly.

Just to be clear: I'm perfectly happy to allow OUTGOING "bulk" header configurable, but I would 
strongly argue against making the INBOUND "bulk" header detection configurable. It can, in 
only average conditions, kill a mail server or two with a single loop.

Is there any progress on this issue?

Our group has several trackers, some of which have fake users whose email 
address is a mailing list. When one of those fake users (e.g. Leads) is in 
the nosy list, email gets send to our university-wide mailman server for 
delivery. When mailman sees 'Precedence: bulk' it silently drops it. What 
I have to do in each case I see it (Ubuntu 14 logwatch does this too) is 
to either comment out the setting of this mail header, or make it 
configurable. Having a config.ini option to specify this would be great 
for me.

Hi there, to make some progress I propose
to keep this issues about sending Precedence:bulk.

a) point to some documents that describe precedence header intended and real behaviour
b) try to find the point (commit) where this was introduced and the reason for it.
c) Check out some examples. The nosy detector for issues.roundup-tracker.org itself
is send to the mailinglist and it seems to work (I don't know precisely why).

I agree that mail loop prevention is important, I have also seen email loop clutter up a full 
company internet connection within a short timeframe.

I did some investigation on items (a) and (b) of Bernhard's msg5271. It's 
pretty long so I put it in the file precedence-bulk.txt. Reading it may 
help your understanding of this header field.

Regarding (c) the problem I had was when the mailing list was managed by 
mailman. Perhaps other mailing list managers don't look at Precedence: 
bulk.

Tom,
thanks for doing the research, so we have some indication that Precedence: bulk can
indicate other email systems to not send an automated reply. But also rfc2076 (an 
informational rfc) that it is "discouraged".  So my next question is: Which software
actually will filter out Precedence:bulk?

I think that roundup's message do not qualify as "bulk" in the sense that a lot of them are 
send right way. Usually nosy lists are quite short I presume.

We have a number of mailman mailinglist and I think roundup nosy messages go right through 
to them, even with the roundup devel mailinglist, I think this is also mailman. 

Tom, are you positive that Mailman is dropping it by default? Or is there something special
about your mailman installation? Which version of mailman anyway

I think the we could drop the header or make it an option, it does not seem to be necessary 
today.

I have some answers to Bernhard's questions about UW's mailman. We are 
running a locally modified 2.1.19 which is the latest version. I asked about 
the modifications and was told they are not related to Precedence: bulk.

The email I received on 12 Apr 2011 when I first encountered this problem has 
the following in the body:
-----
Thank you for the additional information. That was useful in figuring out 
what the problem was. I went to the sendmail log for Mailman and this is what 
I found:

Apr 12 10:18:22 mailman2 sendmail[30654]: p3CHIMsU030654: from=<roundup-
admin@web.labmed.washington.edu>, auth=no, size=2004, class=-60, nrcpts=1, 
msgid=<1302628686.17.0.575133366818.issue2384@web.labmed.washington.edu>, 
proto=ESMTP, daemon=MTA, relay=mx2.cac.washington.edu [140.142.33.18]
bash-3.2$

The message is listed as class=-60 which is Precedence: Bulk and Mailman 
discards those. Unfortunately, there isn't any way for us to change that on 
this end so the sender will need to configure their mail program such that 
this gets changed. It looks like this sender sent several messages to Mailman 
lists today and they all have that Precedence: Bulk setting and so were 
discarded.
-----
I was told that there is a global configuration option to specify where or 
not to drop messages containing Precedence: bulk. Obviously for us, this flag 
is set. I couldn't determine where that is located.

Another mailman expert dug deeper to find out what is going on. Here is 
what he said:

  I need to correct myself. The bulk detection feature is a local
  modification to the UW Mailman service.

So that is why our mailman drops 'Precedence: bulk' emails while the un-
modified mailman doesn't.

Again folks, I would caution messing with the "Precedence: bulk" header
stuff for the mail-server killing reasons noted previously in this issue.

On Sat, 14 Mar 2015 at 08:21 Tom Ekberg <issues@roundup-tracker.org> wrote:

>
> Tom Ekberg added the comment:
>
> Another mailman expert dug deeper to find out what is going on. Here is
> what he said:
>
>   I need to correct myself. The bulk detection feature is a local
>   modification to the UW Mailman service.
>
> So that is why our mailman drops 'Precedence: bulk' emails while the un-
> modified mailman doesn't.
>
> ________________________________________________
> Roundup tracker <issues@roundup-tracker.org>
> <http://issues.roundup-tracker.org/issue2550564>
> ________________________________________________
>

Tom, Richard,

thanks for your remarks!

What I learn from it is:
a) Dropping emails with "Precedence: bulk" does not seem to be a general
problem with mailman, but with a local modification within the
University of Washington. So they could add a customer to strip this
header from emails that have the roundup header set and would be done.
So we probably look for other real world examples where setting
"Precedence: bulk" is a problem.

b) On the other hand, rfc2046 discourages setting "Precedence: bulk",
so it should not at all be necessary. Setting no return-path seems a
regular message, but I don't think we should use it outself. If roundup 
does not react to automatic emails we would be fine to prevent loops.
So here also the question is: In which real world situations would 
roundup emails without this header be a problem?

1) As per https://tools.ietf.org/html/rfc3834, the "Precedence: bulk"
header's SOLE purpose is to signal mailers that this is autogenerated
mail, so they shall not auto-reply to it. Full stop.

2) The same RFC recommends setting "Auto-Submitted: <anything but `no'>"
instead and doesn't recommend to use "Precedence:";
https://tools.ietf.org/html/rfc5436 updates the RFC, suggesting some
more specific values for the header.

3) Setting "Reply-To:<>" instead is a "hard" solution - no one will _be
able_ to reply (though Outlook/Exchange (2003) is reported to use
"From:" instead sometimes
(https://social.technet.microsoft.com/Forums/exchange/en-US/72e7608a-9ff1-43c3-b6dd-454dafc67672/replyto-address-is-not-used-bei-out-of-office-replies?forum=exchangesvrclientslegacy)),
another one is to set it (and possibly "From:") to a "black hole"
address like "noreply@<domain>".

4) But 3) doesn't look like an option for Roundup because one can reply
to threads by replying to notifications AFAIK.

5) Mailer support for "Auto-Submitted:": supported by Lotus Domino
(https://jira.atlassian.com/browse/JRA-15325), unsupported by Exchange
2007 (message 2011, linked blog post 2006) but "Precedence:" supported
(http://stackoverflow.com/a/7316397/648265), supported by GMail
(https://productforums.google.com/forum/#!msg/apps/INFoYJFBDrI/F9YCFjOrYB0J).
Other mailers I know of do not have auto-reply functionality out of the box.

---

C1) As per 1), the university server admins are completely wrong to drop
these unconditionally and shall be directed to the RFCs, this discussion
and other related references here.
C2) As per 5), we cannot drop "Precedence:" quite yet (unless someone
digs up evidence that the bulk of Exchange servers on duty - used by
tech people, at least - now support the new header)
C3) but as per 2), it may be good to complement it with
"Auto-submitted:" in accordance with the newer RFC.

Hello Ivan,

Stefan brought this issue up in 2009 and has probably found a way to
solve his problem outside Roundup. After reading your msg5532 carefully
Is still don't know what you are proposing:  Is it simply an addendum to
the research done by Tom last year and posted as:
http://issues.roundup-tracker.org/file1568/precedence-bulk.txt ?
If yes then I propose we should close this issue with 
Resolution "wont fix".

Best Regards, Peter Funk

> After reading your msg5532 carefully
> Is still don't know what you are proposing:  Is it simply an addendum to
> the research done by Tom last year and posted as:
> http://issues.roundup-tracker.org/file1568/precedence-bulk.txt ?
> If yes then I propose we should close this issue with
> Resolution "wont fix".
And I thought I was clear...
It is a more thorough research, and the conclusions are marked "C<n>)".

And they are that we cannot drop the header now and shouldn't be doing 
that to solve the original problem;
neither letting a user set it makes much sense.
So it's indeed WONTFIX.

So, by saying WONTFIX you are saying one of 2 things to me:
  (1) use some other tracker that doesn't put Precedence: bulk
      on email headers or has an option to remove it, or
  (2) continue to locally patch roundup to remove it so that
      our mailman (managed by another group) will pass our
      mail instead of silently dropping it.

I don't understand why there is resistance to adding a config option 
to not allowing this mail header attribute to be optional. Nobody is 
suggesting to drop Precedence entirely, only to make it a config 
option.

Hello Tom,

Tom Ekberg added the comment:
> So, by saying WONTFIX you are saying one of 2 things to me:
>   (1) use some other tracker that doesn't put Precedence: bulk
>       on email headers or has an option to remove it, or
>   (2) continue to locally patch roundup to remove it so that
>       our mailman (managed by another group) will pass our
>       mail instead of silently dropping it.
> 
> I don't understand why there is resistance to adding a config option 
> to not allowing this mail header attribute to be optional. Nobody is 
> suggesting to drop Precedence entirely, only to make it a config 
> option.

I understand that having to locally patch an upstream version of some 
free software package is a hassle which should be avoided.

So I'm not against such a configuration option.  I've only asked
whether I understood the points made by Ivan in msg5532 this morning.

Tom: Would you care to provide such a patch?

Best Regards,
Peter Funk
-- 
Peter Funk, home: ✉Oldenburger Str.86, D-27777 Ganderkesee
mobile:+49-179-640-8878 phone:+49-421-20419-0 <http://www.artcom-gmbh.de/>
office: ArtCom GmbH, ✉Haferwende 2, D-28357 Bremen, Germany

I sent an email to the group here at UW that manages mailman. This is the response 
I got:

The RFCs don't actually mandate the usage, value or actions for precedence ("the 
use of Precedence is not specifically recommended by this specification", or 
"Precedence: Non-standard, controversial, discouraged"), so we stick to what our 
experience tells us. Most emails with this tag are things such as "Out of office" 
or vacation messages. Accordingly we drop emails that have this tag to save our 
users from unwanted emails.

Am Dienstag, 19. April 2016 16:03:16 schrieb Tom Ekberg:
> I don't understand why there is resistance to adding a config option
> to not allowing this mail header attribute to be optional.

Just in general a config options should only be added if there is
a usecase for several people. Each config option will have to be maintained
in the future and thus multiplies complexity of the product.

I have not looked into the use case for "Precedence: bulk",
for me an important argument would be, if you can make a case about
other people having a need for this.

> C2) As per 5), we cannot drop "Precedence:" quite yet (unless someone
> digs up evidence that the bulk of Exchange servers on duty - used by
> tech people, at least - now support the new header)

Exchange 2010 is claimed to support RFC 3834 (i.e. likely honors the new
header):
https://technet.microsoft.com/en-US/library/ff848256(v=exchg.141).aspx .

According to
http://www.securityspace.com/s_survey/data/man.201603/mxsurvey.html ,
however, the vast majority of installations still use Exchange 2003 (see
http://capitalhead.com/articles/microsoft-exchange-server-build-numbers-and-release-dates.aspx
for build number reference. Note that the builds in the analysis are
smtpsvc.dll builds which still roughly correspond to those of Exchange).

Tom, we have your patch for:

  issue2551018: Add new note_filter parameter to nosymessage.

does that permit you to change the Precedence header? If so then maybe 
this needs to be a ticket about adding a new header:

  Auto-Submitted: by roundup issue tracker

as Ivan noted in https://tools.ietf.org/html/rfc3834

Thoughts?

-- rouilj

The note_filter parameter allows one to have a different body section for a nosy message. It could be modified to allow one to change message properties. That would only be used for the UW trackers. This is a small snippet of ansible code that comments Precedence bulk the line out:

  - name: comment out prececence bulk
    lineinfile:
      path: "{{ TRACKER_SRC }}/roundup/mailer.py"
      regexp: '^(\s*.*Precedence.*bulk.)$'
      backrefs: yes
      line: '#\1'

It gets applied after the roundup code is untarred on the tracker host. This way I can keep up with the roundup tip and still be able to have nosy emails not dropped by the UW changes to mailman.

In response to msg3809 I have made mailgw.py treat any Auto-Submitted 
header with a value other than no the same as we treat precedence bulk 
email. rev6247:ca0915457761

I would like to solve the original issue as well. I see two things we 
need:

1) by default add Auto-Submitted: auto-generated to outgoing mail
2) provide method(s) to set headers Auto-Submitted and Precedence to
   another value or remove the headers entirely.

I was considering in mailer.py creating:

  Mailer::outgoing_headers = { "Precedence": "bulk",
                                "Auto-Submitted": "auto-generated"
                             }

this will be used by set_message_attributes:

  for header, value in self.outgoing_headers:
     if value is not None:
        message[header] = value
     else:
        del(message[header])

to replace current code that sets Precedence to bulk.

Now the question is how does somebody set this?

Can it be set in the tracker's interfaces.py:

  from roundup.mailer import outgoing_headers
  outgoing_headers['Precedence'] = None

does this work?

Can it be set somehow in a detector so IssueClass::nosymessage
which calls IssueClass::send_message which calls 
Mailer::set_message_attributes does the RightThing(TM)?

Currently nosymessage includes add_headers and passes it to 
send_message. If we move the processing for add_headers after 
set_message_attributes is called in send_message and modify
processing to allow the value to be None to indicate deletion
of a header. I think we have a better way to modify the headers.

I kind of like using add_headers as well as outgoing_headers
for Mailer.

A user may want different settings for nosy messages as opposed to 
admin notification messages sent due to a roundup crash.

This allows those two cases to be handled separately.

Thoughts?

-- rouilj