Hi,

we're using roundup as issue tracker and exchange as email server.

We have a problem related to an issue with both:
- me and,
- a mailing list I'm a member of
in the nosy list.

When I do an update of the issue, emails get send to me and the mailing
list (due to configuration option NOSY_EMAIL_SENDING == single), with
each email containing only a single addressee.

when the 2 messages arrive, one stays in my inbox, the other one is
filtered to the mailing list folder.

The problem is that sometimes only one message arrives (any of the two).
I've tracked this down to the following:
when a message arrives at the exchange server, and that message has the
same Message-id and the same Date field as an earlier message, the
second message is dropped (documented here:
http://technet.microsoft.com/en-us/library/dd577073%28v=exchg.80%29.aspx#1
).

When roundup sends the 2 emails, it sends it with the same message-id,
but not necessarily with the same Date.

So if the Date field is different, I receive 2 emails. If the Date field
is the same I receive 1 email (the first one to arrive).

From my user perspective, this is bad: I expect the email addressed to
me to appear in my inbox.

I looked at the sources for roundup-1.4.19 and I think the behaviour is
the same there. This piece of code is of interest:
roundupdb.py:send_message:
...
        # send an individual message per recipient?
        if self.db.config.NOSY_EMAIL_SENDING != 'single':
            sendto = [[address] for address in sendto]
        else:
            sendto = [sendto]

        # tracker sender info
        tracker_name = unicode(self.db.config.TRACKER_NAME, 'utf-8')
        tracker_name = nice_sender_header(tracker_name, from_address,
            charset)

        # now send one or more messages
        # TODO: I believe we have to create a new message each time as we
        # can't fiddle the recipients in the message ... worth testing
        # and/or fixing some day
        first = True
        for sendto in sendto:
            # create the message
            mailer = Mailer(self.db.config)

            message = mailer.get_standard_message(sendto, subject, author,
                multipart=message_files)

            # set reply-to to the tracker
            message['Reply-To'] = tracker_name

            # message ids
            if messageid:
                message['Message-Id'] = messageid
            if inreplyto:
                message['In-Reply-To'] = inreplyto
...
In the single mode (each in the cc-list an individual message), we call
get_standard_message for each recipient, which creates a Date field with
current time. We then stamp the same message-id on each email, but don't
change the date field.

If we force the Date to be the same for all messages, I'll never get two
messages, only one (timing-dependent which one).

If we force the Date to be different for all messages, I always get two
messages, but it's a not so pretty workaround for the specifics of
exchange duplicate message detection, and may not work for other similar
duplicate elimination programs.

I think the cleanest fix would be to have a different message-id for
each message. But, I'm not sure about the drawbacks of doing that.

What I understand from the standard (http://tools.ietf.org/html/rfc5322)
though is that same or different message-id depends on the intent of the
sender:
...
In all cases, it is the meaning that the sender of the message wishes to
convey (i.e., whether this is the same message or a different message)
that determines whether or not the "Message-ID:" field changes, not any
particular syntactic difference that appears (or does not appear) in the
message.
...
One could argue that the 2 emails messages are different because each
has a different addressee.

Thanks,
- Tom

> In all cases, it is the meaning that the sender of the message wishes
> to convey (i.e., whether this is the same message or a different
> message) that determines whether or not the "Message-ID:" field
> changes, not any particular syntactic difference that appears (or
> does not appear) in the message.
>
> One could argue that the 2 emails messages are different because each
> has a different addressee.

From that argumentation I tend to make this configurable in the
config.ini file for the tracker. Because if I change this others will
probably argue that they only want a single message id. What do you think?

> From that argumentation I tend to make this configurable in the
> config.ini file for the tracker. Because if I change this others will
> probably argue that they only want a single message id. What do you
> think?

I agree, that's probably a good idea.

In message
<1333106220.03.0.265807397089.issue2550752@psf.upfronthosting.co.za> ,
Tom de Vries writes:

>New submission from Tom de Vries <tom@codesourcery.com>:
>we're using roundup as issue tracker and exchange as email server.
>
>We have a problem related to an issue with both:
>- me and,
>- a mailing list I'm a member of in the nosy list.
>
>When I do an update of the issue, emails get send to me and the mailing
>list (due to configuration option NOSY_EMAIL_SENDING == single), with
>each email containing only a single addressee.
>
>when the 2 messages arrive, one stays in my inbox, the other one is
>filtered to the mailing list folder.
>
>The problem is that sometimes only one message arrives (any of the two).
>I've tracked this down to the following:
>when a message arrives at the exchange server, and that message has the
>same Message-id and the same Date field as an earlier message, the
>second message is dropped (documented here:
>http://technet.microsoft.com/en-us/library/dd577073%28v=exchg.80%29.aspx#1
>).
>
>When roundup sends the 2 emails, it sends it with the same message-id,
>but not necessarily with the same Date.
>
>So if the Date field is different, I receive 2 emails. If the Date field
>is the same I receive 1 email (the first one to arrive).
>
>From my user perspective, this is bad: I expect the email addressed to
>me to appear in my inbox.
>
> [...]
>If we force the Date to be the same for all messages, I'll never get two
>messages, only one (timing-dependent which one).
>
>If we force the Date to be different for all messages, I always get two
>messages, but it's a not so pretty workaround for the specifics of
>exchange duplicate message detection, and may not work for other similar
>duplicate elimination programs.

True, but I think changing the message-id will be more difficult. It
should definatly be an option regardless of which solution is adopted.

It's really stupid that exchange can't turn that off. I have had
duplicate detection in my email chain for a couple of decades, and
it's only considered a duplicate if it's the same message (messageid)
handled the same way (i.e. is delivered to the same mailbox/folder).

>I think the cleanest fix would be to have a different message-id for
>each message. But, I'm not sure about the drawbacks of doing that.

Well one of the issues is that clients use the In-Reply-To: <msgid>
header for threading in a client.

The comment in:

   ./roundup-1.4.19/roundup/mailgw.py:
        # try in-reply-to to match the message if there's no nodeid

indicates that it does compare the in-reply-to (whose value is the
message id of the message being responded to) and expects a message id
matching what it sent. I am not sure if there is support for a many->1
message-id -> actual message mapping which would be needed if a
message resulted in multiple message-id's.

>What I understand from the standard (http://tools.ietf.org/html/rfc5322)
>though is that same or different message-id depends on the intent of the
>sender:
>...
>In all cases, it is the meaning that the sender of the message wishes to
>convey (i.e., whether this is the same message or a different message)
>that determines whether or not the "Message-ID:" field changes, not any
>particular syntactic difference that appears (or does not appear) in the
>message.
>...
>One could argue that the 2 emails messages are different because each
>has a different addressee.

Well I would claim the message is a response to a single event:

  the roundup update

It really is a single message going to multiple people.  Roundup is
just changing the envelope addressing and not really the message.

On 30/03/12 21:59, John Rouillard wrote:
> 
> John Rouillard <rouilj@users.sourceforge.net> added the comment:
> 
> In message
> <1333106220.03.0.265807397089.issue2550752@psf.upfronthosting.co.za> ,
> Tom de Vries writes:
> 
>> New submission from Tom de Vries <tom@codesourcery.com>:
>> we're using roundup as issue tracker and exchange as email server.
>>
>> We have a problem related to an issue with both:
>> - me and,
>> - a mailing list I'm a member of in the nosy list.
>>
>> When I do an update of the issue, emails get send to me and the mailing
>> list (due to configuration option NOSY_EMAIL_SENDING == single), with
>> each email containing only a single addressee.
>>
>> when the 2 messages arrive, one stays in my inbox, the other one is
>> filtered to the mailing list folder.
>>
>> The problem is that sometimes only one message arrives (any of the two).
>> I've tracked this down to the following:
>> when a message arrives at the exchange server, and that message has the
>> same Message-id and the same Date field as an earlier message, the
>> second message is dropped (documented here:
>>
http://technet.microsoft.com/en-us/library/dd577073%28v=exchg.80%29.aspx#1
>> ).
>>
>> When roundup sends the 2 emails, it sends it with the same message-id,
>> but not necessarily with the same Date.
>>
>> So if the Date field is different, I receive 2 emails. If the Date field
>> is the same I receive 1 email (the first one to arrive).
>>
>>From my user perspective, this is bad: I expect the email addressed to
>> me to appear in my inbox.
>>
>> [...]
>> If we force the Date to be the same for all messages, I'll never get two
>> messages, only one (timing-dependent which one).
>>
>> If we force the Date to be different for all messages, I always get two
>> messages, but it's a not so pretty workaround for the specifics of
>> exchange duplicate message detection, and may not work for other similar
>> duplicate elimination programs.
> 
> True, but I think changing the message-id will be more difficult. It
> should definatly be an option regardless of which solution is adopted.
> 
> It's really stupid that exchange can't turn that off. I have had
> duplicate detection in my email chain for a couple of decades, and
> it's only considered a duplicate if it's the same message (messageid)
> handled the same way (i.e. is delivered to the same mailbox/folder).
> 
>> I think the cleanest fix would be to have a different message-id for
>> each message. But, I'm not sure about the drawbacks of doing that.
> 
> Well one of the issues is that clients use the In-Reply-To: <msgid>
> header for threading in a client.
> 
> The comment in:
> 
>    ./roundup-1.4.19/roundup/mailgw.py:
>         # try in-reply-to to match the message if there's no nodeid
> 
> indicates that it does compare the in-reply-to (whose value is the
> message id of the message being responded to) and expects a message id
> matching what it sent. I am not sure if there is support for a many->1
> message-id -> actual message mapping which would be needed if a
> message resulted in multiple message-id's.
> 

Hi,

Ah, thanks for pointing that out, I never noticed roundup does threading
of messages.

I've considered the following options to still make the threading work:
- Use the 'References:' field to pass the original message-id. Could be a
  solution, but I'm not sure if all mail clients will pass that on reply.
- Use an x-header such as x-round-message-id, but my own mail client 
  doesn't pass x-headers. And it wouldn't work for threading on the
  mail client, only roundup would recognize this field.
- add a serial number (unique for each nosy list member) to the message-id
  when sending mail, and remove it when receiving mail. Works for threading
  in mail client and in roundup, but doesn't thread messages when one
  person on the nosy list replies with cc to another person in the nosy
list.

Any preferences or better ideas, anyone?

Given that changing the message-id will most likely will break threading in 
one way or another, the option of forcing the Date to be different to work 
around the specifics of the exchange duplicate message detection starts to
sound less awkward.

>> What I understand from the standard (http://tools.ietf.org/html/rfc5322)
>> though is that same or different message-id depends on the intent of the
>> sender:
>> ...
>> In all cases, it is the meaning that the sender of the message wishes to
>> convey (i.e., whether this is the same message or a different message)
>> that determines whether or not the "Message-ID:" field changes, not any
>> particular syntactic difference that appears (or does not appear) in the
>> message.
>> ...
>> One could argue that the 2 emails messages are different because each
>> has a different addressee.
> 
> Well I would claim the message is a response to a single event:
> 
>   the roundup update
> 
> It really is a single message going to multiple people.  Roundup is
> just changing the envelope addressing and not really the message.

I think one can argue one way or the other on whether these 2 messages
are the
same or not.
But to respond to your argumentation, the difference in the 2 messages
is not
limited to the envelope. The addressees inside the envelope are also
different,
which is why my feeling is that they're different.
If the addressees inside the envelope were the same, I would have the
guarantee
to receive a message with me as TO or CC, and my filters would keep it
in my inbox,
and the biggest part of my problem would be fixed.

Thanks,
- Tom

> 
> ----------
> nosy: +rouilj
> 
> ________________________________________________
> Roundup tracker <issues@roundup-tracker.org>
> <http://issues.roundup-tracker.org/issue2550752>
> ________________________________________________

NOSY_EMAIL_SENDING has the following interpretation:
...
email_sending – single
Controls the email sending from the nosy reactor. If multiple then a
separate email is sent to each recipient. If single then a single email
is sent with each recipient as a CC address.
...

I got confused with single addressee vs single email.
So the problem I'm having is related to NOSY_EMAIL_SENDING == multiple.

Corrected title.

I wonder if using some form of bcc addressing option would work?

SO we have single (send single email), bcc (single email but everybody
is on bcc list, so can't see each other), multiple.

But yes fuzzing a unique date seems to be a solution.

Tom, is this still a problem? If so, I'll try poking a couple of
exchange admins at work to see if they know of a workaround.