Roundup Tracker - Issues

Issue 2550671

classification
ProgrammingError: ERROR: could not serialize access due to concurrent
Type: crash Severity: normal
Components: Versions: 1.4
process
Status: closed fixed
:
: : ThomasAH, ber, kowey, rouilj, schlatterbeck, stefan
Priority: :

Created on 2010-10-15 13:56 by kowey, last changed 2018-06-07 09:52 by schlatterbeck.

Files
File name Uploaded Description Edit Remove
auto-generated-email kowey, 2010-10-15 13:56
Messages
msg4146 Author: [hidden] (kowey) Date: 2010-10-15 13:56
We have some software which sits behind our version control repository
and which notifies roundup via the email gateway when certain kinds of
commits have been made.  It is frequently the case that several of these
emails are sent out in quick succession (as this process is automated)

Recently, we've been getting error messages of this sort:
An unexpected error occurred during the processing                     
                                                                     
of your message. The tracker administrator is being                    
                                                                     
notified.                                                              
                                                                     
                                                                       
                                                                     
----------------                                                       
                                                                     
Traceback (most recent call last):                                     
                                                                     
  File "/usr/lib/pymodules/python2.5/roundup/mailgw.py", line 738, in
handle_Message                                                         
    return self.handle_message(message)                                
                                                                     
  File "/usr/lib/pymodules/python2.5/roundup/mailgw.py", line 794, in
handle_message                                                         
    return self._handle_message (message)                              
                                                                     
  File "/usr/lib/pymodules/python2.5/roundup/mailgw.py", line 1369, in
_handle_message                                                       
    cl.set(nodeid, **props)                                            
                                                                     
  File "/usr/lib/pymodules/python2.5/roundup/backends/rdbms_common.py",
line 1606, in set                                                    
    propvalues = self.set_inner(nodeid, **propvalues)                  
                                                                     
  File "/usr/lib/pymodules/python2.5/roundup/backends/rdbms_common.py",
line 1816, in set_inner                                              
    self.db.setnode(self.classname, nodeid, propvalues,
multilink_changes)                                                     
             
  File "/usr/lib/pymodules/python2.5/roundup/backends/rdbms_common.py",
line 933, in setnode                                                 
    self.sql(sql, vals)                                                
                                                                     
  File "/usr/lib/pymodules/python2.5/roundup/backends/rdbms_common.py",
line 159, in sql                                                     
    self.cursor.execute(sql, args)                                     
                                                                     
ProgrammingError: ERROR:  could not serialize access due to concurrent
update                                                                
                                                                       
                                                                     
update _issue set _activity='2010-10-15
13:14:44.653',_actor=1837,_resolvedin=2,_status=8 where id=1942 

Strangely it seems that the change we want to make takes effect anyway.

See also http://bugs.darcs.net and the attached mail that our
auto-updater tries to send
msg4147 Author: [hidden] (kowey) Date: 2010-10-15 13:59
Also,$ roundup-admin -v
1.4.13 (python 2.5.2)

This is running on Debian 5.0.3 (Debian stable at the time of this
writing, with roundup installed via an unstable package)
msg4155 Author: [hidden] (ber) Date: 2010-10-18 10:17
Which database backend are you using?
Does the error message occurr each time? (if not, what is the 
frequency?) Does it depend on the message?
msg4168 Author: [hidden] (kowey) Date: 2010-10-19 21:42
It's Postgres.  The conditions we'd need to trigger the bug don't happen
very often (so I can't really provide a useful answer about frequency).
I think it's happened in two batches, with each batch giving me 3 or 4
of these at a time.

I can keep an eye out for this in the future.

The script that generates these updates lives here:
http://darcs.net/contrib/update_roundup.pl
msg4169 Author: [hidden] (stefan) Date: 2010-10-20 03:52
What I find troubling in this report is that I'd expect each mailgw
session to open up a new session, such that multiple sessions are
independent transactions, which can't interfere with each other.

The tracelog however seems to suggest that the internal cursor variable
gets corrupted due to multiple simultaneous sessions.
msg4466 Author: [hidden] (schlatterbeck) Date: 2012-01-05 14:34
I'm also seeing this problem in a production tracker. The reason is that
the cookies are manipulated with the same database cursor as the
production data. I'll have to dig up another issue where this has been
discussed in depth (including a patch as far as I remember).
msg5704 Author: [hidden] (rouilj) Date: 2016-07-02 03:55
Ralf were you thinking of issue1703116?
msg5714 Author: [hidden] (schlatterbeck) Date: 2016-07-02 20:08
On Sat, Jul 02, 2016 at 03:55:05AM +0000, John Rouillard wrote:
> 
> Ralf were you thinking of issue1703116?

Yes, thanks for digging it up.
Especially the linked discussion on the list was what I remembered.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com
msg5771 Author: [hidden] (rouilj) Date: 2016-07-07 00:20
I wonder if this was affected by the code that I discussed in

https://sourceforge.net/p/roundup/mailman/message/35198191/

that was patched in issue2550853. It affects the handling of serial retries.

-- rouilj
msg6080 Author: [hidden] (rouilj) Date: 2018-06-06 23:33
in msg4466 Ralf said:

   The reason is that the cookies are manipulated with the
   same database cursor as the production data.

with the resolution of issue2550955 that separates the production 
database cursor from the OTK/cookie cursor, is this problem also solved?
msg6083 Author: [hidden] (schlatterbeck) Date: 2018-06-07 09:52
This is probably fixed with the latest changes how OTK and session
management are handled with a separate cursor.

Another reason for concurrency problems might have been a mail setup
that runs the mailgw inside the delivery system of the MTA, so that
concurrent MTA invocations will produce concurrent roundup mailgw
invocations. This is generally not a recommended setup for several reasons:

- concurrency issues if several mails referring to the same roundup
  issue are sent in close succession (as is the case in this issue)
- it is hard to disable mta delivery during updates or maintenance of
  the roundup tracker

the recommended setup is a mailbox where the MTA delivers mail and a
separate job that fetches mail from this mailbox via POP or IMAP
protocols. In that case delivery in the MTA and processing of the mails
by the roundup mailgw are decoupled, no concurrency issues will arise
and it is easier to upgrade or disable the mailgw during maintenance
periods.
History
Date User Action Args
2018-06-07 09:52:50schlatterbecksetstatus: new -> closed
resolution: fixed
messages: + msg6083
2018-06-06 23:37:55rouiljsettype: crash
2018-06-06 23:33:04rouiljsetmessages: + msg6080
2016-07-07 00:20:52rouiljsetmessages: + msg5771
2016-07-02 20:08:20schlatterbecksetmessages: + msg5714
title: ProgrammingError: ERROR: could not serialize access due to concurrent -> ProgrammingError: ERROR: could not serialize access due to concurrent
2016-07-02 03:55:05rouiljsetnosy: + rouilj
messages: + msg5704
2014-05-21 14:03:36ThomasAHsetnosy: + ThomasAH
2012-01-05 14:34:02schlatterbecksetnosy: + schlatterbeck
messages: + msg4466
2010-10-20 03:52:37stefansetnosy: + stefan
messages: + msg4169
2010-10-19 21:42:56koweysetmessages: + msg4168
2010-10-18 10:17:05bersetnosy: + ber
messages: + msg4155
2010-10-15 13:59:41koweysetmessages: + msg4147
2010-10-15 13:56:55koweysetversions: + 1.4
2010-10-15 13:56:31koweycreate