Roundup Tracker - Issues

Issue 2551025

Title: Fix travis-ci failures for mysql tests
Type: crash Severity: normal
Components: Database Versions: devel
Status: fixed Resolution: fixed
Dependencies: Superseder:
Assigned To: rouilj Nosy List: ber, jerrykan, rouilj, schlatterbeck
Priority: high Keywords: Blocker

Created on 2019-02-21 00:37 by rouilj, last changed 2019-12-22 01:01 by rouilj.

msg6361 Author: [hidden] (rouilj) Date: 2019-02-21 00:37
Hi Ralf:

In message <>,
Ralf Schlatterbeck writes:
>On Tue, Feb 19, 2019 at 08:45:09PM -0500, John P. Rouillard wrote:
>> Does anybody know how to fix the Travis CI errors with mysql?
>> Jerrykan/John K do you have any ideas? It would be good to release the
>> current head as roundup 2.0.a1 once it passes CI to pypi.
>Do we have an issue for this, I have just looked and haven't found one.

We should as soon as this email hits the tracker 8-). I removed the
roundup-devel mailing list as the creation of the ticket will be
forwarded there and I didn't want to chance having the ML on the nosy

>That said: I looked into the travis errors and investigated this and had
>written some emails about it so I would have to dig this out of my mail
>The failing tests seem to be a configuration issue with mysql that
>should be fixable by changing the mysql config.

Yeah, I think the errors were happening during tear down. A quick
search indicated that the error could be caused by an oversized
packet. So the question is why did it all of sudden start, and how do
you change it in travis?

Have a great week.
msg6362 Author: [hidden] (ber) Date: 2019-02-25 11:04
In order to be able to help it would be nice to have links to the
current findings (ci, and Ralf's previous mails).
msg6383 Author: [hidden] (rouilj) Date: 2019-03-10 16:14
I was unable to find Ralf's emails in the list other than one note that
it should be fixable by changing mysql's my.cnf.

Looking at doc/mysql.txt, a fix for this is condition to change the
max_allowed_packet size setting. See also:

which indicates it could also be an out of memory condition, connection
limit or a timeout being hit.
The interactive_timeout and wait_timeout values are by default: 28800
seconds or 8 hours and the entire run
takes only 20 minutes. There are no settings for these vars in my.cnf
that would reduce the value.
Max_connections is 151 by default, again I don't think this is an issue.

I tried to edit /etc/mysql/my.cnf change that to 500M (from 16M) and
restart the server. This seems to have been done correctly according to
the log:

see lines: 5851 and 5927

but the 14 errors are still there.

Looking at the errors, some look like an example of issue2550743 as
these failing tests are indexer based:

test/ FAILED             
 [ 28%]
test/ FAILED      
 [ 28%]
test/ FAILED             
 [ 28%]
test/ FAILED              
 [ 28%]
test/ FAILED       
 [ 28%]
test/ FAILED        
 [ 28%]
test/ FAILED          
 [ 28%]
test/ FAILED      
 [ 28%]

but these failing tests/errors:

test/ FAILED               
 [ 65%]
test/ FAILED                
 [ 65%]
test/ FAILED                  
 [ 66%]
test/ FAILED            
 [ 66%]
test/ FAILED         
 [ 66%]
test/ FAILED   
 [ 66%]
test/ ERROR    
 [ 66%]

aren't AFAIK indexer based.

The errors always come from:

    def sql_close(self):
>           self.conn.close()
E           OperationalError: (2006, '')
roundup/backends/ OperationalError

So I am stumped.
msg6433 Author: [hidden] (rouilj) Date: 2019-03-25 22:57
I rolled back the mysqlclient installed by pip from 1.3.14 (breaks test) 
to 1.3.13 (tests pass).

I don't know what in 1.3.14 causes mysql to exit with a 2006 error on 
close. Nothing in the changelog stood out as a possible cause/change.
msg6737 Author: [hidden] (rouilj) Date: 2019-10-13 01:46
I reverted the pin to version 1.3.13 and we still have the same 
failure. Plus some new rest erro case failures. I have reinstalled
the pin.

I thought we may be double closing the connection, but by attempt to 
ignore the second close call if the handle was already dead didn;t make 
a difference.

I submitted this question to stack overflow to see if we can get some 

Using the github tracker for this sort of question seems unwanted by 
the developer.

If anybody has an idea here I would appreciate it.
msg6738 Author: [hidden] (rouilj) Date: 2019-10-13 17:19
Hmm, Using:

it looks like  a new case was added that results in 
CR_SERVER_GONE_ERROR which is what error 2006 is IIUC.

I wonder if I was on the right path and we are leaking a double close 
of the database through.

Looking at the sql_close code, it looks like ProgrammingError used to 
be raised with a connection closed message.

So I am trying to emulate the same for OperationalError with:

+        # issue2551025: with revision 1.3.14 of mysqlclient.                   
+        # It looks like you can get an OperationalError 2006                   
+        # raised for closing a closed handle.                                  
+        except MySQLdb.OperationalError as message:                            
+            print str(message)                                                 
+            if str(message) != '':                                             
+                raise                                                          
         except MySQLdb.ProgrammingError as message:                            
             if str(message) != 'closing a closed connection':                  

I hope this will allow me to figure out what message to filter
on and suppress the exception.

msg6750 Author: [hidden] (rouilj) Date: 2019-10-17 02:12
mysql tests are passing with mysqlclient 1.4.4

I ended up using:
   except MySQLdb.OperationalError as message:
       if str(message) != "(2006, '')":  # close connection                

Not sure if this is 100% correct but it appears to do the trick.

Marking issue pending. Anybody got feedback?

-- rouilj
msg6845 Author: [hidden] (rouilj) Date: 2019-12-22 01:01
No feedback on my patch. Closing as fixed.
Date User Action Args
2019-12-22 01:01:23rouiljsetstatus: pending -> fixed
resolution: fixed
messages: + msg6845
2019-10-17 02:12:18rouiljsetstatus: open -> pending
messages: + msg6750
2019-10-13 17:19:36rouiljsetassignee: rouilj
messages: + msg6738
2019-10-13 01:46:03rouiljsetkeywords: + Blocker
status: new -> open
messages: + msg6737
2019-03-25 22:57:12rouiljsetmessages: + msg6433
2019-03-10 16:14:00rouiljsetnosy: + jerrykan
messages: + msg6383
2019-02-25 11:04:50bersetnosy: + ber
messages: + msg6362
2019-02-21 00:55:13rouiljsetpriority: high
title: [Roundup-devel] Does anybody have release plans for current roundup devel? -> Fix travis-ci failures for mysql tests
type: crash
components: + Database
versions: + devel
2019-02-21 00:37:10rouiljcreate