Roundup Tracker - Issues

Issue 2550743

classification
Title: Reindex with MySQL Server failed
Type: crash Severity: normal
Components: Database Versions: 1.4
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: rouilj, telsch
Priority: Keywords:

Created on 2012-01-30 15:43 by telsch, last changed 2016-07-09 18:26 by rouilj.

Messages
msg4492 Author: [hidden] (telsch) Date: 2012-01-30 15:43
When we try to reindex our roundup instance we get the error 'MySql
Server has gone away (2006)'
We found a quick fix, if we set in the configuration from mysql:
    /etc/my.conf
    [mysqld]
    max_allowed_packet = 1M

the 'max_allowed_packet' value from '1M' to '64M'

i hope it is possible to reduce the transaction size, so we don't have
to configure the mysql for the roundup reindex.
msg5794 Author: [hidden] (rouilj) Date: 2016-07-09 18:26
Sorry this has taken so long to address.

Do you have large attached files that could be indexed?

In looking through the code (specifically
backends/indexer_rdbms.py::add_text, I can see getting
this error if you are indexing attachment contents.

It looks like it tries to insert all words in a document at a
single time using:

        # for each word, add an entry in the db
        sql = 'insert into __words (_word, _textid) values (%s, %s)'%(a, a)
        words = [(word, id) for word in words]
        self.db.cursor.executemany(sql, words)

While that could be split up into single execute commands or
smaller batches of X number of words, performance would suffer
as the python mysql driver generates a large insert with values
command to the server if I understand:

 https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-executemany.html

correctly.

We may accept a patch to make the number of inserts
configurable using a setting in the [rdbms] section of the
tracker config file. So if you develop one, please feel free
to open an issue and attach the patch. The default should
be an unlimited number of inserts. 

If you can't increase the packet size, maybe some other search
engine (in 1.6.0 we support whoosh as well as xapian) would work.

I have added this problem to the documentation in mysql.txt under the
header: Other Configuration so others can benefit from your knowledge.

Thanks for the report and sorry this took so long.

-- rouilj
History
Date User Action Args
2016-07-09 18:26:33rouiljsetstatus: new -> closed
resolution: wont fix
messages: + msg5794
nosy: + rouilj
2012-01-31 10:03:57telschsettype: crash
2012-01-30 15:43:25telschcreate