Roundup Tracker - Issues

Issue 2550975

Title: Verify Python 3 compatibility for existing databases
Type: rfe Severity: normal
Components: Infrastructure Versions: devel
Status: fixed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: joseph_myers, rouilj
Priority: Keywords: python3

Created on 2018-07-25 19:40 by joseph_myers, last changed 2018-09-16 16:25 by joseph_myers.

msg6131 Author: [hidden] (joseph_myers) Date: 2018-07-25 19:40
We need to make sure that an existing instance, set up with Python 2,
including UTF-8 data in the database not just plain ASCII, works
correctly when used with Python 3 - that the database content format is
properly compatible across Python versions - with every database back
end (the reverse, creating with Python 3 and using with Python 2, may be
less important, but is still desirable - if someone has problems
updating their instance to Python 3, they should be able to move back to
Python 2 without the stored data being incompatible).  anydbm may have
the highest risk of problems here, as it uses the Python marshal module
in storing data.
msg6208 Author: [hidden] (rouilj) Date: 2018-08-19 00:26
Question would doing a export/import to handle the upgrade from a
python2 to python3 be acceptable or doable?

My thought is upgrading from python2 to python3 could be handled like we
do a database change. Export the tracker using the python 2 instance,
then import the data into a python3 based tracker.

For large trackers this may not be a good mechanism, but for testing
purposes I think it should work.

-- rouilj
msg6209 Author: [hidden] (joseph_myers) Date: 2018-08-19 00:31
I'd hope that for the RDBMS backends, the database content is already 
compatible in fact, but it needs to be tested (with non-ASCII characters).

For anydbm problems are likely and it might be necessary to use 
export/import.  For that we need to fix issue 2550976 (make Roundup 
running under Python 3 read and write CSV files compatible with those used 
by Roundup running under Python 2, because they aren't compatible for 
non-ASCII characters in strings at present).
msg6231 Author: [hidden] (joseph_myers) Date: 2018-09-02 23:55
My fix for CSV import/export compatibility also includes some changes to
make the journal representation in RDBMS databases with Python 3
compatible with that used with Python 2.  Given that, some limited
testing indicates sqlite databases are compatible between Python
versions (including with non-ASCII characters in the main data and the
journal), *if* you delete db/otks and db/sessions (anydbm files) when
changing Python version.  anydbm databases are not compatible and
probably can't be compatible.

It's plausible that, given those changes, mysql and postgresql databases
are also compatible (including with non-ASCII characters), but that
still needs to be tested.
msg6250 Author: [hidden] (joseph_myers) Date: 2018-09-16 16:25
For postgresql, existing Python 2 databases work fine with Python 3.

For mysql, they don't.  I've committed a patch to ensure both database
and connection encoding are set to UTF-8 with mysql with Python 3, to
allow non-ASCII characters to work properly in that case.  But existing
Python 2 databases use whatever the default MySQL encoding is, but store
UTF-8 bytes incorrectly labelled as being in that encoding (if for any
reason the default database encoding at the time of creation differs
from the default connection encoding, things would be worse that than).
 So I've documented in upgrading.txt that for mysql you need an
export/import to move to Python 3 (though it would be good to have a
more efficient way of changing the recorded encoding of the data in the
database without actually carrying out a character set conversion it, so
as not to need such an export/import).
Date User Action Args
2018-09-16 16:25:06joseph_myerssetstatus: new -> fixed
resolution: fixed
messages: + msg6250
2018-09-02 23:55:46joseph_myerssetmessages: + msg6231
2018-08-19 00:31:55joseph_myerssetmessages: + msg6209
2018-08-19 00:26:01rouiljsetnosy: + rouilj
messages: + msg6208
2018-07-26 10:42:00joseph_myerssetkeywords: + python3
2018-07-25 19:40:48joseph_myerscreate