Roundup Tracker - Issues

Issue 2551166

classification
Cross-Platform Imports do not work
Type: crash Severity: urgent
Components: Command-line interface Versions: 2.1.0
process
Status: closed wont fix
:
: : noschvie, rouilj
Priority: :

Created on 2021-11-04 11:30 by noschvie, last changed 2022-01-28 03:42 by rouilj.

Files
File name Uploaded Description Edit Remove
status.csv noschvie, 2021-11-04 11:30 status.csv export file
Messages
msg7352 Author: [hidden] (noschvie) Date: 2021-11-04 11:30
The attached status.csv file is part of an export done with Roundup 2.1.0 running under Windows. Such a dump can not be imported by Roundup running under Linux.

Analysed by Ralf Schlatterbeck:
Every line in your status.csv is terminated by *two* carriage returns and *one* linefeed. This is a variant of Microsoft windows newline conventions with another carriage return.
I don't know where the additional CR comes from.
So I guess the export/import should use the 'universal newline' feature of python to be able to read exports from windows. I guess nobody has ever tested if cross-platform export/import works.
The additional carriage return is probably ignored when evaluating the last column (which always contains the 'is retired' flag in all exports I've seen over the years).
msg7353 Author: [hidden] (rouilj) Date: 2021-11-04 13:59
Hi all: 

Norbert, glad you got the import working (in later message).
But a note here on the weird \r\r\n line ending.

    https://stackoverflow.com/questions/9845681/does-python-csv-writer-always-use-dos-end-of-line-characters#comment76482943_29976091

mentions:

    open('file.csv', 'wb')...

    For others who find this post, don't miss the 'wb' if you're still
    using Python 2. (In Python 3, this problem is handled by
    Python). You won't notice a problem if you're missing it on some
    platforms like GNU/Linux, but it is important to open the file in
    binary mode on platforms where that matters, like
    Windows. Otherwise, the csv file can end up with line endings like
    \r\r\n. If you use the 'wb' and os.linesep, your line endings
    should be correct on all platforms.

So this looks like a combination of:

  Use of python 2 on windows (and missing the 'wb' option)

and 

  import on linux

Norbert, what version of python is used on the turnkey appliance?

I wonder if setting line ending to \n for csv.reader and csv.writer
will work for all combinations of:

  windows, linux
  export python 2, 3
  import python 2, 3

At the very least \r\r\n shouldn't get created as that requires using
dos2unix twice on the file to convert it fully.

Also Universal newline support (mentioned by Ralf as a possible solution)
caused issues in:

  https://issues.roundup-tracker.org/issue1163890

and was removed in changeset:   3283:4fe232899be8 in the 0.8 timeframe.
It may be fixed by now? Still universal line endings won't handle \r\r\n.

(Note I am including the tracker in this email so we get it on ticket.)
msg7354 Author: [hidden] (noschvie) Date: 2021-11-04 14:30
The Turnkey appliance seems to use Python 3.7 because of the error messge, but both are installed, 2.7 and 3.7

Importing issue - 1Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/roundup/admin.py", line 1702, in run_command
    ret = function(args[1:])
  File "/usr/local/lib/python3.7/dist-packages/roundup/admin.py", line 1448, in do_import
    nodeid = cl.import_list(file_props, r)
  File "/usr/local/lib/python3.7/dist-packages/roundup/backends/rdbms_common.py", line 3156, in import_list
    newid = eval_import(proplist[propnames.index("id")])
IndexError: list index out of range

root@roundup ~# dir /usr/local/lib
python2.7  python3.7

root@roundup ~# python --version
Python 2.7.16
msg7444 Author: [hidden] (rouilj) Date: 2022-01-28 03:42
I am going to close this out. It seems like an edge case in multiple ways:

  python 2 -> 3
  windows -> linux

We have a workaround by editing the csv file so this is annoying but not a blocker.
History
Date User Action Args
2022-01-28 03:42:56rouiljsetstatus: new -> closed
resolution: wont fix
messages: + msg7444
2021-11-04 14:30:02noschviesetmessages: + msg7354
2021-11-04 13:59:15rouiljsetnosy: + rouilj
messages: + msg7353
2021-11-04 11:31:48noschviesettype: crash
2021-11-04 11:30:51noschviecreate