Issue 2551166
Created on 2021-11-04 11:30 by noschvie, last changed 2022-01-28 03:42 by rouilj.
File name |
Uploaded |
Description |
Edit |
Remove |
status.csv
|
noschvie,
2021-11-04 11:30
|
status.csv export file |
|
|
msg7352 |
Author: [hidden] (noschvie) |
Date: 2021-11-04 11:30 |
|
The attached status.csv file is part of an export done with Roundup 2.1.0 running under Windows. Such a dump can not be imported by Roundup running under Linux.
Analysed by Ralf Schlatterbeck:
Every line in your status.csv is terminated by *two* carriage returns and *one* linefeed. This is a variant of Microsoft windows newline conventions with another carriage return.
I don't know where the additional CR comes from.
So I guess the export/import should use the 'universal newline' feature of python to be able to read exports from windows. I guess nobody has ever tested if cross-platform export/import works.
The additional carriage return is probably ignored when evaluating the last column (which always contains the 'is retired' flag in all exports I've seen over the years).
|
msg7353 |
Author: [hidden] (rouilj) |
Date: 2021-11-04 13:59 |
|
Hi all:
Norbert, glad you got the import working (in later message).
But a note here on the weird \r\r\n line ending.
https://stackoverflow.com/questions/9845681/does-python-csv-writer-always-use-dos-end-of-line-characters#comment76482943_29976091
mentions:
open('file.csv', 'wb')...
For others who find this post, don't miss the 'wb' if you're still
using Python 2. (In Python 3, this problem is handled by
Python). You won't notice a problem if you're missing it on some
platforms like GNU/Linux, but it is important to open the file in
binary mode on platforms where that matters, like
Windows. Otherwise, the csv file can end up with line endings like
\r\r\n. If you use the 'wb' and os.linesep, your line endings
should be correct on all platforms.
So this looks like a combination of:
Use of python 2 on windows (and missing the 'wb' option)
and
import on linux
Norbert, what version of python is used on the turnkey appliance?
I wonder if setting line ending to \n for csv.reader and csv.writer
will work for all combinations of:
windows, linux
export python 2, 3
import python 2, 3
At the very least \r\r\n shouldn't get created as that requires using
dos2unix twice on the file to convert it fully.
Also Universal newline support (mentioned by Ralf as a possible solution)
caused issues in:
https://issues.roundup-tracker.org/issue1163890
and was removed in changeset: 3283:4fe232899be8 in the 0.8 timeframe.
It may be fixed by now? Still universal line endings won't handle \r\r\n.
(Note I am including the tracker in this email so we get it on ticket.)
|
msg7354 |
Author: [hidden] (noschvie) |
Date: 2021-11-04 14:30 |
|
The Turnkey appliance seems to use Python 3.7 because of the error messge, but both are installed, 2.7 and 3.7
Importing issue - 1Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/roundup/admin.py", line 1702, in run_command
ret = function(args[1:])
File "/usr/local/lib/python3.7/dist-packages/roundup/admin.py", line 1448, in do_import
nodeid = cl.import_list(file_props, r)
File "/usr/local/lib/python3.7/dist-packages/roundup/backends/rdbms_common.py", line 3156, in import_list
newid = eval_import(proplist[propnames.index("id")])
IndexError: list index out of range
root@roundup ~# dir /usr/local/lib
python2.7 python3.7
root@roundup ~# python --version
Python 2.7.16
|
msg7444 |
Author: [hidden] (rouilj) |
Date: 2022-01-28 03:42 |
|
I am going to close this out. It seems like an edge case in multiple ways:
python 2 -> 3
windows -> linux
We have a workaround by editing the csv file so this is annoying but not a blocker.
|
|
Date |
User |
Action |
Args |
2022-01-28 03:42:56 | rouilj | set | status: new -> closed resolution: wont fix messages:
+ msg7444 |
2021-11-04 14:30:02 | noschvie | set | messages:
+ msg7354 |
2021-11-04 13:59:15 | rouilj | set | nosy:
+ rouilj messages:
+ msg7353 |
2021-11-04 11:31:48 | noschvie | set | type: crash |
2021-11-04 11:30:51 | noschvie | create | |
|