Roundup Tracker - Issues

Issue 2550960

classification
Title: Python 3 preparation patches
Type: rfe Severity: normal
Components: Infrastructure Versions: devel
process
Status: new Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ber, joseph_myers, rouilj
Priority: normal Keywords: patch, python3

Created on 2018-06-10 22:58 by joseph_myers, last changed 2018-06-23 00:57 by joseph_myers.

Files
File name Uploaded Description Edit Remove
roundup-python3-patches-1-36.tar.gz joseph_myers, 2018-06-10 22:57 Patches 1 to 36
0037-Python-3-preparation-miscellaneous-Python-scripts-no.patch joseph_myers, 2018-06-12 11:36 Python 3 preparation: miscellaneous Python scripts not named *.py
0038-Python-3-preparation-documentation.patch joseph_myers, 2018-06-12 11:37 Python 3 preparation: documentation
0039-Python-3-preparation-HTML-templates.patch joseph_myers, 2018-06-12 11:37 Python 3 preparation: HTML templates
0040-Python-3-preparation-comparisons.patch joseph_myers, 2018-06-17 12:47 Python 3 preparation: comparisons
0041-Python-3-preparation-use-string.ascii_letters-instea.patch joseph_myers, 2018-06-18 00:53 Python 3 preparation: use string.ascii_letters instead of string.letters.
0042-Python-3-preparation-unicode.patch joseph_myers, 2018-06-18 00:53 Python 3 preparation: unicode
0043-Python-3-preparation-unichr.patch joseph_myers, 2018-06-18 00:54 Python 3 preparation: unichr
0044-Python-3-preparation-StringIO.patch joseph_myers, 2018-06-18 00:54 Python 3 preparation: StringIO
0045-Python-3-preparation-update-ugettext-ungettext-uses.patch joseph_myers, 2018-06-18 00:54 Python 3 preparation: update ugettext, ungettext uses
0046-Python-3-preparation-avoid-string.lower.patch joseph_myers, 2018-06-18 00:55 Python 3 preparation: avoid string.lower
0047-Python-3-preparation-avoid-obsolete-types.-Type-name.patch joseph_myers, 2018-06-18 00:55 Python 3 preparation: avoid obsolete types.*Type names
0048-Python-3-preparation-remove-obsolete-email-module-mo.patch joseph_myers, 2018-06-18 00:55 Python 3 preparation: remove obsolete email module monkey patch
0049-Python-3-preparation-update-email-module-names.patch joseph_myers, 2018-06-18 00:56 Python 3 preparation: update email module names
0050-Python-3-preparation-avoid-string.find.patch joseph_myers, 2018-06-18 00:56 Python 3 preparation: avoid string.find
0051-Python-3-preparation-avoid-logging._levelNames.patch joseph_myers, 2018-06-18 00:56 Python 3 preparation: avoid logging._levelNames
0052-Python-3-preparation-update-__import__-call-for-rela.patch joseph_myers, 2018-06-18 00:57 Python 3 preparation: update __import__ call for relative import
0053-Python-3-preparation-use-byte-string-argument-to-bas.patch joseph_myers, 2018-06-18 00:57 Python 3 preparation: use byte-string argument to base64.decodestring for favicon
0054-Python-3-preparation-use-byte-strings-and-binary-I-O.patch joseph_myers, 2018-06-18 00:58 Python 3 preparation: use byte strings and binary I/O in roundup/install_util.py
0055-Python-3-preparation-use-bytes-more-in-password-hand.patch joseph_myers, 2018-06-20 00:38 Python 3 preparation: use bytes more in password handling
0056-Python-3-preparation-make-fallback-SysCallError-an-a.patch joseph_myers, 2018-06-20 21:17 Python 3 preparation: make fallback SysCallError an actual exception class
0057-Python-3-preparation-use-bytes-in-_gen_nonce.patch joseph_myers, 2018-06-20 21:17 Python 3 preparation: use bytes in _gen_nonce
0058-Python-3-preparation-write-bytes-to-socket-in-roundu.patch joseph_myers, 2018-06-20 21:18 Python 3 preparation: write bytes to socket in roundup_server.py
0059-Python-3-preparation-HTTP-headers-handling-in-roundu.patch joseph_myers, 2018-06-20 21:18 Python 3 preparation: HTTP headers handling in roundup_server.py
0060-Python-3-preparation-avoid-string.split.patch joseph_myers, 2018-06-20 21:18 Python 3 preparation: avoid string.split()
0061-Python-3-preparation-avoid-string.join.patch joseph_myers, 2018-06-20 21:18 Python 3 preparation: avoid string.join()
0062-Python-3-preparation-avoid-string.translate-and-stri.patch joseph_myers, 2018-06-20 21:19 Python 3 preparation: avoid string.translate() and string.maketrans()
0063-Python-3-preparation-update-tokenize-use-in-cgitb.py.patch joseph_myers, 2018-06-20 21:19 Python 3 preparation: update tokenize use in cgitb.py
0064-Python-3-preparation-send-bytes-to-socket-in-cgi-cli.patch joseph_myers, 2018-06-20 21:20 Python 3 preparation: send bytes to socket in cgi/client.py
0065-Python-3-preparation-avoid-basestring.patch joseph_myers, 2018-06-22 00:34 Python 3 preparation: avoid basestring
0066-Python-3-preparation-update-string-translate-method-.patch joseph_myers, 2018-06-22 00:34 Python 3 preparation: update string translate method call in cgi/accept_language.py
0067-Python-3-preparation-use-bytes-in-anti_csrf_nonce.patch joseph_myers, 2018-06-22 00:34 Python 3 preparation: use bytes in anti_csrf_nonce
0068-Python-3-preparation-use-bytes-in-_gen_sid.patch joseph_myers, 2018-06-23 00:55
0069-Python-3-preparation-avoid-assigning-to-instance-__g.patch joseph_myers, 2018-06-23 00:55
Messages
msg6084 Author: [hidden] (joseph_myers) Date: 2018-06-10 22:57
The attached tarball of 36 patches addresses various Python 3
compatibility issues in the Roundup code.  See the descriptions quoted
in https://sourceforge.net/p/roundup/mailman/message/36339407/ for more
information on these patches.
msg6085 Author: [hidden] (joseph_myers) Date: 2018-06-12 11:38
Three more patches now attached, addressing the same issues in,
respectively, miscellaneous Python scripts not named *.py, the
documentation, and the HTML templates.
msg6086 Author: [hidden] (rouilj) Date: 2018-06-13 00:28
Applied file1638 0038-Python-3-preparation-documentation.patch to patch
code examples to be python 2 and 3 compatible. I included the changes to
design.txt as well.
msg6087 Author: [hidden] (joseph_myers) Date: 2018-06-17 12:47
Now added patch 40, which addresses Python 3 differences in comparisons
(a standard Python 3 conversion issue, not covered by automated tools as
the results of an automated conversion would generally be very
unidiomatic).  Testing the same as for previous patches (Roundup
testsuite under Python 2, only the parts not skipped with the modules I
have installed).

Python 3 no longer has the cmp function, or cmp= arguments to sorting
functions / methods (key= must be used instead), and requires rich
comparison methods such as __lt__ to be defined instead of using
__cmp__.  All of the comparison mechanisms supported in Python 3 are
also supported in Python 2.

This patch makes the corresponding changes in Roundup to use key
functions and rich comparison methods.  In the case of the
JournalPassword and Permission classes, only __eq__ and __ne__ are
defined as I don't see ordered comparisons as useful there (and for
Permission, the old __cmp__ function didn't try to provide a valid
ordering).  In the case of the Date class, I kept the __cmp__ method and
implemented the others in terms of it, to avoid excess repetitiveness in
duplicating implementation code for all six rich comparison methods.

In roundup/admin.py, help_commands_html used operator.attrgetter to
produce the second argument of sorted() - which would be reasonable for
a key function, but the second argument is the cmp function in Python 2,
not a key function (and the key function must be a named argument not a
positional argument in Python 3).  That function appears to be
completely unused, so I expect that code never worked.  This patch adds
the missing key= to that sorted() call, but it would also be reasonable
to remove the unused function completely instead.
msg6088 Author: [hidden] (joseph_myers) Date: 2018-06-18 01:13
I've now added 14 more patches (patches 41 through 54).

41. Python 3 preparation: use string.ascii_letters instead of
string.letters.

42. Python 3 preparation: unicode.

This patch introduces roundup/anypy/strings.py, which has a comment
explaining the string representations generally used and common
functions to handle the required conversions.  Places in the code that
explicitly reference the "unicode" type / built-in function are
generally changed to use the new functions (or, in a few places where
those new functions don't seem to fit well, other approaches such as
references to type(u'') or use of the codecs module).  This patch does
not generally attempt to address text conversions in any places not
currently referencing the "unicode" type (although scripts/import_sf.py
is made to use binary I/O in places as fixing the "unicode" reference
didn't seem coherent otherwise).

43. Python 3 preparation: unichr.

44. Python 3 preparation: StringIO.

This generally arranges for StringIO and cStringIO references to use
io.StringIO for Python 3 but io.BytesIO for Python 2, consistent with
the string representations generally used in Roundup.  A special
FasterStringIO in the TAL code, which referenced internals of the old
Python 2 StringIO module, is cut down so it doesn't actually do anything
beyond the StringIO class it inherits from (it would also be reasonable
to remove FasterStringIO completely).  One place in roundup_server.py
clearly needing binary I/O is made to use io.BytesIO unconditionally.

45. Python 3 preparation: update ugettext, ungettext uses.

Python 3 does not have ugettext, ungettext methods as distinct from
gettext and ngettext, so uses in Roundup need adjusting to allow for that.

46. Python 3 preparation: avoid string.lower.

47. Python 3 preparation: avoid obsolete types.*Type names.

There are references to types.ListType (not actually used),
types.TupleType (for which tuple is a sufficient replacement) and
types.InstanceType (also removed from the types module in Python 3, it
was the type of instances of old-style classes only).  Where
InstanceType is used it's testing whether an exception is a class or
not; support for non-class exceptions was removed in Python 3 and patch
3 in this series removed the sole instance of one in Roundup, so making
the code in question unconditional seems reasonable.

48. Python 3 preparation: remove obsolete email module monkey patch.

This monkey patch has a comment saying it's obsolete with a minimum
Python version of 2.7.  Furthermore, it does not work with Python 3,
both because the email.Header name has gone away (only email.header
remains) and because the __dict__ object is a mappingproxy in Python 3
which can't be modified in the way used there.  So the monkey patch is
removed in this patch.

49. Python 3 preparation: update email module names.

Various email.* modules were renamed in Python 2.5, and backwards
compatibility for the old names was removed in Python 3.

50. Python 3 preparation: avoid string.find.

51. Python 3 preparation: avoid logging._levelNames.

logging._levelNames is an internal name, not a public interface, and is
not available in current Python 3.  Fortunately, logger.setLevel accepts
strings for log levels, not just numeric levels, in 2.7 (although that
change is not mentioned in the 2.7 documentation).

52. Python 3 preparation: update __import__ call for relative import.

53. Python 3 preparation: use byte-string argument to
base64.decodestring for favicon.

54. Python 3 preparation: use byte strings and binary I/O in
roundup/install_util.py.

This code is hashing files it copies and inserting comments with the
hash value.  The hash interfaces require bytes objects in Python 3 and
it seems reasonable to use such objects throughout this file.
msg6089 Author: [hidden] (joseph_myers) Date: 2018-06-20 00:38
Patch 55 is also needed to get demo.py to start up from scratch.  I have
a few more patches for issues showing up with demo.py processing
requests that I still need to run the testsuite on with Python 2 before
uploading.

55. Python 3 preparation: use bytes more in password handling.

This patch adds enough fixes to use bytes to work for PBKDF2 passwords,
but probably not for the other hash algorithms supported.

It would make sense for the PBKDF2 code to use the Python hashlib
support for PBKDF2 as the first choice before M2Crypto and the local
PBKDF2 implementation (hashlib supports it in 2.7.8, 3.4 and later), but
it still seems appropriate to fix the local code to handle bytes
properly anyway.  I don't know if we'll end up supporting Python 3.3
(the minimum plausible Python 3 version, but now EOL) in Roundup or not.
msg6090 Author: [hidden] (ber) Date: 2018-06-20 06:28
Am Mittwoch 20 Juni 2018 02:38:29 schrieb Joseph Myers:
>  I don't know if we'll end up supporting Python 3.3
> (the minimum plausible Python 3 version, but now EOL) in Roundup or not.

Propably not if it already has reached its end of life.
msg6091 Author: [hidden] (joseph_myers) Date: 2018-06-20 21:28
The next nine patches get to the point where, while demo.py (with
mailgw.py hacked up) does not successfully serve a login page running
under Python 3, the resulting exception tracebacks at least get to the
browser rather than having a second exception occur during the handling
of the first because of Python 3 issues in the exception handling code.

56. Python 3 preparation: make fallback SysCallError an actual exception
class.

The code defines a fallback version of SysCallError when import of
OpenSSL.SSL.  But in Python 3, defining it as None and then using in
"except" statements results in:

TypeError: catching classes that do not inherit from BaseException is
not allowed

57. Python 3 preparation: use bytes in _gen_nonce.

58. Python 3 preparation: write bytes to socket in roundup_server.py.

59. Python 3 preparation: HTTP headers handling in roundup_server.py.

HTTP headers are handled differently in Python 3 (where they use
email.message.Message) compared to Python 2 (where they use
mimetools.Message).  In some places the code needs to check which
version of the interface is available.  For the common case of getting a
single header, ".get" is available in both versions, and is an alias of
".getheader" in Python 2.  (Note that the Python 3 semantics of ".get"
are slightly different from those in Python 2 if there is more than one
of a given header - it returns an arbitrary one, when in Python 2 it's
specified to return the last one.  Hopefully the places using this
interface rather than explicitly allowing for multiple headers with the
same name are OK with that and it shouldn't actually occur with
well-behaved clients.)

60. Python 3 preparation: avoid string.split().

61. Python 3 preparation: avoid string.join().

62. Python 3 preparation: avoid string.translate() and string.maketrans().

63. Python 3 preparation: update tokenize use in cgitb.py.

Note that the same interface that has changed incompatibly is also used
in tools/pygettext.py.  That file also needs fixing, but this patch does
*not* attempt such a fix.

64. Python 3 preparation: send bytes to socket in cgi/client.py.
msg6092 Author: [hidden] (joseph_myers) Date: 2018-06-22 00:37
The next three patches get to the point where (with mailgw.py hacked up)
demo.py successfully serves a login page under Python 3, although
actually logging in fails with more bytes/str issues.

65. Python 3 preparation: avoid basestring.

66. Python 3 preparation: update string translate method call in
cgi/accept_language.py.

The str translate method in Python 3 is sufficiently different from that
in Python 2 that different arguments are needed here.  Note that the
existing code "ascii = ''.join([chr(x) for x in range(256)])" is
redundant with both versions (in Python 2, None can be used instead as
the first translate argument from Python 2.6 onwards).

67. Python 3 preparation: use bytes in anti_csrf_nonce.
msg6093 Author: [hidden] (joseph_myers) Date: 2018-06-23 00:57
With the next two patches, it's possible to log in with demo.py running
under Python 3, both as demo/demo (patch 68) and as admin/admin (patch 69).

68. Python 3 preparation: use bytes in _gen_sid.

69. Python 3 preparation: avoid assigning to instance __getitem__ in
TruthDict.

In Python 3, special method names are generally only looked up at the
class level, not on instance objects, and so assigning to them for an
instance object doesn't work as expected.
History
Date User Action Args
2018-06-23 00:57:36joseph_myerssetmessages: + msg6093
2018-06-23 00:55:48joseph_myerssetfiles: + 0069-Python-3-preparation-avoid-assigning-to-instance-__g.patch
2018-06-23 00:55:39joseph_myerssetfiles: + 0068-Python-3-preparation-use-bytes-in-_gen_sid.patch
2018-06-22 00:37:49joseph_myerssetmessages: + msg6092
2018-06-22 00:34:52joseph_myerssetfiles: + 0067-Python-3-preparation-use-bytes-in-anti_csrf_nonce.patch
2018-06-22 00:34:38joseph_myerssetfiles: + 0066-Python-3-preparation-update-string-translate-method-.patch
2018-06-22 00:34:19joseph_myerssetfiles: + 0065-Python-3-preparation-avoid-basestring.patch
2018-06-20 21:28:17joseph_myerssetmessages: + msg6091
2018-06-20 21:20:03joseph_myerssetfiles: + 0064-Python-3-preparation-send-bytes-to-socket-in-cgi-cli.patch
2018-06-20 21:19:45joseph_myerssetfiles: + 0063-Python-3-preparation-update-tokenize-use-in-cgitb.py.patch
2018-06-20 21:19:18joseph_myerssetfiles: + 0062-Python-3-preparation-avoid-string.translate-and-stri.patch
2018-06-20 21:18:59joseph_myerssetfiles: + 0061-Python-3-preparation-avoid-string.join.patch
2018-06-20 21:18:40joseph_myerssetfiles: + 0060-Python-3-preparation-avoid-string.split.patch
2018-06-20 21:18:26joseph_myerssetfiles: + 0059-Python-3-preparation-HTTP-headers-handling-in-roundu.patch
2018-06-20 21:18:06joseph_myerssetfiles: + 0058-Python-3-preparation-write-bytes-to-socket-in-roundu.patch
2018-06-20 21:17:37joseph_myerssetfiles: + 0057-Python-3-preparation-use-bytes-in-_gen_nonce.patch
2018-06-20 21:17:14joseph_myerssetfiles: + 0056-Python-3-preparation-make-fallback-SysCallError-an-a.patch
2018-06-20 06:28:24bersetmessages: + msg6090
2018-06-20 00:38:29joseph_myerssetfiles: + 0055-Python-3-preparation-use-bytes-more-in-password-hand.patch
messages: + msg6089
2018-06-18 01:13:20joseph_myerssetmessages: + msg6088
2018-06-18 00:58:08joseph_myerssetfiles: + 0054-Python-3-preparation-use-byte-strings-and-binary-I-O.patch
2018-06-18 00:57:44joseph_myerssetfiles: + 0053-Python-3-preparation-use-byte-string-argument-to-bas.patch
2018-06-18 00:57:22joseph_myerssetfiles: + 0052-Python-3-preparation-update-__import__-call-for-rela.patch
2018-06-18 00:56:55joseph_myerssetfiles: + 0051-Python-3-preparation-avoid-logging._levelNames.patch
2018-06-18 00:56:40joseph_myerssetfiles: + 0050-Python-3-preparation-avoid-string.find.patch
2018-06-18 00:56:23joseph_myerssetfiles: + 0049-Python-3-preparation-update-email-module-names.patch
2018-06-18 00:55:57joseph_myerssetfiles: + 0048-Python-3-preparation-remove-obsolete-email-module-mo.patch
2018-06-18 00:55:37joseph_myerssetfiles: + 0047-Python-3-preparation-avoid-obsolete-types.-Type-name.patch
2018-06-18 00:55:15joseph_myerssetfiles: + 0046-Python-3-preparation-avoid-string.lower.patch
2018-06-18 00:54:51joseph_myerssetfiles: + 0045-Python-3-preparation-update-ugettext-ungettext-uses.patch
2018-06-18 00:54:36joseph_myerssetfiles: + 0044-Python-3-preparation-StringIO.patch
2018-06-18 00:54:19joseph_myerssetfiles: + 0043-Python-3-preparation-unichr.patch
2018-06-18 00:53:57joseph_myerssetfiles: + 0042-Python-3-preparation-unicode.patch
2018-06-18 00:53:29joseph_myerssetfiles: + 0041-Python-3-preparation-use-string.ascii_letters-instea.patch
2018-06-17 12:48:00joseph_myerssetfiles: + 0040-Python-3-preparation-comparisons.patch
messages: + msg6087
2018-06-13 00:28:48rouiljsetpriority: normal
nosy: + rouilj
messages: + msg6086
2018-06-12 11:38:11joseph_myerssetmessages: + msg6085
2018-06-12 11:37:26joseph_myerssetfiles: + 0039-Python-3-preparation-HTML-templates.patch
2018-06-12 11:37:06joseph_myerssetfiles: + 0038-Python-3-preparation-documentation.patch
2018-06-12 11:36:41joseph_myerssetfiles: + 0037-Python-3-preparation-miscellaneous-Python-scripts-no.patch
2018-06-11 07:00:32bersetnosy: + ber
2018-06-10 23:52:01rouiljsetkeywords: + python3
2018-06-10 23:50:29rouiljsetkeywords: + patch
2018-06-10 22:58:00joseph_myerscreate