The attached tarball of 36 patches addresses various Python 3
compatibility issues in the Roundup code.  See the descriptions quoted
in https://sourceforge.net/p/roundup/mailman/message/36339407/ for more
information on these patches.

Three more patches now attached, addressing the same issues in,
respectively, miscellaneous Python scripts not named *.py, the
documentation, and the HTML templates.

Applied file1638 0038-Python-3-preparation-documentation.patch to patch
code examples to be python 2 and 3 compatible. I included the changes to
design.txt as well.

Now added patch 40, which addresses Python 3 differences in comparisons
(a standard Python 3 conversion issue, not covered by automated tools as
the results of an automated conversion would generally be very
unidiomatic).  Testing the same as for previous patches (Roundup
testsuite under Python 2, only the parts not skipped with the modules I
have installed).

Python 3 no longer has the cmp function, or cmp= arguments to sorting
functions / methods (key= must be used instead), and requires rich
comparison methods such as __lt__ to be defined instead of using
__cmp__.  All of the comparison mechanisms supported in Python 3 are
also supported in Python 2.

This patch makes the corresponding changes in Roundup to use key
functions and rich comparison methods.  In the case of the
JournalPassword and Permission classes, only __eq__ and __ne__ are
defined as I don't see ordered comparisons as useful there (and for
Permission, the old __cmp__ function didn't try to provide a valid
ordering).  In the case of the Date class, I kept the __cmp__ method and
implemented the others in terms of it, to avoid excess repetitiveness in
duplicating implementation code for all six rich comparison methods.

In roundup/admin.py, help_commands_html used operator.attrgetter to
produce the second argument of sorted() - which would be reasonable for
a key function, but the second argument is the cmp function in Python 2,
not a key function (and the key function must be a named argument not a
positional argument in Python 3).  That function appears to be
completely unused, so I expect that code never worked.  This patch adds
the missing key= to that sorted() call, but it would also be reasonable
to remove the unused function completely instead.

I've now added 14 more patches (patches 41 through 54).

41. Python 3 preparation: use string.ascii_letters instead of
string.letters.

42. Python 3 preparation: unicode.

This patch introduces roundup/anypy/strings.py, which has a comment
explaining the string representations generally used and common
functions to handle the required conversions.  Places in the code that
explicitly reference the "unicode" type / built-in function are
generally changed to use the new functions (or, in a few places where
those new functions don't seem to fit well, other approaches such as
references to type(u'') or use of the codecs module).  This patch does
not generally attempt to address text conversions in any places not
currently referencing the "unicode" type (although scripts/import_sf.py
is made to use binary I/O in places as fixing the "unicode" reference
didn't seem coherent otherwise).

43. Python 3 preparation: unichr.

44. Python 3 preparation: StringIO.

This generally arranges for StringIO and cStringIO references to use
io.StringIO for Python 3 but io.BytesIO for Python 2, consistent with
the string representations generally used in Roundup.  A special
FasterStringIO in the TAL code, which referenced internals of the old
Python 2 StringIO module, is cut down so it doesn't actually do anything
beyond the StringIO class it inherits from (it would also be reasonable
to remove FasterStringIO completely).  One place in roundup_server.py
clearly needing binary I/O is made to use io.BytesIO unconditionally.

45. Python 3 preparation: update ugettext, ungettext uses.

Python 3 does not have ugettext, ungettext methods as distinct from
gettext and ngettext, so uses in Roundup need adjusting to allow for that.

46. Python 3 preparation: avoid string.lower.

47. Python 3 preparation: avoid obsolete types.*Type names.

There are references to types.ListType (not actually used),
types.TupleType (for which tuple is a sufficient replacement) and
types.InstanceType (also removed from the types module in Python 3, it
was the type of instances of old-style classes only).  Where
InstanceType is used it's testing whether an exception is a class or
not; support for non-class exceptions was removed in Python 3 and patch
3 in this series removed the sole instance of one in Roundup, so making
the code in question unconditional seems reasonable.

48. Python 3 preparation: remove obsolete email module monkey patch.

This monkey patch has a comment saying it's obsolete with a minimum
Python version of 2.7.  Furthermore, it does not work with Python 3,
both because the email.Header name has gone away (only email.header
remains) and because the __dict__ object is a mappingproxy in Python 3
which can't be modified in the way used there.  So the monkey patch is
removed in this patch.

49. Python 3 preparation: update email module names.

Various email.* modules were renamed in Python 2.5, and backwards
compatibility for the old names was removed in Python 3.

50. Python 3 preparation: avoid string.find.

51. Python 3 preparation: avoid logging._levelNames.

logging._levelNames is an internal name, not a public interface, and is
not available in current Python 3.  Fortunately, logger.setLevel accepts
strings for log levels, not just numeric levels, in 2.7 (although that
change is not mentioned in the 2.7 documentation).

52. Python 3 preparation: update __import__ call for relative import.

53. Python 3 preparation: use byte-string argument to
base64.decodestring for favicon.

54. Python 3 preparation: use byte strings and binary I/O in
roundup/install_util.py.

This code is hashing files it copies and inserting comments with the
hash value.  The hash interfaces require bytes objects in Python 3 and
it seems reasonable to use such objects throughout this file.

Patch 55 is also needed to get demo.py to start up from scratch.  I have
a few more patches for issues showing up with demo.py processing
requests that I still need to run the testsuite on with Python 2 before
uploading.

55. Python 3 preparation: use bytes more in password handling.

This patch adds enough fixes to use bytes to work for PBKDF2 passwords,
but probably not for the other hash algorithms supported.

It would make sense for the PBKDF2 code to use the Python hashlib
support for PBKDF2 as the first choice before M2Crypto and the local
PBKDF2 implementation (hashlib supports it in 2.7.8, 3.4 and later), but
it still seems appropriate to fix the local code to handle bytes
properly anyway.  I don't know if we'll end up supporting Python 3.3
(the minimum plausible Python 3 version, but now EOL) in Roundup or not.

Am Mittwoch 20 Juni 2018 02:38:29 schrieb Joseph Myers:
>  I don't know if we'll end up supporting Python 3.3
> (the minimum plausible Python 3 version, but now EOL) in Roundup or not.

Propably not if it already has reached its end of life.

The next nine patches get to the point where, while demo.py (with
mailgw.py hacked up) does not successfully serve a login page running
under Python 3, the resulting exception tracebacks at least get to the
browser rather than having a second exception occur during the handling
of the first because of Python 3 issues in the exception handling code.

56. Python 3 preparation: make fallback SysCallError an actual exception
class.

The code defines a fallback version of SysCallError when import of
OpenSSL.SSL.  But in Python 3, defining it as None and then using in
"except" statements results in:

TypeError: catching classes that do not inherit from BaseException is
not allowed

57. Python 3 preparation: use bytes in _gen_nonce.

58. Python 3 preparation: write bytes to socket in roundup_server.py.

59. Python 3 preparation: HTTP headers handling in roundup_server.py.

HTTP headers are handled differently in Python 3 (where they use
email.message.Message) compared to Python 2 (where they use
mimetools.Message).  In some places the code needs to check which
version of the interface is available.  For the common case of getting a
single header, ".get" is available in both versions, and is an alias of
".getheader" in Python 2.  (Note that the Python 3 semantics of ".get"
are slightly different from those in Python 2 if there is more than one
of a given header - it returns an arbitrary one, when in Python 2 it's
specified to return the last one.  Hopefully the places using this
interface rather than explicitly allowing for multiple headers with the
same name are OK with that and it shouldn't actually occur with
well-behaved clients.)

60. Python 3 preparation: avoid string.split().

61. Python 3 preparation: avoid string.join().

62. Python 3 preparation: avoid string.translate() and string.maketrans().

63. Python 3 preparation: update tokenize use in cgitb.py.

Note that the same interface that has changed incompatibly is also used
in tools/pygettext.py.  That file also needs fixing, but this patch does
*not* attempt such a fix.

64. Python 3 preparation: send bytes to socket in cgi/client.py.

The next three patches get to the point where (with mailgw.py hacked up)
demo.py successfully serves a login page under Python 3, although
actually logging in fails with more bytes/str issues.

65. Python 3 preparation: avoid basestring.

66. Python 3 preparation: update string translate method call in
cgi/accept_language.py.

The str translate method in Python 3 is sufficiently different from that
in Python 2 that different arguments are needed here.  Note that the
existing code "ascii = ''.join([chr(x) for x in range(256)])" is
redundant with both versions (in Python 2, None can be used instead as
the first translate argument from Python 2.6 onwards).

67. Python 3 preparation: use bytes in anti_csrf_nonce.

With the next two patches, it's possible to log in with demo.py running
under Python 3, both as demo/demo (patch 68) and as admin/admin (patch 69).

68. Python 3 preparation: use bytes in _gen_sid.

69. Python 3 preparation: avoid assigning to instance __getitem__ in
TruthDict.

In Python 3, special method names are generally only looked up at the
class level, not on instance objects, and so assigning to them for an
instance object doesn't work as expected.

Rebased patch 1 attached, updated for changes checked in since the
original patch was prepared.

Patches 70 to 73 get to the point where it's possible to do things such
as creating issues and viewing lists of database objects with demo.py
running under Python 3 without exceptions occurring.

70. Python 3 preparation: use // division in HTML templates.

Like patches 25 and 39, but I missed the need for such changes in HTML
templates when preparing those patches.

71. Python 3 preparation: convert dbm keys back from bytes to strings.

The Python 3 dbm module uses bytes for keys and values, converting any
passed in strings to bytes and always returning bytes when keys are
listed or values extracted.  Bytes for values is fine with Roundup
(which uses the marshal module to produce the values stored, which
produces bytes anyway in Python 3), but bytes for keys need converting
back to strings when keys are enumerated.

72. Python 3 preparation: convert string content to bytes for file storage.

When FileClass content comes from a text field in a form rather than an
uploaded file (e.g. creating a msg object in the classic template), this
reaches the blobfiles code as a str object, which thus needs converting
to bytes for storage.

Note that given this fix, while msg objects can be created, they appear
with spurious b'' on the issue pages.  Something needs to handle the
conversion in the other direction as well; I'm not entirely sure what,
but probably the hyperdb property wrapper for any String content
property that is actually stored in a file like this.

(As previously discussed, ideally there might be a distinction between
String and Bytes fields, and then there might be separate text and
binary variants of FileClass.  I haven't attempted to implement any of
that and it should be possible to get Roundup working with Python 3
without needing to do that.)

73. Python 3 preparation: extract full Content-Type header in
roundup_server.py.

This is a fix for code changed in patch 59 and could reasonably be
merged into that patch.  It turns out that for proper parsing of some
form POST submissions, the content type extracted here has to be the
full Content-Type header, including in particular the specified boundary.

Now 1.6 is out (thanks!), can we figure out how to get these patches (or
any other such fixes for the same issues) reviewed and into Roundup
trunk, so that future Python 3 improvements can take place incrementally
in trunk starting from a baseline where many things already work, and so
that future unrelated code changes are less likely to accidentally
introduce more incompatibilities by copying code using old idioms?

A reminder: the version of patch 1 attached directly to this issue is a
rebased one that supersedes the original one in the attached tarball. 
Patch 38 was already applied for 1.6.  These patches don't do anything
about the mailgw.py use of mimetools; if desired I could provide a
version of my mailgw.py hack that avoids breaking things with Python 2
while still not actually making it work with Python 3, but really what's
needed for mailgw.py's use of mimetools is review / commit of John
Kristensen's patches to move it to using the email package.

I am very interested in moving to Python 3 and am testing the patches in 
my local environment.

Not sure what the best way forward is, but maybe the patches can just be 
committed to a separate branch and refined there (before being merged 
into the master branch) - I have done this "locally" now at 
https://bitbucket.org/cmeerw/roundup/src/py3/ (in case anyone is 
interested)

patch 0045 results in infinite recursion for me with Python 3

Anyway, I have made some more changes on a py3-cmeerw branch here: 
https://bitbucket.org/cmeerw/roundup/commits/branch/py3-cmeerw

With this I got a (read-only) issue tracker pretty much working here with 
Python 3 (using jinja2 as the template engine)

changes for share/roundup/templates/jinja2/html/layout/sort.html in patch 
0039 are not needed and actually break the template on both Python 2 and 
Python 3 ("range" is provided by jinja2, not Python)

I have now gone through the patches: there are a few things that might 
affect behaviour on Python 2:

patch 0003: in frontends/ZRoundup/ZRoundup.py, line 211
  uses "raise Exception" instead of "raise" with a string literal (not 
sure if this has any real world impact)

patch 0012: in roundup/backends/rdbms_common.py, line 674
  catches Exception instead of StandardError (probably benign)

patch 0042: in roundup/xmlrpc.py, line 44
  doesn't check for ascii encoding any more

patch 0044: in roudup/cgi/TAL/TALInterpreter.py, line 759
  FasterStringIO optimization has been removed

and I guess there might be some performance impact for Python 2 users 
because of "iteritems" -> "items" changes and extra "list" calls - not 
sure if we should care about these

Thanks for the review!  I believe the changes in question are safe, but
it's good to have people looking at them.

Moving away from raising a string literal is unavoidable since Python 3
only allows raising exception objects.  It's unfortunate we have no test
coverage for the front ends (and to declare Python 3 support complete,
people will need to test all the front ends manually with Python 3), but
this is probably one of the safest changes in the untested code.

Regarding not checking for ascii, I think it's fine to treat a non-ascii
key the same as any other key that's ascii but invalid, and thus
simplify the code by having more places use the same encoding/decoding
functions.  (Whether non-ascii property names ought to be supported by
Roundup, presuming they aren't at present, is an interesting question,
but not one we need to address now.)

Removal of FasterStringIO was noted in the description I gave for patch
44 in the comments on this issue.  It's unavoidable in that it depended
on internals of the old Python 2 StringIO module.  My understanding is
that the main thing that matters for Roundup performance in large
instances is keeping down the number of SQL database queries involved in
producing a page or carrying out an action; even if FasterStringIO was
relevant at one point in the original context for which the TAL code was
written, it probably isn't very relevant now in Roundup.

Possible performance cost from iteritems etc. changes (which was noted
in the description of patch 20 I sent to roundup-devel) is hard to avoid
if you want to write the code directly in a modern (Python 3) style as
much as possible (not using a wrapper layer unless strictly necessary),
because the code in question (using items when you want to iterate,
list() when an explicit list is needed) is the modern style, while also
being compatible but less efficient with Python 2.  Again, I think the
area of biggest performance concern in Roundup is keeping down the
number of SQL queries, which should be unaffected by all these changes.

and 2 patches to get the setup script working (includes an updated 
msgfmt.py):

https://bitbucket.org/cmeerw/roundup/commits/7307133a8ed032b3460930dfc1f5
c160ee7f1671?at=py3-cmeerw
https://bitbucket.org/cmeerw/roundup/commits/160b9f734240545b957f92d70e4e
f38df90896e8?at=py3-cmeerw

just been looking at the FasterStringIO thing a bit more...

so there were comments in the code warning not to use cStringIO in this 
case because of unicode issues; now the patches actually use io.ByteIO 
(in Python 2) which certainly doesn't accept unicode. But maybe that's 
actually ok because of a roundup specific modification to 
TALInterpreter.py: "Modifications for Roundup: 1. implemented ustr as 
str" But then FasterStringIO should be removed completely.

I've attached an updated version of the full set of patches, in a form I
think is suitable for trunk.  Changes from those previously posted:

* Patches with extended descriptions now have those included in the
commit messages in the patch files.

* Patch 38 is omitted as already in 1.6.

* Parts of the original patch 39 that Christof identified as incorrect
are removed.

* Patch 45 is omitted as Christof reported problems with it.

* The original patch 73 is merged into the original 59 (which is now
numbered 57), being a bug fix to a change made in that patch.

I'm happy to push some or all of these to trunk, given write access, if
we think there's now been enough review, or to make a git tree with them
available somewhere.

I think the version of that TAL code in Roundup is deliberately avoiding
Python 2 unicode (such as via ustr = str), hence being OK for it to use
io.BytesIO in Python 2 like other StringIO / cStringIO uses.  (But the
consolidation of different variants of StringIO on io.BytesIO certainly
has the potential to be one of the riskier changes, since it's not
formally a no-op with Python 2 although it's not meant to actually
change how anything behaves with Python 2.)

Once these patches are in trunk, we'll still need at least the following
to consider Python 3 support complete (the order in which these are
listed is arbitrary, any dependencies between them are pretty weak):

* CHANGES.txt and doc/upgrading.txt entries for the changes (not listing
every separate patch, but noting the presence of such changes and the
need to update .py and .html files copied into instances).

* Also merge Christof's followup patches (including a replacement /
fixed version of my patch 45) into trunk.

* Also update Jerrykan's mailgw changes
(https://github.com/jerrykan/herder/tree/mailgw last 6 commits) and get
those into trunk to eliminate mimetools use in mailgw.

* Complete getting the CI working to test commits with Python 3
automatically.

* Get the whole testsuite working cleanly with Python 3.

* Test all the front ends with Python 3 and get them working with it
(front ends are both not well-covered by the testsuite, and a place
where bytes/string conversions are particularly likely to be needed). 
By front ends I mean the different web interfaces where
doc/installation.txt says "There are five web interfaces to choose from"
(and then lists six, not five), not just what's in the frontends
directory - all six need to be made to work with Python 3 for us to be
able to document Python 3 support as on a par with that for Python 3.

* Similarly, make sure that incoming mail via roundup-mailgw works with
Python 3, for each different way documented for it to receive mail.

* Make sure that an existing instance, set up with Python 2, including
UTF-8 data in the database not just plain ASCII, works correctly when
used with Python 3 - that the database content format is properly
compatible across Python versions - with every database back end (the
reverse, creating with Python 3 and using with Python 2, may be less
important, but is still desirable - if someone has problems updating
their instance to Python 3, they should be able to move back to Python 2
without the stored data being incompatible).  anydbm may have the
highest risk of problems here, as it uses the Python marshal module in
storing data.

* Ensure that the CSV format used for export/import is properly
compatible across Python versions.  UTF-8 strings are included in
exports with \x escapes for the individual bytes of the UTF-8
representation (because that's how Python 2 does repr of a str object).
 Special care will be needed to keep this compatible for an export with
one Python version to be importable with another Python version.

* Update tokenize use in tools/pygettext.py (my patches only deal with
the use in cgitb.py).

* Convert scripts/roundup-reminder to use the email package instead of
the MimeWriter module which is no longer available with Python 3.

* Remove 2to3-done.txt once the text in there discussing Python 3
conversion plans is no longer relevant.  (I think the lists of files are
already not relevant.)

* Update all documentation to reflect the support for Python 3 as well
as for Python 2.  Documentation here includes the classifiers for pypi
in setup.py that currently include 'Programming Language :: Python :: 2
:: Only'.

* Get out an actual release with full Python 3 support (and then expect
users to find problems and so more fixed releases to be needed; I think
we should aim to have such releases reasonably frequently until the
Python 3 support seems solid in production use).

* Not actually needed for Python 3 support, but a good idea anyway:
prefer the standard library's PBKDF2 support in password handling
(available with Python 2.7.8, 3.4 and later).

* Not actually needed, but cleanup to consider: remove
roundup/admin.py:help_commands_html as apparently unused.

* Not actually needed, but cleanup to consider: remove FasterStringIO
completely.

* Not actually needed, but might be cleaner: distinguish String and
Bytes fields (and then consider having text and binary variants of
FileClass).

change hardcoded io.BytesIO: 
https://bitbucket.org/cmeerw/roundup/commits/65e47b5d84a0197231ce884d73ba
166169513ecb?at=py3-cmeerw

I have started looking at test failures now, here are some patches:

https://bitbucket.org/cmeerw/roundup/commits/4c6e8f35ec1eda243b15c2be4c
7e80c9aa8d5cc7
https://bitbucket.org/cmeerw/roundup/commits/75d4ad10cd6d60c676ece341b3
2e9e6ca9201ea9
https://bitbucket.org/cmeerw/roundup/commits/ddea2d1453381a17271b042221
785ea906c6e9da
https://bitbucket.org/cmeerw/roundup/commits/17246d7177e600e4d2f3c63b86
6915881775d532
https://bitbucket.org/cmeerw/roundup/commits/7fa11755bb4930e7d37193beb8
372de3aaf8acfb

http://issues.roundup-tracker.org/issue2550968
http://issues.roundup-tracker.org/issue2550969
http://issues.roundup-tracker.org/issue2550970
http://issues.roundup-tracker.org/issue2550971
http://issues.roundup-tracker.org/issue2550972
http://issues.roundup-tracker.org/issue2550973
http://issues.roundup-tracker.org/issue2550974
http://issues.roundup-tracker.org/issue2550975
http://issues.roundup-tracker.org/issue2550976
http://issues.roundup-tracker.org/issue2550977
http://issues.roundup-tracker.org/issue2550978
http://issues.roundup-tracker.org/issue2550979
http://issues.roundup-tracker.org/issue2550980
http://issues.roundup-tracker.org/issue2550981
http://issues.roundup-tracker.org/issue2550982
http://issues.roundup-tracker.org/issue2550983
http://issues.roundup-tracker.org/issue2550984
http://issues.roundup-tracker.org/issue2550985
http://issues.roundup-tracker.org/issue2550986
http://issues.roundup-tracker.org/issue2550987

filed for the individual TODO items mentioned here and on the mailing
list for completing Python 3 support (and a few that came up in the
course of Python 3 work but aren't actually needed to complete that
support).

All the patches for which this issue was created are on trunk, and
separate issues have been opened for known remaining Python 3 issues.