Roundup Tracker - Issues

Issue 2551042

classification
Regular expression error on python3.7
Type: crash Severity: normal
Components: Versions: devel
process
Status: fixed fixed
:
: rouilj : pokoli, rouilj
Priority: normal : patch

Created on 2019-04-21 08:53 by pokoli, last changed 2019-04-29 00:35 by rouilj.

Files
File name Uploaded Description Edit Remove
issue2551042.patch pokoli, 2019-04-21 09:12
Messages
msg6463 Author: [hidden] (pokoli) Date: 2019-04-21 08:53
I've cloned latest trunk, installed using python setup.py install 
and executed: 

$ roundup-admin -i test install jinja2

Which have me the following exception:

Traceback (most recent call last):
  File "/home/pokoli/.virtualenvs/roundup/lib/python3.7/sre_parse.py",
line 1021, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\w'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pokoli/.virtualenvs/roundup/bin/roundup-admin", line 2, in
<module>
    from roundup.scripts.roundup_admin import run
  File
"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/scripts/roundup_admin.py",
line 40, in <module>
    from roundup.admin import AdminTool
  File
"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/admin.py",
line 30, in <module>
    import roundup.instance
  File
"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/instance.py",
line 43, in <module>
    from roundup.cgi import client, templating
  File
"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/cgi/client.py",
line 38, in <module>
    from roundup import rest
  File
"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/rest.py",
line 326, in <module>
    class RestfulInstance(object):
  File
"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/rest.py",
line 587, in RestfulInstance
    @Routing.route("/data/<:class_name>", 'GET')
  File
"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/rest.py",
line 279, in route
    rule = re.compile(cls.__var_to_regex.sub(cls.url_to_regex, rule))
  File "/home/pokoli/.virtualenvs/roundup/lib/python3.7/re.py", line
309, in _subx
    template = _compile_repl(template, pattern)
  File "/home/pokoli/.virtualenvs/roundup/lib/python3.7/re.py", line
300, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "/home/pokoli/.virtualenvs/roundup/lib/python3.7/sre_parse.py",
line 1024, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \w at position 2
msg6464 Author: [hidden] (pokoli) Date: 2019-04-21 09:12
I've attached a patch which fixes the issue for me. 

Note that this is my first contribution to the project, so please let me
know if I've done something wrong in the process and I will fix it.
msg6465 Author: [hidden] (rouilj) Date: 2019-04-22 00:08
Hello Sergi:

In message <1555836824.21.0.366951058554.issue2551042@roundup.psfhosted.org>,
Sergi Almacellas Abellana writes:
>I've cloned latest trunk, installed using python setup.py install 
>and executed: 
>
>$ roundup-admin -i test install jinja2

Which version of python 3.7 are you running?

If in the root of the checked out tree, you run python3 demo.py (as
specified in README.txt) without virtualenv, does that work or does it
generate a crash?

Also for testing purposes, what happens if from the root of the
checked out tree you run:

   mkdir example
   python3 roundup/scripts/roundup_admin.py -i example install classic

Do you still see the error? Try this both with and without virtualenv.

I can't reproduce the crash with python 3.4 using the following
command line:

   python3 ../roundup/scripts/roundup_admin -i . install jinja2

it prompts me for the database to use. No crashes.

Looking at the traceback, I expect any attempt to import rest.py would
cause this crash.  Our CI pipeline tests python 3.4 through 3.6 and
imports rest.py. I don't know of any change to regular expressions in
3.7 that would make \w (all word characters) an invalid character
class.

>Which have me the following exception:
>
>Traceback (most recent call last):
>  File "/home/pokoli/.virtualenvs/roundup/lib/python3.7/sre_parse.py",
>line 1021, in parse_template
>    this = chr(ESCAPES[this][1])
>KeyError: '\\w'

I am guesing the \\w is coming from whatever ESCAPES is doing and it
doubles the \.

>During handling of the above exception, another exception occurred:
>
>Traceback (most recent call last):
>  File "/home/pokoli/.virtualenvs/roundup/bin/roundup-admin", line 2, in
><module>
>    from roundup.scripts.roundup_admin import run
>  File
>"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/scripts/roundup_admin.py",
>line 40, in <module>
>    from roundup.admin import AdminTool
>  File
>"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/admin.py",
>line 30, in <module>
>    import roundup.instance
>  File
>"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/instance.py",
>line 43, in <module>
>    from roundup.cgi import client, templating
>  File
>"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/cgi/client.py",
>line 38, in <module>
>    from roundup import rest
>  File
>"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/rest.py",
>line 326, in <module>
>    class RestfulInstance(object):
>  File
>"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/rest.py",
>line 587, in RestfulInstance
>    @Routing.route("/data/<:class_name>", 'GET')
>  File
>"/home/pokoli/.virtualenvs/roundup/lib/python3.7/site-packages/roundup/rest.py",
>line 279, in route
>    rule = re.compile(cls.__var_to_regex.sub(cls.url_to_regex, rule))

Can you check rest.py and make sure that you have the following three
lines starting at line 244:

class Routing(object):
    __route_map = {}
    __var_to_regex = re.compile(r"<:(\w+)>")
    url_to_regex = r"([\w.\-~!$&'()*+,;=:\%%]+)"

The last line is the regexp that is causing the crash according to
the traceback.

>  File "/home/pokoli/.virtualenvs/roundup/lib/python3.7/sre_parse.py",
>line 1024, in parse_template
>    raise s.error('bad escape %s' % this, len(this))
>re.error: bad escape \w at position 2

Well \w is at position 2 of the raw string but that is supposed to be
a word character class according to the manual.

Let me know how the tests I suggested work.
msg6466 Author: [hidden] (pokoli) Date: 2019-04-22 07:53
Hello John, 

Thanks for your fast reply

On 22/4/19 2:08, John Rouillard wrote:
> Which version of python 3.7 are you running?
Python 3.7.3

>
>
> If in the root of the checked out tree, you run python3 demo.py (as
> specified in README.txt) without virtualenv, does that work or does it
> generate a crash?
Without virtualenv it generates the same crash.
>
>
> Also for testing purposes, what happens if from the root of the
> checked out tree you run:
>
>    mkdir example
>    python3 roundup/scripts/roundup_admin.py -i example install classic
Same crash without virtualenv.
>
>
> Do you still see the error? Try this both with and without virtualenv.
>
> I can't reproduce the crash with python 3.4 using the following
> command line:
>
>    python3 ../roundup/scripts/roundup_admin -i . install jinja2
>
> it prompts me for the database to use. No crashes.
>
> Looking at the traceback, I expect any attempt to import rest.py would
> cause this crash. 
Right, any import of the routes produces the crash.
>  Our CI pipeline tests python 3.4 through 3.6 and
> imports rest.py. I don't know of any change to regular expressions in
> 3.7 that would make \w (all word characters) an invalid character
> class.
>
>> Which have me the following exception:
>>
>> Traceback (most recent call last):
>>  File "/home/pokoli/.virtualenvs/roundup/lib/python3.7/sre_parse.py",
>> line 1021, in parse_template
>>    this = chr(ESCAPES[this][1])
>> KeyError: '\\w'
> I am guesing the \\w is coming from whatever ESCAPES is doing and it
> doubles the \.
This is the representation of the regular expression on which \w is
escaped, so it shows \\w
>
> Can you check rest.py and make sure that you have the following three
> lines starting at line 244:
>
> class Routing(object):
>     __route_map = {}
>     __var_to_regex = re.compile(r"<:(\w+)>")
>     url_to_regex = r"([\w.\-~!$&'()*+,;=:\%%]+)"
>
> The last line is the regexp that is causing the crash according to
> the traceback.
Yes, that's the lines that I have and the one I've patched in order to
fix them.
msg6467 Author: [hidden] (rouilj) Date: 2019-04-29 00:34
Applied in rev 42a713e36def. I still have no idea why this is needed, but 
at least it doesn't seem to have broken anything. Local tests and travis 
ci are passing with the change.
msg6468 Author: [hidden] (rouilj) Date: 2019-04-29 00:35
bah that should be rev d9a3f6957731
History
Date User Action Args
2019-04-29 00:35:31rouiljsetmessages: + msg6468
2019-04-29 00:34:12rouiljsetpriority: normal
assignee: rouilj
messages: + msg6467
status: new -> fixed
resolution: fixed
2019-04-22 07:53:33pokolisetmessages: + msg6466
2019-04-22 00:08:37rouiljsetnosy: + rouilj
messages: + msg6465
2019-04-21 09:12:54pokolisetfiles: + issue2551042.patch
keywords: + patch
messages: + msg6464
2019-04-21 08:53:44pokolicreate