steps to reproduce: 

virtualenv -p 2 ve2
source ve2/bin/activate
pip install roundup -> process hangs at 100% cpu
wget https://files.pythonhosted.org/packages/80/6d/3ccdd66551bb89b90758add63b8814b516d7471348675aa564aab032341f/roundup-2.1.0.tar.gz 
tar -xzvf roundup-2.1.0.tar.gz 
cd roundup-2.1.0
python setup.py install -> process hangs at 100% cpu

debugging yields that get_path('platlib') contains "lib64" instead of "lib" which is searched for in line 94 of setup.py

attached patch corrects this by using .startswith ("lib") which should be sufficient for all archs ?

regards,
marcus.

Hi Marcus:

In message <1636475687.17.0.08336466737.issue2551169@roundup.psfhosted.org>,
Marcus Priesch writes:
>debugging yields that get_path('platlib') contains "lib64" instead of
>"lib" which is searched for in line 94 of setup.py

Thanks for the patch. I don't have any time to reproduce it in the
next few days (and wouldn't be able to reproduce on gentoo).

Where is the infinite loop occurring? In my copy of setup.py the code
look like:

        # get the platform lib path.
        plp = get_path('platlib')
        # nuke suffix that matches lib/* and return prefix
        head, tail = os.path.split(plp)
        while tail != 'lib' and head != '':
            head, tail = os.path.split(head)
        if not head:
            head = sys.prefix
        return head

I don't see how that generates an infinite loop.  It should return
sys.prefix in your case. That code hasn't changed since 2.1.0 so....
Sys.prefix is still wrong, so the patch is probably a good idea, but
it's not an infinite loop. I want to make sure I solve the infinite
loop and not just remove a possible trigger for it.

Thanks and have a great day.

Hi John,

you are definitely right, we should solve the infinite loop ;)

its happening here:

 >          while tail != 'lib' and head != '':
 >              head, tail = os.path.split(head)

 >>> get_path("platlib")
'/home/pr/work/projects/5t-roundup/ve2/lib64/python2.7/site-packages'

it consumes everything (as no "lib" anywhere) and stays at head == "/"

 >>> os.path.split ("/")
('/', '')

so maybe exit the loop with "error" if head == "/" and tail == ""

regards,
marcus.

In message <c093fec4-798d-57b4-a6c6-df7ce48ad111@priesch.co.at>,
Marcus Priesch writes:
>you are definitely right, we should solve the infinite loop ;)
>
>its happening here:
>
> >          while tail != 'lib' and head != '':
> >              head, tail = os.path.split(head)
>
> >>> get_path("platlib")
>'/home/pr/work/projects/5t-roundup/ve2/lib64/python2.7/site-packages'
>
>it consumes everything (as no "lib" anywhere) and stays at head == "/"

Ah I have the wrong stop condition. Ok that's a good catch.

I wonder if we can do better on the lib check. Would:

  if tail not in [ 'lib', 'lib64' ] and head != '/':

work? I am thinking of a user libby who creates a virtual env under
/home/libby/roundup. Matching ^lib would make a mess of her home
directory.  (Yes, a libby born in 1964 could have username lib64
but....)

Thoughts?

Hi John,

>>>           while tail != 'lib' and head != '':
>>>               head, tail = os.path.split(head)

I really dont know what you are trying to achieve with this piece of
code, but shouldnt there be a more general approach to get there?

i am pretty sure that other python packagers have the same problem and
it is already solved somehow ? or am i wrong ?

why do you need to access anything above "platlib" ?

>>>>> get_path("platlib")
>> '/home/pr/work/projects/5t-roundup/ve2/lib64/python2.7/site-packages'

... doesnt it always end with .../pythonX.X/site-packages ?

then you could just strip the last two dirs to get there ...

searching for something like "lib" or "lib64" without context could 
always lead to problems ... like you stated ;)

> I wonder if we can do better on the lib check. Would:
> 
>    if tail not in [ 'lib', 'lib64' ] and head != '/':

yes, until someone comes up with lib128 or lib32 ;)

> work? I am thinking of a user libby who creates a virtual env under
> /home/libby/roundup. Matching ^lib would make a mess of her home
> directory.  (Yes, a libby born in 1964 could have username lib64
> but....)

according usernames: it will find lib or lib64 before the home dir:

'/home/libby/work/projects/5t-roundup/ve2/lib64/python2.7/site-packages'
'/home/lib64/work/projects/5t-roundup/ve2/lib64/python2.7/site-packages'

ps: i worked on roundup with ralf about 15 years ago, but rarely used it
in between ... and now i am back for some project we are working on
together ;)

regards,
marcus.

In message <32c162ba-03f6-40f9-f5d0-d23fde144716@priesch.co.at>,
Marcus Priesch writes:
>>>>           while tail != 'lib' and head != '':
>>>>               head, tail = os.path.split(head)
>
>I really dont know what you are trying to achieve with this piece of
>code, but shouldnt there be a more general approach to get there?
>
>i am pretty sure that other python packagers have the same problem and
>it is already solved somehow ? or am i wrong ?

It isn't solved, it's generally not a problem for them. Here is the
issue. Roundup includes man pages, templates and locale files. Normal
pip installations put all of that under the site-packages directory
(i.e. in platlib). Needless to say, man roundup-server wil not
work. Looking for templates means searching not in
/usr/share/roundup/templates, but in

  /usr/(lib|lib64)/python<version mumble>/site-packages/<sys.prefix>/share/roundup/templates

ugh. Also if you are looking for the locale files (so you can fix them
by undumping the .mo file), don't go to /usr/share/locale bacause they
won't be there.

Most pypi packages don't have this supporting material. So for them
this is a non-existant problem.

In setup.py, the execuables can be specified by defining
entry_points. Then pip creates stubs in /ur/bin or equivalent so the
commands can be run. There is nothing for these other types (man
pages, locale ...) of files.

That being said, Sunday I committed a second pass at allowing roundup
to find it's supporting fles under site-packages/<sys.prefix> commit
5ad7fb912227. But humans who want to use those files are still poorly
served. (Btw the current mechanism is the fix for a bug when somebody
as trying to look at how something was done in one of the templates.)

>why do you need to access anything above "platlib" ?

Because that is one way to determine where these other files should be
placed outside of the platform library. The function itself has a doc
string of:

   Get site specific prefix using --prefix, platform lib or
   sys.prefix.

>>>>>> get_path("platlib")
>>> '/home/pr/work/projects/5t-roundup/ve2/lib64/python2.7/site-packages'
>
>... doesnt it always end with .../pythonX.X/site-packages ?

One would hope so. I have no spec that says that is the case.  Also
--lib-prefix I think can override that (but the current heuristic
would probably fail as well). I seem to remember being very surprised
with a python2 pip installing stuff into dist-packages rather than
site-packages 8-/.

Looking for 'lib' is most definately a (bad) hack.

>then you could just strip the last two dirs to get there ...

I wonder if stripping the tail directory and looking for a 'share'
subdirectory would work better? Using the existence of a share
subdirectory as a marker seems problematic, but the current mechanism
isn't much better.

>ps: i worked on roundup with ralf about 15 years ago, but rarely used it
>in between ... and now i am back for some project we are working on
>together ;)

Welcome back. The more the merrier.

Marcus, thoughts on stripping path elements until you end up at a directory
with a share subdirectory?

Would that work in your case? Does the parent directory of lib64 have a share
subdirectory on gentoo?

Hi John,

yes, there is a share beside lib64, but i have a different thought on
the whole problem (thinking about it one or the other time) on a more
generic level:

wouldnt it make sense to install only the minimum necessary to run
roundup in a standardized way and to let all the other fancy stuff as an
exercise for the individual distribution's package maintainer ?

otherwise you will most likely end up covering all different
distribution's aspects in your setup.py - which you obviously dont want
to do ...

lets say when i use "pip install roundup" in a virtualenv (thats mostly
how i develop python) i dont care about man pages and stuff being
installed at the correct place - i.e. what would the correct place be
after all ? i would assume somewhere inside the virtualenv path (as
that's what virtualenvs are all about in the end) - but i am pretty sure
that "man" wont find it there.

on the other hand, when i install roundup with my distribution's
software installer (e.g. apt install roundup) i would expect to have all
things installed in the correct places and to let "man" find the
installed man pages (if any) ... but thats clear the duty of the package
maintainer.

when i install as superuser using "pip install roundup" or "setup.py
install" (which in my opinion nobody should do at all) - everything goes
into /usr/local ... and ... hmmm, maybe i should be prepared that not
everything ends up being installed in the correct place (if using gentoo
at least ;) - but these aspects again should be covered with correctly
working setuptools et al (correctly patched by the gentoo devs so that
these files end up being installed in the correct place and are
accessible by "man") ...

maybe take a look at how django handles it - as i think it is
comaparable in terms of setup.py needs.

at least it installs one script 'django-admin' and a man page for it:

	https://github.com/django/django

regards,
marcus.

>wouldnt it make sense to install only the minimum necessary to run
>roundup in a standardized way and to let all the other fancy stuff as an
>exercise for the individual distribution's package maintainer ?

No distribution is packaging roundup. Debian gave up on it a number of
years ago. There is one appliance for roundup, but that's the only
non-pip based deployment mechanism I know of.

One of the big issues with packaging roundup is how customizable the
tracker is. You can't just blindly upgrade. Version to version
upgrades are usually not that bad, but I have one person who is
jumping from 1.3 something to 2.1 who is having a hard time of it.

>when i install as superuser using "pip install roundup" or "setup.py
>install" (which in my opinion nobody should do at all) - everything goes
>into /usr/local

Not necessarily depends on the distribution, /usr/lib/pythonxx/site-packages is also 
possible.

In my (admittedly small sample) 80% of the installs are using
setup.py in a VM spun up to support roundup. The initial complaint
that the man pages, templates and locales were buried came from just
such an install.

Also we have a docker install as well where the man command etc. should
just work.

Distutils used to install manpages, etc. in the normal system
places. Changing to setuptools changed that. Setuptools only supports
creating executables wrappers in a place on the path. It has no
support for man, locale or other files. That has to be done in the
setup.py installer.

>but these aspects again should be covered with correctly
>working setuptools et al (correctly patched by the gentoo devs so that
>these files end up being installed in the correct place and are
>accessible by "man") ...

Setuptools provides no such mechanism and will not.  I have lost the
link to the multi year old ticket/forum multi where this issue was
discussed and fixes were proposed but there was no consensus by
developers on fixing this.

>maybe take a look at how django handles it - as i think it is
>comaparable in terms of setup.py needs.
>
>at least it installs one script 'django-admin' and a man page for it:
>
>	https://github.com/django/django

From the setup file, it looks like the man page is just buried on the
system somewhere. I don't see any indication that django's setup.py is
following the documented steps to override man page placement under
the python install directory.

Sadly Os X on mac's put things in really odd places.

The install directory is:

   /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages

and the man directory is /usr/share/man/. So regardless of what
solution happens here won't work in all cases.

> No distribution is packaging roundup. Debian gave up on it a number of

not al all ;) - in gentoo you have it:

$ equery l -p roundup
  * Searching for roundup ...
[-P-] [  ] www-apps/roundup-2.0.0:0

> One of the big issues with packaging roundup is how customizable the
> tracker is. You can't just blindly upgrade. Version to version
> upgrades are usually not that bad, but I have one person who is
> jumping from 1.3 something to 2.1 who is having a hard time of it.

gentoo has webapp-config for this kind of things - i think it would take 
some effort to have roundup supported there:

	https://wiki.gentoo.org/wiki/Webapp-config

but the problem i think is more related to customized tracker settings...

>> when i install as superuser using "pip install roundup" or "setup.py
>> install" (which in my opinion nobody should do at all) - everything goes
>> into /usr/local
> 
> Not necessarily depends on the distribution, /usr/lib/pythonxx/site-packages is also
> possible.

yeah, but then you have mixed your "user space" with your "distribution 
space" - you can easily run into the problem that some other (system) 
package updates you some dependency and your stuff doesnt work any more 
... ;)

> In my (admittedly small sample) 80% of the installs are using
> setup.py in a VM spun up to support roundup. The initial complaint
> that the man pages, templates and locales were buried came from just
> such an install.

hm ... imho if you use tarball + setup.py you have everything at hand 
and you shouldnt need a man page ... at least its only for roundup-admin 
which is covered nicely on the web ... ;) - anyway.

> Also we have a docker install as well where the man command etc. should
> just work.

cool, but here you can have an install script which does the correct 
things for your "distribution" which is running in the container.

> Distutils used to install manpages, etc. in the normal system
> places. Changing to setuptools changed that. Setuptools only supports
> creating executables wrappers in a place on the path. It has no
> support for man, locale or other files. That has to be done in the
> setup.py installer.

hmm, maybe they dropped it just because of the complexity this creates, 
and the things we are discussing here ;)

i just took a look at django, as they also have lots of template files 
and static files you can use in you project or not.

its all in a package there and stuff gets installed by using 
"include_package_data" - maybe this could be a way to go, to not install 
templates into /usr/share/roundup but have them as python packages and 
when you initialise a tracker stuff gets copied to your tracker-home ?

>> but these aspects again should be covered with correctly
>> working setuptools et al (correctly patched by the gentoo devs so that
>> these files end up being installed in the correct place and are
>> accessible by "man") ...
> 
> Setuptools provides no such mechanism and will not.  I have lost the
> link to the multi year old ticket/forum multi where this issue was
> discussed and fixes were proposed but there was no consensus by
> developers on fixing this.

ok.

>> 	https://github.com/django/django
> 
>>From the setup file, it looks like the man page is just buried on the
> system somewhere. I don't see any indication that django's setup.py is
> following the documented steps to override man page placement under
> the python install directory.

ok, i think it is done in the scripts/rpm-install.sh ;)

> Sadly Os X on mac's put things in really odd places.
> 
> The install directory is:
> 
>     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
> 
> and the man directory is /usr/share/man/. So regardless of what
> solution happens here won't work in all cases.

YEAH - but i dont think you can support all these systems ... ;) - i 
think there is also an Amiga port of python existing ;) and of course 
winxxx...

maybe just try to install the man page into /usr/local, /usr depending 
on uid of the installer and give a warning if it fails.

and package templates like django does ?

just installed django and roundup here in gentoo, below are some 
locations of installed files:

django (locale files in the python packages)
 >>> /usr/lib/python3.8/site-packages/django/conf/locale/de/
 >>> /usr/lib/python3.8/site-packages/django/conf/locale/de/LC_MESSAGES/
 >>> 
/usr/lib/python3.8/site-packages/django/conf/locale/de/LC_MESSAGES/django.po
 >>> 
/usr/lib/python3.8/site-packages/django/conf/locale/de/LC_MESSAGES/django.mo
 >>> /usr/lib/python3.8/site-packages/django/conf/locale/de/__init__.py
 >>> /usr/lib/python3.8/site-packages/django/conf/locale/de/formats.py
 >>> /usr/bin/django-admin (symlink)
 >>> /usr/share/doc/django-3.2.8/
 >>> /usr/share/doc/django-3.2.8/README.rst.bz2
 >>> /usr/share/doc/django-3.2.8/AUTHORS.bz2
 >>> /usr/share/bash-completion/completions/django-admin

roundup:
 >>> /usr/lib/python3.9/site-packages/roundup/
 >>> /usr/share/roundup/cgi-bin/roundup.cgi
 >>> /usr/share/roundup/templates/
 >>> /usr/share/man/man1/roundup-server.1.bz2
 >>> /usr/share/man/man1/roundup-demo.1.bz2
 >>> /usr/share/man/man1/roundup-admin.1.bz2
 >>> /usr/share/man/man1/roundup-mailgw.1.bz2
 >>> /usr/share/locale/es/LC_MESSAGES/roundup.mo
 >>> /usr/share/doc/roundup-2.0.0/
 >>> /usr/share/doc/roundup-2.0.0/FAQ.txt.bz2
 >>> /usr/share/doc/roundup-2.0.0/CHANGES.txt.bz2
 >>> /usr/bin/roundup-admin (symlink)

peeking into the ebuild files, both just run some 
"distutils-r1_python_install_all" command and merge the result into the 
rootfs -  django doing some additional bashcompletion stuff.

so there are no man pages installed in the django case ... hm ... if 
they dont do it, should we ?

looking at the gentoo package path i see the discrepancy:

dev-python/django is merely a dev stuff whereas
www-apps/roundup clearly is a web-application - which someone could 
argue - should have a manpage ;)

so my proposal:

  * locales should be handled like in django
  * templates should be packages - or at least package data ?
   -> so everything whats needed is in the python path

  * man pages get only installed by the distribution-package
    -> or try in setup.py like stated above and give a hint when
       it fails

anyway ... think to have to work on the frontend again :(

cheers,
marcus.

Hi Marcus:

Sorry, I missed your update.

You said:
>roundup:
> >>> /usr/lib/python3.9/site-packages/roundup/
> >>> /usr/share/roundup/cgi-bin/roundup.cgi
> >>> /usr/share/roundup/templates/
> >>> /usr/share/man/man1/roundup-server.1.bz2
> >>> /usr/share/man/man1/roundup-demo.1.bz2
> >>> /usr/share/man/man1/roundup-admin.1.bz2
> >>> /usr/share/man/man1/roundup-mailgw.1.bz2
> >>> /usr/share/locale/es/LC_MESSAGES/roundup.mo
> >>> /usr/share/doc/roundup-2.0.0/
> >>> /usr/share/doc/roundup-2.0.0/FAQ.txt.bz2
> >>> /usr/share/doc/roundup-2.0.0/CHANGES.txt.bz2
> >>> /usr/bin/roundup-admin (symlink)

This looks broken, it's missing the roundup-{server,demo,mailgw}
commands unless you elided them.

>so there are no man pages installed in the django case ... hm ... if 
>they dont do it, should we ?

I claim yes. Man pages should be included by default if the command is
included.

>looking at the gentoo package path i see the discrepancy:
>
>dev-python/django is merely a dev stuff whereas
>www-apps/roundup clearly is a web-application - which someone could 
>argue - should have a manpage ;)

I haven't dealt with django in many years. Also we were installing
from source so....

>so my proposal:
>
>  * locales should be handled like in django

Well they kind of are. I modified the installer before this ticket
came up to search the normal install path for a wheel install of
roundup.

>  * templates should be packages - or at least package data ?
>   -> so everything whats needed is in the python path

It's too difficult to publish each of 5 trackers as a separate version
controlled package. Doing a once a year release of roundup is still a
multi week process.

>  * man pages get only installed by the distribution-package
>    -> or try in setup.py like stated above and give a hint when
>       it fails

The man pages are often the most up to date documentation on
commands. I know the docs used to point to the man pages for detailed
usage and examples. Maybe we can do something about that when
overhauling the docs but not now.

Docs and error handling in setuptools/setup.py isn't great. I have
tried making it more flexible in the past and ended up crashing
setuptools itself. Ralf IIRC had his own pain due to setuptools that
results in going back to distutils as he couldn't figure out how to fix
it.

Also we will soon be unable to install at all with the distutils
hierarchy being removed.

I am going to commit what I currently have that gets rid of the
infinite loop, supports lib/lib64. If we have to revisit this in the
future, that would be a good time to consider looking for a marker
directory like share or share/man.