The official Python documentation[1] references the Python Packaging
Guide[2] which recommends using setuptools instead of distutils.

[1] https://docs.python.org/2/distutils/
[2] https://packaging.python.org/en/latest/current/

Also see related:

  * issue2550866 - "pip install --editable ." fails
 *  issue2550898 - recent setup.py change breaks installation of scripts
without setuptools

Also note from setup.py[1]

  # FIXME: setuptools breaks the --manifest-only option to setup.py and
  # doesn't seem to generate a MANIFEST file. Since I'm not familiar with
  # the way setuptools handles the files to include I'm commenting this
  # for now -- Ralf Schlatterbeck

[1] http://sourceforge.net/p/roundup/code/ci/7cfd30cc/tree/setup.py

issue2550816 is the issue for the broken --manifest-only build issue.

I have closed all issues cross referenced from this ticket to make this ticket the one that will be used to fix all the related issues.

Also moved nosy lists from closed tickets to this one.

I've attached a patch series which I think resolves the problems found in the associated issues.

The first patch makes setuptools a hard dependencies for packaging/installing roundup (without the previous fallback to disutils). Using setuptools for python packaging is essentially a defacto standard these days, so I'm hoping adding setuptools as a hard dependency won't be a problem. It also includes documentation on using SOURCES.txt instead of the MANIFEST file.

The second patch fixes the scripts to work with the setuptools develop command (ie. `pip install --editable .`). It is essentially just re-applying a previously reverted commit.

The third patch is just removing code that is no longer needed because of the changes in the second patch.

The patch series is also viewable on my github repo clone: https://github.com/roundup-tracker/roundup/compare/master...jerrykan:setuptools

It might be a good idea if someone familiar with doing a release could do a quick review (particularly that the documentation is correct and the steps work) before the patches are applied.

Why would you like to introduce external dependency on setuptools?

On Mon, Mar 29, 2021 at 2:54 PM John Kristensen
<issues@roundup-tracker.org> wrote:
>
>
> John Kristensen added the comment:
>
> I've attached a patch series which I think resolves the problems found in the associated issues.
>
> The first patch makes setuptools a hard dependencies for packaging/installing roundup (without the previous fallback to disutils). Using setuptools for python packaging is essentially a defacto standard these days, so I'm hoping adding setuptools as a hard dependency won't be a problem. It also includes documentation on using SOURCES.txt instead of the MANIFEST file.
>
> The second patch fixes the scripts to work with the setuptools develop command (ie. `pip install --editable .`). It is essentially just re-applying a previously reverted commit.
>
> The third patch is just removing code that is no longer needed because of the changes in the second patch.
>
> The patch series is also viewable on my github repo clone: https://github.com/roundup-tracker/roundup/compare/master...jerrykan:setuptools
>
> It might be a good idea if someone familiar with doing a release could do a quick review (particularly that the documentation is correct and the steps work) before the patches are applied.
>
> _________________________________________________
> Roundup tracker <issues@roundup-tracker.org>
> <https://issues.roundup-tracker.org/issue2550899>
> _________________________________________________

Anatoly asked:

> Why would you like to introduce external dependency on setuptools?

 * Well it is recommended by the packaging guide.
   https://packaging.python.org/en/latest/current/

 * It is preferred by pip. Also this should allow
   pip install --editable to work

 * It can be used to make a single file deployable egg package.

 * It also builds wheels for single file deployable package.

There is a small ecosystem of programs that can be used to interact
with a roundup install. To date we have no way to distribute these
separately from the main roundup source install. Eggs and I think
wheels do allow this segmentation.

Most distributions include setuputils as a package. redhat and debian 
make the base of many distributions, but even alpine linux includes 
setuptools. So it's not a difficult requirement to fulfill.

I would like a fallback to distutils, but I am not motivated enough
to do it.

I will post to roundup-users to see if anybody raises any blockers.

Have a great day.

-- rouilj

Hi John:

I assume this:

  hg status --rev 1.6.0:tip | sed -ne 's/^A //p' | while read i ; \
   do echo $i; grep "$i" MANIFEST; done | uniq -c

should change to

  hg status --rev 2.0.0:tip| sed -ne 's/^A //p' | while read i ;       
   do echo $i; grep "$i" roundup.egg-info/SOURCES.txt; done | uniq -c

replacing MANIFEST. Also is looks like MANIFEST.in is still
valid for setuptools. I assume the reference for changing
MANIFEST.in should change from:

  http://docs.python.org/2/distutils/sourcedist.html#manifest-template

to:

  https://packaging.python.org/guides/using-manifest-in/#using-manifest-
in

correct?

Also I was able to use:

    python3 setup.py sdist --repository test --sign

successfully. I also did a purse sdist and tested the tarball. All
tests passed.

I also was able to:

    python3 setup.py bdist_egg --repository test --sign
    python3 setup.py bdist_wheel  --repository test --sign

and it seems to work on test.pypi.org.

Do you have a reference on how I can test these? Something similar to
(your) step 11 in RELEASE.txt would be good. I don't want egg (ahem)
on my face if I deploy these methods.

Have a great day.

-- rouilj

I vaguely remember some problems with setup.py years ago shortly before
a release I did. One was with encoding of the announcement text that did
not work with some older python version and is probably long since
fixed, but I don't remember if this was the only problem. I've not found
what I was searching when looking quickly through the commit history.

That said: We should make sure that installation continues to work with
python 2.7 for some time.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com

Ralf said:

> That said: We should make sure that installation continues to
> work with python 2.7 for some time.

I have setuptools installed for python 2.7 under ubuntu/debian.

I was able to generate a sdist, egg and whl using python2.
The untarred sdist passed a run of pytest with 2.7 and 3.6.
I don't test mysql or postgres, sqlite and anydbm are tested.

On Tue, Apr 06, 2021 at 02:51:10PM +0000, John Rouillard wrote:
> 
> John Rouillard added the comment:
> 
> Ralf said:
> 
> > That said: We should make sure that installation continues to
> > work with python 2.7 for some time.
> 
> I have setuptools installed for python 2.7 under ubuntu/debian.
> 
> I was able to generate a sdist, egg and whl using python2.
> The untarred sdist passed a run of pytest with 2.7 and 3.6.
> I don't test mysql or postgres, sqlite and anydbm are tested.

Thanks!
I'm using postgres but in almost all installations I install directly
from hg :-) I just remembered that I had to revert some changes to the
install process directly before a release but that was years ago.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office@runtux.com

Hello John,

On 6/4/21 7:20 am, John Rouillard wrote:
> I assume this:
> 
>    hg status --rev 1.6.0:tip | sed -ne 's/^A //p' | while read i ; \
>     do echo $i; grep "$i" MANIFEST; done | uniq -c
> 
> should change to
> 
>    hg status --rev 2.0.0:tip| sed -ne 's/^A //p' | while read i ;
>     do echo $i; grep "$i" roundup.egg-info/SOURCES.txt; done | uniq -c
> 
> replacing MANIFEST.

Looks like I missed that reference to the MANIFEST file, so yes your 
suggested change would seem to be correct and needs to be included in 
the patch.

> Also is looks like MANIFEST.in is still
> valid for setuptools. I assume the reference for changing
> MANIFEST.in should change from:
> 
>    http://docs.python.org/2/distutils/sourcedist.html#manifest-template
> 
> to:
> 
>    https://packaging.python.org/guides/using-manifest-in/#using-manifest-
> in
> 
> correct?

It seems the Python2 documentation is the only place that documents what 
should go in the MANIFEST.in file.

The packaging.python.org links seems to be a dead link (even though it 
is referenced in other parts of the documentation).

The Python3 documentation no longer seems to outline the list of 
commands that can be used in the MANIFEST.in file... and the note at the 
top gives the impression that distutils (and its documentation) is, or 
will be, deprecated in favour of setuptools.

Looking through the setuptools documentation there are a number of 
references to MANIFEST.in, but again no information about what should be 
in the file.

However it seems that setuptools does provide alternative methods to 
using MANIFEST.in, see:

     https://setuptools.readthedocs.io/en/latest/userguide/datafiles.html

My reading of that seem to indicate that we should be using 
`include_package_data=True` so that backwards compatibility with 
MANIFEST.in is kept even if we don't use that alternative options.

> Also I was able to use:
> 
>      python3 setup.py sdist --repository test --sign
> 
> successfully. I also did a purse sdist and tested the tarball. All
> tests passed.
> 
> I also was able to:
> 
>      python3 setup.py bdist_egg --repository test --sign
>      python3 setup.py bdist_wheel  --repository test --sign
> 
> and it seems to work on test.pypi.org.
> 
> Do you have a reference on how I can test these? Something similar to
> (your) step 11 in RELEASE.txt would be good. I don't want egg (ahem)
> on my face if I deploy these methods.

I'm not sure I follow what you want to test. Are you able to elaborate?

SeeYa,
John.

On 12/4/21 11:57 am, John Kristensen wrote:
> The packaging.python.org links seems to be a dead link (even though it
> is referenced in other parts of the documentation).

It seems this might actually be related to:

   https://github.com/pypa/packaging.python.org/issues/876

So, yes, It may be a viable alternative when the packaging.python.org 
documentation gets fixed.

Hi John:

In message <4d988495-a405-0242-af46-64e263b858ad@jerrykan.com>,
John Kristensen writes:
>Looks like I missed that reference to the MANIFEST file, so yes your 
>suggested change would seem to be correct and needs to be included in 
>the patch.

Wilco.

>> Also is looks like MANIFEST.in is still
>> valid for setuptools. I assume the reference for changing
>> MANIFEST.in should change from:
>> 
>>    http://docs.python.org/2/distutils/sourcedist.html#manifest-template
>> 
>> to:
>> 
>>    https://packaging.python.org/guides/using-manifest-in/#using-manifest-
>> in
>> 
>> correct?
>
>It seems the Python2 documentation is the only place that documents what 
>should go in the MANIFEST.in file.
>
>The packaging.python.org links seems to be a dead link (even though it 
>is referenced in other parts of the documentation).

Eep. I just got the link from the docs, never actually tried it 8-/.

>[...]
>However it seems that setuptools does provide alternative methods to 
>using MANIFEST.in, see:
>
>     https://setuptools.readthedocs.io/en/latest/userguide/datafiles.html
>
>My reading of that seem to indicate that we should be using 
>`include_package_data=True` so that backwards compatibility with 
>MANIFEST.in is kept even if we don't use that alternative options.

So add `include_package_data=True` at the end of the setup call after:

 data_files=data_files

Correct?

>> Also I was able to use:
>> 
>>      python3 setup.py sdist --repository test --sign
>> 
>> successfully. I also did a pure sdist and tested the tarball. All
>> tests passed.
>> 
>> I also was able to:
>> 
>>      python3 setup.py bdist_egg --repository test --sign
>>      python3 setup.py bdist_wheel  --repository test --sign
>> 
>> and it seems to work on test.pypi.org.
>> 
>> Do you have a reference on how I can test these? Something similar to
>> (your) step 11 in RELEASE.txt would be good. I don't want egg (ahem)
>> on my face if I deploy these methods.
>
>I'm not sure I follow what you want to test. Are you able to elaborate?

Well when I have an sdist tarball, I unpack it somewhere in /tmp, run
pytest against it. Then I run setup install --prefix (or --user) and
check that expected files are where they should be in the resulting
install.

What is the equivalent test mechanism for an egg or wheel to verify
that they are properly built and will install correctly.
E.G. a dummy example would be:

   python --wheel roundup.whl --entry roundup-admin

to verify that I can start roundup-admin from the wheel.

Roundup isn't just a module to be imported by an application. It is a
set of cli applications, so maybe egg/wheel distribution doesn't make
sense.

Thanks.

On 12/4/21 1:42 pm, John Rouillard wrote:
> 
> John Rouillard added the comment:
> 
> Hi John:
> 
> In message <4d988495-a405-0242-af46-64e263b858ad@jerrykan.com>,
> John Kristensen writes:
>> Looks like I missed that reference to the MANIFEST file, so yes your
>> suggested change would seem to be correct and needs to be included in
>> the patch.
> 
> Wilco.

I've updated the RELEASE.txt files to correctly refer to 
`roundup.egg-info/SOURCES.txt` in all places where `MANIFEST` was 
previously reference.

> 
>>> Also is looks like MANIFEST.in is still
>>> valid for setuptools. I assume the reference for changing
>>> MANIFEST.in should change from:
>>>
>>>     http://docs.python.org/2/distutils/sourcedist.html#manifest-template
>>>
>>> to:
>>>
>>>     https://packaging.python.org/guides/using-manifest-in/#using-manifest-
>>> in
>>>
>>> correct?
>>
>> It seems the Python2 documentation is the only place that documents what
>> should go in the MANIFEST.in file.
>>
>> The packaging.python.org links seems to be a dead link (even though it
>> is referenced in other parts of the documentation).
> 
> Eep. I just got the link from the docs, never actually tried it 8-/.

The packaging.python.org link has now been fixed, so I've also updated 
the RELEASE.txt to use this link. Though at some point we may want to 
move away from using the MANIFEST.in file (see below)

> 
>> [...]
>> However it seems that setuptools does provide alternative methods to
>> using MANIFEST.in, see:
>>
>>      https://setuptools.readthedocs.io/en/latest/userguide/datafiles.html
>>
>> My reading of that seem to indicate that we should be using
>> `include_package_data=True` so that backwards compatibility with
>> MANIFEST.in is kept even if we don't use that alternative options.
> 
> So add `include_package_data=True` at the end of the setup call after:
> 
>   data_files=data_files
> 
> Correct?

I tried running `python setup.py sdist` with and without this line added 
and it made no difference to the generated 
`roundup.egg-info/SOURCES.txt` file. So it seems the MANIFEST.in file is 
used even without this option. I also had a closer look at the 
documentation and noticed this comment:

 > If using the setuptools-specific include_package_data argument, files
 > specified by package_data will not be automatically added to the
 > manifest unless they are listed in the MANIFEST.in file.

My reading of this seems to indicate that by continuing to use 
MANIFEST.in we may end up having to maintaining the list of data files 
in multiple places. It may be better to move away from using the 
MANIFEST.in file to managing these data files in the setup.py file.

For the time being I've not included the `include_package_data=True` option.

The update commits can be found at 
https://github.com/roundup-tracker/roundup/compare/master...jerrykan:setuptools

On 12/4/21 1:42 pm, John Rouillard wrote:
>>> Also I was able to use:
>>>
>>>       python3 setup.py sdist --repository test --sign
>>>
>>> successfully. I also did a pure sdist and tested the tarball. All
>>> tests passed.
>>>
>>> I also was able to:
>>>
>>>       python3 setup.py bdist_egg --repository test --sign
>>>       python3 setup.py bdist_wheel  --repository test --sign
>>>
>>> and it seems to work on test.pypi.org.
>>>
>>> Do you have a reference on how I can test these? Something similar to
>>> (your) step 11 in RELEASE.txt would be good. I don't want egg (ahem)
>>> on my face if I deploy these methods.
>>
>> I'm not sure I follow what you want to test. Are you able to elaborate?
> 
> Well when I have an sdist tarball, I unpack it somewhere in /tmp, run
> pytest against it. Then I run setup install --prefix (or --user) and
> check that expected files are where they should be in the resulting
> install.
> 
> What is the equivalent test mechanism for an egg or wheel to verify
> that they are properly built and will install correctly.
> E.G. a dummy example would be:
> 
>     python --wheel roundup.whl --entry roundup-admin
> 
> to verify that I can start roundup-admin from the wheel.
> 
> Roundup isn't just a module to be imported by an application. It is a
> set of cli applications, so maybe egg/wheel distribution doesn't make
> sense.
> 
> Thanks.

Would the following commands cover the use-cases for testing the 
installation from the various file formats:

     pip install dist/roundup-2.0.0.tar.gz
     pip install dist/roundup-2.0.0-py3-none-any.whl
     python -m easy_install dist/roundup-2.0.0-py3.9.egg

I'm not sure about how to invoke commands from a python archives 
directly. Are there any specific use-cases where this would be useful? I 
would assume to do anything useful with roundup it would need to be 
installed.

Hi John:

In message <6bc554a3-c2ca-e9d3-1250-5a2aea397247@jerrykan.com>,
John Kristensen writes:
>On 12/4/21 1:42 pm, John Rouillard wrote:
>>>> [...] I also did a pure sdist and tested the tarball. All
>>>> tests passed.
>>>>
>> What is the equivalent test mechanism for an egg or wheel to verify
>> that they are properly built and will install correctly.
>> E.G. a dummy example would be:
>> 
>>     python --wheel roundup.whl --entry roundup-admin
>> 
>> to verify that I can start roundup-admin from the wheel.
>> 
>> Roundup isn't just a module to be imported by an application. It is a
>> set of cli applications, so maybe egg/wheel distribution doesn't make
>> sense.
>
>Would the following commands cover the use-cases for testing the 
>installation from the various file formats:
>
>     pip install dist/roundup-2.0.0.tar.gz
>     pip install dist/roundup-2.0.0-py3-none-any.whl
>     python -m easy_install dist/roundup-2.0.0-py3.9.egg
>
>I'm not sure about how to invoke commands from a python archives 
>directly. Are there any specific use-cases where this would be useful? I 
>would assume to do anything useful with roundup it would need to be 
>installed.

I am not sure. Does distributing wheels and eggs for roundup make any
sense? I thought wheels and eggs were an installation method. However
it looks like using wheels and eggs only make sense for code that
consists only of modules to be imported. If the package has command
line code (e.g. roundup-server, roundup-admin) then distributing
wheels or eggs is worthless.

That being said we do have some client side roundup libraries that are
meant to be used by roundup users writing their own code. E.G. the
xmlrpc client, see: https://issues.roundup-tracker.org/issue2550554.
I could see an equivalent REST library joining it in the future.

Am I interpreting the utility of egg/whl correctly?

Thanks.

Hello John,

On 13/4/21 1:15 pm, John Rouillard wrote:
> 
> John Rouillard added the comment:
> 
> Hi John:
> 
> In message <6bc554a3-c2ca-e9d3-1250-5a2aea397247@jerrykan.com>,
> John Kristensen writes:
>> On 12/4/21 1:42 pm, John Rouillard wrote:
>>>>> [...] I also did a pure sdist and tested the tarball. All
>>>>> tests passed.
>>>>>
>>> What is the equivalent test mechanism for an egg or wheel to verify
>>> that they are properly built and will install correctly.
>>> E.G. a dummy example would be:
>>>
>>>      python --wheel roundup.whl --entry roundup-admin
>>>
>>> to verify that I can start roundup-admin from the wheel.
>>>
>>> Roundup isn't just a module to be imported by an application. It is a
>>> set of cli applications, so maybe egg/wheel distribution doesn't make
>>> sense.
>>
>> Would the following commands cover the use-cases for testing the
>> installation from the various file formats:
>>
>>      pip install dist/roundup-2.0.0.tar.gz
>>      pip install dist/roundup-2.0.0-py3-none-any.whl
>>      python -m easy_install dist/roundup-2.0.0-py3.9.egg
>>
>> I'm not sure about how to invoke commands from a python archives
>> directly. Are there any specific use-cases where this would be useful? I
>> would assume to do anything useful with roundup it would need to be
>> installed.
> 
> I am not sure. Does distributing wheels and eggs for roundup make any
> sense? I thought wheels and eggs were an installation method. However
> it looks like using wheels and eggs only make sense for code that
> consists only of modules to be imported. If the package has command
> line code (e.g. roundup-server, roundup-admin) then distributing
> wheels or eggs is worthless.
> 
> That being said we do have some client side roundup libraries that are
> meant to be used by roundup users writing their own code. E.G. the
> xmlrpc client, see: https://issues.roundup-tracker.org/issue2550554.
> I could see an equivalent REST library joining it in the future.
> 
> Am I interpreting the utility of egg/whl correctly?

My understanding is that a tarball is source distribution while a wheel 
in more like a binary distribution. For a python package that contains C 
code, installing from a tarball will require having a compiler 
installed, while installing from a wheel doesn't (because the code has 
already been pre-compiled).

For a native python package there probably isn't isn't a whole of 
difference, but is sounds like pip will still convert the tarball to a 
wheel first, then install the wheel.

So I there there is value in providing both tarballs and wheels.

Wheels are intended to replace eggs, so I think we can just ignore those.