On 22/08/15 01:58, anatoly techtonik wrote:
> The argumentation "it's old - let's update it" doesn't seem too
> exciting. Right now you don't need anything to run tests, and
> if you want to add external dependency just because it is "new",
> I am -1.

A self-contained pytest script can be, and has been, generated to 
replace the existing script (see part 4 of the patchset[1]). So while 
the pytest generated script is larger than the old script there are no 
external dependencies required to run the tests.

>
> Do you have an example of real problems that you've
> encountered with existing test suite?
>

The existing run_tests.py script is a non-starter with python3, so if we 
want to even look at adding python3 support to Roundup the run_tests.py 
script would need to be updated to add python3 support. This could be 
done, but also begs the question of why are we maintaining our own test 
runner script? Why not just push that maintenance burden off to some 
other project (ie. pytest)?

As best as I can tell the existing script originates from the Zope 
project. I found a few dead links to what I assume was the original 
script, but it seems as though the Zope project now uses the 
zope.testing package. So updating the existing script from upstream also 
doesn't seem like an option.

I personally find the documentation for the existing script 
('run_tests.py -h') a bit confusing, I'm not sure if I have ever figured 
out how to run a single test case using the existing script, and some 
options don't behave as expected (ie. 'run_tests.py --dir test' works as 
one might expect, but 'run_tests.py --dir test/' doesn't).

If there are others like me who have problems with the existing script 
where do they go for more information or examples on how to use the 
script? Doing a search for "pytest" is likely to turn up more useful 
results than searching for "run_tests.py" or "zope test.py" (if they can 
figure out its origins). The pytest tool is widely used and well 
documented[2], which means there is already lots of information out 
there and people coming to Roundup are more likely to be already 
familiar with the tool.

There are also a number of benefits that adopting pytest could provide 
(less boilerplate, fixtures, using 'assert', etc.) if we were to fully 
embrace it, but the one nice benefit I have included in the current 
patchset is the support for test skipping decorators it brings to 
python2.6 (not available in unittest before python2.7). When running 
tests you now know how may tests have been skipped (and why), instead of 
just being told some tests won't be run.

I'm not trying to do a "hard sell" for pytest, but I do think we need to 
move away from the existing run_test.py script, hence why I thought it 
would be useful to put some patches together that would replace it with 
what I think is the best tool for the job.

If the consensus is to use another tool, then I am OK with that.

SeeYa,
John.

[1] 
http://issues.roundup-tracker.org/file1583/0004-Replace-existing-run_tests.py-script-with-a-pytest-s.patch

[2] http://pytest.org/latest/