On 22/08/15 17:42, anatoly techtonik wrote:
>     I personally find the documentation for the existing script
>     ('run_tests.py -h') a bit confusing, I'm not sure if I have ever
>     figured out how to run a single test case using the existing script,
>     and some options don't behave as expected (ie. 'run_tests.py --dir
>     test' works as one might expect, but 'run_tests.py --dir test/'
>     doesn't).
>
>
> Yes. I remember I was doing this part, but I am not sure if it was SCons
> or Roundup, so my requirement for test utility are:
> 1. list tests with quick filter (preview test selection)
> 2. run tests filtered by list
> 3. run single test
> 4. run tests that match substring

Here are a few examples - let me know if they don't cover what you are 
asking for (note: 'py.test' could be substituted for a new 
'run_tests.py' script):

Search for and run all tests in current working directory (recursively):

   py.test

Search for and run all tests in a directory/file (recursively):

  py.test test/test_dates.py

Search for and run tests that match a substring (classes or functions):

  py.test -k SessionTest
or
  py.test test/test_sqlite.py -k SessionTest

Run a specific test:

  py.test test/test_dates.py::DateTestCase::testSorting

To list the collected/matching tests, just add the '--collect-only' option.

>
> In ideal world I'd wrap pytest or similar project and strip all the
> option that are not used in a project unless user specifies -v -h
> (--verbose --help).
>

I'm not opposed to this, but I'm not sure if it is really worth the 
effort to maintain something that just obscures the options. Most 
"users" will probably just run the test script without any options to 
see if all the tests pass. The "developers" are the ones most likely to 
be playing around with options, in which case we just document the few 
options I've mentioned above that covers 90% of the use-cases. It is 
then up to the developers if they want to dig into the options further 
by using the '-h' option.

>     There are also a number of benefits that adopting pytest could
>     provide (less boilerplate, fixtures, using 'assert', etc.) if we
>     were to fully embrace it, but the one nice benefit I have included
>     in the current patchset is the support for test skipping decorators
>     it brings to python2.6 (not available in unittest before python2.7).
>     When running tests you now know how may tests have been skipped (and
>     why), instead of just being told some tests won't be run.
>
>
> This is a much better argumentation. Thanks. =) Are the tests become
> hard depending on pytest, or it is some general practice how to skip stuff?

 From python2.7 the unittest module has decorators for skipping test[1], 
but for the foreseeable future Roundup also supports python2.6. The 
pytest has its own test skipping decorators[2] which has support for 
python2.6, so if we were to take advantage of that we would be tied to 
using pytest. If we drop support for python2.6 then we could use the 
unittest skip decorators instead of the pytest ones.

>
>     If the consensus is to use another tool, then I am OK with that.
>
>
> I don't have a preference, but it would be nice to keep tests in some
> format that is independent of the tool, and for this specific problem
> choose tool that is not huge. =)
>

This just comes down to how much we want to embrace the pytest way of 
doing things. We could stick to using only what is available in unittest 
and keep things fairly portable, or take advantage of the nice extra 
features that pytest provides. Swings and roundabouts[3]

[1] 
https://docs.python.org/2/library/unittest.html#skipping-tests-and-expected-failures
[2] 
http://pytest.org/latest/skipping.html#marking-a-test-function-to-be-skipped
[3] http://idioms.thefreedictionary.com/it%27s+swings+and+roundabouts