After installing the latest py.test and the pytest-cov plugin using pip
I ran the following command in my hg Roundup source repository dir::

  py.test --cov=.

The output of this test run is attached as roundup_coverage_report.txt

My proposal is that we developers try to increase the test coverage
on the roundup code base over time.  
This is an engineering task which will not be easy.  
But I believe it will pay off in the end.  
I create this general issue here now so we have one to refer to in
commit messages.

On Tuesday 08 March 2016 at 13:28:37, Peter Funk wrote:
> My proposal is that we developers try to increase the test coverage
> on the roundup code base over time.  

It is always good to improve automatic tests.
But 100% test coverage usually is a mistake,
it has to be a reasonable coverage,
balancing out the drawbacks of too many tests.
(One drawback is that too many tests make
are additional code that has defects and raise maintenance costs 
and the bars for changing the code.)

Bernhard Reiter, 09.03.2016, 15:48:
> On Tuesday 08 March 2016 at 13:28:37, Peter Funk wrote:
> > My proposal is that we developers try to increase the test coverage
> > on the roundup code base over time.  
> 
> It is always good to improve automatic tests.
> But 100% test coverage usually is a mistake,
> it has to be a reasonable coverage,
> balancing out the drawbacks of too many tests.
> (One drawback is that too many tests make
> are additional code that has defects and raise maintenance costs 
> and the bars for changing the code.)

Yes, I agree:  
Writing maintainable test code is hard and requires engineering.
And I've to admit that I've still a lot to learn in this area.
When I studied computer science TDD wasn't even invented yet! ☺

Regards, Peter Funk
-- 
Peter Funk, home: ✉Oldenburger Str.86, D-27777 Ganderkesee
mobile:+49-179-640-8878 phone:+49-421-20419-0 <http://www.artcom-gmbh.de/>
office: ArtCom GmbH, ✉Haferwende 2, D-28357 Bremen, Germany

I am in the process of adding some tests for some do_*
functions in admin.py. Do we have a percentage coverage
target, or any criteria for what we should check first?

My particular round of additional testing was due to
issue2550572 bug in roundup_admin. So I was testing the
code paths that failed.

-- rouilj

| Do we have a percentage coverage
| target, or any criteria for what we should check first?

AFAIK we don't.

Hello all,

Bernhard Reiter added the comment29.06.2016 um 08:16:
> | Do we have a percentage coverage
> | target, or any criteria for what we should check first?
> 
> AFAIK we don't.

Trying to achieve a certain coverage percentage would not
automatically make Roundup a better software.  

However:  IMHO It would help the project a lot if we could find 
and agree on some way (for example a naming scheme ?) so that 
we can distinguish unit tests from component or integration tests.

Also it would help, if unit tests aleady exist, to document where
to find those unit tests belonging to certain parts of the 
roundup code base.

I'm feeling myself still not experienced enough with test driven 
development to propose the right steps in this direction.
 
Best Regards, Peter Funk

Adding the info on the CI pipeline Jerrycan set up to
allow us to see code coverage.

Git mirror of roundup is at:

  https://github.com/roundup-tracker/roundup

Travis CI is at:

  https://travis-ci.org/roundup-tracker/roundup/builds

codecov is at:

  https://codecov.io/gh/roundup-tracker/roundup

coverage is at 68.42%.

Areas that are uncovered are:

  admin.py (and possibly by extension display and progress in support.py)
  cgi/cgitb.py
  cgi/templating.py
  cgi/ZTUtils/
  cgi/client.py
  cgi/TAL/
  cgi/actions.py

which have significant code (> 100 lines) and are below the average.

Note: While it is good to have some code coverage, 100% is considered
suboptimal (in the only research I could find, reported by Robert Glass,
but it fits my personal experience).

Hi Bern:

In message <1571039971.84.0.988568062794.issue2550905@roundup.psfhosted.org>,
Bernhard Reiter writes:
>Bernhard Reiter added the comment:
>
>Note: While it is good to have some code coverage, 100% is considered
>suboptimal (in the only research I could find, reported by Robert Glass,
>but it fits my personal experience).

I agree. I would be happy with 80-90% coverage. Some areas I would
expect not to be tested:

  exception handlers - e.g triggering random disconnects from remote services
           is probably not going to happen.

  debugging code - probably routines used by the exception handling code.

I have my demo instance running under the profiler to see what code is
actually being hit often. That would be the first code that needs to
be put under test.

We need testing for external interfaces: web, email (pop, imap) and
the way to do these tests is not obvious. We also run the risk of
mocking/developing an external test harness that is broken. So
"passing" code that uses these harnesses is actually broken in real
world use.

With the python 3 changes to the web output, tal (which has almost no
coverage), jinja and chameleon which have no coverage) we are at
greater risk of producing unusable code. It is better to find the
breakage while the original modifiers are around rather than delay
months, years. As years later we have lost the reason for the
modifications if the original people are no longer available.

Peter are you still around and interested in this still?

-- rouilj

See also issue2551101 for testing using a live server instance
triggered via wsgi.

At this time we have coverage of 76%. Still poor coverage of the templating
languages.