Roundup Tracker - Issues

Issue 2550905

classification
Title: Improving the test coverage of the Roundup test suite
Type: Severity: normal
Components: Versions:
process
Status: new Resolution:
Dependencies: Superseder:
Assigned To: pefu Nosy List: ber, pefu, rouilj
Priority: normal Keywords: Effort-High

Created on 2016-03-08 12:28 by pefu, last changed 2019-10-14 11:40 by rouilj.

Files
File name Uploaded Description Edit Remove
roundup_coverage_report.txt pefu, 2016-03-08 12:28
Messages
msg5488 Author: [hidden] (pefu) Date: 2016-03-08 12:28
After installing the latest py.test and the pytest-cov plugin using pip
I ran the following command in my hg Roundup source repository dir::

  py.test --cov=.

The output of this test run is attached as roundup_coverage_report.txt

My proposal is that we developers try to increase the test coverage
on the roundup code base over time.  
This is an engineering task which will not be easy.  
But I believe it will pay off in the end.  
I create this general issue here now so we have one to refer to in
commit messages.
msg5490 Author: [hidden] (ber) Date: 2016-03-09 15:48
On Tuesday 08 March 2016 at 13:28:37, Peter Funk wrote:
> My proposal is that we developers try to increase the test coverage
> on the roundup code base over time.  

It is always good to improve automatic tests.
But 100% test coverage usually is a mistake,
it has to be a reasonable coverage,
balancing out the drawbacks of too many tests.
(One drawback is that too many tests make
are additional code that has defects and raise maintenance costs 
and the bars for changing the code.)
msg5491 Author: [hidden] (pefu) Date: 2016-03-09 15:58
Bernhard Reiter, 09.03.2016, 15:48:
> On Tuesday 08 March 2016 at 13:28:37, Peter Funk wrote:
> > My proposal is that we developers try to increase the test coverage
> > on the roundup code base over time.  
> 
> It is always good to improve automatic tests.
> But 100% test coverage usually is a mistake,
> it has to be a reasonable coverage,
> balancing out the drawbacks of too many tests.
> (One drawback is that too many tests make
> are additional code that has defects and raise maintenance costs 
> and the bars for changing the code.)

Yes, I agree:  
Writing maintainable test code is hard and requires engineering.
And I've to admit that I've still a lot to learn in this area.
When I studied computer science TDD wasn't even invented yet! ☺

Regards, Peter Funk
-- 
Peter Funk, home: ✉Oldenburger Str.86, D-27777 Ganderkesee
mobile:+49-179-640-8878 phone:+49-421-20419-0 <http://www.artcom-gmbh.de/>
office: ArtCom GmbH, ✉Haferwende 2, D-28357 Bremen, Germany
msg5681 Author: [hidden] (rouilj) Date: 2016-06-29 02:49
I am in the process of adding some tests for some do_*
functions in admin.py. Do we have a percentage coverage
target, or any criteria for what we should check first?

My particular round of additional testing was due to
issue2550572 bug in roundup_admin. So I was testing the
code paths that failed.

-- rouilj
msg5682 Author: [hidden] (ber) Date: 2016-06-29 08:16
| Do we have a percentage coverage
| target, or any criteria for what we should check first?

AFAIK we don't.
msg5683 Author: [hidden] (pefu) Date: 2016-06-29 10:03
Hello all,

Bernhard Reiter added the comment29.06.2016 um 08:16:
> | Do we have a percentage coverage
> | target, or any criteria for what we should check first?
> 
> AFAIK we don't.

Trying to achieve a certain coverage percentage would not
automatically make Roundup a better software.  

However:  IMHO It would help the project a lot if we could find 
and agree on some way (for example a naming scheme ?) so that 
we can distinguish unit tests from component or integration tests.

Also it would help, if unit tests aleady exist, to document where
to find those unit tests belonging to certain parts of the 
roundup code base.

I'm feeling myself still not experienced enough with test driven 
development to propose the right steps in this direction.
 
Best Regards, Peter Funk
msg6741 Author: [hidden] (rouilj) Date: 2019-10-13 22:38
Adding the info on the CI pipeline Jerrycan set up to
allow us to see code coverage.

Git mirror of roundup is at:

  https://github.com/roundup-tracker/roundup

Travis CI is at:

  https://travis-ci.org/roundup-tracker/roundup/builds

codecov is at:

  https://codecov.io/gh/roundup-tracker/roundup

coverage is at 68.42%.

Areas that are uncovered are:

  admin.py (and possibly by extension display and progress in support.py)
  cgi/cgitb.py
  cgi/templating.py
  cgi/ZTUtils/
  cgi/client.py
  cgi/TAL/
  cgi/actions.py

which have significant code (> 100 lines) and are below the average.
msg6743 Author: [hidden] (ber) Date: 2019-10-14 07:59
Note: While it is good to have some code coverage, 100% is considered
suboptimal (in the only research I could find, reported by Robert Glass,
but it fits my personal experience).
msg6744 Author: [hidden] (rouilj) Date: 2019-10-14 11:40
Hi Bern:

In message <1571039971.84.0.988568062794.issue2550905@roundup.psfhosted.org>,
Bernhard Reiter writes:
>Bernhard Reiter added the comment:
>
>Note: While it is good to have some code coverage, 100% is considered
>suboptimal (in the only research I could find, reported by Robert Glass,
>but it fits my personal experience).

I agree. I would be happy with 80-90% coverage. Some areas I would
expect not to be tested:

  exception handlers - e.g triggering random disconnects from remote services
           is probably not going to happen.

  debugging code - probably routines used by the exception handling code.

I have my demo instance running under the profiler to see what code is
actually being hit often. That would be the first code that needs to
be put under test.

We need testing for external interfaces: web, email (pop, imap) and
the way to do these tests is not obvious. We also run the risk of
mocking/developing an external test harness that is broken. So
"passing" code that uses these harnesses is actually broken in real
world use.

With the python 3 changes to the web output, tal (which has almost no
coverage), jinja and chameleon which have no coverage) we are at
greater risk of producing unusable code. It is better to find the
breakage while the original modifiers are around rather than delay
months, years. As years later we have lost the reason for the
modifications if the original people are no longer available.
History
Date User Action Args
2019-10-14 11:40:28rouiljsetmessages: + msg6744
2019-10-14 07:59:31bersetmessages: + msg6743
2019-10-13 22:38:52rouiljsetmessages: + msg6741
2016-06-29 10:03:21pefusetmessages: + msg5683
2016-06-29 08:16:06bersetmessages: + msg5682
2016-06-29 02:49:24rouiljsetnosy: + rouilj
messages: + msg5681
2016-03-09 15:58:36pefusetmessages: + msg5491
2016-03-09 15:48:28bersetmessages: + msg5490
2016-03-08 12:28:37pefucreate