Comments on: How Much Test Coverage Is Enough?

By: How Much Test Coverage Is Enough Summa Blog | Weak Bladder

How Much Test Coverage Is Enough Summa Blog | Weak Bladder — Mon, 08 Jun 2009 01:17:23 +0000

[...] How Much Test Coverage Is Enough Summa Blog Posted by root 4 hours ago (http://www.summa-tech.com) Thanks for your comment 3 ladycoder march 24th 2009 at 6 33 pm powered by wordpress log in summa e sales summa tech com p 412 258 3300 Discuss | Bury | News | How Much Test Coverage Is Enough Summa Blog [...]

By: Adriana B.

Adriana B. — Fri, 10 Apr 2009 22:57:36 +0000

I found this article very interesting, and would like to suggest a theme for a future post: what factors need to be taken into consideration when deciding how much testing is needed?

I am Sr. Business Technology Analyst and my projects range from mission critical to cost-saving but not-capable of causing any damage if a defect goes live. I’d like to know how external factors, such the business impact of a software malfunction, might affect your recommendation of how much test coverage is ideal.

By: Marcus Vechiato

Marcus Vechiato — Mon, 06 Apr 2009 12:15:57 +0000

Your point of view about tests and costs are very interesting.

The common aproach is “Everyone do tests without show their coverage and never relate their costs”.

Nice work!

By: John Ferguson Smart

John Ferguson Smart — Thu, 26 Mar 2009 20:31:46 +0000

Nice write-up, Handerson. I like the graph.

There is a lot of debate on this topic, of course, . Tests, and in particular TDD and BDD-style testing, is not just to verify that the code does things right, but also that the code does the right thing. Certainly, don’t test getters and setters directly, but won’t these be tested indirectly by decent functional tests anyway?

Test coverage is a metric, a means of seeing how well your code is exercised by your tests. In practice, it is a reasonable indicator that tests are being taken seriously, but, in itself, it provides no guarantee that the tests themselves are of good quality. That said, I like seeing high test coverage in a project. I use it, in conjunction with code reviews, to check that testing practices are being done well.

By: tim

tim — Thu, 26 Mar 2009 09:09:39 +0000

Wow, subjective tests written against subjective code using subjective metrics

IMPERICAL DATA!

You are a god dude.

Cover every line of that subjective code with that subjective test using those subjectively mocked dependency injections.

A god dude. A god.

By: Jeroen Wenting

Jeroen Wenting — Thu, 26 Mar 2009 08:55:03 +0000

How do you define “100% test coverage” even in the context of unit testing?
Different coverage tools don’t always agree what constitutes “covered code”.
I’ve encountered situations in which one tool would tell me I had 100% coverage on a class, yet another tool counted that same class as being covered something like 60%.
The difference appeared to be that one tool reported non-executable code (like variable and field declarations without initialisers and lines containing only braces) as being uncovered code, and the other skipped them in its analysis.
Neither is tool fully correct here. To count a line containing only a closing brace or an empty statement as uncovered seems wrong, but that variable declaration? If that variable is used yet never initialised, there’s a potential for trouble so the coverage tool should ideally trace testcode to see if the variable initalisation is covered or not, and report the line as covered if and only if all possible uses for the variable are covered by tests. Most tools don’t seem to analyse that deep.

And as Ivo hints at, there’s code to be written in many situations that can be next to impossible to trigger in a test.
Mostly this will be error handling code that’s only triggered in extreme corner cases, that may rely on environmental factors that cannot be reliably replicated in a test (or can’t be without more effort than is warranted given the rarity of the need to execute it).
For such code a lexical analysis of correctness is a better test than spending possibly days writing a test for it when the code itself only gets executed once a year on an application running 24/7.

By: Ivo

Ivo — Thu, 26 Mar 2009 07:36:48 +0000

I think your graph is rather optimistic and should be much steeper towards the 100%: increased test coverage is strongly subject to the law of diminishing returns. There are at least two clear categories that prevent me from reaching 100%:
1) I’m not going to write unit tests for all those bean getters and setters that don’t do anything apart from accessing the field: that’s a lot of work without any benefit.
2) Defensive coding leads to null checks and catch blocks that are very hard to trigger. They are more like debug statements then real parts of the code. Consequently, there are always lines and branches that are very hard to cover by tests. Of course, you shouldn’t have a zillion of them; as always, it’s about balance.

By: LadyCoder

LadyCoder — Tue, 24 Mar 2009 23:33:00 +0000

I work on a team that practices TDD (Test Driven Development). We do not have 100% coverage of all of our code, and we are actually quite happy with the coverage that we have.

One important thing to note, 100% test coverage is a nice goal, but it isn’t the most important thing in terms of testing. What is more important than the amount of code you are covering, is the quality of the tests you have. I can write test code that will cover 100% of my code, but if it isn’t really testing anything useful, then it is useless.

Unfortunately, there aren’t many good tools (I don’t know of any, but I am sure there are some) that can really test the quality of your test code. So, looking at code coverage is a metric that a lot of people use to see how well your code is tested.

I am, by no means saying that code coverage is a completely useless statistic, but it needs to be taken with a grain of salt.

By: Handerson Gomes

Handerson Gomes — Tue, 24 Mar 2009 20:43:40 +0000

Good point Ariel, I should have added some references.

I’m using 100% test coverage is the context of Unit Testing. The bigger the number of lines of code in the application executed by unit tests, the bigger the coverage.

Some tools for Test Coverage Report that I’m familiar with:
Cobertura (http://cobertura.sourceforge.net/) is an excellent Open Source tool that can be easily used with Ant and Maven.

Clover (http://www.atlassian.com/software/clover/) is a commercial option with a more rich feature set.

Thanks for your comment.

By: ariel

ariel — Tue, 24 Mar 2009 20:27:06 +0000

You state in the first paragraph that 100% test coverage is easy to measure. How do you define “100% test coverage” and how do you measured it?