How Much Test Coverage Is Enough?
March 24th, 2009 Handerson Gomes, Consultant (email the author)
There is a lot of talk these days about 100% test coverage and if we should strive to reach it. I honestly think that 100% test coverage is a hot topic because it can be easily measured.
To me the most important question is how much test coverage is enough.
I would like to discuss test coverage by making an analogy to exercise.
We all know that exercising, like testing, is a good thing. We also know that there is a range that dictates the minimum and maximum amount of exercise that will be beneficial. Too little exercise and nothing happens, too much and it can hurt and even cause permanent damage.
How much exercise is enough? Will I be healthy if I start running some miles every week? Probably yes. Would I be even healthier if I (try) to run 1000 miles every week? Most likely not.
It is the same thing with testing. Can I definitely say that a product with 60% test coverage is “healthier” than a product with 1%? Not definitely, but plausibly yes, to the point that I can bet on it, and I’m not a gambler. How about a product with 40% and one with 50% test coverage? From a “healthy” perspective it is hard to assume that one is better than the other by looking only at the coverage. (A theme for another post would be to discuss if High Test Coverage is a cause or consequence of the fitness of a product).
It is easy to agree that a product with 0% test coverage should strive to reach, let’s say 10%. A lot of people will agree that moving from 10% to 20% is also feasible. But when do you stop?
So the question now is about Return on Investment. How much “healthier” does the product get for each test invested? Although I don’t have statistical data, my experience tells me that the cost is not linear. There will be an initial upfront cost (actually investment) to setup the framework for testing. Some projects will be easier other more complicated, depending of the nature of the code base, types of integration, developers skill, etc. After this initial bump the cost is almost linear and most methods will have the same cost to be tested. In most cases teams will postpone some methods due to the difficulty to test them. Maybe the method needs to connect to a third party system, maybe the test cannot be repeated or other reasons. These are the tests responsible for the increase in the cost of coverage. To reach 100% test coverage an extra amount of resources will have to be invested.
Test Coverage Cost
The same way having an active life is a good way to increase our overall health, having an active testing life is a good way to improve our product “testability” and coverage. TDD is a well known technique that can definitely improve the usability of the API, reduce complexity and naturally improve test coverage.
But most importantly, define what the goals of high code coverage are for your organization, and create ways to measure it. If you don’t know what you’re trying to achieve with high coverage, how can you tell your “workout” is working? What do you want to accomplish by increasing the test coverage?
- reduced number of bugs open after each release?
- reduced number of time spent detecting and fixing bugs?
- increased efficiency of the team (delivering more with less)?
There are different ways to achieve the same goals, and from the business perspective the best one is the least expensive one.
It is well known that the earlier a bug is found the cheaper it is to fix it. High test coverage is one of the best ways to detect bugs earlier on and to keep them at bay, but is not one size fits all.
Sometimes there are bigger threats to a project that requires investing the limited amount of time and resources in different initiatives.
Every test created is a test that eventually will require maintenance. Therefore it is important to choose what to test wisely.
It is not because your test coverage can be measured that it must be chase the 100% mark, but the reality is that testing is like exercising, most of us actually need to do more.
Feel free to use the comments section to share your thoughts on this topic.
Entry Filed under: Agile and Development
Pages
Categories
- Agile and Development
- Application Modernization
- Cloud Applications
- Process Integration
- Summa
- Technology + Healthcare
- Uncategorized
Most Recent Posts
- Summa Is Award Finalist at IBM’s Impact 2012
- Working with JqGrid and ASP.NET MVC - Setting up a base jqgrid parameters class
- Rebase a Slave Mercurial Repo to a Subversion Master
- The Social Enterprise Part 2 – How To Set Up Chatter In Less Than 30 Minutes
- Implement Clear Governance for BRMS
Feeds
Calendar
| M | T | W | T | F | S | S |
|---|---|---|---|---|---|---|
| « Feb | Apr » | |||||
| 1 | ||||||
| 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| 9 | 10 | 11 | 12 | 13 | 14 | 15 |
| 16 | 17 | 18 | 19 | 20 | 21 | 22 |
| 23 | 24 | 25 | 26 | 27 | 28 | 29 |
| 30 | 31 | |||||
10 Comments Add your own
1. ariel | March 24th, 2009 at 3:27 pm
You state in the first paragraph that 100% test coverage is easy to measure. How do you define “100% test coverage” and how do you measured it?
2. Handerson Gomes | March 24th, 2009 at 3:43 pm
Good point Ariel, I should have added some references.
I’m using 100% test coverage is the context of Unit Testing. The bigger the number of lines of code in the application executed by unit tests, the bigger the coverage.
Some tools for Test Coverage Report that I’m familiar with:
Cobertura (http://cobertura.sourceforge.net/) is an excellent Open Source tool that can be easily used with Ant and Maven.
Clover (http://www.atlassian.com/software/clover/) is a commercial option with a more rich feature set.
Thanks for your comment.
3. LadyCoder | March 24th, 2009 at 6:33 pm
I work on a team that practices TDD (Test Driven Development). We do not have 100% coverage of all of our code, and we are actually quite happy with the coverage that we have.
One important thing to note, 100% test coverage is a nice goal, but it isn’t the most important thing in terms of testing. What is more important than the amount of code you are covering, is the quality of the tests you have. I can write test code that will cover 100% of my code, but if it isn’t really testing anything useful, then it is useless.
Unfortunately, there aren’t many good tools (I don’t know of any, but I am sure there are some) that can really test the quality of your test code. So, looking at code coverage is a metric that a lot of people use to see how well your code is tested.
I am, by no means saying that code coverage is a completely useless statistic, but it needs to be taken with a grain of salt.
4. Ivo | March 26th, 2009 at 2:36 am
I think your graph is rather optimistic and should be much steeper towards the 100%: increased test coverage is strongly subject to the law of diminishing returns. There are at least two clear categories that prevent me from reaching 100%:
1) I’m not going to write unit tests for all those bean getters and setters that don’t do anything apart from accessing the field: that’s a lot of work without any benefit.
2) Defensive coding leads to null checks and catch blocks that are very hard to trigger. They are more like debug statements then real parts of the code. Consequently, there are always lines and branches that are very hard to cover by tests. Of course, you shouldn’t have a zillion of them; as always, it’s about balance.
5. Jeroen Wenting | March 26th, 2009 at 3:55 am
How do you define “100% test coverage” even in the context of unit testing?
Different coverage tools don’t always agree what constitutes “covered code”.
I’ve encountered situations in which one tool would tell me I had 100% coverage on a class, yet another tool counted that same class as being covered something like 60%.
The difference appeared to be that one tool reported non-executable code (like variable and field declarations without initialisers and lines containing only braces) as being uncovered code, and the other skipped them in its analysis.
Neither is tool fully correct here. To count a line containing only a closing brace or an empty statement as uncovered seems wrong, but that variable declaration? If that variable is used yet never initialised, there’s a potential for trouble so the coverage tool should ideally trace testcode to see if the variable initalisation is covered or not, and report the line as covered if and only if all possible uses for the variable are covered by tests. Most tools don’t seem to analyse that deep.
And as Ivo hints at, there’s code to be written in many situations that can be next to impossible to trigger in a test.
Mostly this will be error handling code that’s only triggered in extreme corner cases, that may rely on environmental factors that cannot be reliably replicated in a test (or can’t be without more effort than is warranted given the rarity of the need to execute it).
For such code a lexical analysis of correctness is a better test than spending possibly days writing a test for it when the code itself only gets executed once a year on an application running 24/7.
6. tim | March 26th, 2009 at 4:09 am
Wow, subjective tests written against subjective code using subjective metrics
IMPERICAL DATA!
You are a god dude.
Cover every line of that subjective code with that subjective test using those subjectively mocked dependency injections.
A god dude. A god.
7. John Ferguson Smart | March 26th, 2009 at 3:31 pm
Nice write-up, Handerson. I like the graph.
There is a lot of debate on this topic, of course, . Tests, and in particular TDD and BDD-style testing, is not just to verify that the code does things right, but also that the code does the right thing. Certainly, don’t test getters and setters directly, but won’t these be tested indirectly by decent functional tests anyway?
Test coverage is a metric, a means of seeing how well your code is exercised by your tests. In practice, it is a reasonable indicator that tests are being taken seriously, but, in itself, it provides no guarantee that the tests themselves are of good quality. That said, I like seeing high test coverage in a project. I use it, in conjunction with code reviews, to check that testing practices are being done well.
8. Marcus Vechiato | April 6th, 2009 at 7:15 am
Your point of view about tests and costs are very interesting.
The common aproach is “Everyone do tests without show their coverage and never relate their costs”.
Nice work!
9. Adriana B. | April 10th, 2009 at 5:57 pm
I found this article very interesting, and would like to suggest a theme for a future post: what factors need to be taken into consideration when deciding how much testing is needed?
I am Sr. Business Technology Analyst and my projects range from mission critical to cost-saving but not-capable of causing any damage if a defect goes live. I’d like to know how external factors, such the business impact of a software malfunction, might affect your recommendation of how much test coverage is ideal.
10. How Much Test Coverage Is&hellip | June 7th, 2009 at 8:17 pm
[...] How Much Test Coverage Is Enough Summa Blog Posted by root 4 hours ago (http://www.summa-tech.com) Thanks for your comment 3 ladycoder march 24th 2009 at 6 33 pm powered by wordpress log in summa e sales summa tech com p 412 258 3300 Discuss | Bury | News | How Much Test Coverage Is Enough Summa Blog [...]
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
Trackback this post | Subscribe to the comments via RSS Feed