Since we ditched the horridly old Workshop for Weblogic that is based on an old version of eclipse and now work in Eclipse 3.5 for the webservices and 3.3 for the OSB we have been able to use the latest and greatest plugins. The Workshop for Weblogic was so archaic that most of the plugins did not work. For unit testing code coverage we are using eclemma. This result popped up in one of our beans today;

The bean itself is about 240 lines of code including import statements, private member variables, etc. It also contains six methods of which five of them have 100% code coverage. The one method remaining to be unit test in that bean is 30% of the bean's code complexity.

Given that there are six methods in the bean you would expect the complexity to be about even between the methods with each method contributing about 16% of the necessary unit testing code coverage. That one requires 30% of the code coverage suggests that it is probably too big and too complex.

Sure enough, on inspection it is.

So between the code coverage tools and unit testing an area of unusually strong complexity can be discovered and narrowed down into smaller and simpler algorithmic structures. Win, win.

We have two layers of automated testing in Engineering. We have the unit tests, which is the standard white box tests using JUnit that focuses on exercising the cyclomatic complexity of a method in the code. Used in conjunction with JMock it satisfies our needs that the code does what it says it will at that level.

We also have functional tests which hit the system through our Webservice calls and Webmethods which we expose to external systems to ours. The primary focus of Engineering is the core system which is the brain amongst all the other systems that go towards providing the end business result.

Our functional tests are run through JUnit and we hit our systems using jax-ws where the interfaces are generated through ant. Essentially they are black box tests which know nothing of the system beneath the calls but ensure that the customers for our Webmethods get the result they desire when the use our system.

Our end goal and current strategy is to get to a point of continuous deployment. Grandiose for a system that only six months ago had zero unit tests to it, but very believable now with the leaps and bounds we have made in quality, confidence and engineering improvements. We want to ensure that any one point in time at any moment through the day and night the code base is of maximum quality and production ready.

Consequently our functional tests are growing in increasing complexity and sophistication. Rather than just exercising our system we are now reaching out to the other systems via jax-rpc, jax-ws and even JDBC in one case to ensure that the results that are expected from any change in state of the system are being propagated correctly through-out the entire system.

Ironically, I probably spend more time in the debugger during the functional tests watching the data flow in and out of the system rather than in the code-base itself. It is a curious position.

Older codebases, and even recent ones that have been implemented without unit testing to support them, tend to have a lot of structures that make the codebase untestable. The codebase we inherited for this project was not in a testable state. We had to modify several aspects of it to abstract different static classes and methods out. We did it with the standard interface and putting a concrete impl class behind which then referenced the static class or method in the desired static manner.

The google testing blog calls this 'static cling' and calls the solution the repository pattern. I think the GoF have a bit to answer for by making everything a pattern. An interface with an implementation behind it is simple Object Oriented programming. Hardly deserving of a pattern.

JMock has an imposterizer which allows you to mock concrete objects rather than interfaces. We mucked around with doing that in the case of the static classes but a couple of things went against us. One was that a static method in a library was final and since the imposterizer extends the class it is mocking to make a proxy it made that impossible.

In the end the interface-impl solution worked well and made that aspect of the codebase testable.

Steve McConnell's Code Complete points to an interesting study where the defect detection rates of many common-defect detection techniques used individually have a modal rate of about 30-50%. Unit testing under that study had a lowest defect detection rate of 15%, a modal rate of 30% and highest rate of 50%. Given the focus that different methodologies such as XP have had on unit testing that is quite low and a little disappointing. However the study noted;

When used individually, no method had a statistically significant advantage over any of the others. The variety of errors people found was so great, however, that any combination of two methods - including independent groups using the same method - increased the total number of defects found by a factor of almost two.

So unit testing used in conjunction with regression testing (for instance in continuous integration) improves the value of unit testing. From the list in that book of the value of the different removal methods, we have decided to use unit testing in conjunction with formal code reviews and formal design inspections. However, our design inspection are based on the stories (use cases) and are a couple of pages to make sure the starting point for coding is ok and agreed on.

Unit Testing has value beyond defect removal rates. It is amazing how developer confidence improves with unit testing. Kent Beck writes:

While there are many rational explanations for why programmers should automate tests, for me the compelling argument is emotional. When I write automated tests I feel more confident in my work than when I don't write automated tests.

Which I agree with. The unit testing adds a sense of completeness to code as well. With all unit testing libraries and the integration of unit testing into IDEs it has become very simple to do so as part of a project. But the study that McConnell points to suggests that used in combination with another defect removal methodology they are even more effective.

Unit testing has become nearly universal in penetration amongst engineering shops. The hump to implement it can be high but the value in terms of quality is so instantaneous a trade off that it is invaluable. Michael Pederson writes:

I first learned about automated unit testing about four years ago, and decided to experiment with it. I quickly found out that code I thought I had thoroughly debugged had errors in it.

My experience is the same. It used to be that when you went back to code you wrote a year ago you would wonder what the hell the developer (ie you) was thinking. Now that little feedback loop is within fifteen minutes.

The current project we are doing makes the Oracle Service Bus [OSB] the central brain between all the different systems. We are architecting it as it should be, but there is no real mechanism to place unit tests on the workflow within the OSB. It is also the most fragile part of the system. Namespaces are particularly susceptible to breakage in that environment.

Interesting comparison of test driven development and writing unit tests after the problem has been solved in an article from Peer Siebel;

Then bloggers were arguing back and forth about what this means. I don't think it means much of anything--I think test-driven design is great. I do that a lot more than I used to do. But you can test all you want and if you don't know how to approach the problem, you're not going to get a solution.

Lately we have been pair programming over the problem and solving it in java code, webservices and the OSB. Afterwards we have been running off and putting unit tests over it.

To be truthful I have been really enjoying this pattern of work. It helps that Patrick is an entertaining fellow to work with, but we have had two eyes on the problem and have talked out design decisions and choices while we are working. Additionally for me there is a large element of training as well as I come up to speed on a new and complicated system.

The added bonus is we are getting a lot of work done by working in this manner.

There is a posting on the perl journal about test driven development and how being dogmatic about a quality approach is never fully fruitful. One of the problems in software is feature volatility and while functional testing offers a mechanism to test features it is also plagued with the same problem.

It has been my experience that unit testing does produce higher quality and more robust code during development. It is more laborious and does take up more time, but the number of times untested code has sucked up two developers for two days each in tracing down difficult and obscure behaviour it is worth it. More so, those that don't cover their code fully with unit tests do have the buggiest code.

There was a time when the bare minimum of quality in a software project was continuous integration and a central source repository. Now, it is unit testing as well. But again, it doesn't require 100% unit testing coverage, or test driven development. One of the things that makes code so durable is that if you get it right once, it tends to work the same forever. So code that has passed QA is normally good for production in most cases, short of gnarly edge cases.

There is also the issue that working code in a crunch environment is valuable, so working code will get pumped out and unit tests skipped. Features are often deemed more important during the development phase and quality only becomes important after feature complete. There is also the issue that quality can be difficult to determine until feature complete anyway as features can be negatively impacted by the different delivery mechanism for data in complex projects when they are incomplete leading to confusing bugs or quality issue reports.

Ultimately however, unit testing has raised the bar on what is the standard for quality in a software project. Codebases without test harnesses of some kind are remiss.