World Wide Webber


My Books
REST in Practice: Hypermedia and Systems Architecture
Amazon:
US, UK
Developing Enterprise Web Services by Sandeep Chatterjee and Jim Webber
Amazon:
US, UK,
Also available: Korean Edition

My Bookshelf
RESTful Web Services Cookbook by Subbu Allamaraju
Programming Clojure by Stuart Halloway
RESTful Web Services by Leonard Richardson and Sam Ruby
Testing Functionality versus Testing Design
Posted: 08 February 2009 @ 00:04 UT from London, UK

In my previous post I talked about how unit tests aren't born tests, but are a design aid which turn into tests as the components they shape mature. But testing doesn't stop at driving out good, decoupled objects. Good developers take both broad and detailed views of system testing, which normally involves writing functional tests to validate scenarios.

But let's start smaller. Given we want to use unit tests to drive out the design of all layers of a system - including those layers which are exposed to users or other systems - there's an chance that we may end up writing code in unit tests that ends up being similar in nature to parts of our functional tests. And if so the argument goes our code may not be DRY, and the team may not be "agile" and Mr Fowler may be upset once more.

In fact this is a rather poor argument. Once you accept that unit tests are not tests but a design device, that any resemblance between unit test code and functional test code is accidental, and is likely to be transitory as system design changes through delivery, you understand that these two kinds of "tests" are inherently different beasts.

To help clarify this, I'd like to use an example from the world of Web-based (though not necessarily RESTful) services. In our book, Savas, Ian and I use a coffee shop called Restbucks to exemplify various techniques and often we capture those techniques as code snippets to help ground them in practical reality. In our chapter on CRUD services one of the implementation options uses a Java servlet to provide a means of placing and manipulating coffee orders via the network.

In designing the service, we need to isolate specific components and model their design and interaction with collaborators. We certainly didn't want to have to run the servlet in a real container at design time because that slows down development and doesn't help with formulating good designs. Instead where we needed to design how the servlet would interact with its collaborators, we used mock HttpServletRequest and HttpServletResponse objects (specifically the MockHttpServletRequest and MockHttpServletResponse from Spring) to drive out a set of behavioural contracts between the servlet and its consumers for given test contexts. By following TDD-ish techniques, we drove out the design of the service's constituent components, incrementally adding functionality until we could honour the behavioural contracts placed on us by our unit tests.

Our QA department - the same three authors with day jobs! - couldn't have cared less. "They" couldn't care how we designed and implemented the servlet, only that the overall service matched their expectations from a functional point of view, from the point of view of the consumer of the service. "They" didn't try to re-use our test code in some relentless pursuit of DRY-ness because functional testing does not (and should not) test component and interaction design, it tests functionality. If you look hard, there's a even a clue in the name!

We did in fact write functional tests which exercised the servlet and which occasionally had passing resemblance to some of our unit tests, especially those tests which deal with the payload of HTTP requests and responses But those functional tests maintain *functionality* throughout the development where the unit tests protect good design. The different is that functionality needs to be unswervingly preserved in hi-fidelity into production whereas design often changes, driven by unit tests of course.

In a dependable system, we need design time contracts expressed as unit tests, much as we need functional tests to guarantee the presence of expected, correct functionality. But these are different horses for different courses. Functional tests can be used to show when a system is ready, but they are no real indicator of the quality of the system, of its ability to be chained, maintained, and updated since that is a function of good design. Similarly unit tests can't be used to show when a system has met its users' requirements, since that it is a matter of functionality. But taken together functional and unit testing - holistically - can be used to demonstrate functionality and design.

And that is goodness.

Comments:
#

You mentioned your book. Which book is that, as I don't think it is the book advertised in your sidebar (though I could be mistaken). I'm very interested to see how you are building these services. 

Thanks!

#

Hi Ryan, 

It's a book in progress, rather than the one in the sidebar (which is a few years old now and about WS-* Web Services). The working title is "GET Connected" and Savas, Ian and I are hoping to have it out later this year. 

Jim

#

Seems to me functional tests are in fact a form of formal specification of behaviour. Well-written ones are implementation-independent, and fairly declarative in nature. Badly written ones are brittle to change and leave you wondering whether the bug is in the tests or the code.  

If you have to keep updating your tests as the code changes, that can be a measure of bad code/test frameworks.

#

Hey Steve, 

I'd not use the word "behaviour" there since it's overloaded, but I agree with the sentiments. They test what your customer is paying for (for some values of customer and payment). 

If functional tests are tightly coupled to code (as are unit tests because they drive out code design) they will be brittle as you describe. 

Separating these out isn't easy, but I'm finding it is a very worthwhile skill to learn and practice.

#

I agree that Unit Testing can be a great design aid if it leads to good decoupling. But it is the "thinking" that is stimulated not just the "modelling". And there's the rub. The code written to test the decoupled module may be as complex or more so than the application module. While the application is a living breathing "thing", responding to the environment it is deployed in, the unit test tends to be a snapshot of the system at the time. First level interfaces may be adjusted to reflect the changes, but deeper code, written solely for testing, become harder to maintain. 

For example, in order to test your RestBucks ajax interface, you may choose to avoid all the database implementation issues, and use a flat csv or XML file in a local directory space. However, while the live application has changed, the test data may not have altered. In many cases, the absence of change may not cause an error, but the testing is now incomplete. Sure, bureaucracy (bless it!) may be perfect and detect these deficiencies. What if it doesn't. 

Okay, the contract has changed, but has the test data? Writing good test harnesses is as difficult as writing good application code, maybe even more so.

#

Hi Jim - not quite on the same subject, but still thinking about 'thinking' - this from from Tim Boudreau's blog: 

"Is programming...analysis done backward? 

Ever stop to think how we do software? When analyzing the real world, people start with existing phenomena, then derive one or more models that describe it - each is an abstraction of one or more physical phenomenon. Occasionally someone starts with an intuition about how something ought to work, and it comes up correct. In software development, we start with the abstractions and the real world emerges. Is that a thing human beings have a lot of practice at?" 

The full post is at: 

http://weblogs.java.net/blog/timboudreau/archive/2009/02/is_programminga.html

Author Name:
Email:
Author URL:
Comment:
Antispam:
Please type the following string (note that if the strings don't match, your comment will be lost... sorry!): 'HWTSY'.
 
Recent entries

Recent comments

Feeds:
RSS 2.0 Atom