Concerns about unit test quality

In a recent blog post, Nat Pryce expressed a concern that I’ve long had:

I have observed that few programmers are skeptical enough about the code they write. They focus on success cases, ignore edge cases and error cases, try to bang out as many features as possible, and don’t notice that those features are incomplete or will not work in some situations.

And then Nat goes on to explain how test-driven development can help developers improve their thinking about the tests that they write:

As programmers get more experienced with writing automated tests, they get better at thinking about how their code might fail. That’s because they get plenty of opportunity to observe where their tests are insufficient and think about how to avoid such problems in the future.

I don’t disagree with Nat at all, but I am still concerned that it’s a case of fitting a case of trying to fit a round peg into a square hole: the developer may get better at writing tests, but she’s still a developer, not a testing specialist.
This is, of course, one of the reasons why we perform other types of testing in addition to unit tests. But if we want to make sure unit tests are robust, it seems like we need to involve the testing specialist in some way. And having the developer pair with the testing specialist to review unit tests is the most obvious solution to em. Even if the testing specialist’s skill-reading codes are weak, the developer can at least describe the tests to the testing specialist (If the testing specialist has the skills to write unit tests, having him (help to) write the unit tests would be another alternative).
I’ve pursued this type of pairing a few times, but it has never worked out for more than a couple of weeks. The primary reason for this failure, in my opinion, is lack of buy-in from the developers. They don’t see the value of having a testing specialist review their unit tests, or unit testing just somehow remains the property of the development club.
Any advice or experiences you can share would be helpful.

Risk-based testing

One of the Ten Tips for Agile Testing is “Use risk based testing.”

You can never test everything with the same (extensive) depth; even in a waterfall project you have to make choices. In an agile project all the activities are time boxed so you have to make choices about how extensively you want to test each feature. We use a risk based approach to determine which test activities we are going to carry out for a feature during the iteration. The risk level of every feature is determined by the customer and the teams. It is a very transparent process so the customer knows exactly which test activities are executed for every feature

I’ve heard a lot about risk-based testing in the last couple of years. I don’t mean to sound like a know-it-all, but I’ve been employing risk-based testing for years (which goes along with risk management mentioned in yesterday’s post), only I didn’t know it had a specific name. What do others think of this?

Exploratory testing

Lately, I’ve been thinking a lot about the role of conventional QA engineers on agile projects. In my earlier post on this topic, I concluded that there is less need for non-coding QA specialists on agile teams, but that the skills that such a person brings to the table are still necessary.
In a recent column over at, titled Does Exploratory Testing Have A Place On Agile Teams?, Johanna Rothman writes the following:

Agile projects require true generalists as testers: people who have requirements-, design-, and code-understanding skills. Without those skills, they can’t think critically enough about the product under development, and they might not be able to create enough variety of tests. If they understand the requirements, design, and code, they can turn that understanding into crafty tests. Some of those tests will be exploratory. Even some of the exploratory tests will need to be automated in order to repeat them. And, I’ve seen great testers on agile projects who can quickly create automated tests to do some of their exploration.

I read that as a confirmation of my thinking. Coders can indeed do testing, but it takes someone on the team who has the big picture in mind in order to do good exploratory testing. QA generalists are needed to do this work.

What does ‘done’ mean?

Here’s a problem that I’ve been thinking about recently. I’d love to hear feedback.
Scrum dictates that each user story that a team commits to in a sprint should be complete at the end of the sprint, and the story’s acceptance criteria define what ‘done’ means.
Certain types of enterprise testing can pose challenges to this principle. Performance requirements, for instance, are often based on scenarios that transcend individual user stories.
Here is an example: the application under test supports two different user roles, and the performance requirements stipulate the required performance of individual functions when certain numbers of users of both roles are using the application concurrently. User stories A, B and C cover functionality performed primarily by one user role and stories D, E, and F cover functionality used by another role.
If the team implemented stories A, B and C in one sprint, the team would not fully know whether stories A, B and C meet performance requirements until stories D, E and F are complete and the entire scenario is run with all user roles.
Certainly, the scrum team can minimize these sorts of problems by changing the order in which they implement stories or other strategies. And the team gains a certain amount of value from performing partial tests, but fundamentally, enterprise testing poses these types of challenges, and scrum teams have to deal with them.
If the scrum team cannot work around this sort of challenge, then they should consciously acknowledge the challenge, figure out a compromise to agile principles or scrum practices in order to deal with the challenge, and specify the risks involved in the compromise. And most importantly, all of this should be communicated to the customer at the sprint review.
Using the example above, a portion of the sprint review would go something like this:

Here’s the challenge we faced: in this sprint we completed stories A, B and C to the best of our ability, but we cannot fully know whether these stories fulfill all performance criteria until we complete stories D, E, and F.
Here’s the compromise we came to: we have completed these stories to the best of our ability. We did as much as we can, short of the full performance testing scenario, to verify that the stories meet performance acceptance criteria.
Here are the risks involved in this compromise: It’s possible that the stories do not perform as well as expected in an enterprise deployment. But since the missing variable pertains to unfinished functionality, users will not be executing that functionality at this time anyway.
Here’s how we plan to address this compromise: we plan to complete stories D, E and F in the next sprint. At that time, we’ll be able to run the entire performance scenario and verify that stories A, B and C meet performance requirements.

At that point, the customer would be able to accept or reject the stories. However, if the team has been working closely with the customer from the beginning of the process, the customer should have known about the challenges and have worked with the team on how to address them. The risks presented at sprint review should not come as a surprise to the customer.