http://www.javaworld.com/javaworld/jw-03-2005/jw-0307-testing.html
Summary
To help developers decide whether they should be doing more automated testing, Ben Teese presents two questions in this article: Are developers being realistic about the testing they will complete on their applications? And when does automatic testing make sense for their applications? (
2,300 words; March 7, 2005)
This article is about whether we, meaning professional software developers, should be doing more automated testing. This article is targeted to those who find themselves repeating manual tests over and over again, be it developers, testers, or anyone else.
In this article I ask:
- Are we realistic about how much testing we're going to do?
- When does it become feasible to automate testing?
Note that this article doesn't discuss whether we should be testing (be it automated or manual). Nor is it about any particular type of testing, be it unit testing, system testing, or user-acceptance testing.
Instead, this article is intended to act as a prompt for discussion and contains opinions based upon my own experience.
A real-world example, Part 1
Let's start with the job that I recently worked on. It involved small changes to a moderately complex online Website—nothing special, just some new dynamically-generated text and images.
Because the system had no unit tests and was consequently not designed in a way that facilitated unit testing, isolating and unit-testing my code changes proved to be difficult. Consequently, my unit tests were more like miniature system tests in that they indirectly tested my changes by exercising the new functionality via the Web interface.
I figured automating the tests would prove pointless, as I guessed I would be running them only a couple of times. So I wrote some plain English test plans that described the manual steps for executing the tests.
Coding and testing the first change was easy. Coding and testing the second change was easy too, but then I also had to re-execute the tests for the first change to make sure I hadn't broken anything. Coding and testing the third change was easy, but then I had to re-execute the tests for the first and second changes to make sure I hadn't broken them. Coding and testing the fourth change was easy, but...well, you get the picture.
What a drag
Whenever I had to rerun the tests, I thought: "Gee running these tests is a drag."
I would then run the tests anyway and, on a couple of occasions, found that I had introduced a defect. On such occasions, I thought: "Gee I'm glad I ran those tests."
Since these two thoughts seemed to contradict each other, I started measuring how long it was actually taking me to run the tests.
Once I had obtained a stable development build, I deployed my changes into a system-testing environment where somebody else would test them. However, because the environment differed, I figured I should re-execute the tests just to make sure they worked there.
Somebody then system-tested my changes and found a defect (something that wasn't covered by my tests). So I had to fix the defect in my development environment, rerun the tests to make sure I hadn't introduced a side effect, and then redeploy.
The end result
By the time I'd finished everything, I had executed the full test suite about eight times. My time measurements suggested that each test cycle took about 10 minutes to execute. So that meant I had spent roughly 80 minutes on manual testing. And I was thinking to myself: "Would it have been easier if I'd just automated those tests early on?"
Do you ever test anything just a couple of times?
I believe the mistake I made in underestimating the effort required to test my work is a mistake also made by other developers. Developers are renowned for underestimating effort, and I don't think that test-effort estimation is any different. In fact, given the disregard many developers have for testing, I think they would be more likely to underestimate the effort required to test their code than they would be to underestimate anything else.
The main cause of this test-effort blow-out is not that executing the test cycle in itself takes longer than expected, but that the number of test cycles that need to be executed over the life of the software is greater than expected. In my experience, it seems that most developers think they'll only test their code a couple of times at most. To such a developer I ask this question: "Have you ever had to test anything just a couple of times?" I certainly haven't.
But what about my JUnit tests?
Sure, you might write lots of low-level JUnit tests, but I'm talking about the higher-level tests that test your system's end-to-end functionality. Many developers consider writing such tests, but put the task off because it seems like a lot of effort given the number of times they believe they will execute the tests. They then proceed to manually execute the tests and often reach a point where the task becomes a drag—which is usually just after the point when they thought they wouldn't be executing the tests any more.
Alternately, the developer working on a small piece of work on an existing product (as I was doing) can also fall into this trap. Because it's such a small piece of work, there's no point in writing an automated test. You're only going to execute it a couple of times—right? Not necessarily, as I learned in my own real-world example.
Somebody will want to change your software
While developers typically underestimate the number of test cycles, I think they're even less likely to consider the effort required for testing the software later. Having finished a piece of software and probably manually testing it more times than they ever wanted to, most developers are sick of the software and don't want to think about it any more. In doing so, they are ignoring the likelihood that at some time, somebody will have to test the code again.
Many developers believe that once they write a piece of software, it will require little change in the future and thus require no further testing. Yet in my experience, almost no code that I write (especially if it's written for somebody else) goes through the code-test-deploy lifecycle once and never touched again. In fact, even if the person that I'm writing the code for tells me that it's going to be thrown away, it almost never is (I've worked on a number of "throw-away" prototypes that were subsequently taken into production and have stayed there ever since).
Even if the software doesn't change, the environment will
Even if nobody changes your software, the environment that it lives within can still change. Most software doesn't live in isolation; thus, it cannot dictate the pace of change.
Virtual machines are upgraded. Database drivers are upgraded. Databases are upgraded. Application servers are upgraded. Operating systems are upgraded. These changes are inevitable—in fact, some argue that, as a best practice, administrators should proactively ensure that their databases, operating systems, and application servers are up-to-date, especially with the latest patches and fixes.
Then there are the changes within your organization's proprietary software. For example, an enterprise datasource developed by another division in your organization is upgraded—and you are entirely dependent upon it. Alternately, suppose your software is deployed to an application server that is also hosting some other in-house application. Suddenly, for the other application to work, it becomes critical that the application server is upgraded to the latest version. Your application is going along for the ride whether it wants to or not.
Change is constant, inevitable, and entails risk. To mitigate the risk, you test—but as we've seen, manual testing quickly becomes impractical. I believe that more automated testing is the way around this problem.
But what about change management?
Some argue that management should be responsible for coordinating changes; they should track dependencies and ensure that if one of your dependencies changes, you will retest. Cross-system changes will be synchronized with releases. However, in my experience, these dependencies are complex and rarely tracked successfully. I propose an alternate approach—that software systems are better able to both test themselves and cope with inevitable change.
his article is about whether we, meaning professional software developers, should be doing more automated testing. This article is targeted to those who find themselves repeating manual tests over and over again, be it developers, testers, or anyone else.
In this article I ask:
- Are we realistic about how much testing we're going to do?
- When does it become feasible to automate testing?
Note that this article doesn't discuss whether we should be testing (be it automated or manual). Nor is it about any particular type of testing, be it unit testing, system testing, or user-acceptance testing.
Instead, this article is intended to act as a prompt for discussion and contains opinions based upon my own experience.
A real-world example, Part 1
Let's start with the job that I recently worked on. It involved small changes to a moderately complex online Website—nothing special, just some new dynamically-generated text and images.
Because the system had no unit tests and was consequently not designed in a way that facilitated unit testing, isolating and unit-testing my code changes proved to be difficult. Consequently, my unit tests were more like miniature system tests in that they indirectly tested my changes by exercising the new functionality via the Web interface.
I figured automating the tests would prove pointless, as I guessed I would be running them only a couple of times. So I wrote some plain English test plans that described the manual steps for executing the tests.
Coding and testing the first change was easy. Coding and testing the second change was easy too, but then I also had to re-execute the tests for the first change to make sure I hadn't broken anything. Coding and testing the third change was easy, but then I had to re-execute the tests for the first and second changes to make sure I hadn't broken them. Coding and testing the fourth change was easy, but...well, you get the picture.
What a drag
Whenever I had to rerun the tests, I thought: "Gee running these tests is a drag."
I would then run the tests anyway and, on a couple of occasions, found that I had introduced a defect. On such occasions, I thought: "Gee I'm glad I ran those tests."
Since these two thoughts seemed to contradict each other, I started measuring how long it was actually taking me to run the tests.
Once I had obtained a stable development build, I deployed my changes into a system-testing environment where somebody else would test them. However, because the environment differed, I figured I should re-execute the tests just to make sure they worked there.
Somebody then system-tested my changes and found a defect (something that wasn't covered by my tests). So I had to fix the defect in my development environment, rerun the tests to make sure I hadn't introduced a side effect, and then redeploy.
The end result
By the time I'd finished everything, I had executed the full test suite about eight times. My time measurements suggested that each test cycle took about 10 minutes to execute. So that meant I had spent roughly 80 minutes on manual testing. And I was thinking to myself: "Would it have been easier if I'd just automated those tests early on?"
Do you ever test anything just a couple of times?
I believe the mistake I made in underestimating the effort required to test my work is a mistake also made by other developers. Developers are renowned for underestimating effort, and I don't think that test-effort estimation is any different. In fact, given the disregard many developers have for testing, I think they would be more likely to underestimate the effort required to test their code than they would be to underestimate anything else.
The main cause of this test-effort blow-out is not that executing the test cycle in itself takes longer than expected, but that the number of test cycles that need to be executed over the life of the software is greater than expected. In my experience, it seems that most developers think they'll only test their code a couple of times at most. To such a developer I ask this question: "Have you ever had to test anything just a couple of times?" I certainly haven't.
But what about my JUnit tests?
Sure, you might write lots of low-level JUnit tests, but I'm talking about the higher-level tests that test your system's end-to-end functionality. Many developers consider writing such tests, but put the task off because it seems like a lot of effort given the number of times they believe they will execute the tests. They then proceed to manually execute the tests and often reach a point where the task becomes a drag—which is usually just after the point when they thought they wouldn't be executing the tests any more.
Alternately, the developer working on a small piece of work on an existing product (as I was doing) can also fall into this trap. Because it's such a small piece of work, there's no point in writing an automated test. You're only going to execute it a couple of times—right? Not necessarily, as I learned in my own real-world example.
Somebody will want to change your software
While developers typically underestimate the number of test cycles, I think they're even less likely to consider the effort required for testing the software later. Having finished a piece of software and probably manually testing it more times than they ever wanted to, most developers are sick of the software and don't want to think about it any more. In doing so, they are ignoring the likelihood that at some time, somebody will have to test the code again.
Many developers believe that once they write a piece of software, it will require little change in the future and thus require no further testing. Yet in my experience, almost no code that I write (especially if it's written for somebody else) goes through the code-test-deploy lifecycle once and never touched again. In fact, even if the person that I'm writing the code for tells me that it's going to be thrown away, it almost never is (I've worked on a number of "throw-away" prototypes that were subsequently taken into production and have stayed there ever since).
Even if the software doesn't change, the environment will
Even if nobody changes your software, the environment that it lives within can still change. Most software doesn't live in isolation; thus, it cannot dictate the pace of change.
Virtual machines are upgraded. Database drivers are upgraded. Databases are upgraded. Application servers are upgraded. Operating systems are upgraded. These changes are inevitable—in fact, some argue that, as a best practice, administrators should proactively ensure that their databases, operating systems, and application servers are up-to-date, especially with the latest patches and fixes.
Then there are the changes within your organization's proprietary software. For example, an enterprise datasource developed by another division in your organization is upgraded—and you are entirely dependent upon it. Alternately, suppose your software is deployed to an application server that is also hosting some other in-house application. Suddenly, for the other application to work, it becomes critical that the application server is upgraded to the latest version. Your application is going along for the ride whether it wants to or not.
Change is constant, inevitable, and entails risk. To mitigate the risk, you test—but as we've seen, manual testing quickly becomes impractical. I believe that more automated testing is the way around this problem.
But what about change management?
Some argue that management should be responsible for coordinating changes; they should track dependencies and ensure that if one of your dependencies changes, you will retest. Cross-system changes will be synchronized with releases. However, in my experience, these dependencies are complex and rarely tracked successfully. I propose an alternate approach—that software systems are better able to both test themselves and cope with inevitable change.
About the author
Ben Teese is a software engineer at Shine Technologies.