Bug Blog

Manual testing is often viewed as expensive and slow. However, it is the most effective way of finding critical defects (read http://www.satisfice.com/articles/et-article.pdf by software testing expert James Bach for more information). Exploratory testing is gaining momentum as 100% automated software shops find themselves drowning in customer reported defects for mobile, web or old-fashioned client-server system platforms.

Repeatability; how sweet it isn’t

Automated testing supporters espouse the benefits of repeatable tests. Repeatability also means selecting specific test data that is known to exist and generating a passing result. The test script may create data, manipulate it and then remove it.

Users do not enter perfect data and then remove it. Repeating the same tests over and over again with simple variations of the same perfectly entered data will not find bugs. Executing automated tests repeatedly finds reportedly between 6-30% of defects (Cem Kaner August 2000; Architecture of Test Automation).

The impact of test stagnation

Test stagnation affects both manual and automated testing but it’s more prominent in automation. Automated tests are typically simple tests covering non-complex code variations. Running those same tests repeatedly, even if a defect is found, it typically is not significant enough to fix.

Often the automated test engineer edits the test to pass and fixes the script after the release. Supplementing this with manual testing enables the release to deploy as planned assuming the automated test failures were false.

Automated test maintenance plan and pay

All application code requires ongoing maintenance, and automated tests are code. If you don’t plan on maintenance time and cost your automated testing project becomes an expensive failure. Anytime the code changes, the automated tests break and must be edited and enhanced to include new functionality. Do you change your application code frequently?

Balance testing for higher quality, less risk, and greater returns

Balance is the key. Balance automated testing with manual exploratory testing. Options exist for organisations that can’t afford more resources or don’t want to hire manual testers. There are services that provide manual exploratory testing at a reasonable cost when you need it. Use a testing service to provide increased test coverage, find user defects and reduce release risk.