Escalating software risk and cost with automated testing

Automating unit tests are essential for testing code integration after check in and build verification. It has value for finding issues when smoke testing new code deployments and executing simple modular functional tests. Automated test creation is software code development, which involves time, money, and ongoing maintenance. It’s the maintenance debt of automation that causes automated test initiatives to fail.
Escalating software risk and cost with automated testing

Contact Us We cut software testing from weeks to days. Let’s talk for 15 minutes to see if we can accelerate your digital delivery too. Schedule a call with our CEO Ash Conway.

Manual testing is often viewed as expensive and slow. However, it is the most effective way of finding critical defects (read http://www.satisfice.com/articles/et-article.pdf by software testing expert James Bach for more information). Exploratory testing is gaining momentum as 100% automated software shops find themselves drowning in customer reported defects for mobile, web or old-fashioned client-server system platforms.

Repeatability; how sweet it isn’t

Automated testing supporters espouse the benefits of repeatable tests. Repeatability also means selecting specific test data that is known to exist and generating a passing result. The test script may create data, manipulate it and then remove it.

Users do not enter perfect data and then remove it. Repeating the same tests over and over again with simple variations of the same perfectly entered data will not find bugs. Executing automated tests repeatedly finds reportedly between 6-30% of defects (Cem Kaner August 2000; Architecture of Test Automation).

The impact of test stagnation

Test stagnation affects both manual and automated testing but it’s more prominent in automation. Automated tests are typically simple tests covering non-complex code variations. Running those same tests repeatedly, even if a defect is found, it typically is not significant enough to fix.

Often the automated test engineer edits the test to pass and fixes the script after the release. Supplementing this with manual testing enables the release to deploy as planned assuming the automated test failures were false.

Automated test maintenance plan and pay

All application code requires ongoing maintenance, and automated tests are code. If you don’t plan on maintenance time and cost your automated testing project becomes an expensive failure. Anytime the code changes, the automated tests break and must be edited and enhanced to include new functionality. Do you change your application code frequently?

Balance testing for higher quality, less risk, and greater returns

Balance is the key. Balance automated testing with manual exploratory testing. Options exist for organisations that can’t afford more resources or don’t want to hire manual testers. There are services that provide manual exploratory testing at a reasonable cost when you need it. Use a testing service to provide increased test coverage, find user defects and reduce release risk.

Bug Newsletter

Keep up to date with the latest in beta, bug & product testing.

User acceptance testing tools & teams

Quickly assess the quality of your software, websites & mobile apps