Bug Blog

Testing and quality assurance are amongst the biggest constraints when it comes to delivering software products on time. While customers and clients generally understand that no software is ever truly bug-free when it first ships, a critical bug or error can be devastating to a developer's sales and reputation. 

With so much of the infrastructure we rely on in our daily lives dependent on accurate and unfailing software, it's more important than ever to catch and correct bugs in the code. As such, there has been a big push lately toward more automation in QA testing. 

The argument goes that automated, script-driven testing can test more components more accurately, rigorously, and consistently than human testers can, and in a fraction of the time. That's true, but those who favor automated testing have to contend with the fact that, while automation can do many things better and faster than human beings, there are still many things it can't do at all. 

Manual testing is still a vital part of the QA process and human testers are far from obsolete. In fact, many businesses do not achieve their desired results with automated testing alone. 

The 2017-18 World Quality Report found that automated testing technologies only perform about 15% of common test activities. A hybrid approach that uses automation while covering the gaps with manual testing may be the ideal approach, but it's important to understand the strengths and unique benefits that manual testing can provide.

Don't get us wrong—we're big fans of automation here at Bugwolf. In fact, we've developed a mature automation suite of our own, but that's only one piece of the digital quality puzzle. As long as you've got humans using your website or app, you need a human perspective on what works and what doesn't. 

Here are six of the key reasons why you can't (or at least shouldn’t) write off the manual testing process.

1. Exploratory Testing

When a human tester finds an element of a program that's behaving in odd or unexpected ways, they can stop and dig into it. They can experiment with different inputs and let their own intuition and curiosity guide them to investigate further, factoring in what they've discovered during the remainder of the testing process.

That exploratory testing is virtually impossible to script. Almost by definition, exploratory testing refers to what happens when you stray away from the beaten path, letting "what happens when I do this?" be your guiding principle.

Automated testing is great for finding bugs in the core functions of the software, the code that always gets run, the predictable test cases. Actual users, however, rarely stay within the expected boundaries, and that's why you need manual testers. They will behave just as unpredictably, poking holes in strange places and finding the bugs that hide from automated processes.

In particular, mobile apps have complicated use cases. Think of all the variables that are completely outside the control of the developer that can affect how a mobile app performs – everything from declining battery life to intermittent Wi-Fi signals to sweaty hands. There's no way an automated script can account for all those things. Yet human testers can, and when they find one unpredictable thing that impacts the software's performance, they can drill down on it to learn more.

2. Human Insights

Automated scripts can check to see if code functions as intended, but no script can wrap itself around the gestalt of an app to tell you if it feels right, if it's doing what it's intended to do at a conceptual level. All it can tell you is whether the code was written correctly or not.

Humans are creative and analytical, and manual testing gives you a human perspective on your software's performance and functionality. People can spot misleading or confusing visual issues that scripts would gloss right over.

They can reproduce customer-caught bugs and errors. They can also bring their own experience to the table, with testing informed by past experiences finding software bugs, writing code, or pushing an app to its limits. This empirical knowledge can yield insights that could never be anticipated in advance, or by a machine.

Human testers can also catch issues that aren't bugs by definition, and don't conform to strict pass-or-fail testing standards. Speed, visual noise, ambiguities, and other usability issues can easily slip right by an automated test if they're technically working as intended.

If you want to test the strength of your code, automated testing is well-equipped to do just that. But if you want to see how your software is actually going to perform in the wild, you need human testers.

3. Bugs in the Gaps

An automated test can only do what it's told to do. No matter how powerful or dynamic a scripted test may be, there will be gaps in its testing methodology. 

Automation can't find things you don't know how to look for, and bugs are often found where you least expect them. Many bug testers report that they often discover bugs when they're trying to test out something completely unrelated to what they actually end up discovering.

Scripts are written by humans, which means they're limited by the experience and imagination of the person writing them. It also means that the scripts themselves can contain bugs or errors, which can generate false positives, overlook bugs that would be obvious to a human tester, or fail to test everything that needs to be tested.

4. The Limits of Scripting

Scripts have other limitations, beyond the skill and knowledge of the people writing them. There are external constraints on scripting as well: the time, labour, and costs involved in building robust, comprehensive directions for an automated testing process.

Some scenarios are too complicated, too expensive, or too weirdly specific to be worth testing with automation. A lot of work goes into writing, refining, and deploying a script that can thoroughly test a large software program, and automation is frequently too expensive to use for small projects.

Good testing is repeatable but also variable. Thus, attempting to modify automated scripts to perform variable testing on a just-discovered bug is almost never efficient, in terms of time or money. Manual testing may be slow, but the high costs of setting up and maintaining automated tests cannot be ignored.

5. Validating the User Interface

In many apps and websites, few aspects receive more attention or focus than the user interface. Automated scripts are ill-suited to providing any insights into the quality of the UI beyond verifying that the underlying functionality works as intended.

Manual testers can take the big picture of the software they're testing into account; they can understand not just how the code is supposed to work, but also whether or not the software meets the needs and expectations of the actual people who comprise its target market. In most cases, the UI is the biggest part of this.

Buttons that don't look like buttons, alerts that fail to catch your attention, and text that's too small or stylized to read easily are just a few of the things that scripts are often blind to, but even untrained testers will pick up on immediately.

6. The Development Environment

Automated scripts can perform testing functions with incredible speed, but setting up a script for the first time can be a slow, labour-intensive, and costly endeavour. And in an Agile environment, where bug fixes and other changes are being made as parts of a continuous development cycle, it's hard to fit scripting updates into the process without bringing everything to a grinding halt.

When your work is organized into sprints, it's difficult to keep scripting updates on track, and they tend to lag behind. Even in a more traditional development environment, it rarely makes sense from an efficiency standpoint to allow delays or reorganize priorities, in order to update or modify automated testing scripts.

The Best Way to Run a Bug Hunt

You can get a lot of testing done with automation – and there are certainly many aspects of testing that automation does well – but if you're looking to automated scripts to free you from any reliance on manual testing, you're going to end up shipping out buggy software. You're also likely to miss many areas of potential improvement that manual tests would have identified right away.

The optimal solution for QA is to use both methods. Automated scripting should be utilized for the obvious and predictable use cases, for stress testing, and for weeding out unambiguous coding errors. For everything else, however – the weird bugs that pop up when you play around with the software in an "off-label" way, the UI problems that electronic eyes can't see, the on-the-fly testing you have to do when code changes as part of a fast-paced development cycle – there's no substitute for human testers. People can explore your software inside and out in ways you might never have expected, bringing all of their experience, imagination, and unconventional modes of thinking to bear on the code you've compiled.

Another way to put this: automated testing will catch the bugs that would show up immediately after launch and result in immediate updates and some bad press. But if you want to find the bugs that won't show up until years later – perhaps at some critical moment – you still need real human beings as a part of the process.