Bug Blog

While some types of testing require detailed planning, exploratory testing has a minimum of planning and a maximum of execution and so relies heavily on the experience and skill of the tester.  It is the tester who determines what the analysis will be and generates logs accordingly.  Logging is done as testing is executed, with concentration on documenting the key elements of what is under test as well as outlines regarding what the tester thinks might be useful in further testing.

Exploratory testing does not substitute for other more formal tests.  There is sometimes the temptation, especially when nearing the end of time or budget, to substitute exploratory testing for more formal test procedures.  Exploratory testing is meant to complement such other forms as regression tests and user experience testing.  It is not intended as a substitute and should never be substituted for other types of detailed testing.

Exploratory testing is sometimes referred to as ad hoc testing.  Ad hoc testing is running random tests.  Test cases are chosen randomly in an effort to locate possible bugs.  Exploratory testing generally requires imagination, the tester is actually trying to break the application by pushing it, if necessary, beyond design limits.  The idea is to determine if the application can operate under unusual or high stress circumstances, and while exploratory testing can certainly find bugs, locating bugs is not always its first priority.  Under such circumstances, there may not even be established cases to test. You must then imagine a possible defect and test for it.  Basically, exploratory testing is unstructured and the quality of such testing depends entirely upon the qualities of the tester or testers involved.  Part of that quality is knowing what to test, when to test and how far to test.  It's rather easy to become a bit too enthusiastic and start testing to a point of unreality.  This is why you need skilled testers.  They are, in a sense, artists.  And the greatest gift of the artist is to know when to stop.

Agile methodology works well with exploratory testing because the two share similar basic principles.  These are:

• The importance of interactions and individuals over tools and processes.  Both agile methodology and exploratory testing place the same kind of emphasis on the individual.
• Interaction is senior to planning.  Rather than following a plan, agile methodology and exploratory testing are about responding to changes as they happen.
• Exploratory testing isn't about simply meeting contractual obligations, but is more interested in producing the highest quality software possible, this is also a part of the agile method.
• Like the actual method, exploratory testing is more concerned with obtaining working software rather than detailed documentation.

Exploratory testing isn't really concerned with coding or acceptance testing and is not really a part of user experience, at least not directly.  It is more of a way of critiquing the application by placing it in as difficult a situation as possible and then seeing how it performs.

Exploratory testing can be considered a type of user testing in that its purpose is to use the system as a user would in order to identify bugs.  While exploratory testing can be a bit more severe than normal user testing it still falls into that category.  However, there are some important differences that make this style of testing unique.  Unlike other forms of UX testing, exploratory testing is not interested in wide coverage.  Its purpose is to find the defects that don't show up under normal testing procedures.  These are defects that occur at the extreme edge of functionality, but just because they're rare doesn't mean they have little impact when they arise.  Defects can often be more severe at the edge of performance, as operations outside standard usage patterns are the most likely to push the edge of the envelope and result in system crashes.

While automated testing exists to test the system based upon how the development team intends it to perform, exploratory testing operates outside of the box and focuses on those elements of the system that don't fall so readily within standard usage patterns, and so are less likely to be tested in detail.  Consequently, exploratory testing should be used in concert with automation, but not as a substitute for automated testing.

Whether we're talking about non functional or performance testing, automation belongs in those areas of high predictability while exploratory testing exists to measure the least predictable.  Nevertheless it is important that both types of testing be carefully coordinated in order to extract the maximum value from software testing.  For example exploratory testing can find a bug that can then be added to an automated testing regime in order to prevent the defect from occurring again.

The greatest benefit of exploratory testing is the ability to find problems that cannot be found by any other means.  Of course, this requires some planning, and so exploratory testing is not totally and utterly random, it is instead a skillful investigation of the software.

While exploratory testing certainly has its place prior to release, it isn't just the development cycle phase.  It is a technique that can be applied throughout development.  It is a type of flow testing and can be applied to any module which has an established flow through.

Ultimately the best exploratory testing is done with the user in mind and reflects realistic scenarios that could happen, even if they are not likely to happen.  While exploratory testing is sometimes considered ad hoc, this does not mean that it is careless or sloppy.  It is the union of a knowledgeable tester and the software under test.  This tester then uses his or her imagination, skill and experience to push the software in ways that it could be logically pushed by users themselves.  If the exploratory tester has done his job, the software will not only have fewer bugs it will also have a greater chance of passing user experience testing with minimal difficulties.