Should we automate our exploratory tests ?
Exploratory tests are by definition always different tests that are designed and executed simultaneously. If we stick to this definition, their automation may seem unnatural. And yet, the implementation of automation based on combinations of predefined and random scenarios could have an interest: the discovery of business bugs.
Exploratory testing: high-level “business” software testing
The principle of exploratory testing is simple, but it requires a calm environment and time on one’s hands: a user, who is generally a business expert in the field implemented by the software, decides to explore a part of the software by carrying out a complete business scenario that he or she deems relevant.
In the typology of software testing, exploratory tests are often associated with functional tests because they constitute a family of software tests qualified as high-level “business” tests: their objective is to verify the ability of the software to carry out the complete procedures for which it has been designed. These procedures may involve several organizational entities of the company, several information systems and various processus, hence the interest in testing scenarios that validation tests do not necessarily address.
In terms of organization, having an exploratory testing policy is very beneficial: the validation team is a mine of information and knowledge, and having them work on high-level scenarios allows them to deal with a customer case that is conducive to new ideas: evolution of functions, of the User interface.
Automation of high-level business tests
In some cases, the automation of high-level business tests is quite complex: the combination of different implemented functions can generate multitudes of cases, all of which are complementary, and all of which have the possibility of being reproduced by an end user.
In addition, the multiplication of execution environments could impose the ability to reproduce these scenarios on different platforms, but implementing the same functions.
To overcome these two difficulties, we have based our Agilitest solution on two very important pillars: the ability to perform multi-channel tests and the possibility of variabilizing the tests, in particular through the use of CSV or JSON files. These two capabilities alone allow us to fix to this problem, by making it possible to replay the same test on several environments or by managing the combinatorial complexity of the tests through data files.
Exploratory tests: the way to discover business bugs
Unfortunately, exploratory testing are often poorly implemented in software testing because teams do not have enough time, yet it is very important and complementary to functional testing.
In some cases, they even lead to the discovery of functional bugs: display problems, data problems, blocked functions, aso.
In other cases, it is also possible to succeed in realizing a business scenario that is impossible in reality, or conversely, not succeed in reproducing a scenario that should be possible.
In this case, we found what we can call a “business” bug.
What to do when faced with a business bug?
When faced with a business bug, there are two possibilities:
- The software does not conform to what was specified, we have a bug that reproduces under particular conditions that are not those tested in our plans, whether they are manual or automated.
- The software conforms to what was specified, but corresponds to a combinatorial that was not defined: it is always very difficult to specify all the functional possibilities of a software, and we have fallen into a grey area. Testers who know the underlying business well know how to define the expected behavior of the application, which is not the one they have just observed. This raises the question of having the test teams work very early in the development process, directly with the functional architects, product owners, etc., to prevent this situation from recurring, and this is generally a good practice because it can avoid unwanted developments.
In both cases, the correction of the bug combined with the production of a new test case will allow you to enrich your plans, to clarify the expected behavior of the application, and to confirm the validation team in the capacity of the tested tool to realize a more complex scenario.
What about automation?
At Agilitest, what we recommend is the correction of the “business bug” by adding controls during the realization of the various functions.
We also propose to carry out an automated test which reproduces the “business bug” and controls that it will not reproduce. This is the core business of what Agilitest can do : manage the complexity and combinatoriality by using sub-scripts performing high-level functions and data files to completely variabilize the replayed scenarios.
Generally speaking, it is important to always cover bug fixes with automated non-regression tests linked to the bugs, ensuring that they cannot reoccur: it is very painful for a customer who has reported a bug in version N to see it corrected in version N+1 and then reappear in version N+2.
I n conclusion, when an exploratory test leads to the discovery of a business bug, it is useful to automate the scenario in order to add it to the functional test battery. And if the automation of an exploratory test may seem too complex because of the combinatorial nature of the test, remember that Agilitest was designed to manage this complexity.
Originally published on Agilitest’ blog.