Need for Test Automation & Evolution of Frameworks

Today, assuring the quality of any software has become very challenging. The size and complexity of applications have increased exponentially, while end-users are becoming more and more demanding. Under these circumstances, verification and validation activities tend to lose their momentum and make compromises. As a result, testing-cycle times tend to shrink significantly. In this situation, to move in step with the ever-changing quality dynamics and end-user expectations, conventional manual testing may not be a suitable option. Test automation becomes an increasingly critical and strategic necessity.

Appreciating the promise of Test Automation

Let’s get started! Here are the most promising benefits of test automation (and outline of the expectations that are met from a tester’s perspective).

1. More tests more often – yes, test automation means faster test execution, which encourages more test iterations. It also makes creating new test cases easy, fast and ‘ready to run’.
2. Consistency and repeatability of Tests – since tests are always run the same way in automation, test results can be consistently compared to results from previous test iterations. Importantly, tests can be repeated on different environments.
3. Reuse of Tests – generally, reusing tests or test scripts from previous projects gives a good head start to new projects.
4. Optimum use of resources – automating repeating or regular checklist of tasks would definitely help test engineers focus on more demanding and prioritized work.
5. Faster time-to-market – reusing tests and shortening test execution times will fasten feedback cycle to developers and internal communications with the testers. Eventually, all this shortens the time to market.
6. Increased confidence over Application / Product roll-out – importantly, this is the most promising benefit of test automation. Running an extensive set of tests often, consistently, and on different environments successfully, increases the confidence that the product is really ready to be released or rolled out.
7. Compatible with Regression Tests – with test automation, one can run previously created tests against any new functionality and environment without any extra effort, which clearly makes testing more efficient.

Understanding the common problems with Test Automation

1. Unrealistic expectations – some managers strongly believe that test automation will solve all their testing problems and make the software quality better. The test automation experts and their team should help these managers set expectations that are correct and clear.
2. Misconception about Test Automation – finding a lot of new defects – it has to be clearly understood that once the automated tests have run successfully, then it is not very likely that new bugs will be found, unless the tested functionality changes.
3. Poor testing practice – if the available testing practices and processes are inadequate, then it is better to start improving them rather than bringing in test automation.
4. Test maintenance – generally, when the System under Test (SUT) changes, then the corresponding tests change. Manual test engineers can handle such changes, even major changes, without any problems. The same is not the case with automation. With even a slight change in the SUT, the automation tests will fail. Further, if maintaining a test automation suite takes more time than the effort involved in manual testing, considering all the changes that the SUT has to go through, such automation will surely be abandoned. This situation typically arises while adding new features to the system.
5. Technical challenges – building or augmenting a test automation suite is definitely a technical challenge, which should be given a thoughtful consideration. A strong team with requisite technical skill-set should be created to build and troubleshoot any issues with automation.

Evolution of Test Automation Frameworks

Generally, a testing team is actually a pool of testers supporting many diverse applications completely unrelated to each other. Now, if each project implements a unique test strategy, then the testers moving amongst different projects can potentially become more of a hindrance than a help. The time needed for a tester to become productive in a new environment may just not be there, thereby reducing the productivity of new testers. In order to address such situations, one should strive to develop a single framework that grows with each new application. A collective thought process on similar lines eventually leads to an evolution of a framework that can meet the test automation requirements.

Test automation frameworks have evolved over time. Following are the three major generations:

1. First generation frameworks are unstructured, have test data embedded into the scripts and normally contain one script per test case automated. Scripts are mainly generated using capture and replay tools but may also be manually coded. This kind of script is virtually non-maintainable; when the tested system changes, the scripts need to be written all over again.
2. Second generation frameworks have scripts that are well designed, modular, robust, documented and (thus) maintainable. The scripts not only handle test execution but also other activities (like set-up, clean-up and error detection and recovery). Test data is still embedded into the scripts though, and there is one driver script per test case automated. Code is mostly written manually. Both implementation and maintenance require programming skills, which test engineers may not have.
3. Third generation frameworks have all the good characteristics already found in the second generation frameworks. However, they go further by taking test data out of the scripts. This has two significant benefits. Firstly, one driver script may execute multiple similar test cases by just altering the data. Adding new tests is trivial then. Secondly, the test design and framework implementation are separate tasks—the former can be given to someone with domain knowledge and the latter to someone with programming skills. This concept is called data-driven testing. Unlike Keyword-driven testing, this takes the concept even further by adding keywords driving the test executing into the test data.

Next: A real life example of the need for a Test Automation framework.