Automated Testing in Agile Environments

Automated Testing of Agile Projects vs. Waterfall Projects

"Automation is a critical component to maintaining agility, and is a priority for the entire team through established practices/disciplines and a focus on continuous improvement. Continuous integration/builds, unit, functional & integration test execution and continuous/automated deployment are common examples of applying automation beyond the scope of traditional automated tests."Automation is a critical component for maintaining agility, and through established practices/disciplines and a focus on continuous improvement is a priority for the entire team. Continuous implementation of integration/builds, unit, functional & integration test execution and continuous/automated deployment are common examples of the application of automation beyond the scope of traditional automated tests."

"In contrast, waterfall projects will tend not to invest in automation as a priority, primarily due to the phased investment and handoff mindset that accompanies that approach. At best, builds may leverage some automation, and a phase of a waterfall project may be dedicated to functional test automation for long-term support/future release needs, but it’s rare to find much more in a back-end-loaded waterfall project."

Agile Tester Skills

Waterfall projects, on the other hand, may prefer not to invest as a priority in automation, mostly because of the staggered investment and handoff mentality that follows that approach. At best, builds can utilize some automation, and a waterfall project phase may be devoted to usable test automation for long-term support/future release needs, but it is unlikely to see anything more in a back-end load. Waterfall projects, on the other hand, may prefer not to invest as a priority in automation, mostly because of the staggered investment and handoff mentality that follows that approach. At best, builds can utilize some automation, and a waterfall project phase may be devoted to usable test automation for long-term support/future release needs, but it is unlikely to see anything more in a back-end load.

Automation Responsibilities on an Agile Team

"The team! Literally anyone who does anything repetitively more than 2 or 3 times in a short period of time should and will automate it, preferably with a well known tool such as TestComplete, open source options or homegrown scripts/utilities. Testers generally shoulder the development of functional test automation, since that activity can parallel the actual feature development." Testers may also work at one time on several stories or functions, so it is important tools to coordinate and juggle various testing activities. As a measure of success, they would have transitioned to a more collaborative, team-based approach to the iterative/continuous delivery of high quality, working applications.

Deciding what Tests get Automated in Agile Environments

Testers may also work at one time on several stories or functions, so it is important tools to coordinate and juggle various testing activities. As a measure of success, they would have transitioned to a more collaborative, team-based approach to the iterative/continuous delivery of high quality, working applications.

Maintaining and Reviewing Automated Tests in Agile Environments

As tests are often generated by pairs of tester-developer and/or tester-product owner pairs, as part of story acceptance, there is an integrated review phase of the test scope and validation stage. Otherwise, the collaborative preparation to identify the tests, then the collaborative acceptance with the owner of the product provides sufficient mutual information and review. The end result is stable, resilient automated tests.

Ideally, all tests, whether unit, functional or some other, are maintained by the entire team. Testers, however, can use specific GUI-level testing software that require their participation to manage tests over time. The tools should be carefully chosen to be accessible to all team members to minimize the potential silo-ing of this information, information should be readily available, and the test inventor should be available.

Agile Teams Automate Black Box and White Box Tests

A mixture of white-box (under-the-covers) and UI automation (black-box) tests is more common for functional/acceptance tests. White-box tests appear to be less fragile, so they are less costly to maintain, whereas black-box UI-driven tests need to be judiciously leveraged due to the cost of developing and maintaining the tests over time. Preferably, all testers are involved, so the black-box UI-driven tests need to be judiciously leveraged due to the cost of developing and maintaining the tests over time.

Implementing Test Driven Development

By definition, TDD is a method of developer coding (via a unit test framework), not an agile practice of tester. Literally, a developer would write a failing unit test, add the associated code to satisfy the test, then re-run those tests to ensure performance. The resulting code is subsequently refactored/redesigned while keeping the unit tests running 'Green.' Red-Green-Refafa is the traditional TDD motto.'

However, for higher-level functional and acceptance testing, whether automated or manually performed, the same principle (i.e. test-first) is certainly applied. Basically, detailed specifications for a user story are articulated in the form of acceptance checks, which can then be captured in advance in a test automation environment so that they can ultimately run 'green' once the functionality is in.'

When Not to Use Automation in an Agile Environment

When and when not to automate experiments, it is necessary to know that this holds true no matter what technique is used. Troy said that it can not be automated if a test is only done once. "One-time tasks and exploratory testing/edge cases shouldn't be automated. Edge cases are, by definition, usually one-off test cases, and the effort to automate edge cases generally does not pay off. Exploratory tests are best used to gain knowledge of a new feature and then tweak or revise tests based on the new knowledge."

Switching  from Waterfall to Agile

Troy had this to say when asked if she had any advice to send testers who are transitioning from a Waterfall project to an Agile project: "Be aware of the change in mindset/culture. Experiment with modern tools. Keep the big picture of the application in mind and consider individual user stories within the bigger context. Think holistically, and encourage the rest of the team to do so as well."

Functional, GUI, Regression and Performance Testing in an Agile Environment

Troy clarified that in an Agile store, all testing forms have a location, it is just a matter of how they are used. "Unit, functional, acceptance, GUI (some of which become regression), performance and load testing are all used. We try to select the most appropriate tool for the team and the type of test we’re performing, so we have several different tools in our arsenal."

"All of these testing types have a place in an agile environment. However, it’s important to incorporate them in an Agile way, rather than a prescriptive way. Balance the quality goals of the team with the requirements and risk of individual user stories."

Automated Testing Starts on Day One