Keeping Up with Change Through Layered Test Automation

A leading provider of consumer marketing data was developing a self-serve portal to provide on-demand reporting to their clients. As the application functionality expanded with each sprint, it became increasingly difficult to complete end-to-end regression testing each time. The organization possessed technical test resources, but not the experience in implementing an effective automated testing solution.

The application under test presented unique challenges as well: The reports used complex charts and graphs that could not be reliably verified using image comparison utilities. The underlying system data also changed frequently, requiring the expected results to be updated.

The SPR team assessed the application, development and testing processes, and test environment to create a strategy for implementing an automated end-to-end regression test. The strategy ensured that the organization had the required foundational components to support long-term success with test automation, including:

• Automation scope
• Framework structure
• Test environment and data management
• Process integration
• Resource requirements

The SPR team then built a Selenium WebDriver-based automated testing framework to support the end-to-end regression test cases at both the API/service and UI levels. The framework was constructed using a layered approach to maximize maintainability and scalability.

• The Core Layer provides base automation functionality including UI interaction, API/service interaction, data management, and logging/reporting. The Core Layer does not require modification to scale automation scope to additional functionality or applications.
• The Application Layer includes application page objects as well as methods that performed common actions that are reused frequently across the end-to-end tests.
• The Test Definition Layer consists of data-driven test classes that utilize methods in the Application Layer to execute testing actions and verifications. This feature was extremely important as the system had a relatively small number of use cases, but a large volume of data combinations for each use case.

To address verifying the complex charts and graphs, the SPR team developed methods to extract the data structures used to draw the charts directly from the page Document Object Model (DOM). The data structures, while still complex, could be reliably validated. The framework also featured a dual execution mode capability to address the need to update expected results frequently. Update Mode allows the test suite to capture or update expected results, and Verification Mode compares actual results against previously captured expected results.

Finally, the SPR team mentored client resources throughout the automation framework buildout to ensure they were capable of executing, maintaining, and enhancing the automated testing solution.

The client now can efficiently create and execute both API and UI level regression tests within the scope of a single development sprint. Further, the target regression test coverage executed during each sprint has been increased, while cycle time has decreased. The Core Layer of the framework will serve as a foundation and accelerator for expanding automation to other applications and project teams.