This site uses cookies and by using the site you are consenting to this. We utilize cookies to optimize our brand’s web presence and website experience. To learn more about cookies, click here to read our privacy statement.

Putting Automation Solutions to the Test

Author: Sara Vallejo Posted In: Testing
SPR recently presented a Testing Leadership Workshop at Chicago’s Willis Tower. The solution-packed afternoon drew testing leaders and practitioners from companies including CDW, Northern Trust, and Paylocity.

SPR’s Nancy Kastl, Practice Director of Testing Services, and Joey Figliulo, Testing Sales Specialist, hosted the event, joined by facilitators Dai Whaley, VP of Quality Engineering at Applied Systems; Ryan Burlage, Solution Architect at Tricentis; and Matt Mead, CTO at SPR.  

Automation a Top Priority

The workshop was designed to allow participants to identify their challenges in software testing and for group discussion of potential solutions based on experiences. Nancy and the SPR team sent a survey prior to the workshop to gather the participants’ most pressing testing challenges, and found that test automation and resources were among the top challenges.  

Automation was the theme throughout the workshop activities. An informal poll of attendees showed that around 90% were doing some form of test automation.  

During two workshop activities, attendees captured issues with automation on sticky notes and stuck the notes to the event room’s walls. Broken into teams, attendees reviewed their challenges, identified duplicates, grouped similar challenges, and then selected their top two automation challenges.  Several common challenges arose.  

Top challenges our attendees are facing include: automation resources and skills, whole team ownership of automation, test script resiliency and test coverage, automation tools, data creation and management, and in-sprint automation.  

Challenge 1: How Test Data is Handled

Challenges abound with data. Data can be sensitive and must be handled appropriately, and identifying and obfuscating sensitive data can be difficult and time consuming. In many cases, only a few people understand the data and it can be hard to determine which data is the right data. Waiting for proper data can take weeks, and masking and stripping PII from sensitive data can add to that timing.  

The attendees then brainstormed solutions, and one solution rose to the top: data governance. Setting rules who can touch which data, data segregation, appropriately reverting data at the end of an automated test, and data assignment or reservation can help make test automation run more smoothly. Attendees agreed that a mindset shift is critical, and that upskilling and training are often necessary.  

Challenge 2: Getting the Whole Team to Own Automation

Automation faces even greater challenges when only Quality professionals own automation. For success with automation, a whole-team approach to ownership is vital. Share on X Attendees highlighted struggles with lack of up-to-date documentation, lack of ownership around application and code quality, and lack of well-defined acceptance criteria for releasing a new build to QA or production.  

One resounding challenge we heard again and again was time. The race against the sprint cycle makes achieving quality software difficult, with quality often sacrificed to achieve a greater speed to market.  

When brainstorming solutions, it became clear that team building and leadership were essential in tackling this challenge. A top-down quality culture and mindset is critical to holding all parties accountable for quality, while bottom-up team building can foster a shared ownership of quality and greater respect between team members. Getting developers and testers to tackle problems together can go a long way. Setting metrics and goals can also drive accountability and entrench quality in the team culture.  

Bar Stool Solutioning

Attendees also participated in a short-form solutioning activity where they could volunteer to sit in the hot seat (a bar stool) and ask the group for help with particular challenges. We covered three challenges during this activity.   

Increasing Test Coverage 

Attendees suggested documenting where you’re currently at in coverage and setting a target or goal. Not everything can and should be automated, so determine what level of automation you want to reach. Always keep the user in mind when determining coverage, and lean into test driven development through pair developers and testers together for technical and business expertise. Finally, define meaningful metrics to gauge how automation improves coverage and decreases defects.  

Experts in Everything 

The QA professionals in attendance brought up a growing trend: QA resources are expected to be experts in everything. Key solutions were to build a hierarchy of subject matter experts and go-to resources in different topics, have SMEs share their expertise in Lunch and Learns, and to build new requirements for testing (e.g. accessibility, mobility, performance) in as user stories so they get prioritized along with functions and features.  

Onboarding Contractors 

In situations where a vast amount of domain knowledge exists, onboarding new employees and contractors can be especially challenging. Documentation is critical to achieving an efficient onboarding process. Domain knowledge only goes so far and should be shared through short introductory videos or other documentation, knowledge transfer activities, and pairing developers and testers who have the domain knowledge with new automation contractors to help quickly transfer that knowledge.  

New Solutions, New Connections

Throughout the afternoon, testing professionals engaged with each other, working together to solve the common problems most are facing. Not only did the attendees leave with inspiration and solutions to try, they also left with new connections within their profession, enabling them to continue the dialogue long beyond our testing leadership event.