Test Automation – Can this be done?

It has been many years since the concept of test automation became reality. There have been hundreds of thousands of projects, many trials and errors, with far more failures than successes. I ask myself, “Does someone have the secret formula to test automation success? Is there a one size fits all solution that we are all desperately searching for?”

As a test architect, test team manager, and consultant, I have attended many conferences and training classes, I have interviewed hundreds of professionals, and worked with many colleagues and clients over the years from different backgrounds and with wide ranges of experiences. Some swear by the tools they are using, some focus on the process, and some say nothing ever really worked as they expected. I have asked this question many times – what would the right solution look like? How would you build it? and what would YOU recommend? And most importantly, why?

I would like to share my thoughts on this subject primarily based on what I have witnessed myself over the years and also what I’ve learned from my research of various sources.

Whether you are starting out from scratch or regrouping, this blog may shed some light on how to prepare for this journey.

Many of us who have been in the industry for a while, know this well – when teams are embarking on a testing automation initiative to improve their existing approach or start from scratch, the first question is, what is the right tool? Of course, picking the right automation tool is an important piece of an overall successful automation strategy. There are so many options from coded Selenium WebDriver solutions to scripless automation, there are dozens of other “mainstream” tools and there hundreds of others used across projects all over the world.

Automation tools have improved drastically over the years. Now, these tools can support virtually any programming language and Integrated Development Environment (IDE), opening an array of possibilities for which resources the team can actually create the scripts. With the growing popularity of CI/CD concepts, a variety of automated deployment tools, the desire to test often, but also test in lower environments, the demand of integrating test scripts into deployment pipelines has been on the rise. So, how do we finally make the choice? What criteria do we need to meet in order to make the right decision?

Now that the technology has become less limiting on tool selection, the key considerations are:

• The type of testing being performed
• Who will be drafting the test
• Who will be executing the test
• What environment the test will be executed in
• Frequency of your release cycles

The experience and skillset on the team makes all the difference and should not be overlooked when looking for sustainable test automation solutions. Keyword here is sustainable. Regardless of the tool you choose, this will drive new processes, and it is important to choose the right balance of who will be responsible for scripting which tests. Historically, developers are usually responsible for unit testing to keep the code base clean and maintain code coverage while testers are responsible for the “other” tests, it is these tests that are usually problematic to maintain.

Here again, there is no one size fits all. We must look at the budget, the ratio of Dev/Testers and the skillsets available within both teams.

Before we select a tool, what does your organization look like? Ask yourself:

• Do our developers understand how to write tests?
• Do testers have the ability to script in the appropriate language? Preferably using the same tech stack as team uses to build the software.
• Do we have a budget to hire a team of automation engineers to help with test coverage or do we have the capacity for our existing team to absorb this effort and we can train our folks to do it?
• Do we go offshore?
• How long will it take, do we have time?
• How will this impact our velocity?

These are all key success factors, but very important questions to ask yourself. Some of them may be more difficult boxes to check than others.

As we are sometimes told to look at ourselves in the mirror - the same rule applies to any organization planning for test automation.

Process: Ideally, automation work needs to be part of a delivery process – “baked” into the process as we say. This approach ensures all features receive complete test coverage without delay, as a part of the development life cycle and dodoes not generate additional technical debt in the backlog - something very challenging to deal with later. I would personally vote to avoid any kind of technical debt at all costs. This strategy also applies If tech-debt already accumulated and there’s a substantial backlog of tests that need to be automated. Many successful teams swear by it. Pulling some backlog items into the sprint work will ensure continues progress in tackling backlog. That means that a test automation initiative - net new, optimization, or clean-up - will require investment up front and an initial hit to velocity, but you will thank yourself for it later.

Tools: Tools can be tailored to a skill set of the team and any knowledge gaps can be filled with training and solid processes in place to accommodate changes. Answer the following questions:

• Are my testers technical enough to learn scripting language or perhaps already have experience?
• Can they be trained in a reasonable time frame to script?
• Do we have someone on the team to supervise, teach best coding practices and review the work?

If the answer to any of the above questions is no, perhaps scriptless solution is something to consider.

If the answer is yes, then exploring an open-source tool might be the right strategy.

Planning: Lastly, many teams make mistakes by jumping right into building scripts without giving it much thought. Careful planning and preparation are very important here. Prioritize the type of tests you want to start with – maybe it is API, Performance or the UI. Asking yourself which area has the highest rate of defects might help make this determination.

Your manual test suite should be cleaned up and prepared for conversion to automation before any automation efforts can start. Define processes, standards, objectives and set some reasonable goals. Bring your teams along on this journey, supporting change management through providing resources, communication, and feedback loops.

Today’s popular trends of Continues Integration and Continues Delivery require support of well-established test automation practices to sustain quality of the product or application with every release. In order to build this practice properly and continue to mature it over time, it has to be defined, managed, measured and controlled by the organization just like any other process. Once a certain level of maturity has been reached, it has been reported (Quantitative Study of open source projects using continues integration – Yuqing Wang, Mika V. Mantyla, Zihau Liu, Jouni Markukula) that supporting test automation does not require a significant increase in overall effort. It becomes a part of the overall delivery process and therefore treated as normal operations as opposed to often being viewed as an additional burden.

While there’s no one-size fits all solution for successful test automation practices, one thing is for certain – it requires careful planning on all organizational levels. Following best practices is essential for success. Evaluate your team, their skillsets, the technology stack at hand, define success criteria .criteria. Research the tools that will meet that criteria and identify a pilot group to try it out. Start small, observe the outcome, make adjustments and do it again. It is a long journey for many, but well worth it.