Quantitative and Qualitative Questions to Define a Test Assessment
The Modern Testing practice at SPR takes great pride in our flexibility in offering a wide variety of assessment and coaching services. We work with our clients to help them take an honest, comprehensive, holistic look at every aspect of their software testing and quality assurance people, processes and tools. We do this with a focus on the latest-and-greatest methodologies and processes within software development: DevOps, Continuous Integration, Development and Testing, Kanban, etc.
Even while looking to our client’s future, the roots of our assessment and coaching process are firmly planted in the original Six Sigma processes defined by Motorola in the 1980s: Define, Measure, Analyze, Improve and Control.
THE SPR APPROACH
Our test assessment process derives from the DMAIC categories. SPR describes it in four steps:
- Scoping – Setting the course for the assessment (Define)
- Investigation – Gathering the necessary information to make the right decisions (Measure as applicable, and Analyze)
- Reporting – Communicating the findings and recommendations (Analyze)
- Transformation – Implementing the findings and recommendations (Improve and Control)
Implementing the recommended changes to Improve and applying a Control to adjust the solution as time goes on is treated as a separate Transformation initiative with Modern Testing. Many clients choose to handle the implementation of recommended changes themselves following the Reporting phase of the assessment.
It is of critical importance the Scoping process goes smoothly and produces a set of agreed-upon deliverables and personnel to be involved with the rest of the assessment. Without understanding the expected goals and an overall agenda for the project as it begins, it is destined to not deliver what the client is looking for. Any errors made in setting the initial trajectory become magnified as we iterate through the Investigation, and our Reporting is nearly assured to be well off-course.
HITTING QUANTITATIVE TARGETS
The Scoping phase of an assessment includes both quantitative and qualitative aspects of a client’s approach to testing. Our quantitative approach tries to focus on four business needs as they relate to testing:
We think of those four points as a “square” which must have a fixed area. The center of the square is a point where each of the criteria is at its optimal possible value. If a project wants to go faster and pull Speed toward the middle, one or more of the other three points must push out: either you will have less Coverage, a higher Cost or an increased Risk.
Our goal at the start of an assessment is to understand, for each client, the priority of improving each of these factors and making sure the stakeholders understand the impact of transforming testing around those improvements.
THE QUALITATIVE “WHO,” “WHAT,” AND “HOW”
In addition to quantitative goals, we also want to understand qualitative feedback from each client around three simple questions:
- Are you happy with who is doing your testing?
- Are you satisfied with what is being tested?
- Are you comfortable with how testing is being done?
The Who, What and How, as stated, is very generic and relies on hard-to-define adjectives (happy, satisfied, comfortable). When we work with each level of people on a client team (analysts and engineers, managers, and the executive level), these questions become more detailed and tailored to focus in on each person’s knowledge and responsibilities. For example, “who” questions for each level might include:
- Director: Do you feel you are getting value for your money from the people currently doing your product validation?
- Manager: Do you have issues with the skill level and technical capabilities of your team of test analysts?
- Analyst: Is it clear who is responsible for identify and executing testing tasks within your current project team structure?
These questions are also adjusted to bring people from outside of a testing organization into the assessment. Developers, project managers, business product owners, and others will all have insight as to what is working well and what is going poorly within test delivery. Looking for the exact same information from them as you would from the test analysts or their management structure would not be very insightful, however.
Coming out of the initial Quantitative and Qualitative discussions are a series of target areas within each client which we focus on for the Investigation portion of a test assessment. We avoid trying to fix things which aren’t broken, change processes which may not be able to change, or recommend solutions to problems that don’t exist. We start out down the right path toward Reporting and future Transformation.
In the next installment of this blog, we’ll look at the Investigation process, and the evidence gathered to confirm or invalidate the guidance we get from the Quantitative and Qualitative scoping.
