Testing Process Audit. The term “audit” might sound a bit like an official financial inspection, doesn’t it? But it’s not. Did you know that “audit” derives from the Latin word audire, which means “to hear”? So, during our testing process audit, we want to hear you. In the following section, we will explain how we get to listen to what is relevant to you.
What is a testing process audit, and how we do it
Our testing process audit can last between one week and a month, a period in which one of our experts will perform a solution-oriented analysis of your testing process. This means that we will strive to find the problematic aspects in your testing process, anything that might be blocking you from achieving your business goals. Based on our findings, we will also offer solutions suitable to your context.
In the following, I will mention different aspects that we might analyze in a testing process audit depending on what’s relevant for your situation.
Test strategy and testing approaches
In this step, we want to understand the testing process’s high-level goals and how they fit your business objectives.
Entry and exit criteria for items in testing
We also wish to help you determine the prerequisites that should be met before starting the testing and the items that should be fulfilled before completing the testing.
Testing visibility of testing activities and testing progress
The visibility of testing activities and testing progress is very important for the team, the stakeholders, and other key people involved in the project. We would be looking and analyzing your reports, working notes (if you have any), and whether or how you share your test results.
Communication flow in the testing/project team
When team members explain what has been tested and how, or when they talk about potential risks, the project moves forward more effectively. Communication is essential in a testing process and is worth doing it as well as you can. We will analyze this aspect by looking at how visibility is shared within the team. We’ll either talk to different team members or participate as observers in relevant meetings/discussions.
Testing techniques used vs. the ones that are suitable
Many techniques can be used in the testing process, and each has core strengths and weaknesses. Based on your focus and what you want to achieve, some techniques will be more relevant than others. We want to look at the currently used ones and propose others that could be more suitable for creating strong tests.
Metrics used to measure testing effectiveness
When we talk about metrics for evaluating the effectiveness of testing, we refer to how well we tested something. We count bugs, the number of tests, the complexity of tests, etc. It is essential to understand that a single metric does not provide a good picture, that there are no simple models that can be just taken and applied. We will help you discover the suitable metrics for your project’s context and set a good interpretation of the data needed to avoid the introduction of malfunctions in the testing and the testing team.
Clarity of the testing quality and product status
When evaluating a testing process, we look at how qualitative the testing is. We assess if we have good coverage and complex tests and how clear is the quality of testing and the condition of the product for the ones who make decisions.
When all the planned tests are run, the product owner should understand what kind of tests were covered (only basic scenarios, or complex ones, or risk-based), which areas remained uncovered, and if additional tests are needed, or if the product is stable enough.
As a test team, we consider it important to be able to give this information and to know how to say this in a report. During our audit, we will give you examples of how you can do this and how you can make an informed assessment of the quality of the testing process and product status.
Skills available in the team vs. the ones that are needed
During the audit, we want to help you find out what set of skills your team has and, based on your context, detect any additional skills needed to perform particular tasks. We will also advise you if any training is needed for developing new skills.
Tester/developer ratio and task distribution between testers and developers
We believe that the industry’s standard ratios are meaningless out of context. To form an opinion about this, we want to understand how the backlog and daily workload are distributed between testers and developers. Also, we’ll analyze if any missing features, blocking bugs, inconsistent designs, or undocumented requirements are aspects that drive down the team’s productivity.
Tracking of test coverage
Tracking the test coverage is useful to have an overview and a good understanding of the test progress. It helps you know how much is left to test and how confident you can be on the quality of the application you’re testing. Whether you’re tracking the code, requirements, or specifications coverage, we want to help you understand if those are enough or if there are other types of coverage worth considering. We will also look over any matrices and checklists and identify other documents needed to track or record the progress of your testing.
Testing tools used and their efficiency
There are many testing tools available to ease our testing process nowadays. For example, when it comes to automation, we could ask ourselves what kind of test tools are needed in our project, whether they are available and what skills we have in the team. Or if we have the right configuration and equipment required to execute the tests. Our audit will also look over the testing tools you use and how efficient they are in your context.
Bug investigation and tracking
Since bugs and bug reports are important outcomes of testers’ work, highlighting the tester’s efforts to investigate a problem, bug investigation and tracking are two other aspects that we would help you with during the audit. For this, we would propose that one of our specialists would do a pair testing session with one or more of your team members to observe how bugs are investigated, reported, and tracked in your project. We will be looking at who takes the load to do the follow-up tests and how much information is considered enough by the programmers for efficient debugging, and we will provide hints on how this aspect can be optimized.
When and why is a testing process audit helpful
If you made it so far, you hopefully have a better idea of what a testing process audit is, so let’s get into the when and why.
It’s important to understand that our testing process audit won’t solve your problems in itself; you will need to act and implement our recommendations. It won’t give you a “silver bullet” type of solution either. It’s meant to answer the questions: “What’s not going well?”, “What am I missing?”, “Where do I need to improve?”, “What do I need to change to have better results or my testing team/product team to perform better?” giving you the power to make smart and effective decisions.
We’ve gathered a few scenarios when you, as a team leader or stakeholder, might be on the verge of making important decisions but can’t because of incomplete information. Our intention here is to help you realize whether or not you are in that position.
An increase/decrease in the number of team members
Many times, you might find yourself in the position of employing new people in your team. But do you really need more people? Or is there something else you need, in fact?
Often when you know you might have a problem that you need to fix, you end up hiring new employees. But, before trying to solve anything, it is essential to understand what the real issue is. Otherwise, you risk jumping to solve a problem, which, in fact, might not be the real one. The audit will give you the opportunity to understand if you need more people or if your team could work on new skills or improve specific ones.
Starting a new collaboration
Another example is when you, as a client, want to start a collaboration with us. In this case, a testing audit could be considered a pilot project that we could have. This type of short-term collaboration would be a perfect opportunity to know each other and work together. It would be a way to decide if a partnership would be beneficial for both parties.
Starting a new testing team
We often hear about companies struggling to organize their new testing team, not knowing where to start, what to focus on, and how to build a testing process. This is a perfect case for you to benefit from our expertise in helping you define a clear testing mission, create your test strategy and test plan, and understand what to pay attention to.
Changing methodologies
It is common to make changes in your process, such as swapping from one methodology to another and then question its outcome. Our testing process audit might come as an extra hand while you do these changes to help you understand if you’re on the right track on adapting the testing process to the context imposed by the new methodology.
The process is not working as expected
Another scenario could be when you have a process in place, but it’s not working the way you or your team anticipated. You feel there is a need for change, but you don’t know what is needed exactly. Changes or improvements within an existing process are sometimes welcomed, and knowing about the problems will help address them and eventually solve them.
Client complaints are piling up
Have you ever gotten complaints about the product or about a feature that is not working as requested? Have you gotten negative feedback from your customers right after the release of new functionality? The feeling of disappointment and failure could eventually pile up in time, leaving little room for impartially understanding why the product, feature, or even the process fails to meet the expectations. Having us join your analysis with an objective and critical eye could eventually help you understand what determines the piled complaints and how you could address them and forestall them in the future.
The team’s hesitant in releasing new features
Have you ever heard “No, the feature is not ready to be delivered yet” close to the release date? When this happens once or twice, it’s nothing to worry about, but when this is your reality, most of the time, you could have a process problem. Some team members say “No,” but others just think it, have doubts about the features, or fear an unknown issue might appear. All these hesitations frequently lead to repeated delays. This is a good moment to turn to an audit of your testing process and get to the bottom of what’s keeping your team from releasing new features with confidence.
Company/department restructuring
One other example is that you did a restructuring, and you want to see if it was successful or not. Restructuring usually implies changes in the organizational structure, reallocation of resources, ownership changes, goal changes, etc. These might impact the testing teams as well, on how and with whom they used to work, ending up with confusion and some degree of uncertainty and indecision. In this case, we can look at your processes and see if there’s something else that needs to be done.
Every one of these scenarios leads to blockers in the delivery process. Without the right information, your very impactful decisions might be suboptimal. Just as testing makes a product better, a test analysis makes the process better. Testing is a natural part of a product, as is an analysis of the testing process.
What dimensions of the audit do you think would be most relevant in your team or project? Let me know in the comments below.