What to test? Pareto analysis, High-risk areas, User feedbacks…

Recently during a team discussion with my Manager, he came up with a question – “What is the best use of our testing resources? How should time be allocated? Which areas should we test more often?” Luckily for me, he clearly understands that it is not possible to test everything (let me tell you, not many managers understand this simple fact). He also knows how from an economic stand point, it does not make sense to spend lots of time testing areas of the application where the chances of having defects/bugs are low, or even if defects/bugs are there the impact of those defects would be low as far as the end users are concerned. So he was curious to explore the kind of strategy we should follow while deciding a roadmap to test our upcoming yearly Corporate Release. This might sound like a simple question, but in real environment one can imagine how complex it can get to formulate a testing strategy to test a release.

Well, testing is not all about hitting the keyboard and clicking the mouse until you stumble upon some hidden defects in the application. Finding corner-case defects that nobody in real environment would ever come across might not make much sense. Similarly finding defects, which are trivial and have least chance to trouble a real-time user can eat away your precious testing time. At the same time, missing important defects could be vital for the reputation of your application after release and also for your own credibility as a tester. Keeping all these things in mind, often it becomes bit tricky to adapt a testing strategy that answers all these possible loopholes and yet helps the testing team to perform adequate testing on schedule.

Coming back to my Manager’s question, he had a suggestion for his own question - to focus most time on high-risk areas of the software and less time on the areas of less risk. In other words, test deeply and frequently in areas of high risk, and spread the rest of the testing effort broadly and less frequently over the areas of less risk. But now the question was how to decide which are the high-risk areas and which are of low risk. Some more brainstorming, and we came up with these points to start with:

1. Pareto Analysis
2. Try and identify High-risk areas
3. Analyze User feedbacks

Pareto Analysis:
As most of you might already know, Pareto analysis is a statistical technique in decision-making that is used for selection of a limited number of tasks that produce significant overall effect. It uses the Pareto principle (80-20 rule) - the idea that for many events, 80% of the effects come from 20% of the causes. Or in terms of quality improvement, a large majority of problems (80%) are produced by a few key causes (20%). [Wikipedia]

In case of our Product, we use JIRA for issue tracking (bugs, user stories, improvements etc). And we decided to filter out last 5 years (coinciding with the first beta release of our Product) JIRA bugs (with Resolutions ranging from Unresolved, Completed, Fixed, Insufficient Details, Cannot Reproduce) and perform a Pareto Analysis on the filtered bugs to see if a pattern exists. Though personally I don’t believe much on statistics, this time the findings were interesting (if not shocking)! In our case, the Pareto Analysis showed that 80% of the bugs that were reported in last 5 years were present in areas constituting just 25% (well, little more than 20% though) of the whole application. Now we had at least something to start considering as the high-risk areas in our Product.

Try and identify High-risk areas:
Risk analysis can be used to decide where testing should be focused. As it is next to impossible to test every possible aspect of an application, every possible combination of events, every dependency, or everything that could go wrong, risk analysis becomes necessary (if not mandatory) for most contexts of software development. This might require analytical/judgment skills, ability to think like an end-user, common sense, and past experience. Some of the considerations can include:

» Functions those are often used by the users.
» Functionalities those are most visible to the user.
» Complex functionalities.
» Functions that have a lot of code refactoring, updates or bug fixes.
» Functions that require a consistent level of performance.
» Functions that use new technologies.
» Functions that require interfacing with external systems.
» Functions that use third party plug-ins/tools.
» Functions developed by more programmers at the same time.
» New functionalities.
» Functions developed under extreme time pressure.
» Functions those are most important to the stakeholders.
» Functions that reflect complex business logic.
» Functionalities that have large financial impact (e.g. shopping cart, online transactions, payment gateways etc).
» Functionalities that have large safety impact (e.g. health care products, life critical systems, software to be used in automobile/airlines, navigational gadgets etc).
» Functionalities that have large security impact (e.g. login forms, database transactions, authentication and authorization criteria etc).
» Talk to your developers and try to identify parts of the code that are most complex, and thus most prone to errors.
» Similar/related functionalities with bad history of defects.

Analyze User feedbacks:
User feedbacks often provide good enough idea about the areas in which the end users have most chance of finding defects/problems. If you have a Product support team for your software, keep your eyes and ears open to the help desk call transcripts. By doing so you can have an overall idea of the risk areas that your end user is prone to hit upon! You can seek help of your Business Analyst to get some ideas on the areas that she thinks as high-risk zone.

Once you have a clear picture of the high-risk drive-slow zones of your software, you would be in a far better position to plan out a strategy for your Testing efforts. Test deeply and frequently in areas of high risk, and spread the rest of the testing effort broadly and less frequently over the areas of less risk. However, I am curious to hear your stories. How do you decide the high-risk zones in your software? How do you allocate testing effort (and time) for different areas/components? Share your ideas via commenting.

Happy Testing…

Share on Google Plus

About Debasis Pradhan

Debasis has over a decade worth of exclusive experience in the field of Software Quality Assurance, Software Development and Testing. He writes here to share some of his interesting experiences with fellow testers.