Priority Report!

Dolly: The more I read on the net about Bug Severity and Bug Priority the more I am getting confused. Everyone out there seems to have his own way of describing it and it leaves me even more confused. Can you help?

Victor: Why do we have two fields in a defect tracker to fill in - Severity and Priority? Are not they all the same? I have noticed that when a defect is of high severity usually it is also assigned with a top priority and vice versa. If they mean the same, why do we have 2 different names and 2 different fields in the defect tracker? Why do we need these two different parameters (severity and priority) for a defect? Can't we do only with one?

Krishna Prasad: I want to know about Priority and Severity of software bugs. Please clarify my doubt with one example of each for the following:

1. Low Severity and Low Priority

2. Low Severity and High Priority

3. High Severity and High Priority

4. High Severity and Low Priority

Once again I am back with few FAQs from my mailbox. And this time they are regarding the age-old question of Bug Severity VS. Bug Priority. I can imagine how confusing it can become for someone new to the S/W testing field. But before I can try to offer any kind of solution to dismiss this confusion, kindly allow me to present you with a small story!

One afternoon at a theater:
One fine Saturday morning when Ashley (the female protagonist of my story) is still enjoying the warmth of her bed, much to her annoyance, the cell phone rings breaking the silence of the room; as if yelling to make her realize that its 8AM already and the world out side her hostel room is up and on its feet. Lazily she reaches out for her cell phone and as she looks at the screen a smile flashes across her face, even without her knowledge. It’s her BF on the phone. He asks if she can bunk her weekend MBA class and watch the noon show of the latest blockbuster in the theater near to her hostel! Ashley thinks for a moment and nods in affirmative with the typical unforgettable blushing smile flashing across her face. :) Everything seems to happen as per the plan [telling lies to her roomies, bunking the important class, reaching the multiplex in time; something that does not happen quite often :)...] until, to her horror and surprise she finds out in the theater that the person sitting behind her is none other than her own cousin who also stays in the city! Oops…

Moral of the story:
How would you describe the situation (in terms of severity)? Critical, Major, Normal, Minor... Well, assuming that Ashley’s cousin is unaware of the love story I would imagine that she would probably consider this as a Critical situation. Won’t she? Now considering that we have a critical situation (bug/defect) at hand... How fast she would want to fix this up (either by taking her cousin on a Lunch and begging her not to tell about it to anybody or by simply trying to get even with her and threaten her saying that if she opens her mouth about it she would doom her by telling about the cousin's own secret love story to others) describes the priority! The more desperate Ashley is to fix this situation up, the higher the priority! Isn't it? :)

Bug Severity:
WordWeb dictionary defines the word “Severity” as - Used of the degree of something undesirable, something hard to endure, extreme plainness. Severity of a defect/bug tells us how undesirable the defect is. For example, a bug that causes the program to crash and terminate would be considered as high severity, while a minor spelling error might be of low severity. In our organization we use a 7-point scale to rate the severity of a bug:

Blocker: This type of bug blocks development and/or testing work. Blocks users from completing the task the function was created to facilitate.

Critical: The software crashes, hangs, or causes you to lose data. E.g. Crashes, loss of data, severe memory leak.

Major: Major loss (or lack) of function. Users cannot do what they NEED to do, with no workarounds.

Minor: Minor loss (or lack) of function. Users cannot do what they WANT to do or a major problem where an easy workaround exists.

Trivial: Cosmetic problem like misspelled words or misaligned text. No loss of function.

Enhancement: Request for new feature or enhancement.

Unprioritized: The reporter has no opinion on the criticality of the issue.

Though this can vary from organization to organization, the overall practice may remain similar. You are free to adopt any terminology that suits you as long as they describe the impact of the bugs accurately and unambiguously.

Bug Priority:
WordWeb dictionary defines the word “Priority” as - Status established in order of importance or urgency. Priority of a defect/bug tells us how soon it is required to fix the problem. Priority reflects a business decision as to how soon that bug should be fixed. Priority of the bug determines what gets fixed next and what does not. The priority of a bug can be decided from either the Project management point of view or from the user’s point of view. A bug that is of high priority from the Project management point of view may or may not be of high priority if assessed from the user’s point of view. Check with your users; their prioritizations may surprise you! Priority of a bug can be classified as:

P1 – FIX the bug ASAP before the release, preferably in the current development iteration itself.
P2 – The Bug can be fixed in the successive iteration. But the fix must be done before the next major release of the Product.

P3 – The bug can be left to be fixed in the subsequent releases.

P4 – There is no hurry. This bug can be fixed as and when time permits.

Bug Severity Vs. Bug Priority:
The severity of a bug does not necessarily translate into the urgency to fix it. A severe bug that crashes the application only once on a Feb 29th (leap year) for 0.0001% of the users is lower priority than a mishandled error condition resulting in the need to re-enter a portion of the input for every user every time. Many bugs cause crashes, but aren't fixed because the crash is very infrequent or on a version/platform/feature low on the vendor's support list. Corner-case crashes, crashes that are dependent on a rare combination of sequence of events (trigger to set off the crash) fall under such classification of low priority bugs. Once I was testing the Parser of my application. And the Parser used to crash when the parameters were: 'MaxNulls' = '999999999' and 'MaxPrefixes' = '999999999'. This was a catastrophic crash, which was causing the Program to terminate. However, it took almost 2 years to get this crash fixed as it was rated with a low Priority (despite of its high severity) for obvious reason! :)

ATM Machine Example!
Let’s take the example of an ATM machine to further understand the different possible combinations of bug severity and priority:

Low Severity and High Priority Bug: Suppose you are testing an ATM machine and you notice that the welcome screen displays a misspelled Bank name! (e.g. in place of “Welcome to ICICI Bank” it rather displays “Welcome to UCICI Bank”. This is quite possible, considering the fact that the letters “U” and “I” share close neighborhood in a typical QWERTY type keyboard). This might be a cosmetic error and would not hamper the functioning of the ATM machine in any possible way; still this could prove to be a real pain in the eyes for a customer. Considering that the spelling error is in a frequently used part of the program, it might give an overall bad impression that could hurt the Bank’s reputation. These kinds of bugs are real annoyance for the customers and no Bank would want to loose revenue for a silly error like this one. So though this bug is of low severity, its priority would be high for obvious reasons. It is quite interesting that often, low severity cosmetic errors like these get a high priority.

High Severity and Low Priority Bug: Think of a bug that cause the ATM machine to black out if an user tries to use an expired ATM card and try it for 13 consecutive times (even after the machine keeps rejecting it)! Sounds like a critical crash? Well, it might be a critical bug as far as its severity is concerned, but don’t be too surprised if you see a low Priority being assigned to it. Reason? I think its obvious!

High Severity and High Priority Bug: These kinds of bugs are easier to imagine. Any kind of bug that results in some major catastrophic failure (crashes, system hang, memory corruptions, data loss, functionalities that does not work and so on) and that occurs in areas of the application that are frequently used can be classified under such types of high severity and high priority bugs. Think of scenarios where the ATM machine - does not detect a valid card even after entering correct PIN number, hangs if tried with an invalid card and/or PIN number, blocks the card after just 1 unsuccessful entry of PIN (instead of 3 wrong entries), does not dispense money even if there is sufficient cash in the a/c and in the ATM machine, dispenses incorrect amount to the user, does not forget the previous session even after the transaction is complete (in case of ATM machines where you have to swipe your card instead of inserting it), does not return the card even after the transaction is complete and so on. As you can imagine all of these cases of bugs can fall under high severity and high priority type.

Low Severity and Low Priority Bug: These are often low impact low on urgency-meter bugs that are quite harmless and can be fixed without a hurry. These kind of bugs do not harm the application in a disastrous way and the chance of an end user/customer being annoyed by it is also very less. Think of a scenario where the ATM machine does not append a title (Mr/Miss/Mrs) to the name of the customer in the welcome screen. This would not affect the business in a drastic way. And there might not be many customers who would mind it if their name is displayed without a title.

Hope I was able to make the distinction (Bug Severity Vs. Bug Priority) clear through the examples. Now here is a small testing exercise for you. Why don’t you try and present few more examples (at least one from each combination viz. high severity and low priority bugs, low severity and high priority bugs and so on) taking the ATM machine as example?

Happy Testing…

Read More »

Tags: Article, Bugs, Critical Thinking, FAQs, How To, Interview Questions, Lessons Learned, Paradox, Quality, Software Testing, Terminologies, Testing Stories, Thinking Tester

Repeatability of Tests - A necessary Evil!

“If you cannot repeat what you are testing then you are NOT testing properly! Think of regression testing. What will happen if your regression tests are exploratory by nature and are non-repeatable”? A tester friend of mine argued strongly, trying to give himself ground to continue his fight against my advocacy for exploratory testing!

This made me cogitate. Not that he had succeeded in proving that Exploratory Testing (ET) sucks! But I had to start thinking if repeatability (read reusability) of a test is really all that bad! I guess “repeatable test” VS “non-repeatable test” argument is just another face of the “Scripted testing” VS “Exploratory testing” debate. Here in this post, I am not going to continue the same debate trying to prove which one is better and which one is not. Rather I would like to think more on the repeatability aspect of a test!

Some questions to start with can be:
1) Can ALL the tests be made repeatable?
2) Should ALL the tests be made repeatable?
3) By making a test repeatable are we not (may be inadvertently) making them too predictable?
4) Can repeatable tests discover NEW problems (defects/bugs) in the system?
5) Blindly trying to make each and every test repeatable. Is it worth the effort and expense?
6) Why should we repeat just a set of tests (test cases, test scripts whatever), when we can utilize the same time exploring much more tests that may uncover unknown defects!
7) When we say a test is repeatable, is the test really repeatable in all its senses? Can someone guarantee that the test runs exactly in the same way (environment, concurrently running processes and applications, exactly same DLLs loaded at that moment, machine condition, configuration settings, and so forth) as it had run the last time around?
8) Do repeatable tests guarantee reproducible defects/bugs?
9) A repeatable test can make sure there is no re-occurrence of an earlier defect. Ahh well! But how long? Will these so-called repeatable tests not loose value over a period of time (after a number of iterations)?
10) Is software testing all about repeatability?

The argument in support of making your tests repeatable may hold good to some extent, in certain cases like Regression testing or for that matter Performance testing. But is there any point of trying hard to make ALL the tests repeatable? Here are few interesting quotes from some honorable Testing Gurus/Experts:

Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.
- James Bach, Test Automation Snake Oil, 1996
- Brian Marick's talk Classic Testing Mistakes

By repeating tests we actually make sure we are avoiding other possible defects (remember minefield analogy? Stepping on someone’s footstep actually makes sure we may avoid stepping on a live landmine) thus minimizing our chance of discovering new defects!

Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.
- "Pesticide Paradox", Boris Beizer, in his book Software Testing Techniques, Second Edition, 1990

"Pesticide paradox" compares software defects with that of pests! Imagine a context when a farmer applies certain pesticide to get rid of insects from his crop. There is every possibility that there will be some insects that will survive the pesticide. If he keeps applying the same pesticide, the insects eventually may build up resistance and the pesticide would no longer work! "Pesticide paradox" describes the problem that a regression test series gets less and less powerful as you use it over and over again. When a set of repeatable tests is run for a period of time they tend to pass more often than failing. A test is valuable when it fails and the failure uncovers a bug. Running a set of tests that seldom fail and have least chance of uncovering defects sounds like a bad idea. Isn’t it?

Having said this, does this mean repeatable tests are waste of time? May be not!
1) Think of Performance tests that a tester may need to run over and over again for days in and days out.
2) Think of Benchmark tests.
3) Think of Build Verification Tests/Sanity Tests that need to be run every time you have a new build!
4) Think of Regression tests.
5) Think of a scenario where you need to run certain tests that are important in nature. The tests that verify some very critical functionalities of the application and these tests must be run periodically to make sure those functionalities continue to work without issues.

So, it appears that having some repeatable tests can be helpful for your test project along with those tests that are non-repeatable. As always, it certainly seems to be context dependant and depends on your testing mission/goal! What do you think? Do you think ALL (what that can mean!) tests MUST be repeatable, as if we are not software testers rather some robotic human beings trying to repeat some algorithm already decided in advance! Repeatable tests! Are they the need of the hour or necessary evil? Your thoughts please.

Happy Testing…

Further Reading:
Reasons to Repeat Tests - By James Bach
Repeatability is Overrated - By Elisabeth Hendrickson

Read More »

Tags: Article, Critical Thinking, FAQs, Paradox, Quality, Software Testing, Terminologies, Thinking Tester, Top 10, Top 5