James asks: “What is the main difference between Smoke Testing and Sanity Testing? Are they part of Blackbox or Whitebox testing approach?”
Gokul asks: “What is Build Verification Testing? When is it done? Is it the responsibility of the developer or the tester?”
Ridhima asks: “I am too confused about Smoke and Sanity Testing. Are they same? Or is there any difference between them? If yes, what are the main differences between the approach followed in Smoke Testing and Sanity Testing?”
Testing FAQs like the above keep flooding my inbox! Also I come across such testing questions in other places like online Testing Forums/Testing Communities/Testing Discussion Groups etc. And it looks like everybody (who knows something about testing) has his/her own answer for these type of questions! This amazes me. Terminologies in Testing can be dangerous! A term might mean differently to different persons in different contexts. And the above FAQs present an almost perfect example of how testing terms can lead to confusion!
However, after seeing a wide (wild!) variety of views (on the differences in Smoke Testing and Sanity Testing), I decided it was about time to throw in my 2 cents and write a post about it.
But before I can explain I will like to tell you a small story! “One fine morning Mr. Tester was about to start for office and, much to his distress, he suddenly found out that his old bike would refuse to start! It was Product Release time in office and Mr. Tester was already getting late. The release deadline was round the corner and he had to test an important new feature of the product the very same day! Accidents happen. When you are lucky they happen with others. This time, Mr. Tester wasn’t!
He kept trying to wake up his sleeping old bike but it looked as if it was dead forever! After a considerable amount of struggle and some black patches on his fingers (due to grease) Mr. Tester realized that it was already too late to take it to a garage and started for office by a taxi instead! How he reached office almost one and half hour late (thanks to the Chennai traffic!) and how he had to face his fuming Test Manager is left to the imagination of the readers!
However, after all this Mr. Tester made up his mind to take a new bike and to get rid of the old one. Next weekend, he was seen in the nearest Bike Showroom, talking to the salesman. He was looking for a bike with fair enough pick-up and with good enough fuel consumption. Soon he was able to spot such a model. [At least the bike Company and the Salesman claimed so]. But being a tester himself, how could he take their words for granted without testing it himself! So he wanted to take a test ride and was keen to see the basic features of the bike. He was interested in checking the pick-up, the balance, the control while riding, disk brakes, the digital display etc. However, he was quite satisfied with the result of his test (ride) and brought home the bike.
Now, the poor bike was to go through another series of tests [Poor bike! Must be cursing it's fate for being sold to a crazy "tester"!] This time Mr. Tester was interested to test the performance of the bike over a period of a couple of months. This time features like the fuel mileage, wear and tear of basic mechanical parts of the bike etc were under the scanner! Fortunately (for the Bike Manufacturer), Mr. Tester was satisfied with his findings and has been riding the bike to office (and elsewhere) till today.”
Moral of the Story: Well, all stories are not necessarily attached with morals. However, this story has certainly got some testing morals.
a) Taking a test ride to test the basic features (functionalities) of the bike can be compared to “Smoke Testing” a product. In the above story, while taking the test ride, Mr. Tester was determining if the basic features of the bike were stable and acceptable. In a typical testing environment, when a build is received for testing, a smoke test is run to determine if the build is stable and can be considered for further testing. Testers usually do a Smoke Testing before accepting the build for further testing. The tester "touches" all areas of the application without getting too deep into the functionality.
b) Testing the bike performance in detail after bringing it home can be compared to “Sanity Testing” a product. Testing those features in detail was not possible in the showroom or while taking test ride. In a typical testing environment, when a new build is received with minor modifications, instead of running a thorough regression test suite, a sanity test is performed so as to determine that the build has actually fixed the issues and no further issue has been introduced by the fixes. Sanity testing is generally a subset of regression testing and a group of test cases are executed that are related with the changes made to the product.
Smoke Testing Vs. Sanity Testing:
1) “Smoke Testing” is usually done on the nightly/interim build to test its stability. Therefore “Smoke testing” is often called as “Build Verification Testing” too. In contrast, “Sanity Testing” is usually done during the later cycles after thorough regression cycles are over. When multiple cycles of testing are executed, “Sanity Testing” is done towards the Product release phase.
2) “Smoke Testing” is done following a shallow and wide approach where all the basic and major areas are tested without going too deep into the functionality. In contrast, “Sanity Testing” is usually a focused but limited form of regression testing, which follows a deep and narrow approach to test a particular functionality in detail.
3) “Smoke Testing” is done by developers before the build is released or by testers before accepting a build for further testing. On the other hand, “Sanity Testing” is done mostly by the testers.
4) “Smoke Tests” are mostly in form of scripted form (either written test cases or automated test scripts) whereas “Sanity Tests” are mostly non-scripted!
5. “Smoke Testing” can be compared with the normal health check-up of the product whereas the “Sanity Testing” can be compared with some specialized tests to reveal possible problems with a particular functionality of the product!
Hope this helps in explaining the concept of “Smoke Vs. Sanity Testing” in a better way. Feel free to comment on my views by leaving behind your comment. If you know any better way to explain it, please share with me by leaving your comment. Thanks.
Happy Testing …
Similar Posts:
Read More »
Update: Please refer to the "Comments" section of this post for more interesting facts/analysis on this bug!
