Smoke Testing Vs. Sanity Testing – Testing Stories from an Indian Tester!

James asks: “What is the main difference between Smoke Testing and Sanity Testing? Are they part of Blackbox or Whitebox testing approach?”

Gokul asks: “What is Build Verification Testing? When is it done? Is it the responsibility of the developer or the tester?”

Ridhima asks: “I am too confused about Smoke and Sanity Testing. Are they same? Or is there any difference between them? If yes, what are the main differences between the approach followed in Smoke Testing and Sanity Testing?”


Testing FAQs like the above keep flooding my inbox! Also I come across such testing questions in other places like online Testing Forums/Testing Communities/Testing Discussion Groups etc. And it looks like everybody (who knows something about testing) has his/her own answer for these type of questions! This amazes me. Terminologies in Testing can be dangerous! A term might mean differently to different persons in different contexts. And the above FAQs present an almost perfect example of how testing terms can lead to confusion!

However, after seeing a wide (wild!) variety of views (on the differences in Smoke Testing and Sanity Testing), I decided it was about time to throw in my 2 cents and write a post about it.

But before I can explain I will like to tell you a small story! “One fine morning Mr. Tester was about to start for office and, much to his distress, he suddenly found out that his old bike would refuse to start! It was Product Release time in office and Mr. Tester was already getting late. The release deadline was round the corner and he had to test an important new feature of the product the very same day! Accidents happen. When you are lucky they happen with others. This time, Mr. Tester wasn’t!

He kept trying to wake up his sleeping old bike but it looked as if it was dead forever! After a considerable amount of struggle and some black patches on his fingers (due to grease) Mr. Tester realized that it was already too late to take it to a garage and started for office by a taxi instead! How he reached office almost one and half hour late (thanks to the Chennai traffic!) and how he had to face his fuming Test Manager is left to the imagination of the readers!

However, after all this Mr. Tester made up his mind to take a new bike and to get rid of the old one. Next weekend, he was seen in the nearest Bike Showroom, talking to the salesman. He was looking for a bike with fair enough pick-up and with good enough fuel consumption. Soon he was able to spot such a model. [At least the bike Company and the Salesman claimed so]. But being a tester himself, how could he take their words for granted without testing it himself! So he wanted to take a test ride and was keen to see the basic features of the bike. He was interested in checking the pick-up, the balance, the control while riding, disk brakes, the digital display etc. However, he was quite satisfied with the result of his test (ride) and brought home the bike.

Now, the poor bike was to go through another series of tests [Poor bike! Must be cursing it's fate for being sold to a crazy "tester"!] This time Mr. Tester was interested to test the performance of the bike over a period of a couple of months. This time features like the fuel mileage, wear and tear of basic mechanical parts of the bike etc were under the scanner! Fortunately (for the Bike Manufacturer), Mr. Tester was satisfied with his findings and has been riding the bike to office (and elsewhere) till today.”

Moral of the Story: Well, all stories are not necessarily attached with morals. However, this story has certainly got some testing morals.

a) Taking a test ride to test the basic features (functionalities) of the bike can be compared to “Smoke Testing” a product. In the above story, while taking the test ride, Mr. Tester was determining if the basic features of the bike were stable and acceptable. In a typical testing environment, when a build is received for testing, a smoke test is run to determine if the build is stable and can be considered for further testing. Testers usually do a Smoke Testing before accepting the build for further testing. The tester "touches" all areas of the application without getting too deep into the functionality.

b) Testing the bike performance in detail after bringing it home can be compared to “Sanity Testing” a product. Testing those features in detail was not possible in the showroom or while taking test ride. In a typical testing environment, when a new build is received with minor modifications, instead of running a thorough regression test suite, a sanity test is performed so as to determine that the build has actually fixed the issues and no further issue has been introduced by the fixes. Sanity testing is generally a subset of regression testing and a group of test cases are executed that are related with the changes made to the product.

Smoke Testing Vs. Sanity Testing:
1) “Smoke Testing” is usually done on the nightly/interim build to test its stability. Therefore “Smoke testing” is often called as “Build Verification Testing” too. In contrast, “Sanity Testing” is usually done during the later cycles after thorough regression cycles are over. When multiple cycles of testing are executed, “Sanity Testing” is done towards the Product release phase.


2) “Smoke Testing” is done following a shallow and wide approach where all the basic and major areas are tested without going too deep into the functionality. In contrast, “Sanity Testing” is usually a focused but limited form of regression testing, which follows a deep and narrow approach to test a particular functionality in detail.

3) “Smoke Testing” is done by developers before the build is released or by testers before accepting a build for further testing. On the other hand, “Sanity Testing” is done mostly by the testers.

4) “Smoke Tests” are mostly in form of scripted form (either written test cases or automated test scripts) whereas “Sanity Tests” are mostly non-scripted!

5. “Smoke Testing” can be compared with the normal health check-up of the product whereas the “Sanity Testing” can be compared with some specialized tests to reveal possible problems with a particular functionality of the product!

Hope this helps in explaining the concept of “Smoke Vs. Sanity Testing” in a better way. Feel free to comment on my views by leaving behind your comment. If you know any better way to explain it, please share with me by leaving your comment. Thanks.

Happy Testing …
Similar Posts:
Share on Google Plus

About Debasis Pradhan

Debasis has over a decade worth of exclusive experience in the field of Software Quality Assurance, Software Development and Testing. He writes here to share some of his interesting experiences with fellow testers.

50 Comments:

  1. Debashish;
    thanks for nice post. It reduces confusionfrom my mind related smoke and sanity testing.

    ReplyDelete
  2. Hi,
    Your explanation was crisp and clear. However, it has lead to another confusion. If sanity testing is testing in detail a particular functionality then how is functional testing different from sanity testing.
    Would appreciate your reply.

    Thanx.

    ReplyDelete
  3. @ Subbu,

    Functional Testing is done on a *set of functionalities* of the entire Product, whereas Smoke Testing is generally done on *a particular functionality* in detail! Functional Testing comprises of an entire arsenal of tests to test different functionalities of the Application Under Test (AUT)! In contrary, Sanity Testing is a subset of Regression Test Pack to make sure a particular functionality works as expected even after going through a series of interim releases!

    Hope this helps. Happy Testing...

    -Debasis

    ReplyDelete
  4. You are right? Terminologies may sound different for different persons.

    During an interview I was asked, whether I know what is DCR and BCR.
    I said NO.

    Interviewer asked, whether I have written testplans.
    I said Yes, I have written for CR's i.e Change Requests.

    Then he countered saying, "Then you told you don't know DCR and BCR".
    I asked what's that?
    He said, DCR stands for Development CR and BCR stands for Business CR.

    He was expecting answers in terminologies as per his company? How can we know what terminologies are used in his company?

    ReplyDelete
  5. @ Murthy Deshkulkarni,

    Your above narration cites a good example of how use of abbreviations and terminologies can lead to confusion. But a good tester "asks questions" when he is not sure about something. You could have asked your interviewer to clarify "what he meant by DCR and BCR" when he asked you whether you know about it. That way he would have expanded the abbreviations and you might have gotten a chance to tell more about it.

    Use of abbreviations and buzz words while taking interviews is dangerous. But a good tester asks questions to clear traps. Asking meaningful and important questions at the right time can help you getting out of traps. See if asking questions can help you when you are in a similar situation (not necessarily in an interview) next time.

    Happy Testing...
    -Debasis

    ReplyDelete
  6. Debasis, It's great example..

    ReplyDelete
  7. excellent Debasis..got a clear differentiation now.But got a mild doubt..so according to ur posts can u tell me who performs the smoke testing.is it the tester or developer?

    -raji

    ReplyDelete
  8. @ Rajasri,

    who performs the smoke testing. Is it the tester or developer?

    Hmm... I would like you to read the post once again! :) Hope that should clarify your doubt!

    Happy Testing...
    -Debasis

    ReplyDelete
  9. What a Superb article....enjoyed reading and getting things clarified ...

    ReplyDelete
  10. Hi Debasis,
    Thanx a lot for good explanation with examples.its very useful for new testers like me,

    Bharati

    ReplyDelete
  11. Hi
    Debasis thax a lot for ur explanation with suitable examples.
    I like very much this site

    Regards
    Mallikarjun

    ReplyDelete
  12. Superb article....
    Nice explanation...
    Im not going to forget what is what in my whole life with this example....

    shashipalreddy@born to test

    ReplyDelete
  13. hi
    debasis
    its really wonder full example to diiferentiate between smoke testing and sanity testing ,even i had confusion on this ,today i have seen ur blog and this article is very nice as that of ur blog
    great job
    keep it up

    ReplyDelete
  14. All of you have one thing in common, you guys have lot of free time, no great responsibility, or owner ship of anything, may be something trivial.

    Your discussion shows how navie you are! How the !!! doest it matter what is called what? If you dont have sense of knowing & coming up tailoring a process as the situation requires your unfit to be QA! Definitions dont matter! you need to know whats the right thing to do as per the situation & time and how much quality you can gain. Product has three angles (triangle) Time, Quality and Money, now take a situation/challenge and balance all three?
    Hope you woke up from your slumber!

    Go do something productive/value add for your team/company

    Good luck!

    ReplyDelete
  15. @ Anonymous,

    If I were you, I would not have dared to make a generalized foolish statement like: All of you have one thing in common, you guys have lot of free time, no great responsibility, or owner ship of anything, may be something trivial. Well, how can you be so sure about All of here? How can you be so sure that they don't have any great responsibility? Why should they believe you especially when you don't show anything to prove that YOU have done something exceptional in testing? Moreover, even you seem to lack the guts to disclose your name and rather preferred to remain Anonymous as you were reluctant to face the follow-up questions that might have been thrown at you! Personally, I respect negative feedback. But yours look like sentences from a frustrated soul. How can you expect others to take you for granted when you yourself are yet to show something that could show your skills and competence?

    How the !!! doest it matter what is called what? I absolutely agree with this point. If you had spent some time on this blog then probably you might have come to know my view on Confusing Terminologies in S/W testing. I understand you are just a onetime surfer who wanted to leave some trail on the blog without even caring to take a deeper look at the Archive!

    Product has three angles (triangle) Time, Quality and Money, now take a situation/challenge and balance all three? Do you *really* have the control to regulate the so-called 3 dimensions of a Project? If your answer is YES, I doubt you are NOT a Tester! Do you (as a tester) really have the control over things like Money and Time invested in a Project? BTW, I am just curious to hear your understanding of Quality! Could you enlighten me?

    Hope you woke up from your slumber! Hope is a good thing. keep up the hope.

    Go do something productive/value add for your team/company. How do you add value to your company? What is value addition BTW? I am eager to learn from you. Do enlighten me with your thoughts.

    At any rate, thanks for leaving behind your comment. I allowed it to be published to let my readers decide the merit of your comment. I hope I could think otherwise, but a comment full of spelling and grammatical errors that talk how someone else sucks, does not quite interest me! If you could really show me how to become a better tester, then I am eager to learn from you. Kindly let me and my readers learn from you. We would really appreciate the effort.

    Happy Testing...
    -Debasis

    ReplyDelete
  16. Just happen to walk in to this software testing zone by accident (well actually was looking for how to address hard to reproduce defects!!). I am impressed with all the posts and the comments. Very informative and a living community in here.
    Thanks to all the posters - Debasis and all :).

    ReplyDelete
  17. @ Meren,

    Welcome to Software Testing Zone. It is so amazing to know that sometimes accidents can yield exciting results. :) Great to know that you liked the posts here. Hope you keep coming back in future for more interesting posts on Software Testing.

    Happy Testing...
    -Debasis

    ReplyDelete
  18. Hi Debasis,
    Nice explanation. I still got a question: can sanity tests use test cases or automated test scripts?

    ReplyDelete
  19. @ Dário,

    Well, it depends how you differentiate a sanity test from a smoke test. If you go by my above explanation, Sanity Test is *usually* [not *always*] non-scripted. That means they are suite of quick tests that are done at the very last minute of a delivery schedule.

    However, my understanding of Smoke and Sanity Testing has undergone refinement over the period of time since I had written the above article. To know my current updated understanding of Smoke Vs. Sanity Vs. Build Verification Testing (BVT) Vs. Build Acceptance Testing (BAT) feel free to check out the new article!

    Hope this helps. Happy Testing...
    -Debasis

    ReplyDelete
  20. Hey,
    Nice way of describing the difference between the 2 Testing methods. One doubt; According to you, sanity testing is a subset of "regression testing". So what's regression testing and what testing methodologies are left while doing sanity testing?
    Thanks in advance

    ReplyDelete
  21. Regarding Sanity Testing, kindly take a look at my comment above. And to know more about Regression Testing, consider going through the following posts:

    > How Important is Regression Testing?

    > Regression Testing Revisited!

    > Repeatibility of Tests; in light of Regression Testing!

    Hope this helps. Happy Testing...

    ReplyDelete
  22. Hi,
    Nice article it helped me to clear my doubt about smoke testing and sanity testing.
    Thanks a lot.

    ReplyDelete
  23. hi Debasis,
    thanx for very good explanation about smoke and sanity testing...this is a common question asked in the interview and it helped me a lot to come out of the confusing definations.!!

    ReplyDelete
  24. Thank God and Debasis,
    Finally I found exactly what is Smoke testing and what is Sanity testing.....

    ReplyDelete
  25. This comment has been removed by the author.

    ReplyDelete
  26. excellent, u have cleared my doubt with this explanations
    http://soft-engineering.blogspot.com

    ReplyDelete
  27. Then what is the difference between sanity testing & re-testing. Re-testing is also a subset of regression testing. In re-testing, group of test cases related to the fixes are executed.

    ReplyDelete
  28. Key Debasis!! Very well written..Keep writting.

    ReplyDelete
  29. from the past 1and 1/2 year,i was confused with them,now clarified with this example............
    Thank u very much.

    ReplyDelete
  30. Amazing article, but as you said different people have different opinions.
    But still I believe you, actually you made me to believe you with such a good story.

    But still I have few queries:
    Q1- Important question: Testing levels are Unit, Integration, System Testing, Acceptance Testing
    at which level we do Smoke testing and at which level do sanity testing?

    Q2- Important question: I know that we check major functionality of built in both(Smoke & Sanity), can you give few examples of major functionality what we supposed to check if we are testing a job search website?

    Q3-How can we convince someone that we are (right because maximum people think differently)specially on a telephonic interview with some top guy of company?

    Q4-Sanity is called subset of regression testing because we test the build we got after smoke testing and because we are not testing the fresh built, so it is kind of regression testing, Am I correct?

    Please reply
    -pushprajsingh05@gmail.com

    ReplyDelete
  31. Hi Debasis,

    After going through your post, what I understood is:
    Smoke testing is a health checkup of the build to verify whether build is acceptable for further testing or not.And it can be scripted or non-scripted.
    Sanity testing, we do when a build has come with small changes and instead of doing regression testing we will perform sanity test.
    e.g.If tester has reported one defect and it has been fixed by developer so tester will retest it(This will be retesting) and he will test some related function to check whether defect has not affected any of other function, instead of doing regression testing(to test other function tester will perform sanity testing). Is this the example correct or I need to understand more on it.
    Sequence of these testing will be: smoke testing, functional testing, retesting, sanity testing and performance testing.Correct me if I am wrong somewhere.
    Thanks,
    Amit

    ReplyDelete
  32. Hi Debasis

    Appreciate your time and input in clearing the confusion between smoke and sanity testing.

    Mohammed

    ReplyDelete
  33. Hi Debasis

    Appreciate your time and input in clearing the confusion between smoke and sanity testing.

    Mohammed

    ReplyDelete
  34. Hi Debasis

    Appreciate your time and input in clearing the confusion between smoke and sanity testing.

    Mohammed

    ReplyDelete
  35. very nice explanation . thanks n keep publishing

    ReplyDelete
  36. Finally, a post that explains the difference clearly!

    ReplyDelete
  37. Really nice explanation

    ReplyDelete
  38. According to this two types of testing i,e ..
    smoke and sanity testing ,,smoke testing is done on first build whereas sanity testing is done on lateral builds.....

    ReplyDelete
  39. Very Good Explanation .....

    Thanks....
    Nayan

    ReplyDelete
  40. Hi,

    Thank you very much for you post. I was confused between these two. I just wanted to add something to this article which I read from Wiki that why Smoke Testing is called "SMOKE TESTING"


    "Smoke testing refers to physical tests made to closed systems of pipes to test for leaks. By metaphorical extension, the term is also used for the first test made after assembly or repairs to a system, to provide some assurance that the system under test will not catastrophically fail. After a smoke test proves that "the pipes will not leak, the keys seal properly, the circuit will not burn, or the software will not crash outright,"[citation needed] the system is ready for more stressful testing."

    ReplyDelete
  41. hi

    i m confused about bug/defect/fault/failure and error

    please anyone explain with any example

    thanx

    ReplyDelete
  42. Hi,

    Can any one tel me the difference between Sanity testing and System testing and when it can be user.

    Thanks in Advance

    ReplyDelete
  43. Hi Debasis

    Nice explanation with superb example and also hats-off answer to the Mr.Anonymous who is reakky a frustrated guy.

    Regards
    Aakash

    ReplyDelete
  44. Its too good. very easy to understand

    ReplyDelete
  45. It was amazing explanation.. thank u so much for this answer... really fantastic.

    ReplyDelete
  46. thanks for the post

    ReplyDelete
  47. If any one had/have confusion about Smoke Vs Sanity, they should refer to this page. Very well explained

    ReplyDelete
  48. finally i got it :)

    ReplyDelete
  49. Hi Debasis,

    Very clear and cool explanation on sanity and smoke testing...

    ReplyDelete
  50. Can anybody explain with example as what is considered major functionality during smoke test and what is in depth consideration in sanity with example??

    ReplyDelete

NOTE: Comments posted on Software Testing Tricks are moderated and will be approved only if they are on-topic. Please avoid comments with spammy URLs. Having trouble leaving comments? Contact Me!