Strategies for selecting and prioritizing smoke test cases

Before deep diving into selecting and prioritizing smoke test cases, let’s introduce the concept of Smoke Testing. Smoke testing is a preliminary level of testing that is performed on an initial build of the software to ensure that the most critical functionalities work correctly.

Interestingly, the term “smoke testing” comes from the hardware testing practice of turning on a new piece of hardware for the first time and considering it a success if it does not catch fire or smoke. 

It is often referred to as “Build Verification Testing” or “Surface Level Testing” because it focuses on ensuring that the core functionalities of the software are working correctly before more in-depth testing is conducted.

The objective of smoke testing is to verify the “stability” of the system to proceed with more rigorous testing types. It is not meant to be exhaustive but rather to quickly determine whether an application is suitable for further, more detailed testing.

Smoke tests should be designed to be quick, typically taking a few minutes to an hour to complete. This allows for frequent execution without significantly slowing down the development process.

Here are some recommended times to perform smoke testing:

While smoke testing is commonly carried out manually, its effectiveness can be significantly enhanced by automating the process to shorten feedback loops. This is particularly important when smoke testing is integrated into continuous integration (CI) pipelines.

Here are the key points regarding the automation of smoke testing:

Unlike other testing approaches such as Regression testing or Functional testing, smoke testing typically does not require repeatedly running every test case.

Instead, it involves the strategic selection of test cases based on their risk and priority to quickly uncover critical issues.

As applications increase in complexity, there is a corresponding rise in the number of smoke tests that need to be executed.

This leads to extended execution times. Occasionally spanning hours or even days, which defeats the primary purpose of smoke testing to promptly identify critical issues.

This presents a unique challenge for testing teams: the necessity to strategically select and prioritize tests due to resource constraints and time limitations.

The team is then confronted with the critical task of identifying and focusing on high-priority smoke tests. They need to employ some type of risk-based testing strategies, and continually maintain and update the test suite to ensure relevance and effectiveness.

In the list below, you’ll discover various methods to prioritize smoke tests. All intended to strike a balance between comprehensive coverage and reducing execution time:

Gravity is a unified platform designed to help testing teams monitor and leverage insights from both production and testing environments. It improves the efficiency of smoke test prioritization.

Its primary function is to produce “Quality Intelligence” by processing the ingested data through machine learning algorithms and Generative AI.

This involves translating raw data into meaningful insights using techniques such as pattern recognition, trend and correlation analysis, anomaly and outlier detection, and more.

Gravity generates data-driven insights for the testing team by comparing real user interaction paths in live production with the tests conducted in testing environments.

These insights enable testing teams to spot gaps in coverage, identify features that are either over-tested or under-tested, and recognize redundant testing efforts in less critical areas. 

These insights assist in selecting and prioritizing smoke test cases, ensuring focused test coverage in critical areas, thereby enhancing the effectiveness of smoke testing without relying on assumptions or guesswork.