State of Test Automation 2023: Keeping our E2E Test Suites Lean and Valuable

What we learned from the survey “State of Test Automation 2023”: Keeping our E2E Test Suites Lean and Valuable

The 2023 edition of Perforce’s “State of Test Automation” survey has just been released and gives us some interesting insights:

A few words about this “State of Test Automation” survey. It has been conducted annually for the last 4 years and targets mainly QA professionals in all areas of Digital.

We can see that the three main areas of concern are

These topics are very closely related: by focusing on the right tests, those that target anomalies in production, we can reduce the number of tests and decrease the efforts for their automation and maintenance. Today, the primary challenge in test automation lies in effectively addressing functional regression testing, particularly in areas like end-to-end (E2E) testing. The key lies in directing automation efforts toward the tests that truly make a difference, specifically in minimizing anomalies encountered during product usage.

This is precisely the objective of the usage-centric testing approach. To optimize end-to-end testing by analyzing actual product usage and prioritizing the most critical user journeys. By focusing on the key user interactions, this approach aims to enhance the effectiveness of testing. It also ensures thorough testing of the most significant aspects of the product.

But let’s get back to the important reasons behind the goal of prioritizing tests correctly.

Testing every aspect of a system thoroughly can be time-consuming and expensive. Prioritizing tests allows for the identification and testing of critical features or components that are more likely to contain defects or have a higher impact on the system’s functionality. By focusing on high-risk areas first, organizations can reduce the likelihood of critical issues escaping into production.

Test prioritization helps allocate limited testing resources effectively. By identifying and prioritizing tests based on their importance and potential impact, teams can ensure that the most crucial tests are executed within the available time and resource constraints. It prevents wasting effort on low-priority tests that may not provide significant value.

Prioritizing tests based on their potential impact helps identify defects earlier in the development process. By focusing on critical functionality or areas prone to defects, teams can uncover issues sooner, enabling quicker remediation. Early defect detection reduces the cost and effort associated with fixing bugs later in the development cycle.

Prioritizing tests allows for a more efficient testing process, enabling faster time to market. By concentrating on high-priority tests and critical functionality, teams can complete essential testing activities within the desired time frame, ensuring a quicker release cycle and enabling organizations to respond promptly to market demands.

Prioritizing tests based on customer needs and requirements ensures that critical functionalities and user scenarios are thoroughly tested. By focusing on customer-centric priorities, organizations can deliver a higher quality product that meets or exceeds customer expectations. This, in turn, enhances customer satisfaction and builds trust in the product and the organization.

Overall, test prioritization is crucial for optimizing testing efforts, minimizing risks, and ensuring the timely delivery of high-quality software products. It aligns testing activities with business goals, customer needs, and project constraints, resulting in more efficient testing processes and improved overall product quality.

Responses on automation priorities are indicative of current concerns, as E2E testing appears at the top, on par with performance testing.

The importance of the automation objective for E2E tests is not natural. These tests are very expensive to automate and long to run. This is where we find the key lesson from the results of this survey: to master the constraints of budget, limited resources, and reduced time to automate, it is necessary to properly target the tests that we automate. This is even more true for E2E tests, which play a very important role in preventing regression anomalies, but are expensive to create and maintain.

End-to-end regression testing typically involves executing many test cases to ensure that the software functions correctly after changes or updates. Automating this process reduces the manual effort required, leading to significant time and cost savings. Automated tests are faster and more frequent than manual tests, facilitating efficient release cycles and prompt feedback on software changes.

Manual regression testing can be prone to human errors and inconsistencies, leading to variations in test execution and results. Automated tests, on the other hand, follow predefined scripts or test cases precisely, ensuring consistent and accurate execution. This reduces the chances of missing critical test scenarios or introducing testing errors due to human oversight.

End-to-end regression testing typically covers various user journeys and functionalities of the software. Automating this process allows for extensive test coverage, ensuring that all critical aspects of the software are thoroughly tested. Furthermore, automation enables scalability, as automated tests can be easily repeated across different environments, configurations, and datasets, ensuring broader coverage and reducing the risk of overlooking potential issues.

Automating end-to-end regression testing provides faster feedback on the stability and quality of the software after changes or updates. Automated tests can be executed promptly and repeatedly, allowing for rapid detection of regression issues. Early problem identification allows prompt resolution by developers, minimizing user impact and reducing the time and effort needed for bug fixes.

By automating repetitive and time-consuming regression tests, resources can be allocated more effectively. Testers can focus on higher-value activities such as exploratory testing, analyzing results, and enhancing test strategies, rather than spending excessive time on repetitive manual testing tasks. This optimizes the utilization of testing resources and enables the team to deliver higher-quality software within the given time constraints.

Automation is a fundamental component of continuous integration and delivery (CI/CD) pipelines. Automating end-to-end regression tests allows for seamless integration of testing into the CI/CD process, enabling frequent and reliable software deployments. Automated tests can be integrated with build systems and triggered automatically. This ensures that each code change is thoroughly tested before being deployed to production.

In summary, automating end-to-end regression testing improves efficiency, consistency, accuracy, test coverage, scalability, feedback cycles, and resource utilization, and supports CI/CD practices. These benefits collectively contribute to higher software quality, faster time-to-market, and improved customer satisfaction.

Looking at the survey respondents’ objectives for improving testing, we observe a constant focus on automating tests (33%) and an increased emphasis on improving test coverage (18%). This objective is directly linked to prioritizing important tests and reducing the risk of anomalies that impact users and customers.

Good test coverage can be formulated using several indicators, such as requirements coverage, code coverage, interface coverage, and real usage coverage. Requirements coverage is the main objective of validation tests based on user stories. Code coverage and interface coverage are good objectives for unit and integration tests, respectively. But when it comes to functional regression tests, especially E2E tests, the coverage of the actual use of the product is the main concern.

By simulating actual product usage, end-to-end testing can replicate real-world scenarios and interactions that users may encounter. This allows for a more accurate assessment of how the software performs in different situations, helping to identify and address potential issues that may arise during actual usage.

When testing is aligned with the actual use of the product, it helps provide comprehensive coverage of the software’s functionalities. This ensures that all critical user journeys and workflows are thoroughly evaluated, reducing the risk of undiscovered bugs or defects that might impact users in production.

End-to-end testing that mirrors the actual product usage focuses on the end user. By considering the user’s perspective, the testing process can better validate whether the software meets the intended requirements and delivers a positive user experience. This approach helps identify usability issues, performance bottlenecks, and other factors impacting user satisfaction.

By aligning end-to-end testing with actual product usage, it becomes easier to identify anomalies or unexpected behavior during testing. This allows for early detection and resolution of potential issues before they impact users in real-world scenarios. It helps improve the overall quality and reliability of the software.

In summary, mirroring actual product usage in end-to-end software testing ensures comprehensive coverage, early issue identification, user-centric requirement validation, and improved software quality and user experience.

Facilitating the proper focus of E2E test suites by analyzing actual usage is fundamentally the contribution of the “Usage-centric Testing” (UcT) approach.

Knowing the coverage of the usage by the tests, allows us to eliminate useless tests and to recommend only the relevant tests concerning the real user paths of our application.

In usage-centric testing, the primary goal is to ensure that the software performs as expected in real-world usage scenarios, mimicking how actual users interact with the system. This approach, strongly supported by Gravity, aims to uncover issues that may arise when multiple features or components interact with each other, providing a comprehensive assessment of the software’s functionality, usability, and performance.

If you are interested in the UcT approach, here are two articles from our Blog that go into detail.

Usage-centric Testing part 1: E2E testing, a love/hate relationship

Usage-centric Testing part 2: a shift-right approach