How to harness generative AI to transform quality engineering

With a strategic and thoughtful approach to generative AI (GenAI) initiatives, organizations can enhance their software testing processes.

In brief

• GenAI can drive efficiency and productivity in quality engineering by automating software test design and execution, among other processes.
• To capitalize on this potential, companies must first resolve the issues GenAI presents, including concerns about bias and transparency.
• Taking a tailored, phased approach to GenAI adoption can help.

Generative AI, commonly referred to as GenAI, surpasses traditional artificial intelligence (AI) capabilities by empowering machines to not only comprehend information but also create, imagine and innovate. The latest EY Reimagining Industry Futures Study¹ confirms GenAI’s status as a breakthrough technology, with 43% of the 1,405 enterprises surveyed investing in it. This represents merely the tip of the iceberg, with boundless possibilities awaiting exploration. Specifically, GenAI technology brings immense value to software test lifecycle management in delivering high-quality software products, expediting test cycles, enhancing test coverage and boosting tester productivity.

Below, we explore the prospective future of software quality test automation, leveraging the transformative capabilities of GenAI technology. Based on early experiments, we outline the priorities of representative use cases. We also discuss the challenges and limitations and put forward actionable recommendations for enterprises, outlining immediate, midterm and long-term strategies to seamlessly integrate GenAI capabilities into their testing practices.

Early experiments and proof points

Following are some of the proof points from our early experimentations for global clients spanning varied sectors.

Challenges and limitations

The findings from the early experiments indicate that GenAI holds promise for automating certain testing tasks, improving test coverage, and enhancing the effectiveness and efficiency of the overall testing process. However, it is important to consider the following limitations and challenges associated with GenAI technology.

Not a panacea to address all problems

GenAI is not a one-size-fits-all solution for addressing all testing needs. While it can be suitable for certain testing requirements, there are situations where traditional testing approaches may be more appropriate and effective. Careful evaluation and a broad solution approach considering alternative testing methodologies and human involvement are necessary to provide holistic test lifecycle management.

Lack of transparency on the usage of GenAI hampers the assessment and resolution of concerns regarding output inaccuracy, data privacy and user consent. In regulated industries like financial services or health care, compliance with ethical guidelines and legal regulations is critical. Organizations should document and report on GenAI-powered testing approaches, including risk mitigation measures and testing methodologies.

Addressing bias in generated output

The application of GenAI technology can lead to inherent bias, discrimination and promote unsafe decisions, which may originate from the training data itself. Organizations must proactively establish ethical frameworks and guidelines to mitigate risks and have measures in place to validate the generated test outputs. The recent executive order by President Joe Biden² underscores the critical need for the development of standards, tools and tests to ensure the safety, security and trustworthiness of AI systems. This noteworthy step reflects the growing acknowledgment of the potential implications and risks associated with AI technologies.

Need for human oversight and validation

Testing often involves human expertise, domain knowledge and judgment to identify relevant test cases, prioritize testing areas and assess the significance of certain scenarios. GenAI, while helpful, may still lack the human intuition, domain context and critical thinking required for effective test planning, design, execution and reporting. Having a human in the loop to validate the output accuracy becomes inevitable.

Actionable roadmap

Organizations must adopt a mindset of thinking big, starting small, embracing failure as a learning opportunity, and scaling their GenAI-enabled testing initiatives to foster continuous innovation and growth. The figure below highlights an indicative roadmap demonstrating the three-phase approach for GenAI adoption in software testing. This roadmap must be tailored based on the organizational goals, adoption readiness and impact potential to quality processes.

Concluding thoughts

According to the Gartner Hype Cycle 2023 for AI,³ the GenAI movement encompasses two pivotal aspects driving the development of more powerful AI systems. On one side, there are the innovations that will be fueled by GenAI capabilities, leading to advancements across various domains including software testing. On the other side, there are the innovations that will fuel the progression and refinement of GenAI itself, enabling it to become even more capable and impactful. Apparently, the future of GenAI for testing holds tremendous potential for advancements in testing effectiveness, efficiency and coverage. Enhanced realism, adaptive generation, domain-specific models, collaboration with human testers, self-learning and self-healing through continuous feedback loop, and seamless integration with enterprise requirement and/or test management tools will shape the next generation of GenAI for testing. To stay relevant, organizations are advised to initiate quick win strategies and gradually increase investments in line with the evolving business needs.

Thank you to the following professionals for their contributions to this piece:

• Narayan Arunan Surya, Managing Director, Technology Consulting, Global Delivery Services
• Santhi Duraisamy, Managing Director, Technology Consulting, Global Delivery Services
• Akila Narayanan, Senior Manager, Other Consulting, Global Delivery Services
• Dhanesh Kumar, Senior Manager, Technology Consulting, Global Delivery Services