At Nationwide, the world’s largest building society, the past two years have seen a major transformation of its software testing practices, with defects in production falling by up to 45%. Its new technical centre of excellence (CoE) has been key to its success, according to test engineering manager Rich Jordan, who will be a keynote speaker at the QA Financial Forum London.
Jordan said the CoE, which has been playfully named ‘DaveOps’, is delivering data, automation, virtualised environments, operations, performance and security capabilities for Nationwide’s project and development teams.
Speed and predictability has been critical to the success of the transformation, and that has enabled automated testing.
But according to Jordan, the change really only happened when the team looked deeper into why, despite automation, testing was not happening faster. “This is when we better understood approaches like behaviour-driven development and the need to create models within teams. It enabled us to understand how our systems worked and as a result, we used modelling tools as an asset to realise the solution,” Jordan said.
In turn, that focused Nationwide on quality, rather than simply increasing the number of software tests. The DaveOps group now has a flexible specification for this model-based approach, which allows the business, development and testing teams to better understand and deploy the system. As a result, a test cycle which could have taken days or weeks at the start of the transformation is now complete in a matter of “single digit minutes”, said Jordan.
“The time saved and the predictability of our approach changes our understanding of test quality,” explained Jordan. “We no longer understand test quality in terms of the volume of tests a team is able to run, but we look at the coverage and value that each test delivers.” For example, while the test CoE team ran 4.6 million tests during January 2020, the more important focus was the number of tests the team needed to run to reach a particular coverage outcome. “Many teams have the capability to run this volume but they don’t need to. It doesn’t provide us value and simply adds to the burden of analysis,” Jordan said.
The model-based approach has enabled Nationwide to eliminate between 25% and 45% of “defects in design” because collaborations are happening earlier in the software development life cycle, said Jordan. “There is a move from testing towards a more holistic quality view,” Jordan explained. “We are now just building models of the solution, rather than test models to articulate test design. Bei it from a designer’s perspective or a tester’s perspective, we are seeing quality as one overall thing to explore and improve.”
Jordan will be talking about Nationwide’s testing transformation at the QA Financial Forum London on March 12th.
