Test automation tool vendor of the year: Sauce Labs
Partner project: Charles Schwab Intelligent Portfolios
The Schwab robo platform was established as a market leader. But app delivery required a major overhaul, with automation being the key objective. Schwab’s quality assurance tool of choice was Sauce Labs.
In volatile markets, investors expect reliability and speed from a robo-advisory platform. Under the hood, the robo’s algorithms may be smoothly re-allocating hundreds of millions of dollars of ETF assets. It may be delivering market returns for far lower fees. But if that efficiency is not reflected in the quality, speed and UX the client sees on their browser or mobile when they are executing or monitoring their investments, then the immutable law of razor thin margins means that market share will be lost.
Charles Schwab Intelligent Portfolios is the robo offering of the Charles Schwab brokerage and asset management business, which has more than $30 billion under management. It is one of Schwab’s most profitable and fast-growing businesses. It is also rated as top performing US robo among the market leaders in terms of generating investor returns.
But the challenge Schwab faced at the start the project in mid-2017 was that the Intelligent Portfolios business was encumbered by the high cost and low speeds of manual testing.
The objective set by Mike Schober, managing director for wealth management digital at Charles Schwab, was to transform the quality assurance process for the Intelligent Portfolios product team by adopting a “shift left” continuous testing approach. Above all, that entailed a move to validating quality with automation.
“The key objective was to transform the team from traditional models of development and QA to a paradigm of individual and team accountability for ‘quality development’,” explained Schober. “This required a cultural change as well as processes, tools and technology shifts.”
Sauce Labs, the cross-device cloud testing firm which, like Schwab, is based in San Francisco was a key partner in this shift. “As our team transitioned from traditional QA, a lot of the work and processes that model entailed needed to evolve or be suspended” said Schober. “We have since transitioned to Sauce Labs.”
“We are now able to better manage our staff resources related to environment configurations and expand the number of browser/OS/mobile combinations that can be tested. Now, the team can scale automated testing to be able to run automation more frequently and earlier in the SDLC.”
According to Schober, Schwab achieved all of the key metrics it set for the transition, and continues to see improvements. Critically, it has also been able to mitigate disruption in the change.
Scrolling back to the picture before the transformation began, functional testing in the Intelligent Portfolios team was almost exclusively manual, with all test design and processes managed by an outsourced QA team based overseas. There were between 1.5 and 2 quality assurance staff employed for each developer. Application updates were released on a monthly basis, following two sprint cycles.
Because testing was not done continuously throughout the development process, there were a significant number of bugs discovered at the end of each cycle. And because the development and QA teams were not co-located, many of the team’s conversations were held late at night, often getting bogged down in detail.
Mike Schober realised that sweeping changes needed to be made, and that developers should take ownership of quality. The major cultural change was embodied in the introduction of the SDET (Software Development Engineers and Test) role, in which automated validation became part of the development cycle, with tooling that embedded quality gates within the entire build process.
The relationship with the offshore provider ended and In October 2017, the Schwab developers immersed themselves in learning Ruby and Cucumber, and adding automated continuous testing with Sauce Labs.
There are clear metrics for the success of the transition. The average cost per team was reduced by 30% after the implementation of the new process centred on automation. The median velocity between Q4 2017 and Q1 2018 improved by as much as 20% with a 50% reduction in bugs, including far fewer bugs released into production. The quality of the automated test scripts continues to improve as developers are applying the same standards to test code as they do feature code, including peer reviews.
And what about the all-important customer experience? “It’s more about what they see sooner and what they won’t see,” said Schober. “In this new model, the teams will be capable of delivering smaller pieces of functionality at a more rapid pace as individual discrete features or assembling larger features. Improved quality is the benefit to the client experience.”