An IBM report into the massive software breakdown at TSB in April reveals that testing failures were a key cause.
A lack of performance testing that provided "the required evidence of capacity" was one of the key causes of the breakdown in TSB's systems last April that led to half its online customers being cut off, according a report by IBM
that has been made public by UK lawmakers.
The report was produced by IBM on April 29th, just three days after it had been brought in by the TSB board to help triage the bank's problems and design a strategy for fixing them, and was disclosed this week in a Treasury Select Committee hearing.
"A combination of new applications, advanced use of micro-services, combined with the use of active-active data centres, have resulted in compounded risks in production," said the IBM report. Furthermore, the report said that TSB had failed in testing for the software migration. "Performance testing did not provide the required evidence of capacity and the lack of active-active test environments have materialised risk due to issues with global load balancing across data centres," it said. "IBM has not seen evidence of ... a rigorous set of go-live criteria."
IBM's team concluded that the underlying IT platform at TSB was less at fault than the custom and package applications it was attempting to roll out, in high volumes for continuous production.
It said that a previous core banking migration project that it had worked on at a similarly large bank had employed multiple trial migrations over a longer period than TSB had attempted.
TSB's problems started in mid-April when it attempting to shift five million customers and 1.3 billion onto the software platforms of its parent, Spain's Banco Sabadell. TSB's internet banking was operating at around 50 percent of capacity two weeks after the problems first emerged, meaning that only five out of every 10 customers trying to use TSB’s online service would succeed, the bank admitted. The previous system was operated by Lloyds Banking Group Plc, which sold TSB three years ago.
At parliamentary hearings last May, TSB's CEO Paul Pester repeatedly apologised to lawmakers and said he will relinquish part of his bonus after he bungled the roll-out. Pester, who’s run the lender since 2013, said he takes “absolute responsibility” for “unacceptable” customer service. However, Pester denied in those May hearings that the new platform was rolled out prematurely without adequate testing or that its Spanish parent company, which provided the system, put pressure on them to speed up the process.
Pester also came under fire from customers and politicians when he initially claimed the IT migration had gone “smoothly” despite the widespread disruption.