Aurélie Méheust: European sales director
In the race to speed up app delivery times, some retail banks and insurance companies are turning to crowd-sourced testing, which makes use of a geographically distributed network of freelancers to test customer-facing applications and websites. But while the crowd-sourced specialists report growing interest from the financial sector, there are understandable reservations about the privacy and security issues involved in using freelance testing networks. First, the advantages. The simple idea behind crowd-sourced testing is that hundreds of man-hours of testing can be generated for an app before it’s released, and that time can be concentrated into a few hours of real time. The other key advantage is that crowd-testing can be used to employ a geographically-focused group of testers to ensure that region-specific features are tested correctly. Aurélie Méheust is the European sales director of StarDust, a French crowd-sourced testing company founded in 2011 whose customers include banks BNP Paribas and Caisse d’Epargne, and insurance company AXA. “Some of our customers in banking are afraid of crowd-sourced testing because it’s not the traditional way of doing things, but that mentality is changing,” said Aurélie. The insurance sector is also showing some interest in crowd-testing, according to 99Tests, an Indian crowd-sourced testing company that employs from a community of 18,000 testers. “We’re currently working with Allianz and CF Insurance among others, and we see a lot of potential in the Australian insurance market,” said Praveen Singh, 99Tests’ co-founder and CEO. Most of the firm’s business is white-labelled through testing companies that have established relationships with financial firms. “They will already be doing their own testing, and then they’ll call on us to do another layer,” said Singh. The strength of crowd-sourced testing is its ability to bring together a large number of testers very quickly. The testers report any bugs found to a platform run the crowd-sourced testing company. That information can then be passed onto the customer, which can integrate the reports into their own test management system. Typically, testers are paid according to the number of bugs they find. Peter Blair, vice president of marketing at Applause, a US testing company that pioneered crowd-sourced testing, said that this gives large financial firms access to speed and scalability. “We can spin up hundreds of testers for a test cycle and run that in a matter of hours, rather than having ten people each working for ten hours if it were done internally,” said Blair. “This also means that companies can flex up and down. We have a financial firm as a customer that does a minor app release every two weeks and major release every six months. We let them triple or quadruple the testers for the release cycles according to their needs.” For Guillaume Gimbert, StarDust’s Head of Product Management, the geographic reach of crowd-sourced testing is a key advantage, especially for testing payments systems which will involve different intermediaries — and different transaction platforms — in different countries. “Something might work perfectly in France, but then break when you try and do it in Spain.” It is the drive among retail banks to grab market share with the best mobile apps that is fuelling interest in crowd-sourced testing, said Applause’s Peter Blair. “Mobile redesign is at the forefront of the agenda of banks, and they lean on us to make sure it’s a seamless experience,” said Blair. “When we test payments, for example, we give our testers some money and tell them to use it throughout a designated time period, buying appropriate items, often in stores. That way we can replicate the user’s experience, where there will be people standing behind him, waiting to pay, and maybe the signal is bad or the WiFi isn’t working well. We can do this across many different urban environments in a way that cannot be replicated in a lab.” The major limitation to crowd-sourced testing is confidentiality, and this is tackled in different ways by the test providers. StarDust focuses on the quality of their testers. “We have a very strict selection of the crowd-testers and our online tests have a 70% fail rate,” said StarDust’s Gimbert. Even then, not all clients can be reassured. “Sometimes financial firms are not ready to test their product in the wild because of privacy issues and they cannot be convinced that it’s safe.” Applause has a different approach. “A good amount of the work that we do with financial firms uses testers who already have some relationship with those firms,” said Peter Blair. “Usually they’re customers of the bank. We put up a notice in our community of testers that we’re looking for people who are clients of the bank. And we will ask for referrals from our community to find as many as are needed.” All those testers are then required to sign NDAs. “There is a misconception that crowd-sourced testing isn’t as secure and reliable as testing done in the lab,” said Blair. “But we’ve been able to reassure our customers in banking that it is safe. As a result we’re doing a lot of business in that area, and it is a sector that continues to grow.”
