Yoram Mizrachi, CTO of Perfecto, is one of the select group of people who can truly claim to have shaped the technology and the business of software testing. While the softly-spoken Mizrachi is not one for bragging, the business he co-founded in 2006 is established as one of the global leaders in test automation.
Perfecto started as a business for testing mobile devices for telco customers. Then it branched out into testing mobile apps and caught the wave generated by huge growth in demand for mobile banking. Three years ago it signalled a broadening of its portfolio with a move into web browser testing.
Last year Perfecto was acquired by Perforce, the Minneapolis-based vendor of software development tools. And in July 2019, private equity fund Francisco Partners announced it was joining as an equal partner with Clearlake Capital as a backer of Perforce. The deal will allow Perforce to continue its buy and build strategy, said Mark Tiers, Perforce CEO.
Who better to ask about the future of testing – and Perfecto – than Yoram Mizrachi? We met him at Perfecto’s development offices in Israel.
Q: What difference has acquisition of Perfecto by Perforce made? Do you have more technology resources now, for example?
A: More or less the same, from the technology point of view. What we’re really enjoying about being part of a bigger company is the marketing effort – a raised level of awareness about Perfecto.
Perfecto over the years grew quite nicely, mainly with business from Fortune 500 companies. And we reached the top and everything was fine. But we do see the need to move down the pyramid to win business from small organisations.
We see massive opportunities in Europe which were not so much the focus up to now. But really we are looking at a global opportunity. Everyone wants to do mobile banking, and everybody wants to automate.
Q: So how will you be scaling up your platform over the next couple of years?
A: There’s going to be consolidation in the market. Customers are looking for those companies that can give them one suite of solutions. We are definitely going in that direction with Perforce, which is acquiring companies in the DevOps space. Tricentis is looking to do the same. Customers are looking for exactly that.
If you are a company going with a best-of-breed solution, it’s just a nice name. I was speaking with a colleague at [one of the largest banks in the world] and he told me that globally, in the DevOps space, the bank is dealing with 7,000 vendors. Seven thousand! It can’t be done! Each one of them might have a very good solution, but managing them is impossible. Managing the security audits, for example, is impossible. So it’s very natural that our marketplace will grow into big clusters.
Q: So let’s say a large bank wants to reduce the number of its DevOps vendors from several thousand to a hundred. That’s a vast reduction. How does it happen?
A: You have to think in terms of solutions, and not technologies. There are hundreds of testing companies, all of which will claim they have the best solution for this, or for that. But if the solution isn’t perfect but it gives me the value that I need, then that’s OK. When you buy a car, you’re not looking for the car with the best steering wheel, you’re looking for the best package.
So it might involve tactical acquisitions or something we develop in house. But this is the opportunity with the bigger Perforce.
We have competitors who are saying we are the very best at this specific thing. It might be true, but it doesn’t help. When you are building a continuous testing solution for the bank of the insurance company, then you have to look at the big picture. And the challenge is managing information and understanding what is going on, even if you are happy with your automation solution.
We have not touched the tip of the iceberg with continuous testing. And Perforce and Perfecto are trying to make it easy.
Q: What will be the secret of staying ahead of the competition in this fast-changing marketplace?
A: I am never happy with what we have. Companies sometimes fall in love with what they have created. We have changed dramatically since we started and we will look completely different again in 10 years’ time.
Let me give you one example. When we started, the big company that we had to take on in sales was DeviceAnywhere [now Keynote DeviceAnywhere]. At the time, there were no smartphones and the way they connected to the devices to test them was to open them and hard-wire them.
Our first major customer was a leading telco and we realised that this approach would no longer work for a client of that size, testing more and more devices – it would be too time-consuming. So our first solution involved pointing an external camera at the screen, which was very fancy from a technology point of view at the time, but very complicated.
Nowadays the issue of connectivity to devices is not such a huge problem. But customers are struggling with other things. Everyone understands that automation is a must-have. But how do you do automation at scale, with hundreds of thousands of executions every day, and stay on top of things?
So the technical challenges change over time. Stuff we were very proud of six or not seven years ago is now a commodity.
We always try to look one step ahead. The market struggled from getting devices, to getting good connectivity to devices. And where I think what we are struggling with today is a huge amount of information, and huge number of tests that need to be done. How do you understand the requirement? We have customers who are struggling with thousands of daily test executions, but they don’t really understand what is going on. The problem is not the ability to automate, it’s a problem of what I call test analytics.
I would say there are four main pillars that have to be built for continuous testing: creation, execution, the lab and finally the analytics . If you want to be successful to you have to be successful in each of these. The huge problem in the industry remains the mis-match between the skills-sets of organisations and the tools they are using.
We have customers who are trying to create Selenium or Appium script, for example, which requires a high skill-set. But what they have as a resource is people who basically used to be manual testers, or just starting their careers. And, guess what; they fail, which is nothing to do with the tool.
On the other hand, we see many companies which are playing on the creation side, and they are doing a good job of creating the tests, but the issue is how they can scale. The requirement five or seven years ago was to create a set of 500 or 1,000 scripts and run them on a monthly basis. With that workload it is easy enough to allocate two or three dayss to analyse the results manually.
But today, with continuous testing, the requirement may be more like 10,000 scripts a day. There is no time.
Q: So in terms of the platform and the product you offer, what are the missing parts of the jigsaw for Perfecto? What’s the agenda for the next couple of years?
A: I don’t know if one to two years is a long enough timeframe. But the testing market – the wider DevOps market – is going in to probably the biggest transition it’s seen in the past 40 years. We moved from manual testing to optimised automated testing and now we’re moving to continuous testing; not as a ‘nice to have’ but as an enabler for DevOps. This is what that marketplace is struggling with now.
So over the next two years our objective at Perfecto is to be able to give our customers one solution. A solution that still respects open-source – I am a great believer in that. If the marketplace decides that Selenium and Appium are the frameworks for doing testing then we will contribute to that – we are contributing to that. Customers will not keep you because they are tied to you.
Our objective at Perfecto and Perforce will be to provide a single platform that can take you through the journey of testing. This is where we see our success story.
I go back to those four pillars I was talking about. How do customers manage the creation gap? Some banks we work with believe in a BDD solution, which is one option. Then the next challenge is do you orchestrate execution so that when you arrive in the morning you can see that everything that you intended to execute has indeed been executed. How do you keep your lab up to date?. How do you makes sense out of all of this?
One advance we’ve made in the Perfecto product over the past year is a much more detailed analysis of the results.
We’ve looked at a few millions of executions by our customers and we found that the typical path fail ratios for a test execution doesn’t really help. Fail ratios don’t really mean anything. What customers need is visibility into the reasons for the failure and categorise the reasons for failure. Saying: “The test failed because we could not find an object on the screen” actually does not mean anything. But we can look back, sometimes using some AI, and say why the object was not found. A reason that makes sense; for example there is a problem with the pop-up or because the network was not connected: a reason that makes sense.
I’ll give you the example of a US customer who was seeing 700 failures a day simply because someone somewhere in the continuous integration process was connecting a single script to a device that was last connected four years ago.
Now, it’s OK to have legacy code if the testing environment can point you at what you need to do. Testing does not improve the quality of the application; you also need the visibility into the main problems that need to be fixed.
We have this technology today and we are extending it. As of today we have modelled several dozens of known failure reasons into the system. But the next step is to get the system to do this for us, to find the reasons that we do not yet know how to categorise.
It’s probably the biggest project that we have today. And we’re looking with the help of Perforce to make it bigger with some possible acquisitions, most likely in the area of more AI, more analytics.
One of the pitfalls of code analysis was always too much information. You execute the code analysis and you get a report of maybe 1,000 pages. It’s been one of the problems for our generation: how do you handle so much information? That’s especially true in the DevOps cycle, where decisions have to be taken very quickly; every day, by a small group of people. You need to be able to arrive in the morning and see immediately what needs to be done.
AI can help make smart decisions on a wider scale. It’s about enhancing your ability to learn; to learn by the system and not by the people.
And insurance companies are pioneers in this space
I’ll give you the example of a US customer who was seeing 700 failures a day simply because someone somewhere in the continuous integration process was connecting a single script to a device that was last connected four years ago.
Now, it’s OK to have legacy code if the testing environment can point you at what you need to do. Testing does not improve the quality of the application; you also need the visibility into the main problems that need to be fixed.
We have this technology today and we are extending it. As of today we have modelled several dozens of known failure reasons into the system. But the next step is to get the system to do this for us, to find the reasons that we do not yet know how to categorise.
It’s probably the biggest project that we have today. And we’re looking with the help of Perforce to make it bigger with some possible acquisitions, most likely in the area of more AI, more analytics.
One of the pitfalls of code analysis was always too much information. You execute the code analysis and you get a report of maybe 1,000 pages. It’s been one of the problems for our generation: how do you handle so much information? That’s especially true in the DevOps cycle, where decisions have to be taken very quickly; every day, by a small group of people. You need to be able to arrive in the morning and see immediately what needs to be done.
AI can help make smart decisions on a wider scale. It’s about enhancing your ability to learn; to learn by the system and not by the people.
And insurance companies are pioneers in this space – the type of solution is very obvious. If you can automate claims processing for your customer and improve your service. That’s a win-win.
Q: So how does AI work in the context of your product and platform?
A: I’ll give you an example. We learned from customers that one of the things that could cause a test to fail is because of something as simple as a pop-up that says “take a survey” appearing in a random place. So we’ve taken an AI engine which knows how to do image analysis – which is something quite ‘bread and butter’ in AI terms – and with data from our executions we’ve fed this programme with just under a million examples of what a pop-up looks like. Now we have the ability to tell the customer that the script failed because there is a pop-up in an unexpected place.
Now you can categorise. We have 5,000 failures and slice according to those that are caused by a pop-up, which is a simple example from an AI perspective. We can create an action. We have a generic pattern, that can be recognised like a face and we can feed this information to our customers so they can save time testing.
If you take this small example and broaden it out, then the power of AI is to take this information and to expose it in a simple way, very fast.
Q: And this could be especially true for financial firms that have big compliance requirements?
A: Yes. The way DevOps works is that you have to do everything. Software has to comply with regulatory standards several times a day. You need to do all the regulatory checks, all the time, on every application. You will not be able to decide to test some apps manually and automate others. The only important question is “how” and not “what”.
It doesn’t matter if you have the best testing tools if you have to tell your customer that they have to build the labs – a mobile testing farm, for example. And if you are executing tens of thousands of tests, you need test analytics, a system probably based on machine learning or AI which can give you an understanding of the data. It’s not a system of record. Test management is a system of record which enables you to go the regulator and say “I’ve executed this test”.
Q: So how far away are you from realising this AI-driven platform?
A: We have the components, the major issue is the adoption by our customers. For the past 20 years the major KPI for testers has been the number of lines of code tested. Developers don’t work like that. There is no longer an opportunity to review these results manually. So the evolution is towards a quality delivery. For Perfecto the main goal is to ease the transition for customers.
We have the scale for connectivity, but the biggest challenge is to work with the customer to make his journey to continuous testing easier. I don’t see the end of this transition. But I see the next steps.
Q: Perfecto was a mobile testing business, now you have a broader business. Is that transition understood by customers?
A: We are still making this move. Six years ago everyone thought everything would be moving to mobile. With customer-facing banking applications that happened, and now something like 75% of the traffic has moved to mobile. But Internal apps are still mainly focused on desktop browsers, because people are still coming into work in offices. The move to mobile hasn’t happened there yet.
So we realised two things. First, that the market for internal apps is looking for an easier solution. And on the other side we realised that the move to continuous testing is not something that is only relevant to B2C applications, which has moved much faster because those apps have a bigger impact on the business. Now all projects within the bank or the insurance company are moving that way.
Banks are our number one customer base [among financial firms] and insurance companies are number two. Banks are driving change because customers are demanding easier channels; and mobile is easiest. Insurers are looking at digital transformation largely because it allows them to make better offers to their customers: they can reduce the costs of policies because they are making much better decisions, because you know how many miles the car has been driven, and so on.
Q: A question about the marketplace you work in. What is your verdict on the biggest deal that’s happened: IBM’s $34 billion acquisition of Red Hat? And how do you see balance between the use of open source tools and proprietary tools evolving?
A: I think Red Hat was a great move for IBM. You cannot assume that you will have the best technology forever and IBM understands that. I think that open source has become the primary go-to market for adopting new technology, especially at the grass roots. We see that in our testing business. So in that sense it is a great deal. But then IBM has a 50/50 history in terms of making deals work.
More generally, I think “open source” can be dangerous term and we see that in testing. It does not mean “free”. Now I’m a technologist so I like the approach of open source: “Here’s everything, you can do what you like with it and you are not tied to it.” Cost-wise it does give you the ability to start small and then scale up before you have committed to the support and services you need to run it. That’s good.
But organisations that go the open source route need to keep in mind that they need to control their own destiny. They have to ask whether they really have open-source, or are they dependent on a proprietary vendor who has given you access to hundreds of millions of lines of code which you have nothing to do with. You have no chance of going into the code and fixing it if something goes wrong.
Q: What do you think is going to happen to the big systems integrators in this marketplace?
A: I think 2017 and 2018 were really hard years for them. This industry used to be tied to the number of people you had. You needed a hundred people to do your testing and you would get them anywhere you could find them. But companies have moved into DevOps and manual testing is no longer an option. There is still room for them, they are still doing massive deals. But they need to change: from quantity to quality.
We have good relations with the major SIs but no partnerships as such. For many years that was a tactical decision but not a strategic decision. The question now is: with a deeper relationship with this SI or another SI could we deliver a better blueprint for the customer to move from waterfall testing to continuous testing? We are definitely looking at deeper relationships.
Q: What’s the secret of hiring good people?
A: One thing which I am very proud of is that we are refreshing our technology all the time. We figured out that if the code is 10 years old then the chances are that the person who wrote it is no longer around and also that it is no longer at the cutting edge.
Three years ago we took our solution and we divided it into dozens of micro-services, each one of which is cutting edge. And we have technology companies coming to us all the time, looking to see how we do stuff. We are completely in the cloud for our own deployment. Everything is refreshed constantly. And now when we go to hire people, it is quite easy, because we are looking for the cutting edge at any given moment. People here stay for a long time because they can remain updated.
