The faster we get feedback, the better. This also applies to comprehensive automated tests. Given the trend towards low code solutions to automate testing at the UI level, we wanted to do some experimentation to compare when to implement some of the most popular options. Low code solutions are great because they have a less steep learning curve, but they are less useful for evaluating new code changes if feedback is delayed due to slow run tests.
In this blog post, we’ll share the results of one of our benchmarks to compare the speed of some of the most popular Low code solutions To automate the test.
We will explain our experience, provide all the details (in case you want to reproduce it) and summarize our analysis. If you just want to get acquainted with the results, go directly to the section Normative results: conclusions and observations.
Why is Rapid UI Test Automation Important?
As part of your testing strategy, you can automate test cases in order to have a set of checks that can provide feedback on the potential impact on the system of the most recent changes in the database. The usual scenario is that you just finished a change you were working on and you want to check if it could damage other parts of the system.
You take your tests while you refresh your mind, maybe take a short walk around your house, have coffee, or chat with your teammates. You want to have your safety check results in a few minutes, especially in CI/CD pipelines. If the feedback loop is too long, you can move on to something else and change the context. Getting back to what you were doing becomes more expensive, and requires more time and effort.
So, let’s say you set your safety checks to take no more than 5 minutes. The faster the automation tool runs tests and delivers results, the better because you will be able to add more test cases to your validation set within the same time limit and increase coverage in your application.
It’s also important to note that for safety checks, you may want to run your own unit tests, API tests, and some UI tests to get a comprehensive view of your application’s most important flows. Keep in mind that what usually happens is that you can run thousands of unit tests in seconds, hundreds of API tests in a minute, and some UI tests can take a few minutes.
Benchmarking the most popular low code solutions
Low code test automation platforms aim to simplify test automation with functionality that does not require the user to write any code. You will have a logger that will allow you to easily create and modify test cases with a simple interface without the need for programming skills. In the past few years, new low code solutions for test automation have been rolling in. To name a few: Functionize, GhostInspector, Katalon, Kobiton, Mabl, Reflect, Testim, TestProject. And the list is growing every day.
A few months ago, Testim released a new feature called Turbo Mode which enables you to run test cases in a faster mode. To increase speed, they run without saving all the stored data normally, which you won’t need in most cases, such as network, console logs, and screenshots of all steps in successful runs (they still store results when the test case reports an error). They claim that they can run between 25% and 30% faster than implementations in standard certification tests.
They also claim in the same article that they “Our tests compared to other low-code or no-code tools and completed implementation in less than half the time of the alternatives.“
We wanted to test their claims. We thought it could be a good experience to learn and share the results with the community, making a benchmark among the most popular low code solutions for test automation (defining most popular like that we They have known for a while.
Let’s see how the experiment was designed before discussing its results. We decided to compare the following low-code platforms:
- Certification (with or without turbo mode)
- test project
We also wanted to compare Katalon, because it’s a popular alternative as well, but it doesn’t offer native cloud support. The same thing happened with TestProject, but it was very easy to integrate with SauceLabs, so we decided to include it anyway.
Another aspect to mention is that Testim, Mabl and TestProject allow you to run locally, so we also compared local execution times. We focused on comparing the time it takes to run a test in the cloud because that’s the typical scenario where you’ll run your tests from your own CI engine. But since you often debug your tests locally, we’ve included a comparison of local execution speeds. To reduce variance across all tests, we ran them on the same laptop.
Laptop specification for local operation:
- Operating System: Windows 10 Home
- Model: ASUS VivoBook 15
- CPU: AMD Ryzen 7 3700U 2.3GHz
- Memory: 8 GB
- Connectivity: Wi-Fi and Viber, access from Uruguay
To keep the experiment as simple as possible, we decided to focus our time on comparing different tools but not on different test design strategies (such as whether or not it is better to use shorter test cases).
We built one test case with a typical length, as in many real projects. It is important to highlight that we designed the test to include different types of elements in the user interface to interact with (filling in entries, clicking on buttons, validating texts, etc.). Next, we automated the same test case in each instrument. To make the comparison fair and reduce the effect of outliers, we decided to test 10 times on each instrument.
To reduce variables, we always ran the tests with Google Chrome, although most tools offer to run the test in different browsers (but we didn’t compare browser performance this time).
We’ve looked at different beta sites to use as the system under test and decided to use Demoblaze (provided by Blazemeter as a training site for their performance testing tool). It’s not run by any of the companies we’ve been evaluating, it’s stable and fast (so there’s less risk that it affects the results of our analysis), it’s similar to a real e-commerce site and has just enough complexity to achieve our goal.
The test case is determined by following these steps:
- Go to Demoblaze
- Click on the title of the first item in the list (Samsung galaxy s6)
- Verify that the product title is “Samsung galaxy s6”
- Click the “Add to Cart” button
- Click OK on the pop-up that says Add Product
- Click “Product Store” to return to the home page
- Click the “Shopping Cart” menu at the top right
- Click the “Submit Request” button
- Fill in each of the x fields with an “x”
- Click the Buy button
- Validate the message “Thank you for your purchase!”
- Click the OK button
In the following screenshots, you can see how the test text is represented in some of the tools we compare.
Certification with a very graphical approach:
TestProject with a very simple script-like style:
So, in short:
- Comparison of Testim, Mabl, TestProject and Functionize.
- Single test case includes different types of actions (validations, fill in entries, click buttons). Site under test: Demoblaze.
- The local executions were always running on the same laptop (for all tools that provide local execution).
- All tests were performed using Google Chrome.
- We ran the test condition 10 times in each configuration to normalize the results.
Normative results: conclusions and observations
The following table and graphs summarize the results. We performed a statistical analysis to verify that the data sets were significantly different. If you want to learn more about why averages shouldn’t be compared clearly, read this article.
In our experience, the Testim turbo was (on average) 27% faster than the standard Testim implementation. This means that we cannot refute what they claimed in their article.
It is easy to analyze and compare the same information in a graph like the following:
Let’s compare implementations only at the local level:
Although the most significant outcome is the time it takes to run from the cloud, since we’ll be running our tests in our pipelines there, we also wanted to compare execution times locally, as this will affect the time to debug the test scripts.
Mapl was the fastest in local playback, with an average time of 22% faster than the second fastest tool.
To make it easier to compare different tools on the cloud, let’s just pay attention to the average implementations of the experiment:
Testim was the fastest to implement the cloud. On average, Testim was between 79% and 87% faster than other tools when it came to executing the same single test case. In other words, while you can run the test case with Mapl only once (the second fastest test in this experiment), you can run the same test almost 5 times with Testim. Also, Mapl was almost twice as fast as TestProject and Functionize.
We did not compare how response time might affect results. All the different tools have their own cloud or can be connected to a platform like Saucelabs or BrowserStack.
Some additional notes:
- It is very easy to learn how to automate with a low code tool, especially after you have already learned to use one. We were impressed by our ability to transfer our knowledge of one low code automation tool to another. For example, we were able to score and record a test case and execute it using Mapl without any additional tool-specific learning. Although we have experience with selenium, it may take at least a few hours to adjust the environment, set up the framework and run the tests. Not to mention configuring it to run in a network, getting nice looking test reports, getting the solution ready to be integrated into a CI/CD environment, etc.
- While it is true that someone with no programming skills can be very good at automating test cases with these tools, having coding experience helps improve test architecture, including driving good practices for module, version, etc. .
Thanks and appreciation
As a final note, I (Federico Toledo) could not have written this article without the help of Juan Pablo Rodriguez and Federico Martinez, of Abstracta. They prepared most of the texts and collected the results.
I hope you enjoyed this article and found it useful!