K6 vs Gatling with Gatlytron: A Guide to Free Load Testing Tools in Action

In todays computing landscape with ever increasing complexity of systems, performance testing gets more important by the day. Many free and paid testing solutions are available, picking the right tool for the right job however can get confusing if you are not experienced in performance testing itself. In this post we will look at two popular free solutions, K6 and Gatling with Gatlytron.

General

K6 is a JavaScript based load testing tool, that is provided by Grafana and therefore also integrates into their landscape. Tests are executed by a small k6 executable that has to be installed.

Gatling is a Scala based solution, where you write your tests in either Java, Scala or Kotlin. You have a development project and the test can be executed with a build tool like maven.

Reporting

K6 reports a lot of raw metrics and a summary report at the end of the test. There is a built-in dashboard that can be enabled to view basic real time graphs locally, but cannot be accessed after a test has finished. There is no HTML report at the end of the test(without extensions) and any file output options are raw metrics. The collected metrics have to be sent to a timeseries solution like TimescaleDB, influxDB or such so that they can be usefully analyzed in a dashboarding tool like Grafana. The alternative to using extensions is to output the raw data to a CSV file, which then could be analyzed in Excel. The CSV output will contain about 1KB of data for every simple request, if you have a test with 100’000 HTTP-Requests you will end up with a 100MB csv file.

Following shows the CSV output for a single request, which is part of a group “MyTestCase“:

CSV output for a single request

In most use cases, the only metrics you are interested in are http_reqs(count), http_req_duration(response time) and http_req_failed(error rate). All other metrics would only be useful if you want to go into such detail, what you rarely do in load test results. Such analysis is normally done after you found there is a problem, and then you check in the Browser Dev Tools what exactly is going on. Therefore these metrics are considered an overhead.

Gatling aggregates and reports metrics in an HTML report at the end of a test, without the need for tools to store and visualize the data. Combined with Gatlytron, aggregated data can be stored to files or sent to a database, and that data can than be displayed in a visualization tool.

Following is a CSV output of data collected by Gatlytron:

CSV output of data collected by Gatlytron

Response Validation

K6 provides a simple function that can be used to execute checks. Checks are entirely separate from your requests and cannot influence if a request is considered failed or successful. The checks in K6 are basically counters that can be used together with thresholds to determine if a test is considered successful or failed.

Gatling provides a check() function that allows to validate http response codes, headers and body. This will influence if a request is considered successful or failing.

Separation of Failed Requests

In K6 there doesn’t seem to be a way to separate metrics of failed requests based on response validation.

Gatling reports metrics for successful and failed requests separately as OK, KO and ALL values. This is a crucial feature in load testing tools, as failing requests can severely distort your measured response times in both directions (lower times for aborted requests, higher times for running into timeouts).

Custom Metrics

K6 allows to create custom metrics using objects as follows:

The downside of how K6 does this is that you cannot declare your metric where you actually use it, what can make it quite cumbersome to declare and manage metrics, especially when you have a lot of them.

For example, the following cannot be done in K6:

If you try the above, you will wind up with one of the following errors:

Therefore, the above way is not really recommended. The better way of doing this for response time metrics is to use groups, as shown later in this post.

Gatling open source does not allow reporting of custom metrics out-of-the-box. Using Gatlytron you can report custom metrics as follows:

Naming of Requests

Naming of requests is useful to make it easier to understand what the script is doing, plus it can be used to get the requests sorted in the reports by the order they occur by using numbers as prefix.

K6 allows custom naming of requests using tags(default name is the URL), the syntax to do so is however rather cumbersome for such a simple action. Also the names will not show up in the summary report, they will only be reported to output destinations(files, time series databases …).

Gatling lets you name the request right when you create them:

Grouping of Metrics

K6 allows grouping of metrics using the group-method. This will group metrics in the summary output be reported for each group and sub groups. However, as it reports so many metrics for each group, this might make the report rather big and hard to read:

Gatling allows to easily create custom counts and response time metrics using groups and naming your requests:

Test Data Files

Test data files are used to make load tests more realistic and test different data constellations. In most cases, you will receive CSV files as database exports from the project you are testing. Both K6 and Gatling support using CSV and JSON Files. However, K6 does not support CSV natively and needs an additional Libary like Papa Parse to do so.

Following is an example how to read and use data from a CSV file in K6:

Gatling is here a bit simpler, having simple methods to iterate data in various ways and substitute string placeholders like “#{paramName}” with the data from the file:

Extracting Values and Using Them

K6 allows to read values directly from a response object, but also has options to read cookies from headers as follows:

Gatling allows to extract values from the response and save them to the session object. Then the values can be retrieved from the session object:

Gatling also has the possibility to propagate extracted values into a shared ProtocolBuilder:

Recording Browser Actions

To easily have a base you can start from, load testing tools often come with a way to record or generate scripts. Both K6 and Gatling provide tools to either record actions in the browser and turn them into scripts, or converting HAR files created with the developer tools into simulation.

Separation of Test Cases and Simulations

Test cases in load testing are separated into different files and should be able to be executed together to make simulate real world load as closely as possible.

K6 allows such separation of test cases and simulations. As in the following code, you can create files containing your scenarios, while having a main file that is used to define the simulation, as in following code:

Gatling has as well a concept of Scenarios and Simulations. These can be defined in the same class or in separate classes. For simplicity, here is a Simulation with a scenario:

Load Calculation

Calculating appropriate load to be executed against your system is one of the most important factors in load testing. The most common way to create a stable and controlled load pattern is to define a pacing time. This allows us to define how often a specific scenario should be executed per time unit. In our years of load testing, we most commonly just want to to be able to define the following 4 factors per scenario:

• Virtual Users: The number of virtual users to simulate

• Executions/hour: The total number of executions a scenario should be executed by all the VUs.

• Start Offset: The time to wait from the start of the test until the scenario starts.

• Users Per Ramp Up: Number of users that should be added per ramp up interval. (to control Parallelization)

  
  

What we don’t want to bother with is the Pacing Time and the RampUpInterval, which both can be calculated based on above factors.

K6 seems not to support the above concept totally, but you can define a scenario with number of VirtualUsers and Executions/hour, while not being in control of ramp up of the users:

Gatling supports the concept of pacing, put leaves the calculation of everything else to the tester. Here Gatlytron comes to the rescue, which provides simple methods to create a scenario based on above 4 factors:

Also what you often want and is rarely provided by the load testing tool, is to be able to just set a percentage of the defined scenario to run. This is also provided by Gatlytron:

Conclusions

From our viewpoint, Gatling is for most use cases the better solution, as it provides higher flexibility for scripting use cases and analyzing data.

• Pros for K6:

  • K6 is easily integrated when you have Grafana.

  • K6 reports detailed HTTP metrics for every stage of the request-response cycle.

  • K6 can have a rather small project size.

  • K6 can have a html report with a free extension (the report contains not much more info than the command line summary output)

  • K6 has lower onboarding time as learning it is simpler.

• Pros for Gatling:

  • Gatling is easier to use after initial training efforts and has a simpler code structure.

  • Gatling is more integrated and provides more features out of the box.

  • Gatling is more powerful and can be used for more use cases.

  • Gatling is less cumbersome to use when creating complex use cases.

  • Gatling provides easy ways to retrieve test data from various file formats.

  • Gatling aggregates metrics with a report to a file and does not need any data store to get easy to analyze test results.

  • Gatling separates the metrics of successful and failed requests.

  • Gatling(with Gatlytron) can store aggregated metrics to various databases and file formats. It stores relevant and aggregated metrics. These can then be displayed in various monitoring views like Grafana or EMP.

Recommendation

From our perspective, K6 can be used to test less critical systems that have low amount of complexity, where a simple solution is sufficient. This might be useful for simple API testing.

For most load testing use cases, where you want to have more control on what you measure and need to be more flexible of what you do, Gatling is the better choice over K6.

 Happy Performance Engineering!