Track and improve k6 memory usage and performance #1167
Comments
|
Do you have any idea why the RPS numbers are dropping from 21k to 15-16k over time? Btw, I just tested (with k6 0.25.1) if starting new repetitions involved any kind of large overhead. I compared a script doing a single http request per iteration with one doing a hundred: At least for k6 0.25.1, the difference doesn't seem to be big. Using the above script that loops through a hundred requests seems to result in 5-10% higher RPS numbers. |
|
No, I currently don't have a good idea why the iterations per second (and thus, the RPS) count dropped from 21k to 15-16k. The biggest drop was between 0.16.0 and 0.17.0, and I tried to spot something suspicious in the diff, but there are just too many changes to tell that way, so we'd have to do it commit by commit... 😞 I initially thought it might be some scheduling change, since it turns out that k6 v0.16.0 was the last k6 version that still used iterations per VU. In v0.17.0 the iterations were switched to be shared across all VUs. But I tried the new Regarding multiple requests per iteration, I also haven't tested that, I planned to explore it today. But I think |
|
Just to confirm @ragnarlonn's results, tweaking the script in various ways ( |
|
@mstoykov, I ran the same benchmark with an updated core-js library. I added it at the bottom of the table, as you can see, there's a ~600KB increase in memory usage per VU 😞 |
|
For posterity/comparison, here's an issue about a previous performance investigation: #540 |
|
Just to add more statistics: I did a test with 1K VUs for 30 minutes and it needed 30 Gb to run. Otherwise, I have " dial tcp XX.XXX.XXX.XX:80: socket: too many open files " |
|
@hassanbakerrts, can you give some more details about the script you ran? What was the size of the JS code you were executing (either in KB or lines of code)? Did you import some browserified library? Or open a big data file? I'm asking because 1k VUs taking 30GB of RAM means each VUs was ~30MB - very different from what we've observed with simple scripts. 😕 Also, which k6 version and OS did you use, and did you run the k6 docker container or a native k6 executable? Regarding |
|
@na-- @hassanbakerrts Could it be results storage that take up the extra space? At least my tests have been 30-second ones, but if you run it for 30 minutes I guess it will collect a lot more results data. |
|
@ragnarlonn, it could be, though usually that becomes a problem for longer running tests. Still, it mostly depends on the number of HTTP requests, so with 1k VUs and little sleep, it might be the cause... Here are the related issues: #763 and #1068 |
|
@na-- @ragnarlonn I imported lodash during setup but not when VUs were running. My VUs code was to ask for four GET requests. There is a sleep(random(1,4)) where random returns seconds for each VU. We had to run it on ECU (AWS) cluster, I think the OS there is linux but not sure which one (those things were handled by DevOps team). I used docker image to run k6. |
|
Ah, given that each k6 VU is its own separate runtime, just importing lodash would add significant memory overhead to all VUs, even if you only use in the |
|
@na-- That sounds like a perhaps interesting feature to add: being able to import & use things in the setup() context only, which means it will not affect memory usage during the test. Would it be hard to implement? (although perhaps it would be necessary to stagger the execution of the setup function for all the VUs, as otherwise peak memory usage would not change. It would just not be constantly high anymore like it is if lodash is present all the time in each VU context) |
|
@ragnarlonn, I though about this some yesterday, but there a big problem with having different imports/dependencies for As you probably recall, one of the first things Unless we can think of a clever and easy solution, I think we shouldn't spend a lot of time on this compared to the other memory-saving improvements we want to add to k6 - HDR histograms, reduction of core.js, goja improvements, shared read-only data files, streaming reading of big files, etc. The reason for that is that this seems like it won't deliver very big benefits, but would require substantial changes. After all, I'd argue that most people who |
|
@na-- Oh yeah, I didn't think about the distributed execution...And I can't think of a simple and user-friendly way to do this either. The only thing that comes to mind is adding some special import statement to init() that allows you to specify which imports you want available in other contexts (i.e. |
|
The problem with having custom import/exports, or custom syntax of any kind, is that would be running afoul of ECMAScript parsers (specifically in our case, Babel). We could do it with an exported method that returns a simple JS string array with extra files that need to be included in any archive bundle, or have a property in the exported |
|
Now that #1206 was merged in var http = require("k6/http");
var sleep = require("k6").sleep;
exports.default = function () {
http.get("https://httpbin.test.loadimpact.com/get");
sleep(Math.random() * 2 + 3);
}Running it with and So, roughly an 8-fold memory consumption improvement (from ~9.11GB to ~1.14GB), much less CPU usage and a lot faster initialization. The initialization time was actually the most noticeable difference, if you have RAM to spare. Running the same commands again, but with And for Almost immediate initialization of 3500 distinct JS runtimes is quite impressive 😄 But yeah, for the context of this particular issue, we should have CI tasks that monitor both the |
|
I decided to throw my test webpack packager for k6 for a test. Table:
All the number fluctuate but they generally stay in the same ballpark and the RPS is apparently not affected in this particular case. For me this confirms that the webpack setup I propose is now viable way to make k6 scripts much more performant when using |
|
I tested the latest goja with some es6 support with and without corejs. The tested script was pretty simple: With
The numbers move somewhat but are around those magnitudes. As the script always takes 1m05.2s (the sleep is apparently pretty unrandom ;) ), it looks like the no corejs version takes only 2 seconds to initialize while the others take 20 or 30 seconds respectively, which for 3500 VUs isn't bad, but isn't as instant. |
|
Hey all, an update on this old issue: we recently updated our "Running large tests" guide, and added benchmark results for v0.42.0 in our k6-benchmarks repository. We plan to automate benchmarking soon:tm:, so that we can continually track k6 performance for all releases. |
We currently don't have a good idea how changes in k6 affects its performance. In particular, we knew that core-js used a lot of memory (#1036 (comment)), but we never systematically tracked that, so we lack any baseline to compare against if we ever want to update our currently bundled version. I did some very unscientific measurements with
/usr/bin/timeon my laptop, and I've shared the formatted results below. I used a lot of current and previous k6 versions to run a script with 200 VUs executing a million iterations of adefaultfunction that just makes a simplehttp.get()operation against a localhost Go server. Here are the results:[1] Not actually very representative, because these tests were run against a localhost server on my laptop, but still useful as a measure to track the overhead that the rest of k6 introduces. As a comparison, I created this simple script that just hammers a URL, and I got out ~42000 requests per second on my machine out of it.
I'm currently willing to chalk up most of the difference between this simple script and k6 to the overhead of JS execution, since when I also replaced the JS VUs with a
lib.MiniRunnerVUso that each iteration didn't have any JS execution and just made a single request to the localhost server, I also got out ~40000 RPS out of k6.And to repeat, none of that is representative or matters much in the real world, since when I tested a server on the local network, both k6 and the simple Go script achieved the same result of ~7000 RPS, likely because then the test was IO and not CPU bound. Probably the only use for tracking that metric is to monitor if it suddenly drops, since that would likely mean we introduced something that worsened the performance significantly.
[2] This memory increase was caused by the addition of core-js in every VU.
[3] Again, this was caused by a core-js and babel update
[4] The huge drop was caused by #1038 🎉 but again, core-js was responsible
[5] just commented out these lines, but useful to know, because we plan to allow the possibility of users choosing not to use core-js: #1049 (comment)
So yeah, in summary, we should have some sort of a CI task that monitors the memory consumption and performance of k6, so that we can spot sudden changes in either. Whether that's on every PR or once daily or something else needs more discussion and investigation. And we probably should avoid CircleCI and do this either in Azure Pipelines or GitHub Actions, whichever turns out to be better (#959). For the performance part, if we track it through the iterations per second metric, we'd need #355, which is currently in progress.
The text was updated successfully, but these errors were encountered: