OOM killed when opening large files

[Breee]

Brief summary

k6 allocates alot of ram (7+ GB) when I try to open and read a big file.

k6 version

k6 v0.42.0 ((devel), go1.19.4, linux/amd64)

OS

Ubuntu 20.04.5 LTS

Docker version and image (if applicable)

No response

Steps to reproduce the problem

1. create file oom.js

import { sleep } from 'k6';

const file = open("bigfile.img", 'b');

export default function () {
  sleep(1);
}

2. create file bigfile.img: fallocate -l 1G bigfile.img
3. k6 run oom.js

Expected behaviour

No OOM kill

Actual behaviour

k6 gets OOM killed quickly

[mstoykov]

Thanks for reporting this @Breee!

Unfortunately this is one of those cases where this works as intended 😬 or at least it is how it has always worked and there isn't a none breaking change way to fix this.

open is basically wrongly named it is actually readfile - and as such instead of just opening it reads the whole contents of the file in memory.

If you do this with enough VUs or/and big enough files - this will always OOM.

While likely just adding read will "fix" this then we need ways to work with whatever read returns and AFAIK js does not have a standard around working with files we can just work from.

You can read the docs to see of ways where for files that are basically very big lists of items it is recommended that you use SharedArray around it.

Your case unfortunately falls in more closely to #1931, but as I explain in #2311 just having some representation of a file that isn't a memory hog is not enough .... as people generally want to do stuff with those files instead of just having them.

In this particular case this is just one more thing that isn't well implemented or not with big files in mind.

Basically 3 things with some good intentions combine to make what should on the surface be 1GB allocation to be 8+GB:

1. go is GC based language which for this case basically means that it more or less use twice as much memory as it needs - usually with the idea you will need this 2x much memory.2
2. k6 caches files read from the fs for many reasons, one of which is that this usually is pretty good idea ;). Another is that k6 can make an archive whose whole point is that it has everything it needs to run the test. It is a lot easier to do this if we keep everything we needed as well.
3. A dependency tries to be too smart when reading and decides that anything that is bigger than 100mb is too big and likely shouldn't preallocate the whole buffer.

Unfortunately 3 basically means that instead of it allocating 1GB for the buffer, filling that up and then making a copy that we will keep. It basically allocates a bunch of buffers each double the previous size which in total go to 3GB of allocations 🤦 .

You can see in this profile here.

We do the caching on fs.open which goes to the layer that actually just caches everything. And funnily enough it does it in a way that doesn't need 3GBs.

We then return go to actually read it (from memory already) and you can even clearly see where the code decides to not make the buffer big enough 🤦

I would like to say that this is an easy fix but I see no point in "fixing" this particular part as the one above will make this whole thing irrelevant once again. As even if you just managed to load this it will still not work once you have to have a bunch of VUs having a copy of it - even if it just takes 1GB each. I would argue that even if it takes 1GB for all of them it will still fall apart.

I don't have better workaround at this point than "write an extension for your particular workcase". I would like to tell you that we will be working on this soon, but I am doubtful that will have a finished solution soon enough. One of the big problems in practice is that the moment you want to actually use this "big file" everything that you can use it will need to support whatever abstraction we end up on. But you can go read #2311 for those same arguments.

I am leaving this open in order to be able to refer to it and so it is easier to find. But this likely will be closed with one of the issues linked.

[Breee]

Thanks a lot for the explanation!