Differences between tests run in cloud vs run environments

[gabrielqs]

Hi,

During my tests with k6 (v0.23.0, brew installed on MacOS), I have found a few differences which have confused me a bit. As I didn't see it mentioned anywhere in the docs, I'm opening this issue. BTW, great work on k6, it's an awesome tool. Differences are:

1. -c parameter works with k6 run and doesn't work with k6 cloud
   When I fire a k6 run -c config.json test.js, the configurations are read properly from the config.json file and added to the options variable. When using it via the cloud environment (k6 cloud -c config.json test.js). K6 fails with the message below:
   
   One thing to notice is that k6 cloud --help does show -c as a possible parameter.
2. options variable is available on the setup function when executing the test locally, but it isn't available when running it with k6 cloud. It would be great to have the options variable available there. One possible use case is: based on the number of VUs being used for a given test, create and authenticate X numbers with my application
3. Parameter --http-debug works when executing a test with k6 run, but produces no output when executing it with k6 cloud
4. When running a test using k6 run, I can see output from console.log calls coming from a test JS script. That's not the same case with k6 cloud, even though the parameter is listed under k6 cloud --help

I'm aware that there might be a performance reason behind these differences, but would like to point them out, as maybe updating the docs for these items while they are different can potentially help other users.

Thanks!
Gabriel

[na--]

Thanks for mentioning those differences and sorry for the delayed response!
I'll start backwards:

• console.log() (point 4) and --http-debug (point 3, although I assume you also meant it in point 4 about being present in k6 cloud --help ) are not shown when running tests with k6 cloud because in the cloud execution mode of k6, the tests are executed remotely on one or more managed virtual machines. We're planning on exposing those execution logs in the interface of the Load Impact web app soon, but it'd be very impractical to pipe them back to the k6 that started the test.
• (Point 2) Some the values for vus, duration, stages, etc. in the script options object in the cloud execution mode won't be properly populated. That's a valid bug and unfortunately an unintended consequence of the way we currently execute tests in the cloud, where they can be distributed between multiple machines and we control the different k6 instances through their REST API...
  Unfortunately this is unlikely to get fixed before we implement a fully native k6 distributed execution support. Until then, you can use environment variables as a workaround. If you run the following script with k6 cloud -e duration=30s -e vus=5 script.js:

  import http from "k6/http";
  import { sleep } from "k6";
  
  export let options = {
      vus: __ENV.vus,
      duration: __ENV.duration,
  };
  
  export function setup() {
      http.get(`https://test.loadimpact.com/?vus=${__ENV.vus}&duration=${__ENV.duration}`);
  }
  
  export default function (data) {
      http.get(`https://test.loadimpact.com/?vus=${__ENV.vus}&duration=${__ENV.duration}&current_vu=${__VU}`);
      sleep(2 + Math.random() * 3);
  }

  You will be able to both configure the total number of VUs in the script and read that number in the setup() function.

And finally, the bug you described in point 1 is also perfectly valid and we will leave this issue open until we fix it

[mstoykov]

@na-- haven't you fixed this with 704dcc3

[na--]

❗ Yeah, well spotted, it seems like I inadvertently fixed the bug from point 1 and forgot that it had its own issue 🎉 Closing this now! 🎊

What's more, the bug from point 2 would be coincidentally fixed by #1007! 3 and 4 are in the roadmap as well, though their implementation depends more on changes in the cloud execution backend, not k6...

[gabrielqs]

thanks!