Allow ramping down of users

[max-rocket-internet]

Resolves: #1488

• Merge stop_user_instances() into stop_users()
• Allow stop_rate to be passed to stop_users()
• Negative or positive value for stop_rate does the same thing
• Copy same ramping behaviour from spawn_users(), it seems robust
• If no stop_rate passed to stop_users() then don't "ramp down", just stop all without sleeping
• Add one extra debug log message when running test is edited with new user_count/hatch_rate values
• Add test runner to ensure ramping down works

This is my first proper PR for this project so any and all feedback welcome 🙂

I have still yet to update any documentation...

[codecov]

Codecov Report

Merging #1502 into master will decrease coverage by 0.16%.
The diff coverage is 82.14%.

@@            Coverage Diff             @@
##           master    #1502      +/-   ##
==========================================
- Coverage   81.44%   81.28%   -0.17%     
==========================================
  Files          27       27              
  Lines        2388     2410      +22     
  Branches      367      372       +5     
==========================================
+ Hits         1945     1959      +14     
- Misses        352      359       +7     
- Partials       91       92       +1     

Impacted Files	Coverage Δ
locust/runners.py	81.67% <82.14%> (+0.85%)	⬆️
locust/clients.py	90.19% <0.00%> (-4.91%)	⬇️
locust/event.py	87.50% <0.00%> (-4.40%)	⬇️
locust/user/users.py	95.83% <0.00%> (-2.78%)	⬇️
locust/stats.py	88.96% <0.00%> (-0.06%)	⬇️
locust/user/task.py	96.21% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5b89d7...dc70aa0. Read the comment docs.

[cyberw]

This timeout will potentially prolong the iteration time for this while loop.

So if you had for example stop_rate 1 and stop_timeout 2, you would end up (if no users stopping within the timeout) taking 3 seconds to complete an iteration (giving you a stop rate of 1/3 users per second).

I dont think that is what our users would would expect.

Ideally these things should be done async (I guess you'll have to spawn "killer greenlets" or something)

[max-rocket-internet]

OK I see what you mean.

What is we just do stop_group = Group() outside the loop and then stop_group.kill(block=True) after loop is done?

[cyberw]

Sounds good! Can you add a test for it? I think it is one of those things that could trip us up :) (now or in the future)

[max-rocket-internet]

I can but what exactly is the test? Ramping down with stop_timeout set? How does the test fail? If ramping down doesn't happen? Or it happens too slowly?

If I understand correctly, if stop_timeout is set, we can't really guarantee any ramp down rate accurately, but could do best effort.

[cyberw]

If I understand correctly, if stop_timeout is set, we can't really guarantee any ramp down rate accurately, but could do best effort.

Well, in my worst-case example, we should still be stopping roughly 1 user per second, over time (except at the very last step where we last users will take around 1s extra)

For example, if we have 4 users and 2s timeout, it should take 6s to ramp down to zero if no users finish their iteration in time and ramp down is 1/s (and the iterations should be killed after 2 seconds and not sooner)

(the current implementation would take 13s i think)

Also, you can use a faster ramp down rate to make the test take less time than in my example :)

[max-rocket-internet]

@cyberw OK I've added a second test under TestStopTimeout that basically repeats the other test but with stop_timeout set. Please have a look and see what you think.

[cyberw]

@cyberw OK I've added a second test under TestStopTimeout that basically repeats the other test but with stop_timeout set. Please have a look and see what you think.

This test is too lenient:

self.assertTrue(user_count < 10, "User count has not decreased at all: %i" % user_count)

After 2.1 seconds of ramp down at 2/s and with one second delay I would expect to see 6 users running, any more than that is an error.

You can run higher ramp down rate to shorten the test.

[max-rocket-internet]

This test is too lenient

OK fair enough.

I did some testing and did see some variation, like +/- 1 when using whole numbers for sleeping. So I increased the hatch rate to 4 and tightened the sleep times also. This makes the tests run faster and it also very tight with higher hatch rate. If I make it tighter than this I see some failures just from variation.

[cyberw]

Looks great now. Only thing left is some documentation + an example.

[max-rocket-internet]

Looks great now

🎉

Only thing left is some documentation + an example.

Can you give me a hint here? I've looked through the docs and there's nothing really specifically mentioning hatch rate when decreasing users, decreasing users at all or really anything specific about hatch rate. And what kind of example would I show?

[cyberw]

I think for the web gui it is kind of supposed to be self explanatory, so maybe a note there about the hatch rate affecting both up and down ramping (tbh I'm not much of a web user myself).

Maybe there should be a an example similar to the unit test in /examples as well?

(edit: ignore what I previously wrote)

[max-rocket-internet]

I think for the web gui it is kind of supposed to be self explanatory, so maybe a note there about the hatch rate affecting both up and down ramping (tbh I'm not much of a web user myself).

OK true. I made some small changes to the wording around starting and editing a test.

Maybe there should be a an example similar to the unit test in /examples as well?

I don't think this is possible? The examples are all various types of locust files and you can't edit a running test in them. Or really do anything to show "ramping down" I don't think. I think this will be addressed in my other PR which we are also close to merging 😉

[cyberw]

Nice!