Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor ramping VUs executor #2124

Closed
wants to merge 6 commits into from
Closed

Refactor ramping VUs executor #2124

wants to merge 6 commits into from

Conversation

inancgumus
Copy link
Member

Refactored Ramping VUs executor—a little bit. This is the first step as I'm learning about the system. I hope this refactoring makes sense.

@codecov-commenter
Copy link

codecov-commenter commented Aug 29, 2021

Codecov Report

❗ No coverage uploaded for pull request base (master@e0d8d93). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head db88710 differs from pull request most recent head f953e94. Consider uploading reports for the commit f953e94 to get more accurate results
Impacted file tree graph

@@            Coverage Diff            @@
##             master    #2124   +/-   ##
=========================================
  Coverage          ?   72.69%           
=========================================
  Files             ?      182           
  Lines             ?    14581           
  Branches          ?        0           
=========================================
  Hits              ?    10599           
  Misses            ?     3338           
  Partials          ?      644           
Flag Coverage Δ
ubuntu 72.69% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0d8d93...f953e94. Read the comment docs.

@na-- na-- requested review from imiric, mstoykov and codebien August 30, 2021 07:52
@na-- na-- added this to the v0.35.0 milestone Aug 30, 2021
@na--
Copy link
Member

na-- commented Aug 30, 2021

Thanks, I just skimmed this and it seems like a solid improvement! That said, all of the new methods end up having a ton of parameters and results, which is not ideal for comprehension. What we ended up doing in another executor (the externally-controlled) one was to split up the Run() state into a separate struct and have all of these things as methods to that struct. Take a look at

// externallyControlledRunState is created and initialized by the Run() method
// of the externally controlled executor. It is used to track and modify various
// details of the execution, including handling of live config changes.
type externallyControlledRunState struct {
ctx context.Context
executor *ExternallyControlled
startMaxVUs int64 // the scaled number of initially configured MaxVUs
duration time.Duration // the total duration of the executor, could be 0 for infinite
activeVUsCount *int64 // the current number of active VUs, used only for the progress display
maxVUs *int64 // the current number of initialized VUs
vuHandles []*manualVUHandle // handles for manipulating and tracking all of the VUs
currentlyPaused bool // whether the executor is currently paused
runIteration func(context.Context, lib.ActiveVU) bool // a helper closure function that runs a single iteration
}

@inancgumus
Copy link
Member Author

inancgumus commented Aug 30, 2021

Thanks! Also, thanks for pointing out the externallyControlledRunState implementation :) It helped.

I have added a struct called rampingVUsRunState to share the state between the methods. I still don't think this refactoring is an ideal implementation, but I hope it paves the way for the next possible refactoring steps.

imiric
imiric previously approved these changes Sep 23, 2021
Copy link
Contributor

@imiric imiric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job breaking down Run() 👏 handleVUs() is clean and the *HandlerStrategy functions are a nice touch and encapsulate the behavior nicely.

Just a minor meta nitpick: I would use a more descriptive commit message and PR description explaining what was done and why instead of just "refactor" :) E.g. "Split up RampingVUs.Run method ..." and maybe a small summary of the changes.

@mstoykov
Copy link
Contributor

@inancgumus can you also provide some benchmark results from before and after the refactor

@inancgumus
Copy link
Member Author

@mstoykov

name                                                                                                      old time/op    new time/op    delta
RampingVUsGetRawExecutionSteps/seq:;segment:/normal-8                                                        221µs ± 0%     221µs ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:/rollercoaster-8                                                2.54ms ± 0%    2.40ms ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/normal-8                                                     221µs ± 0%     219µs ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/rollercoaster-8                                             2.44ms ± 0%    2.45ms ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/normal-8                       66.3µs ± 0%    66.8µs ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/rollercoaster-8                 739µs ± 0%     746µs ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/normal-8           22.9µs ± 0%    23.0µs ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/rollercoaster-8     241µs ± 0%     247µs ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/normal-8                                                89.0µs ± 0%    90.9µs ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/rollercoaster-8                                         1.02ms ± 0%    1.07ms ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/normal-8                                          87.9µs ± 0%    89.6µs ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/rollercoaster-8                                    911µs ± 0%     939µs ± 0%   ~     (p=1.000 n=1+1)

name                                                                                                      old alloc/op   new alloc/op   delta
RampingVUsGetRawExecutionSteps/seq:;segment:/normal-8                                                        246kB ± 0%     246kB ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:/rollercoaster-8                                                2.65MB ± 0%    2.65MB ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/normal-8                                                     246kB ± 0%     246kB ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/rollercoaster-8                                             2.65MB ± 0%    2.65MB ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/normal-8                       73.7kB ± 0%    73.7kB ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/rollercoaster-8                 795kB ± 0%     795kB ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/normal-8           24.6kB ± 0%    24.6kB ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/rollercoaster-8     270kB ± 0%     270kB ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/normal-8                                                98.3kB ± 0%    98.3kB ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/rollercoaster-8                                         1.06MB ± 0%    1.06MB ± 0%   ~     (p=1.000 n=1+1)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/normal-8                                          90.1kB ± 0%    90.1kB ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/rollercoaster-8                                    983kB ± 0%     983kB ± 0%   ~     (p=1.000 n=1+1)

name                                                                                                      old allocs/op  new allocs/op  delta
RampingVUsGetRawExecutionSteps/seq:;segment:/normal-8                                                         1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:/rollercoaster-8                                                  1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/normal-8                                                      1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/rollercoaster-8                                               1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/normal-8                         1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/rollercoaster-8                  1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/normal-8             1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/rollercoaster-8      1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/normal-8                                                  1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/rollercoaster-8                                           1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/normal-8                                            1.00 ± 0%      1.00 ± 0%   ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/rollercoaster-8                                     1.00 ± 0%      1.00 ± 0%   ~     (all equal)

@mstoykov
Copy link
Contributor

@inancgumus seems okay, but in general run with -count 5 or something as otherwise benchcmp will not show differences as it doesn't know of what statistical significance the difference is.

Also please rebase on master as you now have conflicts after the metrics registry merge ... sorry about that :(

@inancgumus
Copy link
Member Author

inancgumus commented Sep 28, 2021

Here are the benchmarks with count=10:

name                                                                                                      old time/op    new time/op    delta
RampingVUsGetRawExecutionSteps/seq:;segment:/normal-8                                                        223µs ± 5%     236µs ±16%    ~     (p=0.156 n=9+10)
RampingVUsGetRawExecutionSteps/seq:;segment:/rollercoaster-8                                                2.59ms ± 5%    2.58ms ± 2%    ~     (p=0.356 n=10+9)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/normal-8                                                     233µs ± 6%     226µs ±11%    ~     (p=0.165 n=10+10)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/rollercoaster-8                                             2.60ms ± 5%    2.57ms ± 8%    ~     (p=0.481 n=10+10)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/normal-8                       67.2µs ± 6%    65.0µs ± 1%  -3.20%  (p=0.015 n=10+10)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/rollercoaster-8                 738µs ± 5%     729µs ± 3%    ~     (p=0.280 n=10+10)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/normal-8           24.1µs ±10%    21.9µs ± 1%  -9.06%  (p=0.000 n=10+8)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/rollercoaster-8     241µs ± 2%     237µs ± 2%  -1.67%  (p=0.023 n=10+10)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/normal-8                                                87.2µs ± 3%    87.1µs ± 3%    ~     (p=0.393 n=10+10)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/rollercoaster-8                                          964µs ± 3%     963µs ± 2%    ~     (p=0.905 n=10+9)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/normal-8                                          85.2µs ± 1%    85.8µs ± 2%    ~     (p=0.278 n=9+10)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/rollercoaster-8                                    905µs ± 3%     896µs ± 0%    ~     (p=0.243 n=10+9)

name                                                                                                      old alloc/op   new alloc/op   delta
RampingVUsGetRawExecutionSteps/seq:;segment:/normal-8                                                        246kB ± 0%     246kB ± 0%    ~     (p=0.591 n=9+10)
RampingVUsGetRawExecutionSteps/seq:;segment:/rollercoaster-8                                                2.65MB ± 0%    2.65MB ± 0%    ~     (p=0.710 n=10+9)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/normal-8                                                     246kB ± 0%     246kB ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/rollercoaster-8                                             2.65MB ± 0%    2.65MB ± 0%    ~     (p=0.130 n=8+10)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/normal-8                       73.7kB ± 0%    73.7kB ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/rollercoaster-8                 795kB ± 0%     795kB ± 0%    ~     (p=0.058 n=7+9)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/normal-8           24.6kB ± 0%    24.6kB ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/rollercoaster-8     270kB ± 0%     270kB ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/normal-8                                                98.3kB ± 0%    98.3kB ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/rollercoaster-8                                         1.06MB ± 0%    1.06MB ± 0%    ~     (p=1.000 n=10+10)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/normal-8                                          90.1kB ± 0%    90.1kB ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/rollercoaster-8                                    983kB ± 0%     983kB ± 0%    ~     (p=0.124 n=10+10)

name                                                                                                      old allocs/op  new allocs/op  delta
RampingVUsGetRawExecutionSteps/seq:;segment:/normal-8                                                         1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:/rollercoaster-8                                                  1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/normal-8                                                      1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:0:1/rollercoaster-8                                               1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/normal-8                         1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/rollercoaster-8                  1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/normal-8             1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/rollercoaster-8      1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/normal-8                                                  1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/rollercoaster-8                                           1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/normal-8                                            1.00 ± 0%      1.00 ± 0%    ~     (all equal)
RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/rollercoaster-8                                     1.00 ± 0%      1.00 ± 0%    ~     (all equal)

Copy link
Contributor

@mstoykov mstoykov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM in general. I have some naming ... considerations and some code golfing (that isn't really necessary and might be worse for readability).

The biggest "request change" is actually adding a test for gracefulShutdown and rampingVUs given the comment that removing the support for it doesn't break tests :)

lib/executor/ramping_vus.go Outdated Show resolved Hide resolved
lib/executor/ramping_vus.go Outdated Show resolved Hide resolved
lib/executor/ramping_vus.go Outdated Show resolved Hide resolved
lib/executor/ramping_vus.go Show resolved Hide resolved
lib/executor/ramping_vus.go Outdated Show resolved Hide resolved
lib/executor/ramping_vus.go Outdated Show resolved Hide resolved
@inancgumus inancgumus closed this Sep 30, 2021
@inancgumus inancgumus self-assigned this Sep 30, 2021
@inancgumus inancgumus reopened this Sep 30, 2021
@inancgumus inancgumus closed this Sep 30, 2021
RampingVUs.Run method was complex and this refactors it by adding
rampingVUsRunState to share run state between functions used by
the Run method.

+ Makes the Run method easier to read and understand
+ Makes it explicit which Goroutines are being fired in the Run
+ Separates responsibilities to other parts like:
  + rampingVUsRunState and its methods.
  + Moves trackProgress Goroutine from the makeProgressFn
    to the Run method
  + Makes initializeVUs to only handle initializing GRs
  + Uses two strategy functions to make them reusable
    and clearer. handleVUs and handleRemainingVUs use them.
+ Makes the handleVUs algorithm easier to understand and
  manage.

`go get` was failing on Go tip.
@inancgumus inancgumus reopened this Sep 30, 2021
@inancgumus inancgumus mentioned this pull request Sep 30, 2021
3 tasks
@inancgumus inancgumus linked an issue Sep 30, 2021 that may be closed by this pull request
3 tasks
Copy link
Contributor

@imiric imiric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a minor suggestion.

I don't have a strong opinion on handleVUs(). The previous version looked OK, though the current one is more succinct, so 👍.

lib/executor/ramping_vus.go Outdated Show resolved Hide resolved
lib/executor/ramping_vus.go Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Polish: RampingVUs
7 participants