Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Option repeat don't care about setup/teardown #161

Closed
ligurio opened this issue Aug 16, 2021 · 0 comments · Fixed by #164
Closed

Option repeat don't care about setup/teardown #161

ligurio opened this issue Aug 16, 2021 · 0 comments · Fixed by #164
Assignees
@Steap2448

Comments

@ligurio
Copy link
Member

ligurio commented Aug 16, 2021 •
edited
Loading

I want to execute testcase for measuring performance for a number of times using option -r N and aggregate results from a test log using option -c. However luatest doesn't make a cleanup before next iteration and test faled. I propose to invoke setup (before_each) and teardown (after_each) when -r option is used.

https://github.com/tarantool/luatest/blob/master/luatest/runner.lua#L393-L402
ligurio added a commit to tarantool/crud that referenced this issue Aug 16, 2021
@ligurio
Update printing results
44e08bb 
For getting stable results we need to run performance tests many times
and then calculate mean, median and standard deviation for results.
luatest has an option `-r N` that allows to run testcase N times, but it
don't execute setup/teardown on each iteration [1] and due to this our
testcases fails due to duplicated id's in a space.
For running tests many times I made an external loop in a shell script:

```
for i in `seq 1 1 30`; do
luatest -v -c test/performance/select_perf_test.lua 2>&1 | grep -E 'INSERT|SELECT'`;
done
```

For more convenient extracting numbers from log I changed format of
printing on each iteration - all numbers that changed from run to run
printed in a single row, and removed printing numbers that are always
static.

1. tarantool/luatest#161
ligurio added a commit to tarantool/crud that referenced this issue Aug 16, 2021
@ligurio
Update printing results
731f927 
For getting stable results we need to run performance tests many times
and then calculate mean, median and standard deviation for results.
luatest has an option `-r N` that allows to run testcase N times, but it
don't execute setup/teardown on each iteration [1] and due to this our
testcases fails due to duplicated id's in a space.
For running tests many times I made an external loop in a shell script:

```
for i in `seq 1 1 30`; do
luatest -v -c test/performance/select_perf_test.lua 2>&1 | grep -E 'INSERT|SELECT'`;
done
```

For more convenient extracting numbers from log I changed format of
printing on each iteration - all numbers that changed from run to run
printed in a single row, and removed printing numbers that are always
static.

1. tarantool/luatest#161
ligurio added a commit to tarantool/crud that referenced this issue Sep 1, 2021
@ligurio
test: update printing results
14ae9bd 
For getting stable results we need to run performance tests many times
and then calculate mean, median and standard deviation for results.
luatest has an option `-r N` that allows to run testcase N times, but it
don't execute setup/teardown on each iteration [1] and due to this our
testcases fails due to duplicated id's in a space.
For running tests many times I made an external loop in a shell script:

```
for i in `seq 1 1 30`; do
luatest -v -c test/performance/select_perf_test.lua 2>&1 | grep -E 'INSERT|SELECT'`;
done
```

For more convenient extracting numbers from log I changed format of
printing on each iteration - all numbers that changed from run to run
printed in a single row, and removed printing numbers that are always
static.

1. tarantool/luatest#161
ligurio added a commit to tarantool/crud that referenced this issue Sep 2, 2021
@ligurio
test: update printing results
3987a4a 
For getting stable results we need to run performance tests many times
and then calculate mean, median and standard deviation for results.
luatest has an option `-r N` that allows to run testcase N times, but it
don't execute setup/teardown on each iteration [1] and due to this our
testcases fails due to duplicated id's in a space.
For running tests many times I made an external loop in a shell script:

```
for i in `seq 1 1 30`; do
luatest -v -c test/performance/select_perf_test.lua 2>&1 | grep -E 'INSERT|SELECT'`;
done
```

For more convenient extracting numbers from log I changed format of
printing on each iteration - all numbers that changed from run to run
printed in a single row, and removed printing numbers that are always
static.

1. tarantool/luatest#161
ligurio added a commit to tarantool/crud that referenced this issue Sep 3, 2021
@ligurio
test: update printing results
7e38ebf 
For getting stable results we need to run performance tests many times
and then calculate mean, median and standard deviation for results.
luatest has an option `-r N` that allows to run testcase N times, but it
don't execute setup/teardown on each iteration [1] and due to this our
testcases fails due to duplicated id's in a space.
For running tests many times I made an external loop in a shell script:

```
for i in `seq 1 1 30`; do
luatest -v -c test/performance/select_perf_test.lua 2>&1 | grep -E 'INSERT|SELECT'`;
done
```

For more convenient extracting numbers from log I changed format of
printing on each iteration - all numbers that changed from run to run
printed in a single row, and removed printing numbers that are always
static.

1. tarantool/luatest#161
ligurio added a commit to tarantool/crud that referenced this issue Sep 3, 2021
@ligurio
test: update printing results
6235428 
For getting stable results we need to run performance tests many times
and then calculate mean, median and standard deviation for results.
luatest has an option `-r N` that allows to run testcase N times, but it
don't execute setup/teardown on each iteration [1] and due to this our
testcases fails due to duplicated id's in a space.
For running tests many times I made an external loop in a shell script:

```
for i in `seq 1 1 30`; do
luatest -v -c test/performance/select_perf_test.lua 2>&1 | grep -E 'INSERT|SELECT'`;
done
```

For more convenient extracting numbers from log I changed format of
printing on each iteration - all numbers that changed from run to run
printed in a single row, and removed printing numbers that are always
static.

1. tarantool/luatest#161
@rosik rosik added the teamS label Sep 3, 2021
ligurio added a commit to tarantool/crud that referenced this issue Sep 3, 2021
@ligurio
test: update printing results
f0445ea 
For getting stable results we need to run performance tests many times
and then calculate mean, median and standard deviation for results.
luatest has an option `-r N` that allows to run testcase N times, but it
don't execute setup/teardown on each iteration [1] and due to this our
testcases fails due to duplicated id's in a space.
For running tests many times I made an external loop in a shell script:

```
for i in `seq 1 1 30`; do
luatest -v -c test/performance/select_perf_test.lua 2>&1 | grep -E 'INSERT|SELECT'`;
done
```

For more convenient extracting numbers from log I changed format of
printing on each iteration - all numbers that changed from run to run
printed in a single row, and removed printing numbers that are always
static.

1. tarantool/luatest#161
@Steap2448 Steap2448 mentioned this issue Sep 7, 2021
Merged
3 tasks
@Steap2448 Steap2448 self-assigned this Sep 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Steap2448 Steap2448

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Repeat _each / _test hooks tarantool/luatest
3 participants
@ligurio @rosik @Steap2448