Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Suspend bench #446

Merged
merged 3 commits into from
Apr 22, 2024
Merged

Suspend bench #446

merged 3 commits into from
Apr 22, 2024

Conversation

justcoon
Copy link
Contributor

@justcoon justcoon commented Apr 21, 2024 •
edited
Loading

fixes: #384

./target/debug/benchmark_suspend_worker --length 3 spawned

result

Results for 'worker-7':
Duration:
  Avg: 10.036015666s
  Min: 10.031044s
  Max: 10.03895s
Results for 'worker-3':
Duration:
  Avg: 10.035762222s
  Min: 10.032683s
  Max: 10.037873s
Results for 'worker-1':
Duration:
  Avg: 10.036104666s
  Min: 10.031829s
  Max: 10.039348s
Results for 'worker-5':
Duration:
  Avg: 10.035954666s
  Min: 10.029747s
  Max: 10.046311s
Results for 'worker-0':
Duration:
  Avg: 10.035976888s
  Min: 10.031721s
  Max: 10.03879s
Results for 'worker-8':
Duration:
  Avg: 10.035595222s
  Min: 10.030308s
  Max: 10.038076s
Results for 'worker-9':
Duration:
  Avg: 10.036201333s
  Min: 10.032083s
  Max: 10.039173s
Results for 'worker-4':
Duration:
  Avg: 10.035727666s
  Min: 10.03134s
  Max: 10.038s
Results for 'worker-2':
Duration:
  Avg: 10.035879555s
  Min: 10.032827s
  Max: 10.038473s
Results for 'invocation':
Duration:
  Avg: 10.035944966s
  Min: 10.029747s
  Max: 10.046311s
Results for 'worker-6':
Duration:
  Avg: 10.036231777s
  Min: 10.031208s
  Max: 10.039941s

./target/debug/benchmark_suspend_worker --length 3 minikube --namespace bench

result

Results for 'worker-3':
Duration:
  Avg: 10.048397555s
  Min: 10.022889s
  Max: 10.110327s
Results for 'worker-7':
Duration:
  Avg: 10.039464666s
  Min: 10.02599s
  Max: 10.075385s
Results for 'worker-4':
Duration:
  Avg: 10.048867555s
  Min: 10.024189s
  Max: 10.121513s
Results for 'worker-1':
Duration:
  Avg: 10.045283777s
  Min: 10.025162s
  Max: 10.080743s
Results for 'worker-5':
Duration:
  Avg: 10.066041333s
  Min: 10.026313s
  Max: 10.110272s
Results for 'invocation':
Duration:
  Avg: 10.0513808s
  Min: 10.021377s
  Max: 10.131838s
Results for 'worker-0':
Duration:
  Avg: 10.044772888s
  Min: 10.026916s
  Max: 10.078585s
Results for 'worker-9':
Duration:
  Avg: 10.045343s
  Min: 10.025903s
  Max: 10.078391s
Results for 'worker-2':
Duration:
  Avg: 10.066876333s
  Min: 10.021377s
  Max: 10.131838s
Results for 'worker-6':
Duration:
  Avg: 10.054322888s
  Min: 10.027499s
  Max: 10.108621s
Results for 'worker-8':
Duration:
  Avg: 10.054438s
  Min: 10.028355s
  Max: 10.110242s

@justcoon
Copy link
Contributor Author

related to: #377
in case of minikube, second iteration was failing on shard-manager startup:

2024-04-21T17:48:02.936725Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=0}: golem_test_framework::dsl::benchmark: Finished iteration
2024-04-21T17:48:02.936791Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::dsl::benchmark: Starting iteration
2024-04-21T17:48:02.936820Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::rdb::k8s_postgres: Creating Postgres pod
2024-04-21T17:48:06.323053Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::rdb::k8s_postgres: Test Postgres started on private host golem-postgres.bench.svc.cluster.local:5432, accessible from localhost as 127.0.0.1:58610
2024-04-21T17:48:06.323160Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::redis::k8s: Creating Redis pod
2024-04-21T17:48:09.732923Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::redis::k8s: Redis pod started, waiting for healthcheck
2024-04-21T17:48:09.738725Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::redis::k8s: Redis started on private host golem-redis.bench.svc.cluster.local:6379, accessible from localhost as 127.0.0.1:58622
2024-04-21T17:48:09.738858Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::redis_monitor::spawned: Starting Redis monitor on port 58622
2024-04-21T17:48:09.740636Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::shard_manager::k8s: Starting Golem Shard Manager pod
thread 'main' panicked at golem-test-framework/src/components/mod.rs:122:17:
Failed to verify that golem-shard-manager is running

it is look like that shard-manager had issues to connect to redis

% kubectl -n bench logs pod/golem-shard-manager -f
2024-04-21T17:48:29.777094Z  INFO golem_shard_manager: Golem Shard Manager starting up...
2024-04-21T17:48:29.777328Z  INFO golem_shard_manager: Using Redis at redis://golem-redis.bench.svc.cluster.local:6379/0
2024-04-21T17:48:29.777382Z  INFO Server::run{addr=0.0.0.0:9021}: warp::server: listening on http://0.0.0.0:9021/
2024-04-21T17:48:29.777539Z  INFO golem_shard_manager: The port read from env is 9020
2024-04-21T17:48:29.777570Z  INFO golem_shard_manager: Listening on port 9020
thread 'main' panicked at golem-shard-manager/src/shard_management.rs:33:87:
called `Result::unwrap()` on an `Err` value: Unknown("IO Error: Custom { kind: Uncategorized, error: \"failed to lookup address information: Name or service not known\" }")
stack backtrace:
   0: rust_begin_unwind
             at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/panicking.rs:647:5
   1: core::panicking::panic_fmt
             at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/result.rs:1649:5
   3: golem_shard_manager::shard_management::ShardManagement::new::{{closure}}
   4: golem_shard_manager::async_main::{{closure}}
   5: tokio::runtime::runtime::Runtime::block_on
   6: golem_shard_manager::server_main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

after max wait time for service startup was changed from 30s to 90s it is working

2024-04-21T18:05:25.109751Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::redis::k8s: Creating Redis pod
2024-04-21T18:05:28.543404Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::redis::k8s: Redis pod started, waiting for healthcheck
2024-04-21T18:05:28.548128Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::redis::k8s: Redis started on private host golem-redis.bench.svc.cluster.local:6379, accessible from localhost as 127.0.0.1:60070
2024-04-21T18:05:28.548258Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::redis_monitor::spawned: Starting Redis monitor on port 60070
2024-04-21T18:05:28.549906Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::shard_manager::k8s: Starting Golem Shard Manager pod
2024-04-21T18:06:17.768949Z  INFO benchmark{name="suspend"}:benchmark{name="suspend" iteration=1}: golem_test_framework::components::shard_manager::k8s: Golem Shard Manager pod started

justcoon
justcoon commented Apr 21, 2024
View reviewed changes
golem-test-framework/src/components/mod.rs
@@ -118,10 +118,10 @@ async fn wait_for_startup_grpc(host: &str, grpc_port: u16, name: &str) {
if success {
break;
} else {
if start.elapsed().as_secs() > 30 {
if start.elapsed().as_secs() > 90 {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix for minikube issues (#377)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what was the "issue"?

afsalthaj
afsalthaj reviewed Apr 22, 2024
View reviewed changes
golem-test-framework/src/components/mod.rs
panic!("Failed to verify that {name} is running");
}
tokio::time::sleep(Duration::from_secs(1)).await;
tokio::time::sleep(Duration::from_secs(2)).await;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm.

@vigoo vigoo merged commit b22bb31 into main Apr 22, 2024
6 checks passed
@vigoo vigoo deleted the suspend_bench branch April 22, 2024 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@afsalthaj afsalthaj afsalthaj left review comments

@vigoo vigoo vigoo approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Suspend benchmark
3 participants
@justcoon @vigoo @afsalthaj