Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

retest: Implement VPC peering for Cloud Cluster with private network #12918

Merged
merged 8 commits into from
Aug 23, 2023

Conversation

savex
Copy link
Contributor

@savex savex commented Aug 22, 2023

Create VPC peering for deployed cloud with private network

Steps:

  1. Get VPC of the ducktape client instance
  2. Detect cloud type and run corresponding workflow: FMC/BYOC

BYOC VPC Peering.

  1. Prepare VPC info using boto3/vpc
  2. Create peering in AWS or reuse existing one
  3. Wait for pending acceptance using AWS if created
  4. Accept it if created
  5. Create routes
  6. Ensure Active on AWS if created

FMC VPC Peering

  1. Prepare vpc info using CloudV2
  2. Create peering in CloudV2
  3. Wait for Pending Acceptance usign CloudV2
  4. Accept it
  5. Create routes
  6. Ensure Ready/Active state on both sides

Prior to implementing, in order not to have long list of arguments in each internal functions, use self.current as a data store. Each of the __payload functions and get will use it as a source of data to create proper body for REST request

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.2.x
  • v23.1.x
  • v22.3.x

Release Notes

  • none

 In order not to have long list of arguments in each internal
 functions, use self.current as a data store. Each of
 the _*_payload functions and _get_* will use it as a source
 of data to create proper body for REST request
    BYOC cloud is created in Operator's provider account. Due
    to this, CloudV2 API can't be used for Peering creation.
    I.e. we will not be able to put routes and acccess data. So,
    BYOC required separate workflow.

    Functions broken down to steps and each workflow uses
    common set of routines as well as own specific ones.
@savex savex requested a review from andrewhsu August 22, 2023 00:56
@savex savex self-assigned this Aug 22, 2023
@savex savex added the team/devprod display on zenhub workspace for devprod team label Aug 22, 2023
@savex savex added this to the v22.2.14 milestone Aug 22, 2023
@andrewhsu
Copy link
Member

checkpoint: all PR checks for a037609 are green

i'll kick off a simple /cdt check using the same packaged artifacts from the green PR checks

@andrewhsu
Copy link
Member

/cdt
tests/rptest/tests/services_self_test.py::SimpleSelfTest

@savex
Copy link
Contributor Author

savex commented Aug 22, 2023

/cdt
rp_version=build
tests/rptest/tests/services_self_test.py::SimpleSelfTest

Copy link
Member

@andrewhsu andrewhsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i encountered an error when running this with a given clusterId

bash$ ducktape --debug --globals=/home/ubuntu/redpanda/tests/globals.json --cluster=ducktape.cluster.json.JsonCluster --cluster-file=/home/ubuntu/redpanda/tests/cluster.json tests/rptest/tests/services_self_test.py::OpenBenchmarkSelfTest
...
[WARNING - 2023-08-22 22:11:02,514 - redpanda_cloud - create - lineno:426]: will not create cluster; already have cluster_id ci0motok30vsi89l501g
[INFO  - 2023-08-22 22:11:04,559 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.services_self_test.OpenBenchmarkSelfTest.test_default_omb_configuration.driver=SIMPLE_DRIVER.workload=SIMPLE_WORKLOAD: FAIL: KeyError('productId')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 133, in run
    self.setup_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 218, in setup_test
    self.test.setup()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/test.py", line 91, in setup
    self.setUp()
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_test.py", line 110, in setUp
    self.redpanda.start()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1490, in start
    cluster_id = self._cloud_cluster.create(superuser=superuser)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 429, in create
    self._update_live_cluster_info()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 403, in _update_live_cluster_info
    self.current.product_id = _c['productId']
KeyError: 'productId'

tests/rptest/services/redpanda_cloud.py Outdated Show resolved Hide resolved
@andrewhsu
Copy link
Member

checkpoint: a037609 also passed the cdt check: https://buildkite.com/redpanda/redpanda/builds/35448

  Includes fixes for findings when running against GCP provider
@savex savex force-pushed the cdt-cloudv2-7904-private-net-peering branch from a037609 to 4ab3b17 Compare August 22, 2023 22:44
@savex savex requested a review from andrewhsu August 22, 2023 22:48
Copy link
Member

@andrewhsu andrewhsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

i verified by running same test on my dev env and stuff works

@savex savex merged commit 1d71111 into dev Aug 23, 2023
48 checks passed
@savex savex deleted the cdt-cloudv2-7904-private-net-peering branch August 23, 2023 14:23
@vbotbuildovich
Copy link
Collaborator

/backport v23.2.x

@vbotbuildovich
Copy link
Collaborator

Failed to run cherry-pick command. I executed the commands below:

git checkout -b backport-pr-12918-v23.2.x-594 remotes/upstream/v23.2.x
git cherry-pick -x 84777cbf8ba503283b15b19e86e6689c2feabf5f b2ffff3e953d7d120024e8548bfe3b9a13d42bc2 20ea07bef3c28d2e768a3c91e995ea6e2bce364c 79580ad99b3a1a5c42abd8ea1a16278dc22c6a73 35bd9831918636e0bad35f04cf21fc425f28b49a 43ecacd1fb858cf5cd8163a4cd173830e5fcaf8a accbff0f6204a075eee02aa7518630967ab8a836 4ab3b176d1b56d60d27ae84cee24271b54629891

Workflow run logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team/devprod display on zenhub workspace for devprod team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants