Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix!: CreateFleet API changes and robust cleanup #35

Merged
merged 1 commit into from
Oct 27, 2023
Merged

fix!: CreateFleet API changes and robust cleanup #35

merged 1 commit into from
Oct 27, 2023

Conversation

jusiskin
Copy link
Contributor

What was the problem/requirement? (What/Why)

  1. The service's CreateFleet API had a breaking change which moved the fleet's min/max capacity fields in the request.
  2. The deadline_resources fixture did not robustly clean up the farm/queue/fleet/queue-fleet-association resources if it encountered an error while part-way through creating them

What was the solution? (How)

  1. Modify the code to conform to the new API schema
  2. Modify the deadline_resources fixture to have robust cleanup using contextlib.ExitStack

What is the impact of this change?

  1. The test fixtures can now create fleets successfully
  2. If an error is encountered when creating the resources, it will clean them up more robustly

How was this change tested?

Ran the test in the worker agent codebase

[2023-10-26 22:52:56,720] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] About to call API (Create farm test-scaffolding-farm)
[2023-10-26 22:53:12,444] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] API call succeeded (Create farm test-scaffolding-farm)
[2023-10-26 22:53:12,445] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] Created farm: farm-22b9b1ad36a74978a7cf808d0d66142b
[2023-10-26 22:53:12,445] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] About to call API (Create queue test-scaffolding-queue in farm farm-22b9b1ad36a74978a7cf808d0d66142b)
[2023-10-26 22:53:22,450] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] API call succeeded (Create queue test-scaffolding-queue in farm farm-22b9b1ad36a74978a7cf808d0d66142b)
[2023-10-26 22:53:22,451] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] Created queue: queue-fbf8aacfc7b04618aa949c46f60a75ee
[2023-10-26 22:53:22,451] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] About to call API (Create fleet test-scaffolding-fleet in farm farm-22b9b1ad36a74978a7cf808d0d66142b)
[2023-10-26 22:53:28,079] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] API call succeeded (Create fleet test-scaffolding-fleet in farm farm-22b9b1ad36a74978a7cf808d0d66142b)
[2023-10-26 22:53:28,080] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] Created fleet: fleet-9844aaa48538406bbdb2710c475d4393
[2023-10-26 22:53:28,080] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] Waiting for fleet fleet-9844aaa48538406bbdb2710c475d4393 to reach desired status ACTIVE
[2023-10-26 22:53:28,080] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] About to call API (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:53:28,268] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] API call succeeded (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:53:28,269] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] Retrying in 10s...
[2023-10-26 22:53:38,279] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] About to call API (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:53:38,396] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] API call succeeded (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:53:38,396] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] Retrying in 10s...
[2023-10-26 22:53:48,399] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] About to call API (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:53:48,525] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] API call succeeded (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:53:48,525] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] Retrying in 10s...
[2023-10-26 22:53:58,535] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] About to call API (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:53:58,644] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] API call succeeded (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:53:58,644] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] Retrying in 10s...
[2023-10-26 22:54:08,654] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] About to call API (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:08,772] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] API call succeeded (Get fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:08,773] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] About to call API (Create queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393 in farm farm-22b9b1ad36a74978a7cf808d0d66142b)
[2023-10-26 22:54:14,628] [test/integ/test_job_submissions.py::TestJobSubmission::test_success] API call succeeded (Create queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393 in farm farm-22b9b1ad36a74978a7cf808d0d66142b)

and later:

[2023-10-26 22:54:14,738] About to call API (Set queue-fleet association to STOPPING_SCHEDULING_AND_CANCELING_TASKS for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:14,909] API call succeeded (Set queue-fleet association to STOPPING_SCHEDULING_AND_CANCELING_TASKS for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:14,909] Waiting for queue-fleet association to reach desired status STOPPED
[2023-10-26 22:54:14,909] About to call API (Get queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:15,063] API call succeeded (Get queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:15,063] Retrying in 10s...
[2023-10-26 22:54:25,073] About to call API (Get queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:25,240] API call succeeded (Get queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:25,241] Retrying in 10s...
[2023-10-26 22:54:35,251] About to call API (Get queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:35,405] API call succeeded (Get queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:35,405] Retrying in 10s...
[2023-10-26 22:54:45,415] About to call API (Get queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:45,567] API call succeeded (Get queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:45,567] About to call API (Delete queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393 in farm farm-22b9b1ad36a74978a7cf808d0d66142b)
[2023-10-26 22:54:45,735] API call succeeded (Delete queue-fleet association for queue queue-fbf8aacfc7b04618aa949c46f60a75ee and fleet fleet-9844aaa48538406bbdb2710c475d4393 in farm farm-22b9b1ad36a74978a7cf808d0d66142b)
[2023-10-26 22:54:45,735] About to call API (Delete fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:46,232] API call succeeded (Delete fleet fleet-9844aaa48538406bbdb2710c475d4393)
[2023-10-26 22:54:46,232] About to call API (Delete queue queue-fbf8aacfc7b04618aa949c46f60a75ee)
[2023-10-26 22:54:46,893] API call succeeded (Delete queue queue-fbf8aacfc7b04618aa949c46f60a75ee)
[2023-10-26 22:54:46,894] About to call API (Delete farm farm-22b9b1ad36a74978a7cf808d0d66142b)
[2023-10-26 22:54:47,464] API call succeeded (Delete farm farm-22b9b1ad36a74978a7cf808d0d66142b)

Was this change documented?

No

Is this a breaking change?

Yes, the resources.Fleet API has changed and has a new required max_worker_count argument.

@jusiskin jusiskin added the bug Something isn't working label Oct 26, 2023
@jusiskin jusiskin requested a review from a team as a code owner October 26, 2023 23:17
ddneilson
ddneilson reviewed Oct 27, 2023
View reviewed changes
src/deadline_test_fixtures/fixtures.py
yield resource
resource.delete(client=deadline_client)

with ExitStack() as context_stack:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is slick

@jusiskin jusiskin merged commit 2ba8987 into mainline Oct 27, 2023
11 checks passed
@jusiskin jusiskin deleted the jusiskin/fix_createfleet_and_cleanup branch October 27, 2023 18:42
@jusiskin jusiskin mentioned this pull request Oct 27, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jericht jericht jericht approved these changes

@ddneilson ddneilson ddneilson approved these changes

Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jusiskin @ddneilson @jericht