Add an option to auto-cancel a pending session after specified timeout #527

vesselofgod · 2022-01-28T01:35:56Z

resolves: lablup/backend.ai#337
Related to: lablup/backend.ai#343

Add an option to auto cancel a pending session after specified timeout

Get pending_timeout option value to the 'scheduler_opt' and if it's not zero, compare pending_timeout with duration in Pending
Get created_at value to the 'PendingSession' and calculate status duration in Pending

Add a pending_timeout option in scheduler_opt using trafaret

…rbidden type's session

…lidation using trafaret.

…HEDULED status

…te time that stay pending status

…t option's time

…ernelInfo for test

…rnelInfo for test

codecov · 2022-01-28T01:38:18Z

Codecov Report

Merging #527 (70b78f5) into main (c48a51b) will decrease coverage by 0.47%.
The diff coverage is 72.50%.

@@            Coverage Diff             @@
##             main     #527      +/-   ##
==========================================
- Coverage   49.28%   48.81%   -0.48%     
==========================================
  Files          55       55              
  Lines        9079     9342     +263     
==========================================
+ Hits         4475     4560      +85     
- Misses       4604     4782     +178

Impacted Files	Coverage Δ
src/ai/backend/manager/scheduler/fifo.py	`95.52% <ø> (-0.13%)`	⬇️
src/ai/backend/manager/scheduler/mof.py	`93.33% <ø> (-0.79%)`	⬇️
src/ai/backend/manager/scheduler/dispatcher.py	`26.73% <48.48%> (+1.99%)`	⬆️
src/ai/backend/manager/models/base.py	`55.03% <72.22%> (+1.22%)`	⬆️
src/ai/backend/manager/models/scaling_group.py	`68.23% <100.00%> (+1.98%)`	⬆️
src/ai/backend/manager/scheduler/drf.py	`92.53% <100.00%> (+0.11%)`	⬆️
src/ai/backend/manager/scheduler/predicates.py	`50.00% <100.00%> (ø)`
src/ai/backend/manager/scheduler/types.py	`78.72% <100.00%> (+2.34%)`	⬆️
src/ai/backend/manager/models/utils.py	`78.01% <0.00%> (-5.68%)`	⬇️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c48a51b...70b78f5. Read the comment docs.

achimnol · 2022-02-15T06:25:36Z

Please merge the main branch to resolve a conflict with addition of StructuredJSONBColumn in both #523 and here.

…ng-session-after-specified-timeout

achimnol

Could you write a test case by setting created_at to a time in the past?

vesselofgod · 2022-02-22T02:45:47Z

you write a test case by setting created_at to a time in

ok! I got it!

achimnol · 2022-02-28T08:21:17Z

you write a test case by setting created_at to a time in

ok! I got it!

What I meant is to create a new test case with a specific created_at time set in the past (maybe by subtracting from the current timestamp) and test actual status transition, not just changing the values of timestamps in the test fixtures...

vesselofgod · 2022-03-01T11:56:32Z

you write a test case by setting created_at to a time in

ok! I got it!

What I meant is to create a new test case with a specific created_at time set in the past (maybe by subtracting from the current timestamp) and test actual status transition, not just changing the values of timestamps in the test fixtures...

I'll add a new test case within this week :)

vesselofgod · 2022-03-06T14:19:31Z

I tried to add a test cases and commit, but error occurred....
I think it's a branch related problem, so I'll solve it and commit.

…ng-session-after-specified-timeout

…om the current timestamp

…ng-session-after-specified-timeout

* Temporarily remove "timeout" example session as it interfers with the DRF/LIFO scheduler test cases. We should use a separate example session list.

* timedelta.seconds is 0 to 86399 as it has a separate field for days. * We need to use total_seconds() to get the int representation or convert all operands to timedelta objects when comparing their absolute values.

* Scheduler configs are now part of it as the "config" field. * etcd's scheduler plugin config format is NOT changed. * Fix merging of global scheduler config and local scheduler config.

achimnol · 2022-03-30T11:14:26Z

Now need to perform an integratino test.

vesselofgod and others added 15 commits January 20, 2022 22:04

feat: Check allowed_session_type in scheduled_opts and don't start fo…

b8d5187

…rbidden type's session

feat: Add a column StructuredJSONBColumn to check scheduler_opts's va…

8cc6884

…lidation using trafaret.

fix: Fix code to pass the lint

6a3690c

fix: Fix base.py to pass the lint

17bb11e

fix: Fix base.py to pass the lint

6926f55

Merge branch 'main' into feature/limit-session-types-per-scaling-group

1de09f0

fix: Let StructuredJSONBColumn accept a trafaret object

ac55895

fix: check allowed_session_types per scaling_group before moved to SC…

b4aeaa9

…HEDULED status

feat: Add a pending_timeout option in scheduler_opt using trafaret

8dea4dd

fix: Add created_at value in PendingSession and KernelInfo to calcula…

ae2866a

…te time that stay pending status

feat: Move a session status PENDING to CANCELLED after pending_timeou…

812b95c

…t option's time

fix: Fix dispatcher.py to pass the lint

b983227

fix: Fix types.py to pass the lint

a2f7bc4

fix: Add created_at value in test_scheduler.py's PendingSession and K…

174bef6

…ernelInfo for test

fix: Add created_at value in test_scheduler.py's ExistingSession's Ke…

d6f2cb6

…rnelInfo for test

vesselofgod requested review from achimnol, kyujin-cho and inureyes January 28, 2022 01:36

docs: Add 527.feature file to write PR commit log

7cda340

vesselofgod changed the title ~~Feature/add an option to auto cancel a pending session after specified timeout~~ feat: add an option to auto cancel a pending session after specified timeout Jan 28, 2022

vesselofgod self-assigned this Jan 28, 2022

fix: Remove the outer extra parentheses, since they are not "pythonic"

de5f3e6

vesselofgod added 2 commits February 16, 2022 10:55

Merge branch 'main' into feature/add-an-option-to-auto-cancel-a-pendi…

818f389

…ng-session-after-specified-timeout

fix: Remove allowed_session_type check part in dispatcher

32429b6

ggstone0523 mentioned this pull request Feb 16, 2022

feat: Add create/update ui for scheduler_opts lablup/backend.ai-webui#1225

Merged

achimnol added the feature label Feb 18, 2022

achimnol suggested changes Feb 22, 2022

View reviewed changes

vesselofgod added 2 commits February 22, 2022 12:14

fix: Fix test case by setting created_at to a time in the past

eb7e7d9

fix: Fix created_at's type string to the datetime

89c083e

vesselofgod and others added 12 commits March 9, 2022 17:48

Merge branch 'main' into feature/add-an-option-to-auto-cancel-a-pendi…

b513822

…ng-session-after-specified-timeout

fix: Add new test case by setting created_at to a time subtracting fr…

9939329

…om the current timestamp

fix: Fix test_scheduler.py to pass the lint

375e5f2

fix: Fix 'TypeError' subtracting datetime.datetime to int

c8a886c

fix: Fix 'AttributeError' in the process of subtracting 'created_at'

bfff99c

Merge branch 'main' into feature/add-an-option-to-auto-cancel-a-pendi…

20e864a

…ng-session-after-specified-timeout

refactor: Better utilize existing scheduler ifaces and db txns

64f6b67

test: Update test codes and refactor out registry_ctx

9f37df2

* Temporarily remove "timeout" example session as it interfers with the DRF/LIFO scheduler test cases. We should use a separate example session list.

fix: lint error

3a82670

fix: Use timedelta correctly

fe876b3

* timedelta.seconds is 0 to 86399 as it has a separate field for days. * We need to use total_seconds() to get the int representation or convert all operands to timedelta objects when comparing their absolute values.

test: Add tests for pending timeout checks

058fe83

test: Remove unused code

a6e0d2e

achimnol added this to the 22.03 milestone Mar 30, 2022

achimnol added 5 commits March 30, 2022 09:49

fix(StructuredJSONColumn): When updating, just validate but not convert.

dfdca13

refactor: Make ScalingGroupOpts a typed attr class

a223fa6

* Scheduler configs are now part of it as the "config" field. * etcd's scheduler plugin config format is NOT changed. * Fix merging of global scheduler config and local scheduler config.

fix: Update populate_fixture() to handle structured json cols

97a0c63

test: Update another missing code sync

20e770e

test: arggggg

6edf9ba

achimnol changed the title ~~feat: add an option to auto cancel a pending session after specified timeout~~ Add an option to auto-cancel a pending session after specified timeout Mar 30, 2022

fix: Produce SessionCancelledEvent for pending-timeout sessions

70b78f5

achimnol merged commit c62ae65 into main Mar 30, 2022

achimnol deleted the feature/add-an-option-to-auto-cancel-a-pending-session-after-specified-timeout branch March 30, 2022 12:50

achimnol added a commit that referenced this pull request Apr 5, 2022

hotfix: A regression in #527

5e90c71

lizable mentioned this pull request May 9, 2022

Feature: num retries config in resource group lablup/backend.ai-webui#1305

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an option to auto-cancel a pending session after specified timeout #527

Add an option to auto-cancel a pending session after specified timeout #527

vesselofgod commented Jan 28, 2022

codecov bot commented Jan 28, 2022 •

edited

Loading

achimnol commented Feb 15, 2022

achimnol left a comment

vesselofgod commented Feb 22, 2022

achimnol commented Feb 28, 2022 •

edited

Loading

vesselofgod commented Mar 1, 2022

vesselofgod commented Mar 6, 2022 •

edited

Loading

achimnol commented Mar 30, 2022

Add an option to auto-cancel a pending session after specified timeout #527

Add an option to auto-cancel a pending session after specified timeout #527

Conversation

vesselofgod commented Jan 28, 2022

codecov bot commented Jan 28, 2022 • edited Loading

Codecov Report

achimnol commented Feb 15, 2022

achimnol left a comment

Choose a reason for hiding this comment

vesselofgod commented Feb 22, 2022

achimnol commented Feb 28, 2022 • edited Loading

vesselofgod commented Mar 1, 2022

vesselofgod commented Mar 6, 2022 • edited Loading

achimnol commented Mar 30, 2022

codecov bot commented Jan 28, 2022 •

edited

Loading

achimnol commented Feb 28, 2022 •

edited

Loading

vesselofgod commented Mar 6, 2022 •

edited

Loading