CSI: unique volume per allocation #10136

tgross · 2021-03-08T21:55:44Z

Adds a PerAlloc field to volume requests that directs the scheduler to test feasibility for volumes with a source ID that includes the allocation index suffix (ex. [0]), rather than the exact source ID. This suffix is also being added at the client when the volume claim RPCs are sent.

Reviewer notes:

Includes a very basic end-to-end smoke test, but I've tested this with multiple volumes as well. Once [Feature Request] CSI Spec Volume Creation #8212 is complete, I'll update the E2E test to create multiple volumes for EBS (which has controllers) so that the entire lifecycle is being exercised.
This PR currently depends on and includes 97bb912 from CSI: remove prefix matching from CSIVolumeByID and fix CLI prefix matching #10158. Once that's merged, d44d62e should be dropped from this PR. Done

shoenig

LGTM!

notnoop

had a couple of questions after skimming the PR. Will review with closer attention later today.

notnoop · 2021-03-18T17:50:12Z

client/allocrunner/csi_hook.go

@@ -92,8 +92,13 @@ func (c *csiHook) Postrun() error {
 			mode = structs.CSIVolumeClaimWrite
 		}

+		source := pair.request.Source
+		if pair.request.PerAlloc {
+			source = source + structs.AllocSuffix(c.alloc.Name)


I recall an issue where alloc indexes aren't unique, and we may run into two allocations sharing the same id in cases of canaries where multiple deployment versions are running. Would that cause an issue here?

It would! But it's one of our invariants that if you're running a job with PerAlloc you can't also use canaries. (It doesn't make sense as a concept with volumes.)

Let's make a note of that and/or add a test for that. I fear one day, we relax or change the requirement a bit and miss this assumption in the code.

Yup, got it right here: https://github.com/hashicorp/nomad/pull/10136/files#diff-897605edf4370ef2c1cc4c1cdc9ade324caef8450e156f513a2f2aaaa8610e1cR1129

I'll definitely add some more commentary to make that clear though.

Just realized I totally missed the docs too!

Done in 3b1d19c

notnoop · 2021-03-18T17:55:09Z

nomad/state/state_store.go

 	if obj == nil {
 		return nil, nil
 	}
+	vol, ok := obj.(*structs.CSIVolume)
+	if !ok {
+		return nil, fmt.Errorf("volume row conversion error")


This seems like a violated invariant, potentially a data corruption issue. Not sure what should happen here. It may be nice to log and include the obj type as well?

It looks like the rest of the state store code actually just tosses this conversion error out and lets us panic later. That's probably the right move.

Done in 4ae9cb8

Add a `PerAlloc` field to volume requests that directs the scheduler to test feasibility for volumes with a source ID that includes the allocation index suffix (ex. `[0]`), rather than the exact source ID.

Read the `PerAlloc` field when making the volume claim at the client to determine if the allocation index suffix (ex. `[0]`) should be added to the volume source ID.

github-actions · 2022-11-27T02:25:32Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

tgross added this to the 1.1.0 milestone Mar 8, 2021

tgross force-pushed the csi-unique-volumes-per-alloc branch from c257b9b to dd628be Compare March 8, 2021 22:08

vercel bot deployed to Preview – nomad-storybook-and-ui March 8, 2021 22:08 View deployment

vercel bot temporarily deployed to Preview – nomad March 8, 2021 22:08 Inactive

This was referenced Mar 9, 2021

scheduler/csi: fix early return when multiple volumes are requested #10143

Merged

CSI: single access_mode is not enforced within a job #10157

Closed

tgross force-pushed the csi-unique-volumes-per-alloc branch from dd628be to a90d367 Compare March 10, 2021 16:34

vercel bot deployed to Preview – nomad-storybook-and-ui March 10, 2021 16:34 View deployment

vercel bot temporarily deployed to Preview – nomad March 10, 2021 16:34 Inactive

tgross mentioned this pull request Mar 10, 2021

CSI: remove prefix matching from CSIVolumeByID and fix CLI prefix matching #10158

Merged

tgross force-pushed the csi-unique-volumes-per-alloc branch from a90d367 to 3b7e58e Compare March 10, 2021 20:23

vercel bot temporarily deployed to Preview – nomad March 10, 2021 20:23 Inactive

vercel bot deployed to Preview – nomad-storybook-and-ui March 10, 2021 20:23 View deployment

vercel bot temporarily deployed to Preview – nomad March 10, 2021 20:29 Inactive

vercel bot deployed to Preview – nomad-storybook-and-ui March 10, 2021 20:29 View deployment

tgross mentioned this pull request Mar 10, 2021

add "count" field to CSI volume register/creation #10159

Open

tgross force-pushed the csi-unique-volumes-per-alloc branch from 2d04b6c to 2d2118b Compare March 10, 2021 20:56

vercel bot deployed to Preview – nomad-storybook-and-ui March 10, 2021 20:56 View deployment

vercel bot temporarily deployed to Preview – nomad March 10, 2021 20:56 Inactive

tgross force-pushed the csi-unique-volumes-per-alloc branch from 2d2118b to d3820d6 Compare March 10, 2021 21:21

vercel bot deployed to Preview – nomad-storybook-and-ui March 10, 2021 21:21 View deployment

vercel bot temporarily deployed to Preview – nomad March 10, 2021 21:21 Inactive

tgross force-pushed the csi-unique-volumes-per-alloc branch from d3820d6 to cebcb23 Compare March 10, 2021 21:24

vercel bot deployed to Preview – nomad-storybook-and-ui March 10, 2021 21:24 View deployment

vercel bot temporarily deployed to Preview – nomad March 10, 2021 21:24 Inactive

tgross self-assigned this Mar 10, 2021

vercel bot temporarily deployed to Preview – nomad March 11, 2021 13:35 Inactive

vercel bot deployed to Preview – nomad-storybook-and-ui March 11, 2021 13:35 View deployment

tgross force-pushed the csi-unique-volumes-per-alloc branch from c80c665 to d77fd59 Compare March 11, 2021 15:11

vercel bot temporarily deployed to Preview – nomad March 11, 2021 15:11 Inactive

vercel bot deployed to Preview – nomad-storybook-and-ui March 11, 2021 15:11 View deployment

tgross force-pushed the csi-unique-volumes-per-alloc branch from d77fd59 to 3ff2917 Compare March 11, 2021 16:24

vercel bot temporarily deployed to Preview – nomad March 11, 2021 16:24 Inactive

vercel bot deployed to Preview – nomad-storybook-and-ui March 11, 2021 16:24 View deployment

tgross requested review from notnoop and shoenig March 11, 2021 16:27

tgross marked this pull request as ready for review March 11, 2021 16:28

vercel bot deployed to Preview – nomad-storybook-and-ui March 11, 2021 19:05 View deployment

vercel bot temporarily deployed to Preview – nomad March 11, 2021 19:05 Inactive

shoenig approved these changes Mar 12, 2021

View reviewed changes

tgross added theme/storage type/enhancement labels Mar 12, 2021

notnoop reviewed Mar 18, 2021

View reviewed changes

vercel bot deployed to Preview – nomad March 18, 2021 18:45 View deployment

vercel bot deployed to Preview – nomad-storybook-and-ui March 18, 2021 18:45 View deployment

tgross added 6 commits March 18, 2021 14:49

CSI: schedule unique volume per allocation

1fcfd5d

Add a `PerAlloc` field to volume requests that directs the scheduler to test feasibility for volumes with a source ID that includes the allocation index suffix (ex. `[0]`), rather than the exact source ID.

CSI: add per-alloc renaming at client

a552eb7

Read the `PerAlloc` field when making the volume claim at the client to determine if the allocation index suffix (ex. `[0]`) should be added to the volume source ID.

e2e: smoke test per-alloc option for CSI

588d889

changelog entry

e036a6c

just panic on corrupt state store

4ae9cb8

docs for unique volumes per alloc

3b1d19c

tgross force-pushed the csi-unique-volumes-per-alloc branch from ac9a9ed to 3b1d19c Compare March 18, 2021 18:49

vercel bot deployed to Preview – nomad-storybook-and-ui March 18, 2021 18:49 View deployment

vercel bot deployed to Preview – nomad March 18, 2021 18:51 View deployment

tgross merged commit 7c75696 into main Mar 18, 2021

tgross deleted the csi-unique-volumes-per-alloc branch March 18, 2021 19:35

tgross mentioned this pull request Mar 18, 2021

[feature request] Ability to use variable interpolation in volume {} stanza #7877

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CSI: unique volume per allocation #10136

CSI: unique volume per allocation #10136

tgross commented Mar 8, 2021 •

edited

Loading

shoenig left a comment

notnoop left a comment

notnoop Mar 18, 2021

tgross Mar 18, 2021

notnoop Mar 18, 2021

tgross Mar 18, 2021

tgross Mar 18, 2021

tgross Mar 18, 2021

tgross Mar 18, 2021

notnoop Mar 18, 2021

tgross Mar 18, 2021

tgross Mar 18, 2021

github-actions bot commented Nov 27, 2022

CSI: unique volume per allocation #10136

CSI: unique volume per allocation #10136

Conversation

tgross commented Mar 8, 2021 • edited Loading

shoenig left a comment

Choose a reason for hiding this comment

notnoop left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 27, 2022

tgross commented Mar 8, 2021 •

edited

Loading