Proper handling of incompatible zedtokens #1723

josephschorr · 2024-01-29T21:28:23Z

NOTE: ZedTokens are a bit longer now as a result of this change, but should still be well within the 1024 limit previously defined

Fixes #1541

josephschorr · 2024-03-13T19:32:43Z

Rebased

josephschorr · 2024-04-01T13:17:53Z

Rebased

josephschorr · 2024-07-01T18:00:01Z

Updated

vroldanbet

Some early feedback, I'll continue tomorrow. Please describe in the PR body what problems are you trying to solve and design choices you took to come up with this solution. It helps folks reviewing the PR with the right context 🙏🏻

internal/datastore/crdb/stats.go

vroldanbet · 2024-07-09T18:45:40Z

pkg/datastore/datastore.go

+	// UniqueID returns a unique identifier for the datastore. This identifier
+	// must be stable across restarts of the datastore if the datastore is
+	// persistent.
+	UniqueID(context.Context) (string, error)


Did you consider alternatives using the UniqueID as a stable identifier for zedtokens?

I assume (not present in the PR body) that the goal is making sure zedtokens from one datastore type are not used in another datastore type. But what happens if you evolve the zedtoken implementation from one version to another for the same datastore, and want to force them to be dropped? the uniqueID wouldn't help, would it?
Shouldn't we store a zedtoken versioning parameter like datastoreType+zedtokenVersion?

Yes, but then it doesn't meet the goal which is to prevent zedtokens not just across types but instances of the datastore as well

But what happens if you evolve the zedtoken implementation from one version to another for the same datastore, and want to force them to be dropped? the uniqueID wouldn't help, would it?
Shouldn't we store a zedtoken versioning parameter like datastoreType+zedtokenVersion?

Can you address this ☝️?

josephschorr · 2024-07-09T20:41:01Z

Some early feedback, I'll continue tomorrow. Please describe in the PR body what problems are you trying to solve and design choices you took to come up with this solution. It helps folks reviewing the PR with the right context 🙏🏻

I added the fixes; it was on the commit but not the PR

vroldanbet

We are transferring UUIDs in every API call for no reason other than to prevent an eventual migration within the same datastore type. That's extremely wasteful.

It's still within the zedtoken spec limit, but it is a very high price to pay for an incredible rare event (a legit one, don't get me wrong). We are storing UUIDs in the datastore as strings, which is the least compact way of storing them and transferring them over the wire. And we are forcing everybody to see the data transferred increase significantly (and latency!) to prevent a scenario that the system will not be subjected to in, I'll dare to say, practically 100% of its lifespan.

We need to rethink this. I get the zedtoken is a stateless token and that it needs to self-contain this information, but I think we can get 99% there by defining the datastore type as a small attribute in the token. I understand migrating from different instances of the same datastore can be problematic, but no one asked for this, and we could find alternative ways for it (e.g. a flag that forces SpiceDB to ignore requested consistency and fallback to full_consistency all the time, or minimize_latency if the customer can take the hit of ignoring the new enemy problem during the migration). I'd argue that by adding such a flag we could completely avoid having to add the datastore type too.

vroldanbet · 2024-07-10T09:37:24Z

internal/middleware/consistency/consistency.go

@@ -55,27 +56,47 @@ func RevisionFromContext(ctx context.Context) (datastore.Revision, *v1.ZedToken,
 		handle := c.(*revisionHandle)
 		rev := handle.revision
 		if rev != nil {
-			return rev, zedtoken.MustNewFromRevision(rev), nil
+			ds := datastoremw.FromContext(ctx)


This middleware already had a dependency on the datastore middleware, but we never declared it via the middleware framework. Please update the middleware chain to state this middleware has such dependency.

Are you asking me to add a comment stating that or add something somewhere? (sorry for forgetting)

No. The middleware framework has API to define dependencies between middleware instances, which makes it fail in dev if the order is not respected.

internal/middleware/consistency/consistency.go

internal/services/v1/relationships.go

internal/services/v1/schema.go

internal/services/v1/watch.go

josephschorr · 2024-07-10T20:46:25Z

We are transferring UUIDs in every API call for no reason other than to prevent an eventual migration within the same datastore type. That's extremely wasteful.

Is it really? We could reduce the size of the ID to just a small prefix if we wish to save some bytes.

It's still within the zedtoken spec limit, but it is a very high price to pay for an incredible rare event (a legit one, don't get me wrong). We are storing UUIDs in the datastore as strings, which is the least compact way of storing them and transferring them over the wire. And we are forcing everybody to see the data transferred increase significantly (and latency!) to prevent a scenario that the system will not be subjected to in, I'll dare to say, practically 100% of its lifespan.

Except users have already been subject to it - this was spurred by two, different reports: one where a user moved between datastore types, but another where they moved between datastores of the same type (but accidentally)

We need to rethink this. I get the zedtoken is a stateless token and that it needs to self-contain this information, but I think we can get 99% there by defining the datastore type as a small attribute in the token. I understand migrating from different instances of the same datastore can be problematic, but no one asked for this,

It has happened before, sadly.

and we could find alternative ways for it (e.g. a flag that forces SpiceDB to ignore requested consistency and fallback to full_consistency all the time, or minimize_latency if the customer can take the hit of ignoring the new enemy problem during the migration). I'd argue that by adding such a flag we could completely avoid having to add the datastore type too.

This PR does add that flag, but we need a means of tracking to ensure that zedtokens are not used across instances of a datastore. The fact that we allow it now is, technically speaking, a small but real risk.

josephschorr · 2024-07-11T19:43:16Z

Updated to only store 8 bytes of prefix of the datastore ID. Since datastores are rarely updated, this should be fine for detecting changes.

josephschorr · 2024-07-30T20:59:30Z

Rebased

an older datastore is used All ZedTokens are now minted with the datastore's unique ID included in the ZedToken and that ID is checked when the ZedToken is decoded. In scenarios where the datastore ID does not match, either an error is raised (watch, at_exact_snapshot) or configurable behavior is used (at_least_as_fresh) Fixes authzed#1541

…token Results in smaller tokens but given that datastore IDs are generated, should still minimize the chances of a conflict

josephschorr · 2024-08-06T17:15:42Z

Rebased

github-actions bot added area/api v1 Affects the v1 API area/CLI Affects the command line area/datastore Affects the storage system area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools) labels Jan 29, 2024

josephschorr force-pushed the incompatible-zedtokens branch 3 times, most recently from d3b259f to 64dda15 Compare January 29, 2024 22:30

josephschorr marked this pull request as ready for review January 29, 2024 22:38

josephschorr requested review from vroldanbet and a team as code owners January 29, 2024 22:38

josephschorr force-pushed the incompatible-zedtokens branch from 64dda15 to 8f8f83c Compare January 30, 2024 00:32

josephschorr force-pushed the incompatible-zedtokens branch from 8f8f83c to 8c87502 Compare March 13, 2024 19:32

josephschorr force-pushed the incompatible-zedtokens branch 2 times, most recently from 52895fe to 9bb170d Compare March 13, 2024 21:43

josephschorr force-pushed the incompatible-zedtokens branch from 9bb170d to d15a06b Compare April 1, 2024 13:17

josephschorr force-pushed the incompatible-zedtokens branch 2 times, most recently from ba65181 to 160e514 Compare April 22, 2024 20:20

josephschorr force-pushed the incompatible-zedtokens branch from 160e514 to c176cb2 Compare July 1, 2024 17:59

josephschorr force-pushed the incompatible-zedtokens branch from c176cb2 to da44237 Compare July 1, 2024 18:29

vroldanbet reviewed Jul 9, 2024

View reviewed changes

vroldanbet requested changes Jul 10, 2024

View reviewed changes

josephschorr force-pushed the incompatible-zedtokens branch from da44237 to a526fc5 Compare July 11, 2024 19:42

josephschorr force-pushed the incompatible-zedtokens branch from a526fc5 to d19b1f7 Compare July 12, 2024 22:13

josephschorr force-pushed the incompatible-zedtokens branch from d19b1f7 to 35fbc6f Compare July 30, 2024 20:59

josephschorr added 3 commits August 6, 2024 13:11

Export the unique, stable, ID from datastore

3bbc373

Change to only store a prefix of the datastore's unique ID in the zed…

9b62b0d

…token Results in smaller tokens but given that datastore IDs are generated, should still minimize the chances of a conflict

josephschorr force-pushed the incompatible-zedtokens branch from 35fbc6f to 9b62b0d Compare August 6, 2024 17:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proper handling of incompatible zedtokens #1723

Proper handling of incompatible zedtokens #1723

josephschorr commented Jan 29, 2024 •

edited

Loading

josephschorr commented Mar 13, 2024

josephschorr commented Apr 1, 2024

josephschorr commented Jul 1, 2024

vroldanbet left a comment •

edited

Loading

vroldanbet Jul 9, 2024

josephschorr Jul 9, 2024 •

edited

Loading

vroldanbet Jul 31, 2024

josephschorr commented Jul 9, 2024

vroldanbet left a comment •

edited

Loading

vroldanbet Jul 10, 2024

josephschorr Jul 12, 2024

vroldanbet Jul 31, 2024

josephschorr commented Jul 10, 2024

josephschorr commented Jul 11, 2024

josephschorr commented Jul 30, 2024

josephschorr commented Aug 6, 2024

Proper handling of incompatible zedtokens #1723

Are you sure you want to change the base?

Proper handling of incompatible zedtokens #1723

Conversation

josephschorr commented Jan 29, 2024 • edited Loading

josephschorr commented Mar 13, 2024

josephschorr commented Apr 1, 2024

josephschorr commented Jul 1, 2024

vroldanbet left a comment • edited Loading

Choose a reason for hiding this comment

vroldanbet Jul 9, 2024

Choose a reason for hiding this comment

josephschorr Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

vroldanbet Jul 31, 2024

Choose a reason for hiding this comment

josephschorr commented Jul 9, 2024

vroldanbet left a comment • edited Loading

Choose a reason for hiding this comment

vroldanbet Jul 10, 2024

Choose a reason for hiding this comment

josephschorr Jul 12, 2024

Choose a reason for hiding this comment

vroldanbet Jul 31, 2024

Choose a reason for hiding this comment

josephschorr commented Jul 10, 2024

josephschorr commented Jul 11, 2024

josephschorr commented Jul 30, 2024

josephschorr commented Aug 6, 2024

josephschorr commented Jan 29, 2024 •

edited

Loading

vroldanbet left a comment •

edited

Loading

josephschorr Jul 9, 2024 •

edited

Loading

vroldanbet left a comment •

edited

Loading