-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ees 5660 replace versioned draft folder with static folder #5403
Ees 5660 replace versioned draft folder with static folder #5403
Conversation
5157854
to
e36dcbf
Compare
…ion of a data set prior to its publishing rather than including version in the folder name, as the version can change multiple times prior to publishing.
…ers to be "draft" rather than version-based names.
…cess the Public API fileshare
… if running integration tests, as otherwise they fire on every individual integration test method run.
4eb268f
to
fb0426b
Compare
"publicDataFileshareMountPath": "/data/public-api-data", | ||
"publicDataFileshareName": "[concat(parameters('subscription'), '-ees-papi-fs-data')]", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps a nitpick, but why have we contracted the term 'fileshare'?
Azure refers to it as 'file share' across the board so it doesn't really make sense to contract it into one word.
If it's not too much trouble, would suggest we re-case this as two words i.e. publicDataFileShareMountPath
and publicDataFileShareName
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a great idea, but I'd prefer not to muddy the waters of this PR with it. Instead, I'll raise a separate PR and find all occurrences that we can change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Created #5427 which targets this PR.
If you're happy with that PR and this one, I'll firstly merge that one into this one and then this one into dev.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, makes sense. However, would suggest that we merge this in first and then get the other one merged in separately to keep things cleaner
new Tuple<int, string>(1, "3"), | ||
new Tuple<int, string>(1, "4"), | ||
new Tuple<int, string>(2, "3"), | ||
new Tuple<int, string>(2, "4") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need to specify the type - can just be new(...)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done along with the change of tuple types ta.
new Tuple<int, string, char>(1, "3", '5'), | ||
new Tuple<int, string, char>(1, "3", '6'), | ||
new Tuple<int, string, char>(1, "4", '5'), | ||
new Tuple<int, string, char>(1, "4", '6'), | ||
new Tuple<int, string, char>(2, "3", '5'), | ||
new Tuple<int, string, char>(2, "3", '6'), | ||
new Tuple<int, string, char>(2, "4", '5'), | ||
new Tuple<int, string, char>(2, "4", '6'), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need to specify the type - can just be new(...)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done along with the change of tuple types ta.
@@ -319,5 +319,30 @@ public static IOrderedEnumerable<T> NaturalThenBy<T>( | |||
{ | |||
return source.ThenBy(keySelector, comparison.WithNaturalSort()); | |||
} | |||
|
|||
public static List<Tuple<T1, T2>> Cartesian<T1, T2>( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any particular reason we're using the Tuple
class over value tuples like List<(T1, T2)>
?
There's a bit of a distinction between the two, but it seems like value tuples are newer and seem to be recommended for use over the Tuple
class. Value tuples importantly are structs and allocate memory more efficiently.
Could switch to using value tuples?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a great idea! Done.
.ToList(); | ||
} | ||
|
||
public static List<Tuple<T1, T2, T3>> Cartesian<T1, T2, T3>( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As above - could use value tuples instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a great idea! Done.
// Assert that the original folder has moved to use the new special "draft" folder. | ||
var expectedDraftFolder = pathResolver.DirectoryPath(dataSetVersion); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Guess it's unlikely, but this test seems a little fragile as we're not actually asserting that it's a drafts
directory and are just assuming that the output from pathResolver.DirectoryPath
is correct.
A similar problem arises with the other test cases below as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this test was gonna hang around for a while I'd be inclined to make a change here, but given we'll be deleting this migration after it gets deployed to Dev and Test, I don't think it's worth the effort in changing now if you're happy with that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(also, I have tested manually!)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, yeh fine with me!
GetMigration().Apply(); | ||
|
||
// Assert that the existing draft folder is left untouched. | ||
Assert.True(Directory.Exists(draftFolder)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Guess it might be overkill - but we might want to assert that the directory with the version number doesn't exist as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd be inclined not to bother, as this pre-existing folder would need to be set up manually in the test setup by us. Given this test'll go as soon as this is deployed to Dev and Test, I'd be inclined to leave as is if that's OK.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, yeh fine with me!
public static bool IsPrivateStatus(this DataSetVersion dataSetVersion) | ||
=> IsPrivateStatus().Compile()(dataSetVersion); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh interesting - does this get around cases where you can't normally use a static method within a LINQ call or something similar?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, it's to allow code to be shareable between normal service code and within LINQ queries I believe (@jack-hive having created the original example for the IsPublicStatus()
method). While the Linq / EF bridge code might otherwise complain when given a DataSetVersion.IsPrivateStatus() method that it can't compile the expression down to SQL, this provides a way to do that, and additionally allows you to use this method as an extension method on the domain model so you can say dataSetVersion.IsPrivateStatus()
.
This way we're much more assured that the logic for determining public / private statuses in queries also matches the same logic as we use in normal code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, I'll bear this one in mind, might come in handy in the future!
…ces with named tuples.
UI test results
Overview
This PR:
This is the second implementation to fix this issue, the first being an attempt whereby the delete code would attempt to look for both a minor AND major version-named folder and delete either if they exist, which felt untidy.
More importantly, it introduced complexity to an area of the codebase that has already generated 2 bugs, and the crux of the issue seemed to be that it was difficult to rely on a folder name that is version-based during a time where a DataSetVersion's version can fluctuate between minor and major at any time.
The implementation
High-level design
To overcome this, the idea here was to move away from a version-based name entirely for a draft DataSetVersion's folder, and instead have a single folder named "draft" which will contain the working files for the current draft version of a DataSet. This "draft" folder would remain up until the version is finalised and can no longer be changed. In this case, it is when the DataSetVersion is published.
The lifecycle of a DataSetVersion and its folder would be:
v1.0
DataSetVersion.v1.0
draft DataSetVersion is imported in its entirety. The files go into adraft
folder within the DataSet's fileshare folder.draft
folder tov1.0.0
.draft
folder again.v1.1
andv2.0
depending on the mapping choices, but the fileshare folder will remain the same -draft
.draft
folder to the appropriate value that is based upon the final version of the new DataSetVersion, to eitherv1.1.0
orv2.0.0
.Implementation details
To facilitate the above flow, the following changes have been put in place:
DataSetVersionPathResolver
code has been updated to resolve to thedraft
folder path if the DataSetVersion in question is in one of the private statuses (the non-public ones), or the standard version-based folder path if the DataSetVersion is public.Public.Data.Services
project that holds theDataSetVersionPathResolver
.PublicData:BasePath
configuration from the ARM template.draft
folder name.Miscellaneous changes
ICustomMigration
has been moved to Common, to allow thePublic.Data.Api
project to leverage it as well as Admin.Public.Data.Api
startup, and both this and the Admin's custom migration code will now look up any ICustomMigration implementations that are registered in DI and run them on startup.// TODO EES-5660
to highlight areas of the code that can be removed once this code has been deployed once to each environment that currently has the Public API enabled.