Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AzDatalake] Lease Clients Implementation #21297

Merged
merged 61 commits into from
Aug 1, 2023
Merged
Show file tree
Hide file tree
Changes from 60 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
e8167a2
Enable gocritic during linting (#20715)
jhendrixMSFT Apr 28, 2023
86627ae
Cosmos DB: Enable merge support (#20716)
ealsur Apr 28, 2023
8ac8c6d
[azservicebus, azeventhubs] Stress test and logging improvement (#20710)
richardpark-msft May 1, 2023
9111616
update proxy version (#20712)
azure-sdk May 1, 2023
d6bf190
Return an error when you try to send a message that's too large. (#20…
richardpark-msft May 1, 2023
e2693bd
Changes in test that is failing in pipeline (#20693)
siminsavani-msft May 2, 2023
03f0ac3
[azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20…
richardpark-msft May 2, 2023
838842d
[azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)
richardpark-msft May 3, 2023
20b4dd8
Update changelog with latest features (#20730)
jhendrixMSFT May 3, 2023
745d967
pass along the artifact name so we can override it later (#20732)
azure-sdk May 3, 2023
6dfd0cb
[azeventhubs] Fixing checkpoint store race condition (#20727)
richardpark-msft May 3, 2023
ed7f3c7
Fix azidentity troubleshooting guide link (#20736)
chlowell May 3, 2023
b2cddab
[Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (…
Alancere May 4, 2023
2a8d96d
add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)
Alancere May 4, 2023
0d22aed
add sdk/resourcemanager/eventhub/armeventhub live test (#20686)
Alancere May 4, 2023
5fa7df4
add sdk/resourcemanager/compute/armcompute live test (#20048)
Alancere May 4, 2023
c005ed6
sdk/resourcemanager/network/armnetwork live test (#20331)
Alancere May 4, 2023
36f766d
add sdk/resourcemanager/cosmos/armcosmos live test (#20705)
Alancere May 4, 2023
9c9d62a
Increment package version after release of azcore (#20740)
azure-sdk May 4, 2023
8bc3450
[azeventhubs] Improperly resetting etag in the checkpoint store (#20737)
richardpark-msft May 4, 2023
e1a6152
Eng workflows sync and branch cleanup additions (#20743)
azure-sdk May 4, 2023
04b463d
[azeventhubs] Latest start position can also be inclusive (ie, get th…
richardpark-msft May 4, 2023
8849196
Update GitHubEventProcessor version and remove pull_request_review pr…
azure-sdk May 5, 2023
27f5ee0
Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)
chlowell May 5, 2023
2eec707
fix (#20707)
Alancere May 6, 2023
22db2d4
AzFile (#20739)
souravgupta-msft May 8, 2023
0cbfd88
azfile: Fixing connection string parsing logic (#20798)
souravgupta-msft May 8, 2023
d54fb08
[azadmin] fix flaky test (#20758)
gracewilcox May 8, 2023
ad8ebd9
Prepare azidentity v1.3.0 for release (#20756)
chlowell May 8, 2023
e2a6f70
Fix broken podman link (#20801)
azure-sdk May 8, 2023
a59d912
[azquery] update doc comments (#20755)
gracewilcox May 8, 2023
bd3b467
Fixed contribution section (#20752)
bobtabor-msft May 8, 2023
132a01a
[azeventhubs,azservicebus] Some API cleanup, renames (#20754)
richardpark-msft May 8, 2023
8db51ca
Add supporting features to enable distributed tracing (#20301) (#20708)
jhendrixMSFT May 9, 2023
4a66b4f
Restore ARM CAE support for azcore beta (#20657)
chlowell May 9, 2023
7d4a3cb
Upgrade to stable azcore (#20808)
chlowell May 9, 2023
068c3be
Increment package version after release of data/azcosmos (#20807)
azure-sdk May 9, 2023
8e0f66e
Updating changelog (#20810)
souravgupta-msft May 9, 2023
ce926c4
Add fake package to azcore (#20711)
jhendrixMSFT May 9, 2023
1a145c5
Updating CHANGELOG.md (#20809)
siminsavani-msft May 9, 2023
90dfc5c
changelog (#20811)
tasherif-msft May 9, 2023
c7eda59
Increment package version after release of storage/azfile (#20813)
azure-sdk May 9, 2023
7fac0b5
Update changelog (azblob) (#20815)
siminsavani-msft May 9, 2023
498a2ef
[azquery] migration guide (#20742)
gracewilcox May 9, 2023
ccb967e
Increment package version after release of monitor/azquery (#20820)
azure-sdk May 9, 2023
f4e6a22
[keyvault] prep for release (#20819)
gracewilcox May 10, 2023
8fd8eda
Merge branch 'main' into feature/azdatalake
tasherif-msft May 11, 2023
c94fa00
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft May 11, 2023
fc0b2b5
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jun 12, 2023
6fb1694
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jun 19, 2023
4f7fe43
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jun 26, 2023
3dac9d0
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 4, 2023
a0a861b
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 7, 2023
124e27e
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 19, 2023
0f5a52c
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 24, 2023
81dabb1
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 27, 2023
d87e78b
Merge remote-tracking branch 'upstream/feature/azdatalake' into featu…
tasherif-msft Jul 31, 2023
450a48e
lease implementation and tests
tasherif-msft Jul 31, 2023
e8a41ef
tests
tasherif-msft Jul 31, 2023
c2891f5
handled errors
tasherif-msft Jul 31, 2023
767cbd9
nit
tasherif-msft Aug 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sdk/storage/azdatalake/assets.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "go",
"TagPrefix": "go/storage/azdatalake",
"Tag": "go/storage/azdatalake_9dd1cc3e0e"
"Tag": "go/storage/azdatalake_78f150eb1d"
}
4 changes: 2 additions & 2 deletions sdk/storage/azdatalake/directory/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -219,14 +219,14 @@ func (d *Client) renamePathInURL(newName string) (string, string, string) {
lastIndex := strings.LastIndex(endpoint, separator)
// Split the string based on the last occurrence of the separator
firstPart := endpoint[:lastIndex] // From the beginning of the string to the last occurrence of the separator
newPathURL, newBlobURL := shared.GetURLs(runtime.JoinPaths(firstPart, newName))
newBlobURL, newPathURL := shared.GetURLs(runtime.JoinPaths(firstPart, newName))
parsedNewURL, _ := url.Parse(d.DFSURL())
return parsedNewURL.Path, newPathURL, newBlobURL
}

// Rename renames a directory (dfs1)
func (d *Client) Rename(ctx context.Context, newName string, options *RenameOptions) (RenameResponse, error) {
newPathWithoutURL, newBlobURL, newPathURL := d.renamePathInURL(newName)
newPathWithoutURL, newPathURL, newBlobURL := d.renamePathInURL(newName)
lac, mac, smac, createOpts := path.FormatRenameOptions(options, newPathWithoutURL)
var newBlobClient *blockblob.Client
var err error
Expand Down
4 changes: 2 additions & 2 deletions sdk/storage/azdatalake/file/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -224,14 +224,14 @@ func (f *Client) renamePathInURL(newName string) (string, string, string) {
lastIndex := strings.LastIndex(endpoint, separator)
// Split the string based on the last occurrence of the separator
firstPart := endpoint[:lastIndex] // From the beginning of the string to the last occurrence of the separator
newPathURL, newBlobURL := shared.GetURLs(runtime.JoinPaths(firstPart, newName))
newBlobURL, newPathURL := shared.GetURLs(runtime.JoinPaths(firstPart, newName))
parsedNewURL, _ := url.Parse(f.DFSURL())
return parsedNewURL.Path, newPathURL, newBlobURL
}

// Rename renames a file (dfs1)
func (f *Client) Rename(ctx context.Context, newName string, options *RenameOptions) (RenameResponse, error) {
newPathWithoutURL, newBlobURL, newPathURL := f.renamePathInURL(newName)
newPathWithoutURL, newPathURL, newBlobURL := f.renamePathInURL(newName)
lac, mac, smac, createOpts := path.FormatRenameOptions(options, newPathWithoutURL)
var newBlobClient *blockblob.Client
var err error
Expand Down
4 changes: 2 additions & 2 deletions sdk/storage/azdatalake/filesystem/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -184,15 +184,15 @@ func (fs *Client) BlobURL() string {
// The new directory.Client uses the same request policy pipeline as the Client.
func (fs *Client) NewDirectoryClient(directoryPath string) *directory.Client {
dirURL := runtime.JoinPaths(fs.generatedFSClientWithDFS().Endpoint(), directoryPath)
dirURL, blobURL := shared.GetURLs(dirURL)
blobURL, dirURL := shared.GetURLs(dirURL)
return (*directory.Client)(base.NewPathClient(dirURL, blobURL, fs.containerClient().NewBlockBlobClient(directoryPath), fs.generatedFSClientWithDFS().InternalClient().WithClientName(shared.DirectoryClient), fs.sharedKey(), fs.identityCredential(), fs.getClientOptions()))
}

// NewFileClient creates a new file.Client object by concatenating file path to the end of this Client's URL.
// The new file.Client uses the same request policy pipeline as the Client.
func (fs *Client) NewFileClient(filePath string) *file.Client {
fileURL := runtime.JoinPaths(fs.generatedFSClientWithDFS().Endpoint(), filePath)
fileURL, blobURL := shared.GetURLs(filePath)
blobURL, fileURL := shared.GetURLs(fileURL)
return (*file.Client)(base.NewPathClient(fileURL, blobURL, fs.containerClient().NewBlockBlobClient(filePath), fs.generatedFSClientWithDFS().InternalClient().WithClientName(shared.FileClient), fs.sharedKey(), fs.identityCredential(), fs.getClientOptions()))
}

Expand Down
14 changes: 14 additions & 0 deletions sdk/storage/azdatalake/internal/testcommon/clients_auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,20 @@ func GetFileClient(fsName, fName string, t *testing.T, accountType TestAccountTy
return fileClient, err
}

func CreateNewFile(ctx context.Context, _require *require.Assertions, fileName string, filesystemClient *filesystem.Client) *file.Client {
fileClient := filesystemClient.NewFileClient(fileName)
_, err := fileClient.Create(ctx, nil)
_require.Nil(err)
return fileClient
}

func CreateNewDir(ctx context.Context, _require *require.Assertions, dirName string, filesystemClient *filesystem.Client) *directory.Client {
dirClient := filesystemClient.NewDirectoryClient(dirName)
_, err := dirClient.Create(ctx, nil)
_require.Nil(err)
return dirClient
}

func GetDirClient(fsName, dirName string, t *testing.T, accountType TestAccountType, options *directory.ClientOptions) (*directory.Client, error) {
if options == nil {
options = &directory.ClientOptions{}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,28 +8,36 @@ package lease

import (
"context"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/lease"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/filesystem"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/base"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/exported"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/generated"
)

// FilesystemClient provides lease functionality for the underlying filesystem client.
type FilesystemClient struct {
leaseID *string
containerClient *lease.ContainerClient
leaseID *string
}

// FilesystemClientOptions contains the optional values when creating a FilesystemClient.
type FilesystemClientOptions struct {
// LeaseID contains a caller-provided lease ID.
LeaseID *string
}
type FilesystemClientOptions = lease.ContainerClientOptions

// NewFilesystemClient creates a filesystem lease client for the provided filesystem client.
// - client - an instance of a filesystem client
// - options - client options; pass nil to accept the default values
func NewFilesystemClient(client *filesystem.Client, options *FilesystemClientOptions) (*FilesystemClient, error) {
// TODO: set up container lease client
return nil, nil
_, _, containerClient := base.InnerClients((*base.CompositeClient[generated.FileSystemClient, generated.FileSystemClient, container.Client])(client))
containerLeaseClient, err := lease.NewContainerClient(containerClient, options)
if err != nil {
return nil, exported.ConvertToDFSError(err)
}
return &FilesystemClient{
containerClient: containerLeaseClient,
leaseID: containerLeaseClient.LeaseID(),
}, nil
}

// LeaseID returns leaseID of the client.
Expand All @@ -42,33 +50,42 @@ func (c *FilesystemClient) LeaseID() *string {
// For more information, see https://docs.microsoft.com/rest/api/storageservices/lease-blob.
func (c *FilesystemClient) AcquireLease(ctx context.Context, duration int32, o *FilesystemAcquireOptions) (FilesystemAcquireResponse, error) {
opts := o.format()
return c.containerClient.AcquireLease(ctx, duration, opts)
resp, err := c.containerClient.AcquireLease(ctx, duration, opts)
return resp, exported.ConvertToDFSError(err)
}

// BreakLease breaks the filesystem's previously-acquired lease (if it exists). Pass the LeaseBreakDefault (-1)
// constant to break a fixed-Duration lease when it expires or an infinite lease immediately.
// For more information, see https://docs.microsoft.com/rest/api/storageservices/lease-blob.
func (c *FilesystemClient) BreakLease(ctx context.Context, o *FilesystemBreakOptions) (FilesystemBreakResponse, error) {
opts := o.format()
return c.containerClient.BreakLease(ctx, opts)
resp, err := c.containerClient.BreakLease(ctx, opts)
return resp, exported.ConvertToDFSError(err)
}

// ChangeLease changes the filesystem's lease ID.
// For more information, see https://docs.microsoft.com/rest/api/storageservices/lease-blob.
func (c *FilesystemClient) ChangeLease(ctx context.Context, proposedLeaseID string, o *FilesystemChangeOptions) (FilesystemChangeResponse, error) {
opts := o.format()
return c.containerClient.ChangeLease(ctx, proposedLeaseID, opts)
resp, err := c.containerClient.ChangeLease(ctx, proposedLeaseID, opts)
if err != nil {
return resp, exported.ConvertToDFSError(err)
}
c.leaseID = &proposedLeaseID
return resp, nil
}

// RenewLease renews the filesystem's previously-acquired lease.
// For more information, see https://docs.microsoft.com/rest/api/storageservices/lease-blob.
func (c *FilesystemClient) RenewLease(ctx context.Context, o *FilesystemRenewOptions) (FilesystemRenewResponse, error) {
opts := o.format()
return c.containerClient.RenewLease(ctx, opts)
resp, err := c.containerClient.RenewLease(ctx, opts)
return resp, exported.ConvertToDFSError(err)
}

// ReleaseLease releases the filesystem's previously-acquired lease.
func (c *FilesystemClient) ReleaseLease(ctx context.Context, o *FilesystemReleaseOptions) (FilesystemReleaseResponse, error) {
opts := o.format()
return c.containerClient.ReleaseLease(ctx, opts)
resp, err := c.containerClient.ReleaseLease(ctx, opts)
return resp, exported.ConvertToDFSError(err)
}
Loading