Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update DataMovement READMEs #37590

Merged
merged 10 commits into from
Jul 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 127 additions & 60 deletions sdk/storage/Azure.Storage.DataMovement.Blobs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,11 @@ We guarantee that all client instance methods are thread-safe and independent of

## Examples

### Examples using BlobContainerClient extension methods to upload and download directories.
This section demonstrates usage of Data Movement for interacting with blob storage.

### Extensions on `BlobContainerClient`

For applicatons with preexisting code using Azure.Storage.Blobs, this package provides extension methods for `BlobContainerClient` to get some of the benefits of the `TransferManager` with minimal extra code.

Instantiate the BlobContainerClient
```C# Snippet:ExtensionMethodCreateContainerClient
Expand Down Expand Up @@ -139,95 +143,158 @@ DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirect
await tranfer.WaitForCompletionAsync();
```

### Examples using BlobContainerClient extension methods to upload and download directories.
### Initializing Blob Storage `StorageResource`

Create Instance of TransferManager with Options
jaschrep-msft marked this conversation as resolved.
Show resolved Hide resolved
```C# Snippet:CreateTransferManagerWithOptions
// Create BlobTransferManager with event handler in Options bag
TransferManagerOptions transferManagerOptions = new TransferManagerOptions();
TransferOptions options = new TransferOptions()
Azure.Storage.DataMovement.Blobs exposes a `StorageResource` for each type of blob (block, page, append) as well as a blob container. Storage resources are initialized with the appropriate client object from Azure.Storage.Blobs.

```C# Snippet:ResourceConstruction_Blobs
StorageResource containerResource = new BlobStorageResourceContainer(blobContainerClient);
StorageResource blockBlobResource = new BlockBlobStorageResource(blockBlobClient);
StorageResource pageBlobResource = new PageBlobStorageResource(pageBlobClient);
StorageResource appendBlobResource = new AppendBlobStorageResource(appendBlobClient);
```

Blob `StorageResource` objects can be constructed with optional "options" arguments specific to the type of resource.

```C# Snippet:ResourceConstruction_Blobs_WithOptions_VirtualDirectory
BlobStorageResourceContainerOptions virtualDirectoryOptions = new()
{
MaximumTransferChunkSize = 4 * Constants.MB,
CreateMode = StorageResourceCreateMode.Overwrite,
DirectoryPrefix = "blob/directory/prefix"
};

StorageResource virtualDirectoryResource = new BlobStorageResourceContainer(
blobContainerClient,
virtualDirectoryOptions);
```

```C# Snippet:ResourceConstruction_Blobs_WithOptions_BlockBlob
BlockBlobStorageResourceOptions leasedResourceOptions = new()
{
SourceConditions = new()
{
LeaseId = leaseId
}
};
TransferManager transferManager = new TransferManager(transferManagerOptions);
StorageResource leasedBlockBlobResource = new BlockBlobStorageResource(
blockBlobClient,
leasedResourceOptions);
```

Start Upload from Local File to Block Blob
When resuming a transfer, a credential to Azure Storage is likely needed. Credentials are not persisted by the transfer manager. When using `BlobStorageResources.TryGetResourceProviders()` to recreate a `StorageResource` for resume, the returned provider can create the resource with a credential specified by the calling code. This allows for workflows like scoping generation of a Shared Access Signature to the given resource path. Your application should provide its own mechanism for getting the appropriate credential, represented by `GenerateMySasCredential()` in the sample below.

```C# Snippet:RehydrateBlobResource
StorageResource sourceResource = null;
StorageResource destinationResource = null;
if (BlobStorageResources.TryGetResourceProviders(
info,
out BlobStorageResourceProvider blobSrcProvider,
out BlobStorageResourceProvider blobDstProvider))
{
sourceResource ??= await blobSrcProvider?.MakeResourceAsync(
GenerateMySasCredential(info.SourcePath));
destinationResource ??= await blobSrcProvider?.MakeResourceAsync(
GenerateMySasCredential(info.DestinationPath));
}
```

### Upload

An upload takes place between a local file `StorageResource` as source and blob `StorageResource` as destination.

Upload a block blob.

```C# Snippet:SimpleBlobUpload
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
sourceResource: new LocalFileStorageResource(sourceLocalPath),
destinationResource: new BlockBlobStorageResource(destinationBlob));
await dataTransfer.WaitForCompletionAsync();
```
Apply Options to Block Blob Download
```C# Snippet:BlockBlobDownloadOptions
await transferManager.StartTransferAsync(
sourceResource: new BlockBlobStorageResource(sourceBlob, new BlockBlobStorageResourceOptions()
{
DestinationConditions = new BlobRequestConditions(){ LeaseId = "xyz" }
}),
destinationResource: new LocalFileStorageResource(downloadPath2));
```

Start Directory Upload
Upload a directory as a specific blob type.

```C# Snippet:SimpleDirectoryUpload
// Create simple transfer directory upload job which uploads the directory and the contents of that directory
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
sourceResource: new LocalDirectoryStorageResourceContainer(sourcePath),
destinationResource: new BlobStorageResourceContainer(
container,
new BlobStorageResourceContainerOptions() { DirectoryPrefix = "sample-directory2" }),
blobContainerClient,
new BlobStorageResourceContainerOptions()
{
// Block blobs are the default if not specified
BlobType = BlobType.Block,
DirectoryPrefix = optionalDestinationPrefix,
}),
transferOptions: options);
await dataTransfer.WaitForCompletionAsync();
```

### Download

A download takes place between a blob `StorageResource` as source and local file `StorageResource` as destination.

Download a block blob.

```C# Snippet:SimpleBlockBlobDownload
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
sourceResource: new BlockBlobStorageResource(sourceBlobClient),
destinationResource: new LocalFileStorageResource(downloadPath));
await dataTransfer.WaitForCompletionAsync();
```

Start Directory Download
```C# Snippet:SimpleDirectoryDownload
DataTransfer downloadDirectoryJobId2 = await transferManager.StartTransferAsync(
sourceDirectory2,
destinationDirectory2);
Download a container which may contain a mix of blob types.

```C# Snippet:SimpleDirectoryDownload_Blob
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
sourceResource: new BlobStorageResourceContainer(
blobContainerClient,
new BlobStorageResourceContainerOptions()
{
DirectoryPrefix = optionalSourcePrefix
}),
destinationResource: new LocalDirectoryStorageResourceContainer(downloadPath));
await dataTransfer.WaitForCompletionAsync();
```

Simple Logger Sample for Transfer Manager Options
jaschrep-msft marked this conversation as resolved.
Show resolved Hide resolved
```C# Snippet:SimpleLoggingSample
// Create BlobTransferManager with event handler in Options bag
TransferManagerOptions options = new TransferManagerOptions();
TransferOptions transferOptions = new TransferOptions();
transferOptions.SingleTransferCompleted += (SingleTransferCompletedEventArgs args) =>
{
using (StreamWriter logStream = File.AppendText(logFile))
{
logStream.WriteLine($"File Completed Transfer: {args.SourceResource.Path}");
}
return Task.CompletedTask;
};
### Blob Copy

A copy takes place between two blob `StorageResource` instances. Copying between to Azure blobs uses PUT from URL REST APIs, which do not pass data through the calling machine.

Copy a single blob. Note the change in blob type on this copy from block to append.

```C# Snippet:s2sCopyBlob
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
sourceResource: new BlockBlobStorageResource(sourceBlockBlobClient),
destinationResource: new AppendBlobStorageResource(destinationAppendBlobClient));
await dataTransfer.WaitForCompletionAsync();
```

Simple Failed Event Delegation for Container Transfer Options
```C# Snippet:FailedEventDelegation
transferOptions.TransferFailed += (TransferFailedEventArgs args) =>
jaschrep-msft marked this conversation as resolved.
Show resolved Hide resolved
{
using (StreamWriter logStream = File.AppendText(logFile))
{
// Specifying specific resources that failed, since its a directory transfer
// maybe only one file failed out of many
logStream.WriteLine($"Exception occured with TransferId: {args.TransferId}," +
$"Source Resource: {args.SourceResource.Path}, +" +
$"Destination Resource: {args.DestinationResource.Path}," +
$"Exception Message: {args.Exception.Message}");
}
return Task.CompletedTask;
};
Copy a blob container.

```C# Snippet:s2sCopyBlobContainer
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
sourceResource: new BlobStorageResourceContainer(
sourceContainer,
new BlobStorageResourceContainerOptions()
{
DirectoryPrefix = sourceDirectoryName
}),
destinationResource: new BlobStorageResourceContainer(
destinationContainer,
new BlobStorageResourceContainerOptions()
{
// all source blobs will be copied as a single type of destination blob
// defaults to block blobs if unspecified
BlobType = BlobType.Block,
DirectoryPrefix = downloadPath
}));
```

## Troubleshooting

All Azure Storage services will throw a [RequestFailedException][RequestFailedException]
with helpful [`ErrorCode`s][error_codes].
***TODO***

## Next steps

Get started with our [Blob DataMovement samples][samples].
***TODO***

## Contributing

Expand Down
Loading