Azure.Storage.DataMovement 12.0.0-beta.3

Azure Storage Data Movement Common client library for .NET

Server Version: 2021-02-12, 2020-12-06, 2020-10-02, 2020-08-04, 2020-06-12, 2020-04-08, 2020-02-10, 2019-12-12, 2019-07-07, and 2020-02-02

Azure Storage is a Microsoft-managed service providing cloud storage that is highly available, secure, durable, scalable, and redundant.

The Azure Storage Data Movement library is optimized for uploading, downloading and copying customer data.

Currently this version of the Data Movement library only supports Blobs.

Source code | API reference documentation | REST API documentation | Product documentation

Getting started

Install the package

Install the Azure Storage client library for .NET you'd like to use with NuGet and the Azure.Storage.DataMovement client library will be included:

dotnet add package Azure.Storage.DataMovement --prerelease

Prerequisites

You need an Azure subscription and a Storage Account to use this package.

To create a new Storage Account, you can use the Azure Portal, Azure PowerShell, or the Azure CLI. Here's an example using the Azure CLI:

az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS

Authenticate the client

In order to interact with the Data Movement library you have to create an instance with the TransferManager class.

Create Instance of TransferManager

TransferManager transferManager = new TransferManager(new TransferManagerOptions());

Create Instance of TransferManager with Options

// Create BlobTransferManager with event handler in Options bag
TransferManagerOptions transferManagerOptions = new TransferManagerOptions();
TransferOptions options = new TransferOptions()
{
    MaximumTransferChunkSize = 4 * Constants.MB,
    CreateMode = StorageResourceCreateMode.Overwrite,
};
TransferManager transferManager = new TransferManager(transferManagerOptions);

Key concepts

The Azure Storage Common client library contains shared infrastructure like authentication credentials and RequestFailedException.

Thread safety

We guarantee that all client instance methods are thread-safe and independent of each other (guideline). This ensures that the recommendation of reusing client instances is always safe, even across threads.

Additional concepts

Client options | Accessing the response | Long-running operations | Handling failures | Diagnostics | Mocking | Client lifetime

Examples

Please see the examples for Blobs DataMovement.

Pause a transfer using the TransferManager using the respective DataTransfer object

DataTransfer dataTransfer = await transferManager.StartTransferAsync(
    sourceResource: sourceResource,
    destinationResource: destinationResource);

// Pause from the Transfer Manager using the DataTransfer object
await transferManager.PauseTransferIfRunningAsync(dataTransfer);

Pause a transfer using the TransferManager using the respective transfer ID

DataTransfer dataTransfer = await transferManager.StartTransferAsync(
    sourceResource: sourceResource,
    destinationResource: destinationResource);
string transferId = dataTransfer.Id;

// Pause from the Transfer Manager using the Transfer Id
await transferManager.PauseTransferIfRunningAsync(transferId);

Pause a transfer using the respective DataTransfer

DataTransfer dataTransfer = await transferManager.StartTransferAsync(
    sourceResource: sourceResource,
    destinationResource: destinationResource);

// Pause from the DataTransfer object
await dataTransfer.PauseIfRunningAsync();

Resume a transfer

async Task<(StorageResource Source, StorageResource Destination)> MakeResourcesAsync(DataTransferProperties info)
{
    StorageResource sourceResource = null, destinationResource = null;
    if (BlobStorageResources.TryGetResourceProviders(
        info,
        out BlobStorageResourceProvider blobSrcProvider,
        out BlobStorageResourceProvider blobDstProvider))
    {
        sourceResource ??= await blobSrcProvider.MakeResourceAsync(GetMyCredential(info.SourcePath));
        destinationResource ??= await blobSrcProvider.MakeResourceAsync(GetMyCredential(info.DestinationPath));
    }
    if (LocalStorageResources.TryGetResourceProviders(
        info,
        out LocalStorageResourceProvider localSrcProvider,
        out LocalStorageResourceProvider localDstProvider))
    {
        sourceResource ??= localSrcProvider.MakeResource();
        destinationResource ??= localDstProvider.MakeResource();
    }
    return (sourceResource, destinationResource);
}
List<DataTransfer> resumedTransfers = new();
await foreach (DataTransferProperties transferProperties in transferManager.GetResumableTransfersAsync())
{
    (StorageResource resumeSource, StorageResource resumeDestination) = await MakeResourcesAsync(transferProperties);
    resumedTransfers.Add(await transferManager.ResumeTransferAsync(transferProperties.TransferId, resumeSource, resumeDestination));
}

Troubleshooting

All Azure Storage services will throw a RequestFailedException with helpful ErrorCodes.

Next steps

Get started with our Blob DataMovement samples.

Contributing

See the Storage CONTRIBUTING.md for details on building, testing, and contributing to these libraries.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit cla.microsoft.com.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.