S3Append 0.0.1

.NET AWS SDK S3 Append

About

S3 Append provides AppendObjectAsync extention method for .NET AWS SDK S3 client, capable of appending data to existing S3 hosted objects.

Performance

Implemented functionality relies on "server side" AWS S3 multipart copy operation, making its performance superior to naive download-update-upload approach. In a nutshel, existing portion of data is being copied by S3 internally while only to-be-appended chunk is uploaded from client's side. For details (and limitations) please refer to relevant documentation.

Implications

The key consideration is costs. Compared to naive approach, at least five AWS S3 requests have to be issued for every and each append operation:

• Yield object metadata to determine its size
• Create multipart upload through which data is to be copied/uploaded
• Copy existing data on the server side
• Upload new data from client side
• Complete multipart upload, overriding original object

Moreover, single UploadPartCopy operation could only copy up to 5 GiB of data. That said, append to an object with size of 5 TB would result in (at least) 1004 requests issued by S3 Append logic.

Usage

When imported into scope, AppendObjectAsync could be used in in a straightforward fashion. Consider S3 bucket 109a6d191b67 hosting object fa5ec9042bc3 with plain text content Hello. Following code would, when executed,

using Amazon.S3;
using S3Append.Extensions;
using S3Append.Models;
using System.Threading.Tasks;

public static async Task Main(string[] args)
{
	var client = new AmazonS3Client();
	var request = new AppendObjectRequest
	{
		BucketName = "109a6d191b67",
		Key = "fa5ec9042bc3",
		ContentBody = " world!"
	};

	await client.AppendObjectAsync(request);
}

result in (the same) fa5ec9042bc3 object containing proverbial Hello world!.