Engineering: Progress aware file upload to Azure Blob Storage

Posted by Nilay Parikh and last modified on Tue Jun 12, 2018.

Azure SDK do not support trackable upload progress, the process is considered as a single task. However, interestingly the internal process split the upload content into small chunks and upload them individually, but the code wait for the entire task to complete.

It is interesting to see that we can implement the same process by code. We would manually split the file into small chunks and upload these chunks asynchronously using PutBlockAsync. Once all parts are successfully uploaded to the container then will call PutBlockListAsync to commit the file.

Please refer the latest Microsoft Blog Post for updated version Asynchronous Parallel Blob Transfers with Progress Change Notification 2.0


CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials("accountname", "accountkey"), true);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
blobClient.SingleBlobUploadThresholdInBytes = 1024 * 1024;
CloudBlobContainer blobContainer = blobClient.GetContainerReference("MyContainerReferance");
CloudBlockBlob cloudBlockBlob = blobContainer.GetBlockBlobReference("bigfile.zip");
var blockSize = 256 * 1024;
cloudBlockBlob.StreamWriteSizeInBytes = blockSize;
var fileName = @"D:\bigfile.zip";
long bytesToUpload = (new FileInfo(fileName)).Length;
long fileSize = bytesToUpload;

if (bytesToUpload < blockSize)
{
    CancellationToken cancellationToken = new CancellationToken();
    var ado = cloudBlockBlob.UploadFromFileAsync(fileName, FileMode.Open, cancellationToken);
    Console.WriteLine(ado.Status);
    ado.ContinueWith(t =>
    {
        Console.WriteLine("Status = " + t.Status);
        Console.WriteLine("Done");
    });
}
else
{
    List<string> blockIds = new List<string>();
    int index = 1;
    long startPosition = 0;
    long bytesUploaded = 0;
    do
    {
        var bytesToRead = Math.Min(blockSize, bytesToUpload);
        var blobContents = new byte[bytesToRead];
        using (FileStream fs = new FileStream(fileName, FileMode.Open))
        {
            fs.Position = startPosition;
            fs.Read(blobContents, 0, (int)bytesToRead);
        }
        ManualResetEvent manualResetEvent = new ManualResetEvent(false);
        var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(index.ToString("d6")));
        blockIds.Add(blockId);
        var blockAsync = cloudBlockBlob.PutBlockAsync(blockId, new MemoryStream(blobContents), null);
        blockAsync.ContinueWith(t =>
        {
            bytesUploaded += bytesToRead;
            bytesToUpload -= bytesToRead;
            startPosition += bytesToRead;
            index++;
            double percentComplete = (double)bytesUploaded / (double)fileSize;
            Console.WriteLine("Percent complete = " + percentComplete.ToString("P"));
            manualResetEvent.Set();
        });
        manualResetEvent.WaitOne();
    }
    while (bytesToUpload > 0);
    var blockListAsync = cloudBlockBlob.PutBlockListAsync(blockIds);
    blockListAsync.ContinueWith(t =>
    {
        Console.WriteLine("Blob uploaded.");
    });
}

Disclaimer

Any views or opinions expressed are solely those of the author and do not represent any other person or organisation. THE ARTICLE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND. IN NO EVENT SHALL THE AUTHOR(S) OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY.

References