Cloud Economics: Computational Load Testing and Stress Testing for Azure App Service Web Api (Performance Benchmark)

Posted by Nilay Parikh and last modified on Tue Jun 12, 2018.

App Service, Service Fabric and Serverless Azure Function are becoming flagship computational service for Microsoft Azure. I am trying to find an economical benchmark for Computing in Azure. It is not a straight-forward task, but I am employing a simple methodology to find cost/computing rational between all three computational platforms.

Summary

Following is the highest throughput and zero error load test’s summary out 5 various load patterns for Azure App Service Server Farm of 3 instances.

Hourly Cost (3 x Instances)0.131183505 GBP x 3 =
0.393550515 GBP per Hour
Azure Pricing
Hourly Test Count579,600 Api CallsLoad Test Stats & Profile
Zero-Fault Average CPU Consumption91.490875%Performance Stats

Load Test - Detail Breakdown

Test Methodology

I ran user load of 600, 1300, 500, 500, 500, 50, 50, 50 respectively. My objective was to find Zero-Fault maximum throughput on the Azure App Service.

The user load was well distributed with planned load spikes for a successful run (I am using the same scenario pattern for all Azure Compute comparisons).

App Service Profile

Sites1
Instances3
Api (SDK)Web Api 2.0

Machine Profile


"sku": {
  "name": "B3",
  "tier": "Basic",
  "size": "B3",
  "family": "B",
  "capacity": 3
}

Machine Configuration

SizeB3 Basic
CPU cores4
Memory GiB7
Local Storage GiB10

Azure Pricing [as of 25th April 2017]

  • 0.131183505 per Hour per Instance - Based on Detail Usage Report from Azure Portal
  • 0.131183505 x 3 = 0.393550515 per Hour per App Service Farm (3xNode)

Web API Harness - Sample Code

Web API deploy two test endpoints,

  • APIEndpoint1 - light-weight and high-volume scenario.
  • APIEndpoint2 - CPU Cruncher with complex graphics and mathematical calculations.
APIEndpoint1

Generate random Guid and apply list and hash operations.


var guidList = new List<dynamic>();
for (var i = 0; i < 300000; i++)
{
    var randomGuid = Guid.NewGuid().ToString();
    var guid = randomGuid;
    var hash = randomGuid.GetHashCode();
    var reverseGuid = Reverse(randomGuid);
}

for (var i = 0; i < 1000; i++)
{
    var randomGuid = Guid.NewGuid().ToString();
    guidList.Add(new { Guid = randomGuid, Hash = randomGuid.GetHashCode(), ReverseGuid = Reverse(randomGuid) });
}

// Return GuidList using JsonConvert

APIEndpoint2

Accept Base64 2048x2048+300dpi High-Resolution image and applying some rigorous image manipulation.


ISupportedImageFormat format = new JpegFormat { Quality = 70 };
Size size = new Size(150, 0)
using (MemoryStream inStream = new MemoryStream(photoBytes))
{
    using (MemoryStream outStream = new MemoryStream())
    {
        using (ImageFactory imageFactory = new ImageFactory(preserveExifData:true))
        {
            imageFactory.Load(inStream)
                        .Hue(180, true)
                        .Brightness(40)
                        .Filter(matrixFilter)
                        .Resize(size)
                        .Pixelate(20)
                        .Format(format)
                        .Save(outStream);
        }

        // Converting Outstream to Base64 and add into HttpResponse.
    }
}

Load Test Stats & Profile

I find TestId/3015 was the best match for comparison. Following is the summary of the test run.

Max User Load500 (No warm up or think time)
Tests/Sec161 (579,600 Tests per Hour)
Tests Failed11
Avg. Test Time (sec)0.37 (370 ms)
Avg. Http Response Size (KiB)137.21
Throughput/Sec (KiB)22,090.81 (21.5731 MiB)

Azure Metrics Query


https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/{{your resource group}}/providers/Microsoft.Web/sites/{{your site farm}}/providers/microsoft.insights/metrics?api-version=2016-09-01&$filter=(name.value eq 'CpuPercentage') and aggregationType eq 'Average' and startTime eq 2017-04-25T21:20:00Z and endTime eq 2017-04-25T21:40:00Z and timeGrain eq duration'PT1M'

Performance Stats

You can also find raw result set at Gist .

Timestamp Average CPU
2017-04-25T21:20:00Z 92.78
2017-04-25T21:21:00Z 97.43
2017-04-25T21:22:00Z 93.185
2017-04-25T21:23:00Z 92.375
2017-04-25T21:24:00Z 96.125
2017-04-25T21:25:00Z 90.585
2017-04-25T21:26:00Z 92.425
2017-04-25T21:27:00Z 91.45
2017-04-25T21:28:00Z 90.35
2017-04-25T21:29:00Z 91.88
2017-04-25T21:30:00Z 93.87
2017-04-25T21:31:00Z 91.29
2017-04-25T21:32:00Z 92.5275
2017-04-25T21:33:00Z 87.2025
2017-04-25T21:34:00Z 90.84
2017-04-25T21:35:00Z 90.32
2017-04-25T21:36:00Z 90.3725
2017-04-25T21:37:00Z 90.5775
2017-04-25T21:38:00Z 83.9275
2017-04-25T21:39:00Z 90.305
Disclaimer

Any views or opinions expressed are solely those of the author and do not represent any other person or organisation. THE ARTICLE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND. IN NO EVENT SHALL THE AUTHOR(S) OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY.