Azure Functions Get More Scalable and Elastic

Back in August this year, I've posted Azure Functions: Are They Really Infinitely Scalable and Elastic? with two experiments about Azure Function App auto scaling. I ran a simple CPU-bound function based on Bcrypt hashing, and measured how well Azure was running my Function under load.

The results were rather pessimistic. Functions were scaling up to many instances, but there were significant delays in doing so, so the processing slowed down up to 40 minutes.

Azure Functions team notified me that they rolled out an updated version of the service, which should significantly improve my results.

So I ran the exact same tests again, and got the new results. I will show these results below.

TL;DR. Scaling responsiveness improved significantly. The max delay reduced from 40 to 6 minutes. There are some improvements still to be desired: sub-minute latency is not yet reachable for similar scenarios.

Setup

See the Function code and the description of the two experiments in my previous article.

Experiment 1: Steady Load

In "Steady Load" scenario 100,000 messages were sent to the queue at constant pace, evenly spread over 2 hours.

Here are the old metrics of Queue Backlog and Instance Count over time:

Function App Scaling (Old)

Old charts are shown in gray background

You can see a huge delay of almost one hour before the function caught up to speed of incoming messages and half-an-hour more before the backlog got cleared.

The new results on the same chart after the runtime update:

Function App Scaling (New)

This looks much better. The maximum backlog is 7 times lower; there's almost no initial delay before the auto scaling kicks in; and overall instance allocation is much more stable.


One more chart is from the same experiment, but it shows slightly different metrics. The old results of Delay (Age) in seconds and Processing Rate in messages per minute:

Function App Delay

The new chart after the runtime update:

Function App Delay

Again, much less delay overall, and processing rate more-or-less stabilizes after the first 15 minutes.

Experiment 2: Spiky Load

The second experiment spanned over 5 hours. The messages were sent mostly at low-ish fixed rate, except for 5 periods of sudden spikes. The green line on the charts below shows these spikes very well.

At the first run 4 months ago, Functions runtime had troubles keeping up to speed even between those bursts of messages.

Here is the chart of the old spiky load processing:

Spicky Load Processing (Old)

You can see that the backlog after each spike goes down really slow. The blue line of processing rate doesn't match the green line almost nowhere, which reveals the struggle to adapt.

The new results of the same chart after the runtime update are quite different:

Spicky Load Processing (New)

Notice how the backlog is empty and the blue processing rate matches exactly the incoming rate during all time except after traffic bursts. The queue goes up during each spike, but the processing rate immediately accelerates too, and the crisis is gone within 15 minutes.

Conclusions

Azure Functions team is clearly working on improvements. While the results in August were puzzling or even embarrassing, the December benchmark makes much more sense.

Looks like Azure Functions are now suitable for CPU-intensive data processing scenarios with flexible load, targeting the maximum delay at about several minutes.

Obviously, the results are not perfect just yet. Here's what still can be done better:

  • Scale faster initially. In the first experiment, the biggest delay appeared right after the start, when the backlog was growing linearly for 10 minutes. "0 to 100" might not be a very realistic scenario, but probably that's how many folks will test Functions against their workloads.

  • Do not scale down that fast after backlog goes to 0. Every time the queue backlog goes to 0, the runtime kills the biggest part of instances almost immediately. During my runs, this caused the queue to grow again without a good reason from user's perspective.

  • Do not allow the backlog to grow without message spikes. Related to the previous item, but slightly different focus. When the load is stable, I would expect the runtime to keep my queue as close to empty as possible. I guess Azure tries to minimize the resources that it consumes behind the scenes, but this should be balanced in favor of user experience.

  • Make scaling algorithms more open. It's a black box right now. I would love to see some documentation, if not code, to be published about what exactly to expect from Consumption Plan auto scaling.

I'll be running more scaling experiments with other types of workloads in the nearest future, so... more benchmarks are coming.

Happy scaling!

Precompiled Azure Functions in F#

This post is giving a start to F# Advent Calendar in English 2017. Please follow the calendar for all the great posts to come.

Azure Functions is a "serverless" cloud offering from Microsoft. It allows you to run your custom code as response to events in the cloud. Functions are very easy to start with; and you only pay per execution - with free allowance sufficient for any proof-of-concept, hobby project or even low-usage production loads. And when you need more, Azure will scale your project up automatically.

F# is one of the officially supported languages for Azure Functions. Originally, F# support started with F# Script files (authored directly in Azure portal or copied from local editor), so you can find many articles online to get started, e.g. Creating an Azure Function in F# from the ground up and Part 2 by Mathias Brandewinder.

However, I find script-based model a bit limited. In today's article I will focus on creating Azure Functions as precompiled .NET libraries. Along the way, I'll use cross-platform tools like .NET Core and VS Code, and I'll show how to integrate Functions with some popular tools like Suave and Paket.

Create a Project

You can follow this walkthrough on Windows or Mac, just make sure that you have .NET Core 2 and Node.js 8.x with npm installed. My editor of choice is Visual Studio Code with Ionide plugin.

I'll show you how to create a new F# Function App from scratch. If you want to jump to runnable project, you can get it from my github.

We start with creating a new F# library project for .NET Standard 2. Run in your command line:

dotnet new classlib --language F# --name HelloFunctions

This command creates a folder with two files: HelloFunctions.fsproj project file and Library.fs source code file.

Now, add a reference to Azure Functions NuGet package:

dotnet add package Microsoft.NET.Sdk.Functions

Define a Function

Open Library.fs code file and change it to the following code:

namespace HelloFunctions

open System
open Microsoft.Azure.WebJobs
open Microsoft.Azure.WebJobs.Host

module Say =
  let private daysUntil (d: DateTime) =
    (d - DateTime.Now).TotalDays |> int

  let hello (timer: TimerInfo, log: TraceWriter) =
    let christmas = new DateTime(2017, 12, 25)

    daysUntil christmas
    |> sprintf "%d days until Christmas"
    |> log.Info

We defined a function hello which should be triggered by Functions runtime based on time intervals. Every time the function is called, we log how many days we still need to wait before Christmas 2017.

To convert this simple F# function to an Azure Function, create a folder called Hello (or choose any other name) next to the project file and add function.json file in there:

{
  "bindings": [
    {
      "name": "timer",
      "type": "timerTrigger",
      "schedule": "0 * * * * *"
    }
  ],
  "scriptFile": "../bin/HelloFunctions.dll",
  "entryPoint": "HelloFunctions.Say.hello"
}

We defined that:

  • Our function is triggered by timer
  • It runs every minute at 0 seconds
  • The entry point is our hello function in the compiled assembly

Prepare Local Runtime

There are a couple more configuration files needed to be able to run the Function App locally. host.json defines hosting parameters; empty file will do for now:

{
}

Most triggers need to connect to a Storage Account. For examples, timer trigger uses it to hold leases to define which running instance will actually execute the action every minute. Copy a connection string to your Storage Account (local Storage emulator is fine too) and put it into local.settings.json file:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "...your connection string..."
  }
}

Note that this file is only used for local development and is not published to Azure by default.

Finally, we need to modify fsproj file to make the build tool copy those files into bin folder. Add the following section in there:

<ItemGroup>
  <Content Include="Hello\function.json">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </Content>
  <Content Include="host.json">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </Content>
  <Content Include="local.settings.json">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </Content>
</ItemGroup>

Run App Locally

The first step is to build and publish our Function App with dotnet commands:

dotnet build
dotnet publish

The first line produces the dll file and the second line copies it and all of its dependencies to publish folder.

The nice thing about Azure Functions is that you can easily run them locally on a development machine. Execute the following command to install the runtime and all the required libraries:

npm install -g [email protected]

This will add a func CLI to your system which is the tool to use for all Function related operations.

Navigate to bin\Debug\netstandard2.0\publish folder and run func start from there. You should see that your app is now running, and your timer function is scheduled for execution:

Function App Start

Once the next minute comes, the timer will trigger and you will see messages in the log:

Timer Trigger Working

Integrate into VS Code

You are free to use full Visual Studio or any editor to develop Function Apps in F#. I've been mostly using VS Code for this purpose, and I believe it's quite popular among F# community.

If you use VS Code, be sure to setup the tasks that you can use from within the editor. I usually have at least 3 tasks: "build" (dotnet build), "publish" (dotnet publish) and "run" (func start --script-root bin\\debug\\netstandard2.0\\publish), with shortcuts configured to all of them.

You can find an example of tasks.json file here.

Also, check out Azure Functions Extension.

Deploy to Azure

You can deploy the exact same application binaries to Azure. Start by creating an empty Function App in the portal, or via Azure CLI (func CLI does not support that).

Then run the following command to deploy your precompiled function to this app:

func azure functionapp publish <FunctionAppName>

At the first run, it will verify your Azure credentials.

In real-life production scenarios your workflow is probably going to be similar to this:

  • Change Function App code
  • Run it locally to test the change
  • Push the code changes to the source control repository
  • Have your CI/CD pipeline build it, run the tests and then push the binaries to Azure Functions environment

HTTP Trigger

Timer-triggered functions are useful, but that's just one limited use case. Several other event types can trigger Azure Functions, and for all of them you can create precompiled functions and run them locally.

The most ubiquotous trigger for any serverless app is probably HTTP. So, for the rest of the article I will focus on several approaches to implement HTTP functions. Nonetheless, the same techique can be applied to other triggers too.

F# code for the simplest HTTP Function can look like this:

namespace PrecompiledApp

open Microsoft.AspNetCore.Mvc
open Microsoft.AspNetCore.Http
open Microsoft.Azure.WebJobs.Host

module PrecompiledHttp =

  let run(req: HttpRequest, log: TraceWriter) =
    log.Info("F# HTTP trigger function processed a request.")
    ContentResult(Content = "HO HO HO Merry Christmas", ContentType = "text/html")

You can find a full example of HTTP Function App here.

This code is using ASP.NET Core classes for request and response. It's still just an F# function, so we need to bind it to a trigger in function.json:

{
  "bindings": [
    {
      "type": "httpTrigger",
      "methods": ["get"],
      "authLevel": "anonymous",
      "name": "req",
      "route": "hellosanta"
    }
  ],
  "scriptFile": "../bin/PrecompiledApp.dll",
  "entryPoint": "PrecompiledApp.PrecompiledHttp.run"
}

If you run the app, the function will be hosted at localhost

HTTP Trigger Working

And a request to http://localhost:7071/api/hellosanta will get responded with our "HO HO HO" message.

This function is of "Hello World" level, but the fact that it's inside a normal F# library gives you lots of power.

Let's see at some examples of how to use it.

Suave Function

What can we do to enhance developer experience? We can use our favourite F# libraries.

Suave is one of the most popular libraries to implement Web API's with. And we can use it in Azure Functions too!

Let's first make a small twist to HTTP trigger definition in function.json:

"bindings": [
  {
    "type": "httpTrigger",
    "methods": ["get"],
    "authLevel": "anonymous",
    "name": "req",
    "route": "{*anything}"
  }
],

Binding now defines a wildcard route to redirect all requests to this function. That's because we want Suave to take care of routing for us.

The definition of such routing will look familiar to all Suave users:

module App =
  open Suave
  open Suave.Successful
  open Suave.Operators
  open Suave.Filters

  let app = 
    GET >=> choose
      [ path "/api/what" >=> OK "Every time we love, every time we give, it's Christmas."
        path "/api/when" >=> OK "Christmas isn't a season. It's a feeling."
        path "/api/how" >=> OK "For it is in giving that we receive." ]

Azure Function is just a one-liner wiring Suave app into the pipeline:

module Http =
  open Suave.Azure.Functions.Context

  let run req =
    req |> runWebPart App.app  |> Async.StartAsTask

The heavy lifting is done by runWebPart function, which is a utility function defined in the same application. You can see the full code of this wiring in my repo.

Run the application and request the URL http://localhost:7071/api/what to see the function in action.

This example is very simple, but you can do lots of powerful stuff with Suave! Most probably, you shouldn't go over the root and try to fit whole mulpti-resource REST API into a single Azure Function. But it might still make sense to keep related HTTP calls together, and Suave can help to keep it cleaner.

Managing Dependencies with Paket

Once your Function App becomes bigger and you start using multiple F# projects, it makes sense to switch to Paket package manager.

It is totally possible to use Paket with Azure Functions. There isn't much specific to Azure Functions, really. Here is an example of paket.dependecies file

source https://www.nuget.org/api/v2

framework: >= netstandard2.0
nuget FSharp.Core
nuget Microsoft.NET.Sdk.Functions
nuget Microsoft.AspNetCore.Mvc.Core

that I used in example which demonstrates Paket + Functions combination.

Attribute-Based Functions

Up until now, we were writing function.json files manually for each function. This is not very tedious, but it is error prone. Microsoft offers an alternative programming model where these files are auto-generated by Functions SDK.

This programming model is based on attributes, which are similar to WebJobs SDK attributes. With this approach, there's no function.json file in the project. Instead, the function declaration is decorated with attributes:

[<FunctionName("AttributeBased")>]
let run([<HttpTrigger>] req: HttpRequest, log: TraceWriter)

The same development flow still works. Once you run dotnet build, a new function.json file will be generated and placed into bin folder. Functions runtime will be able to use it to run the function as usual.

Note that the generated file looks a bit different from the manual equivalent:

  1. It manifests itself with

     "generatedBy": "Microsoft.NET.Sdk.Functions.Generator-1.0.6",
     "configurationSource": "attributes",
    
  2. In case you use input and output bindings, you won't be able to see them in the generated file. Only trigger will be visible in json. Don't worry, input and output bindings will still work.

You can find an example of HTTP function with attributes here.

There are pro's and con's in this model. Obviously, not having to write JSON files manually is beneficial. Some people find the binding attributes really ugly though, especially when you have 3 or 4 bindings and each has multiple parameters.

My preference is to use attributes, but don't mix attribute decoration with real code. I.e. keep the Function's body to a simple 1-liner, and delegate the call to a properly defined F# function with the actual domain logic.

Wrapping Up

Lots of F# users value the language for how quickly one can be productive with it: based on concise syntax, powerful libraries and tools like FSI.

In my opinion, Azure Functions fit nicely into the picture. It takes just several minutes before you can run your first Function App on developer machine, and then seamlessly transfer it into the cloud.

I've prepared a github repository where you can find more Examples of Azure Functions implemented in F#.

Merry Serverless Functional Christmas!

Azure F#unctions Talk at FSharping Meetup in Prague

On November 8th 2017 I gave a talk about developing Azure Functions in F# at FSharping meetup in Prague.

I really enjoyed giving this talk: the audience was great and asked awesome questions. One more prove that F# community is so welcoming and energizing!

All the demos of that session can be found in my github repository.

The slides were only a small portion of my talk, but you can see them below anyways.

Link to full-screen HTML slides: Azure F#unctions

Slides on SlideShare:

Thanks for attending my talk! Feel free to post any feedback in the comments.

Azure Function Triggered by Azure Event Grid

Update: I missed the elephant in the room. There actually exists a specialized trigger for Event Grid binding. In the portal, just select Experimental in Scenario drop down while creating the function. In precompiled functions, reference Microsoft.Azure.WebJobs.Extensions.EventGrid NuGet package.

The rest of the article describes my original approach to trigger an Azure Function from Azure Event Grid with generic Web Hook trigger.

Here are the steps to follow:

Create a Function with Webhook Trigger

I'm not aware of a specialized trigger type for Event Grid, so I decided to use Generic Webhook trigger (which is essentially an HTTP trigger).

I used the Azure Portal to generate a function, so here is the function.json that I got:

{
  "bindings": [
    {
      "type": "httpTrigger",
      "direction": "in",
      "webHookType": "genericJson",
      "name": "req"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    }
  ],
  "disabled": false
}

For precompiled functions, just decorate it with HttpTriggerAttribute with POST method:

public static Task<HttpResponseMessage> Run(
    [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestMessage req)

Parse the Payload

Events from Event Grid will arrive in a specific predefined JSON format. Here is an example of events to expect:

[{
  "id": "0001",
  "eventType": "MyHelloWorld",
  "subject": "Hello World!",
  "eventTime": "2017-10-05T08:53:07",
  "data": {
    "hello": "world"
  },
  "topic": "/SUBSCRIPTIONS/GUID/RESOURCEGROUPS/NAME/PROVIDERS/MICROSOFT.EVENTGRID/TOPICS/MY-EVENTGRID-TOPIC1"
}]

To be able to parse those data more easily, I defined a C# class to deserialize JSON to:

public class GridEvent
{
    public string Id { get; set; }
    public string EventType { get; set; }
    public string Subject { get; set; }
    public DateTime EventTime { get; set; }
    public Dictionary<string, string> Data { get; set; }
    public string Topic { get; set; }
}

Now, the function can read the events (note, that they are sent in arrays) from the body of POST request:

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    string jsonContent = await req.Content.ReadAsStringAsync();
    var events = JsonConvert.DeserializeObject<GridEvent[]>(jsonContent);

    // do something with events

    return req.CreateResponse(HttpStatusCode.OK);
}

Validate the Endpoint

To prevent you from sending events to endpoints that you don't own, Event Grid requires each subsriber to validate itself. For this purpose, Event Grid will send events of the special type SubscriptionValidation.

The validation request will contain a code, which we need to echo back in 200-OK HTTP response.

Here is a small piece of code to do just that:

if (req.Headers.GetValues("Aeg-Event-Type").FirstOrDefault() == "SubscriptionValidation")
{
    var code = events[0].Data["validationCode"];
    return req.CreateResponse(HttpStatusCode.OK,
        new { validationResponse = code });
}

The function is ready!

Create a Custom Event Grid Topic

To test it out, go to the portal and create a custom Event Grid topic. Then click on Add Event Subscription button, give it a name and copy paste the function URL (including key) to Subscriber Endpoint field:

Azure Function URL

Event Grid Subscription

Creating a subscription will immediately trigger a validation request to your function, so you should see one invocation in the logs.

Send Custom Events

Now, go to your favorite HTTP client (curl, Postman, etc) and send a sample event to check how the whole setup works:

POST /api/events HTTP/1.1
Host: <your-eventgrid-topic>.westus2-1.eventgrid.azure.net
aeg-sas-key: <key>
Content-Type: application/json

[{
  "id": "001",
  "eventType": "MyHelloWorld",
  "subject": "Hello World!",
  "eventTime": "2017-10-05T08:53:07",
  "data": {
    "hello": "world"
  }
}]

Obviously, adjust the endpoint and key based on the data from the portal.

You should get a 200-OK back and then see your event in Azure Function invocation logs.

Have fun!

Wanted: Effectively-Once Processing in Azure

This experimental post is a question. The question is too broad for StackOverflow, so I'm posting it here. Please engage in the comments section, or forward the link to subject experts.

TL;DR: Are there any known patterns / tools / frameworks to provide scalable, stateful, effectively-once, end-to-end processing of messages, to be hosted in Azure, preferably on PaaS-level of service?

Motivational Example

Let's say we are making a TODO app. There is a constant flow of requests to create a TODO in the system. Each request contains just two fields: a title and a project ID which TODO should belong to. Here is the definition:

type TodoRequest = {
  ProjectId: int
  Title: string
}

Now, we want to process the request and assign each TODO an identifier, which should be an auto-incremented integer. Numeration is unique per project, so each TODO must have its own combination of ProjectId and Id:

type Todo = {
  ProjectId: int
  Id: int
  Title: string
}

Now, instead of relying on some database sequences, I want to describe this transformation as a function. The function has the type (TodoRequest, int) -> (Todo, int), i.e. it transforms a tuple of a request and current per-project state (last generated ID) to a tuple of a TODO and post-processing state:

let create (request: TodoRequest, state: int) =
  let nextId = state + 1
  let todo = {
    ProjectId = request.ProjectId
    Id = nextId
    Title = request.Title
  }
  todo, nextId

This is an extremely simple function, and I can use it to great success to process local, non-durable data.

But if I need to make a reliable distributed application out of it, I need to take care of lots of things:

  1. No request should be lost. I need to persist all the requests into a durable storage in case of processor crash.

  2. Similarly, I need to persist TODO's too. Presumably, some downstream logic will use the persisted data later on in TODO's lifecycle.

  3. The state (the counter) must be durable too. In case of crash of processing function, I want to be able to restart processing after recovery.

  4. Processing of the requests should be sequential per project ID. Otherwise I might get a clash of ID's in case two requests belonging to the same project are processed concurrently.

  5. I still want requests to different projects to be processed in parallel, to make sure the system scales up with the growth of project count.

  6. There must be no holes or duplicates in TODO numbering per project, even in face of system failures. In worst case, I agree to tolerate a duplicated entry in the output log, but it must be exactly the same entry (i.e. two entries with same project id, id and title).

  7. The system should tolerate a permanent failure of any single hardware dependency and automatically fail-over within reasonable time.

It's not feasible to meet all of those requirements without relying on some battle-tested distributed services or frameworks.

Which options do I know of?

Transactions

Traditionally, this kind of requirements were solved by using transactions in something like SQL Server. If I store requests, TODO's and current ID per project in the same relational database, I can make each processing step a single atomic transaction.

This addresses all the concerns, as long as we can stay inside the single database. That's probably a viable option for the TODO app, but less of so if I convert my toy example to some real applications like IoT data processing.

Can we do the same for distributed systems at scale?

Azure Event Hubs

Since I touched IoT space, the logical choice would be to store our entries in Azure Event Hubs. That works for many criteria, but I don't see any available approach to make such processing consistent in the face of failures.

When processing is done, we need to store 3 pieces: generated TODO event, current processing offset and current ID. Event goes to another event hub, processing offset is stored in Blob Storage and ID can be saved to something like Table Storage.

But there's no way to store those 3 pieces atomically. Whichever order we choose, we are bound to get anomalies in some specific failure modes.

Azure Functions

Azure Functions don't solve those problems. But I want to mention this Function-as-a-Service offering because they provide an ideal programming model for my use case.

I need to take just one step from my domain function to Azure Function: to define bindings for e.g. Event Hubs and Table Storage.

However, reliability guarantees will stay poor. I won't get neither sequential processing per Event Hub partition key, nor atomic state commit.

Azure Service Fabric

Service Fabric sounds like a good candidate service for reliable processing. Unfortunately, I don't have much experience with it to judge.

Please leave a comment if you do.

JVM World

There are products in JVM world which claim to solve my problem perfectly.

Apache Kafka was the inspiration for Event Hubs log-based messaging. The recent Kafka release provides effectively-once processing semantics as long as data stay inside Kafka. Kafka does that with atomic publishing to multiple topics, and state storage based on compacted topics.

Apache Flink has similar guarantees for its stream processing APIs.

Great, but how do I get such awesomeness in .NET code, and without installing expensive ZooKeeper-managed clusters?

Call for Feedback

Do you know a solution, product or service?

Have you developed effectively-once processing on .NET / Azure stack?

Are you in touch with somebody who works on such framework?

Please leave a comment, or ping me on Twitter.

Mikhail Shilkov I'm Mikhail Shilkov, a software developer. I enjoy F#, C#, Javascript and SQL development, reasoning about distributed systems, data processing pipelines, cloud and web apps. I blog about my experience on this website.

LinkedIn@mikhailshilkovGitHubStack Overflow