docker, linux, windows comments edit

Working in an environment of mixed containers - both Windows and Linux - I wanted to run them all on my dev machine at the same time if possible. I found some instructions from a while ago about this but following them didn’t work.

Turns out some things have changed in the Docker world so here are some updated instructions.

As of this writing, I’m on Docker Desktop for Windows (31259) Community Edition. The instructions here work for that; I can’t guarantee more won’t change between now and whenever you read this.

  1. Clean up existing containers before switching to Windows containers. Look to see if you’re using Windows containers. Right-click on the Docker icon in the system tray. If you see “Switch to Windows containers…” then you are not currently using Windows containers. Stop any running containers you have and remove all images. You’ll need to switch to Windows containers and the image storage mechanism will change. You won’t be able to manage the images once you switch.

  2. Switch to Windows Containers. Right-click on the Docker icon in the system tray and select “Switch to Windows containers…” If you’re already using Windows containers, great!

  3. Enable experimental features. Right-click on the Docker icon in the system tray and select “Settings.” Go to the “Daemon” tab and check the box marked “Experimental features.”

Enable experimental features.

That’s it! You’re ready to run side-by-side containers.

The big key is to specify --platform as linux or windows when you run a container.

Open up a couple of PowerShell prompts.

In one of them, run docker images just to make sure things are working. The list of images will probably be empty if you had to switch to Windows containers. If you were already on Windows containers, you might see some.

In that same PowerShell window, run:

docker run --rm -it --platform windows microsoft/nanoserver:1803

This is a command-prompt-only version of Windows Server. You should get a C:\> prompt once things have started up.

Leave that there, and in the other PowerShell window, run:

docker run --rm -it --platform linux ubuntu

This will get you to an Ubuntu shell.

See what you have there? Windows and Linux running side by side!

Windows and Linux containers - side by side!

Type exit in each of these containers to get out of the shell and have the container clean itself up.

Again, the big key is to specify --platform as linux or windows when you run a container.

If you forget to specify the --platform flag, it will default to Windows unless you’ve already downloaded the container image. Once you have used the image, the correct version will be found and used automatically:

# Works because you already used the image once.
docker run --rm -it ubuntu

If you try to run a Linux container you haven’t already used, you may get a message like this:

no matching manifest for windows/amd64 10.0.18362 in the manifest list entries

I’m not sure on the particulars on why sometimes --platform is required and sometimes it’s not. Even after removing all my container images, I was able to run an Ubuntu container without specifying platform, like some cache was updated to indicate which platform should be used by default. YMMV. It doesn’t hurt to include it, however, if you try to use --platform on another machine it may not work - you can only use it when experimental features are enabled.

UPDATE June 14, 2019

I’ve found since working in this mixed environment that there are things that don’t work as one might entirely expect.

  • Networking: With Linux-only containers on Windows you get a DockerNAT virtual network switch you can tweak if needed to adjust network connectivity. Under mixed containers, you use the Windows Container network switch, nat and you really can’t do too much with that. I’ve had to reset my network a few times while trying to troubleshoot things like network connections whilst on VPN.
  • Building container images that reference files from other images: A standard .NET Core build-in-container situation is to create, in one Dockerfile, two container images - the first builds and publishes the app, the second copies the published app into a clean, minimal image. When in mixed container world, I get a lot of errors like, “COPY failed: file does not exist.” I can look in the intermediate containers and the files are all there, so there’s something about being unable to mount the filesystem from one container to copy into the other container.

Unrelated to mixed containers, it seems I can’t get any container to connect to the internet when I’m on my company’s VPN. VPN seems to be a pretty common problem with Docker on Windows. I haven’t solved that.

It appears there’s still a lot to work out here in mixed container land. You’ve been warned.

git comments edit

Are you stuck unable to update your version of Atlassian Sourcetree for Windows because when you update and restart, Sourcetree hangs?

I was stuck on 3.0.17. Every time I updated (to 3.1.2), Sourcetree would restart and then… hang. Unresponsive. Unable to see if there were any new updates, unable to do anything but kill the app.

It turns out the reason for this is that Sourcetree didn’t handle monitor scaling very well. Say you have a 4K monitor set to scale to 150% - that’s when you’d see the hang.

There are two workarounds for this:

The first option is to stop monitor scaling and switch back to 100%. Not the greatest, I know, but that’ll get you through temporarily… and it only needs to be temporary. (I’ll get there.)

The other option is to do a tweak to the Sourcetree configuration file. First, make sure Sourcetree is closed. Now go to %LocalAppData%\SourceTree\app-version like C:\Users\tillig\AppData\Local\SourceTree\app-3.1.2. Open the file SourceTree.exe.config. Find this line:

<AppContextSwitchOverrides value="Switch.System.Windows.DoNotScaleForDpiChanges=false"/>

Update it to look like this:

<AppContextSwitchOverrides value="Switch.System.Windows.DoNotScaleForDpiChanges=false;Switch.System.Windows.Controls.Grid.StarDefinitionsCanExceedAvailableSpace=true"/>

When you start Sourcetree, it should be responsive.

This is the default setting in 3.1.3. If you can get yourself upgraded to 3.1.3, you won’t have to do any workarounds anymore. So if you temporarily set your monitor to 100%, take the upgrades in SourceTree up to 3.1.3 or later, then you can switch your monitor back. (Or, of course, you can tweak the configuration on your hanging version of Sourcetree until you get to 3.1.3 or later.)

I had to upgrade from 3.0.17 to 3.1.2 and then to 3.1.3. For some reason I couldn’t just go straight from 3.0.17 to 3.1.3. YMMV.

net, build, gists comments edit

I’ve been enjoying the addition of dotnet CLI global tools and figured I’d blog the ones I use. I’ll also include a PowerShell script that is used to install them (or, if they’re installed, update them).

The list is current as of this writing, but you can visit the gist (below) to see the set of tools I’m using at any given time.

  • dotnet-depends: Shows the dependency tree for a .NET project.
  • dotnet-outdated: Shows outdated packages referenced by a .NET project.
  • dotnet-format: Formats code based on EditorConfig settings.
  • dotnet-script: Run C# script (.csx) files.
  • dotnet-symbol: Download symbols from the command prompt. [I’m still trying this one out to see if I use it much or like it.]
  • dotnetsdkhelpers: A global tool version of the original SDK installer helpers that addresses the need for external tools and fixes a couple of bugs with the original.
  • gti: Install plugins from a .gti file/manifest. [I’m still trying this one out to see if I like it. If it’s good, I can replace my PowerShell script with a manifest.]
  • microsoft.web.librarymanager.cli: dotnet CLI access to the libman dependency manager for JS.

Here’s the gist with the PowerShell script that I use to install these:

azure, build, net, tfs, vs comments edit

Azure DevOps has the ability to publish things to a private NuGet feed as part of its artifacts handling.

Working with a private feed from a developer machine running builds from the command line or Visual Studio is pretty easy. There is documentation on using a NuGet credential provider to authenticate with Azure DevOps and make that seamless.

However, getting this to work from a pipeline build is challenging. Once you’ve published a package, you may want to consume it from something else you’re building… but the feed is secured. What do you do?

I’m told there are improvements for this coming in Q2 2019. I can’t quantify what those improvements are, but it may mean things start to “just work.” Until then, here are ways to work around the challenges.

Option 1: Separate Restore from Build

The documentation shows how to use NuGet or the dotnet CLI for package restore from your feed. Both of the solutions effectively amount to separating the call to NuGet restore or dotnet restore from the rest of your build.

For NuGet, you’d use a NuGet build step (NuGetCommand@2) and specify the restore. Do that before you execute the build on your solution.

For the dotnet CLI, you’d use a dotnet CLI build step (DotNetCoreCLI@2) and indicate the restore command.

In both cases, the special build command will generate a NuGet.Config file on the fly that contains the system access token. The restore operation will use that custom temporary config during the restore and it will succeed.

However, if you later try running dotnet build or dotnet publish it’ll fail - because there’s an implicit restore that runs during those. These will not have the system credentials in place. You have to use --no-restore on builds, for example, to avoid the auto-restore. It can get painful in a larger build.

If you have a build script, like a bash or PowerShell script, manually executing dotnet restore in that script will also not work. You must use the build tasks to get the magic to happen.

Option 2: Use the Azure Artifacts Credential Provider

Another option in the docs is that you can use the Azure Artifacts credential provider. While it seems this is primarily geared toward running on build agents you host yourself, you can possibly get this working on hosted agents.

I have not tried this myself. I went with option 3, below. However, if you want to give it a shot, here’s what you’d do.

First, you’ll want to be aware of how NuGet credential providers work. I don’t mean the internals, but, like, where you need to put the credential provider executable to make it work and how to troubleshoot it. All of that is documented.


  • Download the latest release of the credential provider. Make sure you get the version that will run on your build agent, not your development machine.
  • Follow the instructions in the readme to find the self-contained executable version of the credential provider in the archive you just downloaded.
  • Extract the credential provider to somewhere in the source you’ll be building. Maybe a separate build folder.
  • As part of your build pipeline, you’ll need to…
    • Set the NUGET_CREDENTIALPROVIDERS_PATH to point to the build folder in your checked-out source that contains the provider.
    • On Linux, you may need to chmod +x that provider.
    • Set the VSS_NUGET_EXTERNAL_FEED_ENDPOINTS to indicate the location of your external NuGet feed and provide the system access token. It takes a JSON format: {"endpointCredentials": [{"endpoint":"http://example.index.json", "username":"vsts", "password":"$(System.AccessToken)"}]}

In the VSS_NUGET_EXTERNAL_FEED_ENDPOINTS you’ll notice the use of the $(System.AccessToken) variable. That’s a predefined system variable in Azure DevOps build pipelines. You’ll see that again later.

Anyway, if all the planets have aligned, when you run your standard dotnet restore or NuGet restore, it will use the credential provider for authentication. The credential provider will use the environment variable and magic will happen.

One other note there - the username vsts isn’t special. It can be any value you want, the endpoint doesn’t actually end up checking. It just can’t be omitted.

Option 3: Update NuGet.Config

The final option is to update your NuGet.Config on the fly with the system access token as part of the build. I went with this option because it was simpler and had fewer moving pieces.

In your source you likely already have a NuGet.Config file that has both the standard feed and your private Azure DevOps feed in it. It should look something like this:

<?xml version="1.0" encoding="utf-8"?>
    <add key="disableSourceControlIntegration" value="true" />
    <clear />
    <add key="NuGet Official" value="" protocolVersion="3" />
    <add key="Azure DevOps" value="" protocolVersion="3" />

The name, Azure DevOps, is the key here. Doesn’t matter what you name it, just make sure you remember it. You’ll need it.

In your build pipeline, before you do any operations to build or restore packages, Use the NuGetCommand@2 task and run a custom command to update the source in that NuGet.Config to have the system credentials attached.

- task: NuGetCommand@2
  displayName: 'Authenticate with Azure DevOps NuGet'
    command: custom
    arguments: sources update -Name "Azure DevOps" -Username "vsts" -Password "$(System.AccessToken)" -StorePasswordInClearText -ConfigFile ./NuGet.Config

What this will do is add the credentials right into the XML of the NuGet.Config as checked out in your source. The NuGet or dotnet commands will already be using that config file to locate your feed, the creds will come along for free.

Items to note:

  • Again, you see that $(System.AccessToken) show up. That’s the magic.
  • If you do this, you need to avoid using DotNetCLI and NuGetCommand tasks that might try to authenticate automatically behind the scenes. The cleartext credentials in the configuration conflict with the credential provider auto-authenticate mechanism and things blow up. This likely means the build steps in your pipeline need to become PowerShell, bash, or MSBuild scripts that do the restore, build, test, publish, etc.
  • You need to have -StorePasswordInClearText or the dotnet CLI won’t be able to use the credentials. If you’re only using NuGet commands, you should be OK not storing in clear text.
  • If you’re on a Linux agent, don’t forget filenames are case-sensitive. If you get an error, make sure you got all the capitalization right for your config file.
  • The source name is case-sensitive. If NuGet.Config has a source named Azure DevOps then the authentication step with NuGet needs to specify the source name as Azure DevOps, too - azure devops won’t work.

azure, kubernetes, build comments edit

Spinnaker is a great tool for continuous deployment of services into Kubernetes. It has a lot of first-class integration support for things like Docker containers and Google Cloud Storage blobs.

For example, say you have a Docker container that gets deployed to Kubernetes via a Helm-chart-style manifest. You might store the manifest in Google Cloud Storage and the Docker container in Google Cloud Registry. Change one of those things, it triggers a pipeline to deploy.

This is not so easy if you are in Azure and storing your manifests in Azure Blob Storage. You can make this work, but it takes a lot of moving pieces. I’ll show you how to get this done.

“Why Don’t You Just…?”

Before we get into the “how,” let’s talk about “why.”

You might wonder why I’m interested in doing any of this when there are things like Azure DevOps that can do deployment and Azure Helm Repository to store Helm charts. Just sidestep the issue, right?

Azure DevOps is great for builds but doesn’t have all the deployment features of Spinnaker. In particular, if you want to use things like Automated Canary Analysis, Azure DevOps isn’t going to deliver.

Further, Azure DevOps is kind of a walled garden. It works great if you keep everything within Azure DevOps. If you want to integrate with other tools, like have artifacts stored somewhere so another tool can grab it, that’s hard. Azure DevOps can publish artifacts to various repo types (NuGet, npm, etc.) but really has no concept of artifacts that don’t get a semantic version (e.g., stuff that would happen in a continuous delivery environment) and aren’t one of the supported artifact types. Universal packages? They need semantic versions.

So what if you have a zip file full of stuff that just needs to be pushed? If Azure DevOps is handling it in a pipeline, you can download build artifacts via a pipeline task… but from an external tool standpoint, you have to use the REST API to get the list of artifacts from the build and then iterate through those to get the respective download URL. All of these calls need to be authenticated but at least it supports HTTP Basic auth with personal access tokens. Point being, there’s no simple webhook.

Azure Helm Repository is currently in beta and has no webhook event that can inform an external tool when something gets published. That may be coming, but it’s not there today. It also doesn’t address other artifact types you might want to handle, like that arbitrary zip file.

How It Works

Now you know why, let me explain how this whole thing will work. There are a lot of moving pieces to align. I’ll explain details in the respective sections, but so you get an overview of where we’re going:

  • Publishing a new/updated blob to Azure Blob Storage will raise an event with Azure EventGrid.
  • A subscription to Azure EventGrid for “blob created” events will route the EventGrid webhooks to an Azure API Management endpoint.
  • The Azure API Management endpoint will transform the EventGrid webhook contents into a Spinnaker webhook format and forward the transformed data to Spinnaker.
  • Spinnaker will use HTTP artifacts to download the Azure Blob.
  • The Spinnaker download request will be handled by a small Node.js proxy that converts HTTP Basic authentication to Azure Blob Storage “shared key” authentication.

Event and retrieval flow for Azure Blob Storage and Spinnaker

Most of this is required because of the security around EventGrid and Azure Blob Storage. There’s not a straightforward way to just use a series of GET and POST requests with HTTP Basic auth.

Create the Azure Storage Account and Blob Container

This is where you will store your artifact blobs.

az storage account create \
  --name myartifacts \
  --resource-group myrg \
  --access-tier StandardLRS \
  --https-only true \
  --kind StorageV2 \
  --location westus

az storage container create \
  --name artifacts \
  --account-name myartifacts

Your blobs will get downloaded via URLs like

Create a Basic Auth Proxy for Blob Storage

If you want to download from Azure Blob Storage without making the artifacts publicly accessible, you need to use SharedKey authentication.. It looks a lot like HTTP Basic but instead of using a username/password combination, it’s a username/signature. The signature is generated by concatenating several headers and generating a signature using your shared key (available through the Azure portal).

It’s complicated and not something built into Spinnaker. If you really want to see how to generate the signature and all that, the docs have a very detailed explanation.

The short version here is that you probably want a client SDK library to do this work. Since Spinnaker really only supports HTTP Basic anyway, you need to take in username:password in some form and convert that into the request Blob Storage expects.

Richard Astbury has a great blog article showing a simple web server that takes in HTTP Basic auth where your storage account name is the username and the storage account key is the password, then proxies that through to Blob Storage for download.

Deploy this proxy somewhere you can access like an Azure App Service.

I converted this to be an Express app without too much trouble, which adds some logging and diagnostics to help with troubleshooting.

Things to keep in mind when you deploy this:

  • Run in HTTPS only mode using a minimum TLS 1.2. These are options in the Azure App Service settings. You’ll be using HTTP Basic auth which isn’t encrypted, so ensuring the communication itself is encrypted is important.
  • Consider adding more layers of security to ensure random people can’t just use your proxy. Azure App Service apps are public, so consider updating the app to only respond to requests to a special endpoint URL that contains a key only you know. You could also ensure you only respond to requests for known storage account info, not just ay info. That would mean folks need the special endpoint URL and the storage account info to get anything.
  • Ensure it’s “always on.” Azure App Service apps go to sleep for a while when they’re idle. The first request can time out as it wakes up, and if Spinnaker is the first request it’ll cause a pipeline failure.

For this walkthrough, we’ll say your Blob Storage proxy app is at You’ll see this show up later as we set up the rest of the pieces. When Spinnaker wants to download an HTTP artifact, it’ll access instead of accessing the Blob Storage URL directly.

You should now be able to access an Azure Blob Storage blob via your proxy using HTTP Basic authentication. Verify this works by dropping a blob into your artifact storage and downloading it through the proxy.

Enable Spinnaker Webhook Artifacts

Spinnaker needs to know that it can expect artifacts to come in via webhooks. To do that, you need to enable the artifacts. Edit the .hal/default/profiles/echo-local.yml file to enable it.

    enabled: true

Enable Spinnaker HTTP Artifacts

The webhook will tell Spinnaker where the artifact lives, but you have to also enable Spinnaker to download the artifact. You also have to provide it with credentials to do so. There is good doc explaining all this but here’s a set of commands to help you.

hal config features edit --artifacts true
hal config artifact http enable
PASSWORD=[big long shared key from Azure Portal]
echo ${USERNAME}:${PASSWORD} > upfile.txt
hal config artifact http account add myartifacts \
  --username-password-file upfile.txt

# You're stuck with that upfile.txt in the current folder until you
# do a backup/restore of the config. First deploy it to get the changes
# out there...
hal deploy apply

# Now back up the config.
hal backup create

# Immediately restore the backup you just made.
hal backup restore --backup-path halbackup-Tue_Dec_18_23-00-10_GMT_2018.tar

# Verify the local upfile.txt isn't being used anymore.
hal config artifact http account get myartifacts

# Assuming you don't see a reference to the local upfile.txt...
rm upfile.txt

Spinnaker now knows not only that webhooks will be supplying artifact locations, but also how to provide HTTP Basic credentials to authenticate with the artifact source.

At the time of this writing I don’t know how Spinnaker handles multiple sets of credentials. For example, if you already have some HTTP Basic artifact credentials set up and add this new set, how will it know to use these new creds instead of the ones you already had? I don’t know. If you know, chime in on this GitHub issue!

You should now have enough set up to enable Spinnaker to download from Azure Blob Storage. If an artifact URL like comes in as part of an HTTP File artifact, Spinnaker can use the supplied credentials to download it through the proxy.

Create an API Management Service for EventGrid Handling

Azure API Management Service is usually used as an API gateway to allow you to expose APIs, secure them, and deal with things like licensing and throttling.

We’re going to take advantage of the API Management policy mechanism to handle Azure EventGrid webhook notifications. This part is based on this awesome blog article from David Barkol. Using API Management policies enables us to skip having to put another Node.js proxy out there.

Why not use API Management for the Blob Storage proxy? I wasn’t sure how to get the Azure Blob Storage SDK libraries into the policy and didn’t want to try to recreate the signature generation myself. Lazy? Probably.

The API Management Service policies will handle two things for us:

  1. Handshaking with EventGrid for webhook subscriptions. EventGrid doesn’t just let you send webhooks wherever you want. To set up the subscription, the receiver has to perform a validation handshake to acknowledge that it wants to receive events. You can’t get Spinnaker to do this, and it’s nigh impossible to get the manual validation information.
  2. Transforming EventGrid schema to Spinnaker Artifact schema. The data supplied in the EventGrid webhook payload isn’t something Spinnaker understands. We can’t even use Jinja templates to transform it at the Spinnaker side because Spinnaker doesn’t want an inbound webhook to be an array - it wants the webhook to be an object that contains the array in a property called artifacts. We can do the transformation work easily in the API Management policy.

First, create your API Management service.

Create API Management Service

This can take a long time to provision, so give it a bit. When it’s done, your API will be at a location based on the API Management Service name, like All the APIs you hang off this management service will be children of this URL.

Once it’s provisioned, go to the “APIs” section. By default, you’ll get a sample “Echo” API. Delete that.

Now, add a new “Blank API.” Call it “EventGrid Proxy” and set the name to eventgrid-proxy. For the URL scheme, be sure to only select HTTPS. Under “Products,” select “Unlimited” so your developer key for unlimited API calls will be in effect.

Create a blank API

In the EventGrid Proxy API, create a new operation called webhook that takes POST requests to /webhook.

Create webhook operation

The webhook operation doesn’t do anything except give us a valid endpoint to which EventGrid can post its webhook notifications.

On the EventGrid Proxy api, select “Design” and then click the little XML angle brackets under “Inbound Processing.” This will get you into the area where you can set up and manage API policies using XML.

Go to XML inbound policy management

In the inbound policy, paste this XML beauty and update the noted locations with your endpoints:

        <set-variable value="@(!context.Request.Headers.ContainsKey("Aeg-Event-Type"))" name="noEventType" />
            <when condition="@(context.Variables.GetValueOrDefault<bool>("noEventType"))">
                    <set-status code="404" reason="Not Found" />
                    <set-body>No Aeg-Event-Type header found.</set-body>
                <set-variable value="@(context.Request.Headers["Aeg-Event-Type"].Contains("SubscriptionValidation"))" name="isEventGridSubscriptionValidation" />
                <set-variable value="@(context.Request.Headers["Aeg-Event-Type"].Contains("Notification"))" name="isEventGridNotification" />
                    <when condition="@(context.Variables.GetValueOrDefault<bool>("isEventGridSubscriptionValidation"))">
                            <set-status code="200" reason="OK" />
                                var events = context.Request.Body.As<string>();
                                JArray a = JArray.Parse(events);
                                var data = a.First["data"];
                                var validationCode = data["validationCode"];
                                var jOutput =
                                    new JObject(
                                        new JProperty("validationResponse", validationCode)
                                return jOutput.ToString();
                    <when condition="@(context.Variables.GetValueOrDefault<bool>("isEventGridNotification"))">
                        <send-one-way-request mode="new">
                            <!-- Replace this with YOUR Spinnaker location! -->
                            <set-header name="Content-Type" exists-action="override">
                                var original = JArray.Parse(context.Request.Body.As<string>());
                                var proxied = new JArray();
                                foreach(var e in original)
                                    var name = e["subject"];
                                    // Replace this with YOUR Blob and proxy info!
                                    var reference = e["data"]
                                    var transformed = new JObject(
                                        new JProperty("type", "http/file"),
                                        new JProperty("name", name),
                                        new JProperty("reference", reference));

                                var wrapper = new JObject(new JProperty("artifacts", proxied));
                                var response = wrapper.ToString();
                                return response;
                            <set-status code="200" reason="OK" />
                            <set-status code="404" reason="Not Found" />
                            <set-body>Aeg-Event-Type header indicates unsupported event.</set-body>
        <base />
        <base />
        <base />

Again, update the URL endpoints as needed in there!

  • The location of your Spinnaker API (deck) endpoint for the webhook. This is always at /webhooks/webhook/sourcename where I’ve used azureblob for the source but you could use whatever you want. The source name is used in the pipeline configuration for webhooks.
  • The location of your Blob Storage and Blob Storage Proxy. These got set up earlier - in the Replace call up there, you need to swap your Blob Storage server location for the location of your proxy so when Spinnaker tries to download, it’ll go through your proxy.

I’m not going to dive too deep into API policy management. I recommend reading the official docs and being aware of what is available. However, I will explain what this policy does and how it works.

Whenever something comes from EventGrid it includes an Aeg-Event-Type header. The two header values we’re concerned with are SubscriptionValidation and Notification.

If there’s no header, return a 404. Whatever the request wants, we can’t support it.

If the header is SubscriptionValidation, EventGrid wants a handshake response to acknowledge we expect the subscription and want to handle it. We grab a validation code out of the inbound request and return a response that includes the code as an acknowledgement.

If the header is Notification, that’s the actual meat of the subscription coming through. The data will be coming through in EventGrid schema format and will look something like this when it’s a “blob created” event:

    "topic": "/subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/xstoretestaccount",
    "subject": "/blobServices/default/containers/container/blobs/filename",
    "eventType": "Microsoft.Storage.BlobCreated",
    "eventTime": "2017-06-26T18:41:00.9584103Z",
    "id": "831e1650-001e-001b-66ab-eeb76e069631",
    "data": {
      "api": "PutBlockList",
      "clientRequestId": "6d79dbfb-0e37-4fc4-981f-442c9ca65760",
      "requestId": "831e1650-001e-001b-66ab-eeb76e000000",
      "eTag": "0x8D4BCC2E4835CD0",
      "contentType": "application/octet-stream",
      "contentLength": 524288,
      "blobType": "BlockBlob",
      "url": "",
      "sequencer": "00000000000004420000000000028963",
      "storageDiagnostics": {
        "batchId": "b68529f3-68cd-4744-baa4-3c0498ec19f0"
    "dataVersion": "",
    "metadataVersion": "1"

We need to transform that into a Spinnaker webhook format that includes artifact definitions. The artifacts coming from Azure Blob Storage will be HTTP File artifacts. It should like this:

  "artifacts": [
      "type": "http/file",
      "reference": "",
      "name": "/blobServices/default/containers/container/blobs/filename"
  • The reference is the download URL location through your proxy to the blob.
  • The name is the subject of the EventGrid event.

The policy does this transformation and then forwards the EventGrid event on to your Spinnaker webhook endpoint.

You should now be able to POST events to Spinnaker via API Management. Using the example EventGrid body above or the one on the Microsoft docs site, make modifications to it so it looks like it’s coming from your blob store - update the locations, container name, and filename as needed. Use the “Test” mechanism inside Azure API Management to POST to your /webhook endpoint and see if it gets forwarded to Spinnaker. Make sure you add an Aeg-Event-Type header to the request or you’ll get the 404. Inside the test area you can see the trace of execution so if it fails you should see exception information and be able to troubleshoot.

Subscribe to EventGrid Blob Created Events

EventGrid needs to send events to your new API endpoint so they can be proxied over to Spinnaker.

In the Azure Portal, in the API Management section, take note of your API key for “Unlimited” - you associated this key with the API earlier when you created it. You’ll need it for creating the subscription.

Get API Management key for Unlimited subscription

When you access your /webhook API endpoint, you’ll pass that key as a query string parameter:

It’s easier to create subscriptions in the Azure Portal because the UI guides you through the right combination of parameters, but if you want to do it via CLI…

az eventgrid event-subscription create \
  --name spinnaker \
  --endpoint-type webhook \
  --included-event-types Microsoft.Storage.BlobCreated \
  --resource-group myrg \
  --resource-id /subscriptions/GUIDHERE/resourceGroups/myrg/providers/Microsoft.Storage/StorageAccounts/myartifacts \

You only want the Microsoft.Storage.BlobCreated event here. If you create or overwrite a blob, the event will be raised. You don’t want the webhook to fire if you delete something.

This may take a few seconds. In the background, EventGrid is going to go do the handshake with API Management and the policy you put in place will respond. This is easier to watch in the Azure Portal UI.

You now have Azure Blob Storage effectively raising Spinnaker formatted webhooks. Azure Blob Storage causes EventGrid to raise a webhook event, API Management policy intercepts that and transforms it, then it gets forwarded on to Spinnaker. You can trigger pipelines based on this and get artifacts into the pipeline.

Test the Full Circle

Here’s what I did to verify it all worked:

Create a simple Helm chart. Just a Kubernetes Deployment, no extra services or anything. You won’t actually deploy it, you just need to see that some sort of retrieval is happening. Let’s say you called it test-manifest so after packaging you have test-manifest-1.0.0.tgz.

Create a simple pipeline in Spinnaker. Add an expected artifact of type http/file. Specify a regex for “Name” that will match the name property coming in from your webhook. Also specify a “Reference” regex that will match reference.

In the “Automated Triggers” section, add a webhook trigger. Give it a source name that lines up with what you put in the API Management policy. In the example, we used azureblob. Also add an artifact constraint that links to the expected artifact you added earlier. This ensures your pipeline only kicks off if it gets the artifact it expects from the webhook payload.

Set up a webhook pipeline trigger with artifact constraints

In the pipeline add a “Bake (Manifest)” step. For the template artifact, select the .tgz artifact that the webhook will be providing. Fill in the name and namespace with whatever you want. Add an override key that will be used in the template process.

Simple/test manifest bake step

Now use the Azure Portal to upload your test-manifest-1.0.0.tgz “artifact” into your Azure Blob Storage container. This should cause the webhook to be raised, eventually making it to Spinnaker. Spinnaker should see the artifact, kick off the pipeline, and be able to download the Helm chart from Blob Storage so it can run the Helm templating step.

On the finished pipeline, you can click the “Source” link in the Spinnaker UI to see a big JSON object that will contain a huge base-64 encoded artifact that is the output of the “Bake (Manifest)” step. If you decode that, you should see your simple Helm chart, templated and ready to go.


What do you look at if it’s not working? There are a lot of moving pieces.

  • Verify the credentials you configured in Spinnaker. The username should be the blob storage account name and the password should be the shared key you got out of the Azure Portal. Try doing a download through your HTTP Basic proxy using those credentials and verify that still works.
  • Run a test in API Management. The “Test” section of API Management lets you simulate an event coming in from EventGrid. Use the “Trace” section in there to see what’s going on and what the response is. Did the transformation fail?
  • Look at Spinnaker “echo” and “deck” pod logs. I’m a big fan of the stern tool from Wercker for tailing logs of pods. It’s easy to watch logs in real time for doing something like: stern 'spin-(echo|deck).*' -n spinnaker -s 5s Make a test request from API Management and watch the logs. Do you see any errors fly by?
  • Ensure you haven’t changed any keys/names. The Azure Blob Storage key, the API Management key, the source name in the Spinnaker webhook… all these names and keys need to line up. If you rotated keys, you need to update the appropriate credentials/subscriptions.
  • Is something timing out? If you don’t have the “Always On” feature for your Azure Blob Storage proxy, it will go to sleep. The first request will cause it to wake up, but it may not be fast enough for Spinnaker and the request to download may time out.

Next Steps

By now you should have things basically working. Things you could do to make it better…

  • Test with large files. I didn’t test the Azure Blob Storage proxy thing with anything larger than a few hundred kilobytes. If you’re going to need support for gigabytes, you should test that.
  • Add security around the Spinnaker webhook. The Spinnaker webhook endpoint doesn’t have any security around it. There’s no API key required to access it. You could make the “source” for the webhook include some random key so no hook would be raised without matching that source. If you have to rotate it that might be a challenge. This also might be something to file as an enhancement on the Spinnaker repo.
  • Add security around the Azure Blob Storage proxy. You definitely need to have your basic creds in place for access here, and you need valid credentials for a storage account… but (at least in the Astbury blog article) there’s no limit on which storage accounts are allowed. You might want a whitelist here to ensure folks aren’t using your proxy to access things you don’t own.