Spinnaker is a great tool for continuous deployment of services into Kubernetes. It has a lot of first-class integration support for things like Docker containers and Google Cloud Storage blobs.

For example, say you have a Docker container that gets deployed to Kubernetes via a Helm-chart-style manifest. You might store the manifest in Google Cloud Storage and the Docker container in Google Cloud Registry. Change one of those things, it triggers a pipeline to deploy.

This is not so easy if you are in Azure and storing your manifests in Azure Blob Storage. You can make this work, but it takes a lot of moving pieces. I’ll show you how to get this done.

“Why Don’t You Just…?”

Before we get into the “how,” let’s talk about “why.”

You might wonder why I’m interested in doing any of this when there are things like Azure DevOps that can do deployment and Azure Helm Repository to store Helm charts. Just sidestep the issue, right?

Azure DevOps is great for builds but doesn’t have all the deployment features of Spinnaker. In particular, if you want to use things like Automated Canary Analysis, Azure DevOps isn’t going to deliver.

Further, Azure DevOps is kind of a walled garden. It works great if you keep everything within Azure DevOps. If you want to integrate with other tools, like have artifacts stored somewhere so another tool can grab it, that’s hard. Azure DevOps can publish artifacts to various repo types (NuGet, npm, etc.) but really has no concept of artifacts that don’t get a semantic version (e.g., stuff that would happen in a continuous delivery environment) and aren’t one of the supported artifact types. Universal packages? They need semantic versions.

So what if you have a zip file full of stuff that just needs to be pushed? If Azure DevOps is handling it in a pipeline, you can download build artifacts via a pipeline task… but from an external tool standpoint, you have to use the REST API to get the list of artifacts from the build and then iterate through those to get the respective download URL. All of these calls need to be authenticated but at least it supports HTTP Basic auth with personal access tokens. Point being, there’s no simple webhook.

Azure Helm Repository is currently in beta and has no webhook event that can inform an external tool when something gets published. That may be coming, but it’s not there today. It also doesn’t address other artifact types you might want to handle, like that arbitrary zip file.

How It Works

Now you know why, let me explain how this whole thing will work. There are a lot of moving pieces to align. I’ll explain details in the respective sections, but so you get an overview of where we’re going:

• Publishing a new/updated blob to Azure Blob Storage will raise an event with Azure EventGrid.
• A subscription to Azure EventGrid for “blob created” events will route the EventGrid webhooks to an Azure API Management endpoint.
• The Azure API Management endpoint will transform the EventGrid webhook contents into a Spinnaker webhook format and forward the transformed data to Spinnaker.
• Spinnaker will use HTTP artifacts to download the Azure Blob.
• The Spinnaker download request will be handled by a small Node.js proxy that converts HTTP Basic authentication to Azure Blob Storage “shared key” authentication.

Most of this is required because of the security around EventGrid and Azure Blob Storage. There’s not a straightforward way to just use a series of GET and POST requests with HTTP Basic auth.

Create the Azure Storage Account and Blob Container

This is where you will store your artifact blobs.

az storage account create \
  --name myartifacts \
  --resource-group myrg \
  --access-tier StandardLRS \
  --https-only true \
  --kind StorageV2 \
  --location westus

az storage container create \
  --name artifacts \
  --account-name myartifacts

Your blobs will get downloaded via URLs like https://myartifacts.blob.core.windows.net/artifacts/artifact-name.zip.

Create a Basic Auth Proxy for Blob Storage

If you want to download from Azure Blob Storage without making the artifacts publicly accessible, you need to use SharedKey authentication.. It looks a lot like HTTP Basic but instead of using a username/password combination, it’s a username/signature. The signature is generated by concatenating several headers and generating a signature using your shared key (available through the Azure portal).

It’s complicated and not something built into Spinnaker. If you really want to see how to generate the signature and all that, the docs have a very detailed explanation.

The short version here is that you probably want a client SDK library to do this work. Since Spinnaker really only supports HTTP Basic anyway, you need to take in username:password in some form and convert that into the request Blob Storage expects.

Richard Astbury has a great blog article showing a simple web server that takes in HTTP Basic auth where your storage account name is the username and the storage account key is the password, then proxies that through to Blob Storage for download.

Deploy this proxy somewhere you can access like an Azure App Service.

I converted this to be an Express app without too much trouble, which adds some logging and diagnostics to help with troubleshooting.

Things to keep in mind when you deploy this:

• Run in HTTPS only mode using a minimum TLS 1.2. These are options in the Azure App Service settings. You’ll be using HTTP Basic auth which isn’t encrypted, so ensuring the communication itself is encrypted is important.
• Consider adding more layers of security to ensure random people can’t just use your proxy. Azure App Service apps are public, so consider updating the app to only respond to requests to a special endpoint URL that contains a key only you know. You could also ensure you only respond to requests for known storage account info, not just ay info. That would mean folks need the special endpoint URL and the storage account info to get anything.
• Ensure it’s “always on.” Azure App Service apps go to sleep for a while when they’re idle. The first request can time out as it wakes up, and if Spinnaker is the first request it’ll cause a pipeline failure.

For this walkthrough, we’ll say your Blob Storage proxy app is at https://mycoolproxy.azurewebsites.net. You’ll see this show up later as we set up the rest of the pieces. When Spinnaker wants to download an HTTP artifact, it’ll access https://mycoolproxy.azurewebsites.net/artifacts/artifact-name.zip instead of accessing the Blob Storage URL directly.

You should now be able to access an Azure Blob Storage blob via your proxy using HTTP Basic authentication. Verify this works by dropping a blob into your artifact storage and downloading it through the proxy.

Enable Spinnaker Webhook Artifacts

Spinnaker needs to know that it can expect artifacts to come in via webhooks. To do that, you need to enable the artifacts. Edit the .hal/default/profiles/echo-local.yml file to enable it.

webhooks:
  artifacts:
    enabled: true

Enable Spinnaker HTTP Artifacts

The webhook will tell Spinnaker where the artifact lives, but you have to also enable Spinnaker to download the artifact. You also have to provide it with credentials to do so. There is good doc explaining all this but here’s a set of commands to help you.

hal config features edit --artifacts true
hal config artifact http enable
USERNAME=myartifacts
PASSWORD=[big long shared key from Azure Portal]
echo ${USERNAME}:${PASSWORD} > upfile.txt
hal config artifact http account add myartifacts \
  --username-password-file upfile.txt

# You're stuck with that upfile.txt in the current folder until you
# do a backup/restore of the config. First deploy it to get the changes
# out there...
hal deploy apply

# Now back up the config.
hal backup create

# Immediately restore the backup you just made.
hal backup restore --backup-path halbackup-Tue_Dec_18_23-00-10_GMT_2018.tar

# Verify the local upfile.txt isn't being used anymore.
hal config artifact http account get myartifacts

# Assuming you don't see a reference to the local upfile.txt...
rm upfile.txt

Spinnaker now knows not only that webhooks will be supplying artifact locations, but also how to provide HTTP Basic credentials to authenticate with the artifact source.

At the time of this writing I don’t know how Spinnaker handles multiple sets of credentials. For example, if you already have some HTTP Basic artifact credentials set up and add this new set, how will it know to use these new creds instead of the ones you already had? I don’t know. If you know, chime in on this GitHub issue!

You should now have enough set up to enable Spinnaker to download from Azure Blob Storage. If an artifact URL like https://mycoolproxy.azurewebsites.net/artifacts/artifact-name.zip comes in as part of an HTTP File artifact, Spinnaker can use the supplied credentials to download it through the proxy.

Create an API Management Service for EventGrid Handling

Azure API Management Service is usually used as an API gateway to allow you to expose APIs, secure them, and deal with things like licensing and throttling.

We’re going to take advantage of the API Management policy mechanism to handle Azure EventGrid webhook notifications. This part is based on this awesome blog article from David Barkol. Using API Management policies enables us to skip having to put another Node.js proxy out there.

Why not use API Management for the Blob Storage proxy? I wasn’t sure how to get the Azure Blob Storage SDK libraries into the policy and didn’t want to try to recreate the signature generation myself. Lazy? Probably.

The API Management Service policies will handle two things for us:

1. Handshaking with EventGrid for webhook subscriptions. EventGrid doesn’t just let you send webhooks wherever you want. To set up the subscription, the receiver has to perform a validation handshake to acknowledge that it wants to receive events. You can’t get Spinnaker to do this, and it’s nigh impossible to get the manual validation information.
2. Transforming EventGrid schema to Spinnaker Artifact schema. The data supplied in the EventGrid webhook payload isn’t something Spinnaker understands. We can’t even use Jinja templates to transform it at the Spinnaker side because Spinnaker doesn’t want an inbound webhook to be an array - it wants the webhook to be an object that contains the array in a property called artifacts. We can do the transformation work easily in the API Management policy.

First, create your API Management service.

This can take a long time to provision, so give it a bit. When it’s done, your API will be at a location based on the API Management Service name, like https://my-azure-api.azure-api.net. All the APIs you hang off this management service will be children of this URL.

Once it’s provisioned, go to the “APIs” section. By default, you’ll get a sample “Echo” API. Delete that.

Now, add a new “Blank API.” Call it “EventGrid Proxy” and set the name to eventgrid-proxy. For the URL scheme, be sure to only select HTTPS. Under “Products,” select “Unlimited” so your developer key for unlimited API calls will be in effect.

In the EventGrid Proxy API, create a new operation called webhook that takes POST requests to /webhook.

The webhook operation doesn’t do anything except give us a valid endpoint to which EventGrid can post its webhook notifications.

On the EventGrid Proxy api, select “Design” and then click the little XML angle brackets under “Inbound Processing.” This will get you into the area where you can set up and manage API policies using XML.

In the inbound policy, paste this XML beauty and update the noted locations with your endpoints:

<policies>
    <inbound>
        <set-variable value="@(!context.Request.Headers.ContainsKey("Aeg-Event-Type"))" name="noEventType" />
        <choose>
            <when condition="@(context.Variables.GetValueOrDefault<bool>("noEventType"))">
                <return-response>
                    <set-status code="404" reason="Not Found" />
                    <set-body>No Aeg-Event-Type header found.</set-body>
                </return-response>
            </when>
            <otherwise>
                <set-variable value="@(context.Request.Headers["Aeg-Event-Type"].Contains("SubscriptionValidation"))" name="isEventGridSubscriptionValidation" />
                <set-variable value="@(context.Request.Headers["Aeg-Event-Type"].Contains("Notification"))" name="isEventGridNotification" />
                <choose>
                    <when condition="@(context.Variables.GetValueOrDefault<bool>("isEventGridSubscriptionValidation"))">
                        <return-response>
                            <set-status code="200" reason="OK" />
                            <set-body>@{
                                var events = context.Request.Body.As<string>();
                                JArray a = JArray.Parse(events);
                                var data = a.First["data"];
                                var validationCode = data["validationCode"];
                                var jOutput =
                                    new JObject(
                                        new JProperty("validationResponse", validationCode)
                                    );
                                return jOutput.ToString();
                            }</set-body>
                        </return-response>
                    </when>
                    <when condition="@(context.Variables.GetValueOrDefault<bool>("isEventGridNotification"))">
                        <send-one-way-request mode="new">
                            <!-- Replace this with YOUR Spinnaker location! -->
                            <set-url>https://your.spinnaker.api/webhooks/webhook/azureblob</set-url>
                            <set-method>POST</set-method>
                            <set-header name="Content-Type" exists-action="override">
                                <value>application/json</value>
                            </set-header>
                            <set-body>@{
                                var original = JArray.Parse(context.Request.Body.As<string>());
                                var proxied = new JArray();
                                foreach(var e in original)
                                {
                                    var name = e["subject"];
                                    // Replace this with YOUR Blob and proxy info!
                                    var reference = e["data"]
                                                     .Value<string>("url")
                                                     .Replace("myartifacts.blob.core.windows.net",
                                                              "mycoolproxy.azurewebsites.net");
                                    var transformed = new JObject(
                                        new JProperty("type", "http/file"),
                                        new JProperty("name", name),
                                        new JProperty("reference", reference));
                                    proxied.Add(transformed);
                                }

                                var wrapper = new JObject(new JProperty("artifacts", proxied));
                                var response = wrapper.ToString();
                                context.Trace(response);
                                return response;
                            }</set-body>
                        </send-one-way-request>
                        <return-response>
                            <set-status code="200" reason="OK" />
                        </return-response>
                    </when>
                    <otherwise>
                        <return-response>
                            <set-status code="404" reason="Not Found" />
                            <set-body>Aeg-Event-Type header indicates unsupported event.</set-body>
                        </return-response>
                    </otherwise>
                </choose>
            </otherwise>
        </choose>
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

Again, update the URL endpoints as needed in there!

• The location of your Spinnaker API (deck) endpoint for the webhook. This is always at /webhooks/webhook/sourcename where I’ve used azureblob for the source but you could use whatever you want. The source name is used in the pipeline configuration for webhooks.
• The location of your Blob Storage and Blob Storage Proxy. These got set up earlier - in the Replace call up there, you need to swap your Blob Storage server location for the location of your proxy so when Spinnaker tries to download, it’ll go through your proxy.

I’m not going to dive too deep into API policy management. I recommend reading the official docs and being aware of what is available. However, I will explain what this policy does and how it works.

Whenever something comes from EventGrid it includes an Aeg-Event-Type header. The two header values we’re concerned with are SubscriptionValidation and Notification.

If there’s no header, return a 404. Whatever the request wants, we can’t support it.

If the header is SubscriptionValidation, EventGrid wants a handshake response to acknowledge we expect the subscription and want to handle it. We grab a validation code out of the inbound request and return a response that includes the code as an acknowledgement.

If the header is Notification, that’s the actual meat of the subscription coming through. The data will be coming through in EventGrid schema format and will look something like this when it’s a “blob created” event:

[
  {
    "topic": "/subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/xstoretestaccount",
    "subject": "/blobServices/default/containers/container/blobs/filename",
    "eventType": "Microsoft.Storage.BlobCreated",
    "eventTime": "2017-06-26T18:41:00.9584103Z",
    "id": "831e1650-001e-001b-66ab-eeb76e069631",
    "data": {
      "api": "PutBlockList",
      "clientRequestId": "6d79dbfb-0e37-4fc4-981f-442c9ca65760",
      "requestId": "831e1650-001e-001b-66ab-eeb76e000000",
      "eTag": "0x8D4BCC2E4835CD0",
      "contentType": "application/octet-stream",
      "contentLength": 524288,
      "blobType": "BlockBlob",
      "url": "https://oc2d2817345i60006.blob.core.windows.net/container/filename",
      "sequencer": "00000000000004420000000000028963",
      "storageDiagnostics": {
        "batchId": "b68529f3-68cd-4744-baa4-3c0498ec19f0"
      }
    },
    "dataVersion": "",
    "metadataVersion": "1"
  }
]

We need to transform that into a Spinnaker webhook format that includes artifact definitions. The artifacts coming from Azure Blob Storage will be HTTP File artifacts. It should like this:

{
  "artifacts": [
    {
      "type": "http/file",
      "reference": "https://mycoolproxy.azurewebsites.net/container/filename",
      "name": "/blobServices/default/containers/container/blobs/filename"
    }
  ]
}

• The reference is the download URL location through your proxy to the blob.
• The name is the subject of the EventGrid event.

The policy does this transformation and then forwards the EventGrid event on to your Spinnaker webhook endpoint.

You should now be able to POST events to Spinnaker via API Management. Using the example EventGrid body above or the one on the Microsoft docs site, make modifications to it so it looks like it’s coming from your blob store - update the locations, container name, and filename as needed. Use the “Test” mechanism inside Azure API Management to POST to your /webhook endpoint and see if it gets forwarded to Spinnaker. Make sure you add an Aeg-Event-Type header to the request or you’ll get the 404. Inside the test area you can see the trace of execution so if it fails you should see exception information and be able to troubleshoot.

Subscribe to EventGrid Blob Created Events

EventGrid needs to send events to your new API endpoint so they can be proxied over to Spinnaker.

In the Azure Portal, in the API Management section, take note of your API key for “Unlimited” - you associated this key with the API earlier when you created it. You’ll need it for creating the subscription.

When you access your /webhook API endpoint, you’ll pass that key as a query string parameter: https://my-azure-api.azure-api.net/webhook?subscription-key=keygoeshere

It’s easier to create subscriptions in the Azure Portal because the UI guides you through the right combination of parameters, but if you want to do it via CLI…

az eventgrid event-subscription create \
  --name spinnaker \
  --endpoint-type webhook \
  --included-event-types Microsoft.Storage.BlobCreated \
  --resource-group myrg \
  --resource-id /subscriptions/GUIDHERE/resourceGroups/myrg/providers/Microsoft.Storage/StorageAccounts/myartifacts \
  --endpoint https://my-azure-api.azure-api.net/webhook?subscription-key=keygoeshere

You only want the Microsoft.Storage.BlobCreated event here. If you create or overwrite a blob, the event will be raised. You don’t want the webhook to fire if you delete something.

This may take a few seconds. In the background, EventGrid is going to go do the handshake with API Management and the policy you put in place will respond. This is easier to watch in the Azure Portal UI.

You now have Azure Blob Storage effectively raising Spinnaker formatted webhooks. Azure Blob Storage causes EventGrid to raise a webhook event, API Management policy intercepts that and transforms it, then it gets forwarded on to Spinnaker. You can trigger pipelines based on this and get artifacts into the pipeline.

Test the Full Circle

Here’s what I did to verify it all worked:

Create a simple Helm chart. Just a Kubernetes Deployment, no extra services or anything. You won’t actually deploy it, you just need to see that some sort of retrieval is happening. Let’s say you called it test-manifest so after packaging you have test-manifest-1.0.0.tgz.

Create a simple pipeline in Spinnaker. Add an expected artifact of type http/file. Specify a regex for “Name” that will match the name property coming in from your webhook. Also specify a “Reference” regex that will match reference.

In the “Automated Triggers” section, add a webhook trigger. Give it a source name that lines up with what you put in the API Management policy. In the example, we used azureblob. Also add an artifact constraint that links to the expected artifact you added earlier. This ensures your pipeline only kicks off if it gets the artifact it expects from the webhook payload.

In the pipeline add a “Bake (Manifest)” step. For the template artifact, select the .tgz artifact that the webhook will be providing. Fill in the name and namespace with whatever you want. Add an override key that will be used in the template process.

Now use the Azure Portal to upload your test-manifest-1.0.0.tgz “artifact” into your Azure Blob Storage container. This should cause the webhook to be raised, eventually making it to Spinnaker. Spinnaker should see the artifact, kick off the pipeline, and be able to download the Helm chart from Blob Storage so it can run the Helm templating step.

On the finished pipeline, you can click the “Source” link in the Spinnaker UI to see a big JSON object that will contain a huge base-64 encoded artifact that is the output of the “Bake (Manifest)” step. If you decode that, you should see your simple Helm chart, templated and ready to go.

Troubleshooting

What do you look at if it’s not working? There are a lot of moving pieces.

• Verify the credentials you configured in Spinnaker. The username should be the blob storage account name and the password should be the shared key you got out of the Azure Portal. Try doing a download through your HTTP Basic proxy using those credentials and verify that still works.
• Run a test in API Management. The “Test” section of API Management lets you simulate an event coming in from EventGrid. Use the “Trace” section in there to see what’s going on and what the response is. Did the transformation fail?
• Look at Spinnaker “echo” and “deck” pod logs. I’m a big fan of the stern tool from Wercker for tailing logs of pods. It’s easy to watch logs in real time for doing something like: stern 'spin-(echo|deck).*' -n spinnaker -s 5s Make a test request from API Management and watch the logs. Do you see any errors fly by?
• Ensure you haven’t changed any keys/names. The Azure Blob Storage key, the API Management key, the source name in the Spinnaker webhook… all these names and keys need to line up. If you rotated keys, you need to update the appropriate credentials/subscriptions.
• Is something timing out? If you don’t have the “Always On” feature for your Azure Blob Storage proxy, it will go to sleep. The first request will cause it to wake up, but it may not be fast enough for Spinnaker and the request to download may time out.

Next Steps

By now you should have things basically working. Things you could do to make it better…

• Test with large files. I didn’t test the Azure Blob Storage proxy thing with anything larger than a few hundred kilobytes. If you’re going to need support for gigabytes, you should test that.
• Add security around the Spinnaker webhook. The Spinnaker webhook endpoint doesn’t have any security around it. There’s no API key required to access it. You could make the “source” for the webhook include some random key so no hook would be raised without matching that source. If you have to rotate it that might be a challenge. This also might be something to file as an enhancement on the Spinnaker repo.
• Add security around the Azure Blob Storage proxy. You definitely need to have your basic creds in place for access here, and you need valid credentials for a storage account… but (at least in the Astbury blog article) there’s no limit on which storage accounts are allowed. You might want a whitelist here to ensure folks aren’t using your proxy to access things you don’t own.

A couple of coworkers were at a conference and we were talking about how the sessions are going. One of them mentioned that there is a lot about “getting started” on various technology but very little content about what you do once you’re past that.

This is something I’ve noticed for a while now. I mentioned it here way back in 2009 talking about enterprise-level ASP.NET application challenges. It’s one of the reasons I wrote a deep dive on .NET Core configuration.

Today it occurred to me why this may be the case.

There will always be more developers out there who are unfamiliar with a topic than those who are familiar. Maybe they’re hobbyists just checking something new out, maybe they’re contractors going from one project to another, maybe they’re students trying to learn new things. Whatever the case may be, those folks will always outnumber the people who already know about the topic.

As such, when a company surveys developers about what documentation is the most valuable, or what features in the IDE are most interesting, or what experience they really need optimized… the “getting started” experience would naturally bubble to the top. Statistically, more people are interested in that. And that’s not a bad thing - you likely do want as many people as possible to use whatever thing it is you’re publishing. Getting new developers and efforts going is a good thing.

What I haven’t ever seen, though, is a survey of what long-term, existing users want. “Of the people who have gotten past the ‘getting started’ bits, what would you like to see?”

I understand why I’ve never seen that - there’s no great way to properly administer that survey. How do you locate the target audience and ensure they’re the ones answering? How much will it cost to do that work? And even if you could figure it out, again, statistically, those folks are in the minority. The value to the company to spend time on those things may or may not necessarily be time well spent.

It’s just frustrating. The “new project” experience from an enterprise perspective can actually be relatively rare. If you already have a product you’re working on and enhancing, likely you’re adding to something existing rather than building something Greenfield. I’d love to see more examples, documentation, and product/framework features that help/reward the long-term users.

I use VirtualBox a lot. I have a few different Vagrant images, not all of which have Hyper-V equivalents. There’s also a lot of mindshare for VirtualBox as a default virtualization provider when it comes to working with Kubernetes and Docker tooling. Defaults are for VirtualBox with Hyper-V added later and not quite as flexible.

Of course, you can’t have Hyper-V and VirtualBox running at the same time. It’s a problem many have run into. The default on Docker for Windows is to use Hyper-V and it pretty well hides the details from you to get things running. If you want to use VirtualBox, the common solution is to add an entry to optionally enable Hyper-V at boot.

I want my VirtualBox / Vagrant images on Windows.

And I want my Docker.

How do I make that happen?

Well, before there was Docker for Windows, there was “Docker Toolbox.” Part of Docker Toolbox was docker-machine, sort of like Vagrant but for bringing up a preconfigured Docker host. Conveniently, docker-machine runs in VirtualBox! So let’s get Docker running.

First, install VirtualBox if you don’t already have that installed. Obviously you can’t have Hyper-V enabled if you’re doing this.

Next, enable Windows Subsystem for Linux and install a Linux distro. I installed Ubuntu.

In your WSL Ubuntu, install Docker CE. Once this is done, you’re going to try running docker run hello-world and you’ll get a message like this:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

No, it’s not. You can’t run Docker inside Windows Subsystem for Linux. At this point, you’ll see solutions where people install Docker for Windows and expose the daemon on port 2375. But… that requires Hyper-V, and we’re not using Hyper-V. So.

Back in Windows land, go download docker-machine. Put that somewhere in your path so you can call it.

Run: docker-machine create docker-host

This is the magic, right here. This will automatically provision a VirtualBox VM running a small Linux host that just exposes the Docker daemon. It might take a second, be patient. When it’s done you’ll have a VM called docker-host running.

You need some info about the Docker host, so run docker-machine env docker-host - this will dump a bunch of values you’ll want. Here’s what a PowerShell output looks like:

PS> .\docker-machine env docker-host
$Env:DOCKER_TLS_VERIFY = "1"
$Env:DOCKER_HOST = "tcp://192.168.99.100:2376"
$Env:DOCKER_CERT_PATH = "C:\Users\username\.docker\machine\machines\docker-host"
$Env:DOCKER_MACHINE_NAME = "docker-host"
$Env:COMPOSE_CONVERT_WINDOWS_PATHS = "true"
# Run this command to configure your shell:
# & "C:\util\docker-machine.exe" env docker-host | Invoke-Expression

Those environment variables are the important bits.

Jump back in WSL Ubuntu and edit your ~/.bash_profile to have those values.

export DOCKER_HOST=192.168.99.100:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=/mnt/c/users/username/.docker/machine/machines/docker-host
export DOCKER_MACHINE_NAME=docker-host
export COMPOSE_CONVERT_WINDOWS_PATHS=true
Note the cert path has changed a bit - WSL mounts the C: drive at /mnt/c so you need to update accordingly. Also, if you're not using Windows-formatted Docker Compose files, you probably don't need that COMPOSE_CONVERT_WINDOWS_PATHS bit, but I left it.

Run source ~/.bash_profile to update the info in your environment.

You should now have:

• A VirtualBox VM called docker-host created by docker-machine up and running.
• A WSL Ubuntu instance with Docker installed and configured to use the docker-host daemon.

In WSL Ubuntu try it again: docker run hello-world

You should get the message from the Docker hello-world container. Yay! Docker on Windows using VirtualBox!

Additional items to note:

• The IP address of the docker-host may change. docker-machine makes a DHCP server in VirtualBox that enables the daemon only for your local machine, but depending on how many Docker hosts you have running or other VMs using that network adapter you may see the IP address shift. You’ll have to update your ~/.bash_profile if that happens.
• You can change how much CPU and memory is associated with the docker-host you create. Run docker-machine to see the available parameters and help.
• You should be able to install the Docker CLI for Windows and tell it (using the docker-host environment variables) to also use that docker-host. That way WSL and Windows itself would share the host. I haven’t tried this myself. My goal was getting WSL running with Docker, so that’s what I did. (Why not just use a Linux VM in VirtualBox and skip all this? Great question. I could make something up like, “This way all your VirtualBox machines and the physical machine can use the same Docker host and share resources,” but… Oh, look at the time! I have to, uh, go now.)

While I have made a lot of costumes and clothing, I’ve never previously sewn a bag. When I saw the YouTube video of Adam Savage building his EDC TWO bag, it made me want to build a bag, too.

In the video he mentioned that the EDC TWO is slightly smaller than the EDC ONE and I wanted to make the full-sized bag. He sells the plans for both, but I bought the EDC ONE pattern. I see that since the time I bought it he’s now offering it in your choice of physical/paper plans or a PDF download. I’m glad I bought the paper plans; it’s huge and I don’t have a printer that would accommodate such a thing. Taping a million sheets of paper together isn’t my style.

Big thanks to Adam Savage and Mafia Bags for making this pattern and publishing it.

The pattern is licensed by Adam Savage under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. I think that means you could technically distribute the pattern for free as long as you give credit, but it’s all of $15 and I feel good contributing to someone who put something cool together. I don’t think my explanation here really needs a license, but assuming it does, let’s also use that same license with the same restrictions, etc.

OK, let’s get this going.

Assumptions

I will make some assumptions as I go along here about a few things. You may have to Google some stuff if you don’t know what I’m talking about. Here’s what I assume:

• You have the EDC ONE pattern. I’m not going to give you the pattern here. I’m going to show you how I used it and call things out about it.
• You have some sewing experience. If I mention that I basted two pieces together, you should know it doesn’t have anything to do with a turkey baster. Tons of YouTube videos and articles out there to catch you up if you don’t recognize what I’m talking about.

Improvements and Errata

Along the way I’ll point out the improvements that could be made in the pattern as well as some of the “bugs” I encountered in the pattern or provided instructions. If you’re building along here, it’d be good to read through all of this first so you might know what’s coming.

General Tips

• Watch your sewing machine tension. I had a heck of a time getting this right and some of my stitching from the inside looks bad. If you normally sew through a couple of layers of medium weight fabric, using heavier-weight fabric means you’re going to have to adjust. For example, my machine usually sits on a tension level 4, but for something close to decent I had to almost double that to 7.75.
• Use the right needle for the job. I used duck canvas for the body of my bag. I had some leather and denim needles but they really didn’t work out in thicker areas. When putting the handle attachments on the bag, for example, you may be sewing through upwards of six thicknesses of fabric. Six layers of duck canvas bends a leather needle. I didn’t find until the end that there are “heavy duty needles” (size 110/18) that are made for this sort of job and I wish I had known earlier.
• Clip corners to reduce bulk. Especially if you’re using thick fabric, you need to clip some of the corners to make sure things lay flat. This includes some of the things like the handle attachments that get sewn to the bag body - without clipping, you see fabric folds and weirdness along the edges.
• Baste things in place. I had a bad time holding super thick fabric with pins. It was pretty good to machine baste things in place close to the edge of the fabric / inside the seam allowance to keep the parts together while assembling. This was especially helpful when attaching the lining, zipper, bag outside, and metal frame holder pieces in one go.
• Watch your seam allowances. This pattern doesn’t work like a Simplicity or Vogue pattern you might buy at the store. That was sort of hard for me.Common patterns use a 5/8” seam allowance. The instructions on this pattern say it’s a 1/2” seam allowance unless marked. However, they may only change the seam allowance on one side of a pattern piece, so if you see a 3/8” marking it might only mean for that side. When in doubt, measure.
• There’s no pattern layout for the fabric. You have to kind of figure out how to place the pattern pieces yourself to make the best use of your fabric.
• Use the drawings and reference photos. Given the instructions are a little vague in places, use all the photos and drawings you can to fill in the gaps. I wouldn’t have been able to finish this without the detailed photos for their finished bag.
• There are bugs in the pattern. I will call them out shortly, but you’re not crazy.

Initial References

• EDC ONE bag - This is what the bag looks like fully put together. Really helpful for reference photos to see if you’re building things right.
• EDC ONE pattern - This is the pattern I bought.

When looking at the pictures of the finished bag and comparing to the pattern, I took some notes.

Front:

• The pattern calls for lobster claws to hold the shoulder strap. The lobster claw they use isn’t like anything I’ve seen. I searched and couldn’t find anything like that.
• You can see the stitching that attaches the metal frame holder channel inside the bag. The pattern doesn’t specifically say you need to sew through all thicknesses when attaching that, so its good to know that’s what’s happening.

Back:

• The pattern calls for “tube magnets” to put in the handles. I’ll get into this later, but I couldn’t figure out what this was and the picture didn’t make it any more clear.
• The “D rings” used to hold the handles on look more square here than standard D rings. I ended up using rectangle rings instead of D rings.
• The shadow on the bag shows that the metal frame you insert in each side of the bag opening may not go the full width of the side - there’s space between where the two frame halves end to allow the bag opening to bend freely.

Inside:

• Again, you can see the frame holder is stitched to the bag through all thicknesses. There’s no crafty seam hiding here.
• Also, again, you can see the metal frame appears slightly shorter than the width of the bag to allow the bag to open and close easily.

Pattern Bugs

Here are the things I found that are wrong on the pattern. Keep these things in mind as you are cutting and sewing.

• The “strap loop” piece is marked “cut one.” Howeer, you need two of these - one for each side of the bag. Cut two.
• The pattern calls for 5” of 3/4” webbing used with the strap loops. Again, this is only enough for one and you need two loops… so you need 10” of 3/4” webbing. (I’ll update that in my parts list below.)
• The bottom piece says you need one for the lining, one for the bottom, implying a total of two pieces. However, you need one for the bottom of the “A panels,” one for the bottom of the “B panels,” and one for the lining - a total of three pieces.
• The instructions say to use 3” Velcro pieces on the main panel for your patch but the pattern drawing measures out at 3.5” for the Velcro pieces. I don’t know which is right, but given the 70” Velcro measurement I think it’s supposed to be 3.5”.
• The handle tab pattern piece is slightly larger than the placeholder shown on Main Panel A. I think the drawing on Main Panel A is wrong.
• There are no instructions explaining what to do with the shoulder pad pieces. For that matter, there’s no “pad” in the shoulder pad in this pattern. I’ll explain what I did in my instructions.
• There are no instructions explaining when you should attach the D rings or the strap loops to the main bag.
• There is no measurement for the metal frame. I used two 22” pieces of 3/16” steel rod. I bent 3-5/8” legs on each end of the rod pieces for the U-shape of the frame.

Parts List

The parts list that ships with the pattern is amazingly vague. For example, they list “self fabric.” Uh… how much? “Lining.” Again, how much?

Here’s a more detailed parts list based on what I actually used. Note I used a different fabric color for the bottom than I did for the main bag body so I list those things separately. Also, I didn’t really pay attention to fabric grainlines since I used duck canvas and it generally looks and works the same regardless of direction. You may need to factor that in.

• 18” of 60” duck for body
• 12” of 60” duck for bottom / accent
• 18” of 60” rip stop nylon for lining
• 29” continuous zipper with two sliders meeting in the middle
• 4 - D rings (rectangle rings) 1” inside width
• 2 - lobster clasps: 1” inside width for strap, clasp big enough for 3/4” webbing
• 1 - webbing slide, 1” inside width
• 70” of 1” wide Velcro
• The pattern calls for 2 “tube magnets” for handles. I used…
  • 2 - 10mm x 60mm neodymium cylinder magnets
  • 8” carbon fiber tube, 10mm inside diameter
• 60” of 1” wide webbing
• 7-3/4” x 15” plastic board for the bottom (corrugated sign)
• 44” of 3/16” steel rod for the metal frame
• 10” of 3/4” webbing
• 3” x 12” craft foam for the shoulder pad if you decide to follow my enhanced instructions

I have no idea what a “tube magnet” is, but that’s used in the handle of the bag. I figured I wanted something that would be nice and strong but still have the magnetism, so I put neodymium magnets inside carbon fiber tubes. To keep the magnets centered in the tubes I put some padding in each end and sealed them up with some silicone sealant.

In the parts list I show using two magnets (one for each handle) and 8” of tube (4” for each handle). You may see in my pictures that I actually went with four magnets (two in each handle) and 12” of tube (6” per handle). This was too much. My handles are a bit too long and the magnet power is a bit stronger than my liking. I may later redo the handles with shorter tubes.

A note on neodymium magnets: They are powerful and they will jump out of your hands to attach to each other. If they do this, they may hit each other with enough force to break. This absolutely happened to me. Luckily I could just jam them into the carbon fiber tubes and it wasn’t a big deal, but be aware this could happen.

Preparation

Cut out all the parts. I didn’t bother transferring pattern markings onto every part because this pattern is very square - you can just measure where things need to be at the time you need to place them.

Attach seven strips of Velcro, each 9” long, to the bottom of the bag lining.

Enhancement: Either don’t use Velcro at all, or ensure you use only the soft half of the Velcro. I’m not sure what value having Velcro at the bottom of the bag provides. I put it in, but probably wouldn’t on future versions. If you do this step, make sure you use the soft half of the Velcro. If you use the rough/stiff half, then you can’t really put any clothing into the bag because it’ll snag.

Enhancement: The next step on the official instruction list is to attach the main panel B to main panel A; and side panel B to side panel A. Don’t do that. It doesn’t make any sense. You’re going to have two “bags” - a short outer bag (from the “B” panels) and a larger inner bag (from the “A” panels). You’re going to need to sandwich the plastic board between those two bag bottoms. Attaching the main panels/side panels like this will just make the sewing way harder. Hard pass.

Fold back and finish the edges for the pen holder. You may want to trim the corners so you don’t have weird extra folds showing at the top.

Instead of marking the pen holder pattern on the pen holder, press the pen holder at the locations where you need to sew. This serves two purposes - first, you won’t have to deal with pattern markings; second, the pen holder will bend a little cleaner at the places that get sewn down.

Attach the pen holder to the pocket.

Enhancement: More pockets, more pen holders, pocket closures. The pattern shows a single pocket and pen holder on the inside of the bag. The pocket is open on the top. I have a lot of small things that I cart around with me, so one pocket is not enough. I’d also really like to have a pocket that can close - snap, Velcro, zip, whatever. Before you sew the pocket down to the lining, consider adding more pockets, having better/bigger pockets, etc.

Attach the pocket + pen holder to the lining.

Cut two lengths of Velcro that are 3.5” long. Attach these to one of the main panel A pieces.

Fold the handle tabs, press, and trim the corners. Attach these with the D rings to the main panel A pieces of the bag.

Cut the 3/4” webbing in half so you have two 5” long pieces.

Fold and press the strap loop attachments. Trim the corners as neeeded.

Attach the shoulder strap loops to the main panel A pieces of the bag.

You should have something that looks like this. Note my Velcro is longer because I wanted a longer patch attachment.

Cut the carbon fiber tube into two 4” lengths. Each one of these will be a handle.

Put the neodymium cylinder magnets into the carbon fiber tubes. Pack each side of the tube so the magnets sit in the center and won’t shake around. However, don’t glue the magnets in because, depending on the orientation, you may need them to be able to rotate in place inside the tube such that the poles will line up when you put the two handles together.

Seal each side of the tube with something waterproof like silicone sealant. Using a waterproof sealant will ensure you can throw the bag in the washing machine if you need to.

I took it a step further and also dipped the handle ends in Plasti-Dip, however if you do that then the carbon fiber tube won’t slide into the handle as easy because the Plasti-Dip is very grippy.

Enhancement: Multi-colored handles. I added a layer of fabric to my handles so they’re multi-colored. The part you hold onto has an extra layer. This also adds some padding around the extra strong magnets.

Lay out your handle pieces and press the long seam allowances in. The pattern/instructions call for topstitching to close the handle.

If you’re doing the multi-color handles, sew the top layer down. Make sure when you lay this out, you pin the top layer to the bottom layer while the bottom layer is wrapped around your handle. If you don’t, especially if you’re using thick fabric, the handle won’t lay flat around the carbon fiber tube.

Once you have that set, sew the handle closed. If you didn’t use Plasti-Dip, you should be able to sew the handle and then slide the carbon fiber tube into the middle of it. I did use Plasti-Dip, so I had to sew the handle around the tube. This means I had to slipstitch it closed by hand instead of doing a machine topstitch. I like how it turned out, but it was more effort.

Enhancement: The official instructions say you should attach the handles to the main panel with the D rings now. Don’t do that. It’s going to make it much harder to get the main panels through your machine if you have the handles getting in the way, magnets attaching to the side of the machine, etc.

Fold and press the zipper ends. Trim as needed.

Attach the zipper ends to the zipper.

Construction

I’m not sure why the official instructions differentiate between “preparation” and “construction” since you’ve been constructing things all along, but it is what it is.

Sew the side panels and the main panels together to form a sort of “tube.” Do this for both the A panels and the B panels.

I attached both sides onto one main panel first, then I attached the final main panel.

Here’s what the bottom (main/side B panels) “tube” looks like.

Attach the bottom to each respective tube. You should end up with a short bag (made out of panels B) and a tall bag (made out of panels A). Here’s the bottom attached to my lower/B bag.

Trim your corners where the bottom attaches to the main bag. There’s a lot of fabric here and it’s going to be lumpy otherwise.

Flip the two bags right-side out. Here’s my main bag “A” flipped right-side out.

Sew the lining panels together into a tube.

Attach the lining bottom to the lining sides.

Fold the seam allowance on the bottom “B” bag over and press it.

Enhancement: Seal the plastic board. I used a corrugated plastic board that I got in the sign section of Home Depot. If you put the bag in the washing machine, that board is going to fill up and retain water. I used some silicone sealant along the edges of the board to ensure it’s waterproof.

Place the plastic board inside the short bag “B.” Slide the larger bag “A” down inside bag B so the board is sandwiched between the two bags. Topstitch along the top of bag B to attach the two bags.

Enhancement: Use Velcro on one of the short edges and only do topstitching on that side for the visual effect. If you want to put the bag in the washer, sewing it in means the board is stuck there. Next time I make this bag, I’m going to use Velcro to close one of the short sides between bag A and bag B so I can open the Velcro and pull the board out.

From the outside it will look like this:

From the inside it will look like this:

Put the lining into the main bag and baste really close to the top edge.

Mark the sides where the zipper should end (ie, where the zipper should stop being attached to the bag). I used pins to do this. Center the zipper on the sides and baste it in place really close to the top of the bag, ending the basting about an inch before your mark.

Press the seam allowances on the channel for the metal frame.

Pin the frame channel to the edge of the bag over the zipper. This part is sort of a geometric mind-bender.

• When the zipper/channel is folded into the bag the channel will end at the top of the bag and you’re going to sew down the bottom of the channel later.
• You want the zipper to stop attaching to the bag at the intersection where the zipper end mark is and where the seam allowance is. This is why you didn’t baste the zipper all the way to the zipper end mark. You’ll see in my picture how the zipper curves down from the edge of the bag to about 3/8” below the edge of the bag when it meets the zipper end mark. This makes a nice smooth transition as the zipper exits the bag.

This picture shows where I started the frame channel - right at the bag midpoint. I have it pinned over the top of the zipper, but the zipper makes that gradual transition/curve I mentioned under the channel. I haven’t finished pinning the frame channel down so you can see what the zipper looks like underneath it.

Sew through all thicknesses along the bag seam allowance (3/8” as marked on the pattern) - lining, bag A, zipper, frame channel top. Once it’s all sewn down you can flip the frame channel up and it should look like this from the outside:

Here’s a view of the same thing from the inside:

Fold the frame channel all the way around to the inside. Pin it down, then sew through all thicknesses - lining, bag A, frame channel bottom. When you’re done, you’ll see the stitching from the outside and it’ll look like this:

Now is a good time to actually attach the handles to the bag. You’re done pushing the bag through the machine so you won’t have the problems of the handles and the tube magnets making life hard.

You’ll note my handles don’t hang totally straight. I used 6” carbon fiber tubes but the D rings on the bag are only 4” apart. I will likely redo my handles at some point to make them better. In my instructions here, I’ve been pretty consistent about the 4” carbon fiber tube lengths to avoid this issue.

Cut the 3/16” steel rod in half so you have two lengths of 22”. On each end of each length, bend the rod down so you have 3-5/8” “legs”. You should end up with two steel rod pieces roughly in the shape of a really wide “U.” File any sharp edges off the ends so they won’t poke through the bag.

Enhancement: Coat the ends of the frame pieces in Plasti-Dip. This will stop them from sliding around in the frame channel, and it’ll also add more protection from the frame poking through the bag.

Put the frame pieces into the frame channel along the top of the bag. This gives the bag a sort of “hinged opening” like an old-school doctor’s bag.

We’re now at a point where the official instructions just stop. There’s no description of what to do with the shoulder pad, and the shoulder pad shown in the reference photos really doesn’t have any pad. So… here’s what I did. Note a lot of it was sort of eyeballing it and hoping so I may not have exact measurements here.

My basic idea was to make a shoulder pad that has an actual piece of padding sewn inside and a channel for the shoulder strap to pass through. I didn’t want it attached to the shoulder strap so I could adjust things as needed.

I cut three of the octagonal shoulderpad pieces from the official pattern. I also cut a piece of craft foam a little smaller than the octagon.

On all of the three pieces I sewed down the ends so they were finished. I also pressed along where the shoulder strap channel would be.

I took two of the finished pieces and sewed them together down the pad channel marking. This makes a sort of tube where I can push the shoulder strap through.

Here it gets a little confusing. Stick with me.

I took the third piece and stuck it on top - right-side to right-side - of the pieces with the channel. I then sewed along the edges with about a 1/4” seam allowance. In the photo below, you can see that sewing along the edge in black. You can see in the photo where I started to flip the whole thing right-side out.

Flip it right-side out and the larger pocket you’ll have is where you’re going to put the foam.

Cut a piece of craft foam and slide it into the shoulder pad. It took me a few tries to get the right size and shape so it’d fill out the shoulder pad but will also sit nice and flat.

Slipstitch the ends of the shoulder pad closed so the foam doesn’t slip out. Be careful not to sew the channel shut where you need to put the strap!

Attach the slide on the end of the shoulder strap and sew it down.

Put one of the lobster claws on the shoulder strap and thread it back through the slide.

Thread the shoulder strap through your custom shoulder pad.

Attach the other lobster claw on the other end of the strap.

There’s also no description of how to make a patch. Ostensibly many people will buy one, but the 2” x 3.5” size doesn’t seem standard to me.

I wanted to make a custom patch, so here’s how I did that.

I made my bag Velcro 2” x 8” to accommodate for the larger patch I had planned. I used my embroidery machine to create the design for the patch.

The design/front half of the patch is 2” x 8” and I created a back half that has the Velcro on it.

I attached the two halves and finished the edge of the patch with a very dense zig-zag machine stitch.

The Finished Product

After attaching the shoulder strap and patch, here’s what my bag looks like:

I’m pretty happy with how this turned out. Hopefully this tutorial helped you in making a bag you’re happy with, too!

I’m using Spinnaker to manage microservice deployments in one of my projects. One of the nice features in Spinnaker is pipeline expressions - a way to do a little dynamic calculation based on the current state of the pipeline.

I’m somewhat new to Spinnaker and I just got through trying to figure out a way to do some conditional stage execution using pipeline expressions but found the learning curve was steep. I don’t plan on repeating everything already in the docs… but hopefully I can help you out.

The Layers: Spinnaker pipeline expressions use the Spring Expression Language syntax to expose the currently executing Spinnaker pipeline object model along with some whitelisted Java classes so you can make your pipeline stages more dynamic.

That’s a lot to build on: The Spinnaker object model, Spring Expression Language, and Java. If you’re, say, a Node or .NET developer, it’s a bit of an uphill battle.

Resources:

• Spinnaker Pipeline Expressions User Guide
• Spinnaker Pipeline Expressions Reference
• Spring Expression Language
• Spinnaker Pipeline Object Model (code)
• Spinnaker Pipeline ExecutionStatus (code)

The Data: The best way you can see what pipeline data you have available is to run a pipeline. Once you have that, expand the pipeline and click the “Source” link in the bottom right. That will get you a JSON document.

The JSON doc is an “execution.” In there you’ll see the “stages” each of which has a “context” object. This is the stuff you’ll get when you use the helper functions like #stage("My Stage") - it’ll find the JSON stage in the execution with the appropriate name and expose the properties so you can query them.

Troubleshooting: There are two ways I’ve found to troubleshoot.

1. Create a test pipeline. Actually just run the expression in a test pipeline. You’ll get the most accurate results from this, but you may not be able to test the queries or operations against a “real pipeline execution” this way.
2. Use the REST API. Spinnaker has a REST API and one of the operations on a pipeline is evaluateExpression. This runs against already-complete executions but is the fastest way to iterate over most queries. I’ll show you how to use that.

If you want to use the REST API to test an expression, first find a pipeline execution you want to use for testing. As with earlier, go get the JSON doc for a pipeline run by clicking the “Source” link on a complete pipeline. Now look at the URL. It’ll look like http://spin-api.your.com/pipelines/01CMAJQ6T8XC0NY39T8M3S6906. Grab that URL.

Now with your favorite tool (curl, Postman, etc.)…

• Create a POST request to the evaluateExpression endpoint for that pipeline, like: http://spin-api.your.com/pipelines/01CMAJQ6T8XC0NY39T8M3S6906/evaluateExpression
• In the body of the request, send a single parameter called expression. The value of expression should be the expression you want to evaluate, like ${ 1 + 1 } or whatever.
• The response will come back with the result of the expression, like { result: 2 }. If you get an error, read the message carefully! The error messages will usually help you know where to look to solve the issue. For example, if you see the error Failed to evaluate [expression] EL1004E: Method call: Method length() cannot be found on java.util.ArrayList type - you know that whatever you’re calling length() on is actually an ArrayList so go look at the ArrayList documentation and you’ll find out it’s size() not length(). The errors will help you search!

Example: The pipeline expression I wanted was to determine if any prior step in the pipeline had a failure. The expression:

${ #stage("My Stage").ancestors().?[status.isFailure() && name != "My Stage"].isEmpty() }

Let’s walk through how I got there.

• ${ ... } wraps every pipeline expression in Spinnaker.
• #stage("...") is a shortcut for getting a stage from the pipeline by name.
• "My Stage" is just the name of the stage running the expression. There isn’t a way I could find to access the current stage. #root gives you the stage context but not the stage definition.
• ancestors() is a method on the Spinnaker Stage object. Notice that ancestors() returns all the ancestor stages and the current stage.
• .? is the Spring Expression Language collection selection operator. It runs the expression in the brackets and returns a filtered ArrayList.
• status is a property on the Stage object and is of type ExecutionStatus.
• isFailure() is a method on ExecutionStatus that determines if the specific status is one of several that qualify as unsuccessful.
• name is a property on the Stage object that has the human-readable name. This clause is how we exclude the current stage from the result set.
• isEmpty() is a method on the Java ArrayList class, which is what comes out of a SpEL collection selection.

How did I figure all that out? Lots of trial and error, lots of poking around through the source. That REST API evaluation mechanism was huge.