process, culture comments edit

Microsoft Teams is a collaborative communications tool in the same vein as Slack. You get your logical ‘teams’ together, folks chat and collaborate in ‘channels,’ and somehow that open office plan your company loves so much is totally justified because just look at that collaboration in action.

I’m a Slack guy. I like Slack over Teams for a few reasons which will probably become clear as I go through this.

The topic of the day is: CONVERSATIONS.

Teams organizes discussions around “conversations.” A conversation is sort of like the first post in a forum thread. It sets the topic of the conversation and replies on that conversation are attached chronologically to that conversation.

This is interesting to understand because it’s very, very different than Slack or other group chat solutions. In a group chat, messages are generally the first-class object. Slack added “threads” that approximate this conversation concept, but it’s still not the same.

Let’s say you want to search for something that was said six months ago. You remember a few key words and want to see the context.

In Slack, you’ll search, find the message, and when you click it you’ll see the message in chronological context of the rest of the messages. You can scroll up and down to find what you need, follow that thread to the next chronological conversation, or follow it back to see what led to the discussion in the first place.

In Teams, you’ll search, find the message, and when you click it you’ll see the conversation that contains the message. Just that conversation. It may not even pop you to the message itself, depending on if you’re lucky or if Teams is misbehaving. Outside of the conversation, there’s no chronological context. What else was discussed that day? Who’s to say?

Now let’s say you want to reply to a discussion that was being had earlier. You thought of something to add and you want to mention it.

In Slack, you can scroll back in the timeline to find the message. You can start a thread from that, or you can grab some text from it, quote it, and add your info to the main channel. Either way, pretty easy to find the earlier conversation.

In Teams, the timeline is not arranged chronologically. At least, not the way you might think. The timeline is ordered by “most recently replied to” conversation. If someone replies to a conversation, that conversation jumps to the bottom of your timeline as “most recent.” The conversation may have started a year ago, if someone comments on it, it’s suddenly most recent.

This means a couple of things in Teams that aren’t necessarily obvious:

  • Search is the only real way to find things. You can’t just scroll back necessarily and see the history.
  • You can never say “scroll up about X lines to see what was said” because… “up” isn’t up. The timeline dynamically rearranges on you so as soon as someone responds to that earlier conversation, everything appears in a different order.

What all this yields is that it’s important to know when to continue an existing conversation and when to start a new one.

That’s a really, really hard thing to do because conversation can be organic. However, you sort of have to “run the conversation” the way you “run a meeting” - curbing things that are off topic or getting folks to start new topics.

Why?

Let’s say there’s a conversation someone starts about the weather in Florida. It’s nice, sunny, might rain next week. That conversation goes for a while, talking about weather patterns and seasons. Then you see something like this:

Bob: The sun can really beat down and hit hard.

Alice: The sun is why my favorite season there is Summer.

Bob: It sure means a hot baseball season, though. I can barely get out to games in that heat.

Carl: How are the Marlins doing recently? I saw they traded pitchers.

Bob: They did, and now they’re not doing so well.

Carl: Maybe they should watch that movie Moneyball where they make smarter trades based on stats.

Bob: You could be right. I did see that the Yankees made a good trade this week.

The topic is “weather in Florida” but suddenly has now been totally derailed by a side conversation about baseball.

At first, you might think, “Who cares? It’s not that big of a deal.”

Well, sort of, and sort of not.

Let’s say this is a long conversation. Let’s also say it’s not about inconsequential topics, but about decisions being made for your business.

In six months, you want to go back and find out what was decided. You do a search… and remember how search results show you conversations not messages? You click on the search result and you’re presented with a wall of text where you have to do a manual search to get past all the totally unrelated junk.

This is generally not helped by the fact Teams “conveniently” hides the main body of longer conversations requiring you to click and expand them. Only the last few replies in a conversation are displayed.

It’s a real challenge in Teams when conversations and collaboration become organic like this. Sometimes it’s good to let open-ended discussions flow, and if there’s a conversation started where that’s intentional, great. Some meetings are like that, too. On the other hand, if you have a conversation on “standards for writing documentation” and someone derails it with argumentation about Oxford comma usage and “Hey, I went to Oxford!” and “Oh, really, were you born there or did you go overseas for college?” then things fall apart quickly: Search results become useless, notifications about changes to the conversation become useless, and the timeline rearranges to show the conversation has been updated with off-topic content.

If you’re trying to make the most of your time in a large set of teams and channels, one or two of these isn’t a problem but everyone taking conversations into odd directions makes managing time and discussion very hard.

Finally, it doesn’t help that chats don’t have conversations. If you get into a group chat then it’s like a standard app - it’s chronological, the timeline doesn’t rearrange, etc. It’s different if you’re in team channels. If you’re switching between chats and channels a lot, this can be really jarring.

“Couldn’t you just get everyone to agree to not use the conversations?” The idea here being if everyone used “conversations” as “individual messages” then the problems go away. I’ve tried this, and if you’re only scrolling this works fine. It breaks down if anyone, at any time, forgets and does a reply to a conversation rather than posting a new message. Any reply and the timeline rearranges. It’s done. Further, search won’t work because you can’t see conversations in relation to each other, so all you will ever get in search results is that one message.

All of this is basically why I like Slack far, far better. The organic conversation flows better, you can start threads if you want but you don’t have to, search works… it’s generally better for how I work.

halloween, costumes comments edit

This year we had 190 trick-or-treaters. This amount is about average for us.

2019: 190 trick-or-treaters.

Average Trick-or-Treaters by Time Block

Year-Over-Year Trick-or-Treaters

Halloween was on a Thursday and the weather was fairly clear. We had some friends come over so there was a small group who went out walking the streets for candy and another small group who stayed home and handed out candy. I was in the “stay home” group.

I noticed I don’t have 2017 or 2018 data.

In 2017 we were doing a fairly major downstairs remodel and didn’t hand out candy. There was a bunch of construction stuff in the driveway that we didn’t want kids getting into or trying to scoot past.

We did hand out candy in 2018 and I’m sure we tallied it but I have no idea where that tally went. I didn’t even blog my costume from last year (I went as Doctor Strange). My mind is going.

Cumulative data:

  Time Block
Year 6:00p - 6:30p 6:30p - 7:00p 7:00p - 7:30p 7:30p - 8:00p 8:00p - 8:30p Total
2006 52 59 35 16 0 162
2007 5 45 39 25 21 139
2008 14 71 82 45 25 237
2009 17 51 72 82 21 243
2010 19 77 76 48 39 259
2011 31 80 53 25 0 189
2013 28 72 113 80 5 298
2014 19 54 51 42 10 176
2015 13 14 30 28 0 85
2016 1 59 67 57 0 184
2019 1 56 59 41 33 190

Our costumes this year:

costumes, personal, halloween comments edit

For Halloween 2019 I decided to make a Star Trek II “Monster Maroon” costume. This is the dress uniform you see folks wearing in the Star Trek II through IV movies.

October 26, 2019: The complete costume (front)

I’ve wanted one for quite some time, so I went for it. I figured folks might be interested in some of the process to get this done.

Research

First, I did a lot of research. I track my research in a Google Doc and I have a folder where I save relevant images and things that I find online that can help. This is a lot of time, searching, finding images, following links, reading on forum posts, looking at movie frames for reference.

I determined that my complete costume would have:

  • Jacket
  • Pants
  • Shirt (the white puffy neck/arm one)
  • Chest insignia
  • Shoulder strap rank pin
  • Shoulder strap security device
  • Left sleeve rank pin
  • Left sleeve “pips and squeaks”
  • Belt
  • Boots
  • Phaser

I figured I’d want a phaser but I didn’t need a communicator. Too many handheld props and nowhere to put them.

I decided this broke down as such:

  • Items to sew:
    • Jacket
    • Pants
    • Shirt
    • Belt
  • Buy or 3D print:
    • Shoulder strap rank pin
    • Shoulder strap security device
    • Left sleeve rank pin
    • Left sleeve “pips and squeaks”
    • Belt buckle
    • Phaser

I skipped the boots because I already had some I planned on reusing. Not perfect, but good enough and it already appeared like a lot of work.

The Anovos version of this costume provided invaluable from a reference photo standpoint. It’s not “screen worn” but it does show how things should roughly line up and helped a lot.

This forum also has some great photos and info that I used while figuring things out.

Cost Breakdown

The probably-incomplete itemized parts list so I can scare myself when I see what got spent:

Item Cost
Jacket pattern $24.95
Pants pattern $18.95
Shirt pattern $14.95
Shoulder strap clasp $7.95
S&H for patterns and clasp $12.95
Fabric swatches for jacket, pants $11.75
Burgundy gabardine, 4 yd (jacket, pants stripes) $26.21
White gabardine (2.75 yd) $9.99
Black gabardine (4 yd) $15.98
Ivory four-way stretch fabric (shirt, 5 yd) $59.97
Thread $25.95
Black bias tape $15.54
Gold bias tape $2.79
Interfacing $1.74
Muslin $22.71
Batting (for shirt and jacket puffy sleeves) $4.97
Snaps $1.99
Stitch Witchery $2.49
Invisible zipper for shirt $2.99
Invisible zipper for pants $4.99
Shoulder pads $4.49
Silver chain $6.38
Spandex $23.78
Knit ribbing for pants cuff $5.99
Black broadcloth (pocket interior, 0.5 yd) $1.00
Black lining (1.5 yd) $5.99
Belt buckle (eBay) $20.99
2” elastic (pants waist, 2yd) $5.99
Gold soutache cord $7.48
Spray paint $50.00
TOTAL $421.90

To soften the blow, I have to consider that I have some fabric left over and I didn’t use all of the thread or bias tape, so I have some I can use on other projects. I also had a few things already that I didn’t have to buy, like white thread, so it ends up kind of evening out.

The shirt fabric would have been half that price but I hosed it up and had to make it twice. Yeah, I’ll get there.

Unless otherwise listed, the vast majority of this went to Joann Fabrics and Crafts. I didn’t break out how much I “saved” using coupons or whatever so it’s not precise. And, of course, there’s stuff you don’t think about - the black bias tape finishes all the seams on the inside, so I used way more than I thought I’d need; I really screwed things up a couple of times and had to remake a couple of pieces, so used more fabric than I thought. That sort of thing.

I didn’t count my time as something that costs, but there was a lot of time here. Calendar time isn’t equal to effort time, though - I work on these things mostly for a few hours on the weekend or on a night or two each week after work. Getting that “me time” is really helpful for me psychologically and I think it makes me less generally cranky. But I don’t know how much “effort time” went in here.

Finally, the patterns - these are as close to screen accurate as you can find. They’re built from real screen-worn costumes as much as possible. However, the instructions are terrible. I don’t care if you’ve done one of the “advanced” patterns from Vogue, this isn’t how you expect. There are missing steps, duplicate steps, one small paragraph where you really need a lot… there’s one step on the pants which is basically, “OK, now put in the waistband.” No photos, no description. It’s like someone giving you an Ikea cabinet and not labeling anything. Just sorta figure it out! This caused a lot of difficulty. Now that I’ve done it once, I see how it should be so I could do it again if needed. I also see how to make it better.

The Phaser

I couldn’t find an affordable phaser and Thingiverse didn’t have quite what I wanted, so I ended up having to make my own using Autodesk Fusion 360. I used reference photos and tried to trace the shape as close as possible.

My first draft looked OK, but was pretty plain.

June 10, 2019: First draft of my phaser

I started adding some details, moving some things around.

June 17, 2019: Iterating over the phaser

After a month or so, in mid-June I finally got something I was happy with and it looked pretty good painted up. If you’d like to make one, I put the model on Thingiverse for free.

July 1, 2019: Final version of the phaser

The Security Device

I followed a similar process as my phaser for the security device. Research tells me the actual screen-used one is a kit-bashed tank wheel from some plastic model kit. I used photos of the screen used props as well as a commercially available replica to create my model. If you want to make one of these, I put the model on Thingiverse for free.

The security device was done about a month after the phaser at the end of July.

July 26, 2019: Iterating over the security device

The Shirt

I started the shirt around the same time as the security device (early July), right after the phaser was finished. I had tried making a muslin version to determine the right size, and while I found the right size around and dove in, it turned out it wasn’t long enough for my torso. I moved too quick with the muslin one to realize I had forgotten to hem it, which shortens it up by an inch or so. Dammit. I had to throw out the whole shirt and start over because you can’t “add length” to the bottom of a shirt.

I also remade the sleeves and neck several times to try to get the “puffiness” right. Luckily these are separate pieces that fit on so when I saw they weren’t right I noticed before they were attached and saved having to remake the shirt more times.

One of the items in the parts list is Stitch Witchery. This is sort of like “fabric tape” - you put it between two pieces of fabric, iron them, and they stick together. That’s how the hem in the shirt works and looks “stitch-free.”

It took about a month (end of July) to get the shirt done.

July 28, 2019: The finished shirt

The Belt

Making the belt was actually pretty easy. I had some vinyl left over from a different costume, so I wrapped the vinyl around some batting and stitched it up. I think this took maybe an hour.

However, the belt buckle was a struggle. I really wanted to 3D print one since I was so deep into making things, but I found that the PLA plastic can’t hold up to the strain of keeping a belt together. The little hook that keeps the belt closed simply breaks, even if it’s solid PLA. I ended up getting a metal belt buckle from eBay and calling it good.

August 2, 2019: The finished belt

The Pants

I started with muslin for the pants which saved a lot of effort. I found that the size I thought I’d need was actually just a little too small.

The pants were my first run-in with how problematic the pattern instructions really are. There are no pattern markings, so when you’re trying to put in pockets which are “optional” you really don’t know how that’s supposed to go. The leg stripes ended up getting made a couple of times because the description of how to make them didn’t actually yield a result that matched the required measurements.

August 24, 2019: Sewing stripes for the pants

The waistband is my biggest gripe in all of this. There are fully four different ways you could make the waistband, each of which is a single paragraph that doesn’t explain enough about what has to happen, and there are no pictures. I sort of muscled something functional into place and I’m absolutely not happy with it. But I decided to skip remaking them (for now) in favor of getting the rest of the costume done.

It took another month (end of August) to get the pants done.

August 25, 2019: Finished shirt and pants

The Jacket

I also started the jacket with a muslin version and, knowing what I know about how short the shirt turned out, decided to add some length both to the arms and the torso.

As noted in the pants section, there are no pattern markings at all, so figuring out where to length pieces (as well as which pieces need to be lengthened) was entirely manual. It also assumed you know how the whole pattern comes together, which I only sort of did. I guessed pretty well and ended up only having to re-cut one piece.

September 6, 2019: Cutting pieces for the jacket

By the end of the first week of September I had all the jacket pieces cut.

September 8, 2019: All jacket pieces cut

The black piping that runs down the sides of the back is all handmade by wrapping cord with bias tape.

September 10, 2019: Starting the jacket piping

Once it’s done, it looks pretty good.

September 14, 2019: The jacket piping is done

There’s some quilting that gets done on the end of each sleeve. My sewing machine doesn’t have one-inch markings up to the four inches required so I used some painter’s tape to make some temporary lines.

September 15, 2019: Quilting the sleeve ends

There’s a white stripe that goes on the left sleeve. This was another instructions challenge. The instructions say to assemble this with Stitch Witchery and attach it to the sleeve.

Problem 1: The stripe has batting in it to make it puffy. You can’t iron batting or it flattens out. Stitch Witchery needs to be ironed on high to adhere. Soooooo that’s not going to work.

Problem 2: The stripe pattern only gives you about 0.25” - 0.5” of working length on each end of the stripe. You rip open the seam on the back of the sleeve, insert one end of the stripe, wrap it around the sleeve (which is all puffy and quilted), and insert the other end of the stripe. With that small amount of working length, it doesn’t really work.

I ended up creating a “tube” of white cloth, using Stitch Witchery to adhere the black and gold bias tape on the sides, then taking the puffy batting and sliding it into the cloth tube. Totally not the way the directions explain it (or roughly diagram it) but the only way I could get it to work.

September 18, 2019: Making the left sleeve stripe

After all the stripes and quilting the sleeves were done.

September 21, 2019: Sleeves are done (front view)

September 21, 2019: Sleeves are done (back view)

The next part is the front facings - those white bits that are on the inside of the jacket.

September 22, 2019: Starting the front jacket facings

The gabardine frays really bad on the edges so you can’t see how cool it really is until it’s finished with the bias tape.

All the chain detail and snaps to hold the front together are hand sewn.

September 22, 2019: Stitching chain details to the facing edge

Once all that work is done, the jacket flap is complete! I added a snap close to the left armpit to hold the right side of the jacket in place as you close it. It seemed to be hard to keep in place otherwise; the pattern didn’t call for any of that, though.

September 28, 2019: The front jacket flap is complete

The shoulder strap is another “tube filled with batting” situation. Once you have that done, gold soutache gets sewn to the edges and the clasp goes on.

September 29, 2019: Making the shoulder strap

Looks pretty good once it’s attached.

September 29, 2019: The shoulder strap attached

There’s very little description in the pattern on how to put the lining in. There’s actually a whole step missing where it seems there’s an assumption that the lining gets inserted “somewhere around here” but no explanation of how it gets attached.

There’s no pattern requirement for shoulder pads, but the shoulders didn’t sit right without them so I added that. I can’t imagine there weren’t shoulder pads in the screen worn costumes.

Finally, because the gabardine frays so horribly, I finished it off using a Hong Kong seam with bias tape. I think it looks really good - finished without a full lining.

October 3, 2019: Jacket lining is complete

After the inside was done, it was time to attach pins, starting with the shoulder strap security device.

October 6, 2019: Security device attached to the shoulder strap

I made these “pips and squeaks” in a couple of hours on the 3D printer based on some reference measurements and photos. If you want to make these, I put the models on Thingiverse for free.

They’re placed in the same manner as Kirk had them in Star Trek 2. However, you’ll also possibly notice my rank pins and the rest of the uniform are Captain, not Admiral, while Kirk was an Admiral in ST2. I’m a little Spock and a little Kirk here. It’s not intended to be an exact replica of either.

October 7, 2019: Pips and squeaks attached to the sleeve

Overall, the jacket ran from early September to early October, so about a month.

Chest Badge

The chest badge gave me a lot of grief and was where a lot of paint and time went.

I found a pretty good Thingiverse model for the badge and figured I’d “just need to paint it.”

I learned a lot about painting here.

  • Don’t try to dilute acrylic enamel with water if you’re hand-painting it. It’ll leave air bubbles and look really lumpy.
  • Reflective metallic spray paint is really hard to work with. It can take days to set, and even then may still leave fingerprints you can’t buff out if you touch it.
  • You can’t really coat a metallic spray paint with anything. Polyurethane causes the paint to get darker because it changes the reflective properties. Lacquer doesn’t change the color but dulls it a lot.
  • Frog Tape is amazing masking tape. It’s expensive but worth it.

What I did was paint the inside bits - ivory and gray - then mask those parts off for the final metallic coat.

October 17, 2019: Masking the chest badges for painting

As you can see, I did several tests. The top one is a glittery but not too shiny paint and hand-painted insides. It’s not too smooth, but the paint doesn’t leave fingerprints. The second one has an airbrushed interior but I tried covering the reflective gold with a clear coat. The third one is an airbrushed interior with that original not-too-shiny metallic paint. The fourth one is the one I ended up using and just choosing to “not touch it” - airbrushed interior, reflective metal paint.

By the time I’d dealt with this I realize I probably should have just bought the damn badge for $20 and called it good. I did that with my rank pins. I dunno. I guess it turned into a mission.

October 19, 2019: Iterating over the chest badges

Final Product

When all was said and done, from a calendar perspective I spent four-and-a-half months on this (early June to mid-October) but I think it turned out fantastic.

October 26, 2019: The complete costume (front)

October 26, 2019: The complete costume (back)

autofac, dotnet comments edit

Back in April 2018 I posted a request for help to own some of the Autofac extension packages.

As I mentioned then, Autofac has effectively two owners: me and Alex Meyer-Gleaves. We maintain core Autofac along with the 20+ extension packages that integrate with different application types (ASP.NET Core, WCF, web forms, and more) as well as feature support packages (configuration, multitenancy). We put out the call for owners to help lighten the ever-growing load.

Since then, we’ve received a small handful of pull requests (thanks to the folks who submitted!) and, unfortunately, no takers to help out on ownership.

When it comes to pull requests, we generally get one of two flavors:

  • Very small fixes - between one and five lines, something that corrects a small error condition or fixes a documentation error.
  • Incredibly large changes - adjusting the way memory gets allocated, changing the way the container gets built, that sort of thing.

In the case of the small changes, these aren’t hard to review or accept, but in the majority case they’re also not addressing any of the issues that users have filed.

In the case of the large changes, it’s more challenging:

  • These are very hard to review. They’re both time intensive and they generally include some breaking API changes we need to consider.
  • The person submitting the change isn’t going to come back and own it if it something goes wrong. It’s a “drive-by submission.” Maybe it introduces a memory leak in an application because it inadvertently holds onto references it shouldn’t. Maybe it adds a few milliseconds on every resolve operation and now under load things are failing. The original submitter isn’t going to fix that.

Finally, there are a lot of things that seem small but are still time sinks. A good recent example is the change in the hosting model for the .NET Core conforming container. Good changes for .NET Core, but for Autofac we have to update docs, come up with examples, adjust how some things get handled, answer StackOverflow questions on it, and so on. What seems like a small change can become a non-trivial time sink.

Unfortunately, we’ve reached a point where a combination of life events, work pressure, and general OSS maintainer burnout has set in and we can’t keep up. We need someone (or several someones) who can come on board and help OWN Autofac. Someone who can review the PRs coming in, understands the challenges with breaking API changes, can help out with support, documentation… all the things I mentioned in our original request for help.

It’s not just extension packages anymore. We need help on all of Autofac - extensions, core Autofac, the whole shmear.

I can’t promise a deep mentoring experience. I apologize in advance for that. The current owners will still be involved and doing everything we can. We’ll be collaborating with anyone new and working to get new team members on-boarded. However, this is likely not a good fit for someone new to C#, .NET, or dependency injection.

Hello… Is it you we’re looking for? Take a second to check out the original post outlining what it means to be an owner. If you think that’s you, tweet us at @AutofacIoC or say hello in the Autofac Google Group.

What Helm Is

Helm is a tool used to create and deploy templates that define entities in Kubernetes. It’s kind of like taking Kubernetes YAML and adding handlebars template support. For example, you might see something like this:

apiVersion: v1
kind: ReplicationController
metadata:
  name: deis-database
  namespace: deis
  labels:
    app.kubernetes.io/managed-by: deis
spec:
  replicas: 1
  selector:
    app.kubernetes.io/name: deis-database
  template:
    metadata:
      labels:
        app.kubernetes.io/name: deis-database
    spec:
      serviceAccount: deis-database
      containers:
        - name: deis-database
          image: {{.Values.imageRegistry}}/postgres:{{.Values.dockerTag}}
          imagePullPolicy: {{.Values.pullPolicy}}
          ports:
            - containerPort: 5432
          env:
            - name: DATABASE_STORAGE
              value: {{default "minio" .Values.storage}}

In this case, you can see there are some values poked in from another YAML document that has some configuration parameters. The values document might look like this:

imageRegistry: "quay.io/deis"
dockerTag: "latest"
pullPolicy: "Always"
storage: "gcs"

When Helm does an installation or upgrade, it takes the parameters, fills in the appropriate blanks in the larger YAML deployment templates, and does the Kubernetes work to execute the deployment.

Since there are a lot of pieces to putting things in Kubernetes - maybe you have some deployments, services, etc. that all need to go in at the same time - Helm uses a concept called “charts” to bundle these up as an atomic entity. You can think of a Helm chart as a zip file with a bunch of Kubernetes YAML in it and a small manifest that explains what the zip file installs.

There are some great benefits to using Helm to deploy things:

  • Separation of deployment template from configuration. You don’t need to keep rafts of YAML around for different environments/clusters/whatever. You can have different parameters and one larger template.
  • Ability to roll back a deployment. Helm tracks the installations and upgrades you make along with the values. If you deploy something that doesn’t work, you can roll it back to the previous version.
  • Ability to list deployments and versions. You can use Helm to list out the charts and versions that have been deployed to Kubernetes. This makes things that are logically spread around namespaces into a nice, central list.
  • Charts can have dependencies. Let’s say your application needs a Redis instance when you deploy it. Cool! You can set up your chart to have a dependency on installing/upgrading Redis at the same time using the Redis Helm chart.

Things to be aware of that will come into play later:

  • Helm installations are global. When you do a Helm installation of a chart, even if it’s into a specific namespace, the installation itself is a global concept. Let’s say you want to install Redis twice (separately from your applications) - once into a test namespace and once into a prod namespace. Make sure you name those installations in a unique way - the output of the installation may go into the namespace but the installation itself is a global thing outside the namespace. (Unclear if this will change with Helm 3.0.)
  • Helm isn’t using kubectl. For version 2.0 Helm uses a service running in the cluster called tiller to do its installations. Tiller is going away in Helm 3.0 but it’s still not using kubectl - it’s using the Kubernetes API directly. What that means is if you’re looking to use kubectl later (even in an automated fashion) to modify things, you’ll get messages like Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply.
  • Helm installation tracks chart version and install version, not container version. When you install something with Helm, later you can do helm list to get the installations. Doing that, you’ll see the chart version and the data last updated but no information about what’s in that installation.

For the purposes of this article, I’m not going into exactly how Helm does its work. If you’re interested in diving deep, the docs are a good place to start. The important things to understand are some of the benefits and limitations.

Continuous Delivery vs. Continuous Deployment

Let’s talk about “CD.” Some folks differentiate between “continuous delivery” and “continuous deployment.” The apparent differentiator is that “delivery” implies software that’s been tested and is ready to be deployed to production but isn’t; while “deployment” implies another step - the software is actually automatically deployed into production and verified.

I guess you can do that, but I don’t really separate those things in my mind. If it’s ready to go to production… why isn’t it going there? I’d argue that if you differentiate then maybe it’s just that your pipeline (or your org, or your process, or whatever) just isn’t ready for a complete check-in-code to deploy-code-in-production execution. It’s an incomplete pipeline, not “two different things.”

Folks Love Helm for CD

There are a lot of articles like this one pointing out how Helm is the missing link in the CI/CD chain for Kubernetes.

It sort of is, but not the way they explain in articles. At least, not the way I see it.

The process most of the articles describe roughly follows this:

  • Create a Helm chart for your application.
  • In the CD pipeline, create a set of parameters that can be used to helm upgrade your app in Kubernetes.
  • If anything fails, use helm rollback to return to the previous state.

Seems simple enough, right? And it is simple. But this is really the “continuous delivery” sort of pipeline - the pipeline where you’re not actually trying to automate things into production. I can’t imagine you’d ever want to stomp your production deployment like this, hope that it works, and depend on helm rollback to return to a previous state if it doesn’t.

That’s a hugely important differentiator. If all you need to do is get things deployed into some sort of development/testing environment and that’s the limit of your pipeline, I guess that works. I feel like the goal should be bigger than that. I’m a fan of test in production and I’m not a fan of trying to replicate environments across dev, test, perf, and production (or however you break it up).

Given that…

Why I Don’t Like Helm for CD

If I’m thinking about actually deploying into production on a regular basis, I want support for more complex scenarios like canary testing. Let’s think about how that works for Kubernetes.

  • Existing deployment of the service in production is taking traffic.
  • New deployment of the service in production goes in alongside the existing deployment, but takes no traffic.
  • Testing against the new deployment runs internal to the cluster.
  • Traffic handling is tweaked to allow some small amount of production traffic to the new deployment whilst the majority of the traffic is still going to the original deployment. This may be accomplished in a few different ways. For example…
    • With a standard Kubernetes service, you can adjust how much traffic goes to old vs. new based on the number of deployed pods. If you want 10% of the traffic on the new and 90% on old, you’d need that proportion of pods - 1 new, 9 old.
    • With a service mesh like Istio, you can use the traffic management built in to control the percentage of traffic routed to each set of pods. This is a lot cleaner than the standard mechanism but means you have some complexity with a service mesh in the mix.
  • Testing runs on the service and both old and new versions of the deployment are monitored. If the new version starts misbehaving, all traffic is routed back to the original version of the service and the “canary” new version is killed. If the new version behaves, more traffic is directed to the new version and removed from the old version until either all traffic is pointed to the new version or the canary is killed.

How do you do something like that with Helm? In its stock form, you really can’t… or your Helm chart is going to be pretty complicated.

  • It’ll have to allow for parameterization of both existing and new deployments or you’re going to have two deployments side-by-side. Remember how deployments are a global concept? That makes it complicated.
  • You’ll have to figure out how to only have one Kubernetes service that can route to both sets of pods (old and new) or you’ll need something external to the Kubernetes load balancer to handle traffic control across the Kubernetes services.
  • If your chart has the Istio bits in it, again, you want one set of Istio load balancing/ingress controls across both pods, so the canary installation wouldn’t want to deploy those.
  • If you control traffic using Kuberenetes constructs (either by adjusting the ratio of deployed pods or tweaking Istio values) those are all helm upgrade operations. If the canary fails, how many helm rollback operations do you need to perform to get back to the original state?

There are other reasons Helm isn’t great for CD, too. Here’s a pretty good article that talks about some of the challenges and shortcomings you can hit.

Helm is Great for Infrequent Deployments

Helm is great for installing things that you don’t deploy often. Need to deploy Istio? Sweet, Helm to the rescue. You don’t update Istio every day. Need to get an ElasticSearch instance installed that your services can share? Boom! Helm, baby! You won’t be upgrading that every day.

Helm as a way to manage infrastructure or shared services is awesome. Things that don’t require canary testing, continuous rollout, that sort of thing.

Helm is Great for Templating in CD Pipelines

Helm is great as a way to package up a set of YAML and handle parameterization and some calcuations to generate a final Kubernetes YAML for deployment… and in a continuous delivery/continuous deployment context, that’s what I would recommend using it for.

A great example of this is the way Spinnaker uses Helm to deploy things. It’s not using helm install or helm upgrade - instead it can take a Helm chart or Helm-formatted YAML document and it uses helm template to generate the final template with the parameters all populated. It then takes that output and executes the deployment in Kubernetes. The note at the top of the Spinnaker “Deploy Helm Charts” page pretty much says it all:

Note: This stage is intended to help you package and deploy applications that you own, and are actively developing and redeploying frequently. It is not intended to serve as a one-time installation method for third-party packages. If that is your goal, it’s arguably better to call helm install once when bootstrapping your Kubernetes cluster.

If you think about what you get with helm list, you’re getting the chart version, right? Honestly, once you have the chart down, the chart version has no meaning in continuous deployment. The important stuff is the version(s) of the container(s) that are deployed and making up the application. That stuff isn’t tracked, so helm list becomes pretty useless.

Things like Helm chart repositories are also really not interesting. In fact, the YAML for the Helm template may just be embedded right in the CD pipeline itself, not a separate thing stored elsewhere. This allows you to separate things like tweaking Istio load balancer settings from the concept of helm install. It also avoids needing to ensure deployment names are unique.

Finally, it means helm rollback won’t save you. You’re not helm installing anything, so there’s nothing to roll back. You’ll need to ensure your CD pipeline can appropriately kill the canary. However, if you’re doing canary testing and not just stomping your existing production deployment with a new untested deployment… you should be able to easily kill the canary and leave the existing deployment untouched, with no one the wiser that something went awry.