Each month my six-year-old daughter comes home with a “homework menu” - a page that looks sort of like a bingo card but each square has a homework assignment in it. Of the 25 assignments listed, you get to pick 15 to do.

The assignments cover things like, “Write a list of words with a long ‘i’ sound in them” to “Count to 100 by fives.” First grade stuff, no problem. For the most part, she’s got this covered… as long as she doesn’t get off in the weeds.

The other night we did, “Make a list of ten odd numbers.” Easy enough. They’d been learning odds and evens in class, so this shouldn’t be much work. In fact, she finished in just a few seconds, listing all the way from one to 19 in short order.

To help cement the pattern in her mind, I thought I’d show her how all the odd numbers end in 1, 3, 5, 7, or 9. I wrote the numbers in a little table like this so the pattern would be easy to see:

``````1    11    21    31
3    13    23    33
5    15    25    35
7    17    27    37
9    19    29    39
``````

She got pretty excited about that, then asked the question that took us off the rails.

Phoenix: Daddy, why are they called “odd” and “even” numbers?

Travis: Go grab me like six alphabet blocks from your room and I’ll show you.

She ran to her room, grabbed the blocks, and returned with them bundled in the front of her shirt. She dumped the whole shirt load squarely on my nuts.

Travis (recovering): OK, you know odd numbers, do you know the even ones?

Phoenix: 2, 4, 6, 8, 10.

Travis: Good. Here’s two blocks. [I hold up two blocks.] Is two even or odd?

Phoenix: Even.

Travis: Right. If I split that into two even piles there aren’t any left over. [I put one block in each hand and hold them apart from each other.] See that? Two even piles. Now here’s three blocks. [I hold up three blocks in one hand.] Is three even or odd?

Phoenix: Odd.

Travis: Correct again. If I split that into two even piles… [I put one block in each hand and hold them apart from each other, leaving the remaining block on the arm of the chair.] …then we have an odd man out. That block on the chair is extra. Odd numbers leave an odd man out. Let’s try with four. Is four even or odd?

Phoenix: Even.

Travis: OK, and when I split them [I held two blocks in each hand] I have two even piles. Is five even or odd?

Phoenix: Odd.

Travis: Yup, and when I split them [I hold two blocks in each hand and leave the fifth on the arm of the chair] I have that odd man out. That’s why they’re “even” and “odd.” If it’s an even number, I get even piles when I split it in half. If it’s an odd number, I get an odd man out. Even number - even piles. Odd number - odd man out.

Phoenix: Oh, OK!

Travis: So, let’s review. [I hold up four blocks, two in each hand.] Four is an even number, so when I split it I have… [I pulled my hands apart with two in each hand.]…?

Phoenix: Four blocks.

Travis: Well, I do have four blocks, but we were talking about even and odd. Four is even, right? So if I split it in half, I have…?

Phoenix: Two in each hand.

Travis: That’s true, but I have even piles. Even number, even piles. Five is an odd number [I hold up five blocks in my hands] so when I split it in half [I drop one block on the arm of the chair and hold two in each of my hands] I have…? [I look down at the block on the arm of the chair hopefully]

Phoenix: An extra.

Travis: Yep, but it’s odd so we can remember it easier if we think “Odd number has an odd man out.” If it’s even you have…? [I hide the block on the chair and hold two in each hand.]

Phoenix: Equal piles.

Travis: Uh… yeah, but I really want to know you’re hearing me here, so “Even number, even piles.” Not “equal piles,” but actually the literal words, “even piles.” I need you to say, “even piles.” Sooooo…. Four is even, even numbers have…

Phoenix: Four.

Travis (really close to losing my shit): Even numbers, even piles. So, even numbers have…?

Phoenix: Even piles.

Travis: Yes, and odd numbers have…? [I put the extra block back on the arm of the chair.]

Phoenix: Odd piles.

Travis: An odd man out. Odd numbers have an odd man out. Even numbers, even piles. Odd numbers, odd man out. Even numbers…?

Phoenix: Even piles.

Travis: Odd numbers…?

Phoenix: Odd man out.

Travis: HALLELUJAH!

I think this is going to be a thing with homework this year since last night we ran into basically the same thing.

Travis: OK, Phoe, you need to draw a hexagon.

Phoenix: How many sides is that?

Travis: A hexagon has six sides.

Phoenix: So, five?

AAAAAAAAAARRRRRRRGGGGGGGGGGGGGGGGHHHHHHH

# Death Star Architecture

Today I want to talk about something I’ve seen in a few places that just frustrates me to no end: the repeated failed strangler pattern.

To make sure we’re all on the same page: The strangler pattern is when you want to overhaul an existing system and you do so by wrapping it in a facade. You swap out components under the facade from old bits to new bits, eventually strangling the old system and removing the facade so only the new stuff is left. The idea is that doing this over time potentially has lower risk than rewriting an existing system from scratch.

There are a couple of articles on this pattern here:

In general, I think this is a decent pattern. However, I take issue with the implication of this line from the Fowler article (emphasis mine):

They aren’t yet at the point where the old application is strangled - but they’ve delivered valuable functionality to the business that gives the team the credibility to go further. And even if they stop now, they have a huge return on investment - which is more than many cut-over rewrites achieve.

My challenge is around the notion that you can do a partial strangler and that’s an OK - or even good - thing.

My experience directly contradicts this. Let’s take a hypothetical example:

Let’s say you started out a long time ago, in a galaxy far, far away, with a simple string processing engine. It was long before SOAP or REST. It did what it needed to do, you sold it, and you got some customers up and running with it.

So far, so good. SOAP comes along, and XML is pretty awesome, so you decide you want to start moving away from arbitrary string-in, string-out and to something with more formal contracts. Cool! Strangler to the rescue. Except… you do still have some customers that won’t really be able to update right away… you need to leave access to the original string processing engine in place. New folks can take the new interface, though, so that’s good, right?

Turns out the strangler to convert strings to XML wasn’t quite SOAP due to some custom extensions you needed to create to make it work with the string processing engine. You still really want some SOAP wrappers on this thing, though, so you can start decoupling things and iterate faster over individual services/features. Let’s wrap the XML message handling with actual SOAP contracts that are pretty close to but not exactly like the XML messages.

Except… you sold the XML messaging to some customers and they can’t really switch to the slightly modified contracts for the services. And you really haven’t pushed those original string processor customers to upgrade yet because they’re threatening to leave if you create any breakages.

OK, this time for reals - REST is a bit lighter weight and would lend itself better to some of the new prospective clients’ needs. Getting some REST microservice support in there could really get things going, especially since most of the developers you’re hiring now are more versed in REST and that’s the direction your market is going.

(I bet you see where this is going…)

Except… now you have customers on all three previous layers: SOAP, XML messaging, and string messaging. Gotta keep access to all three of those things. No breaking changes! Ever!

Does that look at all familiar?

This is why I call it “Death Star Architecture.” You’re not finishing the strangler, so instead of getting the benefit of the pattern, you’re just adding layers to your system that all need to be maintained and tested.

Finish your strangler! In the short term it may seem like you’re getting benefits, but long term the unfinished work results in technical debt that will ultimately cause your destruction.

# Dependency Versioning in Libraries and Applications

Time to address a frequently asked question I see with Autofac and many other open source libraries I use or work on:

I’ve upgraded the (target framework/base libraries/build/.NET SDK version) for my application and now I see build warnings due to transitive dependencies. Why don’t you upgrade the library dependencies to a later version to address an issue I see in my application?

I see this a lot in the .NET Core realm, where use of many small dependencies rather than a big installed framework is a new thing. It’s not so much a question in places like Node.js where the use of many small, chained dependencies has been around for a while.

What it boils down to is that there’s a difference in how you manage dependencies in libraries and applications.

# Libraries Target Compatibility

When you have a library, you want to make sure it’s stable and compatible for your consumers, both at the outset and across upgrades. People want the latest features and bug fixes, right? This means a lot of things including:

• Interfaces need to be stable. If you change an interface, people consuming your library may not be able to take the next update. That means for any public and protected interfaces and classes you have to be very mindful about changes.
• Dependencies need to be stable. If the library takes an update to a dependency, at a minimum that means anyone taking the next version of your library is forced to take an indirect update to the dependency. It may mean forcing a breaking change onto the library consumers who may be directly referencing the dependency and can’t take that update.
• Target the lowest common denominator. That means targeting the lowest version of the .NET Framework or .NET Standard that you can. Lower target version means more compatibility. This also means targeting the lowest version of dependencies you can for the same reason - lower version means generally more compatibility, especially with folks who are already using the transitive dependency.
• Keep the target framework stable. Increasing that target .NET Standard or framework version means the new version of the library may not be compatible with existing applications - people will be forced to update their applications or may just be locked out of using the new version of the library.

As you can imagine, any changes here can cause unforeseen ripple effects. Upgrading a dependency version may fix one issue but could cause downstream consumers problems you can’t anticipate.

The general rule for dependency/framework versions is to target low and keep it stable.

# Applications Target Features

When you write an app, your largest concerns are the features you need and the target environment in which it’s going to run. It means priorities shift as far as compatibility and upgrades are concerned.

• If you need a new dependency feature, you can just take it. If you see something new you need out of a dependency - a feature, a bug fix, whatever - you can take the upgrade when you want.
• Breaking API changes are surfaced differently. This may be a REST API, a command line argument interface, or a plugin abstraction, but bringing in a new application dependency generally doesn’t cause a breaking change for application consumers.

Application dependencies generally don’t end up affecting application consumers.

By and large, when you see an issue with a transitive dependency coming into your application, the solution is to add a direct dependency in your application to the version you want.

Some scenarios you may see in the .NET Core world to help make this concrete:

• Security updates in .NET Core. When security issues are detected in .NET Core base class/framework libraries, the way you resolve that in your application is to manually take dependency updates to fixed versions. (This doesn’t mean libraries shouldn’t take these updates, but it does illustrate how application developers don’t necessarily get to delegate that responsibility away.)
• .NET Standard 2.0 Release. When a new .NET Standard release is issued, that doesn’t require every library to move to that release even if that’s what your application targets. If you look at how .NET Standard works, a higher .NET Standard means more APIs are available to you, but it also means fewer existing apps that target lower .NET Standard versions are able to use your library. You can consume lower .NET Standard libraries in higher targeting applications. For example, a library may target .NET Standard 1.3 and you can totally consume that in your .NET Standard 2.0 app.
• New dotnet SDK/CLI/build system. If you upgrade your development/build systems to use a new dotnet SDK/CLI it can start generating warnings on your applications when it finds transitive dependencies that may be stale. The solution is to add a direct reference to the later version of the stale dependency. The reason that’s the solution is that not everyone has updated to that latest version of the dotnet SDK/CLI. Libraries need to maintain backward compatibility, so forcing the update may indirectly require consumers to upgrade parts of their development or build process that they’re not ready or able to upgrade.

The point is that many of the challenges seen at the application level due to transitive dependencies can be solved by adding direct dependencies; those same challenges may not be solved in the same way at the library level.

# Wrapping It Up

Hopefully this helps clarify why libraries you consume “don’t just take a dependency upgrade” when you notice something in your application.

And, of course, a lot of this is generality. In some cases apps can’t just “take upgrades when they want” because customer environments may not support those changes. In some cases libraries can make changes to public APIs or dependency versions and it doesn’t hurt anything. There’s no hard and fast rule, but basic understanding.

If this sort of discussion interests you, Nick Blumhardt has a similar post on his blog about logging abstraction usage differences between libraries and applications - libraries use abstractions (like `Microsoft.Extensions.Logging` or `LibLog`, applications use implementations (like `Serilog` or `log4net`).

# Hosting Bower Packages in TFS Git

For better or worse, I’m trying to host a Bower package in a Git repository hosted in Team Foundation Server 2013.

Something I noticed when trying to install the package into my project is that the default Git credential manager was totally being ignored. That is, this was happening:

``````PS> bower install --save http://my.tfs.server:8080/tfs/Collection/Project/_git/my-package
bower my-package#*      not-cached http://my.tfs.server:8080/tfs/Collection/Project/_git/my-package#*
bower my-package#*         resolve http://my.tfs.server:8080/tfs/Collection/Project/_git/my-package#*
bower my-package#*           EHTTP Status code of 401
``````

I had authenticated to the TFS server before and credentials should have been stored using the Windows Credential Manager. Doing a `git ls-remote` on the repo didn’t prompt me for credentials.

The answer, as it turns out, is to prefix the URL with `git+` and suddenly the credential manager kicks in.

``````PS> bower install --save git+http://my.tfs.server:8080/tfs/Collection/Project/_git/my-package
bower my-package#*      not-cached http://my.tfs.server:8080/tfs/Collection/Project/_git/my-package#*
bower my-package#*         resolve http://my.tfs.server:8080/tfs/Collection/Project/_git/my-package#*
bower my-package#*        checkout 1.2.3
bower my-package#*        progress Receiving objects:  14% (15702/112154), 71.73 MiB | 9.57 MiB/s
bower my-package#*        progress Receiving objects:  20% (22431/112154), 81.10 MiB | 9.51 MiB/s
...
``````

Pre-emptive disclaimer: This set of anecdotes doesn’t refer to a specific product or a specific repository. Some things come from past life experience, some come from open source projects, some come from other places I’ve encountered. As they say in the movies, “The story, all names, characters, and incidents portrayed in this production are fictitious. No identification with actual persons (living or deceased), places, buildings, and products is intended or should be inferred.”

Some of what I do involves continuous integration and continuous delivery, build tooling, that sort of thing. A particular pain inflicted on me is when I run into a monolith repository - a huge, multi-project, multi-solution repo with tons of different things that are all intended to work together so they all live in one source code repository.

Autofac was in one of these monoliths until a few years ago in our move to GitHub. There were good reasons for it to be that way, but in the end we split it up due to some of the challenges.

Here are a few of the challenges I’ve seen in working with monolith repos.

# Slow Builds

Once a repository gets to a certain size the time it takes to build becomes unbearably long. I’ve seen estimates of about 10 - 15 minutes being a reasonable build time including tests. I think that’s about right. Much longer and folks stop building everything.

Don’t get me wrong, it’s OK to let the build server do its job and offload the task of running huge builds to detect things that are broken. What’s not OK is when your developers can’t build the whole thing - either because the build run takes several hours or because various tooling changes and build optimizations (see below) have made it so only build agents have the right configuration to enable a build to get out the door.

# “Views” on the Source

If it takes too long to build, or the source is so big it’s hard to work in, folks end up working in logical “views” over the source. For the most part this is a reasonable thing to consider, but this has an unforeseen effect on how you build, which leads to some of the challenges I’ll explain below: incorrect build optimizations, inconsistent tools, non-modular modules, and so on.

This becomes more challenging when people want their own custom views. For example, a person who might be working on a specific application also wants to be able to refactor things in dependencies of that application. Creating that custom “view” can give the impression that it’s OK to just change things within that view and that it won’t have any impact on anything outside.

# Incorrect Build Optimizations

Once you have slow builds going on, folks are going to want to speed the build up, especially for their “view” (if you went that direction).

One way you might try optimizing is to run all the tests in a separate build - build one time to compile, provide feedback on whether it even compiled or not, and then run tests with additional feedback on pass/fail. The problem here is that it’s pretty easy to get something compiling that totally won’t work at runtime. How long do those tests take? Who caused them to fail? If you have too many compilation builds queuing up for tests to run, you’ll end up batching them which means finger pointing. “I’m sure it wasn’t me, so someone else can fix it.”

Another way you might try optimizing is to use package management systems to handle dependencies. Each “view” can build independently and in any order, so you can have separate builds if you want, one for each “view” or logical component. That’s actually a pretty decent idea as long as you do that consistently and with a lot of discipline… and as long as you acknowledge that building everything in the entire repository won’t necessarily ensure that the latest versions of everything work together. With package management, the build output of this part of the build won’t necessarily be the build input to some other part.

# Stale and Inconsistent Tools

Once the build gets too large or complicated, it can be an easy pattern to fall into to just tweak one or two code files, check them in, and let the build server do all the work to verify things.

However, if developers never actually run the build, the developers may start using newer tools than the build agent uses, causing the actual build tooling to become stale.

Alternatively, different developers or teams working in different areas of the monolith may want to update the tooling used to build their “view” of the monolith repo. As long as you don’t want to build everything together, that’s fine. Chances are, though, the decision of the team on their component now puts additional requirements on build agents and makes it so all devs can’t build the whole set of components.

A great example of this is MSBuild and Visual Studio: Say you start your project when Visual Studio 2010 is out. As time goes on, developers update their machines to Visual Studio 2012, 2013, and beyond. Parts of the monolith that don’t change much remain back on 2010 while people move on. Eventually no developer has VS 2010 installed and could never actually build the solution. In the meantime, the build agent requires all of those Visual Studio versions be installed because there wasn’t a concerted effort to unify on a version.

Another MSBuild / Visual Studio example: Again, say you start your project with Visual Studio 2010. New features and functions get added but in an effort to not change tooling for everyone, people start manually tweaking files used by the tooling - solution and project files. Now you have a situation where if you actually open the solution in VS 2010 you get an error saying there are project types in the solution that aren’t supported… but if you open the solution in a newer version of Visual Studio you get a notice that the solution is on old tooling and must be upgraded to work.

Generally a build of a codebase corresponds to a single version of something. In the case where the code has multiple logical applications or components, the build might correspond to a single release of those things.

In a more microservice scenario, you likely wouldn’t have each microservice building and deploying as part of a monolith repository. Instead you might have a “build” that continuously runs integration tests over the deployed microservices to ensure they’re still working together as expected.

Anyway, since the build number (or build version, or whatever) can really only track one version number at a time, you have three choices, none of which are awesome:

1. Version everything together. That means if you change one line in one component and the build kicks off, the version on every component in the entire repo changes. If you only ever deploy the entire monolith at the same time and it all builds together (e.g., not through a package management mechanism or using “views” on the repo), that can work.
2. Implement an alternative independent versioning mechanism. In this case you’d have to figure out a custom way to indicate the version of each component in the system that can “version independently.” That version will not be tied to the build server version/number. You may build the same version of a component multiple times if the overall build gets kicked off and the component version hasn’t been incremented. This gets more complex if you want to see in which build a particular component version originated.
3. Never change the version. This really only works if it’s a small project and everything is entirely, like, “software as a service” or something where you always only ever have the latest version deployed and you never have to report on it.

We tried ideas one and two in Autofac before splitting the repos in GitHub. I’ve seen idea three in other projects.

# Circular Builds

If you build as a monolith, it’s really easy to accidentally create a circular build dependency.

Say you have some custom build tasks or scripts that help you with your build, like custom MSBuild tasks. That’s fine as long as the custom tasks are entirely independent of the code they’re building. However, say you have a custom build task that has a dependency on one of the assemblies being built… and that assembly being built requires the use of the build tasks to succeed.

Bad times. You need to unravel that.

Package management decoupling can also make it a piece of cake in the monolith to create a circular build dependency. Component A takes package B as part of its dependencies. Component A builds, publishes package A. Now component B builds… and takes package A as a dependency. This can be a really hard thing to detect, especially if package A and package B don’t themselves properly declare package dependencies.

# Non-Modular Modules

Once the build gets broken up into logical components, applications, microservices, or modules, it’s far too easy to “just add a dependency” on some other piece of the source code repository and ignore the application and process isolation required to ensure that module actually stays modular. A lot of times you’ll see inconsistent application of dependencies - some come from a package management system, some come from the local repository’s build output from some other component.

# Ever-Changing Framework

If the shared libraries or shared dependencies you use are in the same repo as your consuming components, it takes a lot of discipline to not start new functionality by instantly putting new items right in the common code. Your new application or component is going to need to validate phone numbers, why wouldn’t you add a whole shareable framework component for phone number validation? Hey, just throw that in the lowest level dependency so it’s readily available anywhere at any time!

Of course, that means from that point on anyone using your shared library will assume the new functionality is there and removing it will be a breaking change… so… uh… maybe that’s not the best idea.

# It Failed, but Really Succeeded!

The build server may say the build failed, but if you build “a view” of the monolith in isolation, that same piece may actually succeed. Which one is right? Is it a problem with the overall ordering of how the repository builds the logical components? Is the build server actually the system of record anymore?

# It Succeeded, but Really Failed!

After a certain level of complexity gets introduced, it gets pretty easy to start ignoring warnings that get generated or inadvertently cause errors to get ignored.

For example, say your build uses MSBuild in some areas and PowerShell in others. MSBuild calls a PowerShell script which then ends up calling MSBuild. Errors reported in that innermost MSBuild execution may not actually cause the overall build to fail… which means the build will show as successful even though it’s not.

# Illig’s Law of Monolithic Repositories

The amount of discipline required to maintain a build is directly proportional to the size of the source code repository. The amount of discipline actually used is inversely proportional.

Most of the monolith repository problems you see could be avoided with enough developer due diligence and discipline. However, the larger a repository gets, the less personal responsibility folks start to feel for keeping the build performing and running clean. It’s too easy to complain about the size and complexity, passing general housekeeping off as technical debt to be addressed later. Eventually people become complacent (“That’s just how it is, we can’t fix it.”) and nothing ever does get fixed.

It’s the opposite of “too big to fail”: It’s “too big to fix.”