net, json, xml comments edit

When .NET Core was released, a new configuration mechanism came with it: Microsoft.Extensions.Configuration. It’s an improvement over the System.Configuration namespace in a lot of ways and much simpler to use, but there is still a lot to know to effectively take advantage of the features. This post tries to clarify some of the usage patterns and how the new system works based on questions and common issues I’ve seen “in the wild.”

Also, it’s definitely worth looking at the official docs since there are great examples in there, too.

As of this writing, .NET Core 2.1.1 is out. That’s the version I’ll be writing about here. If you show up in a year or three, this could be out of date.

Everything is Key/Value

The most important thing to know about the new configuration system is that everything boils down to key/value pairs. You may have a pseudo-hierarchy of these key/value pairs so you can walk it like a tree, but in the end it’s still key/value pairs, like a dictionary. No matter the input format, it all gets normalized.

  • Keys are case-insensitive.
  • Values are strings.
  • The hierarchy delimiter is : when querying parsed configuration.
  • Every configuration provider flattens their structure down to the same normalized format.

Let’s look at some samples and see how they flatten out.

Here’s a JSON file with some configuration:

{
  "logging": {
    "enabled": true,
    "level": "Debug"
  },
  "components": {
    "database": {
      "connection": "connection-string"
    },
    "files": {
      "path": "/etc/path"
    }
  }
}

When this flattens out, you get:

components:database:connection = "connection-string"
components:files:path = "/etc/path"
logging:enabled = "True"
logging:level = "Debug"

I’ve sorted the configuration keys for easier reading. As you can see, it’s all strings. The Boolean true has been converted to its string representation True.

Let’s look at the same configuration but in XML format:

<?xml version="1.0" encoding="utf-8" ?>
<root>
    <logging enabled="True">
        <level>Debug</level>
    </logging>
    <components>
        <database connection="connection-string" />
        <files path="/etc/path" />
    </components>
</root>

Things to notice in the XML format:

  • The <root> element is throwaway. Configuration ignores the root element.
  • You can specify child configuration items as attributes or as child elements with text.
  • The logging:enabled setting in XML is True to generate the same output as the JSON. If it had been true, since it’s a string, the parsed output would have had logging:enabled = "true".

Something important to note is that name has a special meaning in XML configuration. If you add a name attribute to an XML element it uniquely identifies that element. We’ll get more into that later with Ordinal Collections and Advanced XML.

Now that we’ve seen XML, how about INI format?

[logging]
enabled=True
level=Debug

[components]
files:path=/etc/path

[components:database]
connection=connection-string

Things to notice in the INI format:

  • You can put a : in headings or in keys and it’ll generate the proper flattened format.
  • As with XML, the logging:enabled setting is True since it’s a string by default and won’t be seen as a Boolean.

You can specify configuration as environment variables! Since : doesn’t work well in environment variables in all systems, you use __ in the actual environment variable and it will get converted.

set COMPONENTS__DATABASE__CONNECTION=connection-string
set COMPONENTS__FILES__PATH=/etc/path
set LOGGING__ENABLED=True
set LOGGING__LEVEL=Debug

This will generate the keys in all caps, but that’s OK because keys are case insensitive. You can still access them using lower case names like logging:enabled and you’ll get the right thing.

Note by default the environment variable configuration source will bring all environment variables in. Maybe you want that, maybe you don’t. I’ll talk later about Environment Variable Prefixes to show you how to filter and only get what you want.

And let’s finish up with command line parameters:

mycommand.exe --components:database:connection=connection-string --components:files:path=/etc/path --logging:enabled=True --logging:level=Debug

Each switch gets converted to be a key and the value after the equals sign is the value. You can also do space delimited:

mycommand.exe --components:database:connection connection-string --components:files:path /etc/path --logging:enabled True --logging:level Debug

Don’t mix and match. If some things have equals and some don’t, weird things happen. For example, this:

mycommand.exe --badswitch --goodswitch=value

Yields this config:

badswitch = "--goodswitch=value"

I’ll talk more about this later in the “Advanced Command Line” section.

Overriding Values

One of the coolest things (I think) about the way the Microsoft Configuration system works is that you can use all these providers and set up a configuration precedence to layer differnet config sources on top of each other.

For example, a common way to go in ASP.NET Core is:

  • Base appsettings.json
  • Environment-specific appsettings.{EnvironmentName}.json
  • Environment variables
  • Command line parameters

In this manner, the application can ship with some core defaults (appsettings.json). If you have environment-specific settings that don’t need to change (without re-deploying a file) you could have an environment-specific JSON file like appsettings.Staging.json or appsettings.Production.json. Layer some environment variables on the top for things that get wired up on the fly (like URLs to services) or maybe API keys/secrets that need to be used by the app. Finally, last-minute must-have overrides that can go on the command line, like which port to listen on.

Building that up might look like this:

var config = new ConfigurationBuilder()
  .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
  .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true, reloadOnChange: true)
  .AddEnvironmentVariables()
  .AddCommandLine(args)
  .Build();

Let’s see what happens when we layer some configuration on. Pretend we’re in the Staging environment and here’s what’s out there:

appsettings.json has…

{
  "logging": {
    "includeScopes": false,
    "logLevel": {
      "default": "Debug"
    }
  }
}

appsettings.Staging.json has…

{
  "logging": {
    "logLevel": {
      "default": "Warning"
    }
  }
}

Environment variables:

set ASPNETCORE_ENVIRONMENT=Staging
set LOGGING__LOGLEVEL__MICROSOFT=Information

Command line arguments:

dotnet run --urls=http://*:5005

What does that yield?

aspnetcore_environment = "Staging"
logging:includescopes = "False"
logging:loglevel:default = "Warning"
logging:loglevel:microsoft = "Information"
urls = "http://*:5005"

There are a few interesting things here.

  • If you use the same key in an override configuration, it will replace the corresponding value.
  • You can add configuration at override time, but you can’t remove things. The best you can do is override a value with an empty string.
  • The ASPNETCORE_ENVIRONMENT value is Staging, which is a sort of magic value for ASP.NET Core. I’ll talk more about that later in Environment Variable Prefixes. That said, you’ll also see it in the config as a flattened value if you choose to import everything. It gets the underscore in the name because it’s not double-underscore __ (the usual delimiter).
  • The urls value will be picked up by the ASP.NET Core web host and that’s what port it’ll listen on. If you add arguments to your config, you’ll see it, too.

I mentioned overriding with an empty string as a “fake way” to remove configuration. Specifying null as a value, even in JSON config, doesn’t work. It reads the value and uses an empty string as the value instead of null. Further, there’s no XML analogy to null, nor are there such analogies in most other providers.

Given everything in the config system is a key/value string pair, the closest you can get, then, is to set things to empty strings. When you read values, check for String.IsNullOrEmpty() before using the value.

Since you can’t remove things, specify as little configuration as possible and behave correctly using defaults in your code when configuration isn’t there. This will save you a lot of time when you have some base configuration specifying a value that you don’t want and you can’t figure out how to override and “remove it.”

Ordinal Collections (Arrays)

Ordinal collections (think arrays) are a sort of interesting special case in configuration. It’s pretty easy to think about it when using JSON like this:

{
    "components": [{
        "database": {
            "enabled": true
        }
    }, {
        "files": {
            "enabled": false
        }
    }]
}

It’s an array of two objects. But how does that flatten out into key/value pairs?

This is a big one, since JSON, INI, XML, environment variables, command line parameters, and other config sources all need to work together. You don’t have “arrays” in environment variables. So what does it look like?

The answer is that numeric 0-based keys get generated for each element. The flattened config looks like this:

components:0:database:enabled = "True"
components:1:files:enabled = "False"

Knowing how this works is huge because when you try to intermingle different configuration formats and override values, you have to generate the same key structure.

Let’s look at the same thing in XML:

<?xml version="1.0" encoding="utf-8" ?>
<root>
    <components name="0">
        <database enabled="True" />
    </components>
    <components name="1">
        <files enabled="False" />
    </components>
</root>

Notice in XML we had to manually specify the numeric key. As mentioned earlier, name has a special meaning in XML configuration. If you add a name attribute to an XML element it uniquely identifies that element. For ordinal collections the name is the index in the collection.

For reference let’s look at some bad XML configuration for ordinal collections:

<?xml version="1.0" encoding="utf-8" ?>
<!-- THIS WON'T WORK! MISSING NAMES! -->
<root>
    <components>
        <database enabled="True" />
    </components>
    <components>
        <files enabled="False" />
    </components>
</root>

This version missing names will generate:

components:database:enabled = "True"
components:files:enabled = "False"

Notice it’s missing the index part of the key. If you had JSON and XML config in the same system, the overrides would fail.

Let’s look at INI:

[components:0]
database:enabled=True

[components:1]
files:enabled=False

INI files don’t have a notion of ordinal collections so you need to manually specify the indexes in the keys.

Environment variables also get manual specification:

set COMPONENTS__0__DATABASE__ENABLED=True
set COMPONENTS__1__FILES__ENABLED=False

So do command line parameters:

mycommand.exe --components:0:database:enabled=True --components:0:files:enabled=False

What happens if you skip an index?

<?xml version="1.0" encoding="utf-8" ?>
<root>
    <components name="1">
        <database enabled="True" />
    </components>
    <components name="4">
        <files enabled="False" />
    </components>
</root>

It doesn’t matter. It’s all just string key/value. The above XML will become:

components:1:database:enabled = "True"
components:4:files:enabled = "False"

Given that, how do we override things?

Let’s say we start with a JSON file like we had before:

{
    "components": [{
        "database": {
            "enabled": true
        }
    }, {
        "files": {
            "enabled": false
        }
    }]
}

We want to enable the files component at runtime via the environment. To do that, we’d set an environment variable like this:

set COMPONENTS__1__FILES__ENABLED=True

If you then layer environment variables over the JSON configuration you’ll get the desired effect:

components:0:database:enabled = "True"
components:1:files:enabled = "True"

How would you layer two JSON files to get this to work?

You have to fake out the indexing mechanism. You can do this one of two ways. First, you can use empty objects to pad out your overrides file:

{
    "components": [{}, {
        "files": {
            "enabled": true
        }
    }, ]
}

The presence of the empty object there pushes the index of the “files” object forward so it matches the original index.

The other option is to just specify the index right in a key:

{
    "components:1": {
        "files": {
            "enabled": true
        }
    }
}

Notice the index is in the key and there’s no array at all. Either way it will flatten out and get the desired result.

The complexity around ordinal collections is something to consider when you’re picking a configuration format. Especially if you want to override something in the environment or via command line at runtime, you’ll have to know which index to use in your override.

No Built-In Validation

One of the things that made the old System.Configuration mechanism nice was the ability to put pretty rich type converters, default values, and configuration validation into the ConfigurationSection you write. The new Microsoft.Extensions.Configuration mechanism doesn’t have any of this. You can sort of fake it by binding configuration to objects (which I’ll cover later) but there’s no notion of configuration “schema” or any sort of validation/annotation you can provide to ensure values are in a form you expect.

To that end, make sure you validate your configuration values before using them. Maybe that’s parsing things into strong object models (see below), maybe it’s checking that values are in an expected format. It’s not built in so it’s up to you. In the next section I’ll show you an example of how to use Data Annotations validation with configuration binding.

Also, use String.IsNullOrEmpty() to check for presence/absence of values. If you only check against null then you won’t be able to “remove” configuration later if you need to by setting it to an empty string.

Configuration Object Model

It really helps to know how to navigate around the configuration object model if you’re going to do more than read just a small set of values.

You start your configuration with a ConfigurationBuilder. This is the object to which you’ll attach the various things providing configuration.

var builder = new ConfigurationBuilder();

builder.AddJsonFile("appsettings.json")
       .AddXmlFile("appsettings.xml")
       .AddIniFile("appsettings.ini")
       .AddEnvironmentVariables()
       .AddCommandLine(args);

The builder doesn’t actually invoke any of the things that read configuration. It just gives you an opportunity to specify your configuration sources. It also has a Properties dictionary on it that you can use when people register sources. For example, if you wanted to ensure a particular source only gets registered one time with a ConfigurationBuilder, you could synchronize on that Properties dictionary.

Each time you call one of the extensions (AddJsonFile, AddXmlFile), it adds an IConfigurationSource to the ConfigurationBuilder. The ConfigurationBuilder keeps track of these until you tell it to Build().

Each IConfigurationSource is responsible for building an IConfigurationProvider. When ConfigurationBuilder.Build() is called, each IConfigurationSource.Build() is called in turn to build the configuration providers.

An IConfigurationProvider is what actually reads in and parses the configuration. It provides a normalized view on top of the configuration so the system can query the key/value pairs and have them all nice, flat, and colon-delimited.

You have a set of these configuration providers, so you need something to handle the “merging” of all the providers and provide that single, unified view. This is where the IConfigurationRoot comes in. It keeps track of the final, built set of providers. This is what comes out of ConfigurationBuilder.Build().

When you ask an IConfigurationRoot for a configuration item, it iterates through the set of providers (in reverse order - that’s how the “override” works) until it finds the value, then returns the first value found. If no value is found, it returns null.

From the IConfigurationRoot you can ask directly for a key like logging:level or you can ask for an IConfigurationSection, which gives you a localized view of a sub-tree of the configuration. root.GetSection("logging") will get you the part of configuration that starts with logging:.

Everything under the IConfigurationRoot is an IConfigurationSection. This is where you’ll spend most of your time. An IConfigurationSection has these methods and properties:

  • Key: The local config key based on the current section. If you were looking at the logging:level section, the Key would be level.
  • Path: The full path to the key from the root of config. This would be like logging:level.
  • Value: If the configuration key has a value, this is it. Otherwise this value will be null.
  • this[key]: The configuration value of a child of this section.
  • GetSection(key): Gets a child IConfigurationSection of this section.
  • GetChildren(): Gets the set of all child IConfigurationSection values from this section.
  • GetReloadToken(): Gets the change token that the configuration system is watching for configuration changes.

This is easier to see if we look at some code. Let’s say we have the following configuration:

appsettings.json:

{
  "debug": true,
  "logging": {
    "includeScopes": false,
    "logLevel": {
      "default": "Debug"
    }
  }
}

overrides.json:

{
  "logging": {
    "logLevel": {
      "default": "Warning"
    }
  }
}

The flattened configuration will look like:

debug = "True"
logging:includescopes = "False"
logging:loglevel:default = "Warning"

Let’s build the configuration and wander around.

// This will track the configuration sources we'll merge.
var builder = new ConfigurationBuilder();

// Add two JsonConfigurationProviders to the builder.
builder.AddJsonFile("appsettings.json")
       .AddJsonFile("overrides.json");

// Run through all the providers and build up the sources.
// This will actually read the JSON files and parse them.
var configRoot = builder.Build();

// Ask for a value by absolute path. This will be
// the string "False"
var includeScopes = configRoot["logging:includescopes"]

// Grab the logging section so we can look at it.
var loggingSection = configRoot.GetSection("logging");
foreach(var child in loggingSection.GetChildren())
{
    // Inspect the path, key, and value of each child.
    // This will output:
    //
    // logging:includeScopes (includeScopes) = False
    // logging:logLevel (logLevel) = (null)
    //
    // Notice the path is the absolute path, the key is
    // relative to the parent section. Also notice it's
    // only _immediate_ children you get.
    Console.WriteLine(
      "{0} ({1}) = {2}",
      child.Path,
      child.Key,
      child.Value ?? "(null)")
}

// You can get the logging:loglevel section by absolute path...
var logLevelSection = configRoot.GetSection("logging:loglevel");
// ...or you can get it as a child of the logging section we got earlier...
logLevelSection = loggingSection.GetSection("loglevel");

// Now we can look at the children again:
foreach(var child in logLevelSection.GetChildren())
{
    // This will output:
    //
    // logging:logLevel:default (default) = Warning
    Console.WriteLine(
      "{0} ({1}) = {2}",
      child.Path,
      child.Key,
      child.Value ?? "(null)")
}

// What happens if we ask for something that doesn't exist?
// WE GET BACK AN EMPTY SECTION, NOT AN EXCEPTION. You can't
// ask for a specific section by name to test if it was defined
// in configuration.
var doesNotExist = configRoot.GetSection("does:not:exist");

// This will output:
//
// does:not:exist (exist) = (null)
Console.WriteLine(
  "{0} ({1}) = {2}",
  doesNotExist.Path,
  doesNotExist.Key,
  doesNotExist.Value ?? "(null)")

// If you absolutely need to check to see if something _exists_
// rather than if it's just null (undefined), you need to look at
// the _parent section_ to see if it has a child with the name.
// Make sure you do a case-insensitive comparison! Keys aren't
// case sensitive.
var checkSectionExists =
    configRoot.GetSection("does:not")
              .GetChildren()
              .Any(c =>
                   c.Key.Equals("exist", StringComparison.OrdinalIgnoreCase));

As you can see, there is a lot of power in the new system, but also a little challenge with respect to checking for values or existence of an item. Also, since everything comes out as strings, it means a lot of parsing… but hold on, check this next section out.

Binding to Objects

Since everything comes out as strings one of the first things you’re likely going to do is parse the strings into strongly typed objects. You’re going to have a lot of Int32.TryParse() and stuff all over the place. But wait! There’s another package for you!

Microsoft.Extensions.Configuration.Binder brings the parsing to you. If you don’t want to stay in string-land, this package adds helpful configuration binding/conversion to the mix.

Let’s say we have a configuration that looks like this:

debug = "True"
logging:includescopes = "False"
logging:loglevel:default = "Warning"
logging:maxmessagelength = "255"

Cool. Once we’ve built up the configuration root, we can start getting parsed values pretty easily:

// maxMessageLength will be the integer 255
// Note keys are case-insensitive so we can use camelCase here
// in our code if we really want.
var maxMessageLength = configRoot.GetValue<int>("logging:maxMessageLength");

// debug will be the Boolean true
var debug = configRoot.GetValue<bool>("debug");

// includeScopes will be the Boolean false
var loggingSection = configRoot.GetSection("logging");
var includeScopes = loggingSection.GetValue<bool>("includeScopes");

OK, that works well for getting one-off values. What if we want to get a whole section of values at the same time? Sure. You can create a couple of object classes like this:

public class LoggingSection
{
    public bool IncludeScopes { get; set; }
    public int MaxMessageLength { get; set; }
    public LogLevelSettings LogLevel { get; set; }
}

public class LogLevelSettings
{
    public Microsoft.Extensions.Logging.LogLevel Default { get; set; }
    public Microsoft.Extensions.Logging.LogLevel Identity { get; set; }
}

Now you can read and parse the whole thing at once.

var logSettings = configRoot.GetSection("logging").Get<LoggingSection>();

// This will output:
//
// Max message length: 255
// Include scopes: False
// Default level: Warning
// Identity level: Trace
Console.WriteLine("Max message length: {0}", logSettings.MaxMessageLength);
Console.WriteLine("Include scopes: {0}", logSettings.IncludeScopes);
Console.WriteLine("Default level: {0}", logSettings.LogLevel.Default);
Console.WriteLine("Identity level: {0}", logSettings.LogLevel.Identity);

That’s pretty cool, right?

But, wait, how did logSettings.LogLevel.Identity become LogLevel.Trace?

You noticed there’s no configuration for that. Since that’s not defined the value becomes default(T), in this case, the default value for the Microsoft.Extensions.Logging.LogLevel enumeration, which is LogLevel.Trace.

Under the covers, the configuration binder is mostly using type converters obtained from System.ComponentModel.TypeDescriptor.GetConverter(Type) to convert values from string. If you need more robust support, like deserializing list values or custom types, you’ll need to implement base .NET type conversion on your type and enable string deserialization.

Anyway, once you have it deserialized, you can validate on the values if needed. For example, you could use the System.ComponentModel.DataAnnotations validation pretty easily with some attributes…

public class LoggingSection
{
    public bool IncludeScopes { get; set; }

    // The max length has to be between 10 and 100...
    // but the configuration is set at 255 right now!
    [Range(10, 100)]
    public int MaxMessageLength { get; set; }
    public LogLevelSettings LogLevel { get; set; }
}

Then run the validation yourself:

var loggingSection = config.GetSection("logging").Get<LoggingSection>();
var validationResults = new List<ValidationResult>();

// This will output:
//
// Logging configuration is invalid!
// - The field MaxMessageLength must be between 10 and 100.
if (!Validator.TryValidateObject(
      loggingSection,
      new ValidationContext(loggingSection),
      validationResults,
      true))
{
  Console.WriteLine("Logging configuration is invalid!");
  foreach (var validationResult in validationResults)
  {
    Console.WriteLine("- {0}", validationResult.ErrorMessage);
  }
}

Default Providers

Out of the box there are several providers shipped (you can always see the latest in the repo):

  • Azure Key Vault (bind secrets from the vault into your app)
  • Command Line
  • Environment Variables
  • INI Files
  • JSON Files
  • Key-Per-File (bind a directory of files as config - file name is the key, file contents are the value)
  • User Secrets (dev-time secrets that normally get replaced by something like Azure Key Vault in production)
  • XML Files

Any or all of these can be used together to layer configuration into your application. You can also write your own custom provider to, for example, read configuration out of Redis or SQL.

Refreshing on Change

When configuration is initially built the usual pattern is to read all the configuration from the underlying source and store it in-memory in a dictionary for later retrieval.

Some (but not all) configuration providers allow for the configuration source to change at runtime and have that source reload its configuration to pick up changes.

Out of the box, the providers that allow for reload on change are the file-based providers:

  • INI Files
  • JSON Files
  • XML Files

Conversely, some providers do not handle change events. In this case, you’d have to manually rebuild configuration or handle the change/update yourself. Usually this is either because there’s not a reasonable way to receive a change event; or it doesn’t make sense that the values would change during runtime:

  • Azure Key Vault
  • Command Line
  • Environment Variables
  • Key-Per-File
  • User Secrets (the underlying provider here is JSON files but the code specifically turns off the reload on change flag)

Environment Variable Prefixes

When you use environment variables as a configuration source, you may get a lot of junk you didn’t really want. To avoid that, you can signal the provider to only include variables that have a given prefix.

Let’s say you have these environment variables:

set RANDOM_VALUE=BlipBlipBlip
set COMPONENTS__DATABASE__CONNECTION=connection-string
set COMPONENTS__FILES__PATH=/etc/path
set LOGGING__ENABLED=True
set LOGGING__LEVEL=Debug

If you added all the environment variables like this…

var config = new ConfigurationBuilder()
    .AddEnvironmentVariables()
    .Build();

You’d get everything, even maybe stuff you don’t want:

components:database:connection = "connection-string"
components:files:path = "/etc/path"
logging:enabled = "True"
logging:level = "Debug"
random_value = "BlipBlipBlip"

Uh oh, that random_value snuck in. Instead, let’s prefix the variables we like:

set RANDOM_VALUE=BlipBlipBlip
set CONFIGURATION_COMPONENTS__DATABASE__CONNECTION=connection-string
set CONFIGURATION_COMPONENTS__FILES__PATH=/etc/path
set CONFIGURATION_LOGGING__ENABLED=True
set CONFIGURATION_LOGGING__LEVEL=Debug

Now specify a prefix in the configuration build:

var config = new ConfigurationBuilder()
    .AddEnvironmentVariables("CONFIGURATION_")
    .Build();

The prefix gets trimmed off and the correct environment variables make it in:

components:database:connection = "connection-string"
components:files:path = "/etc/path"
logging:enabled = "True"
logging:level = "Debug"

There are some prefixes that have special meaning.

Out of the box, there are several prefixes that automatically provide help in configuration of connection strings.

  • MYSQLCONNSTR_ - MySQL connection string
  • SQLAZURECONNSTR_ - SQL Azure connection string
  • SQLCONNSTR_ - SQL connection string
  • CUSTOMCONNSTR_ - Custom provider connection string

If you use these, it deserializes differently.

set MYSQLCONNSTR_MYSQLDATABASE=connect-to-mysql
set SQLAZURECONNSTR_SQLAZUREDATABASE=connect-to-sqlazure
set SQLCONNSTR_SQLDATABASE=connect-to-sql
set CUSTOMCONNSTR_CUSTOMRESOURCE=connect-to-custom

If you bring all of the environment variables in (don’t specify a prefix) you get:

connectionstrings:customresource = "connect-to-custom"
connectionstrings:mysqldatabase = "connect-to-mysql"
connectionstrings:mysqldatabase_providername = "MySql.Data.MySqlClient"
connectionstrings:sqlazuredatabase = "connect-to-sqlazure"
connectionstrings:sqlazuredatabase_providername = "System.Data.SqlClient"
connectionstrings:sqldatabase = "connect-to-sql"
connectionstrings:sqldatabase_providername = "System.Data.SqlClient"

The prefix indicates which database provider will be automatically added to configuration. This appears to be primarily useful in moving Azure environment variables over to the application environment.

ASP.NET Core also has several things it uses in the ASPNETCORE_ prefix. The complete list is in the documentation but some common ones are:

  • ASPNETCORE_ENVIRONMENT: Defines the environment name. Typically Development, Staging, or Production.
  • ASPNETCORE_URLS: Defines the addresses/ports on which the server should listen.
  • ASPNETCORE_WEBROOT: If your web root isn’t wwwroot, this override can be used to point to the new location.

Advanced Command Line

When you add command line parameters to configuration, the parser lets you use one of two formats:

  • Each argument is a pair separated by = like --key=value
  • Argument pairs can alternate key/value/key/value with spaces between each, like /key value

It doesn’t matter which format you use, but you have to pick one. You can’t mix and match.

Rules for specifying keys:

  • Keys can start with -, --, or / (and you can mix/match these)
  • If a key starts with - it’s considered to be a shortened version of a longer key so you need to provide a mapping.
  • If you specify the same key on the command line twice, last in wins.

Here are some examples:

# YES
myapp.exe --key1 value1 /key2 value2 -k3 value3
myapp.exe --key1=value1 --key2=value2 /key3=value3

# NO (can't use both space and equals for delimiters)
myapp.exe --key1 value1 /key2=value2

Notice in that first example line there’s a short switch -k3. To use that, we’d need to provide a mapping to a longer key or we’ll get an exception.

var mappings = new Dictionary<string, string>
{
    // Be sure to include the leading dash on the key
    // but leave it off the value!
    { "-k3", "key3" }
};
var config = new ConfigurationBuilder()
    .AddCommandLine(args, mappings)
    .Build();

Finally, be aware that, like with environment variable prefixes, some systems you work with will find special meaning in some command line switches. For example, if you’re working with ASP.NET Core, the web host accepts a --urls command line parameter as the set of locations on which it should listen.

There’s no built-in filtering for command line arguments the way there is for environment variables with prefixes, so be aware when you’re building your app.

Advanced XML

Given the previous .NET configuration system was so rooted in XML it’s understandable that many folks coming in start out looking at how to transform existing XML config into new XML config.

It’s your call if you decide to do that, but be aware of a few things when using XML:

  • You can’t include a DTD or parsing will fail.
  • You can’t include a namespace or parsing will fail.
  • You can’t include elements that may be perceived as “duplicates” or parsing will fail. You must disambiguate with the name attribute.

That last one can easily bite you. Let’s look at an XML file with some gotchas:

<?xml version="1.0" encoding="utf-8" ?>
<!-- THIS IS AN INVALID FILE -->
<root>
  <components route="100">
    <database enabled="True" />
  </components>
  <components route="200">
    <database enabled="False" />
    <files enabled="False" />
  </components>
</root>

So, we don’t have any DTD or namespaces, so that’s fine, but what else is wrong?

There are two components elements that the parser will see as identical. Even though the route values are different, the parser will still see them as redefining the same thing twice. We need to disambiguate with names. Let’s change route to name.

<?xml version="1.0" encoding="utf-8" ?>
<!-- THIS IS A VALID FILE -->
<root>
  <components name="100">
    <database enabled="True" />
  </components>
  <components name="200">
    <database enabled="False" />
    <files enabled="False" />
  </components>
</root>

Note even though there’s a database element in both components elements, they’re not seen as identical because the parent on each has a name. It’ll flatten out to this:

components:100:database:enabled = "True"
components:100:name = "100"
components:200:database:enabled = "False"
components:200:files:enabled = "True"
components:200:name = "200"

As you can see, name shows up in the key hierarchy and becomes a key/value of its own. In any case, you need that to disambiguate.

If you have multiple XML files, you can add a prefix on the elements by setting a name on the root element:

<?xml version="1.0" encoding="utf-8" ?>
<root name="settings">
  <components>
    <database enabled="False" />
    <files enabled="False" />
  </components>
</root>

This will become:

settings:components:database:enabled = "False"
settings:components:files:enabled = "False"
settings:name = "settings"

One thing you can do in XML that you can’t in the other providers is encrypt the XML. Encrypted XML generally looks like this:

<?xml version="1.0" encoding="utf-8" ?>
<root name="settings">
  <components>
    <database enabled="False" />
    <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
                   xmlns="http://www.w3.org/2001/04/xmlenc#">
      <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
      <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
        <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
          <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#kw-aes128" />
          <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
            <KeyName>myKey</KeyName>
          </KeyInfo>
          <CipherData>
            <CipherValue>KPE...SNg==</CipherValue>
          </CipherData>
        </EncryptedKey>
      </KeyInfo>
      <CipherData>
        <CipherValue>Vwl8...64a</CipherValue>
      </CipherData>
    </EncryptedData>
  </components>
</root>

The encrypted XML portion entirely replaces the element being encrypted along with all its contents. This is commonly used to encrypt passwords or other secrets in XML.

A full example showing both how to encrypt and decrypt XML in configuration is seen in the XML configuration provider unit tests.

Testing Using launchSettings.json

You will no doubt want to test your configuration setup to ensure things are being read in/parsed correctly. That’s easy when it’s file based, but what about command line parameters and environment variables?

Visual Studio and .NET Core allow you to put a file called launchSettings.json in the Properties folder under your project. You’ll get one for free with an ASP.NET project, but it works fine with .NET console apps, too.

The full JSON schema for launchSettings.json is available but here’s one that works for a .NET console app:

{
  "profiles": {
    "No Config": {
      "commandName": "Project",
      "environmentVariables": {
      }
    },
    "Environment Config": {
      "commandName": "Project",
      "environmentVariables": {
        "CONFIGURATION_COMPONENTS__DATABASE__CONNECTION": "connection-string",
        "CONFIGURATION_COMPONENTS__FILES__PATH": "/etc/path",
        "MYSQLCONNSTR_MYSQL": "MySQL"
      }
    },
    "Command Line Config": {
      "commandName": "Project",
      "commandLineArgs": "--components:database:connection=connection-string /components:files:path=/etc/path"
    }
  }
}

When you add that to your project, Visual Studio will detect that you have three different run configurations you can choose from. They’ll appear as a dropdown next to the green “Play” button so you can select one before you start debugging.

This is a really great way to do quick configuration changes without modifying base configuration files or code.

In an ASP.NET Core project, you’ll likely see there’s an element under environmentVariables called ASPNETCORE_ENVIRONMENT. You can use that to switch your local dev work to emulate staging or production.

Key Takeaways

It’s a long article, so there are a lot of takeaways.

  • It’s all colon-delimited key/value pairs.
  • Override values by using the same key as a previously defined configuration item.
  • Specify as little configuration as possible and have your app function with reasonable defaults when the configuration isn’t present.
  • Ordinal collections (arrays) use numbered indexes for names.
  • Make sure you validate/parse configuration before using it.
  • Use String.IsNullOrEmpty() to check for presence of a config value.
  • Ordinal collections use the index as part of the key.
  • Use object binding to handle conversion to strong types.
  • Consider data annotations validation to deal with configuration validation.
  • Only file-based providers handle reloading on change events.
  • Use environment variable prefixes to import subsets of the environment.
  • Don’t mix-and-match command line argument specs if you can avoid it.
  • With great XML power comes a great many gotchas.
  • Test environment variables and command line parameters using launchSettings.json.

powershell, windows comments edit

Two things crossed my path in a relatively short period of time that got me thinking:

  1. I read Scott Hanselman’s article on dotnet CLI completion for PowerShell and I liked it a lot. I didn’t know you could have custom completions.
  2. I have been working a lot with Kubernetes and kubectl in PowerShell and wanted completion for that… but you can only get it for bash, not PowerShell.

Challenge accepted.

A lot of Google-fu and some trial-and-error later, and I have a bridge that takes a PowerShell command line, passes it to bash, manually runs the completion function in bash, and hands the set of completions back to PowerShell.

Introducing: PSBashCompletions - a PowerShell module to enable bash completions to surface in PowerShell.

I published it as a module on the PowerShell Gallery so you can install it nice and easy:

Install-Module -Name PSBashCompletions -Scope CurrentUser

(I always install to my own profile because I don’t run as admin.)

You can also go grab it right from GitHub if you want.

To use it:

  1. Make sure bash.exe is in your path. If it’s not, the module will fall back to the location of Git for Windows (assuming git.exe can be found) and try to use the packaged bash.exe there. Failing that… you’re stuck. You need bash.
  2. Locate your bash completion script. Sometimes you can export this from the command (like kubectl); other times you can download it from the project (like git when using Git for Windows).
  3. Register the completer using Register-BashArgumentCompleter. Tell the completer which command you want to complete (kubectl) and where the completion script is (C:\completions\kubectl_completions.sh).

A registration looks like:

Register-BashArgumentCompleter kubectl C:\completions\kubectl_completions.sh

After that, in PowerShell you should be able to use the command and hit tab at the end of the line to get completions.

kubectl c<TAB>

That will compete all the commands starting with ‘c’ for kubectl.

I tried to test it a bunch, but I can’t guarantee it’ll work for every completion or every workstation.

I put troubleshooting instructions in the source readme so if it’s not working there are ways to figure it out. Using the -Verbose option when calling Register-BashArgumentCompleter is the first step to seeing what’s up. If completers in PowerShell encounter any errors, the messages get swallowed. The -Verbose option will tell you the basic bash command line the completer is going to try using so you can run it and see what happens.

I do have some demo/example completions I’ve exported (for git and kubectl) so you can try it out by grabbing those if you want.

Find something wrong? I’d love a PR to fix it.

gists, net, powershell, xml comments edit

I’m doing a lot of analysis work across a huge set of .csproj files to figure out some overall statistics. For example, what’s the target framework version for all of the projects?

PowerShell to the rescue! All the .csproj files are XML, so we can get the whole set of target frameworks for a set of repositories cloned under a common top-level folder with a single command pipeline.

The output of that will look something like…

Count Name                      Group
----- ----                      -----
  698 v4.5                      {v4.5, v4.5, v4.5, v4.5...}
   42 v3.5                      {v3.5, v3.5, v3.5, v3.5...}
  160 v4.0                      {v4.0, v4.0, v4.0, v4.0...}
    6 v4.5.1                    {v4.5.1, v4.5.1, v4.5.1, v4.5.1...}
    3 v2.0                      {v2.0, v2.0, v2.0}
    2 v4.5.2                    {v4.5.2, v4.5.2}
    3 v4.0.1                    {v4.0.1, v4.0.1, v4.0.1}

autofac, net comments edit

The State of Autofac

Autofac is a pretty popular inversion of control container for .NET. Core Autofac is approaching 10 million NuGet downloads. There are over 20 extension packages that are part of the GitHub Autofac organization supporting everything from .NET Core to MVC/Web API to Service Fabric.

The thing is… the whole project is effectively run by just two people who are not actually full-time developers on Autofac. It’s just me and Alex Meyer-Gleaves doing the best we can to:

  • Answer questions on StackOverflow, Gitter, the mailing list, GitHub issues lists, and other areas people ask for help.
  • Address new and existing issues in core Autofac and all the extension libraries.
  • Make ongoing improvements and enhancements including adding new extension libraries as demand comes up.

All of that happens in the “spare time” between full-time jobs, family obligations, and trying to maintain an otherwise decent work/life balance.

We get a few pull requests, but IoC/DI integration is tricky business. Small changes can introduce pretty big penalties when it comes to app performance. To that end, we probably don’t get quite as many PRs as you might think, and when we do there’s a lot of analysis that has to go into many of them to ensure we’re not inadvertently breaking something.

We Need Help!

There’s a lot to maintain and we’re spread thin. We’re looking for people willing to step up and own some of the Autofac extensions and integrations. If we can focus on core Autofac and core extensions we’ll make better headway on things like fixing up the decorator syntax or addressing some of the longer-running issues that aren’t five-minute fixes.

If you become an owner on one of the extensions, you’ll be responsible for:

  • Bug fixes and enhancements: Issues submitted to the extension library’s repository need to be addressed. Determine if the issue is a bug or not, determine if the enhancement request fits with the library or not, etc.
  • API design: Lots of people rely on Autofac and its extensions, you can’t change the signature of methods or introduce breaking changes without a lot of forethought.
  • Requirements determination: Part of the job of an owner is to decide whether a library should support a particular function. You’ll need to make sure fixes and enhancements fit within the bounds of being willing/able to back it with long term support. Is a fix or enhancement worth addressing?
  • Support: Answering questions by actively monitoring autofac tagged questions on StackOverflow, in Gitter, and on the mailing list.
  • Documentation: Changes to the extension or FAQs get documented so future folks can refer to the docs.
  • Examples: There’s an examples repo that shows simple usage of most extensions.

You won’t just be handed a big lump of code and left on your own, but you’ll be considered a primary owner of the extension and you’ll work with Alex and I to bring things forward. It’s a commitment. We’re looking for owners.

If you’re interested, take a look at the list of extension libraries and pick one you think you’d like to take on. Smaller libraries that might be good candidates are Autofac.Extras.Moq, Autofac.Extras.FakeItEasy, Autofac.Mef, and Autofac.Extras.MvvmCross. Feel free to look at others, these are just examples.

Let us know by filing an issue on the repo or tweeting @AutofacIoC. It’ll help if you let us know a little about yourself, any open source work you may be involved in, that sort of thing. It’s not really a job interview, but we are a team so knowing if you’ll be a team fit is valuable.

Not Interested in Owning?

That’s fine, it’s a lot to take on. We’d love your PRs that solve existing issues with a mind toward non-breaking changes. Especially in some of the lesser-used libraries, there’s definitely some TLC needed.

personal comments edit

Xev, my baby princess

On Monday, January 8, 2018, I had to say goodbye to one of my best friends of 15 years: my beautiful baby cat girl, Xev.

From the day she came home, July 2, 2002, she was trouble.

Xev out playing on the balcony

She was very playful. You can see she’s messing with a flower on the balcony. Later, she’d end up hanging from the wall just off the balcony, having been chased there by our cat, Semper, and would eventually fall into the bushes below. (She was just fine. A little scared, but no worse for the wear. And she definitely didn’t learn her lesson.)

She was enough trouble that we couldn’t just let her roam free her first few nights at home. We kept her in the bathroom instead. We made a bed for her in the sink and she loved it.

Xev has a bed in the sink

I think keeping her in the bathroom sort of affected her long-term. She loved the bathroom. If you were in there, she needed to be in there. She’d wrap string around the base of the toilet, she’d drink out of the sink… even after we moved and she was much older, she’d still get in the sink to sleep or play.

Xev still has a bed in the sink

There are a lot of bathroom stories to share. She’d come in and climb into your pants while you were… uh… doing your business. You could yell for “Pooping Cat” and she’d come to that. I’ll stop there, you get the idea.

So naughty. Always in exactly the place you needed to get and totally not interested in moving.

Xev sitting in the clothes dryer

Her favorite toys were those foil/mylar balls that made crinkling noises and anything with tassels on it. She didn’t really wrestle or do that thing some cats do where they roll over and kick with their back feet. She was more of a “grab and bite” kind of cat.

Xev with a tinsel streamer

She mellowed out a little with age and loved to sleep in the windowsill or on the floor in that one spot of sun that would come through. She didn’t fight over much, but she’d totally push the other cats out of the way to get there if needed.

Xev in the windowsill

When my daughter, Phoenix, was born, Xev was there from day one to be the watchful mother cat.

Xev with newborn Phoenix

Our other cats were friendly to Phoenix, but they never really played with her or cuddled with her. Xev… Xev was Phoenix’s older cat-sister. They were best buddies.

Xev with six-year-old Phoenix

As you can see, Xev loved being held like a baby and getting her tummy rubbed. She’d purr so loud that if you were watching TV you could barely hear your show.

This past weekend we noticed Xev had basically stopped eating. She looked a bit thin, but she was drinking. We didn’t think much of it because she was always sort of a “free spirit” and did what she wanted when she wanted. However, after a couple of days of not eating, we really noticed how thin she was getting and that she was becoming pretty lethargic.

I took her to the vet to get her checked out and we found she was in the later stages of kidney failure. She had apparently been headed downhill for quite some time and had masked it really, really well. By the time any real symptoms showed it was too late.

Even the doctor noted that she didn’t look 15 at all. I agree, she aged very well. Her coat was beautiful, full, and clean. She smelled clean. She was so active, even up until that last day.

We made the decision to let her go instead of forcing her through painful treatments that may or may not have actually prolonged her life. I wanted her to live a good life without suffering.

She was my catbaby for 15 years and I loved her more than I can say.

Xev, my 15 year old catbaby

Goodnight, Princess Xev. We will miss you forever.