docker, vs comments edit

Container Tools in Visual Studio offers the ability to develop an application and have it run inside a container while Visual Studio debugs it on your host machine. It’s a cool way to see how your app will behave in the container environment with the flexibility of edit-and-refresh functionality that doesn’t require the overhead of rebuilding the container each time.

I ran into a bunch of trouble getting some things working the other day which caused me to dive a little deeper into how this works and I found a few interesting things. Gotchas? Tips? Sure. That.

I’m primarily using the single-container support - not the Docker Compose multi-container support. If you’re all Docker Compose up in there, this may or may not be helpful to you.

You Only Need ‘base’ for VS

The Dockerfile that gets generated has a bunch of named intermediate containers in it - base, build, publish. This is helpful if you don’t already have a Dockerfile, but if you’re really just trying to get debugging working with VS, you only need the base container. You can delete or modify the others.

When VS builds the container, you can see in the Container Tools output window it’s running a command like this:

docker build -f "C:\src\solution\project\Dockerfile" -t project:dev --target base --label "com.microsoft.created-by=visual-studio" "C:\src\solution"

The --target base is the key - it’s not going to build the rest.

(I think you can change this using <DockerfileStage> in your project - see below.)

VS Controls Container Startup and Teardown

In Visual Studio the container for debugging will get built and will start as soon as you select the Docker configuration for running. Even if you don’t actually start a debug setting, the container will be pulled, built, and run in the background. The container will continue to run until VS shuts down.

docker run -dt -v "C:\Users\yourname\vsdbg\vs2017u5:/remote_debugger:rw" -v "C:\src\solution\project:/app" -v "C:\Users\yourname\AppData\Roaming\Microsoft\UserSecrets:/root/.microsoft/usersecrets:ro" -v "C:\Users\yourname\.nuget\packages\:/root/.nuget/fallbackpackages2" -v "C:\Program Files\dotnet\sdk\NuGetFallbackFolder:/root/.nuget/fallbackpackages" -e "DOTNET_USE_POLLING_FILE_WATCHER=1" -e "ASPNETCORE_ENVIRONMENT=Development" -e "NUGET_PACKAGES=/root/.nuget/fallbackpackages2" -e "NUGET_FALLBACK_PACKAGES=/root/.nuget/fallbackpackages;/root/.nuget/fallbackpackages2" -p 58260:80 --entrypoint tail project:dev -f /dev/null

There are some interesting things to note here:

  • A ton is mounted through volumes. Look at all those -v commands. There’s a remote debugger; your application code/source; user secrets; your local NuGet package cache; and your local installation of fallback packages. You’ll get a warning that pops up if you don’t have volume sharing enabled in Docker. You have to allow the drive with your source code to be mounted. VPNs and firewalls can really mess this up by blocking the SMB port.
  • The remote debugger isn’t associated with a VS 2017 install. The path to the remote debugger you see is C:\Users\yourname\vsdbg\vs2017u5 but this isn’t part of a VS 2017 install. Even if you only have VS 2019, it’s still this path. It could change later, but don’t be fooled.
  • The default environment is Development. The Container Tools put the ASPNETCORE_ENVIRONMENT=Development thing in there. You can override this by updating launchSettings.json (see below).
  • The entrypoint is not your application. Notice the entrypiont is tail -f /dev/null. This just ensures the container keeps running but isn’t tied to your application. A separate docker run call will be issued when it’s time to start your app.

During build, you’ll see something like this in the Build output window:

docker exec -i 40b49d8d963bb682a08fed17248212bcfd939456c8030689e9a28f17f5b067e3 /bin/sh -c "if PID=$(pidof dotnet); then kill $PID; fi"

What this is doing is killing the running dotnet command in the container so any files that might be getting regenerated by Visual Studio or whatever won’t mess up the running process.

When you start debugging, the remote debugger starts in the container. I used Process Explorer and Process Monitor to look for docker commands going by. I see that the command to start the remote debugger is:

"docker" exec -i 40b49d8d963bb682a08fed17248212bcfd939456c8030689e9a28f17f5b067e3 /bin/sh -c "ID=.; if [ -e /etc/os-release ]; then . /etc/os-release; fi; if [ $ID = alpine ] && [ -e /remote_debugger/linux-musl-x64/vsdbg ]; then VSDBGPATH=/remote_debugger/linux-musl-x64; else VSDBGPATH=/remote_debugger; fi; $VSDBGPATH/vsdbg --interpreter=vscode"

UPDATE June 18, 2019: After publishing this post I found out that Visual Studio communicates the dotnet startup command directly to the remote debugger. The debugger is what launches the dotnet command and provides the additional environment variables from launchSettings.json. This allows VS to catch any startup errors.

Using ps -axwwe on a running container being debugged, I can see the command line and the environment for the running dotnet process. The command line looks like:

/usr/bin/dotnet --additionalProbingPath /root/.nuget/fallbackpackages2 --additionalProbingPath /root/.nuget/fallbackpackages bin/Debug/netcoreapp2.1/project.dll

The environment is big so I won’t paste it all here, but I can see environmentVariables things (from launchSettings.json) show up.

launchSettings.json Gets Extra Stuff

Once you’ve right-clicked in VS and added Docker support to your ASP.NET Core project, launchSettings.json will be updated to include a Docker configuration that looks something like this:

{
  "iisSettings": {
    "windowsAuthentication": false,
    "anonymousAuthentication": true,
    "iisExpress": {
      "applicationUrl": "https://localhost:44308",
      "sslPort": 44308
    }
  },
  "$schema": "http://json.schemastore.org/launchsettings.json",
  "profiles": {
    "IIS Express (Development)": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "launchUrl": "",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    },
    "Kestrel (Development)": {
      "commandName": "Project",
      "launchBrowser": true,
      "launchUrl": "",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      },
      "applicationUrl": "https://localhost:44308"
    },
    "Docker": {
      "commandName": "Docker",
      "launchBrowser": true,
      "launchUrl": "{Scheme}://{ServiceHost}:{ServicePort}/",
      "httpPort": 58260,
      "useSSL": false,
      "sslPort": 44308
    }
  }
}

There are some things to note in here.

  • environmentVariables will work, but just in the app. Just like in the other configruations that have an environmentVariables node, you can add that to the Docker node and the environment variables there will be available in your application. However, they won’t be added as global environment variables to the container - they’ll instead get passed in to your application. If you launch a separate shell process in there to poke around, you won’t see them.
  • iisSettings is still important. Even if you don’t use it, the iisSettings.iisExpress.sslPort and iisSettings.iisExpress.applicationUrl values are still important.
  • Docker as the command name is the key. This seems to be a magic thing interpreted by the add-in to know to launch Docker. The name of the configuration doesn’t appear to matter.
  • There are curly-brace magic strings that work in the launchUrl field. I can’t find any beyond {Scheme}, {ServiceHost}, and {ServicePort} that do anything, though in the Microsoft.Docker assembly I see a definition for {ServiceIPAddress} that doesn’t seem used.

You Can Affect docker run Through Project Settings

There are some magic XML elements you can put in your project properties that will affect the docker run command. These are found in the .targets files in the Microsoft.VisualStudio.Azure.Containers.Tools.Targets package, which you can find in your local cache in a spot like C:\Users\yourname\.nuget\packages\microsoft.visualstudio.azure.containers.tools.targets\1.4.10\build.

Ones that seem interesting which I have pulled straight out of the Container.targets file…

  • <DockerDefaultTag>: The default tag used when building the Docker image. When using the container debugging tools the container tag seems to be set to dev but I can’t figure out where that happens.
  • <DockerDefaultTargetOS>: The default target OS used when building the Docker image.
  • <DockerfileBuildArguments>: Additional arguments passed to the Docker build command. I’m not sure what you might do with that, but it may be an interesting hook.
  • <DockerfileRunArguments>: Additional arguments passed to the Docker run command. You can use this to add volume mounts and such to the VS process that starts up the container for your project. You can add environment variables this way, too, if you don’t want to use launchSettings.json.
  • <DockerfileRunEnvironmentFiles>: Semicolon-delimited list of environment files applied during Docker run.
  • <DockerfileStage>: The Dockerfile stage (i.e. target) to be used when building the Docker image. I found this defaults to base but I can’t find where that default is set.

I’ve only really tried the DockerfileRunArguments to see if I could use that for environment variables (before I figured out the launchSettings.json part) and it seemed to work. On the others, YMMV.

Troubleshooting Container Startup

If you have an error starting the container, restart Visual Studio after resolving it. I can’t figure out a way to manually force the container to restart and still have it controlled (start/stop/cleanup) by VS.

If you see…

docker: Error response from daemon: driver failed programming external connectivity on endpoint hungry_nash (09d5705dc88b7afc229be8c3ed8c92bc30c3c4a2e11fdc9ece423cfb4bfe50b3): Error starting userland proxy:.

…then close VS and restart the Docker daemon. I’ve seen this a lot when my machine starts up and maybe has a race condition between networking getting established and the Docker daemon needing networking. A simple restart usually fixes it.

If you see…

Launching failed because the directory '/remote_debugger' in the container is empty. This might be caused by the Shared Drives credentials used by Docker Desktop being out of date. Try resetting the credentials in the Shared Drives page of the Docker Desktop Settings and then restarting Docker.

or…

Error response from daemon: error while creating mount source path '/host_mnt/c/Users/yourname/vsdbg/vs2017u5': mkdir /host_mnt/c: file exists.

…then there’s a problem with drive mounting. Make sure your drive sharing settings in Docker allow the drive with your source and the drive with the remote debugger to both be mounted. Further, if you’re on a VPN like Cisco Anyconnect, chances are the SMB sharing port 445 is blocked. Try getting off the VPN. You’ll need to close VS and restart the Docker daemon once you’ve resolved that.

No, I haven’t found a fix for the VPN thing. I wish I had.

If the container fails to start for whatever reason, you may be left with zombie containers or images.

  • Use docker ps -a to see the containers that are created but will never be used again. When VS is closed these will remain in Created state. Use docker rm image_name_here to clean them up.
  • Use docker images to see the images on your machine. If you see one that’s named like your project with the tag dev, that’s the dev container. Clean that up with docker rmi project:dev (using the appropriate project name in there).

The Actual Documentation

Right now… there’s not any documentation on this stuff. I opened an issue to request some on June 18, 2019 and the team agrees on the need. I hope there will be some coming soon, it’d make life easier.

That said, what’s here is pretty sweet and generally “just works.” Can’t complain about that!

docker, linux, windows comments edit

Working in an environment of mixed containers - both Windows and Linux - I wanted to run them all on my dev machine at the same time if possible. I found some instructions from a while ago about this but following them didn’t work.

Turns out some things have changed in the Docker world so here are some updated instructions.

As of this writing, I’m on Docker Desktop for Windows 2.0.0.3 (31259) Community Edition. The instructions here work for that; I can’t guarantee more won’t change between now and whenever you read this.

  1. Clean up existing containers before switching to Windows containers. Look to see if you’re using Windows containers. Right-click on the Docker icon in the system tray. If you see “Switch to Windows containers…” then you are not currently using Windows containers. Stop any running containers you have and remove all images. You’ll need to switch to Windows containers and the image storage mechanism will change. You won’t be able to manage the images once you switch.

  2. Switch to Windows Containers. Right-click on the Docker icon in the system tray and select “Switch to Windows containers…” If you’re already using Windows containers, great!

  3. Enable experimental features. Right-click on the Docker icon in the system tray and select “Settings.” Go to the “Daemon” tab and check the box marked “Experimental features.”

Enable experimental features.

That’s it! You’re ready to run side-by-side containers.

The big key is to specify --platform as linux or windows when you run a container.

Open up a couple of PowerShell prompts.

In one of them, run docker images just to make sure things are working. The list of images will probably be empty if you had to switch to Windows containers. If you were already on Windows containers, you might see some.

In that same PowerShell window, run:

docker run --rm -it --platform windows microsoft/nanoserver:1803

This is a command-prompt-only version of Windows Server. You should get a C:\> prompt once things have started up.

Leave that there, and in the other PowerShell window, run:

docker run --rm -it --platform linux ubuntu

This will get you to an Ubuntu shell.

See what you have there? Windows and Linux running side by side!

Windows and Linux containers - side by side!

Type exit in each of these containers to get out of the shell and have the container clean itself up.

Again, the big key is to specify --platform as linux or windows when you run a container.

If you forget to specify the --platform flag, it will default to Windows unless you’ve already downloaded the container image. Once you have used the image, the correct version will be found and used automatically:

# Works because you already used the image once.
docker run --rm -it ubuntu

If you try to run a Linux container you haven’t already used, you may get a message like this:

no matching manifest for windows/amd64 10.0.18362 in the manifest list entries

I’m not sure on the particulars on why sometimes --platform is required and sometimes it’s not. Even after removing all my container images, I was able to run an Ubuntu container without specifying platform, like some cache was updated to indicate which platform should be used by default. YMMV. It doesn’t hurt to include it, however, if you try to use --platform on another machine it may not work - you can only use it when experimental features are enabled.

UPDATE June 14, 2019

I’ve found since working in this mixed environment that there are things that don’t work as one might entirely expect.

  • Networking: With Linux-only containers on Windows you get a DockerNAT virtual network switch you can tweak if needed to adjust network connectivity. Under mixed containers, you use the Windows Container network switch, nat and you really can’t do too much with that. I’ve had to reset my network a few times while trying to troubleshoot things like network connections whilst on VPN.
  • Building container images that reference files from other images: A standard .NET Core build-in-container situation is to create, in one Dockerfile, two container images - the first builds and publishes the app, the second copies the published app into a clean, minimal image. When in mixed container world, I get a lot of errors like, “COPY failed: file does not exist.” I can look in the intermediate containers and the files are all there, so there’s something about being unable to mount the filesystem from one container to copy into the other container.

Unrelated to mixed containers, it seems I can’t get any container to connect to the internet when I’m on my company’s VPN. VPN seems to be a pretty common problem with Docker on Windows. I haven’t solved that.

It appears there’s still a lot to work out here in mixed container land. You’ve been warned.

git comments edit

Are you stuck unable to update your version of Atlassian Sourcetree for Windows because when you update and restart, Sourcetree hangs?

I was stuck on 3.0.17. Every time I updated (to 3.1.2), Sourcetree would restart and then… hang. Unresponsive. Unable to see if there were any new updates, unable to do anything but kill the app.

It turns out the reason for this is that Sourcetree didn’t handle monitor scaling very well. Say you have a 4K monitor set to scale to 150% - that’s when you’d see the hang.

There are two workarounds for this:

The first option is to stop monitor scaling and switch back to 100%. Not the greatest, I know, but that’ll get you through temporarily… and it only needs to be temporary. (I’ll get there.)

The other option is to do a tweak to the Sourcetree configuration file. First, make sure Sourcetree is closed. Now go to %LocalAppData%\SourceTree\app-version like C:\Users\tillig\AppData\Local\SourceTree\app-3.1.2. Open the file SourceTree.exe.config. Find this line:

<AppContextSwitchOverrides value="Switch.System.Windows.DoNotScaleForDpiChanges=false"/>

Update it to look like this:

<AppContextSwitchOverrides value="Switch.System.Windows.DoNotScaleForDpiChanges=false;Switch.System.Windows.Controls.Grid.StarDefinitionsCanExceedAvailableSpace=true"/>

When you start Sourcetree, it should be responsive.

This is the default setting in 3.1.3. If you can get yourself upgraded to 3.1.3, you won’t have to do any workarounds anymore. So if you temporarily set your monitor to 100%, take the upgrades in SourceTree up to 3.1.3 or later, then you can switch your monitor back. (Or, of course, you can tweak the configuration on your hanging version of Sourcetree until you get to 3.1.3 or later.)

I had to upgrade from 3.0.17 to 3.1.2 and then to 3.1.3. For some reason I couldn’t just go straight from 3.0.17 to 3.1.3. YMMV.

net, build, gists comments edit

I’ve been enjoying the addition of dotnet CLI global tools and figured I’d blog the ones I use. I’ll also include a PowerShell script that is used to install them (or, if they’re installed, update them).

The list is current as of this writing, but you can visit the gist (below) to see the set of tools I’m using at any given time.

  • dotnet-depends: Shows the dependency tree for a .NET project.
  • dotnet-outdated: Shows outdated packages referenced by a .NET project.
  • dotnet-format: Formats code based on EditorConfig settings.
  • dotnet-script: Run C# script (.csx) files.
  • dotnet-symbol: Download symbols from the command prompt. [I’m still trying this one out to see if I use it much or like it.]
  • dotnetsdkhelpers: A global tool version of the original SDK installer helpers that addresses the need for external tools and fixes a couple of bugs with the original.
  • gti: Install plugins from a .gti file/manifest. [I’m still trying this one out to see if I like it. If it’s good, I can replace my PowerShell script with a manifest.]
  • microsoft.web.librarymanager.cli: dotnet CLI access to the libman dependency manager for JS.

Here’s the gist with the PowerShell script that I use to install these:

azure, build, net, tfs, vs comments edit

Azure DevOps has the ability to publish things to a private NuGet feed as part of its artifacts handling.

Working with a private feed from a developer machine running builds from the command line or Visual Studio is pretty easy. There is documentation on using a NuGet credential provider to authenticate with Azure DevOps and make that seamless.

However, getting this to work from a pipeline build is challenging. Once you’ve published a package, you may want to consume it from something else you’re building… but the feed is secured. What do you do?

I’m told there are improvements for this coming in Q2 2019. I can’t quantify what those improvements are, but it may mean things start to “just work.” Until then, here are ways to work around the challenges.

Option 1: Separate Restore from Build

The documentation shows how to use NuGet or the dotnet CLI for package restore from your feed. Both of the solutions effectively amount to separating the call to NuGet restore or dotnet restore from the rest of your build.

For NuGet, you’d use a NuGet build step (NuGetCommand@2) and specify the restore. Do that before you execute the build on your solution.

For the dotnet CLI, you’d use a dotnet CLI build step (DotNetCoreCLI@2) and indicate the restore command.

In both cases, the special build command will generate a NuGet.Config file on the fly that contains the system access token. The restore operation will use that custom temporary config during the restore and it will succeed.

However, if you later try running dotnet build or dotnet publish it’ll fail - because there’s an implicit restore that runs during those. These will not have the system credentials in place. You have to use --no-restore on builds, for example, to avoid the auto-restore. It can get painful in a larger build.

If you have a build script, like a bash or PowerShell script, manually executing dotnet restore in that script will also not work. You must use the build tasks to get the magic to happen.

Option 2: Use the Azure Artifacts Credential Provider

Another option in the docs is that you can use the Azure Artifacts credential provider. While it seems this is primarily geared toward running on build agents you host yourself, you can possibly get this working on hosted agents.

I have not tried this myself. I went with option 3, below. However, if you want to give it a shot, here’s what you’d do.

First, you’ll want to be aware of how NuGet credential providers work. I don’t mean the internals, but, like, where you need to put the credential provider executable to make it work and how to troubleshoot it. All of that is documented.

Now…

  • Download the latest release of the credential provider. Make sure you get the version that will run on your build agent, not your development machine.
  • Follow the instructions in the readme to find the self-contained executable version of the credential provider in the archive you just downloaded.
  • Extract the credential provider to somewhere in the source you’ll be building. Maybe a separate build folder.
  • As part of your build pipeline, you’ll need to…
    • Set the NUGET_CREDENTIALPROVIDERS_PATH to point to the build folder in your checked-out source that contains the provider.
    • On Linux, you may need to chmod +x that provider.
    • Set the VSS_NUGET_EXTERNAL_FEED_ENDPOINTS to indicate the location of your external NuGet feed and provide the system access token. It takes a JSON format: {"endpointCredentials": [{"endpoint":"http://example.index.json", "username":"vsts", "password":"$(System.AccessToken)"}]}

In the VSS_NUGET_EXTERNAL_FEED_ENDPOINTS you’ll notice the use of the $(System.AccessToken) variable. That’s a predefined system variable in Azure DevOps build pipelines. You’ll see that again later.

Anyway, if all the planets have aligned, when you run your standard dotnet restore or NuGet restore, it will use the credential provider for authentication. The credential provider will use the environment variable and magic will happen.

One other note there - the username vsts isn’t special. It can be any value you want, the endpoint doesn’t actually end up checking. It just can’t be omitted.

Option 3: Update NuGet.Config

The final option is to update your NuGet.Config on the fly with the system access token as part of the build. I went with this option because it was simpler and had fewer moving pieces.

In your source you likely already have a NuGet.Config file that has both the standard NuGet.org feed and your private Azure DevOps feed in it. It should look something like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <solution>
    <add key="disableSourceControlIntegration" value="true" />
  </solution>
  <packageSources>
    <clear />
    <add key="NuGet Official" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="Azure DevOps" value="https://pkgs.dev.azure.com/yourorg/_packaging/yourfeed/nuget/v3/index.json" protocolVersion="3" />
  </packageSources>
</configuration>

The name, Azure DevOps, is the key here. Doesn’t matter what you name it, just make sure you remember it. You’ll need it.

In your build pipeline, before you do any operations to build or restore packages, Use the NuGetCommand@2 task and run a custom command to update the source in that NuGet.Config to have the system credentials attached.

- task: NuGetCommand@2
  displayName: 'Authenticate with Azure DevOps NuGet'
  inputs:
    command: custom
    arguments: sources update -Name "Azure DevOps" -Username "vsts" -Password "$(System.AccessToken)" -StorePasswordInClearText -ConfigFile ./NuGet.Config

What this will do is add the credentials right into the XML of the NuGet.Config as checked out in your source. The NuGet or dotnet commands will already be using that config file to locate your feed, the creds will come along for free.

Items to note:

  • Again, you see that $(System.AccessToken) show up. That’s the magic.
  • You need to have -StorePasswordInClearText or the dotnet CLI won’t be able to use the credentials. If you’re only using NuGet commands, you should be OK not storing in clear text.
  • If you’re on a Linux agent, don’t forget filenames are case-sensitive. If you get an error, make sure you got all the capitalization right for your config file.