Wednesday, February 01, 2012

How to Add a VSIX Installer to a DXCore Plugin

As of CodeRush/Refactor/DXCore 11.2, DXCore supports installation of plugins using Visual Studio Extensibility (VSIX) just like any other extension you might find in the gallery. This is beyond cool for a couple of reasons.

First, I've been [slowly] working on a couple of different ways to create some nature of "plugin gallery" for DXCore extensions. There are so many community plugins, plus things like CR_Documentor and CR_CodeTweet that aren't on the community plugin site, that it's really hard to know what's out there. This would allow DXCore plugins to use the standard Visual Studio Gallery mechanism for browsing.

Second, once you install a DXCore plugin, there's no real "auto-update" mechanism. You have to go check to see if there's a new release and manually do installation yourself. With VSIX and the gallery, you can get notified of new versions of the plugins via the standard Extension Manager in Visual Studio. Auto-update: solved!

I decided my pilot project would be to get a VSIX installer attached to CR_Documentor. If you have a DXCore plugin, you can use these same steps to get a VSIX installer for your plugin. If you're starting a new plugin, you can check out this great article from Alex Skorkin on how to start a fresh plugin with the provided DXCore/VSIX plugin template.

TWO IMPORTANT NOTES before you begin:

  • I converted a C#/.csproj plugin so the steps I took here assume you're using C#, too. If you're using VB, you may have to do some different/additional steps that I'm unaware of.
  • You will lose support for Visual Studio versions before 2010 because you'll need to update the target .NET framework. If you can't afford to lose that support for your plugin, stop now.

First, install the prerequisites. You'll need...

Optional: Create a new, empty DXCore VSIX standard plugin project for reference. You won't actually be doing any coding in here, but having a populated skeleton plugin really helps if you need to grab some code copy/paste style or check to see how something is set up. This was pretty key for me to figure out what I needed to do to add VSIX to my plugin – a skeleton plugin and a diff tool. Once you've created it, build it once and close it. You're done with it unless you need to go refer to something or troubleshoot it.

Open up your plugin project. We're going to make a few modifications to the project properties.

  • Switch the target framework to .NET 4.0. You need to do this because VSIX only supports .NET 4.0. It does mean you'll be giving up support in your plugin for versions of Visual Studio before 2010.
    Set the target framework to .NET 4.0 
  • Switch your build output paths to bin\Debug\ and bin\Release\ for the Debug and Release build configurations, respectively. Most plugin projects have the build output set so the plugin will build right into your Community Plugins folder. This makes it easy to debug. Once you switch to VSIX, you debug in a different way, so you don't want the plugin going into the Community Plugins folder anymore. You do need to have a build output location, though, so switch it back to the standard bin\Debug\ or bin\Release\ location.
  • Remove any post-build copy/deployment tasks. In my plugins, rather than change the build output paths, I use post-build copy tasks to copy the plugin into my Community Plugins folder. Again, you don't want to auto-deploy like this because there's a different mechanism for VSIX, so remove any of these steps from the project.

That's all for now with the project properties. We'll come back to that in a little bit once we've made a few more modifications.

Add some assembly references to your project if you haven't got them already:

  • System.Core
  • System.ComponentModel.Composition

You'll need these so the MEF portion of the VSIX installer will work.

Open your plugin project file in a text editor. It's time to tweak a few things by hand.

We need to add some top-level global properties to the project. In the top-level PropertyGroup node, which appears just below the root Project node, you need to:

  • Add a ProjectTypeGuids node that has the VSIX project GUIDs in it.
    <ProjectTypeGuids>{82b43b9b-a64c-4715-b499-d71e9ca2bd60};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
  • Add the VSIX properties that help direct the VSIX build.
    <GeneratePkgDefFile>false</GeneratePkgDefFile>
    <IncludeAssemblyInVSIXContainer>true</IncludeAssemblyInVSIXContainer>
    <IncludeDebugSymbolsInVSIXContainer>true</IncludeDebugSymbolsInVSIXContainer>
    <IncludeDebugSymbolsInLocalVSIXDeployment>false</IncludeDebugSymbolsInLocalVSIXDeployment>
    <CopyBuildOutputToOutputDirectory>true</CopyBuildOutputToOutputDirectory>
    <CopyOutputSymbolsToOutputDirectory>true</CopyOutputSymbolsToOutputDirectory>

My top-level PropertyGroup node in CR_Documentor now looks like this. I've made the additions bold.

<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="4.0">
  <PropertyGroup>
    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
    <ProductVersion>10.0.20506</ProductVersion>
    <SchemaVersion>2.0</SchemaVersion>
    <ProjectTypeGuids>{82b43b9b-a64c-4715-b499-d71e9ca2bd60};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
    <ProjectGuid>{9229512A-C004-46FD-8CEE-D096C883E827}</ProjectGuid>
    <OutputType>Library</OutputType>
    <AppDesignerFolder>Properties</AppDesignerFolder>
    <RootNamespace>CR_Documentor</RootNamespace>
    <RunPostBuildEvent>OnOutputUpdated</RunPostBuildEvent>
    <TargetFrameworkVersion>v4.0</TargetFrameworkVersion>
    <FileAlignment>512</FileAlignment>
    <GeneratePkgDefFile>false</GeneratePkgDefFile>
    <IncludeAssemblyInVSIXContainer>true</IncludeAssemblyInVSIXContainer>
    <IncludeDebugSymbolsInVSIXContainer>true</IncludeDebugSymbolsInVSIXContainer>
    <IncludeDebugSymbolsInLocalVSIXDeployment>false</IncludeDebugSymbolsInLocalVSIXDeployment>
    <CopyBuildOutputToOutputDirectory>true</CopyBuildOutputToOutputDirectory>
    <CopyOutputSymbolsToOutputDirectory>true</CopyOutputSymbolsToOutputDirectory>
  </PropertyGroup>

These properties are really important to getting the project to build successfully. For example, early on I forgot to add the GeneratePkgDefFile property over and set it to false. For quite some time I got an odd error that I couldn't figure out and stopped the build from finishing:
CreatePkgDef : error : No Visual Studio registration attribute found in this assembly.
Adding the property fixed the build. I was able to find my error by comparing my .csproj to the empty/skeleton VSIX plugin .csproj I had created. That's why I mentioned that as an optional step at the top – it's good to have something working to compare against.

Now add the build targets that allow VSIX to compile. At the bottom of your project file, locate the following line:
<Import Project="$(MSBuildBinPath)\Microsoft.CSharp.targets" />

Just below that, add this line to reference the VS SDK targets:
<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\VSSDK\Microsoft.VsSDK.targets" />

If there's any additional project cleanup you want to do, now's a decent time. For me, CR_Documentor has been around for a while so there was a lot of stuff referring to old versions of .NET and/or old versions of Visual Studio, all of which has persisted over the course of several project upgrades. I can't give you any guidance on this and you do it all at your own risk. You aren't required to do any additional cleanup to get the VSIX stuff to work; I just mention it since you're already neck-deep in .csproj hacking.

Save your changes and close the text editor. You're done manually tweaking the project. If you didn't previously close the project in Visual Studio, you'll now be prompted to load the changed project. Go ahead and do that.

Add a VSIX Plugin Extension class to your plugin. This is the little "shim" that signals DXCore to load your plugin from VSIX. Super easy and not even really any code.

  • Add a new class to your project. I called mine "VsixPluginExtension" because I'm all about naming. :)
  • Set the class to implement DevExpress.CodeRush.Common.IVsixPluginExtension. This is a marker interface; no code to write or implement for it.
  • Add a System.ComponentModel.Composition.ExportAttribute to the class so it exports IVsixPluginExtension.

That's it. Here's a copy/paste version you can grab and just change the namespace, even:

using System.ComponentModel.Composition;
using DevExpress.CodeRush.Common;

namespace YourNamespaceHere
{
  [Export(typeof(IVsixPluginExtension))]
  public class VsixPluginExtension : IVsixPluginExtension
  {
  }
}

Finally, add a VSIX manifest to the project. The VSIX manifest is a little XML file that describes what's in the VSIX package. Unfortunately, there's no template for this file type so you either need to copy one in from a different project (another reason having that empty/skeleton VSIX plugin project is handy) or you need to manually create the file yourself.

  • Add an XML file to the project called source.extension.vsixmanifest and set the "Build Action" on it to "None." If you copy it in from the skeleton project, you're done.
  • Add some empty/placeholder manifest data. You can edit the data later in a nice designer in Visual Studio, but to get you going you have to have something in it. Here's the content from a skeleton VSIX plugin if you want to copy/paste.
<?xml version="1.0" encoding="utf-8"?>
<Vsix xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Version="1.0.0" xmlns="http://schemas.microsoft.com/developer/vsx-schema/2010">
  <Identifier Id="MyDemoVsixPlugin">
    <Name>MyDemoVsixPlugin</Name>
    <Author>MyCompany</Author>
    <Version>1.0</Version>
    <Description xml:space="preserve">Empty DXCore VSIX plugin project.</Description>
    <Locale>1033</Locale>
    <SupportedProducts>
      <VisualStudio Version="10.0">
        <Edition>Pro</Edition>
      </VisualStudio>
    </SupportedProducts>
    <SupportedFrameworkRuntimeEdition MinVersion="4.0" MaxVersion="4.0" />
  </Identifier>
  <References />
  <Content>
    <MefComponent>|%CurrentProject%|</MefComponent>
  </Content>
</Vsix>

Obviously those values won't be right for your project. Your project isn't called "MyDemoVsixPlugin" or whatever. We'll fix that soon.

Open the project properties again. It's time to set up the debugging environment.

The way VSIX debugging works for DXCore plugins is that, rather than deploy your plugin to the Community Plugins folder, Visual Studio can automatically deploy your built VSIX package (which includes your plugin) into an "experimental" version of Visual Studio. This "experimental" version is sort of like running under a "test user profile" so it won't have all of the fancy stuff you've installed into your VS – it'll be pretty bare bones. In fact, the first time you run it, you'll be prompted to set up some default things like your window layout settings and whatnot, just like the first time you ran Visual Studio. Don't be fooled by the "experimental" thing – it's real, full VS. It's just the user profile that's considered "experimental."

An interesting side note is that you can actually see where VS is deploying your plugin for debugging if you look in
%USERPROFILE%\AppData\Local\Microsoft\VisualStudio\10.0Exp\Extensions
which is something like
C:\Users\yourusername\AppData\Local\Microsoft\VisualStudio\10.0Exp\Extensions

If you run into issues where you have to manually remove your extension from the experimental instance, you can delete it from that folder.

On the Debug tab, set the "Start external program" value to start Visual Studio (devenv.exe). Under "Command line arguments," put
/rootsuffix Exp
to instruct Visual Studio to use the "experimental" instance of Visual Studio.

Set Visual Studio to run an Experimental instance for debugging.

On the VSIX tab (which will now appear because your project type was set via the ProjectTypeGuids node you added earlier), for all build configurations, select the "Create VSIX Container during build" and "Deploy VSIX content to experimental instance for debugging" options.

Set the VSIX container to build and deploy

Close the project properties. You're done with that.

Finally, do some minimum update to the manifest to tailor the package for your plugin.

  • Double-click the source.extension.vsixmanifest file you added earlier. This will open it up in a nice designer.
  • At a minimum, change the following properties:
    • ID: This is the "unique ID" for your package. Usually it's the name of your plugin, but you may or may not want that. Just as long as you don't change it later, you'll be fine.
    • Product Name: This is the name of your plugin. You'll see it appear in the Extension Manager in Visual Studio.
    • Version: This is the version of your plugin. You probably want this to match the version of your plugin assembly.

Save your changes and hit F5 to build and start a debugging session. Once the experimental version of Visual Studio starts up, check to see that your plugin is available. You can even go into the Visual Studio Extension Manager (Tools –> Extension Manager...) to see your plugin listed as an extension.

Troubleshoot your build. If you are getting warnings or errors, now's the time to fix them. I had several warnings about different things in CR_Documentor because I updated the .NET target framework from 2.0 to 4.0 and a lot has changed. Unfortunately, I can't give you much guidance on this part. If you've followed the instructions up to now, you should have all of the pieces in place to get this building and debugging properly. If you're getting odd VSIX errors...

  • Go back and make sure you got ALL of the properties in place in your project.
  • Verify you have the right version of the Visual Studio SDK installed for the version of Visual Studio you have installed. If you have VS 2010 SP1, the regular VS 2010 SDK won't work – you need VS 2010 SP1 SDK.
  • Compare your plugin project to an empty/skeleton VSIX DXCore plugin project. Pop the files open in a diff tool and see what the differences are. This was key in my troubleshooting efforts.

Once you've verified the deployment is working, it's time to do a little fine-tuning on the manifest.

The manifest file is what determines how your plugin appears in the Visual Studio Gallery and in the Extension Manager. You can put as much or as little effort into this as you want – the minimum values you need are the ones I mentioned earlier. If you want your plugin to look professional, though, and be discoverable to everyone, a little more work is required.

Open the manifest in the Visual Studio designer. A [somewhat terse] explanation of the main values you see at the top of the designer is on MSDN. I'll tell you how I set up CR_Documentor and you can make the appropriate changes for your project.

  • Author: The name of the person/group responsible for the plugin. Since mine's open source, I put "CR_Documentor Contributors" as the author.
  • Description: A short text description explaining what the plugin does. This can be used to search for plugins, so having keywords in the description can help people locate your plugin.
  • Supported VS Editions: Open this and select all of the standard VS 2010 editions as well as all Express editions. Since DXCore can run on any of these, you want your plugin for DXCore to also be discoverable by people with any of these versions.

    Note that you may start getting a build warning once you select the Express Editions of Visual Studio:
    source.extension.vsixmanifest : warning : VSIX targets Express Versions of Visual Studio but VSIX contains non-template content.
    From what I can tell, this warning is erroneous and can be ignored.
    Select all of the VS editions
  • License Terms: This is a small text file that will contain license information for your plugin. You'll see this information when you install the plugin. I called mine "license.txt" because it's easy.
    License content shows up in the installer
  • Icon: A 32x32 image (png/bmp/jpg/ico) that will appear in the Extension Manager and the Gallery next to your plugin's name and description.
  • Preview Image: A 200x200 image (png/bmp/jpg/ico) that will appear in the Extension Manager and the Gallery showing a screen shot of your plugin.
  • More Info URL: A URL people can click from within the Extension Manager to read more about your plugin. I pointed mine to the CR_Documentor home page.
  • Getting Started Guide: A URL people can click from within the Extension Manager to learn how to get working with your plugin. I pointed mine to the CR_Documentor installation and usage wiki page.

Here's what my manifest looks like, fully populated (click to enlarge):

A fully populated VSIX manifest

Any files you put in the VSIX package (the license, the icon, the preview image) need to be put in the same folder as the .csproj and the source.extension.vsixmanifest. I tried to put them at a different level in the project (solution level) and it caused all sorts of headache. Don't buck the system, just stick them at the project level and life will be great.

Save your changes, rebuild, and debug your plugin. Now when you look in the Extension Manager, you'll see your plugin along with the other VS extensions and it'll look totally professional. (Click to enlarge)

Your plugin will show up in the Extension Manager.

I'll be releasing CR_Documentor via VSIX soon. I have a few more things to tidy up since I've updated to .NET 4.0. In the meantime, if you want to see the working source, you can see it in Subversion in the Google Code repository.

Tuesday, January 31, 2012

Data Type Validation and Model Binding in ASP.NET MVC

When validating input in a web forms application, you need to validate data types on the client and server side because you're working with text boxes and server controls. When you move to MVC, the client-side validation is still an interesting problem to solve, but the server-side validation all happens as a by-product of model binding.

The DefaultModelBinder has some special built-in provisions to handle data type parsing errors and automatically convert those into standardized model state errors.

If you're writing a custom model binder and you want to participate in this...

  • Get the value to parse from the value provider. If there is no value to parse, return null and you're done.
  • Create a ModelState object and set the Value property to the value you're about to parse.
  • Add the ModelState object to the incoming ModelBindingContext.
  • Attempt to parse the value. If it's successful, great. Return the properly parsed value and you're done.
  • If you can't parse the value...
    • Add a FormatException to the ModelState object.
    • Return null.

A simple skeleton binder that does all that looks like this:

using System;
using System.Web.Mvc;

namespace MyNamespace
{
  public class MyBinder : IModelBinder
  {
    public object BindModel(
      ControllerContext controllerContext,
      ModelBindingContext bindingContext)
    {
      var valueResult = bindingContext.ValueProvider.GetValue(bindingConext.ModelName);
      if(valueResult == null)
      {
        return null;
      }
      var modelState = new ModelState
      {
        Value = valueResult
      };
      bindingContext.ModelState.Add(bindingContext.ModelName, modelState);

      // Try to parse the value.
      object parsedValue = null;
      if(!TryParseValue(valueResult, out parsedValue)
      {
        // If you can't parse it, add a FormatException to the error list.
        modelState.Errors.Add(new FormatException());
      }

      // On success, return the parsed value; on fail, return null.
      return parsedValue;
    }
  }
}

The key part of that is the FormatException. After all is said and done, the DefaultModelBinder goes through the model state errors and finds all of the FormatExceptions that have been added. For each one it finds, it removes the FormatException and replaces it with a standardized message in the format:

The value '{0}' is not valid for {1}.

The {0} parameter is the original value result; the {1} parameter is the display name of the model/property being parsed.

The question then becomes: How do you localize/customize the data type validation message?

The DefaultModelBinder uses resources in the System.Web.Mvc assembly for its default set of error messages. There are two resource IDs to be aware of:

  • PropertyValueInvalid: The message for a value that couldn't be parsed and resulted in a FormatException. This gets a String.Format call on it where the first parameter is the attempted value and the second parameter is the name of the property. Default value: The value '{0}' is not valid for {1}.
  • PropertyValueRequired: The message for a value that wasn't available but is required, like a null sent in for an integer. No String.Format on this happens. Default value: A value is required.

If you want to use your own strings, you need to set the DefaultModelBinder.ResourceClassKey static property.

DefaultModelBinder.ResourceClassKey = "MyResources";

Once you do that, whenever one of these resources is required, the DefaultModelBinder will use HttpContext.GetGlobalResourceObject using the resource class key you provided and the ID noted earlier. Looking up the PropertyValueInvalid resource would effectively be a call to:

HttpContext.GetGlobalResourceObject("MyResources", "PropertyValueInvalid");

If you don't have the value in your resources, DefaultModelBinder will fall back and use the default version.

Unfortunately, you can't change the resource IDs, the string formatting arguments, or anything else... but at least you can change the messages.

Monday, January 30, 2012

Explore From Here - Command Line

I'm a big fan of the "command prompt here" context menu extensions for Windows Explorer. I use them all the time. Sometimes, though, I need to go the other-way-around.

That is, I'm at a command prompt and I want Windows Explorer open at the current location of my prompt.

explorer %CD%

Pretty simple, but super helpful. I had one of those "man, I'm stupid" moments when I put two-and-two together on this.

I ended up making a little batch file "explore.bat" and stuck it in my path.

@echo off
echo Opening Explorer on %CD%
explorer %CD%

So now it's just "explore" at the prompt and magic happens. (Yes, I do realize it's only five characters shorter, but I also get a nice little echo message to tell me what's going on, plus I don't have to remember it anymore.)

Note you can get some slightly different functionality if you use some of the command line switches for Windows Explorer, but for me, this works.

Friday, January 27, 2012

Choosing an Exception Type When Unit Testing Error Handling

When I'm testing exception handling code, I have tests for exceptions I know I need to handle and tests for exceptions I'm not expecting.

For example, say I have a component that calls a WCF service. If there's a communication issue, I want to mask that and return some stub/placeholder data. If there’s some other issue, I want to just let the exception bubble up and be handled by a global error handler. Something like this:

public DataObject GetData(SomeParameter p)
{
  if(p == null)
  {
    throw new ArgumentNullException("p");
  }
  DataObject data = null;
  try
  {
    data = this.SomeService.GetRealData(p);
  }
  catch(CommunicationException)
  {
    data = new StubData();
  }
  return data;
}

Not a complex scenario. I'll probably end up with a test component where I can set an exception to be thrown, or use Typemock Isolator to mock a response in the test, like:

Isolate
  .WhenCalled(() => component.SomeService.GetRealData(null))
  .WillThrow(new CommunicationException());

Then you could do your test, like:

[Test]
public void HandlesCommmunicationException()
{
  var component = CreateTheComponent();
  Isolate
    .WhenCalled(() => component.SomeService.GetRealData(null))
    .WillThrow(new CommunicationException());
  var p = new SomeParameter();
  var data = component.GetData(p);
  Assert.IsInstanceOf<StubData>(data);
}

That works well for testing the known exception type. What about the unknown type? You'll have a test like this:

[Test]
public void OtherExceptionsBubbleUp()
{
  var component = CreateTheComponent();
  Isolate
    .WhenCalled(() => component.SomeService.GetRealData(null))
    .WillThrow(new SOME_EXCEPTION_TYPE_HERE());
  var p = new SomeParameter();
  Assert.Throws<SOME_EXCEPTION_TYPE_HERE>(() =>component.GetData(p));
}

Pick an exception type that you'd never expect to get during normal execution.

Which is to say, if I wanted to see what happens when my component throws something other than a CommunicationException, I'm not going to pick something I might see for real.

I WOULD NOT pick...

  • ArgumentNullException
  • ArgumentException
  • NotSupportedException
  • InvalidOperationException

...or any other sort of "commonly used" exceptions that you might see arise from argument validation or something else.

Why not?

Let's use ArgumentNullException as an example. Say you add some more validation to the GetData method so that it inspects values in the SomeParameter p coming in. If there's a specific null value found, you throw an ArgumentNullException. You add tests for that and life is swell.

Except... you didn't remember to modify the test for your exceptions bubbling up. And, hey, look, it still passes! But it passes for the wrong reason. It's never actually getting to the service call where you think you're testing.

Instead, I WOULD pick...

  • DivideByZeroException
  • InvalidTimeZoneException

...or some other exception that you'd never expect to see in the context of what you're doing. Obviously you'll have to adjust based on what you're doing – if you're doing division in your method, you may actually get a DivideByZeroException, so you wouldn't use that.

By choosing the right exception, regardless of the refactoring of the class, your test will still pass... and it will pass for the reason you think.

Mini Cupcake Maker Cookies

Mini cupcake makerIf you happen to have one of these mini cupcake makers, you can put a dollop of cookie dough in each of the cupcake spots and 11 minutes later – cookies.

You may have to “pop” the tops on them at around 8 minutes. They usually bubble a bit.

There was some highly scientific culinary experimentation involved in arriving at that 11 minute point. Earlier and they just fall apart; later and the cookies are pretty hard.

Note we didn’t use liners, we just threw the dough right in.

Oh, and they don’t turn out pretty. But tasty!

Monday, January 16, 2012

This Isn't Scriptable

this isn't scriptable

there is no template
file new
automated
fill-in-the-blanks
cookie cutter
copy/paste
pattern
all input fields validated
rule of thumb
answer

you need to think

solve problems
be creative
break out of the mold
do the right thing

build a great product

don't be a script because this isn't scriptable

be a developer

develop

Tuesday, January 10, 2012

Smart People Not So Great at Murder Mystery Dinner Theater

We had a team celebration today at Kells and it was a rather fun murder mystery dinner (well, lunch) theater from Eddie May Mysteries.

The idea is that, while you eat, there are actors who come in portraying different suspects in a murder mystery. You talk to them, they leave "clues" and "evidence" at your table, and at the end you have to figure out who the killer was.

Our mystery took place supposedly in 1929, so all the actors had costumes on and acted like it was the roaring '20s.

At one point, one of the characters left some "evidence" at our table - a purse with a bloody handkerchief in it.

Now, I'm a fan of mystery movies and such, so the first thing I start thinking is, red herring. I mean, no way it's going to be this blatant, right?

Finally we got a chance to ask the characters questions, sort of near the end. The actress who had the bloody handkerchief sat down and I started thinking... given the time period... woman with a bloody handkerchief... maybe not living the healthiest lifestyle...

"Are you sick?" I asked.

"What?" The actress looked a little shocked.

"Sick. Are you sick? Like, not mentally ill, but physically. Do you have any disease?"

"Why do you ask?" She looked legitimately confused and started acting like I was being offensive to her character.

"Well, given the time frame, somewhat close to turn of the century, I'm wondering if you have tuberculosis. Consumption. You have a bloody handkerchief in your purse and there's no way it's that obvious."

At this point she actually broke character. "This is what you get when you do the show for smart people. I've been doing this show for years and I've literally never heard anyone ask that question. Have you ever heard of Occam's razor?"

I couldn't help but giggle a little. "Yeah."

"OK, good. Think way... way... way simpler than that. Truly, you're overthinking it." She laughed a little.

So... Maybe I should stop watching so many mystery movies. I always expect it to be this crazy convoluted thing now and I ignore the obvious stuff.

I ended up getting it right, despite over-thinking it.

Anyway, it was a pretty sweet event. The actors were good and were obviously having fun with it, which made it all the more fun for us. I would totally do it again.

Thursday, December 29, 2011

Switching from Windows Media Center to XBMC for my Media Front End

I’ve had my media center running for a while using standard Windows Media Center. I like it because it is simple, has few moving pieces, and basically “just works.”

Unfortunately, what I’ve been finding is that, when you have some 900+ DVDs in the DVD Library then it’s a bit slow to open. When you select the “Movie Library” option in WMC, it’ll take upwards of 45 seconds to come up. Not only that, but when you scroll through the movies it’s a bit slow, too.

There’s also some room for improvement when it comes to the UI for the library. Basically you get the poster for the movie, some actor and plot metadata, and that’s about it.

So, since I’d had WMC running for a while, I thought I’d look at some other options for the front end. Again, few moving pieces is a goal. And, in this case, a minimal amount of work is key – I don’t want to redo my whole setup just to switch front ends. After some research, I thought I’d give XBMC a try. It has progressed quite a bit since I originally did my research and appeared to be reasonably easy to get running.

Before I get into how I set it up, if you haven’t read about how I have things set up, what my goals are, and so on, please take a moment to check out my overview post. You’ll see that my use cases are common but not necessarily the most common – for example, all of my movies are in DVD rip (VIDEO_TS) format. When I explain how I got things set up it will be helpful to understand these things since I won’t be explaining, say, how to set up your music… since that’s not one of my use cases. Nor will I cover handling movies that are ripped files (e.g. WMV or MP4 or whatever) because I don’t have that, either. But if you’re looking to set up XBMC and you have a working WMC setup, maybe this will get you pointed in the right direction.

Goals:

  • Minimal rework. I already have a working Windows Media Center installation. I’d like to keep it working and add the ability for XBMC to work.
  • Few moving pieces. Using Windows Media Center, I didn’t have any additional software to install beyond the OS. I don’t mind installing a couple of other things, but I don’t want some complex setup that’s going to require a ton of maintenance and tweaking. I just want it to work.
  • Improved UI response time. The WMC DVD Library takes a long time to come up and show the list of movies. I’d like to see a UI come up in < 5 seconds.
  • Improved UI display. The WMC DVD Library is pretty simple. I’d like to see fan art, metadata, posters, etc.

Given all that… here’s what I did.

First, choose where you want to get your movie metadata from. Windows Media Center uses a single source to get posters, actor info, plot info, and so on. With XBMC you get your choice of where to get your TV and Movie data from. The reason you need to make this decision is that each “scraper” (the thing that goes and gets the data) is a little different and if you want things to just fall into place, you need to make the choice. I recommend using TheTVDB.com for the TV show scraper and IMDb for the Movie scraper.

Next, update your video folder structure so the XBMC metadata scrapers can find the movie or TV show based on the folder names. This is the biggest part of the work. By way of comparison, my old folder structure looked like this:

  • \\server\DVD
    • Alias 1.1
      • VIDEO_TS
      • AUDIO_TS
    • Alias 1.2
      • VIDEO_TS
      • AUDIO_TS
    • Aliens
      • VIDEO_TS
      • AUDIO_TS
    • Blade Runner
      • VIDEO_TS
      • AUDIO_TS
    • Die Hard
      • VIDEO_TS
      • AUDIO_TS

Again, that's the OLD structure so don't use that if you're starting fresh. Things you’ll notice in that old structure:

  • TV and Movies were intermixed. I didn’t separate the two types of DVD rips. With XBMC, you need to separate them because each scraper works against a folder tree.
  • The TV folder names were based on DVD structure and didn’t say which seasons/episodes were actually on each disc. XBMC needs that information and gets it from the folder structure.
  • Movie folders didn’t include the year of the movie in the folder name. While that’s not required, it really helps the scrapers to find the right data for the movie.

I solved those problems by doing a little moving around and renaming of files and folders. After that work, I ended up with a structure that looked like this:

  • \\server\DVD
    • Movies
      • Aliens (1986)
        • AUDIO_TS
        • VIDEO_TS
      • Blade Runner (1982)
        • AUDIO_TS
        • VIDEO_TS
      • Die Hard (1988)
        • AUDIO_TS
        • VIDEO_TS
    • TV
      • Alias
        • Season 01
          • s01e01e02e03
            • AUDIO_TS
            • VIDEO_TS
          • s01e04e05e06e07
            • AUDIO_TS
            • VIDEO_TS

That is the NEW structure – use that.

As you can see, there are some differences.

  • Under the top-level DVD folder, I now have Movies and TV as sub-folders. This allows me to tell XBMC “this folder has movies in it; this one has TV.”
  • Each movie folder includes the full name of the movie along with the year the movie was made. It is important that this matches the title and year of the movie that is in the movie metadata source you chose earlier. Remember how I recommended using IMDb for movies? Go to IMDb and look the information up. If IMDb says the movie is called “The Terminator” and the year it was made is 1984, then the folder should be “The Terminator (1984)” – not “Terminator, The (1984)” or “Terminator (1984)”. It needs to match exactly. Different movie databases may have it listed slightly differently, which is why you need to already know where you want to get your data from.
  • Under the TV folder, each TV series has a folder. Notice the folder doesn’t necessarily include the year of the TV show. The name of the show needs to match the name of the show found in the TV scraper data source. Remember I recommended TheTVDB.com? Go to TheTVDB.com and look up your show. Some shows do have the year listed (if there is more than one show with the same name). Others don’t. Make sure you match what the site says.
  • Under each TV series folder, there are season folders. In the example, you see “Season 01” – the word “Season” with a two-digit season number including a leading zero. Follow that.
  • Under each TV season folder, there are folders with each disc rip. The folder for a disc lists which season(s) and episodes the disc has. In the example, the first disc has season one, episodes one through three on it; the second has season one episodes four through seven. If you have season 4 episodes 8 through 10, you’d have “s04e08e09e10”. It’s OK if there are multiple seasons on one disc – Say you have season 1 episodes 1 and 2, then season 2 episodes 1 and 2 – that would be “s01e01e02s02e01e02” (and I’d put that in the Season 01 folder, but that’s your call).

This structure is really key. It works with both WMC and XBMC. It allows automatic metadata scrapers to work. You can read more about how XBMC uses scrapers and how to avoid wrong title matches on the XBMC site. These pages will also tell you how to handle Movies and TV that are in ripped files rather than DVD VIDEO_TS folders – something I don’t cover.

Now that you have the folder structure whipped into shape, you have to figure out if you want to have any movies grouped into sets. A “movie set” is like a series of movies you want to have appear in the list of movies like a “box set.” For example, if you have all of the Harry Potter movies, you probably want one “Harry Potter” entry in the top-level movie list and when you select it, you want to see the list of Harry Potter movies in the correct order.

To do that, you need to put a file called “movie.nfo” into the VIDEO_TS folder for each movie you want in a set. The file is literally called “movie.nfo,” not named after the movie (so, not “Die Hard.nfo” or anything like that).

The contents of the movie.nfo file look like this:

<?xml version="1.0" encoding="utf-8"?>
<movie xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <title>Harry Potter and the Goblet of Fire</title>
  <set>Harry Potter</set>
  <sorttitle>Harry Potter 4</sorttitle>
</movie>
http://www.imdb.com/title/tt0330373/

The top bit of the file is an XML document with “movie” as the root element. Inside there, you define…

  • The title of the movie – this is what will appear in the UI once you dive into the box set.
  • The name of the movie set – this is what will appear in the UI at the top level in the movie list. Any movie in a set with the same name will be grouped.
  • The sort title for the movie – this is how it sorts inside the set.

At the bottom of the file, outside the XML, put a link directly to the movie on IMDb. You link to IMDb because that’s the metadata scraper you’re going to use (remember?). If you want to use a different scraper, you need to put a link to the movie in the appropriate site.

Again, that movie.nfo file goes inside the VIDEO_TS folder.

You need to do the movie.nfo setup before you index your movies with XBMC or it will be painful to fix later.

You can read about .nfo file formats and movie sets over on the XBMC site.

You now have a decision to make: Do you want to share your library database with multiple XBMC front-ends or do you just have one XBMC front-end? If you only have one XBMC front-end, you can skip the setup of the database and get to the install of XBMC; otherwise, you’ll need to set up an XBMC media database.

There are three things to share across XBMC front ends: the media, the media library database, and the poster/art thumbnails.

  • Sharing the media is as simple as putting it on a network share that all the machines can access. I already have that.
  • Sharing the media library is a little more involved and requires you to set up MySQL. XBMC has a good article on how to do this, as does Lifehacker. I won’t go into the setup of MySQL or the databases here. I will say: I’m using the built-in MySQL on my Synology DS1010+ so it was super minimal to get running. I didn’t need to set up a new server or anything, just check the “enable MySQL” box on the Diskstation. Done.
  • I had to set up Thumbnail sharing after doing the initial XBMC install using the mklink command and replacing the Thumbnail folder in userdata with a directory link… but we’ll get to that in a bit.

Again, if you only have the one XBMC front end, you don’t really need to do any of that.

Now… get XBMC installed and fire it up. You won’t do anything but start it up right now because you need the userdata folder to be created.

XBMC stores all of its data in a user-specific folder called “userdata.” You need to know where this is and be familiar with it. By starting XBMC up that first time, the folder should have been created.

There are a metric ton of “advanced settings” for XBMC that don’t have any UI to them. You have to put these settings in the userdata folder in a file called advancedsettings.xml. This file will not exist – you have to create it.

Create your advancedsettings.xml file and put the following in it (this is from my advancedsettings.xml):

<advancedsettings>
  <sorttokens>
    <token>the</token>
    <token>a</token>
    <token>an</token>
  </sorttokens>
  <tvshowmatching append="no">
    <regexp>\[[Ss]([0-9]+)\]_\[[Ee]([0-9]+)\]?([^\\/]*)(?:(?:[\\/]video_ts)[\\/]video_ts.ifo)?</regexp>
    <regexp>[\._ \[\-\\/]([0-9]+)x([0-9]+)([^\\/]*)(?:(?:[\\/]video_ts)[\\/]video_ts.ifo)?</regexp>
    <regexp>[Ss]([0-9]+)[\.\-]?[Ee]([0-9]+)([^\\/]*)(?:(?:[\\/]video_ts)[\\/]video_ts.ifo)?</regexp>
    <regexp>[\._ \-\\/]([0-9]+)([0-9][0-9])([\._ \-][^\\/]*)(?:(?:[\\/]video_ts)[\\/]video_ts.ifo)?</regexp>
  </tvshowmatching>
  <video>
    <playcountminimumpercent>101</playcountminimumpercent>
  </video>
  <videoextensions>
    <!-- add>.ex1|.ex2</add -->
    <remove>.dat|.bin</remove>
  </videoextensions>

  <!-- Database Sharing/Synchronization -->  
  <videodatabase>
    <type>mysql</type>
    <host>192.168.xxx.xxx</host>
    <port>3306</port>
    <user>xbmc</user>
    <pass>xxxxxxxx</pass>
    <name>xbmc_video</name>
  </videodatabase>
  <musicdatabase>
    <type>mysql</type>
    <host>192.168.xxx.xxx</host>
    <port>3306</port>
    <user>xbmc</user>
    <pass>xxxxxxxx</pass>
    <name>xbmc_music</name>
  </musicdatabase>
  <pathsubstitution>
    <substitute>
      <from>special://masterprofile/Thumbnails</from>
      <to>smb://yourserver/path/to/shared/Thumbnails</to>
    </substitute>
    <substitute>
      <from>special://profile/Thumbnails</from>
      <to>smb://yourserver/path/to/shared/Thumbnails</to>
    </substitute>
  </pathsubstitution>
</advancedsettings>

There's a lot there, but I'll explain it:

  • The sorttokens section explains which things to ignore when sorting titles. I don’t want “A Bug’s Life” to show up under “A" so I added that to the list of tokens. I think the only one in place by default is “The.”
  • The tvshowmatching section defines some regular expressions that help the TV show scraper figure out that “s01e01e02e03” in your folder structure corresponds to season 1, episodes 1 through 3.
  • The video/playcountminimumpercent setting basically says “don’t mark things as played or unplayed.” I don’t want checkmarks to show up by movies based on what I’ve watched.
  • The videoextensions setting is set to ignore some files that show up in VIDEO_TS rips but aren’t actually video files. In this case, I don’t have any .dat or .bin files that are actually video files, so I don’t want them indexed as videos.
  • The last few bits – videodatabase, musicdatabase, pathsubstitution – are for sharing the database across front ends. These are explained in those articles explaining how to set up MySQL as the central database. If you’re not doing that database sharing, omit those sections. If you’re sharing, make sure to update the IP addresses, passwords, and share paths to point to your shared locations.

Now that you have your advancedsettings.xml in place, delete the database and Thumbnails folders out of the userdata folder. They will be recreated as necessary when you restart XBMC.

Note: Using XBMC 10.1, I found the pathsubstitution for getting shared Thumbnails to work… didn’t work. Instead, I manually created a Thumbnails folder on my server in a shared location. I then manually went into the userdata folder at a command prompt and created a link to the shared location:

mklink /d Thumbnails \\yourserver\path\to\shared\Thumbnails

Fire up XBMC again. If you’re sharing the media library database, XBMC will now be using the shared locations.

Go into the system settings and find the section where you can get add-ons. You need to go to the Movie Metadata add-ons section and get the IMDb metadata add-on. This will let you use IMDb as the metadata source; by default it’s not installed.

Add your movie and TV sources to XBMC pointing to the Movie and TV shared folders, respectively. Do this one at a time and let the indexing of each source finish before adding the next; I found that if you interrupt the indexing process by adding a new source, the original indexing never completes. When you add each source, you get to choose what type of data is in each source (Movies, TV, etc.) and which scraper to use. Be sure to select the IMDb scraper for movies and the TheTVDB scraper for TV.

Once everything is indexed, go check out the results. Look at the list of movies and TV shows and make sure everything was properly picked up. You may need to add .nfo files or change some folder names in order to get things scraped right. If you do, you have the choice of trying to just remove one movie/show from the database and re-scraping that or removing the whole movie/TV source and re-indexing from scratch. I didn’t have a ton of luck cherry-picking to fix, so I reindexed my whole library about five times before I got it right.

The last thing you have to do is make it look pretty. Go to the system settings and find the skins. Install the MediaStream Redux skin. Now go visit your movies and TV and use the left/right arrows to get the view menu to pop up. By default the view is “List” – not pretty. Select a different view mode like “Media Info” and you’ll start seeing that super-nice display. Another option is to “show fanart” – do that. You’ll start seeing really nice backdrops related to the movie you’re currently selecting.

That all looks like a lot of work, but it’s actually not that bad. Basically it’s just cleaning up the directory structure and adding a few little XML files. XBMC does the rest of the work.

If you have any other XBMC front ends, just do the advancedsettings.xml placement and Thumbnail sharing steps. You don’t need to re-add the sources or set up the scrapers (I’d recommend against it). You do need to set up the skin. Whenever you add a new movie, tell your original XBMC front end to scan the new movie and add it to the database – once it’s there, the other front ends will find it.

I CAN’T OFFER YOU SUPPORT ON THIS. XBMC has a great wiki and some really nice forums – I recommend asking the experts if you have questions. My setup works for me and I hope this article can help you, but I can’t support it. If you’re interested in other aspects of my media center – my network setup, why I chose what I did, what tools I use to rip DVDs, etc., check out my media center overview.