Testing MySQL queries with NUnit

Even the most adamant unit-testing purist will admit that database queries need to be tested. If possible, these tests should run against the same database engine that will be used in production.

With a SQL Server database, testing queries is fairly easy, since most Visual Studio installations will include SQL Server Express. Just create the database in a SetupFixture, connect with Windows Authentication, and you're all set. MySQL presents a few more challenges, however. First, it needs to be installed. Second, the tests need the correct credentials to connect to the installed MySQL instance.

One simple way to handle this is to set up a central MySQL instance for testing only. However, this means team members must be on the network to run the tests. There is also the issue of multiple team members running the tests at the same time causing unexpected failures in each other's test runs, or worse, causing unexpected failures in an automated build.

A better alternative is to keep the MySQL binaries in version control with a known configuration that can be used in test runs. The binary distribution of MySQL is fairly large; however, with the right command line arguments, we only need two files: bin\mysqld-nt.exe and share\english\errmsg.sys (or whichever language you want to use). Include both of these as content files in a test project and set the build action to "Copy If Newer". In a SetupFixture, start MySQL with some code like this in the setup method:

var process = new Process();
var arguments = new[]
{
    "--standalone",
    "--console",
    "--basedir=.",
    "--language=.",
    "--datadir=.",
    "--skip-grant-tables",
    "--skip-networking",
    "--enable-named-pipe"
};

process.StartInfo.FileName = "mysqld-nt.exe";
process.StartInfo.Arguments = string.Join(" ", arguments);
process.StartInfo.UseShellExecute = false;
process.StartInfo.CreateNoWindow = true;

process.Start();

The first two arguments (--standalone and --console) tell MySQL to run as a standalone instance and to keep the console window open (i.e. do not run as a service). The next three arguments (--basedir=., --language=. and --datadir=.) tell MySQL to run from the current directory, load language files (errmsg.sys) from the current directory, and write data files to the current directory. The --skip-grant-tables argument disables security so that the tests do not need to worry about providing credentials when connecting. The final two arguments (--skip-networking and --enable-named-pipe) tell MySQL not to listen for TCP connections and instead allow named pipe connections. This prevents our standalone MySQL instance from interfering with any other MySQL installations on the machine.

Once the instance has started, we can connect with a connection string like this: Data Source=localhost;Protocol=pipe;. Finally, kill the MySQL process in the SetupFixture teardown method.

On my team, we have rolled this functionality (and a few other goodies) into a NUnit addin, but that's a story for another blog post.

Happy testing!

NServiceBus assembly scanning: avoiding unintended consequences

The default startup behavior for a NServiceBus endpoint is to scan all assemblies in the deployment directory for any types it might be interested in, for example, message modules and message handlers. This is really handy, but occasionally it can lead to unintended consequences. Fortunately, the With method has a few overloads and one of them takes a list of assemblies to scan. If you want to prevent loading types from any stray assembly that might make it into your deployment directory, change your initialization code to look like this:

var assemblies = GetType().Assembly
   .GetReferencedAssemblies()
   .Select(n => Assembly.Load(n))
   .ToList();
assemblies.Add(GetType().Assembly);
NServiceBus.Configure.With(assemblies);

This restricts the assembly scanning process to only the assemblies that your endpoint explicitly references. We can extract this code to an extension method so we can use it in other endpoints:

public static IEnumerable<Assembly> ReferencedAssemblies(this Type type)
{
   var assemblies = type.Assembly
      .GetReferencedAssemblies()
      .Select(n => Assembly.Load(n))
      .ToList();
   assemblies.Add(GetType().Assembly);
   return assemblies;
}

Now our endpoint initialization looks like this:

NServiceBus.Configure.With(GetType()
   .ReferencedAssemblies());

Assert.DoesNotThrow... why does it exist?

Take a look at this test:

[Test]
public void TestSomething()
{
   Assert.DoesNotThrow(MethodUnderTest);
}

It seems to me that it is functionally equivalent to this test:

[Test]
public void TestSomething()
{
    MethodUnderTest();
}

In other words, Assert.DoesNotThrow is the same as writing a test with no assertion. Is it really enough that the code runs without throwing an exception? Presumably the method under test is doing something, so shouldn't you verify those results with a stronger assertion?

AsyncController and custom action invokers

If you are trying to get AsyncControllers to work, you might run into this gem when you try to test your controller:

If this happens to you, a custom action invoker is likely to be the culprit. In my case, I was using Castle Windsor, and I had extended ControllerActionInvoker to provide dependency injection to my action filters. My action invoker looked something like this:



And this is how I had registered the action invoker with Windsor (this assumes you are using a controller factory that is aware of Windsor):




All I had to change to fix the problem was to make my custom action invoker inherit AsyncControllerActionInvoker instead:



AsyncControllerActionInvoker implements IAsyncActionInvoker and adds a few new methods. Depending on what you are trying to accomplish with your custom invoker, you may need to override a few of these new methods as well.

Interestingly enough, I did not have to change how the invoker was registered with Windsor.

Using AsyncController with NServiceBus

With NServiceBus sometimes you do not want to allow a web request to complete until you have received some response from your message handler. For example, suppose you have defined the following message handler that returns a response code on success:

public class TestMessageHandler :
    IHandleMessages<TestMessage>
{
    public IBus Bus { get; set; }

    public void Handle(TestMessage message)
    {
        //Do something interesting...
        Bus.Return(0);
    }
}

This is a perfect scenario for using the asynchronous page API. It allows you to execute a potentially long-running operation on a separate thread outside of the IIS thread pool. While this thread is executing, the original request thread is returned to the thread pool. As a result, the asynchronous page API is great for I/O bound operations, since you keep threads in the thread pool ready to service requests and the new thread spends most of its time blocked on I/O. NServiceBus supports the asynchronous page API out of the box:

Bus.Send(new TestMessage()).RegisterWebCallback(callback, state);

Unfortunately, RegisterWebCallback does not work with ASP.NET MVC. If you are using ASP.NET MVC, you can use AsyncController, which was introduced in ASP.NET MVC 2. The following code illustrates how you might use AsyncController to wait for a response the message handler:

public class TestController : AsyncController
{
    private IBus bus;

    public TestController(IBus bus)
    {
        this.bus = bus;
    }

    public void IndexAsync()
    {
        AsyncManager.OutstandingOperations.Increment();

        bus.Send(new TestMessage()).Register(Callback);
    }
    public ActionResult IndexCompleted()
    {
        return View();
    }
    
    private void Callback(int responseCode)
    {
        AsyncManager.OutstandingOperations.Decrement();
    }
}

As you can see, the AsyncController API is a little awkward (though it is an improvement over the asynchronous page API, in my opinion), but it is a small price to pay for scalability.

Ctrl-C and the NServiceBus generic host

Ten posts and almost a whole year later, my very first post about NServiceBus is still by far my most popular. So here is some more NServiceBus-y goodness.

Last week a coworker came to me asking for help with a self-hosted NServiceBus application. He had a small console application set up to test NServiceBus and it worked beautifully in all but one case. When he killed the application with Ctrl-C, the message being processed was always lost. At first I thought maybe the application was not using a transactional endpoint, but it was configured to do so. I did not see the same behavior with the generic host running in console mode, so I figured the generic host was handling Ctrl-C somehow. A quick search turned up the CancelKeyPress event. I opened up the generic host in reflector, and, on a hunch, navigated to the TopShelf.Internal.Hosts namespace (the generic host internalizes another excellent open source project called TopShelf). Sure enough, the TopShelf console host handles the CancelKeyPress event, and ultimately allows all of the worker threads in NServiceBus to finish handling messages before shutting down.

One question this brought up is what happens if someone pulls the plug on my machine after a message has been received, but before it has been processed? My impression was that a transactional endpoint would take care of this, but now I'm not so sure.

Over-testing: the data-driven edition

Learning to test at the right level of abstraction is tough. A common mistake involves the abuse of data-driven testing tools. Data-driven testing tools, like TestCase in NUnit, are convenient and powerful. You can write a lot of tests really fast, but if you're not careful, you can end up repeating yourself.

For example, suppose we define the following (woefully oversimplified) utility function to test if a string is a valid email address:

public static bool IsValidEmail(string email)
{
    return Regex.IsMatch(email, @"\w+@\w+\.com");
}

The test suite for this function should be complete. A data-driven approach works really well here:

[TestCase("a@test.com")]
[TestCase("1@test.com")]
//... lots more
public void TestValidEmail(string email)
{
    Assert.That(IsValidEmail(email));
}
[TestCase(null)]
[TestCase("not an email address")]
//... and so on
public void TestInvalidEmail(string email)
{
    Assert.That(IsValidEmail(email), Is.False);
}

Now, suppose we use this method in some application logic:

public void SendConfirmation(string message, string email)
{
    if (!IsValidEmail(email))
    {
        throw new ArgumentException("email");
    }
    SendEmail(message, email);
}

Awesome, let's test this bad boy... er, I mean here's the test we wrote before writing the method:

[TestCase("a@test.com")]
[TestCase("1@test.com")]
//... lots more
public void TestSendEmailIfValid(string email)
{
    SendConfirmation("test", email);
    //Assert sends email...
}
[TestCase(null)]
[TestCase("not an email address")]
//... and so on
public void TestSendEmailThrowsIfInvalid(string email)
{
    SendConfirmation("test", email);
    //Assert throws exception...
}

Okay, stop... do we really need to re-test every single valid and invalid email address? Data-driven tests make this easy, but that doesn't make it right. In this function, there are really only two paths that matter: an invalid email address throws an exception, and sending an email successfully. Let's try again:

public void TestSendEmailIfValid(string email)
{
    SendConfirmation("test", "validemail@test.com");
    //Assert sends email...
}
public void TestSendEmailThrowsIfInvalid()
{
    SendConfirmation("test", "not an email address");
    //Assert throws exception...
}

Mockists might even take this so far as to stub out the IsValidEmail function for the purposes of testing this function, but for this simple case, it might be overkill since IsValidEmail has no external dependencies to begin with.

In the real world, this kind of mistake is harder to spot. Generally, if you have an explosion of TestCase attributes covering all kinds of permutations, consider revisiting the code under test. There may be some opportunities to simplify both your code and your test suite.