Over-testing: the data-driven edition

Learning to test at the right level of abstraction is tough. A common mistake involves the abuse of data-driven testing tools. Data-driven testing tools, like TestCase in NUnit, are convenient and powerful. You can write a lot of tests really fast, but if you're not careful, you can end up repeating yourself.

For example, suppose we define the following (woefully oversimplified) utility function to test if a string is a valid email address:

public static bool IsValidEmail(string email)
{
    return Regex.IsMatch(email, @"\w+@\w+\.com");
}

The test suite for this function should be complete. A data-driven approach works really well here:

[TestCase("a@test.com")]
[TestCase("1@test.com")]
//... lots more
public void TestValidEmail(string email)
{
    Assert.That(IsValidEmail(email));
}
[TestCase(null)]
[TestCase("not an email address")]
//... and so on
public void TestInvalidEmail(string email)
{
    Assert.That(IsValidEmail(email), Is.False);
}

Now, suppose we use this method in some application logic:

public void SendConfirmation(string message, string email)
{
    if (!IsValidEmail(email))
    {
        throw new ArgumentException("email");
    }
    SendEmail(message, email);
}

Awesome, let's test this bad boy... er, I mean here's the test we wrote before writing the method:

[TestCase("a@test.com")]
[TestCase("1@test.com")]
//... lots more
public void TestSendEmailIfValid(string email)
{
    SendConfirmation("test", email);
    //Assert sends email...
}
[TestCase(null)]
[TestCase("not an email address")]
//... and so on
public void TestSendEmailThrowsIfInvalid(string email)
{
    SendConfirmation("test", email);
    //Assert throws exception...
}

Okay, stop... do we really need to re-test every single valid and invalid email address? Data-driven tests make this easy, but that doesn't make it right. In this function, there are really only two paths that matter: an invalid email address throws an exception, and sending an email successfully. Let's try again:

public void TestSendEmailIfValid(string email)
{
    SendConfirmation("test", "validemail@test.com");
    //Assert sends email...
}
public void TestSendEmailThrowsIfInvalid()
{
    SendConfirmation("test", "not an email address");
    //Assert throws exception...
}

Mockists might even take this so far as to stub out the IsValidEmail function for the purposes of testing this function, but for this simple case, it might be overkill since IsValidEmail has no external dependencies to begin with.

In the real world, this kind of mistake is harder to spot. Generally, if you have an explosion of TestCase attributes covering all kinds of permutations, consider revisiting the code under test. There may be some opportunities to simplify both your code and your test suite.