Category Archives: Unit Testing

No Null Checks

After running the automated tests generated by Pex, it’s amazing how many functions in the old codebase received various class or structure arguments and immediately started working with the data in them without ever checking for null. It was a paradigm followed by the old public code release of Diku, Envy, Merc, etc. and was unthinkingly carried on by the Basternae 2 programmers, myself included. This lack of data validation is being corrected.

It’s no surprise that the old codebase crashed a lot. After seeing the extent of the damage, I feel like I need to go back and revisit the Magma codebase and add a lot of stabilization.

In fact, one of the main things with Basternae 3 that will help stability is the fact that all player-entered commands are executed within a try-catch block. This means that typing in a broken or unsupported command should never crash the MUD, which was a huge problem in the “bad old days”. Instead it will log an error and appear to do absolutely nothing from the player’s viewpoint.

Pex, Glorious Pex

Back when I was programming C++, I worked with a great test tool: Parasoft C++ Test.  It was great for auto-generating unit tests, which in my opinion is the best place to start.  Even if they need to be modified by a Human to get proper coverage and real-world test results, the ability to generate tests that automatically check for things like NULLs and ridiculous value ranges is a good start to build from.

Even better is the savings in time.  Manually typing out all of the tags, code, and various other “boilerplate” pieces is a pain.  With automated test generation, all you have to do is go in afterward and add rows for the tests you want to run.

This is especially helpful for larger projects (more than 100K lines of code), such as Basternae.

After creating a test project, I told Visual Studio 2008 to auto-generate tests for me.  It took about twenty minutes with the processor thinking at full speed.  I can’t imagine how long it would have taken a human, but since the Human brain functions at about 300Hz it would have taken a mighty long while.

After auto-generating 1,966 tests, I decided to run them.  It got just over a quarter of the way through with *LOTS* of failures (approximately none of this new code has ever been tested) and Visual Studio crashed.

Pex Crash Screenshot
 
One Visual Studio restart later and I was running tests again, with a massive amount of failures.

In the “CharData” class alone, 212 tests were generated and 199 of them threw exceptions on the first run.  The rest were “Inconclusive”, which is the default result for auto-generated tests.

Already, Mr. Pex has made it obvious that the application has some architectural flaws, chief among which is the random number generator.  The generator lives in an instance of the “SystemData” class when it should be in the “RandomNumber” class, which should be a singleton.  When running without the controlled startup environment of the entire codebase, things break horribly.  That’s one of the things a test framework is supposed to point out, right?

While automated test generation is not all-inclusive, Pex is extremely helpful for getting to the point where your tests are at least “Inconclusive” and ready to edit by a Human.  If you get exceptions without even getting that far, you have some work to do (as I have).

I have seen no better tool for getting a quick and easy start on unit testing.  Writing meaningful tests?  Well, that’s something you have to figure out on your own, but at least the boilerplate is done for you.

Here’s where you can get more info on Pex, including a video demonstration:

http://research.microsoft.com/en-us/projects/Pex/

It is worth noting that this Pex is extremely beta.  Every time I run it, I get the “VSTestHost.exe has stopped working” dialog box.  Even with that, all of the tests do manage to run.

Unit Testing With MbUnit

In general either code works or it doesn’t and it’s easy to tell whether it does or doesn’t work. At least until you reach a certain level of complexity. At some point a project gets large enough that you can’t tell which project/dll your error is coming from, let alone which of the 100K or even 50M lines of code is doing naughty things.

Today I had a problem that was easier to solve by creating unit tests to point out where the flaw was. The tool uset? MbUnit.

Here’s an example of an MbUnit test I wrote in order to find a string processing bug:


using System;
using System.Collections.Generic;
using System.Text;
using MbUnit.Framework;
namespace UnitTests
{
[TestFixture]
public class MUDStringTests
{
[RowTest]
[Row(12, "12.box", "box")]
[Row(1, "cheese", "cheese")]
[Row(1, "1.pie", "pie")]
[Row(1, ".chicken", "chicken")]
public void NumberArgument(int expectedValue, string inputString, string expectedOutput)
{
string str = String.Empty;
Assert.AreEqual(expectedValue, BasternaeMud.MUDString.NumberArgument(inputString, ref str));
Assert.AreEqual(expectedOutput, str);
}
}
}

What this does is check whether we’re getting the expected output when we give a certain piece of input. It’s handy for detecting anomalies without having to boot up the server and/or mess with production code. Mind you, there’s no such thing as ‘production code’ since we don’t have a server up yet, but this sort of thing is likely to be handy in the future.

I’m not the type that has any chance of turning into a ‘write the unit tests first’ and ‘unit tests are more important than code’ evangelistic freak, but in most cases it is far easier than using trial-and-error to track down a bug.

I still hate the way WordPress formats my code.