Do you write installers? A lot of times I hear that all the features or stories are completed, but nobody has given any thought to how actually to ship the software and install it. This is not a trivial story, and usually it is the user's first experience with the software, so if it doesn't go well, your reputation starts out in the hole and that's not easy to overcome. Let's assume that we are more thorough and professional than the average joe, and we create an installer for our application that does three things: Install, Upgrade, and Uninstall.

Most of the time I write installers that are MSI or Setup.exe based, since I sort of get those for free with Visual Studio and .NET. If I have a file or dependency, I simply add it to the package and tell it where on the target machine it is supposed to go. It gets compiled and built (only by DEVENV.EXE by the way - MSBUILD doesn't know how to build setup projects...) after the main assemblies.

The main assemblies all have unit tests of course that run with the build, so everybody is green before anything gets built to install. Installers can be kind of tricky. Don't be tempted to run it on your native development box. If you haven't got lots of proven experience with them, be sure to run the installer for the first few iterations ONLY on a virtual machine that you can restore to its original image with a couple clicks. Let's just say I have seen some interesting recursive registry damage from an installer...

We need tests for the installer, but in this case we are not talking about "unit" tests, but rather functional tests. We need the tests to actually install and uninstall the software.

Testing the Installer Code
First up - test the UNinstall. From my experience, I have seen this path as the best one to start with. Install it manually only after the failing test has been created (and test it only on the VM). You can use either NUnit or MSTest as the framework to run the installer. HINT: to uninstall the software exec this command line:
MSIEXEC /X {guid}
where guid is the identifier for your project. This guid can be found in the .VDPROJ file as the key called "ProductCode." Don't change this value ever, or it will disconnect your next install from your current uninstall, and all kinds of weirdness can happen.

Assert that the registry keys get removed from CurrentVersion as discussed below, and that any registry entries created by the installer are removed.

Gotcha
So it didn't work as planned, and there's an error... an Assert fails, which throws an exception, and your tests end and have left the software installed on the system. The next time the install test runs, it won't work because the software will already be installed. So... make sure you have try-catch logic on everything before you assert anything. The finally block should call a function that uninstalls the product. We know that it will work, because we have tests for it...

Unlike most unit tests where we want to be able to run tests in any order, and have complete independence, functional tests for an installer are much simpler to write if they can be run in the Install, Upgrade, and Uninstall order. However, we don't write them in this order, we start with the uninstall first, then add tests for install, and finally the upgrade cases.

Make sure to catch everything done by the installer in the uninstall test, and do not proceed to write install tests until you are absolutely certain that the uninstall works properly and is adequately tested.

When you move on to the install case, it should be a bit easier, as all you need do is reverse the tests in the uninstall. Make sure that you place this test code before the uninstall test code, so that it is now automated.

Now, voila - you have automated tests for your install code. Finally, your upgrade test can be written to confirm the behavior if the installer is run on a system where the software is already installed. You may want to just fail the install and tell the user to uninstall the previous version, or you may want to upgrade in place. A true upgrade is far nicer to the user, but it is a lot of complexity some times when data migration, customized configurations, or other scenarios are needed. Make sure that any of this functionality is captured in stories, and documented with thorough testing.

What do I test?

• Make sure to open the registry after you install your application, and find the key to make sure you can uninstall it later:
• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{guid}
• This key's value should have the above mentioned MSIEXEC uninstall statement. Check it with an assert to make sure it's correct.
• Make sure each of the assemblies and executables are sent to the correct target folder on the target machine.
• If you support command line options or fancy things for the installer, make sure to test each and every one.
• Write the tests first of course...
• Make sure that config files, documentation, and support files (such as XML or other things) are delivered to the correct locations.
• If any files are dynamically generated or modified by the installer with custom actions after delivery, make sure that the modifications are correct.
  
• Make sure that directories are created on install
• Make sure that the ACL security on the folders and files is correct, if this is something your installer does.
  
• Make sure that directories are deleted on uninstall (except if logs or other artifacts are in them, they won't be deleted)
  
• Make sure these things are written in such a way that they are easy to call from a test, and that they handle errors and take care to leave the system in the same state as before the tests ran (with the software uninstalled).
  

Where do I run the tests?
A word of advice: DO NOT RUN THESE TESTS ON THE BUILD SERVER... it is tempting since they look like "unit" tests, but don't do it. Build servers are not something we want to be messing with installing and uninstalling software. Especially software that gets built every few minutes... The best way to automate the whole process is to have the build server build the app, the installer, and the installer tests and deliver them to an install server, or sandbox server. This can be done through a build step that copies all the executables to a UNC share on the sandbox server after each build. The sandbox server can be set to watch for new files in a folder and automatically begin to run the tests. If you are using CruiseControl.net, this is nicely accomplished by running CC.NET on both the sandbox server and the build server. The build server can "import" the red/green status of the functional tests from the installer to its own dashboard, just as if they were local. It takes a bit of work to get this all figured out, but once it's done it is a nice way to live. I wish I had all my CC.NET code to post, but unfortunately it belongs to someone else and I don't have the rights to post it. If someone has wrapped this concept up in an easier to run turn-key package I would love to hear about it with a comment.

If there are any questions about this, please feel free to comment below and I will try to reply as soon as I am able.

Why to Do It
When running lots of unit tests, there are distinct advantages in using all of those cores and cpu's that most of our modern computers have. We can parallelize testing so that we gain two distinct advantages. First, the testing will complete much faster. If you have 4 cpu's or 4 cores, we should see at least a halving of test time, if not more. Secondly, we prove our code is able to run in the closer-to-real-world scenario that other things might be going on while our code under test is executing. Plus its just cool to do multi-threading.

The by-product or side effect benefit we get also in multi-threading our tests is that we need to put more careful thought into the design and construction of test cases, and by that virtue, our software under test. The ability to multi-thread tests is I think one of the highest bars we have in demonstrating conclusively that our software is robust.

Advice on How to Do It
Separate tests into separate test assemblies that can be launched by the build server in parallel. There is no real reason to sequence tests, unless they sometimes interfere with each other. If they do interfere with each other, the tests are poorly designed and should be re-written as independent.

See the Pragmatic Unit Testing book or see this article for good advice on how to write good tests that you can parallelize.

Tests must be thread-safe. We probably should be able to run the same test at the same time without collisions.
Tests must be TRULY independent. This usually means that tests need to generate data that is keyed specific to that test run.
Tests must not have side-effects. They must leave the test system in the state they found it.

The new version of NUnit (2.5) is rumored to have functionality to be able to run tests in parallel threads. The currently released version has only one thread to run all the tests. In some preliminary research I have done, when manually coding up multiple threads to run each test in parallel from a single "unit test" method, it has some contradictory behavior. NUnit reports that all tests pass, but the GUI bar turns red.

More to follow on multi-threading test with MS Test...

I was explaining to someone about different types of testing I do with my code, and how mock objects can be used in unit tests. I used the example of the paint truck at the airport that spray-paints lines on the tarmac. Its job is to put down wide lines, narrow lines, in yellow, and white, and of a certain length, at specific places on the runways and taxiways. The truck is driven by a person, and the paint spray system is operated by driver. It's not a perfect example, but it may get the point across.

When we model this with truck software and try to test it, most developers seem to tend to go right for testing the whole solution end to end. That means trying to figure out how to fire up the engine, position the truck at certain coordinates, be at a certain speed, and then test the spray gun and see if it paints the correct stripes. It's a hard problem to solve, to try to set up with all these things and get it just right - just to see if the painter works properly. It can be a daunting task to try to test software this way, yet most developers who do "unit testing" are trying to do just this kind of thing. My take is that this is not really "unit testing" but actually "functional" testing. While it is definitely good to have functional tests, I prefer my development team to focus on more narrow areas such as individual classes. If these are thoughtfully designed and tested individually, they should work well when combined. This is also where integration and scenario testing come in. These types of tests are beyond the scope of unit tests.

Let me give an example that explains how we might be able to go about testing this a bit more carefully. Let's take the truck as a single "system" and the paint sprayer as a separate "system." These systems are really not very coupled - the sprayer runs on the truck's electric system, but that's about it. It makes sense to decouple them in trying to test them.

If we separate out the sprayer as a separate system, the only inputs it needs from the truck are the electrical connection and the trigger input. We can "mock" out these connections by supplying a new power source and trigger input connector that has the same "interface" (that is - implements the same interface class in code). Now we can test the system without needing to have the actual truck around. This is the concept of mock objects at its best in unit testing.

We can separate even the power mock object from the trigger mock object. We can control how these mock objects behave precisely, and keep them in specific states so that the only variables under test are in the sprayer system. We can then test what happens to the sprayer if we make the power go away, what happens if we trigger both paint guns at the same time, and making sure we can turn on and off each gun. These make great unit tests and document how our code is supposed to work. They should run very fast, and not require any diesel fuel...

All of the ability we have to be able to mock things like this, depends on programming to interfaces, which is a long-time engineering best practice to avoid decoupling. If we design our software to implement interfaces, we can then easily substitute dependency systems at test time with our own controlled version, and focus the testing on a more narrow area, while making the tests run faster at the same time.

I have been using TestDriven.net as a tool now for a few weeks, and I have to say that it is a fine addition to my tools collection. It gives me the flexibility I need to run tests in any manner I choose, simply by right-clicking. It's a Visual Studio add-in for 2005, and 2008, and it works seamlessly with a variety of test frameworks (NUnit, MbUnit, and MSTest being the most notable).

Speaking of NUnit, another excellent tool that I find invaluable to help me test my code. NUnit now has RowTest extensions (like MbUnit) in the production release (2.4.7 and if you aren't using this version, you should upgrade), and Parameterized Tests coming soon, and they look really cool. I find that I can replace a lot of repetition in my tests with this row based system. It enables me to write more tests faster, and that's always good. I am hoping that the new release goes into production mode very soon.

UI testing with Selenium is another favorite topic, I will write up another article later with examples of how to do real automated user interface testing using NUnit and Selenium. However, feel free to start playing with it now... its a great tool also, and plays nicely with the other tools in the toolbox.

Use these tools in good health, to help get your code into production as bug-free as possible by testing the heck out of it!

P.S.
Please make your unit tests into separate assemblies... no one-massive-monolithic-test assemblies please... and the tests should run FAST. If you mock out all the database, network, web service, etc. calls, you should be able to get all of your tests in an assembly run in under 5 seconds. Aim for a target of 100 tests per second, and you will be a much happier camper.

If you are a real stickler for this (and I recommend it...) add this code to your test assemblies:

private static DateTime assemblyStart;

 [ClassInitialize]
 public static void MyClassInitialize(TestContext testContext)
 {
     assemblyStart = DateTime.Now;
 }

 [TestCleanup]
 public static void TestCleanup()
 {
     Assert.IsTrue(
         assemblyStart.AddMilliseconds(5000) > DateTime.Now,
         "Tests took too long to execute.");
 }

Enjoy!