Why to Do It
When running lots of unit tests, there are distinct advantages in using all of those cores and cpu's that most of our modern computers have. We can parallelize testing so that we gain two distinct advantages. First, the testing will complete much faster. If you have 4 cpu's or 4 cores, we should see at least a halving of test time, if not more. Secondly, we prove our code is able to run in the closer-to-real-world scenario that other things might be going on while our code under test is executing. Plus its just cool to do multi-threading.

The by-product or side effect benefit we get also in multi-threading our tests is that we need to put more careful thought into the design and construction of test cases, and by that virtue, our software under test. The ability to multi-thread tests is I think one of the highest bars we have in demonstrating conclusively that our software is robust.

Advice on How to Do It
Separate tests into separate test assemblies that can be launched by the build server in parallel. There is no real reason to sequence tests, unless they sometimes interfere with each other. If they do interfere with each other, the tests are poorly designed and should be re-written as independent.

See the Pragmatic Unit Testing book or see this article for good advice on how to write good tests that you can parallelize.

Tests must be thread-safe. We probably should be able to run the same test at the same time without collisions.
Tests must be TRULY independent. This usually means that tests need to generate data that is keyed specific to that test run.
Tests must not have side-effects. They must leave the test system in the state they found it.

The new version of NUnit (2.5) is rumored to have functionality to be able to run tests in parallel threads. The currently released version has only one thread to run all the tests. In some preliminary research I have done, when manually coding up multiple threads to run each test in parallel from a single "unit test" method, it has some contradictory behavior. NUnit reports that all tests pass, but the GUI bar turns red.

More to follow on multi-threading test with MS Test...

I was explaining to someone about different types of testing I do with my code, and how mock objects can be used in unit tests. I used the example of the paint truck at the airport that spray-paints lines on the tarmac. Its job is to put down wide lines, narrow lines, in yellow, and white, and of a certain length, at specific places on the runways and taxiways. The truck is driven by a person, and the paint spray system is operated by driver. It's not a perfect example, but it may get the point across.

When we model this with truck software and try to test it, most developers seem to tend to go right for testing the whole solution end to end. That means trying to figure out how to fire up the engine, position the truck at certain coordinates, be at a certain speed, and then test the spray gun and see if it paints the correct stripes. It's a hard problem to solve, to try to set up with all these things and get it just right - just to see if the painter works properly. It can be a daunting task to try to test software this way, yet most developers who do "unit testing" are trying to do just this kind of thing. My take is that this is not really "unit testing" but actually "functional" testing. While it is definitely good to have functional tests, I prefer my development team to focus on more narrow areas such as individual classes. If these are thoughtfully designed and tested individually, they should work well when combined. This is also where integration and scenario testing come in. These types of tests are beyond the scope of unit tests.

Let me give an example that explains how we might be able to go about testing this a bit more carefully. Let's take the truck as a single "system" and the paint sprayer as a separate "system." These systems are really not very coupled - the sprayer runs on the truck's electric system, but that's about it. It makes sense to decouple them in trying to test them.

If we separate out the sprayer as a separate system, the only inputs it needs from the truck are the electrical connection and the trigger input. We can "mock" out these connections by supplying a new power source and trigger input connector that has the same "interface" (that is - implements the same interface class in code). Now we can test the system without needing to have the actual truck around. This is the concept of mock objects at its best in unit testing.

We can separate even the power mock object from the trigger mock object. We can control how these mock objects behave precisely, and keep them in specific states so that the only variables under test are in the sprayer system. We can then test what happens to the sprayer if we make the power go away, what happens if we trigger both paint guns at the same time, and making sure we can turn on and off each gun. These make great unit tests and document how our code is supposed to work. They should run very fast, and not require any diesel fuel...

All of the ability we have to be able to mock things like this, depends on programming to interfaces, which is a long-time engineering best practice to avoid decoupling. If we design our software to implement interfaces, we can then easily substitute dependency systems at test time with our own controlled version, and focus the testing on a more narrow area, while making the tests run faster at the same time.

I have been using TestDriven.net as a tool now for a few weeks, and I have to say that it is a fine addition to my tools collection. It gives me the flexibility I need to run tests in any manner I choose, simply by right-clicking. It's a Visual Studio add-in for 2005, and 2008, and it works seamlessly with a variety of test frameworks (NUnit, MbUnit, and MSTest being the most notable).

Speaking of NUnit, another excellent tool that I find invaluable to help me test my code. NUnit now has RowTest extensions (like MbUnit) in the production release (2.4.7 and if you aren't using this version, you should upgrade), and Parameterized Tests coming soon, and they look really cool. I find that I can replace a lot of repetition in my tests with this row based system. It enables me to write more tests faster, and that's always good. I am hoping that the new release goes into production mode very soon.

UI testing with Selenium is another favorite topic, I will write up another article later with examples of how to do real automated user interface testing using NUnit and Selenium. However, feel free to start playing with it now... its a great tool also, and plays nicely with the other tools in the toolbox.

Use these tools in good health, to help get your code into production as bug-free as possible by testing the heck out of it!

P.S.
Please make your unit tests into separate assemblies... no one-massive-monolithic-test assemblies please... and the tests should run FAST. If you mock out all the database, network, web service, etc. calls, you should be able to get all of your tests in an assembly run in under 5 seconds. Aim for a target of 100 tests per second, and you will be a much happier camper.

If you are a real stickler for this (and I recommend it...) add this code to your test assemblies:

private static DateTime assemblyStart;

 [ClassInitialize]
 public static void MyClassInitialize(TestContext testContext)
 {
     assemblyStart = DateTime.Now;
 }

 [TestCleanup]
 public static void TestCleanup()
 {
     Assert.IsTrue(
         assemblyStart.AddMilliseconds(5000) > DateTime.Now,
         "Tests took too long to execute.");
 }

Enjoy!

OK people, for some reason this little gem has been overlooked by an awful lot of developers. Every time I see someone write an if statement that checks to see if a string is null, then if it is empty, I have to cringe. This static method on the String class is a very handy mechanism to do something that is extremely common in code I see.

instead of this

if (str == null || str == String.Empty)
{
    doSomething();
}

use this:


if (String.IsNullOrEmpty(str))
{
    doSomething();
}

It is so much cleaner, and faster too I think.

Just remember that this should probably be the first thing we write in the main line code after the second unit test is failing, for any method that takes a string parameter... [you know, the test where we send null in as the string parameter... then we have to write the if statement in the main code]. Which then follows closely with the third unit test in which we pass String.Empty as the value. But then again you already knew that.

We all know what design patterns are in software design and development. These kinds of patterns also are recognized in unit and other kinds of tests as well.

While it is not necessarily a new idea, it is a good idea. Here are some links I have found on the subject. Further research should yield a whitepaper soon, if I ever get time to write it.

Brian Marick's testing.com: http://www.testing.com/test-patterns/patterns/

A great example at CodeProject: http://www.codeproject.com/KB/architecture/autp5.aspx

RBSC: http://www.rbsc.com/pages/TestPatternList.htm

TypeMock Unit test patterns for .net: http://www.typemock.com/Docs/TestPatterns.html

Book: xUnit Test Patterns: Refactoring Test Code

Enjoy

Acceptance criteria should be our measuring stick when we write software. These criteria should give us the direction to go in writing features, and ultimately the final answer on whether the software is "done" or meets the customer's need. When the customer (or the customer representative, product owner, or business analyst) gives us a notion of what the software is to do for them, it represents a guideline and a goal for our development.

This article is a discussion of ATDD in an Agile development context with Scrum project management and Extreme Programming [XP] development methodology. However it does not apply specifically to Agile. These same techniques can be applied with almost any mechanism.

Test-Driven Development [TDD] says that we should first write a test before we write the code. This forces us to understand clearly what the code is to do, because we have to write the test to verify it. Writing the test is usually the hard part - the code next is easier. This is where the discipline of TDD is most needed. The test itself should be failing when executed, since we don't have the code under test [CUT]. This is a good sign - we need to actually "test the test." We then write as little CUT code as needed, just to make the test pass. Then, we start all over again - write a test, some more code, etc. We refactor the CUT as needed, to make it a cohesive, maintainable design, while still keeping all the tests passing. In this way, we have a complete test framework wrapped around our high-quality code, which makes certain that the code always functions the way we intended no matter what kinds of changes we make internally to it.

This process of TDD as described, is the traditional mechanism we now refer to at Unit Test-Driven Development, or UTDD. There are higher-level tests that we can write also, that have the same impact on our development process. Applying this same concept of a failing test driving us to write code is where this discussion now leads, but at one level of abstraction higher. We now look to the Acceptance Criteria to drive our TDD, or now ATDD.

1. Acceptance Criteria gives us a target for writing Automated Acceptance Tests [AAT's] a. Acceptance Criteria need to be written into an executable set of automated tests. b. AAT's should be able demonstrate to the customer that their criteria for the functionality has been met. c. Ideally, this should include as much detail as the requirements bear out, and as many tests and checks are necessary to convey that each of the criteria has been accounted for.

2. AAT's give us a reason to write code (they are failing - no CUT yet) a. AAT's can be written before writing the code, or written one at a time following this process.

3. We now write the empty framework for the CUT. Some AAT's may still be failing, but that's OK. a. This step deviates from strict TDD in that we are writing code, but not for the intent of passing a test. b. One example is that if a criteria states that a customer is able to bring up a web page with a set of information, obviously we need to provide a web page. c. This framework concept is creating the empty page or container for the code we are going to write. d. The empty framework should not cause AAT's to pass, because AAT's should written to test at a higher level of functionality.

4. To make the AAT's pass, we need to start providing the functionality inside the empty framework. Here we need to apply the standard TDD concepts, and write a failing unit test.

5. We now have a failing unit test, so we write enough CUT to pass that test.

6. We repeat at step 4, refactor, and continue following the UTDD process as described above.

7. When the functionality that the AAT's require is there, they will be passing, Unit Tests [UT's] are passing, and our story (feature) is complete. We now continue to step 1 again for the next story.

How do I do it? What do I need to get started? Acceptance Test Driven Development is a practice that can yield good returns on the time invested. Customers will be pleased with the software they receive, bugs will be fewer, and delivery times will be shorter. However, there are a few things that are needed to make this practice effective in an organization. Buy-in People must be bought-in to the concept that test driven development is a good thing. Management must support this practice, for if not, it will be a much more difficult thing to accomplish. Developers must be willing to put aside their "classical" training and try something new. Testing Tools If we can't write an automated acceptance test for our product, we can't do ATDD. We need a tool designed for testing the particular type of product we are shipping. Story Test IQ [STIQ] is a great tool for automated testing of web applications. Straight Selenium RC under C# code, driven by NUnit also works great, in my experience. VSTS has some good tools also, as do Fiddler, and probably lots of other products. The framework must be in place on day 1 of the iteration to begin writing acceptance tests. If it isn't available, we won't be able to do ATDD. Skills Developers need the skills to be able to envision what is to be delivered, and be able to write automated test scenarios for the acceptance criteria. Testers need to be able to validate that the tests cover enough of the functionality through testing the criteria, and provide other functional, integration, load, and performance tests in addition to the developers' unit tests and acceptance tests. Management needs to be able to direct the right resources at tasks, no matter whether they are Development or Test tasks. Sometimes it's best to blur the lines between the disciplines to make this practice most effective. Customers need to be aware that they are actually going to get what they specify (and perhaps ONLY that much). Education on criteria evolution is almost always required for customers. Don't be surprised if only 50% of the criteria are discovered in sprint planning and story generation for the first couple of iterations. Experience TDD takes discipline, and ATDD even more so. This is not a practice for an organization just beginning down the road of TDD. Put in at least 6 iterations of using strict TDD first before trying ATDD. The experience gained with using TDD will naturally flow into ATDD, and the organization and the customers will see the benefits. Perseverance Good things don't happen instantly. It takes some time to see the benefits of ATDD. Be patient for a few sprints. It should take approximately three iterations to get it dialed in. Customer Cooperation Customers must be available for providing their criteria for acceptance. Customers generally are not experts in this type of "specification" of criteria, and must often be helped through the process at first, to generate usable criteria.

Aren't developers just supposed to write their code and let the test team test it? ahem. no. As a developer, I take pride in delivering *working* software. If I don't test my own software, I think that I'm really failing to do my job. Testing is not just for "testers." As a professional software engineer, I need have not only unit tests for my code (a Development practice by the way), but I also need to write automated functional tests to make sure that all the functionality works as I think it should. Then, I need to have automated Acceptance tests that tell me if indeed the whole system works together to deliver the business solution that the story or requirements describe.

Acceptance Tests
	Why do I start with acceptance tests? Because... they should drive the entire development process. Test-Driven Development doesn't necessarily have to *always* mean Unit Test-Driven Development... The Acceptance tests drive me to write code. In order to write code, I need a unit test. Then I can write the code that passes the unit test and the acceptance test. job well done.

Unit Tests
	I write enough unit tests to be able to make sure that the functionality I write works as I intended. Not too many - tests require maintenance as well as the code they test - but enough to ensure that it's safe to refactor just about anything in the code and make sure that it still works as intended. This is the safety net that unit tests provide.

Code
	Write some. It should make the tests pass. Failure is not an option.

Functional Tests
	We need to make sure that all the pieces of functionality behave in a known way. This is what functional testing does.

Integration Tests
	We need to make sure that all the systems co-operate. Integration tests are deeper yet than functional tests, these are more like "end-to-end" tests of particular scenarios. Usually these scenarios cover all the happy paths of end-to-end, and most likely a couple of failure scenarios as well, to illustrate how the system overall behaves in failure modes.

Deployment Tests
	You DID want to INSTALL the software and use it, right? So... you got a test for that? Sure you do. Deployment tests are essential for making sure the software delivers and configures the binary bits in a way that's useful to the user. Uninstall is a particularly critical issue as well. Make sure that you have complete testing around these critical fundamental features of the software.

Advancing the Practice

Better unit testing. Mocks. High code coverage. Fast test suite execution.

What are some of the things we need to get to these goals?

Teaching TDD

Some developers have an open mind to learn TDD. We need an effective mechanism to teach them the skills of how to think test-first in a fairly rapid manner. TDD is not just about test-first, it's about code quality. To ensure code quality, we need good tests. If your developers aren't writing good code, chances are they are writing even lower quality tests, if any at all. How do we get good quality tests that make sure our code is in good shape?

Test authoring, like most things in life, requires skill, and above all - Practice.

Take some well-written unit tests, and go over them with your developers. Teach them about the parts of each test, the setup, the test, and the assertions (and cleanup). Show examples. Have the experienced developers review the unit tests the other developers write. Have your test lead review unit tests too.

(No, I don't mean Domain-Driven Design.)

Ah, but haven't we all seen the results of Date-Driven Development in action. Don't let this happen to you.

Report all non-agile software development practices to your local scrum master. http://UnScrum.com