Test Driven Developer - TDD

Balance Between Test-Driven Design and Test Maintenance Overhead

John E. Boal — Mon, 28 Jun 2010 05:26:54 GMT

Test-driven Development/Design when practiced properly, yields organic design growth as tests cause feature code to be written in response to failing tests. In *proper* practice, the entire code base is refactored as needed to ensure that all code and design smells are eliminated (while keeping tests passing). This essentially causes the design at any given time to integrate all the features written up to that point. Once this process is completed, the feature code should be relatively complete, if the tests were written well to dictate the behavior required by the customer. If Acceptance-Test Driven Development is used, the Acceptance Tests for the feature passing will indicate that it’s ready to ship. At any given point, the code base that is checked in should withstand the scrutiny of an outside code reviewer, and should embody all the best design principles. If it doesn’t, the “refactor” stage has not yet been completed.

Refactoring to Remove Code and Design Smells

I find many teams that either poorly implement or flat-out skip the step of refactoring the code base and overall code design as part of this cycle. This kind of organic development - sticking a new feature on a code base without integrating it into the design and making changes as required is like sticking a wad of gum on a bowling ball. It tends to be the entire code base lumpy, obtuse, and generally not pretty to look at. Teams can get away with this kind of development for a while more or less successfully (unfortunately). However this behavior causes LARGE problems for the team or those inheriting maintenance and new feature work on this code base. Often the team will have to stop to rewrite the whole code base at some point, because it’s too problematic to maintain the bowling ball and gum code. In practical business situations, it’s far far easier to maintain the overall design and integrate features into the design, course-correcting the design as needed to absorb all of the features into a coherent architecture.

This concept of going through the code base and refactoring all of it to embody all the features in it, and removing design and code smells does imply that if the design changes, so must the tests that correspond to the design. If the tests are tightly coupled to the code base, some design changes can be painful in terms of test maintenance overhead. There are methodologies that assist in decoupling the test from the code, and have the test just confirm the behavior without much knowledge of how the implementation is done. The use of polymorphism, mock objects to decouple behaviors from implementation, and the principle of dependency injection in general can assist with decoupling, and make test maintenance less painful (although perhaps somewhat tougher to write at the outset). There is overhead still however with design changes, and that should be expected, embraced, and *estimated* in planning…

Plan for Change

In iteration planning, changes to the design and implementation, along with test maintenance should be factored in to the estimates for work being done. When breaking out tasks for a story, I like to at least mention if not record a specific task for the design refactoring, and another for the test maintenance for design changes. I personally think these should be more or less estimable by most teams with at least rudimentary skills. If the team is new, add a task with a ballpark guess for the time, but be sure to add the tasks so they have visibility… Scoping the effort for the design changes and test maintenance will be easier and easier going forward as the team gains understanding of what impact new features have. And it is likely that for a team with a good solid design, these changes will be minimal if they are done incrementally.

Some teams prefer to finish the tasks for the features and their tests, and then have a single story for the design and test maintenance per iteration, and keep copying it from one iteration to the next. This works too, however I have seen issues where the iteration tasks don’t get completed, and the maintenance story gets bumped off or not completed in the sprint. This engineering debt is carried forward, and the code that is checked in for “completed” features for the sprint now carries debt with it as well.

In my experience, I have seen more success with making the elimination of design and code smells part of the “Done” criteria for each story, and in fact pre-check-in criteria. This way we can’t call a story “Done” [meaning completed, and shippable] until it *really* is. Design and test maintenance changes on a per-story level should be small, and usually bite-size enough for a team to be able to handle operating this way even in a short iteration period.

Don’t omit the design and test maintenance, plan for it specifically each iteration. This practice will help keep engineering debt from accruing from iteration to iteration. It is a key practice that will help keep a team, a project, and a code base (and a business) all on a successful course for the short, and the longer term.

Find a BUG, Write A TEST

John E. Boal — Wed, 23 Sep 2009 03:24:00 GMT

There are many different kinds of bugs that we encounter in software development:

missing functionality (doesn't do what's required)
unanticipated/undesired behavior (it did what??)
failure to check boundaries (or boundary conditions [e.g null])
user-interface issues (usability)
miscalculations (and logic errors)
control flow errors (failing to break out of a loop, etc.)
failure to handle errors (unhandled exceptions)
security issues (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of privilege)
data interpretation errors (what date is 11/12/13 anyway?)
interoperability errors (sending/receiving incorrect messages or data)

and most likely lots of others. Each of these areas has bugs that are all examples of opportunities to write a test that were missed along the development path.

Here is a simple real-world example. Today my Log file writing routine threw an unhandled exception when the file it was trying to write to was in use by another application (this is failure to handle errors, above). This caused the app to crash with an unhandled exception. In truth, I had written the code TDD, so it had tests, but none of them held the file open while it attempted to log data.

This was an unanticipated condition, but probably one that should have been pretty obvious too. However, I was trying to keep the code as simple as possible, and it's not a multi-threaded app, so there wouldn't have been *internal* collisions anyway. Regardless, the app crashing because its log file is being edited is probably not the desired behavior. So (being the test driven developer) I wrote this test first, to illustrate the bug:

[Test]
public void TestLogFileInUse()
{
    using (FileStream stream = File.OpenWrite(Utilities.LogFileName))
    {
        Utilities.Log("test");
    }
}

it's really simple, but it definitely did fail when the Log() method threw an exception because his file was being held open by the test. Only after I had a failing test, did I then modify the code to catch the IOException and handle it without crashing the application. I could also check for an event log entry if the file write fails, if that was the desired behavior.

The test is fast and simple, and it now guarantees not only that I fixed the bug by making the test pass [not crashing, the only requirement here] but also that the issue will never come back again.

This may be an almost trivial case, but it does illustrate how to use TDD when fixing bugs. I shouldn't really ever be adding new functionality to the mainline code without a failing test of some kind. Refactoring doesn't count, because we only refactor when all the tests are passing. Refactoring doesn't add any new functionality, it is just reorganizing the existing code to make it better.

The worst thing about fixing bugs is when they pop back up later... We hate regressions, because then we have to re-do re-done work YET AGAIN... Lean says that defects are waste, and if they come back after being fixed, then that makes the time spent on fixing them the second time even more wasteful.

This method of "find a bug >> write a test" is a Test-Driven approach to solving issues and making sure they never come back. Let's make regressions an artifact of history, and never fix a bug without having a failing test first!

Red, Green, Done.

John E. Boal — Thu, 04 Jun 2009 20:17:50 GMT

TDD is a great practice to use for code success, as long as the *entire* process is followed...

write tests
write code
refactor
all tests green
we're done right?

Refactoring is the Key to Successful TDD.
We need to look at the changes we made as part of the whole, NOT just in the small. We may have just added a method to a class that makes the class now responsible for more than just the one thing it was before... That's now a code smell (or design smell). When we've refactored the code to make it testable, we're not really done...

We now need to take a step back, and look at the overall design, the archtecture, and how our implementation is satisfying the problems we are solving. We should make sure at this point in time that we can say everything is "well-designed" and "well-implemented."

So many teams that are new to TDD, or that just practice test-first development, tend to forget this CRUCIALLY IMPORTANT step... It's not done, it's not estimated, and often the consequences are simply ignored until it gets to the point that the entire code base is impossible to maintain. This Technical Debt or Engineering Debt goes unaccounted in so many cases. Sometimes, this result ends up being one of the factors that teams use to stop using TDD. When compared to traditional waterfall design models, the output of teams that omit this critical step does not nearly measure up to the designs of waterfall methodology.

There's no free lunch... We still need to have time in our plan for DESIGN. In waterfall, it's all up-front. With TDD and refactoring, we need to do it after we have written the code. We can't just "omit it" - otherwise we're not really "done."

Each time I complete a story, part of my done criteria is to hold up the entire system to a bright light, and look through it for code and design smells. If there are any, it really isn't Done Done Done. I may check in the working code, but I don't close the story as complete until this task is finished.

I estimate in a guess at least for time for sniffing out code and design smells on each story. My guideline for this is at least 10% of the time the rest of the tasks on the story take. Sometimes this clearly isn't enough, as when a story or feature causes significant re-design, or the adaption of an existing design to a new strategy. Use judgement on the estimate, but *AT LEAST* include the task for each story, even if it doesn't get estimated. This method will keep the issue out in the open and remind everyone that it still needs to be done.

If your product owner balks at this additional time (and they most always do) remind them that its a little penalty now, or a HUGE one later. Unless it's a very short-term project on a temporary system, this is almost always a required task, so that maintenance and upkeep - even in the next sprint - aren't blown out of proportion.

At the end of the day, the TEAM is responsible for deciding the best course of action to satifsy the the stakeholders, and even if the PO resists, the development management chain is *still* a stakeholder... As professional developers we really should have the ability to push back on a resistant PO with good quality engineering standards.

Educate your PO's. Make sure people understand how important this critical step is in the story. Place a task on each story for design refactoring. And, last but not least, you can always place a large poster of a nose with a red line through it in the team space and caption it *No Code Smells*" ...

< Rant >

John E. Boal — Wed, 08 Apr 2009 19:05:23 GMT

OK people. STOP putting cowboy code on my desk. Seriously. TEST your software. REALLY. Please STOP writing BRAND NEW LEGACY code.
< /Rant >

OK I feel better. But not really. Just because a system is not considered to be a "production release" (meaning that people outside the particular organization or business won't have access to it) it DOES NOT mean that we can cowboy it up and plop a steaming pile of untested code out there for our colleagues or coworkers to consume.

ALL CODE NEEDS TESTS. Period.

Untested code is NOT finished. Please don't just assume that if the code "works" that it's ready to inflict on other people (or even yourself). Self-inflicted legacy code casualties are the second highest cause of career paralysis, according to 4 out of 5 dentists. By definition, code is "legacy" if it doesn't have tests.

Here's a plan to get your tests caught up.

Start Small
At least we need a set of "sanity" checks... if nothing else. Create a "test suite." Use xUnit or the like, keep it simple. Start with the checks for the obvious output... if that doesn't show up, then something is clearly broken.

Add Some More
One at a time if you have to, add another test for something. Any kind of test will do... unit, functional, acceptance... I recommend starting with acceptance and working backward. We already blew the op to be able to do TDD and get unit testing done, so now let's just make sure the surface behavior is adequate. Definitely do go in deeper if time permits.

New Feature
Each time a new feature is added, update the test suite to make sure everything is happy with changes that ripple out from the effect of the feature. Again - at least make sure that the obvious functionality is tested on the new code. Don't slack on it, keep at least the new code from being legacy.

Lather, Rinse, Repeat
Pick a time each week (say 4:30PM on Wednesdays) to put in a half-hour on writing one new test. Schedule it. Block the time out on your calendar so someone can't schedule a meeting... And hide if you have to - to avoid distractions.

What to Write
Focus on happy-path code tests only if there are none for that feature. Otherwise, do alternate and failure path tests, so that the behavior of the system is understood in circumstances other than the best-case. 90% of bugs occur in the not-happy path.

Using this strategy will get testing started, and contribute to a more robust and reliable product, no matter what it is. Remember that most "side projects" or "utilities" that are developed in-house will be used by your own people. We want them to have the best.

"But it's just a spreadsheet." OK, yes it is. I say we need to make sure to test even the lowly spreadsheet. Because, next week someone in Finance is going to discover it and begin to use it for their payroll planning... if there is a critical flaw in it, it might be YOUR paycheck that gets eliminated!

We should realize that even internal tools and web sites are usually used to make business decisions of some kind. We want to make sure that they are made using reliable tools, to the extent possible. Business relies on its data, now more than ever. Competition is fierce in a down economy - even small mistakes can be a big problem.

"Bread's not done until it's baked" - and neither is code! SO PLEASE: Test it. Bake it. Slice it... and don't get burned!

Writing Unit Tests With Test Driven Development

John E. Boal — Tue, 20 Jan 2009 01:11:28 GMT

I recently recorded a tutorial video of a presentation I am working on. The video is intended to be an introduction to TDD, and how actually to go about writing tests and code using TDD. I hear a lot of people using the term "TDD" without really understanding it. They typically are referring to "unit testing" or sometimes even test-first development, neither of which are really TDD. So my thought was to show it actually being done.

It is my first attempt ever to record a video presentation, and there is some kind of hum on the audio I couldn't get rid of - sorry. I was able to convert it to a down-res format (quick-time), using the HandBrake converter tool. Hopefully it's still readable. This is unscripted and off-the-cuff, so I am sure there's lots of room for improvement. If you have suggestions, please feel free to post a comment below. Otherwise, enjoy!

Writing Unit Tests Using Test-Driven Development Apple QuickTime format, 65MB, 28 min.

Scrum, TDD and Seven Other Buzzwords

John E. Boal — Wed, 17 Dec 2008 02:33:12 GMT

"We need to do TDD" said the client.
"Our team is not producing quality code, and aren't performing up to our expectations."
"So, we need to introduce TDD to make sure we get what we need."

Uh-huh I thought. Thinking to myself... so, why aren't your developers producing quality code? What are they doing instead? Are they even unit testing at all now? Do they know how to test or are unit tests just an afterthought? oh yeah, and are there any other methodologies you need to add for your team to be buzzword-compliant!?

TDD is not just a buzzword. TDD is not just a methodology a team can adopt if they are in trouble. Not hardly. TDD is the top of the pyramid. It requires a mind-shift that is not tremendously difficult, but at the same time is very difficult to accomplish without discipline. Not every team is ready for this step. I would hope to have a cohesive, collaborative, and performant team, teach them how to write good tests (first or last), show them how acceptance criteria can be automated into tests that guide the team toward completion. After going through all of these pieces, I would then introduce TDD as a small thing, and the discipline required would probably already be there at that point.

While it may be in vogue to say a team is "agile" and uses "tdd" (note: here in lower case) it's much more important that a team is producing quality software - buzzwords or not. I am not exactly advocating non-tdd practices, but on the other hand we need to be practical about delivery also. If a team can truly do test-driven development, then congratulations I say, they are well positioned to be at the top of the heap. If not, I would try to lead the team in the direction to be ready for TDD, whether or not it gets used. Strong unit testing along with other kinds of testing will deliver the quality that customers need. So, evolve. Learn, Test well. And, if you can, use TDD.. it's great!

TDD or not TDD? That isn't the question.

John E. Boal — Tue, 09 Dec 2008 20:17:56 GMT

"TDD makes code development take longer."
Yes, it does. 15-30% longer.

"TDD *may* reduce bug counts..."
Real TDD WILL reduce bug counts. Expect about an order of magnitude. 60-90% comparably.

"I don't want to pay up front for testing or make the code take longer to be feature complete."
That is one choice - but it will be a costly one... Defects will not only be more frequent, but they will probably take longer to fix and be more complex as well.

From a financial perspective, (aside from the customer satisfaction) TDD just makes fiscal sense. I think we would need a compelling reason to give management and shareholders why we are NOT using it.

There is a new video posted on Channel 9 of an interview of one of the Microsoft researchers who contributed to a set of case studies published earlier this year. The case studies were done on 4 teams (3 Microsoft and 1 IBM), and they seem to agree with the statistics and team performance numbers I have been citing for years.

These numbers are motivators for me, and some of the primary reasons I practice TDD, support it, and try to spread the word.

Please take a look at the report and see for yourself if you agree...

Unit Testing and Test Driven Development - a Practical Guide

John E. Boal — Sat, 15 Nov 2008 21:08:59 GMT

Here is a presentation I wrote on what Unit Testing is all about, and how TDD fits into the ATDD cycle.

There are specific things here on testing the UI code with Selenium and JSUnit, and recommendations on how to do unit testing on your database code.

This presentation is in PDF format, but I can post the PPTX format also if needed.

A Practical Guide to Unit Testing1.pdf (503.29 KB)

Test-Driven Process Improvement with Refactoring, Value Stream Mapping, and Windows Workflow

John E. Boal — Thu, 30 Oct 2008 20:47:03 GMT

Value Stream Mapping [VSM] is a lean concept that helps us improve our processes by focusing on the whole process and making stepwise refinements on smaller parts of the process, to gain improvement for the whole. It is in a sense, "Refactoring" a process.

In an ideal case, a process can be thought of as a workflow, and perhaps we might be able to represent it using the WF Windows Workflow Foundation code delivered in .NET 3.5. It would be nice to show a code example here of how we can represent a process with workflow steps, but I didn't have time to write up an example as of yet.

With VSM, we are trying to improve a process by mapping it into parts, each of which we can study. We can gather metrics on each of the process steps, and metrics on how long things wait between steps. The waiting between steps is usually far more significant time-wise than any of the steps themselves. WF can be instrumental in providing an automated framework for collecting and reporting these metrics.

Indeed there are also other workflow alternatives of course. BizTalk Server 2006 R2 is a fine candidate for workflow representation as well, when adapters are available for systems that interoperate. I have a specific Microsoft focus in my career, but I have used CapeClear as a Java-based workflow system, and there are others in the ESB space which have features that would allow us to collect metrics on an automated representation of our processes.

Automation is key to refactoring, as in any development mechanism. We can't fix what we don't know is broken. With VSM, we focus on the entire process as a whole, so we need to collect measurements on where it is today, and then we can see total improvement on the whole as we make small changes (refactoring the process).

Many enterprises today use Service-Oriented Architecture [SOA] as a design pattern for the construction of enterprise systems. This architecture lends itself well to the use of WF or BizTalk (substiitute your favorite flavor here) workflow components. SOA helps us to map the entire process to blocks of functionality that we can measure and monitor easily, contributing to our VSM overall picture.

My theory is that like with software development, process improvement can also have a test-driven, scientific approach. VSM is a tool that helps us change the right things to improve our overall process, but it typically has measurements in large units - days, even weeks. With the right automation framework contributing to very rapid feedback with the metrics collected on processes, we can write tests that indicate Red or Green for not only each component being analyzed, but for the entire process as well.

Tests don't have to be in the form of an NUnit DLL or test fixture. We can use simple reports actually to give us the feedback we need to make decisions on how we refactor. Think of it as moving test-driven concepts a couple elevations higher in the plan view. From the 500 foot view to the 5000 foot view if you will. It's the exact same concept, just applied at a somewhat higher level of abstraction.

For each of our services we monitor the time taken in the process and the time between processes, and compare them to our initial baseline. We might even be able to get real-time data on a visible dashboard in the best of cirumstances. When we are better than our initial baseline, we are Green, if not, then we're Red. There is no yellow - same result means we haven't improved so it's still red. This concept is brutally simple, but it can be very effective, especially if the Red/Green report is made visible on a web-based report.

Mapping the workflow using WF, BizTalk, or other tools gives us a way to take automated measurements and compare them to our baseline to render a decision on whether our process has improved or not, and whether we are reaching our overall goal of total improvement. Individual portions of the whole can still be red as long as we achieve green for the overall process.

A business process really is a workflow, we just need to map it as such using software and automation. In this way we can apply the fine concepts of test-driven and refactoring to process improvement, once again using automation and measurement as our top key tools for leveraging our own technologies to help us improve them.

Code examples here would be good of course... Stay tuned for future posts with WF code examples of a process flow that has metrics and baseline comparisons for Red/Green.

Unit Testing in Python

John E. Boal — Tue, 30 Sep 2008 13:26:26 GMT

I have been using IronPython lately, and am having a good time testing code with its built-in unit test framework. Here is a brief example of how to write some unit tests for IronPython (or CPython if that's your flavor). These unit tests leverage the "unittest" module available in Python 2.2 and above.

Here is a short test class:

class TestCode:
    localInt = 1
    @classmethod
    def Method1(self):
        return 100
    def Method2(self, a, b):
        self.localInt = a * b

Here are some sample unit tests for it, using the unittest framework:

import unittest
class UnitTests(unittest.TestCase):
    def testMethod1(self):
        cut = TestCode()
        actual = cut.Method1()
        self.assertEqual(100, actual)

def teststaticInt(self):
self.assertEqual(1, TestCode.localInt)

    def testMethod2(self):
        cut = TestCode()
        cut.Method2(7, 4)
        self.assertEqual(28, cut.localInt)

if __name__ == '__main__':
unittest.main()

This example should illustrate that we can write some unit tests for our Python code fairly easily and quickly. It's nice to have a built-in framework for unit testing. There is also a "doctest" module available for doing more like system testing. I am pretty new to Python, so I haven't got examples of this as yet.

If anyone has any suggestions on tools or experience with integration with Visual Studio those comments would be appreciated.

Sample Code for Selenium and WatiN tests

John E. Boal — Tue, 26 Aug 2008 15:03:11 GMT

I posted a simple sample of accepance test code in Selenium and WatiN along with a sample web site to test. You can download the zip file here.

I also have posted a Fitnesse fixture in that zip file that illustrates how we can create a simple test fixture for Fitnesse acceptance testing. The Fitnesse tests aren't in the set, but here is the page wiki code that makes use of the fixture:

!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnet\FitServer.exe}
!define PATH_SEPARATOR {;}
!path dotnet\*.dll

Here is an acceptance test using our BusinessObjectTestFixture test class:

!|FitnesseFixture.BusinessObjectTestFixture|
|UserId|Password|Authenticate()|
|administrator|secret0|ADMIN|
|admin|secret0|NONE|
|administrator|secret|NONE|
|user11|secret11|USER|
|user11|secret0|NONE|
|user|secret|NONE|

I also created some STIQ tests, here is the code for the tests and components. Extract this zip file under repository\ProjectRoot folder and it should be able to test the sample site also.

ATDDSTIQ.zip (3.57 KB)

Do you test your installer?

John E. Boal — Tue, 01 Jul 2008 04:41:41 GMT

Do you write installers? A lot of times I hear that all the features or stories are completed, but nobody has given any thought to how actually to ship the software and install it. This is not a trivial story, and usually it is the user's first experience with the software, so if it doesn't go well, your reputation starts out in the hole and that's not easy to overcome. Let's assume that we are more thorough and professional than the average joe, and we create an installer for our application that does three things: Install, Upgrade, and Uninstall.

Most of the time I write installers that are MSI or Setup.exe based, since I sort of get those for free with Visual Studio and .NET. If I have a file or dependency, I simply add it to the package and tell it where on the target machine it is supposed to go. It gets compiled and built (only by DEVENV.EXE by the way - MSBUILD doesn't know how to build setup projects...) after the main assemblies.

The main assemblies all have unit tests of course that run with the build, so everybody is green before anything gets built to install. Installers can be kind of tricky. Don't be tempted to run it on your native development box. If you haven't got lots of proven experience with them, be sure to run the installer for the first few iterations ONLY on a virtual machine that you can restore to its original image with a couple clicks. Let's just say I have seen some interesting recursive registry damage from an installer...

We need tests for the installer, but in this case we are not talking about "unit" tests, but rather functional tests. We need the tests to actually install and uninstall the software.

Testing the Installer Code
First up - test the UNinstall. From my experience, I have seen this path as the best one to start with. Install it manually only after the failing test has been created (and test it only on the VM). You can use either NUnit or MSTest as the framework to run the installer. HINT: to uninstall the software exec this command line:
MSIEXEC /X {guid}
where guid is the identifier for your project. This guid can be found in the .VDPROJ file as the key called "ProductCode." Don't change this value ever, or it will disconnect your next install from your current uninstall, and all kinds of weirdness can happen.

Assert that the registry keys get removed from CurrentVersion as discussed below, and that any registry entries created by the installer are removed.

Gotcha
So it didn't work as planned, and there's an error... an Assert fails, which throws an exception, and your tests end and have left the software installed on the system. The next time the install test runs, it won't work because the software will already be installed. So... make sure you have try-catch logic on everything before you assert anything. The finally block should call a function that uninstalls the product. We know that it will work, because we have tests for it...

Unlike most unit tests where we want to be able to run tests in any order, and have complete independence, functional tests for an installer are much simpler to write if they can be run in the Install, Upgrade, and Uninstall order. However, we don't write them in this order, we start with the uninstall first, then add tests for install, and finally the upgrade cases.

Make sure to catch everything done by the installer in the uninstall test, and do not proceed to write install tests until you are absolutely certain that the uninstall works properly and is adequately tested.

When you move on to the install case, it should be a bit easier, as all you need do is reverse the tests in the uninstall. Make sure that you place this test code before the uninstall test code, so that it is now automated.

Now, voila - you have automated tests for your install code. Finally, your upgrade test can be written to confirm the behavior if the installer is run on a system where the software is already installed. You may want to just fail the install and tell the user to uninstall the previous version, or you may want to upgrade in place. A true upgrade is far nicer to the user, but it is a lot of complexity some times when data migration, customized configurations, or other scenarios are needed. Make sure that any of this functionality is captured in stories, and documented with thorough testing.

What do I test?

Make sure to open the registry after you install your application, and find the key to make sure you can uninstall it later:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{guid}
This key's value should have the above mentioned MSIEXEC uninstall statement. Check it with an assert to make sure it's correct.

Make sure each of the assemblies and executables are sent to the correct target folder on the target machine.
If you support command line options or fancy things for the installer, make sure to test each and every one.

Write the tests first of course...

Make sure that config files, documentation, and support files (such as XML or other things) are delivered to the correct locations.
If any files are dynamically generated or modified by the installer with custom actions after delivery, make sure that the modifications are correct.
Make sure that directories are created on install
Make sure that the ACL security on the folders and files is correct, if this is something your installer does.
Make sure that directories are deleted on uninstall (except if logs or other artifacts are in them, they won't be deleted)
Make sure these things are written in such a way that they are easy to call from a test, and that they handle errors and take care to leave the system in the same state as before the tests ran (with the software uninstalled).

Where do I run the tests?
A word of advice: DO NOT RUN THESE TESTS ON THE BUILD SERVER... it is tempting since they look like "unit" tests, but don't do it. Build servers are not something we want to be messing with installing and uninstalling software. Especially software that gets built every few minutes... The best way to automate the whole process is to have the build server build the app, the installer, and the installer tests and deliver them to an install server, or sandbox server. This can be done through a build step that copies all the executables to a UNC share on the sandbox server after each build. The sandbox server can be set to watch for new files in a folder and automatically begin to run the tests. If you are using CruiseControl.net, this is nicely accomplished by running CC.NET on both the sandbox server and the build server. The build server can "import" the red/green status of the functional tests from the installer to its own dashboard, just as if they were local. It takes a bit of work to get this all figured out, but once it's done it is a nice way to live. I wish I had all my CC.NET code to post, but unfortunately it belongs to someone else and I don't have the rights to post it. If someone has wrapped this concept up in an easier to run turn-key package I would love to hear about it with a comment.

If there are any questions about this, please feel free to comment below and I will try to reply as soon as I am able.

Multi-threading Unit Tests

John E. Boal — Wed, 04 Jun 2008 21:20:10 GMT

Why to Do It
When running lots of unit tests, there are distinct advantages in using all of those cores and cpu's that most of our modern computers have. We can parallelize testing so that we gain two distinct advantages. First, the testing will complete much faster. If you have 4 cpu's or 4 cores, we should see at least a halving of test time, if not more. Secondly, we prove our code is able to run in the closer-to-real-world scenario that other things might be going on while our code under test is executing. Plus its just cool to do multi-threading.

The by-product or side effect benefit we get also in multi-threading our tests is that we need to put more careful thought into the design and construction of test cases, and by that virtue, our software under test. The ability to multi-thread tests is I think one of the highest bars we have in demonstrating conclusively that our software is robust.

Advice on How to Do It
Separate tests into separate test assemblies that can be launched by the build server in parallel. There is no real reason to sequence tests, unless they sometimes interfere with each other. If they do interfere with each other, the tests are poorly designed and should be re-written as independent.

See the Pragmatic Unit Testing book or see this article for good advice on how to write good tests that you can parallelize.

Tests must be thread-safe. We probably should be able to run the same test at the same time without collisions.
Tests must be TRULY independent. This usually means that tests need to generate data that is keyed specific to that test run.
Tests must not have side-effects. They must leave the test system in the state they found it.

The new version of NUnit (2.5) is rumored to have functionality to be able to run tests in parallel threads. The currently released version has only one thread to run all the tests. In some preliminary research I have done, when manually coding up multiple threads to run each test in parallel from a single "unit test" method, it has some contradictory behavior. NUnit reports that all tests pass, but the GUI bar turns red.

More to follow on multi-threading test with MS Test...

Testing the Paint Truck

John E. Boal — Tue, 03 Jun 2008 18:55:32 GMT

I was explaining to someone about different types of testing I do with my code, and how mock objects can be used in unit tests. I used the example of the paint truck at the airport that spray-paints lines on the tarmac. Its job is to put down wide lines, narrow lines, in yellow, and white, and of a certain length, at specific places on the runways and taxiways. The truck is driven by a person, and the paint spray system is operated by driver. It's not a perfect example, but it may get the point across.

When we model this with truck software and try to test it, most developers seem to tend to go right for testing the whole solution end to end. That means trying to figure out how to fire up the engine, position the truck at certain coordinates, be at a certain speed, and then test the spray gun and see if it paints the correct stripes. It's a hard problem to solve, to try to set up with all these things and get it just right - just to see if the painter works properly. It can be a daunting task to try to test software this way, yet most developers who do "unit testing" are trying to do just this kind of thing. My take is that this is not really "unit testing" but actually "functional" testing. While it is definitely good to have functional tests, I prefer my development team to focus on more narrow areas such as individual classes. If these are thoughtfully designed and tested individually, they should work well when combined. This is also where integration and scenario testing come in. These types of tests are beyond the scope of unit tests.

Let me give an example that explains how we might be able to go about testing this a bit more carefully. Let's take the truck as a single "system" and the paint sprayer as a separate "system." These systems are really not very coupled - the sprayer runs on the truck's electric system, but that's about it. It makes sense to decouple them in trying to test them.

If we separate out the sprayer as a separate system, the only inputs it needs from the truck are the electrical connection and the trigger input. We can "mock" out these connections by supplying a new power source and trigger input connector that has the same "interface" (that is - implements the same interface class in code). Now we can test the system without needing to have the actual truck around. This is the concept of mock objects at its best in unit testing.

We can separate even the power mock object from the trigger mock object. We can control how these mock objects behave precisely, and keep them in specific states so that the only variables under test are in the sprayer system. We can then test what happens to the sprayer if we make the power go away, what happens if we trigger both paint guns at the same time, and making sure we can turn on and off each gun. These make great unit tests and document how our code is supposed to work. They should run very fast, and not require any diesel fuel...

All of the ability we have to be able to mock things like this, depends on programming to interfaces, which is a long-time engineering best practice to avoid decoupling. If we design our software to implement interfaces, we can then easily substitute dependency systems at test time with our own controlled version, and focus the testing on a more narrow area, while making the tests run faster at the same time.

Tools and Tips

John E. Boal — Tue, 03 Jun 2008 06:31:40 GMT

I have been using TestDriven.net as a tool now for a few weeks, and I have to say that it is a fine addition to my tools collection. It gives me the flexibility I need to run tests in any manner I choose, simply by right-clicking. It's a Visual Studio add-in for 2005, and 2008, and it works seamlessly with a variety of test frameworks (NUnit, MbUnit, and MSTest being the most notable).

Speaking of NUnit, another excellent tool that I find invaluable to help me test my code. NUnit now has RowTest extensions (like MbUnit) in the production release (2.4.7 and if you aren't using this version, you should upgrade), and Parameterized Tests coming soon, and they look really cool. I find that I can replace a lot of repetition in my tests with this row based system. It enables me to write more tests faster, and that's always good. I am hoping that the new release goes into production mode very soon.

UI testing with Selenium is another favorite topic, I will write up another article later with examples of how to do real automated user interface testing using NUnit and Selenium. However, feel free to start playing with it now... its a great tool also, and plays nicely with the other tools in the toolbox.

Use these tools in good health, to help get your code into production as bug-free as possible by testing the heck out of it!

P.S.
Please make your unit tests into separate assemblies... no one-massive-monolithic-test assemblies please... and the tests should run FAST. If you mock out all the database, network, web service, etc. calls, you should be able to get all of your tests in an assembly run in under 5 seconds. Aim for a target of 100 tests per second, and you will be a much happier camper.

If you are a real stickler for this (and I recommend it...) add this code to your test assemblies:

private static DateTime assemblyStart;

 [ClassInitialize]
 public static void MyClassInitialize(TestContext testContext)
 {
     assemblyStart = DateTime.Now;
 }

 [TestCleanup]
 public static void TestCleanup()
 {
     Assert.IsTrue(
         assemblyStart.AddMilliseconds(5000) > DateTime.Now,
         "Tests took too long to execute.");
 }

Enjoy!

C# String.IsNullOrEmpty Method - Making Sure Our Strings Are Valid

John E. Boal — Wed, 14 May 2008 15:21:24 GMT

OK people, for some reason this little gem has been overlooked by an awful lot of developers. Every time I see someone write an if statement that checks to see if a string is null, then if it is empty, I have to cringe. This static method on the String class is a very handy mechanism to do something that is extremely common in code I see.

instead of this

if (str == null || str == String.Empty)
{
doSomething();
}

use this:

if (String.IsNullOrEmpty(str))
{
doSomething();
}

It is so much cleaner, and faster too I think.

Just remember that this should probably be the first thing we write in the main line code after the second unit test is failing, for any method that takes a string parameter... [you know, the test where we send null in as the string parameter... then we have to write the if statement in the main code]. Which then follows closely with the third unit test in which we pass String.Empty as the value. But then again you already knew that. :-)

Acceptance Test Driven Development, Explained

John E. Boal — Wed, 02 Apr 2008 06:10:03 GMT

Acceptance criteria should be our measuring stick when we write software. These criteria should give us the direction to go in writing features, and ultimately the final answer on whether the software is "done" or meets the customer's need. When the customer (or the customer representative, product owner, or business analyst) gives us a notion of what the software is to do for them, it represents a guideline and a goal for our development.

This article is a discussion of ATDD in an Agile development context with Scrum project management and Extreme Programming [XP] development methodology. However it does not apply specifically to Agile. These same techniques can be applied with almost any mechanism.

Test-Driven Development [TDD] says that we should first write a test before we write the code. This forces us to understand clearly what the code is to do, because we have to write the test to verify it. Writing the test is usually the hard part - the code next is easier. This is where the discipline of TDD is most needed. The test itself should be failing when executed, since we don't have the code under test [CUT]. This is a good sign - we need to actually "test the test." We then write as little CUT code as needed, just to make the test pass. Then, we start all over again - write a test, some more code, etc. We refactor the CUT as needed, to make it a cohesive, maintainable design, while still keeping all the tests passing. In this way, we have a complete test framework wrapped around our high-quality code, which makes certain that the code always functions the way we intended no matter what kinds of changes we make internally to it.

This process of TDD as described, is the traditional mechanism we now refer to at Unit Test-Driven Development, or UTDD. There are higher-level tests that we can write also, that have the same impact on our development process. Applying this same concept of a failing test driving us to write code is where this discussion now leads, but at one level of abstraction higher. We now look to the Acceptance Criteria to drive our TDD, or now ATDD.

1. Acceptance Criteria gives us a target for writing Automated Acceptance Tests [AAT's] a. Acceptance Criteria need to be written into an executable set of automated tests. b. AAT's should be able demonstrate to the customer that their criteria for the functionality has been met. c. Ideally, this should include as much detail as the requirements bear out, and as many tests and checks are necessary to convey that each of the criteria has been accounted for.

2. AAT's give us a reason to write code (they are failing - no CUT yet) a. AAT's can be written before writing the code, or written one at a time following this process.

3. We now write the empty framework for the CUT. Some AAT's may still be failing, but that's OK. a. This step deviates from strict TDD in that we are writing code, but not for the intent of passing a test. b. One example is that if a criteria states that a customer is able to bring up a web page with a set of information, obviously we need to provide a web page. c. This framework concept is creating the empty page or container for the code we are going to write. d. The empty framework should not cause AAT's to pass, because AAT's should written to test at a higher level of functionality.

4. To make the AAT's pass, we need to start providing the functionality inside the empty framework. Here we need to apply the standard TDD concepts, and write a failing unit test.

5. We now have a failing unit test, so we write enough CUT to pass that test.

6. We repeat at step 4, refactor, and continue following the UTDD process as described above.

7. When the functionality that the AAT's require is there, they will be passing, Unit Tests [UT's] are passing, and our story (feature) is complete. We now continue to step 1 again for the next story.

How do I do it? What do I need to get started?

Acceptance Test Driven Development is a practice that can yield good returns on the time invested. Customers will be pleased with the software they receive, bugs will be fewer, and delivery times will be shorter. However, there are a few things that are needed to make this practice effective in an organization.

Buy-in

People must be bought-in to the concept that test driven development is a good thing.
Management must support this practice, for if not, it will be a much more difficult thing to accomplish.
Developers must be willing to put aside their "classical" training and try something new.

Testing Tools

If we can't write an automated acceptance test for our product, we can't do ATDD. We need a tool designed for testing the particular type of product we are shipping.
Story Test IQ [STIQ] is a great tool for automated testing of web applications. Straight Selenium RC under C# code, driven by NUnit also works great, in my experience. VSTS has some good tools also, as do Fiddler, and probably lots of other products.
The framework must be in place on day 1 of the iteration to begin writing acceptance tests. If it isn't available, we won't be able to do ATDD.

Skills

Developers need the skills to be able to envision what is to be delivered, and be able to write automated test scenarios for the acceptance criteria.
Testers need to be able to validate that the tests cover enough of the functionality through testing the criteria, and provide other functional, integration, load, and performance tests in addition to the developers' unit tests and acceptance tests.
Management needs to be able to direct the right resources at tasks, no matter whether they are Development or Test tasks. Sometimes it's best to blur the lines between the disciplines to make this practice most effective.
Customers need to be aware that they are actually going to get what they specify (and perhaps ONLY that much). Education on criteria evolution is almost always required for customers. Don't be surprised if only 50% of the criteria are discovered in sprint planning and story generation for the first couple of iterations.

Experience

TDD takes discipline, and ATDD even more so. This is not a practice for an organization just beginning down the road of TDD. Put in at least 6 iterations of using strict TDD first before trying ATDD. The experience gained with using TDD will naturally flow into ATDD, and the organization and the customers will see the benefits.

Perseverance

Good things don't happen instantly. It takes some time to see the benefits of ATDD.
Be patient for a few sprints. It should take approximately three iterations to get it dialed in.

Customer Cooperation

Customers must be available for providing their criteria for acceptance.
Customers generally are not experts in this type of "specification" of criteria, and must often be helped through the process at first, to generate usable criteria.

Testing? Aren't I a Developer?

John E. Boal — Mon, 24 Mar 2008 05:47:08 GMT

Aren't developers just supposed to write their code and let the test team test it?

ahem.

no.

As a developer, I take pride in delivering *working* software. If I don't test my own software, I think that I'm really failing to do my job. Testing is not just for "testers."

As a professional software engineer, I need have not only unit tests for my code (a Development practice by the way), but I also need to write automated functional tests to make sure that all the functionality works as I think it should. Then, I need to have automated Acceptance tests that tell me if indeed the whole system works together to deliver the business solution that the story or requirements describe.

Acceptance Tests

Why do I start with acceptance tests?

Because... they should drive the entire development process. Test-Driven Development doesn't necessarily have to *always* mean Unit Test-Driven Development...

The Acceptance tests drive me to write code.
In order to write code, I need a unit test.
Then I can write the code that passes the unit test and the acceptance test.

job well done.

Unit Tests
	I write enough unit tests to be able to make sure that the functionality I write works as I intended. Not too many - tests require maintenance as well as the code they test - but enough to ensure that it's safe to refactor just about anything in the code and make sure that it still works as intended. This is the safety net that unit tests provide.

Code
	Write some. It should make the tests pass. Failure is not an option.

Functional Tests
	We need to make sure that all the pieces of functionality behave in a known way. This is what functional testing does.

Integration Tests
	We need to make sure that all the systems co-operate. Integration tests are deeper yet than functional tests, these are more like "end-to-end" tests of particular scenarios. Usually these scenarios cover all the happy paths of end-to-end, and most likely a couple of failure scenarios as well, to illustrate how the system overall behaves in failure modes.

Deployment Tests

You DID want to INSTALL the software and use it, right?

So... you got a test for that?

Sure you do.

Deployment tests are essential for making sure the software delivers and configures the binary bits in a way that's useful to the user. Uninstall is a particularly critical issue as well. Make sure that you have complete testing around these critical fundamental features of the software.

Teaching TDD, and Advancing the Practice

John E. Boal — Wed, 19 Mar 2008 05:44:35 GMT

Advancing the Practice

Better unit testing. Mocks. High code coverage. Fast test suite execution.

What are some of the things we need to get to these goals?

Teaching TDD

Some developers have an open mind to learn TDD. We need an effective mechanism to teach them the skills of how to think test-first in a fairly rapid manner. TDD is not just about test-first, it's about code quality. To ensure code quality, we need good tests. If your developers aren't writing good code, chances are they are writing even lower quality tests, if any at all. How do we get good quality tests that make sure our code is in good shape?

Test authoring, like most things in life, requires skill, and above all - Practice.

Take some well-written unit tests, and go over them with your developers. Teach them about the parts of each test, the setup, the test, and the assertions (and cleanup). Show examples. Have the experienced developers review the unit tests the other developers write. Have your test lead review unit tests too.

ATDD - Acceptance Test Driven Development

John E. Boal — Fri, 14 Mar 2008 05:39:09 GMT

Use ATDD as an advanced extension of TDD, keeping the software's development guided by the principles of satisfying the customer's needs through acceptance criteria and automated testing.

BDD, ATDD, UTDD, DSL's ... when will it all end...

The drive toward business-driven testing has never been stronger. Developers are seemingly now finding a higher and higher bar when it comes to customers' expectations of quality and features. Our tools are getting better, and we can deliver more software, faster. But, our methodologies haven't necessarily changed enough to satisfy today's customer expectations.

Enter Business-Driven Design...

Business driven design is a concept that enables us to take business requirements and current priorities and turn them into a software design through Acceptance Test-Driven Development. The business requirements that drive the need for the software are turned into specific criteria that allow the business to decide what the criteria are that will allow them to use a feature and have it meet their business need. Rather than the old-school way of gathering requirements, and having a requirements document and a functional specification, we now turn to individual small criteria that decide if the software is acceptable to meet the need. Some of the criteria map directly from functional requirements, and others may not have been captured in a traditional requirements gathering and specifying model.

Domain-Specific Languages (DSL's) are key to success in Acceptance Test-Driven Development. DSL's give us a way to communicate with the customer and domain experts in their terms. When we capture criteria in this manner, it becomes quite clear to those with domain knowledge, what is meant and what is desired. There is no need for a "translator" between the customer and the developers (this used to be called "Business Analyst"). The developers model the code in terms of the language the customer already uses. This mechanism leads to better communication, better encapsulation, and better object-oriented development.

Acceptance Test-Driven Development [ATDD] gives us a mechanism to use DSL's and direct customer involvement in making sure the software we deliver meets the needs. When we take the criteria and turn them into automated acceptance tests, it is far easier for the customers to see that they are getting what they asked for. It's also easier for the developers to have a target to shoot for, and have a goal to meet. This way, they are more focused on delivering a specific unit of functionality that the customer needs rather than (as so often happens) some "new feature" that they thought might be useful.

Much care needs to be put into the way that acceptance criteria are gathered and then automated. If there is something that is missed, it could critically affect the design. This is an opportunity for customers and developers to collaborate and get it right. The customer needs to understand that if it isn't on the acceptance criteria list, it isn't going to be in the software... Performance criteria, interoperability with other systems, and other criteria like these are often missed. Customers should have many opportunities to review and re-review the criteria before they are approved. Even still, sometimes things are missed. This is why it is important for the customer to be involved at all stages of the development process. The customer shouldn't just be involved in the criteria gathering, then come back later for their product. If things are missed, they will likely become apparent and turn up in daily work. If the customer is there to be consulted, decisions can be made about how to integrate missed criteria, and how to capture these better in the future.

Business-Driven Design is a business-centric, collaborative, agile mechanism for delivering quality software to today's demanding customer.

Red, Green, Refactor

John E. Boal — Tue, 12 Feb 2008 07:17:18 GMT

RED. Why does it have to fail first?

Can't we just skip this step?

Nope.

We write the test to fail first... because we need to test... the test itself! How can we depend on a test that isn't ever going to fail?

Test is failing because there's no code yet.

Write just enough code to make the test pass (hard code the result).

Now that the test is passing, we can refactor.

But... I actually like to take this opportunity to write the next test.

New test is failing because the hard-coded value for the first test doesn't work (by design).

NOW... we can refactor. Adding just enough code to make both tests work. Here is where we put in the actual work that makes the code function.

Every time we have a new piece of functionality, we write a test.

Our tests will be an "executable" document on how we (programmers) intend the code to work.

Even if we didn't get the correct logic to satisfy the requirements (which we found out in our Acceptance testing)... At least we know what we intended our code to do.

GREEN - Test Passes!

Now that the test is passing, you're half way there. Move on to Refactoring, and keep the test green each time a refactoring is completed.

BUGS?

Got a bug? Great! You're just missing a test then...

DON'T JUST FIX IT! That's why bugs come back..

TDD: Find a bug ->Write a test!

The test expects the correct behavior... and fails of course! NOW fix the bug ... and the test passes.

That bug will NEVER be back again because your test will always ensure that the code will do the right thing.

This is the safety net that TDD, Continuous Integration, and test automation give us. We can now refactor mercilessly...

Onward to Refactoring!

Refactor

Use CI tools. Build often.

Change the order of your tests... do they still all run green?
Good tests are made to be independent. They set-up the system, test it, determine success, and then CLEAN UP. Especially if there is an error, CLEAN UP is essential.

Be a good test citizen. Just like when hiking - pack it in, pack it out - After the tests run, always make sure leave things as pristine as you found them.

Write tests as appropriate

We need to make sure to have adequate unit testing. So what does that mean exactly? It depends on the project.
Some projects will require unit testing for every single control flow, and some will only require cursory tests. It may vary even from class to class.
An example I give is a DTO class (data-transfer object). This class has no logic in it, and no functionality - only properties. Do we need to test it? I would estimate in most cases - no. If we have an automated tool that generates property tests for it, then great, use it. Otherwise I think it's a low test risk. Not zero risk though.

Remember when you're so gung-ho about testing that you generate all those unit tests... those tests will have to be maintained along with the mainline code... Test what makes sense, but don't over-do it. Test maintenance costs time and money too.

Make small incremental changes. Once the method has its functionality completed and the tests are passing, check it in! Ideally check-in's to the source repository should occur several times a day (perhaps as often as every couple of hours). If there is a code review process (and there should be) it should take a very small amount of time to explain and review one incremental change. If you're using pair programming (and you should try it if you can) then your code has already been reviewed (pairing is also called real-time code review), and it's ready to check in as often as every half-hour to an hour.

There are lots of times when code gets refactored from a public method to a private one. Many times there are good reasons to combine and refine the internal workings of a public method into a private one. Many developers over the years have insisted they should be writing specific tests on these private methods.

In the .NET world with C#, we can use the "internal" keyword rather than private to allow only classes in the assembly to access the method. We can also use the .NET "InternalsVisibleTo" attribute to allow the test assembly to access the internal methods so they can be tested, yet not be accessible publicly.

However... if we are using real TDD - then the code got written as a result of tests that already exist. So, the coverage should remain virtually the same without adding a single new test. And that seems like a good plan to me. Plus the whole testing of private methods thing is a discussion anyway.

About TestDrivenDeveloper.com

John E. Boal — Tue, 12 Feb 2008 06:33:29 GMT

Test Driven Developer

Assert.AreEqual(13, stripes);
Assert.AreEqual(50, stars);
seems like enough tests to me...

NOT!

Welcome to Test Driven Developer! This site is dedicated to those who understand the benefits of test-first development, and the advancement of the practice.

It is sometimes hard to convince traditionally trained developers that it is better to write a test first, before writing the code. In Test Driven Development, we always write a test first before writing the code.

Many developers are hard to convince to use this methodology. It is very often difficult for a developer to understand completely what the code needs to do. Yet - they want to proceed to write it anyway? It's kind of a flaw in the thinking process.

Lucky that you are smart enough to know that there's a better way..

TDD is not a Java thing. It's not a .NET thing. It's not language dependent. Anyone can do test-first development, for literally any kind of software development.

Figuring out the test for some code is always the hard part... that's why we like to do it first, and get it out of the way.

TestDrivenDeveloper.com is a John E. Boal web site. Please see also my personal blog and my Agile Development blog