TDD is a great practice to use for code success, as long as the *entire* process is followed...

write tests
write code
refactor
all tests green
we're done right?

Refactoring is the Key to Successful TDD.
We need to look at the changes we made as part of the whole, NOT just in the small. We may have just added a method to a class that makes the class now responsible for more than just the one thing it was before... That's now a code smell (or design smell). When we've refactored the code to make it testable, we're not really done...

We now need to take a step back, and look at the overall design, the archtecture, and how our implementation is satisfying the problems we are solving. We should make sure at this point in time that we can say everything is "well-designed" and "well-implemented."

So many teams that are new to TDD, or that just practice test-first development, tend to forget this CRUCIALLY IMPORTANT step... It's not done, it's not estimated, and often the consequences are simply ignored until it gets to the point that the entire code base is impossible to maintain. This Technical Debt or Engineering Debt goes unaccounted in so many cases. Sometimes, this result ends up being one of the factors that teams use to stop using TDD. When compared to traditional waterfall design models, the output of teams that omit this critical step does not nearly measure up to the designs of waterfall methodology.

There's no free lunch... We still need to have time in our plan for DESIGN. In waterfall, it's all up-front. With TDD and refactoring, we need to do it after we have written the code. We can't just "omit it" - otherwise we're not really "done."

Each time I complete a story, part of my done criteria is to hold up the entire system to a bright light, and look through it for code and design smells. If there are any, it really isn't Done Done Done. I may check in the working code, but I don't close the story as complete until this task is finished.

I estimate in a guess at least for time for sniffing out code and design smells on each story. My guideline for this is at least 10% of the time the rest of the tasks on the story take. Sometimes this clearly isn't enough, as when a story or feature causes significant re-design, or the adaption of an existing design to a new strategy. Use judgement on the estimate, but *AT LEAST* include the task for each story, even if it doesn't get estimated. This method will keep the issue out in the open and remind everyone that it still needs to be done.

If your product owner balks at this additional time (and they most always do) remind them that its a little penalty now, or a HUGE one later. Unless it's a very short-term project on a temporary system, this is almost always a required task, so that maintenance and upkeep - even in the next sprint - aren't blown out of proportion.

At the end of the day, the TEAM is responsible for deciding the best course of action to satifsy the the stakeholders, and even if the PO resists, the development management chain is *still* a stakeholder... As professional developers we really should have the ability to push back on a resistant PO with good quality engineering standards.

Educate your PO's. Make sure people understand how important this critical step is in the story. Place a task on each story for design refactoring. And, last but not least, you can always place a large poster of a nose with a red line through it in the team space and caption it *No Code Smells*" ...

Here is a presentation I wrote on what Unit Testing is all about, and how TDD fits into the ATDD cycle.

There are specific things here on testing the UI code with Selenium and JSUnit, and recommendations on how to do unit testing on your database code.

This presentation is in PDF format, but I can post the PPTX format also if needed.

A Practical Guide to Unit Testing1.pdf (503.29 KB)

Value Stream Mapping [VSM] is a lean concept that helps us improve our processes by focusing on the whole process and making stepwise refinements on smaller parts of the process, to gain improvement for the whole. It is in a sense, "Refactoring" a process.

In an ideal case, a process can be thought of as a workflow, and perhaps we might be able to represent it using the WF Windows Workflow Foundation code delivered in .NET 3.5. It would be nice to show a code example here of how we can represent a process with workflow steps, but I didn't have time to write up an example as of yet.

With VSM, we are trying to improve a process by mapping it into parts, each of which we can study. We can gather metrics on each of the process steps, and metrics on how long things wait between steps. The waiting between steps is usually far more significant time-wise than any of the steps themselves. WF can be instrumental in providing an automated framework for collecting and reporting these metrics.

Indeed there are also other workflow alternatives of course. BizTalk Server 2006 R2 is a fine candidate for workflow representation as well, when adapters are available for systems that interoperate. I have a specific Microsoft focus in my career, but I have used CapeClear as a Java-based workflow system, and there are others in the ESB space which have features that would allow us to collect metrics on an automated representation of our processes.

Automation is key to refactoring, as in any development mechanism. We can't fix what we don't know is broken. With VSM, we focus on the entire process as a whole, so we need to collect measurements on where it is today, and then we can see total improvement on the whole as we make small changes (refactoring the process).

Many enterprises today use Service-Oriented Architecture [SOA] as a design pattern for the construction of enterprise systems. This architecture lends itself well to the use of WF or BizTalk (substiitute your favorite flavor here) workflow components. SOA helps us to map the entire process to blocks of functionality that we can measure and monitor easily, contributing to our VSM overall picture.

My theory is that like with software development, process improvement can also have a test-driven, scientific approach. VSM is a tool that helps us change the right things to improve our overall process, but it typically has measurements in large units - days, even weeks. With the right automation framework contributing to very rapid feedback with the metrics collected on processes, we can write tests that indicate Red or Green for not only each component being analyzed, but for the entire process as well.

Tests don't have to be in the form of an NUnit DLL or test fixture. We can use simple reports actually to give us the feedback we need to make decisions on how we refactor. Think of it as moving test-driven concepts a couple elevations higher in the plan view. From the 500 foot view to the 5000 foot view if you will. It's the exact same concept, just applied at a somewhat higher level of abstraction.

For each of our services we monitor the time taken in the process and the time between processes, and compare them to our initial baseline. We might even be able to get real-time data on a visible dashboard in the best of cirumstances. When we are better than our initial baseline, we are Green, if not, then we're Red. There is no yellow - same result means we haven't improved so it's still red. This concept is brutally simple, but it can be very effective, especially if the Red/Green report is made visible on a web-based report.

Mapping the workflow using WF, BizTalk, or other tools gives us a way to take automated measurements and compare them to our baseline to render a decision on whether our process has improved or not, and whether we are reaching our overall goal of total improvement. Individual portions of the whole can still be red as long as we achieve green for the overall process.

A business process really is a workflow, we just need to map it as such using software and automation. In this way we can apply the fine concepts of test-driven and refactoring to process improvement, once again using automation and measurement as our top key tools for leveraging our own technologies to help us improve them.

Code examples here would be good of course... Stay tuned for future posts with WF code examples of a process flow that has metrics and baseline comparisons for Red/Green.