Value Stream Mapping [VSM] is a lean concept that helps us improve our processes by focusing on the whole process and making stepwise refinements on smaller parts of the process, to gain improvement for the whole. It is in a sense, "Refactoring" a process.

In an ideal case, a process can be thought of as a workflow, and perhaps we might be able to represent it using the WF Windows Workflow Foundation code delivered in .NET 3.5. It would be nice to show a code example here of how we can represent a process with workflow steps, but I didn't have time to write up an example as of yet.

With VSM, we are trying to improve a process by mapping it into parts, each of which we can study. We can gather metrics on each of the process steps, and metrics on how long things wait between steps. The waiting between steps is usually far more significant time-wise than any of the steps themselves. WF can be instrumental in providing an automated framework for collecting and reporting these metrics.

Indeed there are also other workflow alternatives of course. BizTalk Server 2006 R2 is a fine candidate for workflow representation as well, when adapters are available for systems that interoperate. I have a specific Microsoft focus in my career, but I have used CapeClear as a Java-based workflow system, and there are others in the ESB space which have features that would allow us to collect metrics on an automated representation of our processes.

Automation is key to refactoring, as in any development mechanism. We can't fix what we don't know is broken. With VSM, we focus on the entire process as a whole, so we need to collect measurements on where it is today, and then we can see total improvement on the whole as we make small changes (refactoring the process).

Many enterprises today use Service-Oriented Architecture [SOA] as a design pattern for the construction of enterprise systems. This architecture lends itself well to the use of WF or BizTalk (substiitute your favorite flavor here) workflow components. SOA helps us to map the entire process to blocks of functionality that we can measure and monitor easily, contributing to our VSM overall picture.

My theory is that like with software development, process improvement can also have a test-driven, scientific approach. VSM is a tool that helps us change the right things to improve our overall process, but it typically has measurements in large units - days, even weeks. With the right automation framework contributing to very rapid feedback with the metrics collected on processes, we can write tests that indicate Red or Green for not only each component being analyzed, but for the entire process as well.

Tests don't have to be in the form of an NUnit DLL or test fixture. We can use simple reports actually to give us the feedback we need to make decisions on how we refactor. Think of it as moving test-driven concepts a couple elevations higher in the plan view. From the 500 foot view to the 5000 foot view if you will. It's the exact same concept, just applied at a somewhat higher level of abstraction.

For each of our services we monitor the time taken in the process and the time between processes, and compare them to our initial baseline. We might even be able to get real-time data on a visible dashboard in the best of cirumstances. When we are better than our initial baseline, we are Green, if not, then we're Red. There is no yellow - same result means we haven't improved so it's still red. This concept is brutally simple, but it can be very effective, especially if the Red/Green report is made visible on a web-based report.

Mapping the workflow using WF, BizTalk, or other tools gives us a way to take automated measurements and compare them to our baseline to render a decision on whether our process has improved or not, and whether we are reaching our overall goal of total improvement. Individual portions of the whole can still be red as long as we achieve green for the overall process.

A business process really is a workflow, we just need to map it as such using software and automation. In this way we can apply the fine concepts of test-driven and refactoring to process improvement, once again using automation and measurement as our top key tools for leveraging our own technologies to help us improve them.

Code examples here would be good of course... Stay tuned for future posts with WF code examples of a process flow that has metrics and baseline comparisons for Red/Green.

I posted a simple sample of accepance test code in Selenium and WatiN along with a sample web site to test. You can download the zip file here.

I also have posted a Fitnesse fixture in that zip file that illustrates how we can create a simple test fixture for Fitnesse acceptance testing. The Fitnesse tests aren't in the set, but here is the page wiki code that makes use of the fixture:

!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnet\FitServer.exe}
!define PATH_SEPARATOR {;}
!path dotnet\*.dll

Here is an acceptance test using our BusinessObjectTestFixture test class:

!|FitnesseFixture.BusinessObjectTestFixture|
|UserId|Password|Authenticate()|
|administrator|secret0|ADMIN|
|admin|secret0|NONE|
|administrator|secret|NONE|
|user11|secret11|USER|
|user11|secret0|NONE|
|user|secret|NONE|


I also created some STIQ tests, here is the code for the tests and components. Extract this zip file under repository\ProjectRoot folder and it should be able to test the sample site also.

ATDDSTIQ.zip (3.57 KB)

I have been asked recently about tools I prefer to use in my every-day development. Here is a list of tools, and where to get them.

Visual Studio 2008      Development IDE                   Microsoft
TestDriven.net           Test Runner                         TestDriven.net
WinMerge                  Diff / Merge tool                   WinMerge.org integrates with VS and Tortoise too!
nUnit                        Unit Testing Framework          nUnit.org     Also see mbUnit and Gallio
Selenium                   UI test framework                 OpenQA       Also see WatiN
ReSharper                  Integrated toolkit for VS        JetBrains     (OK I don't actually use it but it's good.)
Tortoise SVN              Shell Integration with SVN      Tigris
Ankh SVN                  Visual Studio Plugin for SVN     CollabNet
Subversion (Server)     Version Control System          VisualSVN
Cruise Control.net       CI system                            ThoughtWorks
RhinoMocks                Mock Object System              Ayende
nAnt                         .NET Build Tool                     nAnt
Fitnesse                    Acceptance Test Tool            Fitnesse
STIQ                         Story Test Tool                    Solutions IQ
GIMP                         GNU Image Manipulation Prog. SourceForge
Notepad++                 Smart Text Editor                 SourceForge UK when Visual Studio just won't do...

Not development tools exactly, but extremely handy:
Process Explorer          Smarter Task Manager           SysInternals
FileZilla                      Upload/FTP client                  FileZilla
DivX                          Decoder                              DivX         Because sometimes we need to watch movies...

That's all I can think of at the moment, but am probably missing some things. I'm sure you'll all (please) chime in with what I forgot...

We the practitioners of Agile and users of Beer, hereby proclaim Agile Beer Night.

The Agile Beer Users Group (Seattle Chapter) will be meeting on Thursday Aug 21, for information sharing, networking, agile discussions, a brief topic presentation on Automated Acceptance Testing, oh - and BEER. Come meet us for a discussion and brief presentation on Agile practices each month.

There is good food, good beer, and a good chance to meet some of the folks who do agile development in the Pacific Northwest, get to know their tricks, and share in their success. The Owl has a fine selection of food, drinks, and spirits for those who choose them.

ABN next meeting will be held in Seattle at the Owl and Thistle Irish Pub and Restaurant at 808 Post Avenue in downtown Seattle from 5PM to 8PM on Thursday August 21 2008.

The September ABN meeting will be on the East side in Redmond or Bellevue, but plans are not firm yet - stay tuned...


Please link this post, track it back, and let everyone know!

Next ABN on Thursday August 21 2008 5PM to 8PM at the Owl and Thistle Irish Pub and Restaurant at 808 Post Avenue in downtown Seattle

I learned a valuable lesson today using Selenium. It seems that the DragAndDropToObject() function doesn't scale well when the HTML is being updated in a loop by the mainline code... This method uses two locator strings, one to find the object to drag, and the other to find the object to drop.

It apparently has to examine the entire XML DOM for each locator, each time through. So, when I created a loop that did a drag and drop 250 times on a page, the first few were not too bad only a couple hundred milliseconds. However, as more and more DHTML was added to the page, it took longer and longer to drop the next objects. I measured this time, and when I charted the data, it was a moderately increasing exponential slope. Definitely not even linear...

It seems that the more HTML was added to the DOM, the longer it took both of the locators to locate their objects, even though those objects were static on the page. It walked the DOM each time, for each locator. Thus the increasing non-linear time slope, because the DOM grows with every loop iteration.

By the time it got to the 50th iteration through the loop, the time taken was almost 2 orders of magnitude higher than the initial one. It made testing this particular feature not doable through this framework. And in measuring how long it took the actual javascript to render the new DHTML, it was just around 100ms, when the Selenium locates for the command took upwards of 7000ms for the 50th loop, but was only 200ms at the initial loop.

The Solution
I didn't find a complete solution to the excruciating amount of time the locator code needs to walk the DOM, but I did end up switching from the DragAndDropToObject() method to the DragAndDrop() method. The latter uses one locator string and a relative X,Y coordinate string (NOT a locator). It's just about zero time for it to process the relative coordinate string rather than the locator. So, in my specific case with this test, I was able to cut the time down to 2.5 seconds for the 250th drag and drop. Still a very LONG running test, but it worked (oh yeah and it found a nice juicy bug too...).

OpenQA people - it might behoove you to take a look at optimizing how you interact with a DOM that is growing or being updated by the code under test... A bit of performance gain in this area would really come in handy.

Both Selenium and WatiN have been around for a while, and both are fairly good tools when it comes to UI automated testing. Both tools now drive both IE7 and FF2, and are free to download. I have been using Selenium RC for a while now, and it seems to be fairly robust, although sometimes rather frustrating and somewhat slow. Both have a recorder mechanism, and both emit C# code for me. However the WatiN recorder did blow sky high with an unhandled exception when a pop-up window came up, but the recorder code was written by someone else so I won't hold it against WatiN.

I tried WatiN for the first time today, and decided to go with the V.2 CTP instead of the .NET 1.1 release. It seems that some of the API's have been refactored into more sensible structures (factory patterns, etc.), but somehow the things that would have made the most sense to me in the API were structured much differently that I would have expected. There were some problems with my IE (it kept crashing unless I ran the debug version of W2). However, I blame this on IE, not on W2. FF has a plug-in that needs to be manually installed. It comes with the CTP, but doesn't get installed. Hopefully this will be integrated by release.

Most of the basic features are similar in function but not implementation. I can see writing an interface class and a couple of wrapper classes that implement the interface, one each for WatiN and Selenium, so we can have the best of both worlds.

WatiN Pro's

• No proxy server required. It just runs. Runs both FF and IE, switchable with the parameter to the factory method that creates the instance.
  

WatiN Con's

• Documentation is less than sparse.
• I could not find a way to do a "drag and drop" which is easily accomplished with Selenium.
• Does not handle pop-up windows. This was the show-stopper for me.

Selenium Pro's

• polished code
• well-documented functionality
• lots of examples
• handles lots of weird types of conditions and situations (most of which we will all need at some point.)
  

Selenium Con's

• have to run a java proxy in a separate process and set it up and tear it down
  
• slow in places, due to proxy and not-so-well-written sections of code
• locators aren't always easy to figure out. Tools like Firebug and XPather help identify locator strings.
  

Since it does not handle pop-up windows (or I just couldn't figure it out in about 4h of research), this was the end of the line for it. Selenium does handle them, and has been able to accomplish 100% of the weird scenario tests I have needed so far. Loads of tweaking and frustrating trial and error notwithstanding.

So, until we have a bit closer to release for WatiN, Selenium still wins (for now). However, I definitely plan to write the interface and shim both out so I can use either or both later on. Stay tuned for that posting, that will be the money shot...

In closing, let me just reiterate that every developer should be thinking about UI automated testing for web code. Each user-interactive element on the page NEEDS AN ID. That's every link, every button, every text box, checkbox, radio button, and other user input elements. The ID's need to be simple, and either fixed, or predictable. With these things in place in the production code, it will be testable and easier to validate. Better quality code will make it out to the hands of the users, and all will be right with the world. Amen.

<steps down from soapbox./>

"Is it VSTS or Selenium that is going to be our standard tool for testing UI elements and automated acceptance/functional/scenario testing?"

That's like saying "do we want steak or lobster?" The answer is YES. There is no reason we can't do both. Both test tools have their merits, and their drawbacks. In my tool belt, I have both a Phillips head and a flat blade screwdriver. Like my grandfather taught me long ago, use the right tool for the right job.

Selenium actually operates a browser, so it executes JavaScript and gives the test the same experience as the user would get. We can measure response times to render a page, inspect UI elements for formatting, values, and location, and generally things take longer to execute because a browser has to render the page.

VSTS tests use only HTTP level inputs and outputs, so they couldn't care less about JavaScript or having the browser render something. They are typically much faster, because they only test the server and its communication with the client. Conversely, they are much more difficult to hand-code. If we can record a session and alter the code to provide data-driven inputs to the tests, this is an effective way to test that the server is doing the right thing for each request.

Both kinds of tests can be run under both MSTest and NUnit unit test frameworks. There is no reason we have to choose one over the other. When we are talking about UI testing, we are not talking about unit tests anymore, even though we can leverage the same mechanism for executing the tests. Most of the unit test rules do not apply, like testing times, and focused testing. We tend to have scenarios and sometimes fairly elaborate ones that run in these types of test suites.

Let me suggest a course of action for those wishing to have their cake and eat it too. In a single test solution, we can have three separate projects, the Selenium test project, the VSTS test project, and a simple DLL libraray where we can place common helper functions and test setup functions for both types of tests.

If we are careful about how we construct our scenario tests, we can build a complete beginning to end login, accomplish a set of tasks, and log out as a complete user scenario or use case. Then we can refactor the specific things out for re-use, such as log in and log out, and probably each of the specific tasks. This will help us to be able to construct scenarios much easier and quicker, using these building blocks. Then, any "common" non-UI related functionality can be refactored out in to the "helper" classes in the DLL project, making them available for all test types.

Using these techniques, I was able to construct a complete end to end scenario to accomplish a task (and verify the existence (or not) of a specific bug in about 4 hours using Selenium and its IDE. [Note: it shouldn't have taken this long - there were some mainline code change I had to make to be able to get Selenium to be able to locate the controls I needed to operate for the tests.] I was able to then refactor the code in about another hour into these basic blocks. The following scenario and bug confirmation tests took only a few minutes a piece to write for complete scenarios, each of which created their own per-test data using the building blocks and common code.

Granted, some of the per-test data should have been refactored into common test functions that entered data directly into the database rather than going through the UI to create the data, but that sometimes has its problems, and since I assert that everything is going smoothly at each step along the way of the UI process, any BVT-breaking bugs will show up there as those tasks will fail for the setup for those tests as well.

Just a side note, if you plan to use Selenium, the most complicated part will be how to fire up and tear down the Java proxy server for Selenium RC to talk to when it is testing. My suggestion is that in the assembly that houses the Selenium tests, we place Java startup code in a method with a TestFixtureSetup or ClassInitialize attribute [MSTest also has the AssemblyInitialize attribute which can be handy. Here is a reference to a comparison between MSTest and NUnit attributes for those of us who do both.] The Java.exe file will need to be located at run time by searching the file system, then its path can be used to launch it with the jar files that make the proxy work. Make sure to check the running tasks before launching it, just in case someone started it manually. At tear down time, we simply look for the Java.exe task using the process information API, and kill it if it's running (and only if the test system started it).

Using these concepts, we should be able to generate a completely automated suite that we can use for build verification, functional testing, scenario tests, and acceptance tests. We can use both Selenium and MSTest, each where appropriate. All of them can be wrapped up using the unit test framework we choose, and can be run automatically by our build system once the new build has been propagated to the test server. Some folks do this only daily, but in my experience, every couple of hours is a bit better time mark to shoot for.

Comments, suggestions, or other experiences? Let's all please collaborate and enter suggestions below - we'll all benefit!