I posted a simple sample of accepance test code in Selenium and WatiN along with a sample web site to test. You can download the zip file here.

I also have posted a Fitnesse fixture in that zip file that illustrates how we can create a simple test fixture for Fitnesse acceptance testing. The Fitnesse tests aren't in the set, but here is the page wiki code that makes use of the fixture:

!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnet\FitServer.exe}
!define PATH_SEPARATOR {;}
!path dotnet\*.dll

Here is an acceptance test using our BusinessObjectTestFixture test class:

!|FitnesseFixture.BusinessObjectTestFixture|
|UserId|Password|Authenticate()|
|administrator|secret0|ADMIN|
|admin|secret0|NONE|
|administrator|secret|NONE|
|user11|secret11|USER|
|user11|secret0|NONE|
|user|secret|NONE|


I also created some STIQ tests, here is the code for the tests and components. Extract this zip file under repository\ProjectRoot folder and it should be able to test the sample site also.

ATDDSTIQ.zip (3.57 KB)

I have been asked recently about tools I prefer to use in my every-day development. Here is a list of tools, and where to get them.

Visual Studio 2008      Development IDE                   Microsoft
TestDriven.net           Test Runner                         TestDriven.net
WinMerge                  Diff / Merge tool                   WinMerge.org integrates with VS and Tortoise too!
nUnit                        Unit Testing Framework          nUnit.org     Also see mbUnit and Gallio
Selenium                   UI test framework                 OpenQA       Also see WatiN
ReSharper                  Integrated toolkit for VS        JetBrains     (OK I don't actually use it but it's good.)
Tortoise SVN              Shell Integration with SVN      Tigris
Ankh SVN                  Visual Studio Plugin for SVN     CollabNet
Subversion (Server)     Version Control System          VisualSVN
Cruise Control.net       CI system                            ThoughtWorks
RhinoMocks                Mock Object System              Ayende
nAnt                         .NET Build Tool                     nAnt
Fitnesse                    Acceptance Test Tool            Fitnesse
STIQ                         Story Test Tool                    Solutions IQ
GIMP                         GNU Image Manipulation Prog. SourceForge
Notepad++                 Smart Text Editor                 SourceForge UK when Visual Studio just won't do...

Not development tools exactly, but extremely handy:
Process Explorer          Smarter Task Manager           SysInternals
FileZilla                      Upload/FTP client                  FileZilla
DivX                          Decoder                              DivX         Because sometimes we need to watch movies...

That's all I can think of at the moment, but am probably missing some things. I'm sure you'll all (please) chime in with what I forgot...

500 lines of code in one method? uh... U R DOING IT WRONG!

Long methods are a code smell. And a loud one at that. Methods should be focused on doing ONE thing. That thing can be simple or complex, but inner complexity should be refactored out into other (small) methods, keeping them each small and focused on a single task. A method with 4 nested levels of for/foreach/while statements is just so wrong in so many ways... Keep it simple. Keep it straightforward. Did I mention keep it simple?

Jeremy Miller posted a nice article a while back about short methods, and Ward Cunningham has another on C2 that Jeremy references.
We definitely need a Visual Studio plug-in to turn the background yellow when a method exceeds 20 lines, and turn red when it goes over 40 lines...

Here is a simple example. Code by intent means writing code by declaring what it does by well named (and usually long-named) methods. This method is only three lines long, but it conveys that there are two steps to be done, the second with the result of the first. This should be clear to anyone reading it. Further, the XML comment summary (missing from this example) should provide the reader with any information needed not conveyed with the title.

public void DoOneThing()
{
   int n = 0;
   n = DoStep1ReturnNumber(n);
   DoStep2WithResultOfStep1(n);
}

private void DoStep1ReturnNumber(int number)
{
   // simple method
}

private void DoStep2WithResultOfStep1(int number)
{
   // more simple stuff
}

Seriously, if there is something out there already that does this, please let me know! If not, I am going to have to write one...

We the practitioners of Agile and users of Beer, hereby proclaim Agile Beer Night.

The Agile Beer Users Group (Seattle Chapter) will be meeting on Thursday Aug 21, for information sharing, networking, agile discussions, a brief topic presentation on Automated Acceptance Testing, oh - and BEER. Come meet us for a discussion and brief presentation on Agile practices each month.

There is good food, good beer, and a good chance to meet some of the folks who do agile development in the Pacific Northwest, get to know their tricks, and share in their success. The Owl has a fine selection of food, drinks, and spirits for those who choose them.

ABN next meeting will be held in Seattle at the Owl and Thistle Irish Pub and Restaurant at 808 Post Avenue in downtown Seattle from 5PM to 8PM on Thursday August 21 2008.

The September ABN meeting will be on the East side in Redmond or Bellevue, but plans are not firm yet - stay tuned...


Please link this post, track it back, and let everyone know!

Next ABN on Thursday August 21 2008 5PM to 8PM at the Owl and Thistle Irish Pub and Restaurant at 808 Post Avenue in downtown Seattle

I learned a valuable lesson today using Selenium. It seems that the DragAndDropToObject() function doesn't scale well when the HTML is being updated in a loop by the mainline code... This method uses two locator strings, one to find the object to drag, and the other to find the object to drop.

It apparently has to examine the entire XML DOM for each locator, each time through. So, when I created a loop that did a drag and drop 250 times on a page, the first few were not too bad only a couple hundred milliseconds. However, as more and more DHTML was added to the page, it took longer and longer to drop the next objects. I measured this time, and when I charted the data, it was a moderately increasing exponential slope. Definitely not even linear...

It seems that the more HTML was added to the DOM, the longer it took both of the locators to locate their objects, even though those objects were static on the page. It walked the DOM each time, for each locator. Thus the increasing non-linear time slope, because the DOM grows with every loop iteration.

By the time it got to the 50th iteration through the loop, the time taken was almost 2 orders of magnitude higher than the initial one. It made testing this particular feature not doable through this framework. And in measuring how long it took the actual javascript to render the new DHTML, it was just around 100ms, when the Selenium locates for the command took upwards of 7000ms for the 50th loop, but was only 200ms at the initial loop.

The Solution
I didn't find a complete solution to the excruciating amount of time the locator code needs to walk the DOM, but I did end up switching from the DragAndDropToObject() method to the DragAndDrop() method. The latter uses one locator string and a relative X,Y coordinate string (NOT a locator). It's just about zero time for it to process the relative coordinate string rather than the locator. So, in my specific case with this test, I was able to cut the time down to 2.5 seconds for the 250th drag and drop. Still a very LONG running test, but it worked (oh yeah and it found a nice juicy bug too...).

OpenQA people - it might behoove you to take a look at optimizing how you interact with a DOM that is growing or being updated by the code under test... A bit of performance gain in this area would really come in handy.

Both Selenium and WatiN have been around for a while, and both are fairly good tools when it comes to UI automated testing. Both tools now drive both IE7 and FF2, and are free to download. I have been using Selenium RC for a while now, and it seems to be fairly robust, although sometimes rather frustrating and somewhat slow. Both have a recorder mechanism, and both emit C# code for me. However the WatiN recorder did blow sky high with an unhandled exception when a pop-up window came up, but the recorder code was written by someone else so I won't hold it against WatiN.

I tried WatiN for the first time today, and decided to go with the V.2 CTP instead of the .NET 1.1 release. It seems that some of the API's have been refactored into more sensible structures (factory patterns, etc.), but somehow the things that would have made the most sense to me in the API were structured much differently that I would have expected. There were some problems with my IE (it kept crashing unless I ran the debug version of W2). However, I blame this on IE, not on W2. FF has a plug-in that needs to be manually installed. It comes with the CTP, but doesn't get installed. Hopefully this will be integrated by release.

Most of the basic features are similar in function but not implementation. I can see writing an interface class and a couple of wrapper classes that implement the interface, one each for WatiN and Selenium, so we can have the best of both worlds.

WatiN Pro's

• No proxy server required. It just runs. Runs both FF and IE, switchable with the parameter to the factory method that creates the instance.
  

WatiN Con's

• Documentation is less than sparse.
• I could not find a way to do a "drag and drop" which is easily accomplished with Selenium.
• Does not handle pop-up windows. This was the show-stopper for me.

Selenium Pro's

• polished code
• well-documented functionality
• lots of examples
• handles lots of weird types of conditions and situations (most of which we will all need at some point.)
  

Selenium Con's

• have to run a java proxy in a separate process and set it up and tear it down
  
• slow in places, due to proxy and not-so-well-written sections of code
• locators aren't always easy to figure out. Tools like Firebug and XPather help identify locator strings.
  

Since it does not handle pop-up windows (or I just couldn't figure it out in about 4h of research), this was the end of the line for it. Selenium does handle them, and has been able to accomplish 100% of the weird scenario tests I have needed so far. Loads of tweaking and frustrating trial and error notwithstanding.

So, until we have a bit closer to release for WatiN, Selenium still wins (for now). However, I definitely plan to write the interface and shim both out so I can use either or both later on. Stay tuned for that posting, that will be the money shot...

In closing, let me just reiterate that every developer should be thinking about UI automated testing for web code. Each user-interactive element on the page NEEDS AN ID. That's every link, every button, every text box, checkbox, radio button, and other user input elements. The ID's need to be simple, and either fixed, or predictable. With these things in place in the production code, it will be testable and easier to validate. Better quality code will make it out to the hands of the users, and all will be right with the world. Amen.

<steps down from soapbox./>

I often talk of Acceptance Test driven development, and how we can turn acceptance criteria into automated tests that illustrate that our software is doing the job the customer wants. However, sometimes it can be difficult to "extract" these criteria. This is a scenario where a customer needs a web control to take some raw data and turn it into a chart, and stream it to the user on the fly.

Conversation (we'll keep track of acceptance criteria in italics)

We need a chart control that will stream an image to the user.
What kind of image? PNG for now. Possibly GIF later.
1. Needs to produce streaming PNG image. Don't bother with GIF yet, probably YAGNI anyway.

Great. Where does the data for the chart come from?
A database. SQL? Yes. Is there a stored procedure?
Yes, the page can talk directly to the DB. OK, good.
I can get specifics and the schema from the DBA? Yes.
2. Data is read from calling a stored proc in a SQL database. Talk to DBA.

Ok, what kind of a chart is it? Line, bar, XY, pie, or something else?
Basically a line chart. OK.
3. Line chart

What are the axes? Time on the horizontal and data on the vertical.
4. Data are plotted chronologically.

If the chart is 640x480 pixels, is that OK?
No, size needs to be able to vary from 320x240 to 1024x768, depending on the caller.
Gotcha - so we'll need to parameterize the size. Yes, that would be good.
5. Height and Width need to be specified.
6. Min is 320x240, max is 1024x768.

If someone requests over the max or under the min, can we substitute these end values
instead of the actual request? Yes.
7. if invalid size, substitute valid size.

What color is the background? Doesn't matter. How about the foreground? Don't care.
Well, would yellow on white work? No, that isn't enough contrast. I see, OK.
8. Contrast between foreground and background.

Back to the line chart, are there markers on the data? Yes, we need tick marks for each point.
Should these ticks be a different color than the line? Yes.
9. Ticks are a different but still high contrast color from the line.

Are there labels on the axes? Yes, time in HH:00 on each hour for horizontal, and unit values on the vertical.
How many unit labels? It varies, has to be speficied. OK, so are the labels on the time axis
vertically or horizontally oriented? Normal - horizontal. Oh, and I only want every third hour labeled.
OK, got it.
10. Horizontal labels on X axis, alternating every third data point.
11. Parameterize the number of data labels on the Y axis.
...

So you see that in just a few seconds conversation, we can come up with 11 acceptance criteria. Each of these criteria can be written up as a test, so that the customer can press a button and see a scenario that illustrates each of these criteria. We don't need a line of code written to have this conversation, record this criteria, or even to automate it. Of course our tests will fail until we write the code...

We can also use tools like Fitnesse to show simple criteria and the outcome to a customer. However in the above example, it might be a bit difficult. We could have a "scenario" test that illustrates some of the conditions for the chart, and have the chart image itself in the results, so the customer can see the actual output in that scenario. I might be tempted instead to write an nUnit suite that I could run that would demonstrate the correct behavior in those scenarios. It depends also on the team's skill set, and the customer's expectations.

If we do this kind of automation around acceptance criteria, we will make it very easy on our customers, and they will feel very confident that they are getting what they want. While it isn't all that easy to demonstrate automated acceptance criteria in all cases, it is worth an attempt at trying to see what we can do, given our project constraints.

We also get the benefit of being able to run our acceptance automation, and in just a few seconds, we can tell if we are really "done" with the story. This kind of assurance gives us a definite advantage in being able to deliver future software down the road, because we can verify that the existing features still work as we add new ones.

In practical terms, however, this kind of testing can be expensive - at first. Many customers are much less concerned about testing the software vs. just getting some code. In these cases, customers' expectations need to be set that there is actual value in the work done to verify that the code is going to work. The more critical the code, the easier it will be to sell. Even if the customer finds no value in the tests, they will still be impressed in the demo when they see a button pressed, and all tests come up green. I think there is really a psychological benefit for the customer, showing them a bunch of green tests. After all, they are giving us a bunch of green too, aren't they? They will see this and naturally be more confident, accepting and more trusting of our work.

So, be sure to collect this criteria carefully. Make sure both the team and the customer are clear that the criteria is the agreed yard-stick on whether the software is working. Ask probing questions, and clarify the answers. Make sure you document the answers. And even if you find you don't have too much time, make sure to automate at least some of your acceptance criteria each sprint - and be sure to show it off in the demo.

"Is it VSTS or Selenium that is going to be our standard tool for testing UI elements and automated acceptance/functional/scenario testing?"

That's like saying "do we want steak or lobster?" The answer is YES. There is no reason we can't do both. Both test tools have their merits, and their drawbacks. In my tool belt, I have both a Phillips head and a flat blade screwdriver. Like my grandfather taught me long ago, use the right tool for the right job.

Selenium actually operates a browser, so it executes JavaScript and gives the test the same experience as the user would get. We can measure response times to render a page, inspect UI elements for formatting, values, and location, and generally things take longer to execute because a browser has to render the page.

VSTS tests use only HTTP level inputs and outputs, so they couldn't care less about JavaScript or having the browser render something. They are typically much faster, because they only test the server and its communication with the client. Conversely, they are much more difficult to hand-code. If we can record a session and alter the code to provide data-driven inputs to the tests, this is an effective way to test that the server is doing the right thing for each request.

Both kinds of tests can be run under both MSTest and NUnit unit test frameworks. There is no reason we have to choose one over the other. When we are talking about UI testing, we are not talking about unit tests anymore, even though we can leverage the same mechanism for executing the tests. Most of the unit test rules do not apply, like testing times, and focused testing. We tend to have scenarios and sometimes fairly elaborate ones that run in these types of test suites.

Let me suggest a course of action for those wishing to have their cake and eat it too. In a single test solution, we can have three separate projects, the Selenium test project, the VSTS test project, and a simple DLL libraray where we can place common helper functions and test setup functions for both types of tests.

If we are careful about how we construct our scenario tests, we can build a complete beginning to end login, accomplish a set of tasks, and log out as a complete user scenario or use case. Then we can refactor the specific things out for re-use, such as log in and log out, and probably each of the specific tasks. This will help us to be able to construct scenarios much easier and quicker, using these building blocks. Then, any "common" non-UI related functionality can be refactored out in to the "helper" classes in the DLL project, making them available for all test types.

Using these techniques, I was able to construct a complete end to end scenario to accomplish a task (and verify the existence (or not) of a specific bug in about 4 hours using Selenium and its IDE. [Note: it shouldn't have taken this long - there were some mainline code change I had to make to be able to get Selenium to be able to locate the controls I needed to operate for the tests.] I was able to then refactor the code in about another hour into these basic blocks. The following scenario and bug confirmation tests took only a few minutes a piece to write for complete scenarios, each of which created their own per-test data using the building blocks and common code.

Granted, some of the per-test data should have been refactored into common test functions that entered data directly into the database rather than going through the UI to create the data, but that sometimes has its problems, and since I assert that everything is going smoothly at each step along the way of the UI process, any BVT-breaking bugs will show up there as those tasks will fail for the setup for those tests as well.

Just a side note, if you plan to use Selenium, the most complicated part will be how to fire up and tear down the Java proxy server for Selenium RC to talk to when it is testing. My suggestion is that in the assembly that houses the Selenium tests, we place Java startup code in a method with a TestFixtureSetup or ClassInitialize attribute [MSTest also has the AssemblyInitialize attribute which can be handy. Here is a reference to a comparison between MSTest and NUnit attributes for those of us who do both.] The Java.exe file will need to be located at run time by searching the file system, then its path can be used to launch it with the jar files that make the proxy work. Make sure to check the running tasks before launching it, just in case someone started it manually. At tear down time, we simply look for the Java.exe task using the process information API, and kill it if it's running (and only if the test system started it).

Using these concepts, we should be able to generate a completely automated suite that we can use for build verification, functional testing, scenario tests, and acceptance tests. We can use both Selenium and MSTest, each where appropriate. All of them can be wrapped up using the unit test framework we choose, and can be run automatically by our build system once the new build has been propagated to the test server. Some folks do this only daily, but in my experience, every couple of hours is a bit better time mark to shoot for.

Comments, suggestions, or other experiences? Let's all please collaborate and enter suggestions below - we'll all benefit!

Do you write installers? A lot of times I hear that all the features or stories are completed, but nobody has given any thought to how actually to ship the software and install it. This is not a trivial story, and usually it is the user's first experience with the software, so if it doesn't go well, your reputation starts out in the hole and that's not easy to overcome. Let's assume that we are more thorough and professional than the average joe, and we create an installer for our application that does three things: Install, Upgrade, and Uninstall.

Most of the time I write installers that are MSI or Setup.exe based, since I sort of get those for free with Visual Studio and .NET. If I have a file or dependency, I simply add it to the package and tell it where on the target machine it is supposed to go. It gets compiled and built (only by DEVENV.EXE by the way - MSBUILD doesn't know how to build setup projects...) after the main assemblies.

The main assemblies all have unit tests of course that run with the build, so everybody is green before anything gets built to install. Installers can be kind of tricky. Don't be tempted to run it on your native development box. If you haven't got lots of proven experience with them, be sure to run the installer for the first few iterations ONLY on a virtual machine that you can restore to its original image with a couple clicks. Let's just say I have seen some interesting recursive registry damage from an installer...

We need tests for the installer, but in this case we are not talking about "unit" tests, but rather functional tests. We need the tests to actually install and uninstall the software.

Testing the Installer Code
First up - test the UNinstall. From my experience, I have seen this path as the best one to start with. Install it manually only after the failing test has been created (and test it only on the VM). You can use either NUnit or MSTest as the framework to run the installer. HINT: to uninstall the software exec this command line:
MSIEXEC /X {guid}
where guid is the identifier for your project. This guid can be found in the .VDPROJ file as the key called "ProductCode." Don't change this value ever, or it will disconnect your next install from your current uninstall, and all kinds of weirdness can happen.

Assert that the registry keys get removed from CurrentVersion as discussed below, and that any registry entries created by the installer are removed.

Gotcha
So it didn't work as planned, and there's an error... an Assert fails, which throws an exception, and your tests end and have left the software installed on the system. The next time the install test runs, it won't work because the software will already be installed. So... make sure you have try-catch logic on everything before you assert anything. The finally block should call a function that uninstalls the product. We know that it will work, because we have tests for it...

Unlike most unit tests where we want to be able to run tests in any order, and have complete independence, functional tests for an installer are much simpler to write if they can be run in the Install, Upgrade, and Uninstall order. However, we don't write them in this order, we start with the uninstall first, then add tests for install, and finally the upgrade cases.

Make sure to catch everything done by the installer in the uninstall test, and do not proceed to write install tests until you are absolutely certain that the uninstall works properly and is adequately tested.

When you move on to the install case, it should be a bit easier, as all you need do is reverse the tests in the uninstall. Make sure that you place this test code before the uninstall test code, so that it is now automated.

Now, voila - you have automated tests for your install code. Finally, your upgrade test can be written to confirm the behavior if the installer is run on a system where the software is already installed. You may want to just fail the install and tell the user to uninstall the previous version, or you may want to upgrade in place. A true upgrade is far nicer to the user, but it is a lot of complexity some times when data migration, customized configurations, or other scenarios are needed. Make sure that any of this functionality is captured in stories, and documented with thorough testing.

What do I test?

• Make sure to open the registry after you install your application, and find the key to make sure you can uninstall it later:
• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{guid}
• This key's value should have the above mentioned MSIEXEC uninstall statement. Check it with an assert to make sure it's correct.
• Make sure each of the assemblies and executables are sent to the correct target folder on the target machine.
• If you support command line options or fancy things for the installer, make sure to test each and every one.
• Write the tests first of course...
• Make sure that config files, documentation, and support files (such as XML or other things) are delivered to the correct locations.
• If any files are dynamically generated or modified by the installer with custom actions after delivery, make sure that the modifications are correct.
  
• Make sure that directories are created on install
• Make sure that the ACL security on the folders and files is correct, if this is something your installer does.
  
• Make sure that directories are deleted on uninstall (except if logs or other artifacts are in them, they won't be deleted)
  
• Make sure these things are written in such a way that they are easy to call from a test, and that they handle errors and take care to leave the system in the same state as before the tests ran (with the software uninstalled).
  

Where do I run the tests?
A word of advice: DO NOT RUN THESE TESTS ON THE BUILD SERVER... it is tempting since they look like "unit" tests, but don't do it. Build servers are not something we want to be messing with installing and uninstalling software. Especially software that gets built every few minutes... The best way to automate the whole process is to have the build server build the app, the installer, and the installer tests and deliver them to an install server, or sandbox server. This can be done through a build step that copies all the executables to a UNC share on the sandbox server after each build. The sandbox server can be set to watch for new files in a folder and automatically begin to run the tests. If you are using CruiseControl.net, this is nicely accomplished by running CC.NET on both the sandbox server and the build server. The build server can "import" the red/green status of the functional tests from the installer to its own dashboard, just as if they were local. It takes a bit of work to get this all figured out, but once it's done it is a nice way to live. I wish I had all my CC.NET code to post, but unfortunately it belongs to someone else and I don't have the rights to post it. If someone has wrapped this concept up in an easier to run turn-key package I would love to hear about it with a comment.

If there are any questions about this, please feel free to comment below and I will try to reply as soon as I am able.