I learned a valuable lesson today using Selenium. It seems that the DragAndDropToObject() function doesn't scale well when the HTML is being updated in a loop by the mainline code... This method uses two locator strings, one to find the object to drag, and the other to find the object to drop.

It apparently has to examine the entire XML DOM for each locator, each time through. So, when I created a loop that did a drag and drop 250 times on a page, the first few were not too bad only a couple hundred milliseconds. However, as more and more DHTML was added to the page, it took longer and longer to drop the next objects. I measured this time, and when I charted the data, it was a moderately increasing exponential slope. Definitely not even linear...

It seems that the more HTML was added to the DOM, the longer it took both of the locators to locate their objects, even though those objects were static on the page. It walked the DOM each time, for each locator. Thus the increasing non-linear time slope, because the DOM grows with every loop iteration.

By the time it got to the 50th iteration through the loop, the time taken was almost 2 orders of magnitude higher than the initial one. It made testing this particular feature not doable through this framework. And in measuring how long it took the actual javascript to render the new DHTML, it was just around 100ms, when the Selenium locates for the command took upwards of 7000ms for the 50th loop, but was only 200ms at the initial loop.

The Solution
I didn't find a complete solution to the excruciating amount of time the locator code needs to walk the DOM, but I did end up switching from the DragAndDropToObject() method to the DragAndDrop() method. The latter uses one locator string and a relative X,Y coordinate string (NOT a locator). It's just about zero time for it to process the relative coordinate string rather than the locator. So, in my specific case with this test, I was able to cut the time down to 2.5 seconds for the 250th drag and drop. Still a very LONG running test, but it worked (oh yeah and it found a nice juicy bug too...).

OpenQA people - it might behoove you to take a look at optimizing how you interact with a DOM that is growing or being updated by the code under test... A bit of performance gain in this area would really come in handy.

Both Selenium and WatiN have been around for a while, and both are fairly good tools when it comes to UI automated testing. Both tools now drive both IE7 and FF2, and are free to download. I have been using Selenium RC for a while now, and it seems to be fairly robust, although sometimes rather frustrating and somewhat slow. Both have a recorder mechanism, and both emit C# code for me. However the WatiN recorder did blow sky high with an unhandled exception when a pop-up window came up, but the recorder code was written by someone else so I won't hold it against WatiN.

I tried WatiN for the first time today, and decided to go with the V.2 CTP instead of the .NET 1.1 release. It seems that some of the API's have been refactored into more sensible structures (factory patterns, etc.), but somehow the things that would have made the most sense to me in the API were structured much differently that I would have expected. There were some problems with my IE (it kept crashing unless I ran the debug version of W2). However, I blame this on IE, not on W2. FF has a plug-in that needs to be manually installed. It comes with the CTP, but doesn't get installed. Hopefully this will be integrated by release.

Most of the basic features are similar in function but not implementation. I can see writing an interface class and a couple of wrapper classes that implement the interface, one each for WatiN and Selenium, so we can have the best of both worlds.

WatiN Pro's

• No proxy server required. It just runs. Runs both FF and IE, switchable with the parameter to the factory method that creates the instance.
  

WatiN Con's

• Documentation is less than sparse.
• I could not find a way to do a "drag and drop" which is easily accomplished with Selenium.
• Does not handle pop-up windows. This was the show-stopper for me.

Selenium Pro's

• polished code
• well-documented functionality
• lots of examples
• handles lots of weird types of conditions and situations (most of which we will all need at some point.)
  

Selenium Con's

• have to run a java proxy in a separate process and set it up and tear it down
  
• slow in places, due to proxy and not-so-well-written sections of code
• locators aren't always easy to figure out. Tools like Firebug and XPather help identify locator strings.
  

Since it does not handle pop-up windows (or I just couldn't figure it out in about 4h of research), this was the end of the line for it. Selenium does handle them, and has been able to accomplish 100% of the weird scenario tests I have needed so far. Loads of tweaking and frustrating trial and error notwithstanding.

So, until we have a bit closer to release for WatiN, Selenium still wins (for now). However, I definitely plan to write the interface and shim both out so I can use either or both later on. Stay tuned for that posting, that will be the money shot...

In closing, let me just reiterate that every developer should be thinking about UI automated testing for web code. Each user-interactive element on the page NEEDS AN ID. That's every link, every button, every text box, checkbox, radio button, and other user input elements. The ID's need to be simple, and either fixed, or predictable. With these things in place in the production code, it will be testable and easier to validate. Better quality code will make it out to the hands of the users, and all will be right with the world. Amen.

<steps down from soapbox./>

I often talk of Acceptance Test driven development, and how we can turn acceptance criteria into automated tests that illustrate that our software is doing the job the customer wants. However, sometimes it can be difficult to "extract" these criteria. This is a scenario where a customer needs a web control to take some raw data and turn it into a chart, and stream it to the user on the fly.

Conversation (we'll keep track of acceptance criteria in italics)

We need a chart control that will stream an image to the user.
What kind of image? PNG for now. Possibly GIF later.
1. Needs to produce streaming PNG image. Don't bother with GIF yet, probably YAGNI anyway.

Great. Where does the data for the chart come from?
A database. SQL? Yes. Is there a stored procedure?
Yes, the page can talk directly to the DB. OK, good.
I can get specifics and the schema from the DBA? Yes.
2. Data is read from calling a stored proc in a SQL database. Talk to DBA.

Ok, what kind of a chart is it? Line, bar, XY, pie, or something else?
Basically a line chart. OK.
3. Line chart

What are the axes? Time on the horizontal and data on the vertical.
4. Data are plotted chronologically.

If the chart is 640x480 pixels, is that OK?
No, size needs to be able to vary from 320x240 to 1024x768, depending on the caller.
Gotcha - so we'll need to parameterize the size. Yes, that would be good.
5. Height and Width need to be specified.
6. Min is 320x240, max is 1024x768.

If someone requests over the max or under the min, can we substitute these end values
instead of the actual request? Yes.
7. if invalid size, substitute valid size.

What color is the background? Doesn't matter. How about the foreground? Don't care.
Well, would yellow on white work? No, that isn't enough contrast. I see, OK.
8. Contrast between foreground and background.

Back to the line chart, are there markers on the data? Yes, we need tick marks for each point.
Should these ticks be a different color than the line? Yes.
9. Ticks are a different but still high contrast color from the line.

Are there labels on the axes? Yes, time in HH:00 on each hour for horizontal, and unit values on the vertical.
How many unit labels? It varies, has to be speficied. OK, so are the labels on the time axis
vertically or horizontally oriented? Normal - horizontal. Oh, and I only want every third hour labeled.
OK, got it.
10. Horizontal labels on X axis, alternating every third data point.
11. Parameterize the number of data labels on the Y axis.
...

So you see that in just a few seconds conversation, we can come up with 11 acceptance criteria. Each of these criteria can be written up as a test, so that the customer can press a button and see a scenario that illustrates each of these criteria. We don't need a line of code written to have this conversation, record this criteria, or even to automate it. Of course our tests will fail until we write the code...

We can also use tools like Fitnesse to show simple criteria and the outcome to a customer. However in the above example, it might be a bit difficult. We could have a "scenario" test that illustrates some of the conditions for the chart, and have the chart image itself in the results, so the customer can see the actual output in that scenario. I might be tempted instead to write an nUnit suite that I could run that would demonstrate the correct behavior in those scenarios. It depends also on the team's skill set, and the customer's expectations.

If we do this kind of automation around acceptance criteria, we will make it very easy on our customers, and they will feel very confident that they are getting what they want. While it isn't all that easy to demonstrate automated acceptance criteria in all cases, it is worth an attempt at trying to see what we can do, given our project constraints.

We also get the benefit of being able to run our acceptance automation, and in just a few seconds, we can tell if we are really "done" with the story. This kind of assurance gives us a definite advantage in being able to deliver future software down the road, because we can verify that the existing features still work as we add new ones.

In practical terms, however, this kind of testing can be expensive - at first. Many customers are much less concerned about testing the software vs. just getting some code. In these cases, customers' expectations need to be set that there is actual value in the work done to verify that the code is going to work. The more critical the code, the easier it will be to sell. Even if the customer finds no value in the tests, they will still be impressed in the demo when they see a button pressed, and all tests come up green. I think there is really a psychological benefit for the customer, showing them a bunch of green tests. After all, they are giving us a bunch of green too, aren't they? They will see this and naturally be more confident, accepting and more trusting of our work.

So, be sure to collect this criteria carefully. Make sure both the team and the customer are clear that the criteria is the agreed yard-stick on whether the software is working. Ask probing questions, and clarify the answers. Make sure you document the answers. And even if you find you don't have too much time, make sure to automate at least some of your acceptance criteria each sprint - and be sure to show it off in the demo.

"Is it VSTS or Selenium that is going to be our standard tool for testing UI elements and automated acceptance/functional/scenario testing?"

That's like saying "do we want steak or lobster?" The answer is YES. There is no reason we can't do both. Both test tools have their merits, and their drawbacks. In my tool belt, I have both a Phillips head and a flat blade screwdriver. Like my grandfather taught me long ago, use the right tool for the right job.

Selenium actually operates a browser, so it executes JavaScript and gives the test the same experience as the user would get. We can measure response times to render a page, inspect UI elements for formatting, values, and location, and generally things take longer to execute because a browser has to render the page.

VSTS tests use only HTTP level inputs and outputs, so they couldn't care less about JavaScript or having the browser render something. They are typically much faster, because they only test the server and its communication with the client. Conversely, they are much more difficult to hand-code. If we can record a session and alter the code to provide data-driven inputs to the tests, this is an effective way to test that the server is doing the right thing for each request.

Both kinds of tests can be run under both MSTest and NUnit unit test frameworks. There is no reason we have to choose one over the other. When we are talking about UI testing, we are not talking about unit tests anymore, even though we can leverage the same mechanism for executing the tests. Most of the unit test rules do not apply, like testing times, and focused testing. We tend to have scenarios and sometimes fairly elaborate ones that run in these types of test suites.

Let me suggest a course of action for those wishing to have their cake and eat it too. In a single test solution, we can have three separate projects, the Selenium test project, the VSTS test project, and a simple DLL libraray where we can place common helper functions and test setup functions for both types of tests.

If we are careful about how we construct our scenario tests, we can build a complete beginning to end login, accomplish a set of tasks, and log out as a complete user scenario or use case. Then we can refactor the specific things out for re-use, such as log in and log out, and probably each of the specific tasks. This will help us to be able to construct scenarios much easier and quicker, using these building blocks. Then, any "common" non-UI related functionality can be refactored out in to the "helper" classes in the DLL project, making them available for all test types.

Using these techniques, I was able to construct a complete end to end scenario to accomplish a task (and verify the existence (or not) of a specific bug in about 4 hours using Selenium and its IDE. [Note: it shouldn't have taken this long - there were some mainline code change I had to make to be able to get Selenium to be able to locate the controls I needed to operate for the tests.] I was able to then refactor the code in about another hour into these basic blocks. The following scenario and bug confirmation tests took only a few minutes a piece to write for complete scenarios, each of which created their own per-test data using the building blocks and common code.

Granted, some of the per-test data should have been refactored into common test functions that entered data directly into the database rather than going through the UI to create the data, but that sometimes has its problems, and since I assert that everything is going smoothly at each step along the way of the UI process, any BVT-breaking bugs will show up there as those tasks will fail for the setup for those tests as well.

Just a side note, if you plan to use Selenium, the most complicated part will be how to fire up and tear down the Java proxy server for Selenium RC to talk to when it is testing. My suggestion is that in the assembly that houses the Selenium tests, we place Java startup code in a method with a TestFixtureSetup or ClassInitialize attribute [MSTest also has the AssemblyInitialize attribute which can be handy. Here is a reference to a comparison between MSTest and NUnit attributes for those of us who do both.] The Java.exe file will need to be located at run time by searching the file system, then its path can be used to launch it with the jar files that make the proxy work. Make sure to check the running tasks before launching it, just in case someone started it manually. At tear down time, we simply look for the Java.exe task using the process information API, and kill it if it's running (and only if the test system started it).

Using these concepts, we should be able to generate a completely automated suite that we can use for build verification, functional testing, scenario tests, and acceptance tests. We can use both Selenium and MSTest, each where appropriate. All of them can be wrapped up using the unit test framework we choose, and can be run automatically by our build system once the new build has been propagated to the test server. Some folks do this only daily, but in my experience, every couple of hours is a bit better time mark to shoot for.

Comments, suggestions, or other experiences? Let's all please collaborate and enter suggestions below - we'll all benefit!

Do you write installers? A lot of times I hear that all the features or stories are completed, but nobody has given any thought to how actually to ship the software and install it. This is not a trivial story, and usually it is the user's first experience with the software, so if it doesn't go well, your reputation starts out in the hole and that's not easy to overcome. Let's assume that we are more thorough and professional than the average joe, and we create an installer for our application that does three things: Install, Upgrade, and Uninstall.

Most of the time I write installers that are MSI or Setup.exe based, since I sort of get those for free with Visual Studio and .NET. If I have a file or dependency, I simply add it to the package and tell it where on the target machine it is supposed to go. It gets compiled and built (only by DEVENV.EXE by the way - MSBUILD doesn't know how to build setup projects...) after the main assemblies.

The main assemblies all have unit tests of course that run with the build, so everybody is green before anything gets built to install. Installers can be kind of tricky. Don't be tempted to run it on your native development box. If you haven't got lots of proven experience with them, be sure to run the installer for the first few iterations ONLY on a virtual machine that you can restore to its original image with a couple clicks. Let's just say I have seen some interesting recursive registry damage from an installer...

We need tests for the installer, but in this case we are not talking about "unit" tests, but rather functional tests. We need the tests to actually install and uninstall the software.

Testing the Installer Code
First up - test the UNinstall. From my experience, I have seen this path as the best one to start with. Install it manually only after the failing test has been created (and test it only on the VM). You can use either NUnit or MSTest as the framework to run the installer. HINT: to uninstall the software exec this command line:
MSIEXEC /X {guid}
where guid is the identifier for your project. This guid can be found in the .VDPROJ file as the key called "ProductCode." Don't change this value ever, or it will disconnect your next install from your current uninstall, and all kinds of weirdness can happen.

Assert that the registry keys get removed from CurrentVersion as discussed below, and that any registry entries created by the installer are removed.

Gotcha
So it didn't work as planned, and there's an error... an Assert fails, which throws an exception, and your tests end and have left the software installed on the system. The next time the install test runs, it won't work because the software will already be installed. So... make sure you have try-catch logic on everything before you assert anything. The finally block should call a function that uninstalls the product. We know that it will work, because we have tests for it...

Unlike most unit tests where we want to be able to run tests in any order, and have complete independence, functional tests for an installer are much simpler to write if they can be run in the Install, Upgrade, and Uninstall order. However, we don't write them in this order, we start with the uninstall first, then add tests for install, and finally the upgrade cases.

Make sure to catch everything done by the installer in the uninstall test, and do not proceed to write install tests until you are absolutely certain that the uninstall works properly and is adequately tested.

When you move on to the install case, it should be a bit easier, as all you need do is reverse the tests in the uninstall. Make sure that you place this test code before the uninstall test code, so that it is now automated.

Now, voila - you have automated tests for your install code. Finally, your upgrade test can be written to confirm the behavior if the installer is run on a system where the software is already installed. You may want to just fail the install and tell the user to uninstall the previous version, or you may want to upgrade in place. A true upgrade is far nicer to the user, but it is a lot of complexity some times when data migration, customized configurations, or other scenarios are needed. Make sure that any of this functionality is captured in stories, and documented with thorough testing.

What do I test?

• Make sure to open the registry after you install your application, and find the key to make sure you can uninstall it later:
• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{guid}
• This key's value should have the above mentioned MSIEXEC uninstall statement. Check it with an assert to make sure it's correct.
• Make sure each of the assemblies and executables are sent to the correct target folder on the target machine.
• If you support command line options or fancy things for the installer, make sure to test each and every one.
• Write the tests first of course...
• Make sure that config files, documentation, and support files (such as XML or other things) are delivered to the correct locations.
• If any files are dynamically generated or modified by the installer with custom actions after delivery, make sure that the modifications are correct.
  
• Make sure that directories are created on install
• Make sure that the ACL security on the folders and files is correct, if this is something your installer does.
  
• Make sure that directories are deleted on uninstall (except if logs or other artifacts are in them, they won't be deleted)
  
• Make sure these things are written in such a way that they are easy to call from a test, and that they handle errors and take care to leave the system in the same state as before the tests ran (with the software uninstalled).
  

Where do I run the tests?
A word of advice: DO NOT RUN THESE TESTS ON THE BUILD SERVER... it is tempting since they look like "unit" tests, but don't do it. Build servers are not something we want to be messing with installing and uninstalling software. Especially software that gets built every few minutes... The best way to automate the whole process is to have the build server build the app, the installer, and the installer tests and deliver them to an install server, or sandbox server. This can be done through a build step that copies all the executables to a UNC share on the sandbox server after each build. The sandbox server can be set to watch for new files in a folder and automatically begin to run the tests. If you are using CruiseControl.net, this is nicely accomplished by running CC.NET on both the sandbox server and the build server. The build server can "import" the red/green status of the functional tests from the installer to its own dashboard, just as if they were local. It takes a bit of work to get this all figured out, but once it's done it is a nice way to live. I wish I had all my CC.NET code to post, but unfortunately it belongs to someone else and I don't have the rights to post it. If someone has wrapped this concept up in an easier to run turn-key package I would love to hear about it with a comment.

If there are any questions about this, please feel free to comment below and I will try to reply as soon as I am able.

Why to Do It
When running lots of unit tests, there are distinct advantages in using all of those cores and cpu's that most of our modern computers have. We can parallelize testing so that we gain two distinct advantages. First, the testing will complete much faster. If you have 4 cpu's or 4 cores, we should see at least a halving of test time, if not more. Secondly, we prove our code is able to run in the closer-to-real-world scenario that other things might be going on while our code under test is executing. Plus its just cool to do multi-threading.

The by-product or side effect benefit we get also in multi-threading our tests is that we need to put more careful thought into the design and construction of test cases, and by that virtue, our software under test. The ability to multi-thread tests is I think one of the highest bars we have in demonstrating conclusively that our software is robust.

Advice on How to Do It
Separate tests into separate test assemblies that can be launched by the build server in parallel. There is no real reason to sequence tests, unless they sometimes interfere with each other. If they do interfere with each other, the tests are poorly designed and should be re-written as independent.

See the Pragmatic Unit Testing book or see this article for good advice on how to write good tests that you can parallelize.

Tests must be thread-safe. We probably should be able to run the same test at the same time without collisions.
Tests must be TRULY independent. This usually means that tests need to generate data that is keyed specific to that test run.
Tests must not have side-effects. They must leave the test system in the state they found it.

The new version of NUnit (2.5) is rumored to have functionality to be able to run tests in parallel threads. The currently released version has only one thread to run all the tests. In some preliminary research I have done, when manually coding up multiple threads to run each test in parallel from a single "unit test" method, it has some contradictory behavior. NUnit reports that all tests pass, but the GUI bar turns red.

More to follow on multi-threading test with MS Test...

I was explaining to someone about different types of testing I do with my code, and how mock objects can be used in unit tests. I used the example of the paint truck at the airport that spray-paints lines on the tarmac. Its job is to put down wide lines, narrow lines, in yellow, and white, and of a certain length, at specific places on the runways and taxiways. The truck is driven by a person, and the paint spray system is operated by driver. It's not a perfect example, but it may get the point across.

When we model this with truck software and try to test it, most developers seem to tend to go right for testing the whole solution end to end. That means trying to figure out how to fire up the engine, position the truck at certain coordinates, be at a certain speed, and then test the spray gun and see if it paints the correct stripes. It's a hard problem to solve, to try to set up with all these things and get it just right - just to see if the painter works properly. It can be a daunting task to try to test software this way, yet most developers who do "unit testing" are trying to do just this kind of thing. My take is that this is not really "unit testing" but actually "functional" testing. While it is definitely good to have functional tests, I prefer my development team to focus on more narrow areas such as individual classes. If these are thoughtfully designed and tested individually, they should work well when combined. This is also where integration and scenario testing come in. These types of tests are beyond the scope of unit tests.

Let me give an example that explains how we might be able to go about testing this a bit more carefully. Let's take the truck as a single "system" and the paint sprayer as a separate "system." These systems are really not very coupled - the sprayer runs on the truck's electric system, but that's about it. It makes sense to decouple them in trying to test them.

If we separate out the sprayer as a separate system, the only inputs it needs from the truck are the electrical connection and the trigger input. We can "mock" out these connections by supplying a new power source and trigger input connector that has the same "interface" (that is - implements the same interface class in code). Now we can test the system without needing to have the actual truck around. This is the concept of mock objects at its best in unit testing.

We can separate even the power mock object from the trigger mock object. We can control how these mock objects behave precisely, and keep them in specific states so that the only variables under test are in the sprayer system. We can then test what happens to the sprayer if we make the power go away, what happens if we trigger both paint guns at the same time, and making sure we can turn on and off each gun. These make great unit tests and document how our code is supposed to work. They should run very fast, and not require any diesel fuel...

All of the ability we have to be able to mock things like this, depends on programming to interfaces, which is a long-time engineering best practice to avoid decoupling. If we design our software to implement interfaces, we can then easily substitute dependency systems at test time with our own controlled version, and focus the testing on a more narrow area, while making the tests run faster at the same time.

I have been using TestDriven.net as a tool now for a few weeks, and I have to say that it is a fine addition to my tools collection. It gives me the flexibility I need to run tests in any manner I choose, simply by right-clicking. It's a Visual Studio add-in for 2005, and 2008, and it works seamlessly with a variety of test frameworks (NUnit, MbUnit, and MSTest being the most notable).

Speaking of NUnit, another excellent tool that I find invaluable to help me test my code. NUnit now has RowTest extensions (like MbUnit) in the production release (2.4.7 and if you aren't using this version, you should upgrade), and Parameterized Tests coming soon, and they look really cool. I find that I can replace a lot of repetition in my tests with this row based system. It enables me to write more tests faster, and that's always good. I am hoping that the new release goes into production mode very soon.

UI testing with Selenium is another favorite topic, I will write up another article later with examples of how to do real automated user interface testing using NUnit and Selenium. However, feel free to start playing with it now... its a great tool also, and plays nicely with the other tools in the toolbox.

Use these tools in good health, to help get your code into production as bug-free as possible by testing the heck out of it!

P.S.
Please make your unit tests into separate assemblies... no one-massive-monolithic-test assemblies please... and the tests should run FAST. If you mock out all the database, network, web service, etc. calls, you should be able to get all of your tests in an assembly run in under 5 seconds. Aim for a target of 100 tests per second, and you will be a much happier camper.

If you are a real stickler for this (and I recommend it...) add this code to your test assemblies:

private static DateTime assemblyStart;

 [ClassInitialize]
 public static void MyClassInitialize(TestContext testContext)
 {
     assemblyStart = DateTime.Now;
 }

 [TestCleanup]
 public static void TestCleanup()
 {
     Assert.IsTrue(
         assemblyStart.AddMilliseconds(5000) > DateTime.Now,
         "Tests took too long to execute.");
 }

Enjoy!

OK people, for some reason this little gem has been overlooked by an awful lot of developers. Every time I see someone write an if statement that checks to see if a string is null, then if it is empty, I have to cringe. This static method on the String class is a very handy mechanism to do something that is extremely common in code I see.

instead of this

if (str == null || str == String.Empty)
{
    doSomething();
}

use this:


if (String.IsNullOrEmpty(str))
{
    doSomething();
}

It is so much cleaner, and faster too I think.

Just remember that this should probably be the first thing we write in the main line code after the second unit test is failing, for any method that takes a string parameter... [you know, the test where we send null in as the string parameter... then we have to write the if statement in the main code]. Which then follows closely with the third unit test in which we pass String.Empty as the value. But then again you already knew that.

We all know what design patterns are in software design and development. These kinds of patterns also are recognized in unit and other kinds of tests as well.

While it is not necessarily a new idea, it is a good idea. Here are some links I have found on the subject. Further research should yield a whitepaper soon, if I ever get time to write it.

Brian Marick's testing.com: http://www.testing.com/test-patterns/patterns/

A great example at CodeProject: http://www.codeproject.com/KB/architecture/autp5.aspx

RBSC: http://www.rbsc.com/pages/TestPatternList.htm

TypeMock Unit test patterns for .net: http://www.typemock.com/Docs/TestPatterns.html

Book: xUnit Test Patterns: Refactoring Test Code

Enjoy