"Is it VSTS or Selenium that is going to be our standard tool for testing UI elements and automated acceptance/functional/scenario testing?"

That's like saying "do we want steak or lobster?" The answer is YES. There is no reason we can't do both. Both test tools have their merits, and their drawbacks. In my tool belt, I have both a Phillips head and a flat blade screwdriver. Like my grandfather taught me long ago, use the right tool for the right job.

Selenium actually operates a browser, so it executes JavaScript and gives the test the same experience as the user would get. We can measure response times to render a page, inspect UI elements for formatting, values, and location, and generally things take longer to execute because a browser has to render the page.

VSTS tests use only HTTP level inputs and outputs, so they couldn't care less about JavaScript or having the browser render something. They are typically much faster, because they only test the server and its communication with the client. Conversely, they are much more difficult to hand-code. If we can record a session and alter the code to provide data-driven inputs to the tests, this is an effective way to test that the server is doing the right thing for each request.

Both kinds of tests can be run under both MSTest and NUnit unit test frameworks. There is no reason we have to choose one over the other. When we are talking about UI testing, we are not talking about unit tests anymore, even though we can leverage the same mechanism for executing the tests. Most of the unit test rules do not apply, like testing times, and focused testing. We tend to have scenarios and sometimes fairly elaborate ones that run in these types of test suites.

Let me suggest a course of action for those wishing to have their cake and eat it too. In a single test solution, we can have three separate projects, the Selenium test project, the VSTS test project, and a simple DLL libraray where we can place common helper functions and test setup functions for both types of tests.

If we are careful about how we construct our scenario tests, we can build a complete beginning to end login, accomplish a set of tasks, and log out as a complete user scenario or use case. Then we can refactor the specific things out for re-use, such as log in and log out, and probably each of the specific tasks. This will help us to be able to construct scenarios much easier and quicker, using these building blocks. Then, any "common" non-UI related functionality can be refactored out in to the "helper" classes in the DLL project, making them available for all test types.

Using these techniques, I was able to construct a complete end to end scenario to accomplish a task (and verify the existence (or not) of a specific bug in about 4 hours using Selenium and its IDE. [Note: it shouldn't have taken this long - there were some mainline code change I had to make to be able to get Selenium to be able to locate the controls I needed to operate for the tests.] I was able to then refactor the code in about another hour into these basic blocks. The following scenario and bug confirmation tests took only a few minutes a piece to write for complete scenarios, each of which created their own per-test data using the building blocks and common code.

Granted, some of the per-test data should have been refactored into common test functions that entered data directly into the database rather than going through the UI to create the data, but that sometimes has its problems, and since I assert that everything is going smoothly at each step along the way of the UI process, any BVT-breaking bugs will show up there as those tasks will fail for the setup for those tests as well.

Just a side note, if you plan to use Selenium, the most complicated part will be how to fire up and tear down the Java proxy server for Selenium RC to talk to when it is testing. My suggestion is that in the assembly that houses the Selenium tests, we place Java startup code in a method with a TestFixtureSetup or ClassInitialize attribute [MSTest also has the AssemblyInitialize attribute which can be handy. Here is a reference to a comparison between MSTest and NUnit attributes for those of us who do both.] The Java.exe file will need to be located at run time by searching the file system, then its path can be used to launch it with the jar files that make the proxy work. Make sure to check the running tasks before launching it, just in case someone started it manually. At tear down time, we simply look for the Java.exe task using the process information API, and kill it if it's running (and only if the test system started it).

Using these concepts, we should be able to generate a completely automated suite that we can use for build verification, functional testing, scenario tests, and acceptance tests. We can use both Selenium and MSTest, each where appropriate. All of them can be wrapped up using the unit test framework we choose, and can be run automatically by our build system once the new build has been propagated to the test server. Some folks do this only daily, but in my experience, every couple of hours is a bit better time mark to shoot for.

Comments, suggestions, or other experiences? Let's all please collaborate and enter suggestions below - we'll all benefit!