Test Driven Developer - ATDD

Unit Testing and Test Driven Development - a Practical Guide

John E. Boal — Sat, 15 Nov 2008 21:08:59 GMT

Here is a presentation I wrote on what Unit Testing is all about, and how TDD fits into the ATDD cycle.

There are specific things here on testing the UI code with Selenium and JSUnit, and recommendations on how to do unit testing on your database code.

This presentation is in PDF format, but I can post the PPTX format also if needed.

A Practical Guide to Unit Testing1.pdf (503.29 KB)

Sample Code for Selenium and WatiN tests

John E. Boal — Tue, 26 Aug 2008 15:03:11 GMT

I posted a simple sample of accepance test code in Selenium and WatiN along with a sample web site to test. You can download the zip file here.

I also have posted a Fitnesse fixture in that zip file that illustrates how we can create a simple test fixture for Fitnesse acceptance testing. The Fitnesse tests aren't in the set, but here is the page wiki code that makes use of the fixture:

!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnet\FitServer.exe}
!define PATH_SEPARATOR {;}
!path dotnet\*.dll

Here is an acceptance test using our BusinessObjectTestFixture test class:

!|FitnesseFixture.BusinessObjectTestFixture|
|UserId|Password|Authenticate()|
|administrator|secret0|ADMIN|
|admin|secret0|NONE|
|administrator|secret|NONE|
|user11|secret11|USER|
|user11|secret0|NONE|
|user|secret|NONE|

I also created some STIQ tests, here is the code for the tests and components. Extract this zip file under repository\ProjectRoot folder and it should be able to test the sample site also.

ATDDSTIQ.zip (3.57 KB)

Acceptance Criteria: Interview with the Customer

John E. Boal — Thu, 17 Jul 2008 04:37:30 GMT

I often talk of Acceptance Test driven development, and how we can turn acceptance criteria into automated tests that illustrate that our software is doing the job the customer wants. However, sometimes it can be difficult to "extract" these criteria. This is a scenario where a customer needs a web control to take some raw data and turn it into a chart, and stream it to the user on the fly.

Conversation (we'll keep track of acceptance criteria in italics)

We need a chart control that will stream an image to the user.
What kind of image? PNG for now. Possibly GIF later.
1. Needs to produce streaming PNG image. Don't bother with GIF yet, probably YAGNI anyway.

Great. Where does the data for the chart come from?
A database. SQL? Yes. Is there a stored procedure?
Yes, the page can talk directly to the DB. OK, good.
I can get specifics and the schema from the DBA? Yes.
2. Data is read from calling a stored proc in a SQL database. Talk to DBA.

Ok, what kind of a chart is it? Line, bar, XY, pie, or something else?
Basically a line chart. OK.
3. Line chart

What are the axes? Time on the horizontal and data on the vertical.
4. Data are plotted chronologically.

If the chart is 640x480 pixels, is that OK?
No, size needs to be able to vary from 320x240 to 1024x768, depending on the caller.
Gotcha - so we'll need to parameterize the size. Yes, that would be good.
5. Height and Width need to be specified.
6. Min is 320x240, max is 1024x768.

If someone requests over the max or under the min, can we substitute these end values
instead of the actual request? Yes.
7. if invalid size, substitute valid size.

What color is the background? Doesn't matter. How about the foreground? Don't care.
Well, would yellow on white work? No, that isn't enough contrast. I see, OK.
8. Contrast between foreground and background.

Back to the line chart, are there markers on the data? Yes, we need tick marks for each point.
Should these ticks be a different color than the line? Yes.
9. Ticks are a different but still high contrast color from the line.

Are there labels on the axes? Yes, time in HH:00 on each hour for horizontal, and unit values on the vertical.
How many unit labels? It varies, has to be speficied. OK, so are the labels on the time axis
vertically or horizontally oriented? Normal - horizontal. Oh, and I only want every third hour labeled.
OK, got it.
10. Horizontal labels on X axis, alternating every third data point.
11. Parameterize the number of data labels on the Y axis.
...

So you see that in just a few seconds conversation, we can come up with 11 acceptance criteria. Each of these criteria can be written up as a test, so that the customer can press a button and see a scenario that illustrates each of these criteria. We don't need a line of code written to have this conversation, record this criteria, or even to automate it. Of course our tests will fail until we write the code...

We can also use tools like Fitnesse to show simple criteria and the outcome to a customer. However in the above example, it might be a bit difficult. We could have a "scenario" test that illustrates some of the conditions for the chart, and have the chart image itself in the results, so the customer can see the actual output in that scenario. I might be tempted instead to write an nUnit suite that I could run that would demonstrate the correct behavior in those scenarios. It depends also on the team's skill set, and the customer's expectations.

If we do this kind of automation around acceptance criteria, we will make it very easy on our customers, and they will feel very confident that they are getting what they want. While it isn't all that easy to demonstrate automated acceptance criteria in all cases, it is worth an attempt at trying to see what we can do, given our project constraints.

We also get the benefit of being able to run our acceptance automation, and in just a few seconds, we can tell if we are really "done" with the story. This kind of assurance gives us a definite advantage in being able to deliver future software down the road, because we can verify that the existing features still work as we add new ones.

In practical terms, however, this kind of testing can be expensive - at first. Many customers are much less concerned about testing the software vs. just getting some code. In these cases, customers' expectations need to be set that there is actual value in the work done to verify that the code is going to work. The more critical the code, the easier it will be to sell. Even if the customer finds no value in the tests, they will still be impressed in the demo when they see a button pressed, and all tests come up green. I think there is really a psychological benefit for the customer, showing them a bunch of green tests. After all, they are giving us a bunch of green too, aren't they? They will see this and naturally be more confident, accepting and more trusting of our work.

So, be sure to collect this criteria carefully. Make sure both the team and the customer are clear that the criteria is the agreed yard-stick on whether the software is working. Ask probing questions, and clarify the answers. Make sure you document the answers. And even if you find you don't have too much time, make sure to automate at least some of your acceptance criteria each sprint - and be sure to show it off in the demo.

Acceptance Test Driven Development, Explained

John E. Boal — Wed, 02 Apr 2008 06:10:03 GMT

Acceptance criteria should be our measuring stick when we write software. These criteria should give us the direction to go in writing features, and ultimately the final answer on whether the software is "done" or meets the customer's need. When the customer (or the customer representative, product owner, or business analyst) gives us a notion of what the software is to do for them, it represents a guideline and a goal for our development.

This article is a discussion of ATDD in an Agile development context with Scrum project management and Extreme Programming [XP] development methodology. However it does not apply specifically to Agile. These same techniques can be applied with almost any mechanism.

Test-Driven Development [TDD] says that we should first write a test before we write the code. This forces us to understand clearly what the code is to do, because we have to write the test to verify it. Writing the test is usually the hard part - the code next is easier. This is where the discipline of TDD is most needed. The test itself should be failing when executed, since we don't have the code under test [CUT]. This is a good sign - we need to actually "test the test." We then write as little CUT code as needed, just to make the test pass. Then, we start all over again - write a test, some more code, etc. We refactor the CUT as needed, to make it a cohesive, maintainable design, while still keeping all the tests passing. In this way, we have a complete test framework wrapped around our high-quality code, which makes certain that the code always functions the way we intended no matter what kinds of changes we make internally to it.

This process of TDD as described, is the traditional mechanism we now refer to at Unit Test-Driven Development, or UTDD. There are higher-level tests that we can write also, that have the same impact on our development process. Applying this same concept of a failing test driving us to write code is where this discussion now leads, but at one level of abstraction higher. We now look to the Acceptance Criteria to drive our TDD, or now ATDD.

1. Acceptance Criteria gives us a target for writing Automated Acceptance Tests [AAT's] a. Acceptance Criteria need to be written into an executable set of automated tests. b. AAT's should be able demonstrate to the customer that their criteria for the functionality has been met. c. Ideally, this should include as much detail as the requirements bear out, and as many tests and checks are necessary to convey that each of the criteria has been accounted for.

2. AAT's give us a reason to write code (they are failing - no CUT yet) a. AAT's can be written before writing the code, or written one at a time following this process.

3. We now write the empty framework for the CUT. Some AAT's may still be failing, but that's OK. a. This step deviates from strict TDD in that we are writing code, but not for the intent of passing a test. b. One example is that if a criteria states that a customer is able to bring up a web page with a set of information, obviously we need to provide a web page. c. This framework concept is creating the empty page or container for the code we are going to write. d. The empty framework should not cause AAT's to pass, because AAT's should written to test at a higher level of functionality.

4. To make the AAT's pass, we need to start providing the functionality inside the empty framework. Here we need to apply the standard TDD concepts, and write a failing unit test.

5. We now have a failing unit test, so we write enough CUT to pass that test.

6. We repeat at step 4, refactor, and continue following the UTDD process as described above.

7. When the functionality that the AAT's require is there, they will be passing, Unit Tests [UT's] are passing, and our story (feature) is complete. We now continue to step 1 again for the next story.

How do I do it? What do I need to get started?

Acceptance Test Driven Development is a practice that can yield good returns on the time invested. Customers will be pleased with the software they receive, bugs will be fewer, and delivery times will be shorter. However, there are a few things that are needed to make this practice effective in an organization.

Buy-in

People must be bought-in to the concept that test driven development is a good thing.
Management must support this practice, for if not, it will be a much more difficult thing to accomplish.
Developers must be willing to put aside their "classical" training and try something new.

Testing Tools

If we can't write an automated acceptance test for our product, we can't do ATDD. We need a tool designed for testing the particular type of product we are shipping.
Story Test IQ [STIQ] is a great tool for automated testing of web applications. Straight Selenium RC under C# code, driven by NUnit also works great, in my experience. VSTS has some good tools also, as do Fiddler, and probably lots of other products.
The framework must be in place on day 1 of the iteration to begin writing acceptance tests. If it isn't available, we won't be able to do ATDD.

Skills

Developers need the skills to be able to envision what is to be delivered, and be able to write automated test scenarios for the acceptance criteria.
Testers need to be able to validate that the tests cover enough of the functionality through testing the criteria, and provide other functional, integration, load, and performance tests in addition to the developers' unit tests and acceptance tests.
Management needs to be able to direct the right resources at tasks, no matter whether they are Development or Test tasks. Sometimes it's best to blur the lines between the disciplines to make this practice most effective.
Customers need to be aware that they are actually going to get what they specify (and perhaps ONLY that much). Education on criteria evolution is almost always required for customers. Don't be surprised if only 50% of the criteria are discovered in sprint planning and story generation for the first couple of iterations.

Experience

TDD takes discipline, and ATDD even more so. This is not a practice for an organization just beginning down the road of TDD. Put in at least 6 iterations of using strict TDD first before trying ATDD. The experience gained with using TDD will naturally flow into ATDD, and the organization and the customers will see the benefits.

Perseverance

Good things don't happen instantly. It takes some time to see the benefits of ATDD.
Be patient for a few sprints. It should take approximately three iterations to get it dialed in.

Customer Cooperation

Customers must be available for providing their criteria for acceptance.
Customers generally are not experts in this type of "specification" of criteria, and must often be helped through the process at first, to generate usable criteria.

Testing? Aren't I a Developer?

John E. Boal — Mon, 24 Mar 2008 05:47:08 GMT

Aren't developers just supposed to write their code and let the test team test it?

ahem.

no.

As a developer, I take pride in delivering *working* software. If I don't test my own software, I think that I'm really failing to do my job. Testing is not just for "testers."

As a professional software engineer, I need have not only unit tests for my code (a Development practice by the way), but I also need to write automated functional tests to make sure that all the functionality works as I think it should. Then, I need to have automated Acceptance tests that tell me if indeed the whole system works together to deliver the business solution that the story or requirements describe.

Acceptance Tests

Why do I start with acceptance tests?

Because... they should drive the entire development process. Test-Driven Development doesn't necessarily have to *always* mean Unit Test-Driven Development...

The Acceptance tests drive me to write code.
In order to write code, I need a unit test.
Then I can write the code that passes the unit test and the acceptance test.

job well done.

Unit Tests
	I write enough unit tests to be able to make sure that the functionality I write works as I intended. Not too many - tests require maintenance as well as the code they test - but enough to ensure that it's safe to refactor just about anything in the code and make sure that it still works as intended. This is the safety net that unit tests provide.

Code
	Write some. It should make the tests pass. Failure is not an option.

Functional Tests
	We need to make sure that all the pieces of functionality behave in a known way. This is what functional testing does.

Integration Tests
	We need to make sure that all the systems co-operate. Integration tests are deeper yet than functional tests, these are more like "end-to-end" tests of particular scenarios. Usually these scenarios cover all the happy paths of end-to-end, and most likely a couple of failure scenarios as well, to illustrate how the system overall behaves in failure modes.

Deployment Tests

You DID want to INSTALL the software and use it, right?

So... you got a test for that?

Sure you do.

Deployment tests are essential for making sure the software delivers and configures the binary bits in a way that's useful to the user. Uninstall is a particularly critical issue as well. Make sure that you have complete testing around these critical fundamental features of the software.

ATDD - Acceptance Test Driven Development

John E. Boal — Fri, 14 Mar 2008 05:39:09 GMT

Use ATDD as an advanced extension of TDD, keeping the software's development guided by the principles of satisfying the customer's needs through acceptance criteria and automated testing.

BDD, ATDD, UTDD, DSL's ... when will it all end...

The drive toward business-driven testing has never been stronger. Developers are seemingly now finding a higher and higher bar when it comes to customers' expectations of quality and features. Our tools are getting better, and we can deliver more software, faster. But, our methodologies haven't necessarily changed enough to satisfy today's customer expectations.

Enter Business-Driven Design...

Business driven design is a concept that enables us to take business requirements and current priorities and turn them into a software design through Acceptance Test-Driven Development. The business requirements that drive the need for the software are turned into specific criteria that allow the business to decide what the criteria are that will allow them to use a feature and have it meet their business need. Rather than the old-school way of gathering requirements, and having a requirements document and a functional specification, we now turn to individual small criteria that decide if the software is acceptable to meet the need. Some of the criteria map directly from functional requirements, and others may not have been captured in a traditional requirements gathering and specifying model.

Domain-Specific Languages (DSL's) are key to success in Acceptance Test-Driven Development. DSL's give us a way to communicate with the customer and domain experts in their terms. When we capture criteria in this manner, it becomes quite clear to those with domain knowledge, what is meant and what is desired. There is no need for a "translator" between the customer and the developers (this used to be called "Business Analyst"). The developers model the code in terms of the language the customer already uses. This mechanism leads to better communication, better encapsulation, and better object-oriented development.

Acceptance Test-Driven Development [ATDD] gives us a mechanism to use DSL's and direct customer involvement in making sure the software we deliver meets the needs. When we take the criteria and turn them into automated acceptance tests, it is far easier for the customers to see that they are getting what they asked for. It's also easier for the developers to have a target to shoot for, and have a goal to meet. This way, they are more focused on delivering a specific unit of functionality that the customer needs rather than (as so often happens) some "new feature" that they thought might be useful.

Much care needs to be put into the way that acceptance criteria are gathered and then automated. If there is something that is missed, it could critically affect the design. This is an opportunity for customers and developers to collaborate and get it right. The customer needs to understand that if it isn't on the acceptance criteria list, it isn't going to be in the software... Performance criteria, interoperability with other systems, and other criteria like these are often missed. Customers should have many opportunities to review and re-review the criteria before they are approved. Even still, sometimes things are missed. This is why it is important for the customer to be involved at all stages of the development process. The customer shouldn't just be involved in the criteria gathering, then come back later for their product. If things are missed, they will likely become apparent and turn up in daily work. If the customer is there to be consulted, decisions can be made about how to integrate missed criteria, and how to capture these better in the future.

Business-Driven Design is a business-centric, collaborative, agile mechanism for delivering quality software to today's demanding customer.