Acceptance criteria are the real key to being able to develop software in the agile model. Acceptance criteria is not really adequately represented by a use case, even with several alternate path scenarios... I have seen this and it just doesn't do the trick. A scenario is good, but it's really not acceptance criteria. We can however, analyze these scenarios and alternate paths, and usually come up with a pretty good set of criteria.

Beware of the incomplete criteria. It's darn hard to get it all. We know this. But, make an attempt anyway. Use the thinking that if it isn't in the criteria it won't be delivered, and that the developers can implement it any way they want... This thought should be scary enough for a customer that they will be forced to think hard about what they really want the software to do.

When we think we know what we are doing, we can write some tests, write some code. We think we have the idea, only to find out later that we had misunderstood or didn't have all the requirements. Acceptance criteria are absolutely the must-have thing for agile processes to succeed. We need the Customer or product owner [PO] to provide clear, concise, and specific criteria for our software.

I'm not saying that generating/gathering acceptance criteria is easy. It takes practice. It takes cooperation. It takes understanding on both the development team and the customer/PO that "if it isn't requested or specified in the acceptance criteria, it isn't going to be delivered." It's really kind of harsh actually to practice this, so I wouldn't advocate it. However, it might be a good idea to talk about it in those kind of terms while in the sprint planning or story negotiation phase.

Without the ability to know that we are doing the right thing, surely we will fail to deliver it. The customer knows exactly what they want. OK, most customers really don't know what they want. It is up to us as a development team to be able to know this, and help them out by asking lots of questions, and examining exactly what we propose to deliver. After all, that is why they hired us right?

Train your team. Let them know that the ancient role of "Business Analyst" is no place to be found in an agile team, and the whole team is really responsible for making sure to ask the right questions. I do like the whole user story concept, but it is far more practical in sprint planning to capture some of the actual acceptance criteria and store it attached to the story somehow, so that it is accessible by the team and can be implemented as automated tests.

Test automation is critical, mandatory, required, and not optional. Did I mention that test automation is required. Perhaps I should mention it again, in case my previous sentences weren't clear. Automate, Automate, Automate. Double-click run test suite, should be the demo in the sprint review and customer demonstration at the end of the sprint. This is really and truly the only way that agile teams can operate successfully.

If you have people doing any kind of manual testing (other than exploratory), this is a great opportunity to get the team to be far more productive by assisting the manual testers to get these tasks automated. Sometimes it is not practical to automate some manual tests. But in my experience - NOT OFTEN. If it looks hard to automate, someone should be asking why the code is so "untestable"... In most cases, the three or four dev or test hours it takes to automate the most difficult manual test case scenario is more than paid back in the number of times the automation can be run from then on. The number of bugs and breaks that it will prevent is a definite value add to the organization, and in my opinion, money in the bank.

I often talk of Acceptance Test driven development, and how we can turn acceptance criteria into automated tests that illustrate that our software is doing the job the customer wants. However, sometimes it can be difficult to "extract" these criteria. This is a scenario where a customer needs a web control to take some raw data and turn it into a chart, and stream it to the user on the fly.

Conversation (we'll keep track of acceptance criteria in italics)

We need a chart control that will stream an image to the user.
What kind of image? PNG for now. Possibly GIF later.
1. Needs to produce streaming PNG image. Don't bother with GIF yet, probably YAGNI anyway.

Great. Where does the data for the chart come from?
A database. SQL? Yes. Is there a stored procedure?
Yes, the page can talk directly to the DB. OK, good.
I can get specifics and the schema from the DBA? Yes.
2. Data is read from calling a stored proc in a SQL database. Talk to DBA.

Ok, what kind of a chart is it? Line, bar, XY, pie, or something else?
Basically a line chart. OK.
3. Line chart

What are the axes? Time on the horizontal and data on the vertical.
4. Data are plotted chronologically.

If the chart is 640x480 pixels, is that OK?
No, size needs to be able to vary from 320x240 to 1024x768, depending on the caller.
Gotcha - so we'll need to parameterize the size. Yes, that would be good.
5. Height and Width need to be specified.
6. Min is 320x240, max is 1024x768.

If someone requests over the max or under the min, can we substitute these end values
instead of the actual request? Yes.
7. if invalid size, substitute valid size.

What color is the background? Doesn't matter. How about the foreground? Don't care.
Well, would yellow on white work? No, that isn't enough contrast. I see, OK.
8. Contrast between foreground and background.

Back to the line chart, are there markers on the data? Yes, we need tick marks for each point.
Should these ticks be a different color than the line? Yes.
9. Ticks are a different but still high contrast color from the line.

Are there labels on the axes? Yes, time in HH:00 on each hour for horizontal, and unit values on the vertical.
How many unit labels? It varies, has to be speficied. OK, so are the labels on the time axis
vertically or horizontally oriented? Normal - horizontal. Oh, and I only want every third hour labeled.
OK, got it.
10. Horizontal labels on X axis, alternating every third data point.
11. Parameterize the number of data labels on the Y axis.
...

So you see that in just a few seconds conversation, we can come up with 11 acceptance criteria. Each of these criteria can be written up as a test, so that the customer can press a button and see a scenario that illustrates each of these criteria. We don't need a line of code written to have this conversation, record this criteria, or even to automate it. Of course our tests will fail until we write the code...

We can also use tools like Fitnesse to show simple criteria and the outcome to a customer. However in the above example, it might be a bit difficult. We could have a "scenario" test that illustrates some of the conditions for the chart, and have the chart image itself in the results, so the customer can see the actual output in that scenario. I might be tempted instead to write an nUnit suite that I could run that would demonstrate the correct behavior in those scenarios. It depends also on the team's skill set, and the customer's expectations.

If we do this kind of automation around acceptance criteria, we will make it very easy on our customers, and they will feel very confident that they are getting what they want. While it isn't all that easy to demonstrate automated acceptance criteria in all cases, it is worth an attempt at trying to see what we can do, given our project constraints.

We also get the benefit of being able to run our acceptance automation, and in just a few seconds, we can tell if we are really "done" with the story. This kind of assurance gives us a definite advantage in being able to deliver future software down the road, because we can verify that the existing features still work as we add new ones.

In practical terms, however, this kind of testing can be expensive - at first. Many customers are much less concerned about testing the software vs. just getting some code. In these cases, customers' expectations need to be set that there is actual value in the work done to verify that the code is going to work. The more critical the code, the easier it will be to sell. Even if the customer finds no value in the tests, they will still be impressed in the demo when they see a button pressed, and all tests come up green. I think there is really a psychological benefit for the customer, showing them a bunch of green tests. After all, they are giving us a bunch of green too, aren't they? They will see this and naturally be more confident, accepting and more trusting of our work.

So, be sure to collect this criteria carefully. Make sure both the team and the customer are clear that the criteria is the agreed yard-stick on whether the software is working. Ask probing questions, and clarify the answers. Make sure you document the answers. And even if you find you don't have too much time, make sure to automate at least some of your acceptance criteria each sprint - and be sure to show it off in the demo.