Those who know me well know I'm a big proponent of test-driven design and automated tests. I recently wrote an article about why test-driven development makes sense. In that article, I proposed to automate your tests as much as possible. But your system may pass all the tests you so meticulously wrote, while still not doing what the user wants or needs. This is where acceptance testing helps.

A History Lesson

First, let's take a quick look at the history of acceptance testing. Acceptance testing goes beyond software development. It's a practice taken from the broader world of engineering. When a product or a prototype is finished, it goes through a series of acceptance tests. These are predefined tests that follow a series of steps to assert that the product satisfies the requirements of the user.

For example, when the US Navy wants to test the catapults on its aircraft carriers, it launches a car with the approximate weight of a plane. I can imagine they measure certain parameters on the catapult system, as well as the distance the car traveled, etc.

I saw another example when I worked at a television factory as a student. At the end of the line, when the television was fully assembled, as the last step before packaging, someone had to hook it up to the mains, turn it on and put it in front of a machine that analyzed the image quality (brightness, color, etc).

The idea is that the separate components might have been tested previously, but now we test the finished product. We test it according to several test scenario's that we have defined up front. We no longer care about the internal quality, i.e. how well-engineered the separate components are. We're looking at the external quality. This means testing to see if the solution does what we wanted it to do.

In Software

Software development has taken a lot of cues from engineering, and acceptance testing is one of them. In software, acceptance testing can span from ad-hoc tests of the system by developers, over professional testers or end users testing a release, to automated tests of the entire system.

An acceptance test will cover as many components of the system as possible or necessary. The idea is to use the real application, full bells and whistles. However, opinions vary on whether or not to include external systems that are not under your control. Either way, it will often run in an environment that is isolated from the production environment. Naturally, you don't want a system in production if it doesn't pass the acceptance tests.

In essence, an acceptance test is a black-box test. We're not looking under the hood. We just want to know if the system satisfies the requirements as defined by the end-user. So it's important that you have good and frequent conversations with your user.

Different Opinions

I should mention here that some developers define acceptance tests differently. You might see acceptance tests put between integration tests and UI tests, or instead of integration tests. The concept of an acceptance test I subscribe to is closer to that of the Agile Alliance. Another great resource is the page of Software Testing Fundamentals. Both define it as a test performed by or described by the customer, the end-user. So for me, it includes the UI, the database, network calls, etc.

The Software Development Process

You have probably figured out that acceptance tests come at the end of the software development process. After all, you can't test the entire system, if it isn't finished yet. But it can be a little more nuanced than that.

First, you can already define the acceptance tests before doing any programming work. Indeed, acceptance tests are a way of defining the requirements of a system.

Second, the acceptance test suite grows with the application. In an agile world, we ideally construct acceptance tests for each sprint. At the end of the sprint, we can then perform these acceptance tests to see if the sprint was successful. Even better would be to perform the tests during the sprint, after the team has finished implementing a story. If the tests fail, something is wrong with the implementation.

As for who performs these tests, opinions will differ. I've worked at companies where we, the developers, executed the acceptance tests. I've also worked at companies where the business analysts/testers did them. Another option would be to have a select group of end-users come in to test the system and provide feedback. Ideally, the tester has sufficient knowledge of the problem that the application is trying to solve.

What works best for you might not be the best for someone else. The key point to acceptance tests is to test the bigger picture. To test the finished feature, without regard for the underlying technical implementation. All sorts of issues may come out of acceptance tests that don't surface from other types of tests:

• Plain old bugs ("if I enter a special character, it doesn't render correctly")
• Performance issues ("the application feels slow")
• Design matters ("it doesn't look nice")
• User experience problems ("it's not really intuitive")
• Changing requirements ("I know we agreed on this, but now that I'm actually using the application, I realize it's not what I need")

Should I Automate?

As you can imagine, when a system grows, it can become quite labor- and time-intensive to perform all the acceptance tests manually. At a certain moment, you will wonder if it's all worth it. Should you just drop the older tests and only test the most recent features? I know I've worked in a successful team that did exactly that. We only tested the features of the finished sprint.

However, if you could automate this you wouldn't lose the great value that acceptance tests provide. You won't be able to automate every aspect of a manual acceptance test by a human being. For example, a computer can only go so far as to test the intuitiveness of an application. This is inherently a human trait, and what is intuitive for one, might be less so for the other. It's even harder for a computer to realize that the original requirements don't fulfill the business needs. A system may pass all tests and requirements that the user or client defined. But often users will acquire new insights and change the requirements after actually using the system.

But at least part of acceptance tests can be automated. You can use load tests to check if new features don't reduce the performance and responsiveness of the application. You can use UI tests or the Gherkin language to create a test suite that tests entire features automatically, including edge cases.

So automate as much as you can, but realize it won't fully replace manual acceptance testing.

Caveats

When automating your acceptance tests be sure to avoid too much technical jargon. An acceptance test uses the language of the end-user. If you write up your tests in a way that is too close to the technical implementation, your tests could break because of a technical change, even though the product still does what it's supposed to do.

This encourages you to collaborate with the customer or user to write up sensible tests. This in turn fosters trust and improves communication and collaboration.

Another Tool in Your Tool Belt

Unit tests and integration tests will drive you to build the system right, i.e. in the right way. But they don't guarantee you build the right system, i.e. what the user wants. Acceptance tests are a vital tool to check if you've built the right system. This requires that you work together closely with the end-user or customer of your work. As always, automate what you can, but keep in mind that automation won't replace every aspect of acceptance tests.

This post was written by Peter Morlion. Peter is a passionate programmer that helps people and companies improve the quality of their code, especially in legacy codebases. He firmly believes that industry best practices are invaluable when working towards this goal, and his specialties include TDD, DI, and SOLID principles.