As software engineers, we know why we test our code. Because if it's not tested, it's broken. But how should we test our codebase? Should we be writing an end-to-end test for every feature gets delivered? After all, an end-to-end test might reveal a problem that unit tests can't catch.
You wouldn't be the first developer to prioritize writing end-to-end tests over unit tests. However, end-to-end tests are notoriously slow and unreliable. The ice-cream cone and hourglass shape testing suites are used to describe anti-patterns in software testing due to their over-reliance on end-to-end tests. Test suites with these shapes frustrate developers and they will start to lose confidence in the automated builds.
 |
| Hourglass distribution: end-to-end tests and unit tests are equally distributed |
 |
| Ice-cream cone distribution: over-reliance on manual and end-to-end tests and not enough unit tests |
Mike Cohn, one of the founders of the Scrum Alliance, talks about the Testing Pyramid which looks like this:
 |
| Testing Pyramid |
The pyramid consists of 3 layers: unit testing, service(integration) testing, and UI(end-to-end) testing.
Unit tests
At the base layer of the pyramid, we have the unit test. Unit testing is the primary tool you have as a developer to test your code's logic. Unit tests are quick(on the scale of milliseconds), reliable and can be run locally - giving you a fast feedback loop. The majority of the automated tests you write should be an unit test. Because unit tests can be run on a single process, there's little to no room for infrastructure issues that may contribute to a unit test failing(e.g. network issues, failure to deploy a service).
Integration tests
Integration tests are the middle layer of the testing pyramid. They test your code's integration with another service(database, third party, another microservice). In my experience, database integration tests are still quick(on the scale of seconds) and reliable. You can spin up a docker container of the database image every time you run a database integration test locally, resulting in a (still) fast and reliable testing experience.
In a standalone environment, all components of the integration test should run on the same machine. This prevents potential network level problems and reduces the likelihood of infrastructure issues breaking our tests. If our code runs on node A and the database on node B, a failure of either or those nodes will break our test run and result in a failed build. Here, we have two points of failures: node A and B. By placing both our code and database on a single node, we've restricted our infrastructure failure scenarios to a single node.
End-to-end tests
The allure of the end-to-end test is in it's holistic nature. With an end-to-end test, we can test the end-users' experience with our application. We can test everything a unit test can and a whole lot more. However, just because we can doesn't mean we should.
While end-to-end tests promise a lot at the surface, they are known for being slow + unreliable + expensive. Before we can start running end-to-end tests, we need to deploy the entire cluster of microservices to a standalone environment specifically for this build. This collection of service deployments consumes most of the time spent running e2e tests. Because services usually depend on other services, we can't infinitely parallelize our deployments to shave off time.
While there's still a time and place for end-to-end tests, it's important to realize the trade-off we're making. End-to-end tests should be used sparingly to test assertions that we can't with a simpler unit test.
Where does manual testing fit?
As developers, we try to automate everything and despise doing manual work when we can automate it. However, there's a time and place for manual testing. Manual testing can be useful when the problem only surfaces in the production environments or conditions(race conditions, load).
Final Word
It can be easy to fall prey to the guarantees of end-to-end tests and how easy it is to test the completeness of your code in a single test. However, we must be cognizant of the hidden costs of end-to-end testing and use them sparingly in our code base.
References
https://martinfowler.com/articles/practical-test-pyramid.html
