I watched a fantastic talk from J.B. Rainsberger called “Integrated Tests are a Scam.” It was an insightful discussion about the types of tests that we write during software development and the types of tests that actually help you ensure proper code coverage.
As the title indicates, he spoke about the effort that we take when developing integration tests and specifically, the tremendous amount of effort this can often require to ensure coverage of all the different paths and connections between classes, and regretfully, the lack of actual coverage can provide.
This really spoke to me. As a person with a QA background, I’ve spent a long time going down the rabbit hole of testing, both manual and automated, only to find that it didn’t cover “all” the cases and often times, in order to actually automate something, the time it took to write the test and setting up the dependencies and required data took much longer than seemingly writing the feature/method itself.
Writing tests should not feel like pushing a rock uphill through molasses…but often times, it does.
Evaluating test failures should not be ethereal, where you don’t quite know what failed because there are many players that interact during the test.
During the talk, he mentions that there could be between 1–80% coverage from integrated tests, and shockingly, or maybe not so shockingly, the difference between hitting the 1% vs 80% is sheer luck. When you think about it, there is a sheer volume of permutations that you need to go through and that volume rises exponentially.
However, there is light at the end of the tunnel and it is not a train.
He provides strategies on how to apply proper techniques both during design and test writing. In particular, he mentions 4 types of tests:
1) State-based tests — these validate the entry point and that the methods do the right things.
2) Collaboration tests — these validate that a class, or a layer, talks to it’s neighbor correctly.
3) Contract tests — these validate that the class implements the interface correctly.
4) Integration tests — these should actually lie on the very edges of the software, where the code talks to third-party/external resources. These tests should occupy only the boundary between what is inside your software vs outside. You don’t need these tests to cover the internals of your software because they are covered by the other tests above.
For example, in a web application where a client communicates with a server,
The collaboration tests ask of the client:
– Do I ask the right questions?
– Do I handle the answers correctly?
The contract tests ask of the server:
– Do I try to answer the questions correctly?
– Do I really answer in the way that is expected?
Collaboration tests and contract-based tests cover both sides of the equation within your own software.
Designing your code with interfaces and dependency injection allows you to write more of faster and resilient unit tests instead of slower and more brittle integration tests.
These tests also help you write better design which frees up more time to write more unit tests… and the cycle continues down a more happier path that when you may have started reading this blog post 🙂