Even in the agile world, testing is important to assure the delivered software will meet its expectations. On agile teams, testing provides the neccessary feedback to move the workitems to ‘Done’, but there is less time to prepare, execute and report than in a traditional development approach. Therefore, testing is a whole team effort – not only the dedicated testers do the work, but also the developers and functional analists. Since the team cannot deliver without having tested the software, the software must be testable.
As shown in this data, the defect detection rate of a single testing approach is limited. In most cases, unit testing alone will not uncover more than 30% of the errors introduced. This means manual tests, integration tests and exploration testing are required as well. To cope with this, software must be designed for testability on all levels and in all contexts. For software designed for testabiility, testing will take less effort and the feedback faster.
The most important aspects for testability can be summarized by the acronym SOCKS. It stands for: Simplicity, Observability, Controllability, Knowledge and Stability.
- Simplicity is an important quality because complex software is hard to test. When the system is less complex, setting up the test, executing it and interpreting the results is easier.
- Observability means that the internal state and the results of the executed algorithms can be inspected. The tester must have access to UI, Reports, logfiles, and diagnostics. Other ways of improving the observability are supplying custom a custom API or a dedicated test UI – without compromising the security of the system.
- Controllability determines the extend to which the system can be put in a desired state. When testing, you do not want to execute a lot of (manual) steps to reach the initial state for the function to test. It is better to have an option to import the data, or to directly manipulate the data to do an isolated test. For testautomation it is easier to use an API than having to manipulate (script and playback) the user interface.
- Knowledge of the system under test and the technology used makes testing more effective. Good traceability from the requirements to the implemented code helps to select the essential parts of the software to test. Good documentation makes is easier to setup the tests and interpret the results.
- Stability is required to prevent having to test the same components over and over again. Code following the SOLID principles will have a positive effect on the stability of the code. A second reason why stability is important is when errors in one component propagate to another component making it impossible to test. To increase testability the design of the code could introduce bulkheads to confine errors to the component where they occured.
During the developement of the system different types of tests must be executed. The design of the system can support the different phases in the test proces. This is especially important when the system under test must connect with external systems. In these cases i would apply the adapter or facade pattern to the design and set up the connection to the external system through configuration rather than code. In the early developer and unit tests mocks or stubs can be used. When moving to the integration test phase, the stub can be replaced by an actual testsystem and finally the test is executed against the test environment on the target platform. Note that, you will have to think about observability in all these context. When doing a development stage, white box test, monitoring the state of the application is often straightforward, but moving to an actual external system makes inspecting the results much more complicated. It can be neccessary to add fields to the UI or write identity information to the logfiles to be able to lookup the results in the external system (for instance when you cannot use you own technical key to find the data you did send to the external system).
Designing for testability means different things for each phase in testing. For unit tests and developer tests the main focus will be on the design of code. The way the code is structured can have a great impact on how good the code can be unit tested. Testability is increased by preventing anti-patterns like non-deterministic code, methods with side-effects, use of singletons, but use patterns like Dependency-Injection and Inversion of Contol. A detailed discussion can be found here.
For testability in integration and acceptance test phases, higher level design decisions are needed. What components and APIs are defined at the architecture level can have a major impact on te testability.
Architecture & Design
- Are export and import capabilities planned?
- Can the complete system be restored from a backup set?
- Can external dependencies be mocked?
- Is there an API to inspect the internal state of the application?
- Is there a standard test interface?
- Are multiple installs with different configurations on the same machine possible?
- Is installation/deinstallation scriptable?
- Is the system partioned in different compartments that prevent errors propagating?
- Is documentation available descibing the components and their interaction?
- Is there a logging framework to facilitate logging from the application?
- Are control/observation points available where testing code can be injected?
- Are built-in tests available?
- Can the component receive test messages?
- Are performance and usage metrics collected?
- Are there options to provide all exception conditions?
- Is there distinct output per process/input (can all ouput be correlated to the input)
- Are dependencies injected instead of using the service location or singleton pattern?
- do pages/objects have unique names (is system automation friendly)?
- If methods alter data that is not an input argument, is the data returned as a function result?
- Are all state transitions logged?
- Are there useful comments in the code
- Is coupling to concrete implementations prevented?
- Are external interfaces wrapped using adapter or mediator pattern?
- Is the API documention in the source code (and published to service consumers)?