Automated unit tests are hard to write. Software architecture must be designed carefully to allow unit testing. You have to spend time to write tests as well and it's not easy to write good tests. It's easy to make big mess that is hard to maintain after few weeks.
On the other hand automated integration tests are hard to maintain and are fragile. You can "write" them pretty easy in a record-and-replay tool, but later they show their real cost during maintenance.
But there's an answer for problems mentioned above. Do you know Eiffel language? The language has special built-in constructs that support executable contract specification. It's called Design By Contract (DBC). DBC is easier to write and to maintain because no special test cases need to be specified, just conditions for method parameters, expected results and state invariant that must be preserved. How DBC can substitute old-fashioned tests? Read on!
General concepts for DBC are:
- pre-condition: you can specify rules for every input parameter to check contract fulfillment from caller side. If a pre-condition is not met that means there's something wrong with caller
- post-condition: rules for post-method execution state can be specified as well. You can refer to input parameters, current object state and original object state (before method call)
- class invariants: refers to object members and must be valid after every public class call
As an example I'll use DBC implementation called "pycontract" (Python language). It's pretty easy to introduce in a project and uses standard doc-strings (like doctest module). No preprocessing is needed.
Pre-condition is specified by pre keyword:
pre:
someObject != None and someObject.property > 0
Post-condition syntax uses some special syntax to allow to refer to old values, you have to mention mutable variables object that should be available for state change comparisions:
post[self]:
self.property = __old__.self.property + 1
Above statement shows that property should be incremented by one after method return.
Invariant syntax is as follows:
inv:
self.state in [OPEN, CLOSED, SENT]
All above statements must be embedded in method/class comments, here's full example:
class SampleClass:
"""
inv:
self.state in [OPEN, CLOSED, SENT]
"""
def open(self, x):
"""
pre: x > 0
post: self.state != CLOSED
"""
DBC assertions (like doc-strings) are disabled by default (they are just comments). You can decide to enable them for example in development builds (there's an overhead related to DBC). An example how to enable pycontract:
import contract contract.checkmod(modulename1) contract.checkmod(modulename2)
Example failure (postcondition not met) caught by DBC:
DokumentFiskalny.testDokumentu() Traceback (most recent call last):
File "test.py", line 164, in
run_tests()
File "test.py", line 144, in run_tests
run_test_module(db, dao, moduleName)
File "test.py", line 54, in run_test_module
exec("%s.%s(env)" % (sName, sItem))
File "", line 1, in
File "", line 3, in __assert_testDokumentu_chk
File "lib/contract.py", line 1064, in call_public_function_all
return _call_all([func], func, va, ka)
File "lib/contract.py", line 1233, in _call_all
result = func.__assert_orig(*va, **ka)
File "/home/darek/public_html/kffirma/DokumentFiskalny.py", line 433, in testDokumentu
df.zapisz(args)
File "/home/darek/public_html/kffirma/DokumentFiskalny.py", line 254, in zapisz
args["idPodmiotuGospodarczego"],
File "", line 3, in __assert_DokumentFiskalny__create_chk
File "lib/contract.py", line 1165, in call_private_method_all
return _method_call_all(getmro(cls), method, va, ka)
File "lib/contract.py", line 1205, in _method_call_all
return _call_all(a, func, va, ka)
File "lib/contract.py", line 1241, in _call_all
p(old, result, *va, **ka)
File "", line 3, in __assert_DokumentFiskalny__create_post
File "/home/darek/public_html/kffirma/DokumentFiskalny.py", line 155, in _create_post
assert len(result) >= 2
The last part of this puzzle is coverage generation. DBC assertions are evaluated at runtime, so you have to ensure enough percentage code is execute during test runs. Im my experience you can rely on:
- integration-like scenarios (without explicit result verification)
- random input generation
For both cases you can integrate at high, UI level (keyboard, remote controller, HTTP, ...).
Focus on quality techniques is crucial for any software project. If you (1) pay some effort in making your codebase resistant to regressions and (2) maintain that state during project lifetime you can gain more stable software. The real craft lies in selecting proper tools and having enough energy to implement them.