How to write a good integration test¶
This aims at providing a brief list of points one should take into account while writing integration tests (often incorrectly called unit tests).
Make creative use of the different
unittest.TestCase.assert*functions. In contrast, plain
assertstatements should only be used for mypy to have failure reporting work as intended.
Failure cases are important. Always test that validation errors are handled correctly, especially in the frontend.
Test with the least privileged user possible. If there is reason to believe code might behave differently for other users, add them as well, otherwise omit them to save runtime. Since logging in and out during a test is possible, it is possible to switch privileges during a test. This is e.g. useful to test logging.
If you write a regression test, always check that your test actually fails if the fix is not in place. Otherwise, it is worthless.
Join related tests together, if possible. For example, go through the whole lifecycle of simple entities. This reduces the total runtime of the tests.
Avoid raw IDs. They are hard to recognize. Use terms like
self.user_infunction simplifies this for users since it accepts full user objects (
USER_DICT["inga"]), user ids (
9) or names (
"inga"). Beware that for filling in a webtest form field with an enum, you need to use the
Do not imitate the existing
test_logtests. Those represent an anti-pattern. Instead, test the presence of log entries together with their generation. For backend tests, copying
pprintoutput is helpful to compile an expectation. For frontend tests, it usually suffices to check that
change_noteare a match.
@prepsqldecorator may be used to perform some sql queries to the database before the actual test. Needs to be speficied after
@as_usersto work correctly.
Remember to add the
@storagedecorator when accessing storage. Otherwise, your test will fail.
If you require as much time to write a unit test as you needed to write the original functionality, you are doing it right.
For frontend tests, additionally take into account the following points:
Get familiar with the
tests/common.py. There, many helpers are defined which simplify common testing tasks.
If you check whether things are present, be precise. Specify as precise as possible where a string should be, and make it as long as possible.
If you check whether things are not present, be vague. Do not specify where a string should be, and make it as short as possible.
It is possible to access the backend directly inside frontend tests, but this should usually be avoided. However, it is acceptable if you really want to check if data has been written to the database correctly if there is no simple frontend way to do so.
If you would like to test something, but you do not know how, it is usually possible. Look into
tests/common.py, into webtest internals, or ask people. Writing additional helpers can be quite tedious, but is usually worth it in the long run.
assertValidationErrorare some examples.
In general, we aim at 100 % test coverage for our Python code. This means that ideally, not only every statement, but also every control structure branch should be tested.
However, not every statement should in fact be tested. As a rule of thumb: If something is expected to potentially happen in practice, it should be tested, if strucuturally possible. (We will not start messing with subprocess calls manipulating the system to test some failsaves, for example.)
This means the following should not be covered:
Logical consistency checks, usually in the backend, either due to programmatic logic or due to database consistency.
Optional[str]in frontend endpoints – it is impossible to fail this as an incoming request.
On the other hand, just because something is hard or annoying to test, or requiring mocking, it would not warrant an exception. Nevertheless, it is always a trade-off between cost and use: Just because you can imagine a wild way to anyway perform a test, this does not mean you should do it.
Regarding statement coverage, we are at about 90 % and aim to increase this amount in the future. Starting in the end of 2021, we have a mid-term goal to significantly increase this number. Achieving at least 95 % should be possible without reaching vanishing benefit.
To run coverage checks, take a look at the Code coverage.