Test Suite
This page describes the technical details reagrding the test suite. To get a high-level overview of our testing philosophy, take a look at Testing instead.
Sample Data
The sample data lives in tests/ancillary_files/sample_data.json. You can fill
the database with the sample data by calling python3 -m cdedb dev apply-sample-data.
This will create a sample_data.sql file, drop the current
database state and at last repopulate the database with the sample data. Now,
you can use the DB as you would use it online, log in with the users described
below.
If you want to extend the sample data, you may simply change the sample_data.json
file. Be aware that changing existing and sometimes also adding new sample data
can have side effects on some tests, so you should run the whole test suite
afterwards to check that no test breaks.
As an alternative to fill the sample_data.json file manually, we provide
a script to dump the current database state into a JSON file. This can be
very helpfull if you want to add a huge amount of new sample data, which
sometimes comes with dependencies between different SQL-tables and can be quite
frustrating to be done manually. We also provide a make target for this purpose,
simply call make sample-data-dump inside /cdedb2.
Users
There is a default data set for the development it contains some users (according to the table below).
User |
ID |
Password |
Notes |
|---|---|---|---|
DB-1-9 |
secret |
admin with all privileges |
|
DB-2-7 |
secret |
canonical example member, moderator of lists |
|
DB-3-5 |
secret |
member, but not searchable |
|
DB-4-3 |
secret |
former member (but not disabled) |
|
DB-5-1 |
secret |
event user |
|
DB-6-X |
secret |
admin in all realms, but not globally |
|
DB-7-8 |
secret |
orga of an event |
|
hades |
DB-8-6 |
secret |
archived member |
DB-9-4 |
secret |
minor member |
|
DB-10-8 |
secret |
mailinglist user |
|
DB-11-6 |
secret |
assembly user |
|
lisa |
DB-12-4 |
secret |
member with whacked data |
DB-13-2 |
secret |
second meta admin to confirm privilege changes |
|
DB 14-0 |
secret |
mailinglist admin and event user |
|
DB-15-9 |
secret |
a disabled user (and member and CdE admin) |
|
DB-16-7 |
secret |
core admin and cde user |
|
DB-17-5 |
secret |
cde admin and not searchable member |
|
DB-18-3 |
secret |
assembly and event but not cde user |
|
DB-22-1 |
secret |
former member, corresponding to Verwaltungsteam |
|
DB-23-X |
secret |
former member, corresponding to Versammlungsleitung (presider) |
|
DB-27-2 |
secret |
former member, corresponding to Akademieteam |
|
DB-32-9 |
secret |
former member, corresponding to Finanzvorstand |
|
DB-37-X |
secret |
auditor (Kassenprüfer) |
|
DB-38-8 |
secret |
former member, corresponding to CdElokal-Koordination |
|
DB-42-6 |
secret |
honorary member |
|
DB-48-5 |
secret |
assembly admin |
|
DB-100-7 |
secret |
equal to Anton - to test sorting |
Secrets
Every attendee at an assembly got a personal secret to verify their votes afterwards. If an assembly is concluded, all secrets are deleted from the database. Here, we kept the secrets of those users of the sample data.
Assembly |
User |
Secret |
|---|---|---|
2 |
Rowena |
asdgeargsd |
LDAP
We expose some information about our users via ldap. This contains some general information like name and mail address, and group privileges of the user in particular. The data is directly retrieved from the sql tables of the CdEDB and therefore needs no syncing.
To locally test our ldap integration, we add some ldap agents to the CdEDB sample data. They can be used to connect to and retrieve data from the ldap system. To test the permissions of the live duas properly, we also add them in our sample-data.
CN |
Password |
Notes |
|---|---|---|
admin |
secret |
olcRootDN |
apache |
secret |
|
cloud |
secret |
|
cyberaka |
secret |
|
dokuwiki |
secret |
|
test |
secret |
does not exist in live instance |
Running tests
To allow simultaneous development and testing on the same machine without interference between the development and test process, we strictly separate the stateful parts (in particular sql database, file storage directory and logs) of development and test instance.
Note
The majority of our tests do not need the file storage. Since the setup is costly, every test
who needs it has to get the @storage decorator from tests.common for the
storage directory to be created. After this test has finished, the directory will
be deleted.
To achieve this, we use the same mechanisms as for development (or production)
environments. This even allows running multiple test instances in parallel!
Each instance of the test suite gains its own configuration file in tests/config/,
which extends the existent default configuration from cdedb/config.py.
This configuration may (in contrast to cdedb/localconfig.py, which is not
taken into account for test instances) include additional keys which are not
present in the default configuration, if they are needed during the test process.
The setup process uses the Makefile and overwrites the default values of the
make variables with the values specified in the config file.
To prepare and run the testsuite, we provide a central script: bin/check.py
You can pass some pattern to run only specific tests, or use the command line
arguments to run only specific parts of the test suite. For detailed information
run:
bin/check.py --help
In the following, we will explain the pattern matching mechanism and shortly introduce each part of the test suite.
Pattern matching
You can pass an arbitrary amount of patterns to check.py, which will then get matched
against the fully qualified test method name.
Such a full specifier looks like
tests.frontend_tests.event.TestEventFrontend.test_create_event, but you can also pass
an unambiguous part of it, like e.g. just create_eve, for convenience.
These parts of course can also specify complete test files, like backend_tests.core,
where unambiguous parts suffer too.
Pattern matching is performed by unittest, which uses fnmatch.fnmatchcase
internally [1].
If a pattern without an asterisk is passed it will be wrapped with one on both ends.
Application tests
This is the main part of our test suite, providing tests for the CdEDB WSGI application,
including the frontend tests (tests/frontend_tests), backend tests (tests/backend_tests),
database tests and tests for the gluing parts (like validation, all in tests/other_tests).
To decrease runtime, we split this tests in our CI in three parts, using the
configuration present in tests/config/test_1.py to tests/config/test_4.py.
To avoid test clashes when different parts use the same configuration, we use
a simple locking mechanism with lockfiles inside /tmp and let the test script
choose a free test configuration automatically.
LDAP tests
This includes all tests of our LDAP interface. This is a bit more tricky, since it additionally involves the ldap server, which is not able to serve the same ldap tree for different databases (the development and the test instance) simultaneously. So, we decided to let our ldap server serve the test database only during test runs. This avoids resetting the development instance each time the ldap tests are run, but also prevents accessing the development ldap tree during test runs. This may be fixed in the future.
Inside the tests, we mock a ldap client querying our ldap server and check if
the results satisfy our expectations. The configuration for this part of the
testsuite is present in tests/config/test_ldap.py.
XSS tests
To prevent XSS mitigation, we test if our code performs proper HTML escaping
on user input. For this, we use the bin/escape_fuzzing.py script to inject
a payload containing HTML tags inside the database and check if they are
escaped properly during serving.
The configuration for this part of the testsuite is present in tests/config/test_xss.py.
Code coverage
The coverage html reports for easier inspection are accessible on the local dev instance via Apache at localhost:8443/coverage for docker and localhost:20443/coverage for the VM.