Manual Setup

This describes the steps necessary to get the project running on a machine.

Note

This is somewhat complicated and it is advised to use the virtual machine in most cases. Instructions for obtaining and using the image can be found at Setup the VM.

Below may be some Gentoo-specific bits, for Debian-specific stuff look at the setup scripts in related/auto-build.

Prerequisites

We need some dependencies:

  • python (at least 3.9)

  • PostgreSQL (at least 13, for drop with force)

  • Apache (with mod_wsgi)

  • git

  • texlive (incl. luatex; for generating pdf documents)

Further we depend on a number of python packages:

  • passlib

  • psycopg2 (at least 2.5.4, for jsonb support)

  • werkzeug (at least 0.15, for correct multi-params in urls)

  • dateutil (only needed for dev script, namely analyze_timing.py)

  • babel

  • docutils

  • jinja2

  • markdown

  • bleach

  • python-magic

  • python-imaging-library (more specifically pillow)

  • zxcvbn

  • icu

  • mailmanclient (at least 3.3.3, for unsubscription handling)

At last there are some recommended dependencies:

  • sphinx (for building the documentation)

  • guzzle-sphinx-theme (a documentation theme)

  • webtest (for tests, at least 2.0.17 for handling of multiple elements with the same name)

  • lxml (the python module; used in the test suite)

  • pgbouncer (otherwise database performance may be degraded)

  • fail2ban (for preventing brute-force attacks)

  • pylint (for code analysis)

  • requests (the python module; used in some scripts)

Checkout the repository

Use git to clone the code:

git clone ssh://gitea@tracker.cde-ev.de:20009/cdedb/cdedb2.git

For this to work you need to create an account on the tracker website which has then to be granted access to the cdedb repository (send a mail to the db list cdedb@lists.cde-ev.de). All commands below assume, that you are in the root directory of the repository. Now you can build the documentation by issuing:

make doc

Configure the application

First of all, you need to create your personal configuration and make the path to the file available via environment variable, like described in Environment Setup. The configuration file may be empty if you do not want to override the defaults from cdedb.config.

A sample configuration for development instances can be found in related/auto-build/files/stage3/localconfig.py.

Prepare environment

Now we set up all auxiliary stuff. We assume, that postgres is configured for peer authentication (i.e. the system user xy has access to the postgres user xy). First execute as user with enough permissions (that is the ability to run sudo -u postgres ... and sudo -u cdb ...) and with running postgres:

python3 -m cdedb db create-users
python3 -m cdedb db create

The first one will create the database users, the second one the actual tables. To seed them with sample data, run additionally:

python3 -m cdedb db populate

Now configure pgbouncer in pgbouncer.ini (in /etc) with the following:

[databases]
cdb =
cdb_test =
cdb_test_1 =
cdb_test_2 =
cdb_test_3 =
cdb_test_4 =
cdb_test_ldap =
cdb_test_xss =

[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
unix_socket_dir = /run/postgresql
listen_addr = 127.0.0.1
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer_users.txt
pool_mode = session
server_reset_query = DISCARD ALL
max_client_conn = 100
default_pool_size = 20

Additionally place copy file related/pgbouncer_users.txt to /etc/pgbouncer_users.txt for authentication (otherwise pgbouncer will refuse connections):

cp related/pgbouncer_users.txt /etc
chown pgbouncer:root /etc/pgbouncer_users.txt
chmod 600 /etc/pgbouncer_users.txt

This file may be regenerated with the mkauth.py tool from the pgbouncer tar-ball.

Now we set up the Apache server, first add the following lines to /etc/apache2/httpd.conf:

LoadModule wsgi_module modules/mod_wsgi.so
ServerName localhost

and then insert the following close to the end of /etc/apache2/vhosts.d/00_default_ssl_vhost.conf:

WSGIDaemonProcess cdedb processes=4 threads=4
WSGIScriptAlias /db /path/to/repo/wsgi/cdedb.wsgi

<Directory /path/to/repo/wsgi>
Require all granted
</Directory>

Alias /static /path/to/repo/static
<Directory /path/to/repo/static/static>
Require all granted
</Directory>

note, that this is syntax for apache-2.4 (which differs from apache-2.2).

Next we need to create the directory for uploaded data (where www-cde is the user running Apache):

python3 -m cdedb filesystem --owner www-cde storage create

To populate the storage with sample data, run additionally:

python3 -m cdedb filesystem --owner www-cde storage populate

Finally we need a directory where logging files resist. The directory needs to be writable by the user running Apache (default www-cde). To create the default log directory, you can call:

python3 -m cdedb filesystem --owner www-cde log create

Running it

Last step before startup is compiling the GNU gettext .mo files for i18n:

make i18n-compile

Now, check if postgres and pgbouncer are running. Optionally you can run the test suite first to see whether everything is ready:

./bin/check.py

Now start the apache and access https://localhost/db/ with a browser.

Refreshing the running instance

Changes to the code can be propagate as follows to the current instance. For templates no action is necessary. For the python code updating the mtime of the wsgi file resets the apache workers:

sudo systemctl restart apache2

You can use the make target reload to re-compile i18n and trigger the worker reload:

make reload

For the database you should restart pgbouncer (which probably has some open connections left) before doing a python3 -m cdedb dev apply-sample-data.