Manual Setup
This describes the steps necessary to get the project running on a machine.
Note
This is somewhat complicated and it is advised to use the virtual machine in most cases. Instructions for obtaining and using the image can be found at Setup the VM.
Below may be some Gentoo-specific bits, for
Debian-specific stuff look at the setup scripts in related/auto-build
.
Prerequisites
We need some dependencies:
python (at least 3.9)
PostgreSQL (at least 13, for drop with force)
Apache (with mod_wsgi)
git
texlive (incl. luatex; for generating pdf documents)
Further we depend on a number of python packages:
passlib
psycopg2 (at least 2.5.4, for jsonb support)
werkzeug (at least 0.15, for correct multi-params in urls)
dateutil (only needed for dev script, namely analyze_timing.py)
babel
docutils
jinja2
markdown
bleach
python-magic
python-imaging-library (more specifically pillow)
zxcvbn
icu
mailmanclient (at least 3.3.3, for unsubscription handling)
At last there are some recommended dependencies:
sphinx (for building the documentation)
guzzle-sphinx-theme (a documentation theme)
webtest (for tests, at least 2.0.17 for handling of multiple elements with the same name)
lxml (the python module; used in the test suite)
pgbouncer (otherwise database performance may be degraded)
fail2ban (for preventing brute-force attacks)
pylint (for code analysis)
requests (the python module; used in some scripts)
Checkout the repository
Use git to clone the code:
git clone ssh://gitea@tracker.cde-ev.de:20009/cdedb/cdedb2.git
For this to work you need to create an account on the tracker website which
has then to be granted access to the cdedb repository (send a mail to the db
list cdedb@lists.cde-ev.de
). All commands below assume, that you are in
the root directory of the repository. Now you can build the documentation by
issuing:
make doc
Configure the application
First of all, you need to create your personal configuration and make the path to
the file available via environment variable, like described in
Environment Setup. The configuration file may be empty if you do not want
to override the defaults from cdedb.config
.
A sample configuration for development instances can be found in
related/auto-build/files/stage3/localconfig.py
.
Prepare environment
Now we set up all auxiliary stuff. We assume, that postgres is configured
for peer authentication (i.e. the system user xy has access to the postgres
user xy). First execute as user with enough permissions (that is the ability
to run sudo -u postgres ...
and sudo -u cdb ...
) and with running
postgres:
python3 -m cdedb db create-users
python3 -m cdedb db create
The first one will create the database users, the second one the actual tables. To seed them with sample data, run additionally:
python3 -m cdedb db populate
Now configure pgbouncer in pgbouncer.ini
(in /etc
) with the following:
[databases]
cdb =
cdb_test =
cdb_test_1 =
cdb_test_2 =
cdb_test_3 =
cdb_test_4 =
cdb_test_ldap =
cdb_test_xss =
[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
unix_socket_dir = /run/postgresql
listen_addr = 127.0.0.1
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer_users.txt
pool_mode = session
server_reset_query = DISCARD ALL
max_client_conn = 100
default_pool_size = 20
Additionally place copy file related/pgbouncer_users.txt
to
/etc/pgbouncer_users.txt
for authentication (otherwise pgbouncer will
refuse connections):
cp related/pgbouncer_users.txt /etc
chown pgbouncer:root /etc/pgbouncer_users.txt
chmod 600 /etc/pgbouncer_users.txt
This file may be regenerated with the mkauth.py
tool from the pgbouncer
tar-ball.
Now we set up the Apache server, first add the following lines to
/etc/apache2/httpd.conf
:
LoadModule wsgi_module modules/mod_wsgi.so
ServerName localhost
and then insert the following close to the end of
/etc/apache2/vhosts.d/00_default_ssl_vhost.conf
:
WSGIDaemonProcess cdedb processes=4 threads=4
WSGIScriptAlias /db /path/to/repo/wsgi/cdedb.wsgi
<Directory /path/to/repo/wsgi>
Require all granted
</Directory>
Alias /static /path/to/repo/static
<Directory /path/to/repo/static/static>
Require all granted
</Directory>
note, that this is syntax for apache-2.4 (which differs from apache-2.2).
Next we need to create the directory for uploaded data (where
www-cde
is the user running Apache):
python3 -m cdedb filesystem --owner www-cde storage create
To populate the storage with sample data, run additionally:
python3 -m cdedb filesystem --owner www-cde storage populate
Finally we need a directory where logging files resist. The directory
needs to be writable by the user running Apache (default www-cde
). To
create the default log directory, you can call:
python3 -m cdedb filesystem --owner www-cde log create
Running it
Last step before startup is compiling the GNU gettext .mo files for i18n:
make i18n-compile
Now, check if postgres and pgbouncer are running. Optionally you can run the test suite first to see whether everything is ready:
./bin/check.py
Now start the apache and access https://localhost/db/
with a
browser.
Refreshing the running instance
Changes to the code can be propagate as follows to the current instance. For templates no action is necessary. For the python code updating the mtime of the wsgi file resets the apache workers:
sudo systemctl restart apache2
You can use the make target reload to re-compile i18n and trigger the worker reload:
make reload
For the database you should restart pgbouncer (which probably has some open
connections left) before doing a python3 -m cdedb dev apply-sample-data
.