DEPLOYMENT.md

Deployment

This document covers how code gets from a developer's machine to production, including the CI/CD pipeline, manual deployment, rollback procedures, and backups.

How deployment works

Every push to master that passes CI is automatically deployed to production via GitHub Actions. The pipeline:

1. Runs the full test suite against a PostGIS service container
2. Builds Tailwind CSS to verify frontend compilation
3. If tests pass and the push is to master, deploys to Dokku via git push

Dokku then:

1. Builds a Docker image using the multi-stage Dockerfile (CSS build + Python app)
2. Runs collectstatic (inside the Dockerfile)
3. Runs migrations and creates the cache table (from the app.json predeploy hook)
4. Starts the new container with gunicorn
5. Runs the health check (/health/) before routing traffic
6. Removes the old container

CI/CD pipeline

CI workflow (.github/workflows/ci.yml)

Runs on every push and pull request.

Test job:

• Spins up a PostGIS 17 service container
• Installs Python 3.14 + uv, Node.js 22
• Installs GIS system libraries (gdal, geos, proj)
• Builds Tailwind CSS (npm ci && npm run build:css)
• Runs the test suite (uvx nox -s tests)

Deploy job (master only, after tests pass):

• Checks out the full git history
• Configures SSH with the Dokku deploy key
• Pushes to the Dokku remote
• Syncs ops scripts (backup.sh, restore.sh) to the VPS via SCP
• Purges the Cloudflare cache

Docs workflow (.github/workflows/docs.yml)

Runs on pushes to master when docs or API files change. Exports the OpenAPI schema from a live Django instance, builds the documentation site with Zensical, and deploys to GitHub Pages at developers.bookcorners.org.

Required GitHub secrets

Secret	Description
DOKKU_SSH_PRIVATE_KEY	SSH private key for pushing to Dokku
DOKKU_HOST	VPS hostname (e.g., vps.bookcorners.org)

Manual deployment

When CI/CD is unavailable or you need to deploy a hotfix:

# From your local machine, in the book-corners directory
git push dokku master

The Dokku remote should already be configured:

# If not, add it:
git remote add dokku dokku@vps.bookcorners.org:book-corners

Monitoring the deploy

# Watch build output in real-time during git push

# After deploy, check status on the VPS:
sudo dokku ps:report book-corners
sudo dokku logs book-corners --tail

Post-deploy smoke checklist

After every deploy, verify these manually or review in monitoring:

• Homepage loads with styles: https://bookcorners.org/
• Static CSS returns 200: https://bookcorners.org/static/css/app.css
• Login page works: https://bookcorners.org/login/
• Map page loads with markers: https://bookcorners.org/map/
• Admin panel accessible: https://bookcorners.org/admin/
• Health check passes: https://bookcorners.org/health/

Library bulk export operations

The approved-library GeoJSON export is generated independently from application deployments. app.json schedules python manage.py generate_library_export for 02:00 server time every day; predeploy intentionally runs no export work.

Storage and retention

Artifacts are written under Django's persistent MEDIA_ROOT/library_exports/ directory. In production this maps to /var/lib/dokku/data/storage/book-corners/media/library_exports/ on the host, so deployments do not remove the active manifest or its files and the normal media backup includes them.

Each successful changed run publishes raw GeoJSON, gzip GeoJSON, and metadata files, then atomically replaces latest.json. A successful unchanged run only advances the manifest's checked_at. Cleanup retains the active version and seven previous complete versions. Retention is version-based; retained inactive versions are recovery material and are not exposed by download routes.

Manual generation and verification

Generate or refresh the export without deploying:

sudo dokku run book-corners python manage.py generate_library_export

The command reports published, unchanged, or an overlapping-run skip. Any generation failure exits non-zero, logs library_export_failed, and leaves the previous valid manifest active.

Fully validate the active manifest and checksum its three artifacts:

sudo dokku run book-corners python manage.py shell -c '
from pprint import pprint
from libraries.library_export import load_active_library_export_manifest
pprint(load_active_library_export_manifest(verify_checksums=True))
'

The result must be a manifest rather than None. Confirm that checked_at is about one daily interval old or newer, the record count is plausible, and generated_at advances when approved data or the schema changes. Investigate when no successful check occurs within 26 hours.

Privacy review and rollout

There is no permanent staging environment. Before first rollout or after changing the export allowlist, use a temporary staging deployment with synthetic or otherwise approved-for-review data:

1. Keep LIBRARY_EXPORT_DELIVERY_ENABLED=false so no website or API download is advertised.
2. Run the generator manually and inspect representative features, the schema definition, metadata, absolute photo URLs, and both compressed and raw checksums.
3. Confirm that only approved libraries are present and that account, ownership, moderation, staged changes, reports, audit data, and pending photos are absent.
4. Review the English and Italian privacy notices and the authenticated download page.
5. Repeat the manual generation and inspection in production, then enable delivery:

sudo dokku config:set book-corners LIBRARY_EXPORT_DELIVERY_ENABLED=true

The feature flag affects authenticated delivery only. Set it back to false to hide all export routes while investigating a privacy, licensing, or integrity problem; scheduled generation remains enabled.

Monitoring

The normal /health/ endpoint does not validate export freshness. Check the dedicated structured events after the daily schedule:

logcli query '{app="book-corners"} |= "library_export_"' --since 26h --limit 20
logcli query '{app="book-corners"} |= "library_export_failed"' --since 26h --limit 20

Expected healthy outcomes are library_export_published or library_export_unchanged. library_export_skipped_locked is harmless for one overlapping run, but repeated skips require checking for duplicate schedules or a long-running job. Alert on library_export_failed, no published/unchanged event in 26 hours, a manifest older than 26 hours, or an authenticated metadata request returning 503.

Failure recovery

Candidate files are validated before publication and latest.json is swapped atomically, so an interrupted or failed run keeps the last valid version available. To recover:

1. Disable delivery immediately if the active data has a privacy, licensing, or integrity problem.
2. Inspect library_export_failed and nearby application logs. Check free disk space, the persistent media mount and permissions, SITE_URL, database availability, and the scheduled command configuration.
3. Correct the data, configuration, storage, or application problem, then run generate_library_export manually.
4. Verify the manifest with checksums and review the authenticated download page before re-enabling delivery.
5. If persistent artifacts were lost or damaged, restore the media backup and verify it against the database backup from the same archive. Prefer generating a fresh export from the current verified database when possible.

Do not copy a retained historical file over the active filenames. Delivery trusts only the validated files named by latest.json, and a fresh successful run is the normal recovery path.

Contributor agreement rollout

Contributor agreement registration enforcement is deliberately separate from deploying the storage and API contract. The safe production default is:

sudo dokku config:set book-corners CONTRIBUTOR_AGREEMENT_REGISTRATION_REQUIRED=false

With the setting disabled, legacy credential and native social registration requests continue working. Valid current acceptance sent by updated clients is recorded, while missing, false, or stale acceptance creates an unaccepted account instead of rejecting registration.

Do not enable enforcement until the epic readiness checklist is complete: the web flow is deployed, the supported iOS release is available, old and new client payloads have been tested, monitoring is ready, and the minimum supported app-version policy is agreed. Enabling enforcement is a separate scheduled operational action:

sudo dokku config:set book-corners CONTRIBUTOR_AGREEMENT_REGISTRATION_REQUIRED=true

Immediately monitor credential registration, Apple/Google account creation, and validation errors. If existing clients fail or the readiness assumptions are wrong, restore compatibility without rolling back the database migration:

sudo dokku config:set book-corners CONTRIBUTOR_AGREEMENT_REGISTRATION_REQUIRED=false

Disabling enforcement does not delete or alter acceptance records already collected. Contribution enforcement, legacy holds, and outreach require their own independent default-off controls and must never be activated implicitly with registration enforcement.

Rollback

Revert to the previous release

If a deploy introduces a bug, revert the commit on master and push again:

git revert HEAD
git push origin master
# CI will run tests and auto-deploy the revert

For an immediate rollback without waiting for CI:

git revert HEAD
git push dokku master

Rebuild from the current deployed code

If the container is misbehaving but the code is correct:

sudo dokku ps:rebuild book-corners

Deploy a specific commit

# Push a specific commit to Dokku
git push dokku <commit-sha>:refs/heads/master

Database migrations

Migrations run automatically on every deploy via the app.json predeploy hook:

{
  "scripts": {
    "dokku": {
      "predeploy": "python manage.py migrate --noinput && python manage.py createcachetable --database default"
    }
  }
}

If a migration fails, the deploy is aborted and the previous container keeps running.

Running migrations manually

sudo dokku run book-corners python manage.py migrate

Checking migration status

sudo dokku run book-corners python manage.py showmigrations

Backups

Backups use Borg with BorgBase as the offsite target.

What gets backed up

• Database — Full PostgreSQL dump via dokku postgres:export
• Media files — User-uploaded photos from /var/lib/dokku/data/storage/book-corners/media/

Backup scripts

Operational scripts are versioned in the repo under scripts/:

• scripts/backup.sh — nightly backup (database dump + media archive to Borg)
• scripts/restore.sh — interactive restore with selective mode support

Both scripts read configuration from /home/deploy/.env.backup on the VPS. They must be run as the deploy user (not with sudo), because borg authenticates to BorgBase using the deploy user's SSH key.

Prerequisite: The deploy user needs passwordless sudo for dokku commands (used by the scripts for database export/import):

echo 'deploy ALL=(ALL) NOPASSWD: /usr/bin/dokku' | sudo tee /etc/sudoers.d/deploy-dokku

Backup schedule

Nightly at 3 AM server time via cron (under the deploy user):

0 3 * * * /home/deploy/backup.sh >> /var/log/book-corners-backup.log 2>&1

Restore procedures

Use the restore.sh script on the VPS:

# List available archives
/home/deploy/restore.sh --list

# Preview what would be restored (no changes)
/home/deploy/restore.sh --dry-run

# Restore a specific archive (both DB and media)
/home/deploy/restore.sh book-corners-2026-02-28T03:00:00

# Restore only the database
/home/deploy/restore.sh --db-only

# Restore only media files
/home/deploy/restore.sh --media-only

The script prompts for confirmation before any destructive operation and suggests creating a test database first.

Manual restore (without script)

If the restore script is unavailable:

export BORG_PASSPHRASE="<your-passphrase>"
export BORG_REPO="<your-borg-repo-url>"

# Extract the dump from a borg archive
mkdir -p /tmp/restore && cd /tmp/restore
borg extract "$BORG_REPO::<archive-name>" --pattern "*/db.dump"

# Restore to a test database first
sudo dokku postgres:create book-corners-db-test
sudo dokku postgres:import book-corners-db-test < $(find . -name db.dump)

# Verify the data
sudo dokku postgres:connect book-corners-db-test
# In psql: SELECT count(*) FROM libraries_library;

# If everything looks good, restore to production
sudo dokku postgres:import book-corners-db < $(find . -name db.dump)

# Clean up
sudo dokku postgres:destroy book-corners-db-test --force
rm -rf /tmp/restore

Updating ops scripts on the VPS

The deploy job in CI automatically syncs scripts/backup.sh and scripts/restore.sh to /home/deploy/ on the VPS after every successful deploy. No manual git pull or repo checkout is needed.

The sync step uses SCP (atomic copy to /tmp/ then mv into place) and runs with continue-on-error: true so a sync failure never blocks a deploy.

Prerequisite: The CI SSH key must be authorized for the deploy user:

cat ~/.ssh/dokku_deploy.pub | ssh deploy@vps.bookcorners.org 'cat >> ~/.ssh/authorized_keys'

Dockerfile overview

The production image is built in two stages:

Stage 1 — CSS builder (node:22-alpine):

• Installs npm dependencies
• Compiles Tailwind CSS to static/css/app.css

Stage 2 — Python app (python:3.14-slim):

• Installs system dependencies (GDAL, GEOS, Proj, libjpeg, zlib, gettext)
• Installs Python packages via uv
• Copies compiled CSS from stage 1
• Runs compilemessages (compiles .po translation files to .mo)
• Runs collectstatic
• Serves via gunicorn on $PORT

Health checks

The app exposes a /health/ endpoint that Dokku checks during deployment. If the health check fails after 3 attempts, the deploy is rolled back automatically.

Configuration in app.json:

{
  "healthchecks": {
    "web": [{
      "type": "startup",
      "name": "web check",
      "path": "/health/",
      "attempts": 3,
      "timeout": 5,
      "wait": 5
    }]
  }
}

Troubleshooting

Deploy fails during build

Check the build output for errors. Common causes:

• PostGIS extension not available — verify the database image
• collectstatic fails — check WhiteNoise and STATIC_ROOT configuration
• npm install fails — check package.json and package-lock.json are committed

Deploy fails during migration

# Check migration status
sudo dokku run book-corners python manage.py showmigrations

# View the full error
sudo dokku logs book-corners --tail

The old container keeps running if migration fails. Fix the migration and push again.

App is running but returning errors

# View recent logs
sudo dokku logs book-corners --tail

# Check the process status
sudo dokku ps:report book-corners

# Check environment variables
sudo dokku config:show book-corners

# Run a management command to debug
sudo dokku run book-corners python manage.py check --deploy

Static files not loading

# Verify collectstatic ran during build (check deploy output)
# WhiteNoise serves static files — no separate nginx config needed

# Check the static file URL
curl -I https://bookcorners.org/static/css/app.css

SSL certificate issues

# Check certificate status
sudo dokku letsencrypt:list

# Renew manually if needed
sudo dokku letsencrypt:enable book-corners

# Verify Cloudflare SSL mode is "Full (Strict)"