clp-pangolin-sync

Single-shot sync between a CloudPanel SQLite database and the Pangolin WAF: it registers the sites (domains) that CloudPanel manages for this machine in Pangolin, and removes the ones that are gone.

Rewritten from scratch in Python; the .NET prototype lives in the e.soysal/clp-pangolin-sync repository and was used as reference only.

How it works

Every run (designed for cron, every 5–10 minutes):

1. reads the wanted domains: SELECT domain_name FROM site from the CloudPanel database (opened read-only);
2. reads the current state of the Pangolin org (sites, domains, public resources);
3. for domains in the DB but not in Pangolin: creates the base domain if needed (wildcard), creates the public resource, sets tlsServerName, and attaches the standard target (127.0.0.1:443);
4. for domains in Pangolin but not in the DB: deletes the resource — or, if the resource is shared with other sites, only detaches this machine's targets;
5. garbage-collects org domains that no resource references anymore — except domains the WAF itself manages (configManaged), which cannot be deleted via the API and are left in place.

The machine is identified by its site name in Pangolin (default: host name, overridable via PANGOLIN_SITE_NAME). A flock-based lock makes overlapping cron runs skip cleanly.

Install (target machine, Debian/Ubuntu)

curl -fsSL https://git.beyondedge.de/titeq/clp-pangolin-sync/releases/download/latest/setup.sh | bash
# including the cron entry:
curl -fsSL https://git.beyondedge.de/titeq/clp-pangolin-sync/releases/download/latest/setup.sh | bash -s -- --cron

The installer downloads the latest release into /opt/clp-pangolin-sync, creates a virtualenv (uv preferred, plain venv as fallback) and prepares pangolin.env (an existing one is preserved on upgrades). Options (pass after bash -s --): --cron (also installs the every-5-minutes cron entry into /etc/cron.d), --dir DIR, --repo URL.

Before the first release exists, fetch the installer from the repository:

curl -fsSL https://git.beyondedge.de/titeq/clp-pangolin-sync/raw/branch/main/setup.sh | bash

Manual setup

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
cp pangolin.env.example pangolin.env   # then fill in the real values

Configuration

All settings are environment variables; ./pangolin.env (KEY=VALUE, git-ignored) is loaded automatically if present. Real environment variables always take precedence.

Variable	Required	Default	Meaning
PANGOLIN_API_URL	yes	—	Base URL of the Pangolin WAF API
PANGOLIN_API_KEY	yes	—	Bearer key for the API
PANGOLIN_ORG	no	titeq	Pangolin organisation slug
PANGOLIN_DB_PATH	yes	—	Path to the CloudPanel SQLite database
PANGOLIN_SITE_NAME	no	host name	This machine's site name in Pangolin

Usage

.venv/bin/python -m clp_pangolin_sync             # sync
.venv/bin/python -m clp_pangolin_sync --dry-run   # show what would change

• INFO lines go to stdout, WARNING/ERROR to stderr (timestamped).
• Exit code 0: success, or skipped because a previous run is active.
• Exit code 1: configuration, database, API or sync failure — cron will mail the stderr output.

cron example

*/5 * * * * cd /opt/clp-pangolin-sync && .venv/bin/python -m clp_pangolin-sync >> /var/log/clp-pangolin-sync.log 2>&1

Releasing

Push a commit to main with a version marker in its message:

git commit --allow-empty -m "chore: bump version #patch"
git push

#patch, #minor or #major bumps the version from the latest v* tag (the Autorelease workflow), which creates the tag and the release. The tag push then triggers the Release workflow: it runs the test suite, builds the package and uploads clp-pangolin-sync.tar.gz and setup.sh as release assets.

This requires the RELEASE_TOKEN repository secret: a token of a regular user with write access (tokens of the Actions user cannot create tags that trigger the Release workflow).

Development

pip install -r requirements-dev.txt
python -m pytest

Tests mock the HTTP layer (responses) and the WAF client, so they never touch the production API.

Notes on the reference implementation

Differences from the .NET prototype (deliberate fixes):

• the API base URL comes from configuration everywhere (the prototype hard-coded one endpoint);
• the new resource ID is taken from the create response (with a by-name lookup fallback) instead of re-reading the whole list and taking the max ID;
• all targets of this machine are detached for shared resources (the prototype kept only the last one);
• no silent skip of the first domain during garbage collection;
• no Console.ReadKey() / interactive leftovers; the DB is opened read-only; failures go to stderr with exit code 1.