Free chapter · Excerpt from chapter 10 of 10

Troubleshooting: real failures and their fixes

This is the unedited opening of the kit's troubleshooting chapter. Each entry happened on our own test cluster while we built the kit. The full chapter has 14 entries and a support-bundle script; the full guide has 10 chapters.

Each entry: the symptom as you'll see it, the cause, the fix. Entries marked (seen in testing) happened on the kit's own lab while building it.

Old leader stays in start failed; pg_rewind: password authentication failed for user "rewind_user" (seen in testing)

Cause: Patroni was started from a shell or a unit with PGPASSWORD in its environment. libpq uses PGPASSWORD before the .pgpass file Patroni writes, so pg_rewind sends the wrong password, fails, and the old leader can't rejoin. Replication from that process uses the same .pgpass (primary_conninfo has passfile=), so it can't stream either. Nodes started cleanly are unaffected, which is why it hides: it bites the node restarted by hand after an incident.

Fix: never start Patroni from an interactive shell; use the systemd unit, which has a clean environment. Check a running Patroni: tr '\0' '\n' < /proc/$(pgrep -f bin/patroni | head -1)/environ | grep ^PG. Then restart Patroni on the failed node. bin/preflight warns about exported PG* variables.

Patroni ignores a setting in patroni.yml, e.g. logs to the wrong directory (seen in testing)

Cause: every environment variable named PATRONI_* is read as configuration and overrides the YAML (PATRONI_LOG_DIR, PATRONI_NAME, PATRONI_RESTAPI_LISTEN…). A deployment script that exports variables with that prefix leaks them into Patroni.

Fix: remove them from the service environment. The kit's own variables use the HA_ prefix for this reason; render-configs refuses PATRONI_* names in cluster.env.

Patroni loops on Failed to get list of machines … AttributeError("'int' object has no attribute 'get'") (seen in testing)

Cause: etcd 3.4 (e.g. Ubuntu 24.04's package). Its gRPC gateway doesn't serve the v3 JSON endpoints Patroni's etcd3 client calls.

Fix: etcd 3.5+ from the official release (chapter 2). bin/preflight fails on etcd < 3.5.

archive command failed repeatedly right after the cluster is created (seen in testing)

Cause: archiving starts at bootstrap; the pgBackRest stanza doesn't exist yet. Harmless for a few minutes, but WAL piles up in pg_wal until it's fixed.

Fix: pgbackrest --stanza=<cluster> stanza-create on the leader right after bootstrap (chapter 3, step 7). ha-check reports archiving as CRITICAL until then.

Replica: could not start WAL streaming: replication slot "pg2" does not exist (seen in testing)

Cause: normal during the first seconds of a new replica or after a failover: Patroni creates the slot on the leader on its next loop.

Fix: none if it clears within loop_wait × 2. If it persists: patronictl list for a paused cluster, and the leader's Patroni log for slot errors.