Articles › Troubleshooting

Patroni: pg_rewind “password authentication failed” and the PGPASSWORD trap

Published 27 September 2026 · 7 min read · Patroni 4.1, PostgreSQL 16

Your cluster passes every switchover test. Then the leader crashes for real, a replica takes over as it should, and the old leader never comes back: it sits in start failed. We hit this in our own test lab. This article covers the cause, a one-line check, and the fix.

The symptom

patronictl list shows the crashed node stuck:

| pg2    | 10.0.0.12:5432 | Replica | start failed |    |     unknown |     |    unknown |     |

PostgreSQL's log on that node repeats every few seconds:

FATAL:  requested timeline 3 is not a child of this server's history
DETAIL:  Latest checkpoint is at 0/8000028 on timeline 2, but in the history of the
         requested timeline, the server forked off from that timeline at 0/7008BA0.

That message only says the old leader wrote WAL the new leader never saw, so it can't follow the new timeline. Patroni is supposed to fix exactly this with pg_rewind. The Patroni log shows why it didn't:

INFO: running pg_rewind from pg3
INFO: running pg_rewind from dbname=postgres user=rewind_user host=10.0.0.13 port=5432 ...
INFO: pg_rewind exit code=1
INFO:  stderr=pg_rewind: error: connection to server at "10.0.0.13", port 5432 failed:
       FATAL:  password authentication failed for user "rewind_user"
ERROR: Failed to rewind from healthy primary: pg3
INFO: starting as a secondary

The cause: PGPASSWORD beats .pgpass

Patroni doesn't pass passwords on the command line. It writes them to a .pgpass file (the postgresql.pgpass setting) and points pg_rewind and pg_basebackup at it. But libpq, the library under every PostgreSQL client, looks for a password in this order:

1. a password in the connection string itself,
2. the PGPASSWORD environment variable,
3. the password file.

So if Patroni was started with PGPASSWORD in its environment, typically the superuser's password left over from someone's shell, pg_rewind sends that password as rewind_user. Authentication fails, and the rewind is skipped.

Why your tests didn't catch it

The variable only hurts the Patroni process that inherited it. Nodes started cleanly by systemd weeks ago keep replicating happily. The dangerous moment is the restart after an incident: the leader crashed, someone logs in, runs a few psql commands with PGPASSWORD exported, and starts Patroni from that same shell, or a recovery script does it for them. That process is the one that has to run pg_rewind, and it's the one carrying the wrong password.

It gets worse: Patroni points replication at the same .pgpass file (primary_conninfo uses passfile=…, not a password), so even after a manual rebuild, a node running with that environment can't stream from the leader either. Planned switchovers in a clean test environment never show any of this. In our lab, switchovers passed every time, while every crash test failed until we found the variable in the restart path.

How the variable gets there

• Starting Patroni by hand after an incident, from a shell where you exported PGPASSWORD to run psql, e.g. nohup patroni patroni.yml &.
• Wrapper and deploy scripts that source an environment file with set -a, which exports every variable in it, and then start services.
• systemd units with Environment=PGPASSWORD=… or an EnvironmentFile= shared with other tools.
• Container images that set PGPASSWORD for convenience.

Check in 10 seconds

Look at the environment of the running Patroni process on every node:

sudo tr '\0' '\n' < /proc/$(pgrep -f 'bin/patroni' | head -1)/environ | grep -E '^PG|^PATRONI_'

Any PGPASSWORD, PGUSER, PGHOST or PGSERVICE line is a problem. So is any unexpected PATRONI_* line: Patroni reads every PATRONI_-prefixed variable as configuration, and it overrides patroni.yml. That's a related trap we also hit, with a PATRONI_LOG_DIR from a script sending logs somewhere unexpected.

The fix

1. Start Patroni only through systemd, with a clean environment:

   [Service]
   User=postgres
   Environment=PATH=/opt/patroni/bin:/usr/local/bin:/usr/bin:/bin
   ExecStart=/opt/patroni/bin/patroni /etc/patroni/patroni.yml
   KillMode=process

2. Remove any PGPASSWORD from Environment= and EnvironmentFile= lines, then systemctl daemon-reload && systemctl restart patroni. PostgreSQL keeps running while Patroni restarts.
3. On the stuck node, restarting Patroni is usually enough: it retries pg_rewind with the right password. If the rewind can't work any more (the WAL it needs has been recycled), rebuild it: patronictl reinit <cluster> <node>.

Prove it's fixed

A switchover won't prove anything here; only a crash will. In a test environment, kill the leader's Patroni and PostgreSQL with SIGKILL while writing, wait for the new leader, restart the old node, and confirm it comes back as a streaming replica. That's exactly what our failover-drill --mode crash automates. After the fix, our lab's crashed leaders rejoined in about 34 seconds, every time.