Articles › High availability

What happens to Patroni when etcd loses quorum (measured)

Published 8 October 2026 · 4 min read · Measured in a lab: Patroni 4.1.5, etcd 3.6.5, PostgreSQL 16, default timings

Short answer: with default settings, when etcd loses quorum the Patroni primary demotes itself and your application can no longer write, even though PostgreSQL is healthy and no node has failed. Here is what we saw, line by line, and the one setting that changes it.

The setup

Three etcd members (ports 2379, 2479, 2579) and three Patroni nodes (pg1, pg2, pg3) with pg1 as leader, asynchronous replication, all on one lab machine. A client inserted one row per second into pg1 and logged each result. We stopped etcd members with a plain kill. These are lab measurements from one run each, not production figures.

Lose one etcd member: nothing happens

We stopped one of three members. Quorum needs two, so etcd stayed up. Patroni kept logging no action. I am (pg1), the leader with the lock, the replicas kept streaming, and every insert succeeded. This is why a cluster can sit one failure away from trouble without anyone noticing: nothing in Patroni's output changes.

Lose a second member: the primary stops accepting writes

At 20:05:36 we stopped a second member, so one of three remained and quorum was gone. The last successful insert was at 20:05:54, 18 seconds later. Then Patroni logged:

ERROR: Error communicating with DCS
INFO: demoting self because DCS is not accessible and I was a leader
INFO: demoted self because DCS is not accessible and I was a leader

PostgreSQL restarted read-only on pg1, and from 20:05:55 every write failed with:

ERROR:  cannot execute INSERT in a read-only transaction

The two replicas did not take over. Electing a leader needs a write to etcd, and etcd could not accept one. So during the outage the cluster had no primary at all: three healthy PostgreSQL servers, all read-only. patronictl list failed with Etcd is not responding properly.

Restore quorum: writes come back

We started the stopped member at 20:06:54. Patroni logged received promote request, and the first insert succeeded at 20:06:55, one second later. The timeline moved from 1 to 2 because the node had been demoted and promoted again. Both replicas were streaming again with zero lag. In this run the outage lasted about 60 seconds, almost all of it the time we took to bring etcd back.

The setting that changes it: failsafe_mode

Patroni has a failsafe_mode option, off by default. We turned it on with:

patronictl edit-config -s failsafe_mode=true --force

and repeated the test, taking etcd down to no quorum at all. This time the primary stayed primary. For the 70 seconds we watched, every insert succeeded. The log shows how:

ERROR: Error communicating with DCS
INFO: Got response from pg3 http://127.0.0.1:8083/failsafe: Accepted
INFO: Got response from pg2 http://127.0.0.1:8082/failsafe: Accepted
INFO: continue to run as a leader because failsafe mode is enabled and all members are accessible

With failsafe mode the leader asks the other members over the Patroni REST API whether it may keep running. While all of them answer, it does. The message tells you the condition: "all members are accessible". We did not test what happens when a member is unreachable during the outage, or when the primary crashes while etcd is down. In both cases, expect no automatic failover until etcd is back, because there is nowhere to record a new leader.

What to do about it

• Keep etcd healthy. Three members survive one failure; the check that matters is how many more you can lose. ha-check checks etcd health and quorum, so a cluster that is one failure from losing quorum shows up before it happens.
• Decide deliberately about failsafe_mode. Off: a quorum loss costs you writes but never risks two primaries. On: writes continue, at the price of relying on the REST API between nodes. Test it in a lab before enabling it on a production cluster.
• Alert on etcd member health, not only on Patroni. In our run Patroni said nothing unusual after the first member went down.
• Rehearse it: stop two etcd members in a lab and watch what your application does.