One writer, checked on every connection. HAProxy in front of Sentinel

Here’s a pretty common setup for Redis or Valkey. One primary, one replica, a few Sentinels deciding who’s in charge, and HAProxy as the one address every client talks to. The interesting part is the health check, because clients should always reach the one node that is allowed to write, including during and right after a failover.

The usual setup sends ROLE (or INFO replication) and routes to whichever node answers master. That works in steady state and fails in exactly the minutes that matter.

After Sentinel promotes the replica, the old primary doesn’t know yet. Until Sentinel reaches it and turns it into a replica, it keeps answering master. If HAProxy can still reach it, both nodes pass the check.

With balance first, HAProxy keeps sending clients to the first healthy server in its list. If that’s the old primary, clients happily write to a node that is about to resync and throw those writes away.

The fix is to ask for two answers before a node counts as healthy.

1. Sentinel names this node as the primary, and
2. the node itself answers ROLE with master.

HAProxy’s tcp-check can hold a short conversation, so both questions fit into one check.

backend primary
    balance first
    option tcp-check
    tcp-check connect port 26379
    tcp-check send "AUTH healthcheck {{ password }}\r\n"
    tcp-check expect string +OK
    tcp-check send "SENTINEL get-master-addr-by-name {{ name }}\r\n"
    tcp-check expect string-lf "\r\n%[srv_name]\r\n"
    tcp-check connect port 6379
    tcp-check send "AUTH healthcheck {{ password }}\r\n"
    tcp-check expect string +OK
    tcp-check send "ROLE\r\n"
    tcp-check expect string master
    default-server inter 1s fall 2 rise 3 on-marked-down shutdown-sessions

Two small details do most of the heavy lifting.

• The server name is the address Sentinel reports. %[srv_name] makes each server check that Sentinel names it, not just that Sentinel names someone.
• on-marked-down shutdown-sessions. When a node fails the check, its existing connections are closed instead of left to drain, so clients reconnect to the new primary.

Give the health check its own user that can do nothing except what the check needs.

An unplanned failover always has a window. A planned one doesn’t have to. Pause writes on the primary first, then ask Sentinel to fail over.

valkey-cli CLIENT PAUSE 15000 WRITE
valkey-cli -p 26379 SENTINEL FAILOVER <name>

CLIENT PAUSE … WRITE holds writes without rejecting them, so the old primary acknowledges nothing after the pause starts. The replica catches up, gets promoted, and HAProxy moves clients across. A client that retries unacknowledged writes on reconnect loses nothing.

Check whether your health check only asks the node about itself, and what happens to open connections when a node is marked down. If you automate the cluster, make sure the automation can tell real drift from a legitimate failover. Otherwise it may happily “fix” things back to the old primary.