Always On: mastering the timeouts
Categories:
To avoid spurious failovers in an Always On environment, adjust these settings:
- Lease Timeout: raise it if your server goes through CPU load spikes.
- HealthCheck Timeout: raise it if your server is slow to answer diagnostic requests.
When you run AlwaysOn Availability Groups, it is not unusual to get spurious failovers while your system is working perfectly well.
It tends to happen at night, and more rarely during the day. The most common cause is a temporary unavailability of the system, which makes the cluster health checks time out. A typical example is a VM freeze while backup software takes a snapshot.
Since this has probably already happened to you, or is going to, here are the timeout values you can act on to prevent spurious failovers.
For an availability group to work, several components have to keep proving to each other that they are alive. If one of them takes too long to answer, the failover mechanism kicks in.
Communication between SQL Server and the cluster
The lease timeout
This is the direct communication mechanism between the SQL Server engine and the cluster service (RHS.exe). SQL Server is the one talking to the cluster service, and this communication only happens on the primary node of the availability group.
- Default value:
20000ms, 20 seconds. - How it works: if the SQL engine is too busy — under heavy CPU pressure, say — or if the thread stalls for more than 20 seconds, the lease is broken and the cluster forces the instance down to fail over.
The HealthCheck timeout
This one relies on the sp_server_diagnostics system stored procedure. Here the cluster service queries SQL Server to check its health. This check runs on every node of the cluster, primary and secondaries alike.
- Default value:
30000ms, 30 seconds. - How it works: SQL Server reports on the state of the system — memory, blocked queries, system errors. If SQL Server does not answer within 30 seconds, the cluster considers the instance unhealthy.
The failure condition level
This is the tolerance to errors.
- Default value:
3. - Level 3: SQL Server fails over on a critical system error or an internal spinlock condition.
The risk of unstable networks: the WSFC heartbeat
Beyond SQL Server’s own settings, WSFC (Windows Server Failover Cluster) has its own network settings.
If your nodes are spread across two geographical sites, in a multi-subnet configuration, network latency can cause a loss of quorum. The cluster believes the other node is dead simply because the internal ping, the heartbeat, took too long.
Recommended settings for multi-site
If you suffer quorum losses, check these parameters through PowerShell:
- SameSubnetThreshold: the number of missed pings allowed on the same network. Default: 5.
- CrossSubnetThreshold: the number of missed pings allowed between two sites. Default: 20.
Raising these values slightly lets the cluster ride out a brief network glitch without taking production down.
Why not simply set everything to the maximum?
If you set the LeaseTimeout to five minutes, your server will never fail over “for nothing”. But the day a real crash happens, your users will sit in front of a frozen screen for five minutes before high availability wakes up.
The golden rule: raise the timeouts only to cover identified load spikes or known network latency, never to hide an underlying performance problem.
How to change these settings
They live in the properties of the SQL Server role resource, in Failover Cluster Manager.
You can also change them in T-SQL, which is more precise:
ALTER AVAILABILITY GROUP [MyAG]
SET (HEALTH_CHECK_TIMEOUT = 60000); -- raised to 60 seconds
Or through PowerShell for the cluster:
(Get-Cluster).SameSubnetThreshold = 10
Finding your balance
A robust AlwaysOn configuration is a balance between responsiveness — failing over quickly on a crash — and tolerance — not failing over for a micro-latency. The default values, 20 s and 30 s, are excellent for 95 % of cases. If you have to change them, do it in steps of five or ten seconds and watch your logs.
Are your AlwaysOn failovers unexplained? Get in touch for a diagnostic, and let us stabilise your architecture.