[DRBD-user] pacemaker failover problem without allow-two-primaries

[Bryan K. Walton]

Hi,

I've building an HA cluster with two storage nodes.  The two nodes are
running DRBD 9 on CentOS 7. Storage1 is primary and storage2 is
secondary.

The two nodes do their DRBD replication over a bonded, directly cabled
connection. Upstream, both storage nodes are connected to two Brocade
ICX-7450 switches. Each node has a connection to both switches.

I'm using digimer's fence_dlink_snmp agent for fencing.  My desire
here is, obviously, to fence storage1, should it start having some
problems and for storage2 to promoted to primary and the filesystem
to be mounted on storage2. Obviously, I would want storage1 to be
fenced by having its switch ports disabled.

I'm testing my fencing with the following command:

pcs stonith fence storage1-drbd --off

When I test my fencing, things seem to be working, for the most part.
storage1's switch port DO get disabled.  However, pacemaker then fails
in its efforts to get storage2 promoted to secondary.  I see the
following in /var/log/messages on storage2:

ERROR: r0: Called drbdadm -c /etc/drbd.conf primary r0

When I run that command on the command line, I see:
r0: State change failed: (-1) Multiple primaries not allowed by config
Command 'drbdsetup primary r0' terminated with exit code 11

I do NOT have allow-two-primaries set in my config.  The man page seems
to suggest that I should only set this if I'm using OCFS2 or GFS, which
I'm not. My resource filesystem is just XFS.

I believe my root problem is that the replication link is still working,
after my fence agent runs.  So, how do I resolve this?

Thanks!
Bryan Walton