[DRBD-user] I shall become SyncTarget, but I am primary!

Robert Altnoeder robert.altnoeder at linbit.com
Fri Dec 9 12:11:43 CET 2016

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Hello Tyler,

what was the sequence of events that led to this situation?

>From the symptoms described here, my guess would be:
- You had two nodes, A and B
- A was primary
- Someone ran a bad DB update on A, so you decided to restore a backup
- You shut down A (possibly also B), replaced A's data by what's in the
backup, started up A again
- A switched to the Primary role again, then attempted to create a DRBD
connection with B
- B still has the most recent data (with the bad DB update)

If this is what happened, then obviously from B's point of view, A's
DRBD looks like a volume that has some different version of the data,
which neither of the nodes has tracking information about (they don't
know how to update each other incrementally, so the only possible
solution is a full sync).

Since A is Primary, the data on A may be in use (e.g., a filesystem on
the device may be mounted), and it is therefore not a good idea to let a
resync change the data on A's disk, because then you could end up with
inconsistencies on A between what's on the disk and what's in the page
cache, thereby potentially breaking the filesystem's structure or
otherwise resulting in data loss. This is why DRBD refuses to start a
resync to a resource that is in the Primary role.

If A has the correct data now and is supposed to overwrite whatever is
on B, you would at least have to invalidate the data on B.

As you have potentially changed the disk state of A or B to
'Inconsistent' or 'Outdated' in the meantime, this might not work
anymore (DRBD does not resync from an Inconsistent or Oudated source).

Your best course of action to resolve this situation now would be:
(assuming that A has the correct data (the backup) and is supposed to
overwrite all data on B):
1. Shut down the DRBD resource on both nodes ('drbdadm down
<resource_name>')
    The resource must not be in use to be able to do this (e.g., not
mounted)
2. Create new metadata for this resource on both nodes ('drbdadm
create-md <resource_name>')
3. Start the DRBD resource on both nodes ('drbdadm up <resource_name>')
4. Make sure the resource comes up 'Connected' and 'Inconsistent' on
both nodes
5. Force A into 'UpToDate' and 'Primary', which will start a full sync to B
    (on A, enter 'drbdadm primary --force <resource_name>')

I hope that helps. In any case, do not discard the snapshot before you
have verified that the result is what you were expecting, just in case
that anything goes wrong (or had already gone wrong before you even
attempted this).

Best regards,
Robert

-- 
Robert Altnoeder
DRBD - Corosync - Pacemaker
+43 (1) 817 82 92 - 0 <tel:43181782920>
robert.altnoeder at linbit.com <mailto:robert.altnoeder at linbit.com>

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

On 12/05/2016 05:09 PM, Tyler Hains wrote:
> Hello,
>
> I had to restore a VM snapshot of a DRBD machine today to revert a catastrophic DB update. However, after restoring the snapshot, I can't seem to get the two nodes to connect. I have already followed the documented recommendations for split-brain recovery, and the results are shown below.
>
> This is my primary:
> [root at MCM5-DB4 ~]# cat /proc/drbd
> version: 8.3.16 (api:88/proto:86-97)
> GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil at Build64R6, 2014-11-24 14:51:37
>  0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r-----
>     ns:0 nr:0 dw:3136608 dr:9182265 al:31009 bm:15031 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:1191752
>
> And from /var/log/messages on the primary:
> Dec  5 10:20:07 localhost kernel: block drbd0: conn( StandAlone -> Unconnected )
> Dec  5 10:20:07 localhost kernel: block drbd0: Starting receiver thread (from drbd0_worker [1629])
> Dec  5 10:20:07 localhost kernel: block drbd0: receiver (re)started
> Dec  5 10:20:07 localhost kernel: block drbd0: conn( Unconnected -> WFConnection )
> Dec  5 10:20:07 localhost kernel: block drbd0: Handshake successful: Agreed network protocol version 97
> Dec  5 10:20:07 localhost kernel: block drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC
> Dec  5 10:20:07 localhost kernel: block drbd0: conn( WFConnection -> WFReportParams )
> Dec  5 10:20:07 localhost kernel: block drbd0: Starting asender thread (from drbd0_receiver [6450])
> Dec  5 10:20:07 localhost kernel: block drbd0: data-integrity-alg: <not-used>
> Dec  5 10:20:07 localhost kernel: block drbd0: drbd_sync_handshake:
> Dec  5 10:20:07 localhost kernel: block drbd0: self 5E31C2DC5B55B225:D72E026811DB74F3:20B3ACD5A61DC8E5:20B2ACD5A61DC8E5 bits:297938 flags:0
> Dec  5 10:20:07 localhost kernel: block drbd0: peer A98F08F32D2FCB34:0000000000000000:5E32C2DC5B55B224:5E31C2DC5B55B225 bits:183494167 flags:1
> Dec  5 10:20:07 localhost kernel: block drbd0: uuid_compare()=-2 by rule 60
> Dec  5 10:20:07 localhost kernel: block drbd0: I shall become SyncTarget, but I am primary!
> Dec  5 10:20:07 localhost kernel: block drbd0: conn( WFReportParams -> Disconnecting )
> Dec  5 10:20:07 localhost kernel: block drbd0: error receiving ReportState, l: 4!
> Dec  5 10:20:07 localhost kernel: block drbd0: asender terminated
> Dec  5 10:20:07 localhost kernel: block drbd0: Terminating drbd0_asender
> Dec  5 10:20:07 localhost kernel: block drbd0: Connection closed
> Dec  5 10:20:07 localhost kernel: block drbd0: conn( Disconnecting -> StandAlone )
> Dec  5 10:20:07 localhost kernel: block drbd0: receiver terminated
> Dec  5 10:20:07 localhost kernel: block drbd0: Terminating drbd0_receiver
>
> And this is my secondary:
> [root at MCM5-DB5 log]# cat /proc/drbd
> version: 8.3.16 (api:88/proto:86-97)
> GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil at Build64R6, 2014-11-24 14:51:37
>  0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----s
>     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:733976668
>
> With the log:
> Dec  5 10:20:07 localhost kernel: block drbd0: Handshake successful: Agreed network protocol version 97
> Dec  5 10:20:07 localhost kernel: block drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC
> Dec  5 10:20:07 localhost kernel: block drbd0: conn( WFConnection -> WFReportParams )
> Dec  5 10:20:07 localhost kernel: block drbd0: Starting asender thread (from drbd0_receiver [2171])
> Dec  5 10:20:07 localhost kernel: block drbd0: data-integrity-alg: <not-used>
> Dec  5 10:20:07 localhost kernel: block drbd0: drbd_sync_handshake:
> Dec  5 10:20:07 localhost kernel: block drbd0: self A98F08F32D2FCB34:0000000000000000:5E32C2DC5B55B224:5E31C2DC5B55B225 bits:183494167 flags:0
> Dec  5 10:20:07 localhost kernel: block drbd0: peer 5E31C2DC5B55B225:D72E026811DB74F3:20B3ACD5A61DC8E5:20B2ACD5A61DC8E5 bits:297938 flags:2
> Dec  5 10:20:07 localhost kernel: block drbd0: uuid_compare()=2 by rule 80
> Dec  5 10:20:07 localhost kernel: block drbd0: Writing the whole bitmap, full sync required after drbd_sync_handshake.
> Dec  5 10:20:07 localhost kernel: block drbd0: meta connection shut down by peer.
> Dec  5 10:20:07 localhost kernel: block drbd0: conn( WFReportParams -> NetworkFailure )
> Dec  5 10:20:07 localhost kernel: block drbd0: asender terminated
> Dec  5 10:20:07 localhost kernel: block drbd0: Terminating drbd0_asender
> Dec  5 10:20:08 localhost kernel: block drbd0: bitmap WRITE of 5600 pages took 183 jiffies
> Dec  5 10:20:08 localhost kernel: block drbd0: 700 GB (183494167 bits) marked out-of-sync by on disk bit-map.
> Dec  5 10:20:08 localhost kernel: block drbd0: error receiving ReportState, l: 4!
> Dec  5 10:20:08 localhost kernel: block drbd0: Connection closed
> Dec  5 10:20:08 localhost kernel: block drbd0: conn( NetworkFailure -> Unconnected )
> Dec  5 10:20:08 localhost kernel: block drbd0: receiver terminated
> Dec  5 10:20:08 localhost kernel: block drbd0: Restarting drbd0_receiver
> Dec  5 10:20:08 localhost kernel: block drbd0: receiver (re)started
> Dec  5 10:20:08 localhost kernel: block drbd0: conn( Unconnected -> WFConnection )
>
> Can anyone tell me how to get my Primary to connect and push its data over to the secondary?
>
> Thanks!
> Tyler Hains
>
>
>
>
>
> The information contained in this email and any attachments is private and is the confidential property of ROAM Data, Inc. If you are not the intended recipient(s) or have otherwise received this email in error, please delete this email and inform the sender as soon as possible. Neither this email nor the information contained in any attachments may be disclosed, stored, used, published or copied by anyone other than the intended recipient(s). All orders for ROAM Data, Inc. products and services are accepted by ROAM Data, Inc. subject to the terms and conditions of sale set forth on the ROAM Data, Inc. website, as such terms and conditions of sale may be changed from time to time without notice.
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20161209/dae3fc86/attachment.htm>

More information about the drbd-user mailing list