<font size=2 face="sans-serif">Hi all,</font>
<br>
<br><font size=2 face="sans-serif">I'm sure there is a simple answer to
this one..</font>
<br>
<br><font size=2 face="sans-serif">2 Node cluster with heartbeat.</font>
<br><font size=2 face="sans-serif">Migrated resources to the secondary
node, (with primary online still) all migrated fine.</font>
<br>
<br><font size=2 face="sans-serif">As a test pulled the plug from the secondary
node, resources migrated back to primary, no issues at all.</font>
<br><font size=2 face="sans-serif">Turned the power back on, on the secondary
and everything I thought came up ok ,but checking I now have the primary
is saying the secondary is inconsistent but the secondary is saying it's
UpToDate?</font>
<br>
<br><font size=2 face="sans-serif">How do I fix this?</font>
<br>
<br><font size=2 face="sans-serif">storage00 = primary node</font>
<br><font size=2 face="sans-serif">storage01 = secondary node</font>
<br>
<br><font size=2 face="sans-serif">root@storage00:/var/log# /etc/init.d/drbd
status</font>
<br><font size=2 face="sans-serif">drbd driver loaded OK; device status:</font>
<br><font size=2 face="sans-serif">version: 8.3.7 (api:88/proto:86-91)</font>
<br><font size=2 face="sans-serif">GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917
build by root@storage00, 2010-07-20 12:47:09</font>
<br><font size=2 face="sans-serif">m:res cs
ro ds
p mounted
fstype</font>
<br><font size=2 face="sans-serif">0:meta Connected Primary/Secondary
UpToDate/UpToDate C /meta ext3</font>
<br><font size=2 face="sans-serif">1:data Connected Primary/Secondary
UpToDate/Inconsistent C</font>
<br>
<br><font size=2 face="sans-serif">root@storage00:/var/log# cat /proc/drbd</font>
<br><font size=2 face="sans-serif">version: 8.3.7 (api:88/proto:86-91)</font>
<br><font size=2 face="sans-serif">GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917
build by root@storage00, 2010-07-20 12:47:09</font>
<br><font size=2 face="sans-serif"> 0: cs:Connected ro:Primary/Secondary
ds:UpToDate/UpToDate C r----</font>
<br><font size=2 face="sans-serif"> ns:8580 nr:12 dw:400 dr:9872
al:7 bm:13 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0</font>
<br><font size=2 face="sans-serif"> 1: cs:Connected ro:Primary/Secondary
ds:UpToDate/Inconsistent C r----</font>
<br><font size=2 face="sans-serif"> ns:0 nr:0 dw:0 dr:2860
al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0</font>
<br>
<br><font size=2 face="sans-serif">root@storage00:~# dmesg</font>
<br><font size=2 face="sans-serif">[ 8406.964516] block drbd0: PingAck
did not arrive in time.</font>
<br><font size=2 face="sans-serif">[ 8406.965307] block drbd0: peer( Primary
-> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate ->
DUnknown )</font>
<br><font size=2 face="sans-serif">[ 8406.965417] block drbd0: short read
expecting header on sock: r=-512</font>
<br><font size=2 face="sans-serif">[ 8406.965423] block drbd0: asender
terminated</font>
<br><font size=2 face="sans-serif">[ 8406.965427] block drbd0: Terminating
asender thread</font>
<br><font size=2 face="sans-serif">[ 8406.967232] block drbd0: Connection
closed</font>
<br><font size=2 face="sans-serif">[ 8406.967238] block drbd0: conn( NetworkFailure
-> Unconnected )</font>
<br><font size=2 face="sans-serif">[ 8406.967482] block drbd0: receiver
terminated</font>
<br><font size=2 face="sans-serif">[ 8406.967515] block drbd0: Restarting
receiver thread</font>
<br><font size=2 face="sans-serif">[ 8406.967520] block drbd0: receiver
(re)started</font>
<br><font size=2 face="sans-serif">[ 8406.967524] block drbd0: conn( Unconnected
-> WFConnection )</font>
<br><font size=2 face="sans-serif">[ 8407.004451] block drbd1: PingAck
did not arrive in time.</font>
<br><font size=2 face="sans-serif">[ 8407.005244] block drbd1: peer( Primary
-> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate ->
DUnknown )</font>
<br><font size=2 face="sans-serif">[ 8407.005326] block drbd1: asender
terminated</font>
<br><font size=2 face="sans-serif">[ 8407.005329] block drbd1: Terminating
asender thread</font>
<br><font size=2 face="sans-serif">[ 8407.005388] block drbd1: short read
expecting header on sock: r=-512</font>
<br><font size=2 face="sans-serif">[ 8407.007276] block drbd1: Connection
closed</font>
<br><font size=2 face="sans-serif">[ 8407.007280] block drbd1: conn( NetworkFailure
-> Unconnected )</font>
<br><font size=2 face="sans-serif">[ 8407.007288] block drbd1: receiver
terminated</font>
<br><font size=2 face="sans-serif">[ 8407.007289] block drbd1: Restarting
receiver thread</font>
<br><font size=2 face="sans-serif">[ 8407.007291] block drbd1: receiver
(re)started</font>
<br><font size=2 face="sans-serif">[ 8407.007294] block drbd1: conn( Unconnected
-> WFConnection )</font>
<br><font size=2 face="sans-serif">[ 8408.897428] block drbd0: role( Secondary
-> Primary )</font>
<br><font size=2 face="sans-serif">[ 8408.898538] block drbd0: Creating
new current UUID</font>
<br><font size=2 face="sans-serif">[ 8409.017511] kjournald starting. Commit
interval 5 seconds</font>
<br><font size=2 face="sans-serif">[ 8409.018224] EXT3 FS on drbd0, internal
journal</font>
<br><font size=2 face="sans-serif">[ 8409.018227] EXT3-fs: recovery complete.</font>
<br><font size=2 face="sans-serif">[ 8409.018230] EXT3-fs: mounted filesystem
with ordered data mode.</font>
<br><font size=2 face="sans-serif">[ 8409.055569] block drbd1: role( Secondary
-> Primary )</font>
<br><font size=2 face="sans-serif">[ 8409.056602] block drbd1: Creating
new current UUID</font>
<br><font size=2 face="sans-serif">[ 8409.667204] iSCSI Enterprise Target
Software - version 1.4.19</font>
<br><font size=2 face="sans-serif">[ 8409.667321] iscsi_trgt: Registered
io type fileio</font>
<br><font size=2 face="sans-serif">[ 8409.667322] iscsi_trgt: Registered
io type blockio</font>
<br><font size=2 face="sans-serif">[ 8409.667324] iscsi_trgt: Registered
io type nullio</font>
<br><font size=2 face="sans-serif">[ 8512.289805] block drbd0: Handshake
successful: Agreed network protocol version 91</font>
<br><font size=2 face="sans-serif">[ 8512.289816] block drbd0: conn( WFConnection
-> WFReportParams )</font>
<br><font size=2 face="sans-serif">[ 8512.289898] block drbd1: Handshake
successful: Agreed network protocol version 91</font>
<br><font size=2 face="sans-serif">[ 8512.289902] block drbd1: conn( WFConnection
-> WFReportParams )</font>
<br><font size=2 face="sans-serif">[ 8512.289937] block drbd0: Starting
asender thread (from drbd0_receiver [948])</font>
<br><font size=2 face="sans-serif">[ 8512.290780] block drbd1: Starting
asender thread (from drbd1_receiver [951])</font>
<br><font size=2 face="sans-serif">[ 8512.291528] block drbd1: data-integrity-alg:
<not-used></font>
<br><font size=2 face="sans-serif">[ 8512.291552] block drbd1: drbd_sync_handshake:</font>
<br><font size=2 face="sans-serif">[ 8512.291555] block drbd1: self 51ACB128365276DD:03BD4338E0A8C438:19942D6D67C68490:55E6E96AB00ACDBB
bits:0 flags:0</font>
<br><font size=2 face="sans-serif">[ 8512.291558] block drbd1: peer 03BD4338E0A8C438:0000000000000000:19942D6D67C68490:55E6E96AB00ACDBB
bits:0 flags:2</font>
<br><font size=2 face="sans-serif">[ 8512.291561] block drbd1: uuid_compare()=1
by rule 70</font>
<br><font size=2 face="sans-serif">[ 8512.291565] block drbd1: peer( Unknown
-> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown
-> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 8512.302672] block drbd0: data-integrity-alg:
<not-used></font>
<br><font size=2 face="sans-serif">[ 8512.302722] block drbd0: drbd_sync_handshake:</font>
<br><font size=2 face="sans-serif">[ 8512.302726] block drbd0: self 591E74B9107F0F85:BF8A5A7F3C4BFEE4:AE4808A77E5CD3C2:036AAA560A3BB0FF
bits:1 flags:0</font>
<br><font size=2 face="sans-serif">[ 8512.302730] block drbd0: peer BF8A5A7F3C4BFEE4:0000000000000000:AE4808A77E5CD3C2:036AAA560A3BB0FF
bits:2048 flags:2</font>
<br><font size=2 face="sans-serif">[ 8512.302733] block drbd0: uuid_compare()=1
by rule 70</font>
<br><font size=2 face="sans-serif">[ 8512.302740] block drbd0: peer( Unknown
-> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown
-> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 8512.317339] block drbd0: conn( WFBitMapS
-> SyncSource ) pdsk( UpToDate -> Inconsistent )</font>
<br><font size=2 face="sans-serif">[ 8512.317379] block drbd1: aftr_isp(
0 -> 1 )</font>
<br><font size=2 face="sans-serif">[ 8512.317383] block drbd0: Began resync
as SyncSource (will sync 8192 KB [2048 bits set]).</font>
<br><font size=2 face="sans-serif">[ 8512.325110] block drbd1: peer_isp(
0 -> 1 )</font>
<br><font size=2 face="sans-serif">[ 8512.528583] block drbd1: conn( WFBitMapS
-> PausedSyncS ) pdsk( UpToDate -> Inconsistent )</font>
<br><font size=2 face="sans-serif">[ 8512.528680] block drbd1: Began resync
as PausedSyncS (will sync 0 KB [0 bits set]).</font>
<br><font size=2 face="sans-serif">[ 8512.615174] block drbd0: Resync done
(total 1 sec; paused 0 sec; 8192 K/sec)</font>
<br><font size=2 face="sans-serif">[ 8512.615183] block drbd0: conn( SyncSource
-> Connected ) pdsk( Inconsistent -> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 8512.615192] block drbd1: aftr_isp(
1 -> 0 )</font>
<br><font size=2 face="sans-serif">[ 8513.078766] block drbd1: Resync done
(total 1 sec; paused 0 sec; 0 K/sec)</font>
<br><font size=2 face="sans-serif">[ 8513.078777] block drbd1: conn( PausedSyncS
-> Connected ) pdsk( Inconsistent -> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 8513.078819] block drbd1: pdsk( UpToDate
-> Inconsistent ) peer_isp( 1 -> 0 )</font>
<br>
<br><font size=2 face="sans-serif">root@storage01:~# /etc/init.d/drbd status</font>
<br><font size=2 face="sans-serif">drbd driver loaded OK; device status:</font>
<br><font size=2 face="sans-serif">version: 8.3.7 (api:88/proto:86-91)</font>
<br><font size=2 face="sans-serif">GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917
build by root@storage01, 2010-07-20 15:13:41</font>
<br><font size=2 face="sans-serif">m:res cs
ro ds
p mounted fstype</font>
<br><font size=2 face="sans-serif">0:meta Connected Secondary/Primary
UpToDate/UpToDate C</font>
<br><font size=2 face="sans-serif">1:data Connected Secondary/Primary
UpToDate/UpToDate C</font>
<br>
<br><font size=2 face="sans-serif">root@storage01:~# cat /proc/drbd</font>
<br><font size=2 face="sans-serif">version: 8.3.7 (api:88/proto:86-91)</font>
<br><font size=2 face="sans-serif">GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917
build by root@storage01, 2010-07-20 15:13:41</font>
<br><font size=2 face="sans-serif"> 0: cs:Connected ro:Secondary/Primary
ds:UpToDate/UpToDate C r----</font>
<br><font size=2 face="sans-serif"> ns:0 nr:8192 dw:8192 dr:0
al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0</font>
<br><font size=2 face="sans-serif"> 1: cs:Connected ro:Secondary/Primary
ds:UpToDate/UpToDate C r----</font>
<br><font size=2 face="sans-serif"> ns:0 nr:0 dw:0 dr:0 al:0
bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0</font>
<br>
<br><font size=2 face="sans-serif">root@storage01:~# dmesg</font>
<br><font size=2 face="sans-serif">[ 16.336431] block drbd0: Starting
worker thread (from cqueue [1190])</font>
<br><font size=2 face="sans-serif">[ 16.336585] block drbd0: disk(
Diskless -> Attaching )</font>
<br><font size=2 face="sans-serif">[ 16.340633] block drbd0: Found
4 transactions (4 active extents) in activity log.</font>
<br><font size=2 face="sans-serif">[ 16.340637] block drbd0: Method
to ensure write ordering: barrier</font>
<br><font size=2 face="sans-serif">[ 16.340641] block drbd0: max_segment_size
( = BIO size ) = 32768</font>
<br><font size=2 face="sans-serif">[ 16.340646] block drbd0: drbd_bm_resize
called with capacity == 1900408</font>
<br><font size=2 face="sans-serif">[ 16.340653] block drbd0: resync
bitmap: bits=237551 words=3712</font>
<br><font size=2 face="sans-serif">[ 16.340655] block drbd0: size
= 928 MB (950204 KB)</font>
<br><font size=2 face="sans-serif">[ 16.341756] block drbd0: recounting
of set bits took additional 0 jiffies</font>
<br><font size=2 face="sans-serif">[ 16.341760] block drbd0: 0 KB
(0 bits) marked out-of-sync by on disk bit-map.</font>
<br><font size=2 face="sans-serif">[ 16.341767] block drbd0: Marked
additional 8192 KB as out-of-sync based on AL.</font>
<br><font size=2 face="sans-serif">[ 16.373949] block drbd0: disk(
Attaching -> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 16.378189] block drbd1: Starting
worker thread (from cqueue [1190])</font>
<br><font size=2 face="sans-serif">[ 16.378349] block drbd1: disk(
Diskless -> Attaching )</font>
<br><font size=2 face="sans-serif">[ 16.395248] block drbd1: No
usable activity log found.</font>
<br><font size=2 face="sans-serif">[ 16.395254] block drbd1: Method
to ensure write ordering: barrier</font>
<br><font size=2 face="sans-serif">[ 16.395259] block drbd1: max_segment_size
( = BIO size ) = 32768</font>
<br><font size=2 face="sans-serif">[ 16.395264] block drbd1: drbd_bm_resize
called with capacity == 83879352</font>
<br><font size=2 face="sans-serif">[ 16.399814] block drbd1: resync
bitmap: bits=10484919 words=163827</font>
<br><font size=2 face="sans-serif">[ 16.399819] block drbd1: size
= 40 GB (41939676 KB)</font>
<br><font size=2 face="sans-serif">[ 16.434454] block drbd1: recounting
of set bits took additional 0 jiffies</font>
<br><font size=2 face="sans-serif">[ 16.434459] block drbd1: 0 KB
(0 bits) marked out-of-sync by on disk bit-map.</font>
<br><font size=2 face="sans-serif">[ 16.434466] block drbd1: Marked
additional 0 KB as out-of-sync based on AL.</font>
<br><font size=2 face="sans-serif">[ 16.434492] block drbd1: disk(
Attaching -> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 16.467318] block drbd0: conn(
StandAlone -> Unconnected )</font>
<br><font size=2 face="sans-serif">[ 16.468011] block drbd0: Starting
receiver thread (from drbd0_worker [1193])</font>
<br><font size=2 face="sans-serif">[ 16.469226] block drbd1: conn(
StandAlone -> Unconnected )</font>
<br><font size=2 face="sans-serif">[ 16.469585] block drbd1: Starting
receiver thread (from drbd1_worker [1199])</font>
<br><font size=2 face="sans-serif">[ 16.471272] block drbd1: receiver
(re)started</font>
<br><font size=2 face="sans-serif">[ 16.471279] block drbd1: conn(
Unconnected -> WFConnection )</font>
<br><font size=2 face="sans-serif">[ 16.472862] block drbd0: receiver
(re)started</font>
<br><font size=2 face="sans-serif">[ 16.472870] block drbd0: conn(
Unconnected -> WFConnection )</font>
<br><font size=2 face="sans-serif">[ 16.573970] block drbd0: Handshake
successful: Agreed network protocol version 91</font>
<br><font size=2 face="sans-serif">[ 16.573983] block drbd0: conn(
WFConnection -> WFReportParams )</font>
<br><font size=2 face="sans-serif">[ 16.574064] block drbd1: Handshake
successful: Agreed network protocol version 91</font>
<br><font size=2 face="sans-serif">[ 16.574070] block drbd1: conn(
WFConnection -> WFReportParams )</font>
<br><font size=2 face="sans-serif">[ 16.574174] block drbd0: Starting
asender thread (from drbd0_receiver [1220])</font>
<br><font size=2 face="sans-serif">[ 16.574276] block drbd1: Starting
asender thread (from drbd1_receiver [1222])</font>
<br><font size=2 face="sans-serif">[ 16.575190] block drbd1: data-integrity-alg:
<not-used></font>
<br><font size=2 face="sans-serif">[ 16.575966] block drbd1: drbd_sync_handshake:</font>
<br><font size=2 face="sans-serif">[ 16.575971] block drbd1: self
03BD4338E0A8C438:0000000000000000:19942D6D67C68490:55E6E96AB00ACDBB bits:0
flags:0</font>
<br><font size=2 face="sans-serif">[ 16.575975] block drbd1: peer
51ACB128365276DD:03BD4338E0A8C438:19942D6D67C68490:55E6E96AB00ACDBB bits:0
flags:0</font>
<br><font size=2 face="sans-serif">[ 16.575978] block drbd1: uuid_compare()=-1
by rule 50</font>
<br><font size=2 face="sans-serif">[ 16.575986] block drbd1: peer(
Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown
-> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 16.589030] block drbd0: data-integrity-alg:
<not-used></font>
<br><font size=2 face="sans-serif">[ 16.590543] block drbd0: drbd_sync_handshake:</font>
<br><font size=2 face="sans-serif">[ 16.590547] block drbd0: self
BF8A5A7F3C4BFEE4:0000000000000000:AE4808A77E5CD3C2:036AAA560A3BB0FF bits:2048
flags:0</font>
<br><font size=2 face="sans-serif">[ 16.590551] block drbd0: peer
591E74B9107F0F85:BF8A5A7F3C4BFEE4:AE4808A77E5CD3C2:036AAA560A3BB0FF bits:1
flags:0</font>
<br><font size=2 face="sans-serif">[ 16.590554] block drbd0: uuid_compare()=-1
by rule 50</font>
<br><font size=2 face="sans-serif">[ 16.590561] block drbd0: peer(
Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown
-> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 16.595192] block drbd0: conn(
WFBitMapT -> WFSyncUUID )</font>
<br><font size=2 face="sans-serif">[ 16.601669] block drbd0: helper
command: /sbin/drbdadm before-resync-target minor-0</font>
<br><font size=2 face="sans-serif">[ 16.605313] block drbd0: helper
command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)</font>
<br><font size=2 face="sans-serif">[ 16.605323] block drbd0: conn(
WFSyncUUID -> SyncTarget ) disk( UpToDate -> Inconsistent )</font>
<br><font size=2 face="sans-serif">[ 16.605336] block drbd1: aftr_isp(
0 -> 1 )</font>
<br><font size=2 face="sans-serif">[ 16.605342] block drbd0: Began
resync as SyncTarget (will sync 8192 KB [2048 bits set]).</font>
<br><font size=2 face="sans-serif">[ 16.730537] block drbd1: conn(
WFBitMapT -> WFSyncUUID )</font>
<br><font size=2 face="sans-serif">[ 16.730577] block drbd1: peer_isp(
0 -> 1 )</font>
<br><font size=2 face="sans-serif">[ 16.812628] block drbd1: helper
command: /sbin/drbdadm before-resync-target minor-1</font>
<br><font size=2 face="sans-serif">[ 16.817954] block drbd1: helper
command: /sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)</font>
<br><font size=2 face="sans-serif">[ 16.817964] block drbd1: conn(
WFSyncUUID -> PausedSyncT ) disk( UpToDate -> Inconsistent )</font>
<br><font size=2 face="sans-serif">[ 16.817977] block drbd1: Began
resync as PausedSyncT (will sync 0 KB [0 bits set]).</font>
<br><font size=2 face="sans-serif">[ 16.906724] block drbd0: Resync
done (total 1 sec; paused 0 sec; 8192 K/sec)</font>
<br><font size=2 face="sans-serif">[ 16.906734] block drbd0: conn(
SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 16.906741] block drbd0: helper
command: /sbin/drbdadm after-resync-target minor-0</font>
<br><font size=2 face="sans-serif">[ 16.909092] block drbd0: helper
command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)</font>
<br><font size=2 face="sans-serif">[ 16.909102] block drbd1: aftr_isp(
1 -> 0 )</font>
<br><font size=2 face="sans-serif">[ 17.361894] block drbd1: Resync
done (total 1 sec; paused 0 sec; 0 K/sec)</font>
<br><font size=2 face="sans-serif">[ 17.361908] block drbd1: conn(
PausedSyncT -> Connected ) disk( Inconsistent -> UpToDate )</font>
<br><font size=2 face="sans-serif">[ 17.361951] block drbd1: helper
command: /sbin/drbdadm after-resync-target minor-1</font>
<br><font size=2 face="sans-serif">[ 17.364882] block drbd1: helper
command: /sbin/drbdadm after-resync-target minor-1 exit code 0 (0x0)</font>
<br><font size=2 face="sans-serif">[ 17.364898] block drbd1: peer_isp(
1 -> 0 )</font>
<br>
<br><font size=2 face="sans-serif">Regards,</font>
<br><font size=2 face="sans-serif">Adam Niedzwiedzki<br>
</font>