Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi all,
I'm sure there is a simple answer to this one..
2 Node cluster with heartbeat.
Migrated resources to the secondary node, (with primary online still) all
migrated fine.
As a test pulled the plug from the secondary node, resources migrated back
to primary, no issues at all.
Turned the power back on, on the secondary and everything I thought came
up ok ,but checking I now have the primary is saying the secondary is
inconsistent but the secondary is saying it's UpToDate?
How do I fix this?
storage00 = primary node
storage01 = secondary node
root at storage00:/var/log# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
root at storage00, 2010-07-20 12:47:09
m:res cs ro ds p mounted
fstype
0:meta Connected Primary/Secondary UpToDate/UpToDate C /meta ext3
1:data Connected Primary/Secondary UpToDate/Inconsistent C
root at storage00:/var/log# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
root at storage00, 2010-07-20 12:47:09
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
ns:8580 nr:12 dw:400 dr:9872 al:7 bm:13 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:0
1: cs:Connected ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:0 nr:0 dw:0 dr:2860 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
root at storage00:~# dmesg
[ 8406.964516] block drbd0: PingAck did not arrive in time.
[ 8406.965307] block drbd0: peer( Primary -> Unknown ) conn( Connected ->
NetworkFailure ) pdsk( UpToDate -> DUnknown )
[ 8406.965417] block drbd0: short read expecting header on sock: r=-512
[ 8406.965423] block drbd0: asender terminated
[ 8406.965427] block drbd0: Terminating asender thread
[ 8406.967232] block drbd0: Connection closed
[ 8406.967238] block drbd0: conn( NetworkFailure -> Unconnected )
[ 8406.967482] block drbd0: receiver terminated
[ 8406.967515] block drbd0: Restarting receiver thread
[ 8406.967520] block drbd0: receiver (re)started
[ 8406.967524] block drbd0: conn( Unconnected -> WFConnection )
[ 8407.004451] block drbd1: PingAck did not arrive in time.
[ 8407.005244] block drbd1: peer( Primary -> Unknown ) conn( Connected ->
NetworkFailure ) pdsk( UpToDate -> DUnknown )
[ 8407.005326] block drbd1: asender terminated
[ 8407.005329] block drbd1: Terminating asender thread
[ 8407.005388] block drbd1: short read expecting header on sock: r=-512
[ 8407.007276] block drbd1: Connection closed
[ 8407.007280] block drbd1: conn( NetworkFailure -> Unconnected )
[ 8407.007288] block drbd1: receiver terminated
[ 8407.007289] block drbd1: Restarting receiver thread
[ 8407.007291] block drbd1: receiver (re)started
[ 8407.007294] block drbd1: conn( Unconnected -> WFConnection )
[ 8408.897428] block drbd0: role( Secondary -> Primary )
[ 8408.898538] block drbd0: Creating new current UUID
[ 8409.017511] kjournald starting. Commit interval 5 seconds
[ 8409.018224] EXT3 FS on drbd0, internal journal
[ 8409.018227] EXT3-fs: recovery complete.
[ 8409.018230] EXT3-fs: mounted filesystem with ordered data mode.
[ 8409.055569] block drbd1: role( Secondary -> Primary )
[ 8409.056602] block drbd1: Creating new current UUID
[ 8409.667204] iSCSI Enterprise Target Software - version 1.4.19
[ 8409.667321] iscsi_trgt: Registered io type fileio
[ 8409.667322] iscsi_trgt: Registered io type blockio
[ 8409.667324] iscsi_trgt: Registered io type nullio
[ 8512.289805] block drbd0: Handshake successful: Agreed network protocol
version 91
[ 8512.289816] block drbd0: conn( WFConnection -> WFReportParams )
[ 8512.289898] block drbd1: Handshake successful: Agreed network protocol
version 91
[ 8512.289902] block drbd1: conn( WFConnection -> WFReportParams )
[ 8512.289937] block drbd0: Starting asender thread (from drbd0_receiver
[948])
[ 8512.290780] block drbd1: Starting asender thread (from drbd1_receiver
[951])
[ 8512.291528] block drbd1: data-integrity-alg: <not-used>
[ 8512.291552] block drbd1: drbd_sync_handshake:
[ 8512.291555] block drbd1: self
51ACB128365276DD:03BD4338E0A8C438:19942D6D67C68490:55E6E96AB00ACDBB bits:0
flags:0
[ 8512.291558] block drbd1: peer
03BD4338E0A8C438:0000000000000000:19942D6D67C68490:55E6E96AB00ACDBB bits:0
flags:2
[ 8512.291561] block drbd1: uuid_compare()=1 by rule 70
[ 8512.291565] block drbd1: peer( Unknown -> Secondary ) conn(
WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )
[ 8512.302672] block drbd0: data-integrity-alg: <not-used>
[ 8512.302722] block drbd0: drbd_sync_handshake:
[ 8512.302726] block drbd0: self
591E74B9107F0F85:BF8A5A7F3C4BFEE4:AE4808A77E5CD3C2:036AAA560A3BB0FF bits:1
flags:0
[ 8512.302730] block drbd0: peer
BF8A5A7F3C4BFEE4:0000000000000000:AE4808A77E5CD3C2:036AAA560A3BB0FF
bits:2048 flags:2
[ 8512.302733] block drbd0: uuid_compare()=1 by rule 70
[ 8512.302740] block drbd0: peer( Unknown -> Secondary ) conn(
WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )
[ 8512.317339] block drbd0: conn( WFBitMapS -> SyncSource ) pdsk( UpToDate
-> Inconsistent )
[ 8512.317379] block drbd1: aftr_isp( 0 -> 1 )
[ 8512.317383] block drbd0: Began resync as SyncSource (will sync 8192 KB
[2048 bits set]).
[ 8512.325110] block drbd1: peer_isp( 0 -> 1 )
[ 8512.528583] block drbd1: conn( WFBitMapS -> PausedSyncS ) pdsk(
UpToDate -> Inconsistent )
[ 8512.528680] block drbd1: Began resync as PausedSyncS (will sync 0 KB [0
bits set]).
[ 8512.615174] block drbd0: Resync done (total 1 sec; paused 0 sec; 8192
K/sec)
[ 8512.615183] block drbd0: conn( SyncSource -> Connected ) pdsk(
Inconsistent -> UpToDate )
[ 8512.615192] block drbd1: aftr_isp( 1 -> 0 )
[ 8513.078766] block drbd1: Resync done (total 1 sec; paused 0 sec; 0
K/sec)
[ 8513.078777] block drbd1: conn( PausedSyncS -> Connected ) pdsk(
Inconsistent -> UpToDate )
[ 8513.078819] block drbd1: pdsk( UpToDate -> Inconsistent ) peer_isp( 1
-> 0 )
root at storage01:~# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
root at storage01, 2010-07-20 15:13:41
m:res cs ro ds p mounted fstype
0:meta Connected Secondary/Primary UpToDate/UpToDate C
1:data Connected Secondary/Primary UpToDate/UpToDate C
root at storage01:~# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
root at storage01, 2010-07-20 15:13:41
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
ns:0 nr:8192 dw:8192 dr:0 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:0
1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
root at storage01:~# dmesg
[ 16.336431] block drbd0: Starting worker thread (from cqueue [1190])
[ 16.336585] block drbd0: disk( Diskless -> Attaching )
[ 16.340633] block drbd0: Found 4 transactions (4 active extents) in
activity log.
[ 16.340637] block drbd0: Method to ensure write ordering: barrier
[ 16.340641] block drbd0: max_segment_size ( = BIO size ) = 32768
[ 16.340646] block drbd0: drbd_bm_resize called with capacity == 1900408
[ 16.340653] block drbd0: resync bitmap: bits=237551 words=3712
[ 16.340655] block drbd0: size = 928 MB (950204 KB)
[ 16.341756] block drbd0: recounting of set bits took additional 0
jiffies
[ 16.341760] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk
bit-map.
[ 16.341767] block drbd0: Marked additional 8192 KB as out-of-sync based
on AL.
[ 16.373949] block drbd0: disk( Attaching -> UpToDate )
[ 16.378189] block drbd1: Starting worker thread (from cqueue [1190])
[ 16.378349] block drbd1: disk( Diskless -> Attaching )
[ 16.395248] block drbd1: No usable activity log found.
[ 16.395254] block drbd1: Method to ensure write ordering: barrier
[ 16.395259] block drbd1: max_segment_size ( = BIO size ) = 32768
[ 16.395264] block drbd1: drbd_bm_resize called with capacity ==
83879352
[ 16.399814] block drbd1: resync bitmap: bits=10484919 words=163827
[ 16.399819] block drbd1: size = 40 GB (41939676 KB)
[ 16.434454] block drbd1: recounting of set bits took additional 0
jiffies
[ 16.434459] block drbd1: 0 KB (0 bits) marked out-of-sync by on disk
bit-map.
[ 16.434466] block drbd1: Marked additional 0 KB as out-of-sync based on
AL.
[ 16.434492] block drbd1: disk( Attaching -> UpToDate )
[ 16.467318] block drbd0: conn( StandAlone -> Unconnected )
[ 16.468011] block drbd0: Starting receiver thread (from drbd0_worker
[1193])
[ 16.469226] block drbd1: conn( StandAlone -> Unconnected )
[ 16.469585] block drbd1: Starting receiver thread (from drbd1_worker
[1199])
[ 16.471272] block drbd1: receiver (re)started
[ 16.471279] block drbd1: conn( Unconnected -> WFConnection )
[ 16.472862] block drbd0: receiver (re)started
[ 16.472870] block drbd0: conn( Unconnected -> WFConnection )
[ 16.573970] block drbd0: Handshake successful: Agreed network protocol
version 91
[ 16.573983] block drbd0: conn( WFConnection -> WFReportParams )
[ 16.574064] block drbd1: Handshake successful: Agreed network protocol
version 91
[ 16.574070] block drbd1: conn( WFConnection -> WFReportParams )
[ 16.574174] block drbd0: Starting asender thread (from drbd0_receiver
[1220])
[ 16.574276] block drbd1: Starting asender thread (from drbd1_receiver
[1222])
[ 16.575190] block drbd1: data-integrity-alg: <not-used>
[ 16.575966] block drbd1: drbd_sync_handshake:
[ 16.575971] block drbd1: self
03BD4338E0A8C438:0000000000000000:19942D6D67C68490:55E6E96AB00ACDBB bits:0
flags:0
[ 16.575975] block drbd1: peer
51ACB128365276DD:03BD4338E0A8C438:19942D6D67C68490:55E6E96AB00ACDBB bits:0
flags:0
[ 16.575978] block drbd1: uuid_compare()=-1 by rule 50
[ 16.575986] block drbd1: peer( Unknown -> Primary ) conn(
WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
[ 16.589030] block drbd0: data-integrity-alg: <not-used>
[ 16.590543] block drbd0: drbd_sync_handshake:
[ 16.590547] block drbd0: self
BF8A5A7F3C4BFEE4:0000000000000000:AE4808A77E5CD3C2:036AAA560A3BB0FF
bits:2048 flags:0
[ 16.590551] block drbd0: peer
591E74B9107F0F85:BF8A5A7F3C4BFEE4:AE4808A77E5CD3C2:036AAA560A3BB0FF bits:1
flags:0
[ 16.590554] block drbd0: uuid_compare()=-1 by rule 50
[ 16.590561] block drbd0: peer( Unknown -> Primary ) conn(
WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
[ 16.595192] block drbd0: conn( WFBitMapT -> WFSyncUUID )
[ 16.601669] block drbd0: helper command: /sbin/drbdadm
before-resync-target minor-0
[ 16.605313] block drbd0: helper command: /sbin/drbdadm
before-resync-target minor-0 exit code 0 (0x0)
[ 16.605323] block drbd0: conn( WFSyncUUID -> SyncTarget ) disk(
UpToDate -> Inconsistent )
[ 16.605336] block drbd1: aftr_isp( 0 -> 1 )
[ 16.605342] block drbd0: Began resync as SyncTarget (will sync 8192 KB
[2048 bits set]).
[ 16.730537] block drbd1: conn( WFBitMapT -> WFSyncUUID )
[ 16.730577] block drbd1: peer_isp( 0 -> 1 )
[ 16.812628] block drbd1: helper command: /sbin/drbdadm
before-resync-target minor-1
[ 16.817954] block drbd1: helper command: /sbin/drbdadm
before-resync-target minor-1 exit code 0 (0x0)
[ 16.817964] block drbd1: conn( WFSyncUUID -> PausedSyncT ) disk(
UpToDate -> Inconsistent )
[ 16.817977] block drbd1: Began resync as PausedSyncT (will sync 0 KB [0
bits set]).
[ 16.906724] block drbd0: Resync done (total 1 sec; paused 0 sec; 8192
K/sec)
[ 16.906734] block drbd0: conn( SyncTarget -> Connected ) disk(
Inconsistent -> UpToDate )
[ 16.906741] block drbd0: helper command: /sbin/drbdadm
after-resync-target minor-0
[ 16.909092] block drbd0: helper command: /sbin/drbdadm
after-resync-target minor-0 exit code 0 (0x0)
[ 16.909102] block drbd1: aftr_isp( 1 -> 0 )
[ 17.361894] block drbd1: Resync done (total 1 sec; paused 0 sec; 0
K/sec)
[ 17.361908] block drbd1: conn( PausedSyncT -> Connected ) disk(
Inconsistent -> UpToDate )
[ 17.361951] block drbd1: helper command: /sbin/drbdadm
after-resync-target minor-1
[ 17.364882] block drbd1: helper command: /sbin/drbdadm
after-resync-target minor-1 exit code 0 (0x0)
[ 17.364898] block drbd1: peer_isp( 1 -> 0 )
Regards,
Adam Niedzwiedzki
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20100722/b2fffcae/attachment.htm>