Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
----- Original Message -----
> From: "venkatesh prabhu" <vikieethechip at gmail.com>
> To: drbd-user at lists.linbit.com
> Sent: Thursday, February 23, 2012 6:13:24 AM
> Subject: [DRBD-user] Problem with StandAlone
>
> Hi,
> I configured a two node cluster and able to mirror them.
> But facing issue when testing some scenarios.
>
> Steps i followed and end up in standalone mode is
> 1. Shutdowm the primary node <N1>.
> 2. make the other node<N2> primary - make some changes in file
> system.
> 3. shutdown the N2.
> 4. start the N1, make it primary , do some changes in file system and
> reboot N1 and start N2.
> 5. when they come up N1 state is
> 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C
> r-----
> 6. The state of N2 is
> 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown
> r-----
> But i have added auto recovery split brain algorithm in my config
> file. Please see the config file at the end.
> Please let me know how to automatically solve this standalone
> situation.
>
> Logs from N2:
> kernel: drbd: initialized. Version: 8.3.12 (api:88/proto:86-96)
> Feb 23 16:03:19 lab1602 kernel: drbd: GIT-hash:
> e2a8ef4656be026bbae540305fcb998a5991090f build by dag at Build32R6,
> 2011-11-20 10:55:07
> Feb 23 16:03:19 lab1602 kernel: drbd: registered as block device
> major 147
> Feb 23 16:03:19 lab1602 kernel: drbd: minor_table @ 0xf40ac700
> Feb 23 16:03:19 lab1602 kernel: block drbd0: Starting worker thread
> (from cqueue [1538])
> Feb 23 16:03:19 lab1602 kernel: block drbd0: disk( Diskless ->
> Attaching )
> Feb 23 16:03:19 lab1602 kernel: block drbd0: Found 4 transactions (6
> active extents) in activity log.
> Feb 23 16:03:19 lab1602 kernel: block drbd0: Method to ensure write
> ordering: drain
> Feb 23 16:03:19 lab1602 kernel: block drbd0: max BIO size = 131072
> Feb 23 16:03:19 lab1602 kernel: block drbd0: drbd_bm_resize called
> with capacity == 2097152
> Feb 23 16:03:19 lab1602 kernel: block drbd0: resync bitmap:
> bits=262144 words=8192 pages=8
> Feb 23 16:03:19 lab1602 kernel: block drbd0: size = 1024 MB (1048576
> KB)
> Feb 23 16:03:19 lab1602 kernel: block drbd0: bitmap READ of 8 pages
> took 1 jiffies
> Feb 23 16:03:19 lab1602 kernel: block drbd0: recounting of set bits
> took additional 0 jiffies
> Feb 23 16:03:19 lab1602 kernel: block drbd0: 0 KB (0 bits) marked
> out-of-sync by on disk bit-map.
> Feb 23 16:03:19 lab1602 kernel: block drbd0: Marked additional 12 MB
> as out-of-sync based on AL.
> Feb 23 16:03:19 lab1602 kernel: block drbd0: bitmap WRITE of 0 pages
> took 0 jiffies
> Feb 23 16:03:19 lab1602 kernel: block drbd0: 12 MB (3072 bits) marked
> out-of-sync by on disk bit-map.
> Feb 23 16:03:19 lab1602 kernel: block drbd0: disk( Attaching ->
> UpToDate )
> Feb 23 16:03:19 lab1602 kernel: block drbd0: attached to UUIDs
> 1CAAF7BF141F6FAB:77403D46BDEFCA05:022A37BAC8B2DC72:022937BAC8B2DC72
> Feb 23 16:03:19 lab1602 kernel: block drbd0: conn( StandAlone ->
> Unconnected )
> Feb 23 16:03:19 lab1602 kernel: block drbd0: Starting receiver thread
> (from drbd0_worker [1547])
> Feb 23 16:03:19 lab1602 kernel: block drbd0: receiver (re)started
> Feb 23 16:03:19 lab1602 kernel: block drbd0: conn( Unconnected ->
> WFConnection )
At this point it looks like Node 2 doesn't see Node 1 yet and has gone primary and is waiting for connection. Is your filesystem auto mounted? Are you using a cluster manager?
Then 18 seconds later it sees Node 1.
> Feb 23 16:03:37 lab1602 kernel: block drbd0: Handshake successful:
> Agreed network protocol version 96
> Feb 23 16:03:37 lab1602 kernel: block drbd0: Peer authenticated using
> 20 bytes of 'sha1' HMAC
> Feb 23 16:03:37 lab1602 kernel: block drbd0: conn( WFConnection ->
> WFReportParams )
> Feb 23 16:03:37 lab1602 kernel: block drbd0: Starting asender thread
> (from drbd0_receiver [1565])
> Feb 23 16:03:37 lab1602 kernel: block drbd0: data-integrity-alg:
> <not-used>
> Feb 23 16:03:37 lab1602 kernel: block drbd0: drbd_sync_handshake:
> Feb 23 16:03:37 lab1602 kernel: block drbd0: self
> 1CAAF7BF141F6FAB:77403D46BDEFCA05:022A37BAC8B2DC72:022937BAC8B2DC72
> bits:3072 flags:0
> Feb 23 16:03:37 lab1602 kernel: block drbd0: peer
> 1C7B811C8FAD3496:77403D46BDEFCA04:022A37BAC8B2DC72:022937BAC8B2DC72
> bits:25 flags:0
> Feb 23 16:03:37 lab1602 kernel: block drbd0: uuid_compare()=100 by
> rule 90
> Feb 23 16:03:37 lab1602 kernel: block drbd0: helper command:
> /sbin/drbdadm initial-split-brain minor-0
> Feb 23 16:03:37 lab1602 kernel: block drbd0: helper command:
> /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
> Feb 23 16:03:37 lab1602 kernel: block drbd0: State change failed:
> Device is held open by someone
I would look into this line - pretty sure the split brain resolution can't happen because device is open - probably from mounted filesystem? It can't resolve split brain because the device is open and then it gives up after a few tries (I think)
> Feb 23 16:03:37 lab1602 kernel: block drbd0: state = {
> cs:WFReportParams ro:Primary/Unknown ds:UpToDate/DUnknown r----- }
> Feb 23 16:03:37 lab1602 kernel: block drbd0: wanted = {
> cs:WFReportParams ro:Secondary/Unknown ds:UpToDate/DUnknown r----- }
> Feb 23 16:03:37 lab1602 kernel: block drbd0: State change failed:
> Device is held open by someone
> Feb 23 16:03:37 lab1602 kernel: block drbd0: state = {
> cs:WFReportParams ro:Primary/Unknown ds:UpToDate/DUnknown r----- }
> Feb 23 16:03:37 lab1602 kernel: block drbd0: wanted = {
> cs:WFReportParams ro:Secondary/Unknown ds:UpToDate/DUnknown r----- }
> Feb 23 16:03:37 lab1602 kernel: block drbd0: helper command:
> /sbin/drbdadm pri-lost-after-sb minor-0
> Feb 23 16:03:37 lab1602 kernel: block drbd0: helper command:
> /sbin/drbdadm pri-lost-after-sb minor-0 exit code 0 (0x0)
> Feb 23 16:03:37 lab1602 kernel: block drbd0: Split-Brain detected but
> unresolved, dropping connection!
> Feb 23 16:03:37 lab1602 kernel: block drbd0: helper command:
> /sbin/drbdadm split-brain minor-0
> Feb 23 16:03:37 lab1602 kernel: block drbd0: helper command:
> /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
> Feb 23 16:03:37 lab1602 kernel: block drbd0: conn( WFReportParams ->
> Disconnecting )
> Feb 23 16:03:37 lab1602 kernel: block drbd0: error receiving
> ReportState, l: 4!
> Feb 23 16:03:37 lab1602 kernel: block drbd0: asender terminated
> Feb 23 16:03:37 lab1602 kernel: block drbd0: Terminating asender
> thread
> Feb 23 16:03:37 lab1602 kernel: block drbd0: Connection closed
> Feb 23 16:03:37 lab1602 kernel: block drbd0: conn( Disconnecting ->
> StandAlone )
> Feb 23 16:03:37 lab1602 kernel: block drbd0: receiver terminated
> Feb 23 16:03:37 lab1602 kernel: block drbd0: Terminating receiver
> thread
>
>
>
>
> Logs From N1:
>
> Feb 23 16:03:54 lab1601 kernel: drbd: initialized. Version: 8.3.12
> (api:88/proto:86-96)
> Feb 23 16:03:54 lab1601 kernel: drbd: GIT-hash:
> e2a8ef4656be026bbae540305fcb998a5991090f build by dag at Build32R6,
> 2011-11-20 10:55:07
> Feb 23 16:03:54 lab1601 kernel: drbd: registered as block device
> major 147
> Feb 23 16:03:54 lab1601 kernel: drbd: minor_table @ 0xf3d3e300
> Feb 23 16:03:54 lab1601 kernel: block drbd0: Starting worker thread
> (from cqueue [1537])
> Feb 23 16:03:54 lab1601 kernel: block drbd0: disk( Diskless ->
> Attaching )
> Feb 23 16:03:54 lab1601 kernel: block drbd0: Found 4 transactions (6
> active extents) in activity log.
> Feb 23 16:03:54 lab1601 kernel: block drbd0: Method to ensure write
> ordering: drain
> Feb 23 16:03:54 lab1601 kernel: block drbd0: max BIO size = 131072
> Feb 23 16:03:54 lab1601 kernel: block drbd0: drbd_bm_resize called
> with capacity == 2097152
> Feb 23 16:03:54 lab1601 kernel: block drbd0: resync bitmap:
> bits=262144 words=8192 pages=8
> Feb 23 16:03:54 lab1601 kernel: block drbd0: size = 1024 MB (1048576
> KB)
> Feb 23 16:03:54 lab1601 kernel: block drbd0: bitmap READ of 8 pages
> took 2 jiffies
> Feb 23 16:03:54 lab1601 kernel: block drbd0: recounting of set bits
> took additional 0 jiffies
> Feb 23 16:03:54 lab1601 kernel: block drbd0: 100 KB (25 bits) marked
> out-of-sync by on disk bit-map.
> Feb 23 16:03:54 lab1601 kernel: block drbd0: disk( Attaching ->
> UpToDate )
> Feb 23 16:03:54 lab1601 kernel: block drbd0: attached to UUIDs
> 1C7B811C8FAD3497:77403D46BDEFCA04:022A37BAC8B2DC72:022937BAC8B2DC72
> Feb 23 16:03:54 lab1601 kernel: block drbd0: conn( StandAlone ->
> Unconnected )
> Feb 23 16:03:54 lab1601 kernel: block drbd0: Starting receiver thread
> (from drbd0_worker [1547])
> Feb 23 16:03:54 lab1601 kernel: block drbd0: receiver (re)started
> Feb 23 16:03:54 lab1601 kernel: block drbd0: conn( Unconnected ->
> WFConnection )
> Feb 23 16:03:55 lab1601 kernel: block drbd0: Handshake successful:
> Agreed network protocol version 96
> Feb 23 16:03:55 lab1601 kernel: block drbd0: Peer authenticated using
> 20 bytes of 'sha1' HMAC
> Feb 23 16:03:55 lab1601 kernel: block drbd0: conn( WFConnection ->
> WFReportParams )
> Feb 23 16:03:55 lab1601 kernel: block drbd0: Starting asender thread
> (from drbd0_receiver [1564])
> Feb 23 16:03:55 lab1601 kernel: block drbd0: data-integrity-alg:
> <not-used>
> Feb 23 16:03:55 lab1601 kernel: block drbd0: drbd_sync_handshake:
> Feb 23 16:03:55 lab1601 kernel: block drbd0: self
> 1C7B811C8FAD3496:77403D46BDEFCA04:022A37BAC8B2DC72:022937BAC8B2DC72
> bits:25 flags:0
> Feb 23 16:03:55 lab1601 kernel: block drbd0: peer
> 1CAAF7BF141F6FAB:77403D46BDEFCA05:022A37BAC8B2DC72:022937BAC8B2DC72
> bits:3072 flags:2
> Feb 23 16:03:55 lab1601 kernel: block drbd0: uuid_compare()=100 by
> rule 90
> Feb 23 16:03:55 lab1601 kernel: block drbd0: helper command:
> /sbin/drbdadm initial-split-brain minor-0
> Feb 23 16:03:55 lab1601 kernel: block drbd0: helper command:
> /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
> Feb 23 16:03:55 lab1601 kernel: block drbd0: Split-Brain detected, 1
> primaries, automatically solved. Sync from this node
Node 1 has no problem resolving the split brain using the rules you set in the config
Might want to check if the clocks on the nodes are in sync? Hard to line up events when this node shows 16:03:55 which I'm guessing is the same as 16:03:37 on Node 2? Or you gave us non-correlating log sections...
HTH
Jake
> Feb 23 16:03:55 lab1601 kernel: block drbd0: peer( Unknown -> Primary
> ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
> Feb 23 16:03:55 lab1601 kernel: block drbd0: sock was shut down by
> peer
> Feb 23 16:03:55 lab1601 kernel: block drbd0: peer( Primary -> Unknown
> ) conn( WFBitMapS -> BrokenPipe ) pdsk( Consistent -> DUnknown )
> Feb 23 16:03:55 lab1601 kernel: block drbd0: short read expecting
> header on sock: r=0
> Feb 23 16:03:55 lab1601 kernel: block drbd0: meta connection shut
> down by peer.
> Feb 23 16:03:55 lab1601 kernel: block drbd0: asender terminated
> Feb 23 16:03:55 lab1601 kernel: block drbd0: Terminating asender
> thread
> Feb 23 16:03:55 lab1601 kernel: block drbd0: bitmap WRITE of 0 pages
> took 0 jiffies
> Feb 23 16:03:55 lab1601 kernel: block drbd0: 100 KB (25 bits) marked
> out-of-sync by on disk bit-map.
> Feb 23 16:03:55 lab1601 kernel: block drbd0: Connection closed
> Feb 23 16:03:55 lab1601 kernel: block drbd0: conn( BrokenPipe ->
> Unconnected )
> Feb 23 16:03:55 lab1601 kernel: block drbd0: receiver terminated
> Feb 23 16:03:55 lab1601 kernel: block drbd0: Restarting receiver
> thread
> Feb 23 16:03:55 lab1601 kernel: block drbd0: receiver (re)started
> Feb 23 16:03:55 lab1601 kernel: block drbd0: conn( Unconnected ->
> WFConnection )
>
> --
> global {
> usage-count no;
> }
>
> common {
>
> protocol C;
>
> startup {
> degr-wfc-timeout 3; #3 sec..
>
> wfc-timeout 120; # 3 min.
>
> # become-primary-on both;
> } # end of startup
>
> handlers {
> } # end of handlers
>
> disk {
> on-io-error detach;
> no-disk-barrier;
> no-disk-flushes;
> no-md-flushes;
> } # end of disk
>
> net {
> timeout 60; # 6 seconds (unit = 0.1
> seconds)
> connect-int 10; # 10 seconds (unit = 1 second)
> ping-int 10; # 10 seconds (unit = 1 second)
> ping-timeout 5; # 500 ms (unit = 0.1 seconds)
> cram-hmac-alg sha1;
> shared-secret "DRBD disk mirroring for BTP-R";
> after-sb-0pri discard-older-primary;
> #after-sb-1pri discard-secondary;
> #after-sb-2pri violently-as0p;
> after-sb-1pri call-pri-lost-after-sb;
> after-sb-2pri call-pri-lost-after-sb;
> rr-conflict call-pri-lost;
>
> } # end of net
> } # end of commo
> resource mfs_drbd {
>
> syncer {
> rate 100M;
> }
>
> on lab1602 {
> device /dev/drbd0;
> disk /dev/vg0/mirror;
> address 10.203.230.136:7788;
> meta-disk /dev/vg0/drbdmeta[0];
> #meta-disk internal;
> }
>
> on lab1601 {
> device /dev/drbd0;
> disk /dev/vg0/mirror;
> address 10.203.230.135:7788;
> meta-disk /dev/vg0/drbdmeta[0];
> #meta-disk internal;
> }
> } #end of resource sfs_drbd
>
> Thanks in advance.
>
> vengatesh
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
>