Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Before the reboot, the two systems see each other and are in sync. When
I try to start drbd on hydrogen (who was master) after rebooting it I get
hydrogen:~ # service drbd start
Starting DRBD resources: [ d0 s0 n0 ].
..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- In case this node was already a degraded cluster before the
reboot the timeout is 0 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot the timeout will
expire after 0 seconds. [wfc-timeout]
(These values are for resource 'r0'; 0 sec -> wait forever)
To abort waiting enter 'yes' [ 520]:
So right away there is a problem. The logs show drbd complaining about
a split brain:
Mar 5 17:26:49 hydrogen kernel: drbd0: conn( WFConnection ->
WFReportParams )
Mar 5 17:26:49 hydrogen kernel: drbd0: Handshake successful: DRBD
Network Protocol version 86
Mar 5 17:26:49 hydrogen kernel: drbd0: Split-Brain detected, dropping
connection!
Mar 5 17:26:49 hydrogen kernel: drbd0: self
C9710AB94F619A7F:F920CFF31F2A1606:C2B9EF60E881089C:2F33912A597BE6F2
Mar 5 17:26:49 hydrogen kernel: drbd0: peer
CD986B54BF6D0C8B:F920CFF31F2A1607:C2B9EF60E881089D:2F33912A597BE6F2
Mar 5 17:26:49 hydrogen kernel: drbd0: conn( WFReportParams ->
Disconnecting )
Mar 5 17:26:49 hydrogen kernel: drbd0: error receiving ReportState, l: 4!
Mar 5 17:26:49 hydrogen kernel: drbd0: asender terminated
Mar 5 17:26:49 hydrogen kernel: drbd0: tl_clear()
Mar 5 17:26:49 hydrogen kernel: drbd0: Connection closed
Mar 5 17:26:49 hydrogen kernel: drbd0: conn( Disconnecting -> StandAlone )
Mar 5 17:26:49 hydrogen kernel: drbd0: receiver terminated
At the same time, oxygen (now primary) is logging:
Mar 5 17:26:49 oxygen kernel: drbd0: conn( WFConnection -> WFReportParams )
Mar 5 17:26:49 oxygen kernel: drbd0: Handshake successful: DRBD Network
Protocol version 86
Mar 5 17:26:49 oxygen kernel: drbd0: Split-Brain detected, dropping
connection!
Mar 5 17:26:49 oxygen kernel: drbd0: self
CD986B54BF6D0C8B:F920CFF31F2A1607:C2B9EF60E881089D:2F33912A597BE6F2
Mar 5 17:26:49 oxygen kernel: drbd0: peer
C9710AB94F619A7F:F920CFF31F2A1606:C2B9EF60E881089C:2F33912A597BE6F2
Mar 5 17:26:49 oxygen kernel: drbd0: conn( WFReportParams ->
Disconnecting )
Mar 5 17:26:49 oxygen kernel: drbd0: error receiving ReportState, l: 4!
Mar 5 17:26:49 oxygen kernel: drbd0: meta connection shut down by peer.
Mar 5 17:26:49 oxygen kernel: drbd0: asender terminated
Mar 5 17:26:49 oxygen kernel: drbd0: tl_clear()
Mar 5 17:26:49 oxygen kernel: drbd0: Connection closed
Mar 5 17:26:49 oxygen kernel: drbd0: conn( Disconnecting -> StandAlone )
Mar 5 17:26:49 oxygen kernel: drbd0: receiver terminated
At this point I am completely confused. I thought hydrogen (the
rebooted system) should see that it is out of date and become secondary
and resync itself, instead I'm getting split brain. The file system on
the drbd partition is XFS and is mounted read only, so no one is writing
to partition before, during or after the reboot of hydrogen.
Is there a way to print the metadata line (like the one you see in the
logs) manually, I'd like to see if it matches before and after reboot.
Maybe something is altering the data during shutdown or bootup?
Francesco Ciocchetti wrote:
> is DRBD correctly starting on hydrogen? do you have session established
> beetween nodes (it does not seem so).
> what about the logs? there is something there that can justify a
> situation like this?
> what if you try to force connection and primary state or to invalidate peer?
>
> bye
> Francesco
>
> David wrote:
>
>> Francesco Ciocchetti wrote:
>>
>>> I' ve a newbie about DRBD but i experienced a problem like your one. In
>>> my case the problem was the setting of the following configuration
>>> instructions:
>>>
>>> I had to change the first one to this value to be able to regain from
>>> the SB.
>>>
>>>
>>> after-sb-0pri discard-younger-primary;
>>> after-sb-1pri consensus;
>>> after-sb-2pri disconnect;
>>>
>>>
>>> Bye
>>>
>>> David wrote:
>>>
>>>
>>>> Before reboot:
>>>>
>>>> hydrogen:/etc/ha.d # cat /proc/drbd
>>>> version: 8.0.1 (api:86/proto:86)
>>>> SVN Revision: 2784 build by root at hydrogen, 2007-03-05 08:47:01
>>>> 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
>>>> ns:264 nr:0 dw:256 dr:580 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
>>>> resync: used:0/31 hits:20 misses:2 starving:0 dirty:0 changed:2
>>>> act_log: used:0/127 hits:25 misses:0 starving:0 dirty:0 changed:0
>>>>
>>>> oxygen:~ # cat /proc/drbd
>>>> version: 8.0.1 (api:86/proto:86)
>>>> SVN Revision: 2784 build by root at oxygen, 2007-03-05 08:43:02
>>>> 0: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
>>>> ns:0 nr:264 dw:264 dr:0 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
>>>> resync: used:0/31 hits:20 misses:2 starving:0 dirty:0 changed:2
>>>> act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
>>>>
>>>>
>>>> During hydrogen reboot:
>>>> oxygen:~ # cat /proc/drbd
>>>> version: 8.0.1 (api:86/proto:86)
>>>> SVN Revision: 2784 build by root at oxygen, 2007-03-05 08:43:02
>>>> 0: cs:WFConnection st:Primary/Unknown ds:UpToDate/DUnknown C r---
>>>> ns:0 nr:264 dw:520 dr:316 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
>>>> resync: used:0/31 hits:20 misses:2 starving:0 dirty:0 changed:2
>>>> act_log: used:0/127 hits:25 misses:0 starving:0 dirty:0 changed:0
>>>>
>>>>
>>>> Started drbd (no heartbeat) on hydrogen
>>>> oxygen:~ # cat /proc/drbd
>>>> version: 8.0.1 (api:86/proto:86)
>>>> SVN Revision: 2784 build by root at oxygen, 2007-03-05 08:43:02
>>>> 0: cs:StandAlone st:Primary/Unknown ds:UpToDate/DUnknown r---
>>>> ns:0 nr:264 dw:520 dr:316 al:0 bm:2 lo:0 pe:0 ua:0 ap:0
>>>> resync: used:0/31 hits:20 misses:2 starving:0 dirty:0 changed:2
>>>> act_log: used:0/127 hits:25 misses:0 starving:0 dirty:0 changed:0
>>>>
>>>> On hydrogen, I'm seeing:
>>>> hydrogen:~ # service drbd start
>>>> Starting DRBD resources: [ d0 s0 n0 ].
>>>> ..........
>>>> ***************************************************************
>>>> DRBD's startup script waits for the peer node(s) to appear.
>>>> - In case this node was already a degraded cluster before the
>>>> reboot the timeout is 0 seconds. [degr-wfc-timeout]
>>>> - If the peer was available before the reboot the timeout will
>>>> expire after 0 seconds. [wfc-timeout]
>>>> (These values are for resource 'r0'; 0 sec -> wait forever)
>>>> To abort waiting enter 'yes' [ 520]:
>>>>
>>>>
>>>>
>>>> So just starting drbd on hydrogen causes a split brain and oxygen, now
>>>> the primary, to go into a standalone state. Why is that? The file
>>>> system is mounted as a read only file system so no changes should be
>>>> taking place. This is not a primary/primary setup so there is only one
>>>> "active" node at a time. I was under the impression that the rebooting
>>>> node, hydrogen, should see that it is out of date and become
>>>> secondary, resync itself with the primary and stay in the secondary
>>>> state until that is changed? Am I wrong?
>>>>
>>>> Both systems are identical:
>>>> SLES 10
>>>> kernel 2.6.16.27-0.9-bigsmp
>>>> drbd 8.0.1 compiled from source
>>>>
>>>>
>>>> Thanks ahead,
>>>>
>>>> David
>>>>
>>>>
>>>> _______________________________________________
>>>> drbd-user mailing list
>>>> drbd-user at lists.linbit.com
>>>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>>>
>>>>
>>>>
>>>
>>>
>> Thanks for the response. I'm currently using the settings you talk
>> about (sorry, should have included this before):
>>
>> resource r0 {
>>
>> protocol C;
>>
>> net {
>> after-sb-0pri discard-younger-primary;
>> after-sb-1pri consensus;
>> after-sb-2pri disconnect;
>> }
>>
>> syncer {
>> rate 120M;
>> }
>>
>> on hydrogen {
>> device /dev/drbd0;
>> disk /dev/sda4;
>> address 172.16.0.2:7788;
>> meta-disk /dev/sda3[0];
>> }
>>
>> on oxygen {
>> device /dev/drbd0;
>> disk /dev/sda4;
>> address 172.16.0.1:7788;
>> meta-disk /dev/sda3[0];
>> }
>> }
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user at lists.linbit.com
>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>
>>
>
>