Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi,
While I understand the risks associated with forcing a single node
online by using the re-elect options, I am currently documenting a
pre-prod cluster and have to document destructive testing and recovery
procedures.
The situation I am trying to validate would be one where we have a 3
node cluster which spans datacenters, and we have ended up in a
position where one datacenter is off the air which has in turn taken
out 2 nodes of the 3 node cluster, the remaining node has for what
ever reason crashed / restarted and we now need to get this node
online (post cleanup tasks will be captured as part of the docs)
I know that I can simply copy the content of /var/lib/drbd.d/ to
/etc/drbd.d/ do a quick rename and then use drbdadm to bring the
resources online, but since I am provisioning all my drbd resources
via drbdmanage I would like to be able to force this service online,
I have tried the drbdmanage reelect (and force-win) options but am
still unable to connect to the drbdmange process (This said I am able
to see all drbd resources using drbdadm status)
[root at node1 ~]# drbdmanage reelect --force-win
Operation completed successfully
unknown
[root at node1 ~]# drbdadm status
.drbdctrl role:Primary
volume:0 disk:UpToDate
volume:1 disk:UpToDate
node2.domain.name connection:Connecting
node3.domain.name connection:Connecting
resource-sda role:Secondary
disk:UpToDate
node2.domain.name connection:Connecting
[root at lpisscl0001 ~]# drbdmanage ping
pong
[root at node1 ~]# drbdmanage v
ERROR:dbus.proxies:Introspect error on :1.30:/interface:
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did
not receive a reply. Possible causes include: the remote application
did not send a reply, the message bus security policy blocked the
reply, the reply timeout expired, or the network connection was
broken.
Waiting for server: ...............
Error: Startup not successful (no quorum? not *both* nodes up in a 2
node cluster?)
No resources defined
So if I am correct the (completely unsupported / do so at your own
risk) process to force access to the drbdmanaged resources in the
event of loss of quorum for the drbdmanaged process would be
surviving node: node1
[root at node1 ~]# drbdmanage reelect --force-win
recover node2 / node3 from DR procedure / backups
post reintroduction of additional nodes:
restart drbdmanaged process on node1 / reboot node1
Thanks
Jay
On 3 October 2017 at 16:08, Jason Fitzpatrick <jayfitzpatrick at gmail.com> wrote:
> Thanks I will try that now
>
> On 3 Oct 2017 12:05, "Yannis Milios" <yannis.milios at gmail.com> wrote:
>>
>> I think you have to use 'drbdmanage reelect' command to reelect a new
>> leader first.
>>
>> man drbdmanage-reelect
>>
>> Yannis
>>
>>
>>
>> On Mon, Oct 2, 2017 at 2:12 PM, Jason Fitzpatrick
>> <jayfitzpatrick at gmail.com> wrote:
>>>
>>> Hi all
>>>
>>> I am trying to get my head around the quorum-control features within
>>> drbdmanage,
>>>
>>> I have deliberately crashed my cluster, and spun up one node, and as
>>> expected I am unable to get drbdmanage to start due to the lack of
>>> quorum,,
>>>
>>> I was under the impression that I should have been able to override
>>> the quorum state and get the drbdmanaged process online using DBUS /
>>> manually calling the service, but am drawing a blank..
>>>
>>> for the sake of this example it is a 2 node cluster node1 is online
>>> and node2 is still powered off,
>>>
>>> [root at node1]# drbdmanage quorum-control --override ignore node2
>>> Modifying quorum state of node 'node2':
>>> Waiting for server: ...............
>>> Error: Startup not successful (no quorum? not *both* nodes up in a 2
>>> node cluster?)
>>> Error: Startup not successful (no quorum? not *both* nodes up in a 2
>>> node cluster?)
>>>
>>> Any advice?
>>>
>>> Thanks
>>>
>>> Jay
>>>
>>> --
>>>
>>> "The only difference between saints and sinners is that every saint
>>> has a past while every sinner has a future. "
>>> — Oscar Wilde
>>> _______________________________________________
>>> drbd-user mailing list
>>> drbd-user at lists.linbit.com
>>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>
>>
>
--
"The only difference between saints and sinners is that every saint
has a past while every sinner has a future. "
— Oscar Wilde