[DRBD-user] drbd sync killing machine with OOM

[Rene Mayrhofer]

Am Freitag, 4. Mai 2007 19:03:04 schrieb Siim Vahtre:
> On Fri, 4 May 2007, Rene Mayrhofer wrote:
> > Exactly. To enable XEN live migration, both nodes need to be primary -
> > with one drbd8 volume for each domU, I can modify the migration scripts
> > to go primary on both for the period of the migration and secondary on
> > the "old" node after migration has finished.
>
> Why only for the period of migration? For safety? If only one node is
> using the certain block range at one time, there shouldn't be problem
> (please correct me if I'm wrong!).
Yes, for safety against split-brain. The problem is not that both nodes would 
modify the same blocks, but that drbd8 would not be aware of the block ranges 
that each host is responsible for. That is, if for any reason they went 
split-brain, then an automatic "merge" would, as far as I understand drbd8 
until now, be impossible. With independent drbd8 volumes for each domU 
domain, the nodes could run independently/disconnected from each other (each 
host being primary for a number of domUs) and resync without any issue. 

One scenarios where something like this may happen is when the "sync" network 
between the nodes failed for any reason (that can happen), but when the nodes 
were still perfectly alive and connected to the outside world otherwise (via 
their other network interfaces). I have configured heartbeat to use multiple 
links, so this situation would not be considered as any node failing, only 
drbd8 would go into disconnected mode, and thus into split-brain if both 
nodes were primary for the LVM2 volume group.

Or am I missing something here? If my analysis is wrong, then please correct 
me, as I would be more than happy to have LVM2 sitting on top of drbd8 and 
enjoy less administrative overhead.
If I did not explain myself clearly enough, feel free to ask and I'll try 
harder ;-)

> > This has, IMHO, two advantages:
> > - live migration should work
> > - in normal operation, only one node is primary, and thus the chances of
> > going split-brain should be _very_ slim (the network would need to fail
> > exactly during migration, which seems unlikely)
>
> What do you mean?
>
> AFAIK Xen guarantees that while migrating only one node is writing to the
> storage at time. So migration either happens or it does not.
Yes, but as far as I have read, the XEN migration scripts can't easily deal 
with the target being read-only. That is, they are not yet able to make 
drbd8 "switch" primaries at the right moment in time. So (without having a 
proof of concept yet due to the crashes), I plan on:
1. switching both into primary
2. calling XN live migration
3. switching the "old" node into secondary
Which, as far as I am aware, should work with drbd8 and current XEN releases. 
It just needs a "master" script for heartbeat to do this.

> AFAIK you can have LVM2 active on two primary nodes at the same time but
> if you do ANY changes to volume group structure (change volume sizes etc)
> then you have to do 'vgchange -a n' in all but one node and THEN do the
> changes on only that one node that is still active. For my needs, this has
> been enough.
This would be enough for me as well. The problem is, however, not changing the 
volume group itself, but the "normal" writes to the logical volumes.

Can anybody confirm a working setup with >10 drbd8 volumes between two nodes 
in primary/secondary modes, mirroring about 60GB in total? It seems that this 
part of my setup is the problematic one, as everything was fine before I 
switched the additional 10 drbd volumes on.

with best regards,
Rene

-- 
-------------------------------------------------
Gibraltar firewall       http://www.gibraltar.at/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.linbit.com/pipermail/drbd-user/attachments/20070504/126de7e6/attachment.pgp