Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Thanks for your prompt reply. Comments/questions below....
On Thu, Oct 17, 2013 at 12:18 PM, Digimer <lists at alteeve.ca> wrote:
> On 17/10/13 15:08, Lonni J Friedman wrote:
>> [ 1727.959118] block drbd0: state = { cs:StandAlone
>> ro:Secondary/Unknown ds:Inconsistent/DUnknown r----- }
>> [ 1728.069711] block drbd0: wanted = { cs:StandAlone
>> ro:Primary/Unknown ds:Inconsistent/DUnknown r----- }
>
> This is your problem. I suspect that you invalidated the node that had
> been UpToDate. This is a pretty dangerous state to be in. If you have
> critical data on this cluster, I _strongly_ recommend calling LinBit and
> getting professional guidance. It will be worth the money.
Sadly the damage was done by a co-worker while I was out, and I'm
stuck trying to fix the mess. The data on this cluster doesn't change
all that often, so the odds of anything being lost are quite slim.
>
> That said;
>
> The process will be a matter of deciding which node is actually UpToDate
> and then forcing it into that role. Once done, connect should work and
> the Inconsistent node should sync from UpToDate. You'll certainly want
> to do an FS consistency check after.
OK, but what's the actual command to force a node to be primary?
There used to be a -f switch to force a node to become primary, but
that options seems to no longer exist (at least in 8.4.x).
>
> To avoid this in the future, setup fencing (aka stonith). If you're
> using pacemaker or cman+rgmanage, you can do this easily by hooking DRBD
> into their existing fence/stonith facilities.
Agreed.
>
> In any case, please proceed very carefully. If you promote the wrong
> node, you could lose data / corrupt your FS.