[DRBD-user] Help: Replacing A Dead Node

[Mike Sweetser - Adhost]

Just wanted to follow up and say that everything went just fine!  I
didn't even have to do anything regarding the adjustments - the two
nodes started resyncing by themselves once I started up DRBD on the new
node.

The only issue was that the load of resyncing the five devices was too
much to do at once (the DRBD setup supports a full hosting environment
of Apache and MySQL) so I had to cut off some of the resyncs and do them
one by one, but that'll be just fine.

Lars, thanks once again for your help!

Mike Sweetser

-----Original Message-----
From: drbd-user-bounces at lists.linbit.com
[mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Lars Ellenberg
Sent: Monday, July 28, 2008 2:47 PM
To: drbd-user at lists.linbit.com
Subject: Re: [DRBD-user] Help: Replacing A Dead Node

On Mon, Jul 28, 2008 at 02:31:49PM -0700, Mike Sweetser - Adhost wrote:
> > it is now StandAlone Secondary Inconsistent.
> > since an Inconsistent device cannot become SyncSource
> > (and a Primary device will refuse to become SyncTarget)
> > just make sure your good data is Primary still,
> > and say "drbdadm adjust all" now.
> 
> So basically, as long as I'm sure that the good data is Primary (which
> it should be under Heartbeat) then doing "drbdadm adjust all" will
> reconnect the two DRBDs and begin the resync process?  Am I on target
> there?

as long as your good data is "Primary",
the "worst" that can happen to it during drbd re-connect is,
that it detects split brain and refuses to talk to the peer.

> > you probably should not start the cluster manager yet, though,
unless
> > you are sure that your sedtup knows how to deal with a degraded
drbd.
> 
> Right, Heartbeat's staying off on the new node until the
> resynchronization is completed - it will only be running for now on
the
> Primary node.
> 
> > if you did not make it Primary, and there is no cluster manager,
> > and you just created fresh meta data that marks it as Inconsistent,
> > there is no way it possibly could be Primary...
> 
> I figured as much, but considering there's a good amount of work
> involved in trying to redo this if I get it wrong, I just wanted to
make
> 100% sure. :)
> 
> So my new plan would be:
> 
> * Reinit the metadata on the new node (new: drbdadm create-md all)
> * Attach the nodes (new: drbdadm attach all)
> * Verify that the new node is StandAlone and Secondary/Inconsistent
> * Verify that the original node is Primary
> * Run the adjust on the new node (new and possibly old: drbdadm adjust
> all)
> 
> Is that about right?

give it a go.

be prepared to re-adjust the sync rate, in case the additional IO load
has too much impact on your active services,