Hej Lars, I personally would love to become a paid customer, for that would save me a lot of pain in the ... headache I mean :-). But I am just an offshore contractor and that is what we are paid for. Posting "tons" of logs was a lame effort to share this pain with you, and I appologise for that. I hope you did not took it offensive too much. Anyway you provided a very good answer and now it is much clearer in what direction we should go to fix our problem.<br>
<br>Thank you for very much for you help,<br>Maxim Vladimirsky<br><br><div class="gmail_quote">On Tue, Jun 2, 2009 at 5:37 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">On Thu, May 28, 2009 at 10:16:07PM +0400, Maxim Vladimirsky wrote:<br>
> On Thu, May 28, 2009 at 12:22 PM, Lars Ellenberg wrote:<br>
...<br>
<div class="im">> > I am sure they do _log_ why they are going StandAlone.<br>
> > Maybe you want to let us know?<br>
</div> ...<br>
<div class="im">> > My guess is that somehow your init script start/stop order got screwed up.<br>
</div> ...<br>
<div class="im">><br>
> Hi Lars,<br>
><br>
> Please find message logs from both nodes attached (drbd-8.3.1.zip). As you<br>
> can see in there node1 (primary) was rebooted (I did that myself to check<br>
> how cluster reacts) but on boot-up it went in StandAlone mode instead of<br>
> Secondary. Just for comparison I attached logs from a cluster running DRBD<br>
> 0.7.24, as you can see in the same situation rebooted primary node come up<br>
> as secondary - this is the behavior I expect from DRBD 8.3.1 cluster.<br>
<br>
</div>No Sir.<br>
<br>
I did not mean you to post thousands of log lines.<br>
Going through thousands of lines of log files is a service we provide to<br>
paying customers. To those we also provide help integrating DRBD into<br>
whatever solution they have.<br>
wanna become a paying customer?<br>
;)<br>
<br>
I meant you to look into those logs,<br>
and find out why they go StandAlone.<br>
<br>
But, again: the hint was init script start/stop order.<br>
you probably stop the network/down the replication link before you<br>
umount and make it secondary. which results in a split brain situation,<br>
and (potentially) diverging datasets.<br>
which is detected uppon next handshake once the communication is restored.<br>
drbd 0.7 did not have any such detection.<br>
<br>
checkout the users guide, read about split brain.<br>
<br>
to avoid a home grown split brain and data set divergence on every<br>
orderly reboot, you need to first stop services, then umount, then<br>
secondary drbd. and _then_ you may take down your communications.<br>
<div class="im"><br>
> I our case DRBD is started and to some degree controlled (though<br>
> monitored would probably be more correct term) by drbd_controller -<br>
> our proprietary application.<br>
<br>
</div>neat. then it is probably not even init script ordering,<br>
but just broken home grown "controller" application.<br>
I guess you have to fix it.<br>
<div class="im"><br>
> It is logs can be seen in the message log. But it has nothing to do<br>
> with DRBD going StandAlone<br>
<br>
</div>I disagree.<br>
<div class="im"><br>
> for drbd_controller just started DRBD and it came up StandAlone before<br>
> drbd_controller had chance to do something wise.<br>
<br>
</div>yeah sure ;) but that is only the _reaction_, not the cause. the cause<br>
is your "controller" doing stupid things (or not enough, or the wrong<br>
things, or the right things in the wrong order) on reboot.<br>
<br>
(only my educated guess, of course)<br>
<div class="im"><br>
> The intention to set such a short timeouts was to make sure that<br>
> drbdadm/drbdsetup does not block drbd_controller for long time. It is<br>
> drbd_controller responsibility to poll DRBD state on regular basis.<br>
><br>
> Kind regards, Maxim Vladimirsky<br>
<br>
</div>hth.<br>
<font color="#888888"><br>
--<br>
</font><div><div></div><div class="h5">: Lars Ellenberg<br>
: LINBIT | Your Way to High Availability<br>
: DRBD/HA support and consulting <a href="http://www.linbit.com" target="_blank">http://www.linbit.com</a><br>
<br>
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>
__<br>
please don't Cc me, but send to list -- I'm subscribed<br>
_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</div></div></blockquote></div><br>