Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Tue, Jun 24, 2008 at 10:42:37AM +0200, Eric Marin wrote:
> I did this once and got identical checksums :
> ldap-a:~# dd if=/dev/urandom of=/tmp/foobar bs=1M count=256
> ldap-a:~# md5sum /tmp/foobar
> ldap-b:~# netcat -l -p 4711 | md5sum
> ldap-a:~# netcat -q0 192.168.0.2 4711 < /tmp/foobar
well. not exactly the best possible test.
you have a total of 20 "bit flips" in how much GB of payload?
and it is still a difference to send from/to userspace than from within
the kernel, the latter may expose more timing critical things.
>> maybe first enable the "integrity checking",
>> see if that detects something,
>> then disable the checksum offloading,
>> see if it still occurs,
>> if not leave offloading disabled,
>> but disable again the "integrity checking" for performance reasons.
> OK, I'm going to try that next.
>
>>> ldap-a has crashed twice in three weeks,
>>
>> what exactly does "crashed" mean?
>> just "unresponsive"? panic? oops? BUG()?
>> any logs? anything from the console?
> The screen is completely black and the keyboard unresponsive. Even
check wether your console is setup to "auto blank" (dpms or some such).
if it is, it is likely the reason why the screen is blank.
and afer a hard crash, there would be nothing to "un-blank" it again.
disable that feature.
> Alt+Print+B doesn't reboot the server, I have to press the power button
> for a few seconds.
I've seen sysrq-b do nothing when sysrq-o would still switch off the
machine. but yes, that sounds pretty unresponsive :(
> I can't check right now, but the first time it crashed, I couldn't
> find anything in the logs, except for out-of-sync warnings and a
> corrupted FS on a partition mounted by DRBD. Only ldap-a has crashed
> for now, but drbdadm verify all was always executed on ldap-a.
>
>> oh, and in that case (crashed a few times),
>> we need to consider more failure modes:
>> is there any volatile disk cache involved?
>> is no-disk-flushes set?
>> is no-md-flushes set?
> Do you mean volatile disk cache on the RAID card ?
> I enabled no-disk-flushes and no-md-flushes so as not to pollute logs
> with "local disk flush failed", I was getting lots of that.
so you have one of those broken
> I think this is safe with a hardware RAID card with battery-backed
> cache (PERC 6i), isn't it (I hope so !) ? The write policy is :
> write back.
sounds good.
> A few other things of interest :
> -kernel = 2.6.18-6-686-bigmem #1 SMP Fri Jun 6 23:31:15 UTC 2008 i686
>
> -in /etc/sysctl.conf, I put :
> ----8<-------------------------------------------------------------------
> # In case of a 'kernel panic' or 'oops', I want the node to reboot
> # so that the resources are taken by the other node.
> kernel.panic_on_oops = 1
> kernel.panic = 1
> ------------------------------------------------------------------->8----
maybe one second is a little short?
> But it didn't reboot both times it crashed. Maybe these settings could
> be a source of problems ?
>
> -about the RAID card, I get this in DELL Open Manage Server Administrator :
> Firmware version 6.0.2-0002
> Driver version 00.00.03.01
> Minimal required driver version 00.00.03.13
> =>I first dismissed this warning, maybe I shouldn't... though I'm not
> sure how to update this driver while keeping the stock Debian kernel.
> Maybe I could find a new module...
hmm.
> -drbd0 contains live databases for OpenLDAP and MySQL, perhaps these
> could have the usage patterns you refer to ? They are not often
> modified, though : only a few entries a day.
>
>> possibly, but I'd say unlikely, until proven otherwise (by an oops stack
>> trace for example).
> I'm not used to this. Do I need to recompile the kernel to get this
> stack trace ?
no.
you just need to capture any last words it says.
disable screen auto-blank.
use a serial console to some terminal server. or a usb-to-serial, and
on the usb side, attach a terminal program with logging enabled (e.g.
screen). then add console=ttyS0 (or S1) to your kernel boot parameters.
careful if you have a serial link for heartbeat or anything else
configured, don't use the serial console for anything else.
> I'd say the additional load of the verify caused
>> some unexpected memory pressure/io-load, and you box was unable to
>> handle that. serialize your resources, reduce the syncer rate, maybe
>> reduce "max-buffers".
> By serializing resources, do you mean this for drbd1 ? : syncer { after
> drbd0; }
> Syncer rate is 33M for now, as suggested by the official documentation
> for a gigabit connection.
fine.
> What would you suggest for max-buffers : 32, which is the minimum ?
no, just leave it at the defaults, should be fine. anything below 512
will probably seriously hurt performance. it may well be that with very
low settings we have cornercases in drbd where drbd starts behaving bad.
--
: Lars Ellenberg http://www.linbit.com :
: DRBD/HA support and consulting sales at linbit.com :
: LINBIT Information Technologies GmbH Tel +43-1-8178292-0 :
: Vivenotgasse 48, A-1120 Vienna/Europe Fax +43-1-8178292-82 :
__
please don't Cc me, but send to list -- I'm subscribed