Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Sun, Sep 23, 2012 at 12:18:57PM +0200, Markus Müller wrote:
> Hello DRBD Users,
>
> I have a drbd two-node setup running, and got alarmed by the LINBIT
> mail about sync problems with newer kernels. So I updated to 8.4.2
> and tried to make sure anything is fine now.
>
> Even if the mail of LINBIT says that no action is required after
> upgrading, I tried the "drbdadm verify" feature. And it found "oos",
> means blocks not in sync. I thought okay, good that you thought for
> that, and tried to fix this as described in the LINBIT mail by doing
> "drbdadm disconnect/connect". It synced the found "oos:" and I
> thought everything is fine, so I did rerun the "drbdadm verify" to
> just be sure. And I saw... just found more "oos:"! I did again a
> "drbdadm disconnect/connect" but there were still more "oos:" after
> the next "drbdadm verify". I made this some loops and saw that this
> is not working at all to fix this!
If this is while the device was idle,
it is an indication that your hardware flips bits.
If it happens while the device is in use,
certain usage patterns can cause blocks to be different,
search for "digest integrity explained" in the list archives.
> To spot the estimation of damage I stopped drbd, made a network
> block device via "nbd-client / nbd-server" and run the following
> perl script on the devices on primary and secondary from drbd
> directly. The perl program verifies the differences in 1 MB blocks,
> and writes a line for each 1 MB block wich is not fully in sync by
> each bit. This shows that in the 500 GB partition there are still 18
> blocks of 1 MB which have one or more bits different on my secondary
> to my primary. The rerun of the "drbdadm verify/disconnect/connect"
> don't change anything here!
You did eliminate any caching effects (use direct IO)?
Did you compare (diff -U0) a hexdump (xxd) of the differing blocks?
Do you realize that you must only compare the *data* area?
> By the way: Further I saw that the increasing of the MTU size of the
> network interface above 1500 leads that drbd cannot be connected at
> all!
Works for very many people just fine. ==> "You are doing it wrong."
> Network block device of linux just works fine, also with a MTU
> of 9.000 and is in RAID1 much more faster than drbd ever was. Hmm...
> Does anybody see real bennefits of drbd against a solution with nbd
> and raid1?
Whatever works for you.
> $deva = "/dev/md2";
> $devb = "/dev/nbd0";
> $| = 1;
> $size = 1*1000*1000;
> $gb = 1000*1000*1000;
How about power of two block sizes?
Makes it easier to relate to 4k blocks or 512 byte sector numbers
in the kernel log...
> $pos = 0; $bad = 0;
> system("blockdev --setra 200000 ".$deva);
> system("blockdev --setra 200000 ".$devb);
> open(A, "<", $deva) || die($!);
> open(B, "<", $devb) || die($!);
> while((my $reada = sysread(A, $bufa, $size)) == $size) {
> my $readb = sysread(B, $bufb, $size);
> die "a != b\t'".$reada."!=".$readb."'"
> if ($reada != $readb);
> $bad++ if ($bufa ne $bufb);
> print $bad." bad ".(int(++$pos)/($gb/$size))." GB ".(($lastbad
> == $bad) ? "\r" : "\n");
> $lastbad = $bad;
> }
>
> My primary is ubuntu with kernel 3.0, my secondary is debian wWheezy
> with kernel 3.2. Both have drbd 8.4.2 and are 64 bit.
--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com