Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi guys
I am experiencing very strange behaviour of DRBD.
My setup is:
Two absolutely identical Supermicro nodes using LSI 9265-8i controller
with SAS disks.
Scientific Linux 6.2 with latest OpenVZ stable kernel that uses drbd 8.3.10.
the excerpt from my drbd.conf is following:
...
resource r1 {
net
{
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 2M;
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
# data-integrity-alg crc32c;
ping-int 25;
}
startup {
become-primary-on both;
}
syncer {
rate 100M;
al-extents 3383;
csums-alg crc32c;
verify-alg crc32c;
}
disk {
fencing resource-only;
# no-disk-barrier;
# no-disk-flushes;
}
handlers
{
split-brain "/usr/lib/drbd/notify-split-brain.sh root";
out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
}
protocol C;
on srv10 {
device /dev/drbd1;
disk /dev/sdf1;
address 192.168.27.11:7789;
meta-disk internal;
}
on srv11 {
device /dev/drbd1;
disk /dev/sdf1;
address 192.168.27.12:7789;
meta-disk internal;
}
}
...
The problem is that after initial synchronization if I run "drbdadm
verify r1", I get a bunch of out-of-sync blocks. Then I do disconnect
and connect of this resource and run "drbdadm verify r1" again and then
I again do get a bunch of out-of-sync blocks. Some of them are
false-positives and some of them are really out of sync as dd shows
(both with iflags=direct and without). And what is important: there is
no write operations on this device. Nothing is written to it, but I get
a bunch of out-of-sync blocks any time I run verify and resync DRBD.
[root at srv10 vvk]# drbdadm verify r1
[root at srv10 vvk]# cat /proc/drbd
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by
phil at fat-tyre, 2011-01-28 12:17:35
1: cs:VerifyS ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1568452 nr:0 dw:4 dr:39777412 al:1 bm:1300 lo:333 pe:0 ua:367
ap:0 ep:1 wo:b oos:484868
[>....................] verified: 2.3% (9328/9536)M
finish: 0:01:29 speed: 106,784 (106,784) want: 102,400 K/sec
[root at srv10 vvk]# cat /proc/drbd
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by
phil at fat-tyre, 2011-01-28 12:17:35
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1568452 nr:0 dw:4 dr:49330856 al:1 bm:1300 lo:0 pe:0 ua:0 ap:0
ep:1 wo:b oos:724448
[root at srv10 vvk]# dmesg | tail -10
[ 2267.737437] block drbd1: helper command: /sbin/drbdadm
before-resync-source minor-1 exit code 0 (0x0)
[ 2267.737445] block drbd1: conn( WFBitMapS -> SyncSource ) pdsk(
Consistent -> Inconsistent )
[ 2267.737455] block drbd1: Began resync as SyncSource (will sync 724448
KB [181112 bits set]).
[ 2267.737494] block drbd1: updated sync UUID
5B5B2B2EE43F8757:7BE37645CD25FF6D:7BE27645CD25FF6D:0001000000000000
[ 2276.569106] block drbd1: Resync done (total 8 sec; paused 0 sec;
90556 K/sec)
[ 2276.569113] block drbd1: 51 % had equal check sums, eliminated:
371216K; transferred 353232K total 724448K
[ 2276.569120] block drbd1: updated UUIDs
5B5B2B2EE43F8757:0000000000000000:7BE37645CD25FF6D:7BE27645CD25FF6D
[ 2276.569129] block drbd1: conn( SyncSource -> Connected ) pdsk(
Inconsistent -> UpToDate )
[ 2276.569727] block drbd1: bitmap WRITE of 52 pages took 0 jiffies
[ 2276.569750] block drbd1: 0 KB (0 bits) marked out-of-sync by on disk
bit-map.
[root at srv10 vvk]# cat /proc/drbd
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by
phil at fat-tyre, 2011-01-28 12:17:35
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:353232 nr:0 dw:4 dr:50055636 al:1 bm:1410 lo:0 pe:0 ua:0 ap:0
ep:1 wo:b oos:0
[root at srv10 vvk]# drbdadm verify r1
[root at srv10 vvk]# cat /proc/drbd
version: 8.3.10 (api:88/proto:86-96)
GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by
phil at fat-tyre, 2011-01-28 12:17:35
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:353232 nr:0 dw:4 dr:59822788 al:1 bm:1410 lo:0 pe:0 ua:0 ap:0
ep:1 wo:b oos:470532
Is this normal DRBD behaviour?
--
With best regards
Vladimir Kuklin
Senior Linux Administrator
LLC "Mango Telecom",
tel.: +7 (495) 540-44-44 ext. 3576
mob.: +7 (925) 083-86-53
email: vl.kuklin at mangotele.com
www.mango.ru