Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hello,
I started looking at DRBD as a HA ISCSI target. I am experiencing very poor
performance and decided to run some tests. My current setup is as follows:
Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GH
CentoS 6.5 - 2.6.32-431.17.1.el6.x86_64
drbd version: 8.3.16 (api:88/proto:86-97)
md RAID10 using 7200rpm drives
The 2 drbd nodes are synced using an intel 82579LM Gigabit card
I have created an logical drive using LVM and configured a couple drbd
resources on top of that. drbd0 is my iscsi configuration file, which is
shared between the 2 nodes and drbd1 is a 1.75TB ISCSI target.
I run heartbeat on the two nodes and expose a virtual IP to the ISCSI
initiators.
Originally I was running ISCSI with write-cache off (for data integrity
reasons) but have recently switched to write-cache on during testing (with
little to no gain).
My major concern is the extremely high latency test results I got when when
dd against drbd0 mounted on the primary node.
dd if=/dev/zero of=./testbin bs=512 count=1000 oflag=direct
512000 bytes (512 kB) copied, 32.3254 s, 15.8 kB/s
I have pinged the second node as a very basic network latency test and get
0.209ms response time. I have also run the same test on both nodes with
drbd disconnected (or on partitions not associated with drbd) and get
typical results:
node1
dd if=/dev/zero of=./testbin bs=512 count=1000 oflag=direct
12000 bytes (512 kB) copied, 0.153541 s, 3.3 MB/s
node2
dd if=/dev/zero of=~/testbin bs=512 count=1000 oflag=direct
512000 bytes (512 kB) copied, 0.864994 s, 592 kB/s
512000 bytes (512 kB) copied, 0.328994 s, 1.6 MB/s
node2's latency (without drbd connected) is inconsistent but always falls
between those two ranges.
These tests were run with no ISCSI targets exposed, no initiators
connected, essentially on an idle system.
My question is why are my drbd connected latency tests showing results 35
to 100 times slower than my results when dbrd is not connected (or against
partitions not backed by drbd)?
This seems to be the source of my horrible performance on the ISCSI targs
(300~900 K/sec dd writes on the initiators) and very high iowait (35-75%)
on mildly busy initiators.
Any advice pointers, etc. would be highly appreciated. I have already tried
numerous performance tuning settings (suggested by the drbd manual). But I
am open to any suggestion and will try anything again if it might solve my
problem.
Here are the important bits of my current drbd.conf
net {
cram-hmac-alg sha1;
shared-secret "password";
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 0;
}
syncer {
rate 100M;
verify-alg sha1;
al-extents 3389;
}
I've played with the watermark setting and a few others and latency only
seems to get worse or stay where it's at.
Thank you,
Bret
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20140604/6dfc5c6d/attachment.htm>