Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
I am playing around with an asynchronous setup of drbd. In my
application there are mostly reads, but from time to time large batches
of data (10-100s GB) will be written. Write performance is very
important here - the loss of the current batch of data or parts of it is
acceptable on primary node failure, as long as the fs stays alive in
general.
My frist idea was to switch to protocol A without touching any other
parameters. The write performance of this setup was quite disapointing
with about 30MB/s. Thats less than 50% of the sustained write rate of
each node (simple dual core PCs with desktop hdds, writing between 130
and 70 MB/s). The link between the nodes is an exclusive Gigabit cable,
iperf'd with about 900MBit.
So I tried to tune some parameters:
-----------------------------------------------------------------------------------------
global {
usage-count yes;
}
common {
disk {
# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
resync-rate 200M;
al-extents 389;
}
net {
protocol A;
sndbuf-size 1M;
on-congestion pull-ahead;
congestion-fill 800k;
congestion-extents 300;
}
}
resource main_disk {
device /dev/drbd0;
meta-disk internal;
on nas2 {
address 10.1.0.2:7789;
disk /dev/sdb1;
}
on nas3 {
address 10.1.0.3:7789;
disk /dev/sdb1;
}
}
-----------------------------------------------------------------------------------------
Unfortunately, everything got worse: this configuration shows a very
inconsistent performance, from 10..60MB/s of the capabilities of the
harddisks. I monitored /proc/drbd therefore:
dd if=/dev/zero of=/dev/drbd0 bs=1M count=20000&
while true; do cat /proc/drbd; sleep 0.2; done
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38436460 nr:0 dw:39806236 dr:16508544 al:9720 bm:875 lo:1470
pe:0 ua:1 ap:1466 ep:1 wo:d oos:1645552
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38436460 nr:0 dw:39812380 dr:16511616 al:9722 bm:875 lo:1981
pe:0 ua:1 ap:1978 ep:1 wo:d oos:1653744
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:ProtocolError ro:Primary/Unknown ds:UpToDate/Inconsistent A r-----
ns:38437996 nr:0 dw:39817500 dr:16513664 al:9723 bm:875 lo:1725
pe:0 ua:0 ap:1722 ep:1 wo:d oos:1656304
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:ProtocolError ro:Primary/Unknown ds:UpToDate/Inconsistent A r---d-
ns:38437996 nr:0 dw:39824388 dr:16513680 al:9724 bm:875 lo:2 pe:0
ua:0 ap:0 ep:1 wo:d oos:1657844
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/Inconsistent A r-----
ns:38437996 nr:0 dw:39829504 dr:16513680 al:9725 bm:875 lo:771 pe:0
ua:0 ap:769 ep:1 wo:d oos:1662960
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/Inconsistent A r-----
ns:38437996 nr:0 dw:39834624 dr:16513680 al:9726 bm:875 lo:515 pe:0
ua:0 ap:513 ep:1 wo:d oos:1668080
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:WFReportParams ro:Primary/Unknown ds:UpToDate/Inconsistent A r---d-
ns:38437996 nr:0 dw:39840768 dr:16513680 al:9728 bm:875 lo:1028
pe:0 ua:0 ap:1025 ep:1 wo:d oos:1674224
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38437996 nr:0 dw:39847120 dr:16513680 al:9729 bm:875 lo:463 pe:0
ua:0 ap:461 ep:1 wo:d oos:1682416
[>....................] sync'ed: 0.3% (1682416/1682416)K
finish: 1:10:06 speed: 0 (0) K/sec
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38438996 nr:0 dw:39854288 dr:16513680 al:9731 bm:875 lo:720
pe:1074 ua:0 ap:1799 ep:1 wo:d oos:1685312
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38443292 nr:0 dw:39862340 dr:16513728 al:9733 bm:875 lo:755 pe:0
ua:1 ap:1957 ep:1 wo:d oos:1693504
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38443292 nr:0 dw:39868484 dr:16513792 al:9735 bm:875 lo:1268
pe:0 ua:1 ap:1264 ep:1 wo:d oos:1701696
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38443292 nr:0 dw:39874628 dr:16516352 al:9737 bm:875 lo:1779
pe:0 ua:0 ap:1776 ep:1 wo:d oos:1709888
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38443292 nr:0 dw:39881796 dr:16518400 al:9739 bm:875 lo:2035
pe:0 ua:1 ap:2032 ep:1 wo:d oos:1718080
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38443292 nr:0 dw:39890500 dr:16520448 al:9741 bm:875 lo:1907
pe:0 ua:0 ap:1904 ep:1 wo:d oos:1726272
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38443292 nr:0 dw:39899204 dr:16522496 al:9743 bm:875 lo:1779
pe:0 ua:0 ap:1776 ep:1 wo:d oos:1734464
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
dag at Build64R6, 2012-09-06 08:16:10
0: cs:Ahead ro:Primary/Secondary ds:UpToDate/Inconsistent A r-----
ns:38443292 nr:0 dw:39903300 dr:16523520 al:9744 bm:875 lo:1780
pe:0 ua:1 ap:1776 ep:1 wo:d oos:1738560
Any ideas or suggestions?
Regards,
Holger
--
Dipl.-Wirtsch.-Inf. Holger Steinhaus
Softwareentwicklung und Beratung
Straacher Feldstr. 3, 06889 Wittenberg
Festnetz 034929-65950
Fax 034929-65951
mobil 0151-50992485
eMail holger at steinhaus-home.de