DRBD 8.4.11 - disabling barriers
Sanjay Agrawal
sagrawal at janestreet.com
Wed Aug 21 15:26:35 CEST 2024
Hi,
We are using DRBD as part of the ganeti. But it seems by default DRBD sets
disk barriers to flush. Based on our testing, we think due to this we are
getting around half write speed on the VM compared to hypervisor. We tested
by disabling all barriers including disk-drain, which brings our write
speed closer to hypervisor. But man pages from drbdsetup clearly states not
to disable the disk-drain. We are wondering if we can get more information
about why it is recommended. Our understanding is once the writes are
passed to DRBD device from qemu (VM), nothing under that layer should
reorder writes and write request order will be maintained till it is
passed to RAID controller on hypervisor side. So disabling all barriers
should have no impact wrt to data consistency. LVM device is exclusive to
each VM. Same is happening on the secondary node. We are using protocol C
for replication. Since DRBD also uses TCP, the write order to the secondary
node would be maintained as well. Any information related to recommendation
would be very helpful.
Following is out setup
Hypervisor
DRBD device ------------ presented to VM as disk, which is used under
qemu
|
LVM
|
DISK (SD device)
|
RAID controller (battery back cache) DELL PERC H740P or similar RAID
controllers.
|
SSDs
Out testing using fio
┌────────────┬─────────────┬───────────────┬──────────────┬───────────────┬─────────────────┬───────────────┬──────────────────────────────────────────────────────────────────────────────────────────┐
│ hostname │ randread 4K │ seq read 256K │ seq read 4K │ randwrite 4K │
seq write 256K │ seq write 4K │
comment │
├────────────┼─────────────┼───────────────┼──────────────┼───────────────┼─────────────────┼───────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
*│ Hypervisor │ 1170MiB/s │ 6733MiB/s │ 527MiB/s │ 378MiB/s
│ 4603MiB/s │ 591MiB/s │ R6515
RAID6 write policy wb but linux show write through ││ Hypervisor │
1157MiB/s │ 7029MiB/s │ 527MiB/s │ 381MiB/s │ 4366MiB/s
│ 591MiB/s │ R6515 RAID6 write policy wb but
linux show write through 2nd run │*│ VM │ 587MiB/s │
6917MiB/s │ 2199MiB/s │ 159MiB/s │ 1178MiB/s │ 291MiB/s
│ VM running on Hypervisor
default setting │
│ VM │ 585MiB/s │ 6942MiB/s │ 2126MiB/s │ 175MiB/s │
1245MiB/s │ 285MiB/s │
VM running on Hypervisor default setting │
│ VM │ 590MiB/s │ 6944MiB/s │ 1911MiB/s │ 175MiB/s │
1270MiB/s │ 292MiB/s │ VM running on
Hypervisor flush disabled on cluster side run 1 │
│ VM │ 581MiB/s │ 6934MiB/s │ 2096MiB/s │ 75.6MiB/s │
1187MiB/s │ 382MiB/s │ VM running on
Hypervisor flush disabled on cluster side run 2 │
│ VM │ 586MiB/s │ 6887MiB/s │ 2261MiB/s │ 129MiB/s │
1239MiB/s │ 385MiB/s │ VM running on
Hypervisor flush+barrier disabled on cluster side run 1 │
│ VM │ 588MiB/s │ 6915MiB/s │ 2210MiB/s │ 136MiB/s │
1242MiB/s │ 250MiB/s │ VM running on
Hypervisor flush+barrier disabled on cluster side run 2 │
*│ VM │ 591MiB/s │ 6925MiB/s │ 1753MiB/s │ 299MiB/s
│ 2664MiB/s │ 499MiB/s │ VM running on Hypervisor
flush+barrier and drain disabled on cluster side run 1 │*│ VM │
588MiB/s │ 6927MiB/s │ 1583MiB/s │ 301MiB/s │ 1795MiB/s
│ 1001MiB/s │ VM running on Hypervisor flush+barrier and drain
disabled on cluster side run 2 │
│ VM │ 588MiB/s │ 6932MiB/s │ 1890MiB/s │ 295MiB/s │
2242MiB/s │ 885MiB/s │ VM running on Hypervisor
flush+barrier and drain disabled on cluster side run 3 │
│ VM │ 591MiB/s │ 6916MiB/s │ 1638MiB/s │ 301MiB/s │
2660MiB/s │ 923MiB/s │ VM running on
Hypervisor flush+barrier and drain disabled after reboot │
│ VM │ 585MiB/s │ 6921MiB/s │ 1845MiB/s │ 296MiB/s │
2206MiB/s │ 888MiB/s │ VM running on Hypervisor
flush+barrier and drain disabled after reboot run 2 │
│ VM │ 581MiB/s │ 6948MiB/s │ 1464MiB/s │ 296MiB/s │
2754MiB/s │ 884MiB/s │ VM running on Hypervisor flush+barrier
and drain disabled also set scheduler to none │
│ │ │ │ │ │
│ │
run 1 │
│ VM │ 588MiB/s │ 6936MiB/s │ 1481MiB/s │ 293MiB/s │
2723MiB/s │ 842MiB/s │ VM running on Hypervisor flush+barrier
and drain disabled also set scheduler to none │
│ │ │ │ │ │
│ │
run 2 │
│ VM │ 581MiB/s │ 6880MiB/s │ 2080MiB/s │ 176MiB/s │
1246MiB/s │ 347MiB/s │ VM running on
Hypervisor default + --max-buffers=262144 - run1 │
│ VM │ 585MiB/s │ 6953MiB/s │ 2071MiB/s │ 159MiB/s │
1214MiB/s │ 259MiB/s │ VM running on
Hypervisor default + --max-buffers=262144 - run2 │
│ VM │ 588MiB/s │ 6880MiB/s │ 1976MiB/s │ 74.1MiB/s │
1243MiB/s │ 263MiB/s │ VM running on Hypervisor
default + max-epoch-size/max-buffers=262144 - run1 │
│ VM │ 579MiB/s │ 6953MiB/s │ 2099MiB/s │ 154MiB/s │
1181MiB/s │ 269MiB/s │ VM running on Hypervisor
default + max-epoch-size/max-buffers=262144 - run2 │
│ VM │ 585MiB/s │ 6887MiB/s │ 2054MiB/s │ 145MiB/s │
1295MiB/s │ 738MiB/s │ VM running on
Hypervisor default + --unplug-watermark=1024 - run1 │
│ VM │ 583MiB/s │ 6937MiB/s │ 2103MiB/s │ 170MiB/s │
1242MiB/s │ 358MiB/s │ VM running on
Hypervisor default + --unplug-watermark=1024 - run2 │
│ VM │ 581MiB/s │ 6884MiB/s │ 1747MiB/s │ 174MiB/s │
1169MiB/s │ 327MiB/s │ VM running on
Hypervisor default + --unplug-watermark=131072 - run1 │
│ VM │ 579MiB/s │ 6940MiB/s │ 2036MiB/s │ 137MiB/s │
1240MiB/s │ 251MiB/s │ VM running on
Hypervisor default + --unplug-watermark=131072 - run2 │
│ VM │ 586MiB/s │ 6883MiB/s │ 1732MiB/s │ 146MiB/s │
1221MiB/s │ 276MiB/s │ VM running on Hypervisor
default+front_merges=0 read_expire=150 write_expire=1500 │
│ VM │ 578MiB/s │ 6948MiB/s │ 2016MiB/s │ 179MiB/s │
1171MiB/s │ 262MiB/s │ VM running on Hypervisor
default+front_merges=0 read_expire=150 write_expire=1500 │
│ VM │ 582MiB/s │ 6924MiB/s │ 1807MiB/s │ 171MiB/s │
1177MiB/s │ 380MiB/s │ VM running on Hypervisor
default+front_merges=0 read_expire=50 write_expire=150 │
│ VM │ 579MiB/s │ 6929MiB/s │ 1891MiB/s │ 175MiB/s │
1244MiB/s │ 302MiB/s │ VM running on Hypervisor
default+front_merges=0 read_expire=50 write_expire=150 run2 │
└────────────┴─────────────┴───────────────┴──────────────┴───────────────┴─────────────────┴───────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘
Thanks,
--
Sanjay Agrawal
https://corpdir:8443/user/sagrawal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20240821/deb65eaf/attachment-0001.html>
More information about the drbd-user
mailing list