[DRBD-user] DRBD9 performance with RAM drives on Linux
Majzerowicz-Jaszcz, Krzysztof
krzysztof.majzerowicz-jaszcz at intel.com
Tue Dec 7 13:24:41 CET 2021
Hello,
We would like to properly configure DRBD9 using RAM disk bdev's as an underlying devices on both ends for testing purposes.
So far we have seen quite unsatisfactory results using Linux kernel's BRD module for RAM disks and the following DRBD resource config:
resource no_cas_test {
device /dev/drbd0;
on lab_1 {
disk /dev/ram0; #cache dev
meta-disk internal;
address XXXX
}
on lab_2 {
disk /dev/ram0;
meta-disk internal;
address XXXX
}
}
These are HW machines running RHEL and the raw performance we get from RAM disks is over 16GiB/s (over 4400k IOPS) using fio:
# fio --filename=/dev/ram0 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=24 --name=4kiops_randwrite --group_reporting --size=4G
4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 24 processes
Jobs: 16 (f=14): [w(1),_(1),w(5),_(1),w(1),_(1),f(2),_(5),w(7)][85.7%][w=15.3GiB/s][w=4023k IOPS][eta 00m:01s]
4kiops_randwrite: (groupid=0, jobs=24): err= 0: pid=3976: Fri Dec 3 15:53:43 2021
write: IOPS=4402k, BW=16.8GiB/s (18.0GB/s)(96.0GiB/5717msec); 0 zone resets
However, as soon as we start a single, primary DRBD node on lab_1, we only get about 1300MiB/s (350k IOPS).
# fio --filename=/dev/drbd0 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=24 --name=4kiops_randwrite --group_reporting --size=4G
4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 24 processes
Jobs: 24 (f=24): [w(24)][98.6%][w=1379MiB/s][w=353k IOPS][eta 00m:01s]
4kiops_randwrite: (groupid=0, jobs=24): err= 0: pid=4274: Fri Dec 3 15:59:22 2021
write: IOPS=351k, BW=1371MiB/s (1438MB/s)(96.0GiB/71704msec); 0 zone resets
Now if we use physical NVMe drives, there is no difference in performance - both direct hardware (/dev/nvmeXXXXX) test and DRBD test yield maximum drive write performance:
Non-DRBD test:
# fio --filename=/dev/nvme1n1p1 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=12 --name=4kiops_randwrite --group_reporting --size=4G
4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 12 processes
Jobs: 1 (f=1): [_(11),w(1)][98.7%][w=607MiB/s][w=155k IOPS][eta 00m:01s]
4kiops_randwrite: (groupid=0, jobs=12): err= 0: pid=5312: Fri Dec 3 12:34:05 2021
write: IOPS=165k, BW=646MiB/s (677MB/s)(48.0GiB/76092msec); 0 zone resets
slat (nsec): min=1419, max=310283, avg=3730.55, stdev=2879.91
clat (usec): min=5, max=23109, avg=4231.60, stdev=2497.87
lat (usec): min=9, max=23113, avg=4235.39, stdev=2497.50
clat percentiles (usec):
| 1.00th=[ 86], 5.00th=[ 545], 10.00th=[ 1074], 20.00th=[ 1991],
| 30.00th=[ 2638], 40.00th=[ 3425], 50.00th=[ 4080], 60.00th=[ 4817],
| 70.00th=[ 5538], 80.00th=[ 6063], 90.00th=[ 7177], 95.00th=[ 8848],
| 99.00th=[11469], 99.50th=[12256], 99.90th=[14091], 99.95th=[14877],
| 99.99th=[16909]
bw ( KiB/s): min=351792, max=1961536, per=100.00%, avg=725184.65, stdev=26445.60, samples=1653
iops : min=87948, max=490384, avg=181296.12, stdev=6611.40, samples=1653
lat (usec) : 10=0.05%, 20=0.23%, 50=0.44%, 100=0.38%, 250=1.11%
lat (usec) : 500=2.36%, 750=2.48%, 1000=2.23%
lat (msec) : 2=10.84%, 4=28.80%, 10=48.27%, 20=2.80%, 50=0.01%
cpu : usr=3.19%, sys=7.10%, ctx=8562282, majf=0, minf=718
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,12582912,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=646MiB/s (677MB/s), 646MiB/s-646MiB/s (677MB/s-677MB/s), io=48.0GiB (51.5GB), run=76092-76092msec
Disk stats (read/write):
nvme1n1: ios=205/12567629, merge=0/0, ticks=197/53105077, in_queue=46882592, util=100.00%
DRBD test:
# fio --filename=/dev/drbd0 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=12 --name=4kiops_randwrite --group_reporting --size=4G
4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 12 processes
Jobs: 1 (f=1): [_(6),w(1),_(5)][95.1%][w=636MiB/s][w=163k IOPS][eta 00m:04s]
4kiops_randwrite: (groupid=0, jobs=12): err= 0: pid=5104: Fri Dec 3 12:28:03 2021
write: IOPS=162k, BW=634MiB/s (665MB/s)(48.0GiB/77474msec); 0 zone resets
slat (nsec): min=1951, max=198869k, avg=49104.65, stdev=79888.62
clat (usec): min=7, max=207233, avg=4469.84, stdev=2174.11
lat (usec): min=13, max=207304, avg=4519.00, stdev=2178.99
clat percentiles (usec):
| 1.00th=[ 498], 5.00th=[ 1500], 10.00th=[ 1860], 20.00th=[ 2376],
| 30.00th=[ 3064], 40.00th=[ 3818], 50.00th=[ 4555], 60.00th=[ 5211],
| 70.00th=[ 5669], 80.00th=[ 6128], 90.00th=[ 6849], 95.00th=[ 7832],
| 99.00th=[10159], 99.50th=[11076], 99.90th=[13435], 99.95th=[15270],
| 99.99th=[27395]
bw ( KiB/s): min=411961, max=1893680, per=100.00%, avg=677991.74, stdev=20345.92, samples=1765
iops : min=102987, max=473420, avg=169496.99, stdev=5086.49, samples=1765
lat (usec) : 10=0.01%, 20=0.02%, 50=0.04%, 100=0.06%, 250=0.19%
lat (usec) : 500=0.76%, 750=1.45%, 1000=0.74%
lat (msec) : 2=9.48%, 4=29.65%, 10=56.43%, 20=1.15%, 50=0.03%
lat (msec) : 250=0.01%
cpu : usr=1.90%, sys=62.47%, ctx=811802, majf=0, minf=1305
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,12582912,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=634MiB/s (665MB/s), 634MiB/s-634MiB/s (665MB/s-665MB/s), io=48.0GiB (51.5GB), run=77474-77474msec
Disk stats (read/write):
drbd0: ios=0/12560419, merge=0/0, ticks=0/47859223, in_queue=47859223, util=100.00%, aggrios=0/12582912, aggrmerge=0/0, aggrticks=0/47319227, aggrin_queue=41223461, aggrutil=100.00%
nvme1n1: ios=0/12582912, merge=0/0, ticks=0/47319227, in_queue=41223461, util=100.00%
We have seen even worse performance on the VM's running DRBD9 on RAM disks. Is that behavior expected?
Best regards,
Krzysztof Majzerowicz-Jaszcz
---------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20211207/d2d0c86f/attachment-0001.htm>
More information about the drbd-user
mailing list