[DRBD-user] DRBD9 performance with RAM drives on Linux

Majzerowicz-Jaszcz, Krzysztof krzysztof.majzerowicz-jaszcz at intel.com
Tue Dec 7 13:24:41 CET 2021


Hello,

We would like to properly configure DRBD9 using RAM disk bdev's as an underlying devices on both ends for testing purposes.

So far we have seen quite unsatisfactory results using Linux kernel's BRD module for RAM disks and the following DRBD resource config:

resource no_cas_test {
        device /dev/drbd0;
        on lab_1 {
                disk /dev/ram0; #cache dev
                meta-disk internal;
                address   XXXX
        }
        on lab_2  {
                disk /dev/ram0;
                meta-disk internal;
                address XXXX
        }
}

These are HW machines running RHEL and the raw performance we get from RAM disks is over 16GiB/s (over 4400k IOPS) using fio:

# fio --filename=/dev/ram0 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=24 --name=4kiops_randwrite --group_reporting --size=4G

4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 24 processes
Jobs: 16 (f=14): [w(1),_(1),w(5),_(1),w(1),_(1),f(2),_(5),w(7)][85.7%][w=15.3GiB/s][w=4023k IOPS][eta 00m:01s]
4kiops_randwrite: (groupid=0, jobs=24): err= 0: pid=3976: Fri Dec  3 15:53:43 2021
  write: IOPS=4402k, BW=16.8GiB/s (18.0GB/s)(96.0GiB/5717msec); 0 zone resets

However, as soon as we start a single, primary DRBD node on lab_1, we only get about 1300MiB/s (350k IOPS).
# fio --filename=/dev/drbd0 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=24 --name=4kiops_randwrite --group_reporting --size=4G
4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 24 processes
Jobs: 24 (f=24): [w(24)][98.6%][w=1379MiB/s][w=353k IOPS][eta 00m:01s]
4kiops_randwrite: (groupid=0, jobs=24): err= 0: pid=4274: Fri Dec  3 15:59:22 2021
  write: IOPS=351k, BW=1371MiB/s (1438MB/s)(96.0GiB/71704msec); 0 zone resets

Now if we use physical NVMe drives, there is no difference in performance - both direct hardware (/dev/nvmeXXXXX) test and DRBD test yield maximum drive write performance:

Non-DRBD test:
# fio --filename=/dev/nvme1n1p1 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=12 --name=4kiops_randwrite --group_reporting --size=4G
4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 12 processes
Jobs: 1 (f=1): [_(11),w(1)][98.7%][w=607MiB/s][w=155k IOPS][eta 00m:01s]
4kiops_randwrite: (groupid=0, jobs=12): err= 0: pid=5312: Fri Dec  3 12:34:05 2021
  write: IOPS=165k, BW=646MiB/s (677MB/s)(48.0GiB/76092msec); 0 zone resets
    slat (nsec): min=1419, max=310283, avg=3730.55, stdev=2879.91
    clat (usec): min=5, max=23109, avg=4231.60, stdev=2497.87
     lat (usec): min=9, max=23113, avg=4235.39, stdev=2497.50
    clat percentiles (usec):
     |  1.00th=[   86],  5.00th=[  545], 10.00th=[ 1074], 20.00th=[ 1991],
     | 30.00th=[ 2638], 40.00th=[ 3425], 50.00th=[ 4080], 60.00th=[ 4817],
     | 70.00th=[ 5538], 80.00th=[ 6063], 90.00th=[ 7177], 95.00th=[ 8848],
     | 99.00th=[11469], 99.50th=[12256], 99.90th=[14091], 99.95th=[14877],
     | 99.99th=[16909]
   bw (  KiB/s): min=351792, max=1961536, per=100.00%, avg=725184.65, stdev=26445.60, samples=1653
   iops        : min=87948, max=490384, avg=181296.12, stdev=6611.40, samples=1653
  lat (usec)   : 10=0.05%, 20=0.23%, 50=0.44%, 100=0.38%, 250=1.11%
  lat (usec)   : 500=2.36%, 750=2.48%, 1000=2.23%
  lat (msec)   : 2=10.84%, 4=28.80%, 10=48.27%, 20=2.80%, 50=0.01%
  cpu          : usr=3.19%, sys=7.10%, ctx=8562282, majf=0, minf=718
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,12582912,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=646MiB/s (677MB/s), 646MiB/s-646MiB/s (677MB/s-677MB/s), io=48.0GiB (51.5GB), run=76092-76092msec

Disk stats (read/write):
  nvme1n1: ios=205/12567629, merge=0/0, ticks=197/53105077, in_queue=46882592, util=100.00%

DRBD test:
# fio --filename=/dev/drbd0 --ioengine=libaio --direct=1 --rw=randwrite --bs=4k --iodepth=64 --numjobs=12 --name=4kiops_randwrite --group_reporting --size=4G
4kiops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 12 processes
Jobs: 1 (f=1): [_(6),w(1),_(5)][95.1%][w=636MiB/s][w=163k IOPS][eta 00m:04s]
4kiops_randwrite: (groupid=0, jobs=12): err= 0: pid=5104: Fri Dec  3 12:28:03 2021
  write: IOPS=162k, BW=634MiB/s (665MB/s)(48.0GiB/77474msec); 0 zone resets
    slat (nsec): min=1951, max=198869k, avg=49104.65, stdev=79888.62
    clat (usec): min=7, max=207233, avg=4469.84, stdev=2174.11
     lat (usec): min=13, max=207304, avg=4519.00, stdev=2178.99
    clat percentiles (usec):
     |  1.00th=[  498],  5.00th=[ 1500], 10.00th=[ 1860], 20.00th=[ 2376],
     | 30.00th=[ 3064], 40.00th=[ 3818], 50.00th=[ 4555], 60.00th=[ 5211],
     | 70.00th=[ 5669], 80.00th=[ 6128], 90.00th=[ 6849], 95.00th=[ 7832],
     | 99.00th=[10159], 99.50th=[11076], 99.90th=[13435], 99.95th=[15270],
     | 99.99th=[27395]
   bw (  KiB/s): min=411961, max=1893680, per=100.00%, avg=677991.74, stdev=20345.92, samples=1765
   iops        : min=102987, max=473420, avg=169496.99, stdev=5086.49, samples=1765
  lat (usec)   : 10=0.01%, 20=0.02%, 50=0.04%, 100=0.06%, 250=0.19%
  lat (usec)   : 500=0.76%, 750=1.45%, 1000=0.74%
  lat (msec)   : 2=9.48%, 4=29.65%, 10=56.43%, 20=1.15%, 50=0.03%
  lat (msec)   : 250=0.01%
  cpu          : usr=1.90%, sys=62.47%, ctx=811802, majf=0, minf=1305
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,12582912,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=634MiB/s (665MB/s), 634MiB/s-634MiB/s (665MB/s-665MB/s), io=48.0GiB (51.5GB), run=77474-77474msec

Disk stats (read/write):
    drbd0: ios=0/12560419, merge=0/0, ticks=0/47859223, in_queue=47859223, util=100.00%, aggrios=0/12582912, aggrmerge=0/0, aggrticks=0/47319227, aggrin_queue=41223461, aggrutil=100.00%
  nvme1n1: ios=0/12582912, merge=0/0, ticks=0/47319227, in_queue=41223461, util=100.00%

We have seen even worse performance on the VM's running DRBD9 on RAM disks. Is that behavior expected?

Best regards,
Krzysztof Majzerowicz-Jaszcz
---------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20211207/d2d0c86f/attachment-0001.htm>


More information about the drbd-user mailing list