[DRBD-user] Having a high load phenomenon with DRBD 9.2.5, DIRECT-IO and asynclib

[Andreas Mock]

Hi all,

to get a feeling for IOPS for a MySQL database on top of DRBD storage
I run the following command:

fio --rw=randwrite --name=IOPS-write --bs=16k --direct=1 --numjobs=12 --ioengine=libaio --iodepth=32 --refill_buffers --group_reporting --size=4G

blocksize = 16k
paralle processes = 12 (12 instances of fio)
directIO
libaio (async IO)
iodepth = 32 (32 open requests per process)

System environment:
- AlmaLinux 9.2
- DRBD 9.2.5
- 10GBit-Interconnect (without Switch)
- Server: 64 Logical Processors; System idle

After starting the above command anything seems to be normal.
io utilization on the receiving server gets high as expected for 
many little writes (16k writes).

After some runtime load increases suddenly to a high value.
After some investigation I could find out that there are very short
(< 0.5 sec) but high peeks of many kworker threads in process
state 'D' which look like [kworker/35:252+dio/drbd0] shown by
ps ax. So, load calculation is done more or less only based
on a very high amount of kernel threads waiting for IO.

Currently it seems that the inflow of async direct IO requests
is higher than the drain of completed io request and kworker threads
start to block themself somewhere somehow.

Some numbers in my test:
machine idle:
# cat /proc/loadavg
0.04 0.02 0.00 1/706 233272
# ps ax | wc -l
632
# ps ax | grep '\[kwo' | wc -l
272
# ps ax | grep '\[kwo' | grep -v 'events_highpri' | wc -l
159
ps ax | grep '\[kwo' | grep 'block'  | wc -l
16


After starting the above command I query the processes
with the following command in intervals:

# ps ax | grep '\[kwo' | grep 'dio/drbd0]' | wc -l
4390

The highest amount of kworker threads related to drbd
is 4390 while the above test is running.


Here my questions:
a) Is the behaviour I can see here an expected behaviour?
b) How can I investigate this kind of peek stalling of
the kworker threads (all waitimg for IO => high load calculation)
c) Are there any possibilities to tweak the system (e.g. DRBD
settings) to avoid this massive spawning of kworker threads/procs.

Sidenotes:
1) Running the same test on the same underlying disksystem without
DRBD let the amount of kworker threads related to IO raise upto 1470.
The load is never getting high. 

2) I have an old CentOS 7 based system with MySQL running on top
of DRBD. I never had load issues with it. I'm pretty sure that
updatimg from DRBD 9.0.x to 9.2.x brought the load issues
as mentioned above.

I hope someone can give some hints. Probably my case is interesting
enough for the Linbit developers.

Thank you in advance.

Best regards
Andreas