[DRBD-user] drbd sync issues on *all* deployments

[Harald Dunkel]

Hi folks,

I ran a drbdadm verify on all my drbd clusters (2 nodes each). It
is still running, but by now the huge oos numbers look pretty
scary:


il06:~# ssh node24a cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7

  1: cs:VerifyS ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
     ns:0 nr:281562940 dw:281562940 dr:1143238964 al:0 bm:0 lo:0 pe:3696 ua:16 ap:0 ep:1 wo:f oos:250816
	[================>...] verified: 86.2% (4238132/30520400)M
	finish: 6:17:03 speed: 191,816 (164,868) want: 191,280 K/sec


il06:~# ssh mydb01a cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7

  1: cs:VerifyT ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
     ns:1687031744 nr:134488 dw:1687230080 dr:1214075705 al:69078 bm:0 lo:352 pe:416 ua:352 ap:0 ep:1 wo:f oos:232568
	[===============>....] verified: 83.7% (218016/1329440)M
	finish: 2:34:08 speed: 24,124 (25,388) K/sec
  2: cs:VerifyS ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
     ns:0 nr:11554944 dw:11554944 dr:1402828076 al:0 bm:0 lo:217 pe:0 ua:714 ap:0 ep:1 wo:f oos:48884
	[=================>..] verified: 94.8% (76236/1446184)M
	finish: 0:35:50 speed: 36,280 (31,296) want: 33,080 K/sec


il06:~# ssh mydb02a cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7

  1: cs:VerifyT ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
     ns:740391624 nr:285874452 dw:1026266084 dr:1530185919 al:648235 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:337696
	[==>.................] verified: 18.5% (5982824/7339804)M
	finish: 53:05:24 speed: 32,040 (31,112) K/sec
  2: cs:VerifyS ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
     ns:5061656 nr:23922468 dw:2050697240 dr:1770666372 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
	[===>................] verified: 22.5% (5933012/7655380)M
	finish: 42:04:46 speed: 40,088 (39,528) want: 92,160 K/sec


il06:~# ssh srvl060a cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7

  1: cs:VerifyT ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
     ns:1421874360 nr:0 dw:1421874360 dr:1172461205 al:5593028 bm:0 lo:1895 pe:633 ua:14826 ap:0 ep:1 wo:f oos:1700
         [==>.................] verified: 16.9% (9473768/11395236)M
         finish: 18:52:05 speed: 142,804 (192,312) K/sec
  2: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
     ns:0 nr:553645172 dw:553645172 dr:179644044 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0


il06:~# ssh nasl006a cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: 32DFEF1F0DADCBF174877F7

  1: cs:VerifyT ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
     ns:436146808 nr:231152 dw:437189280 dr:295512896 al:6421234 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:3748
         [==>.................] verified: 16.9% (18920872/22753312)M
         finish: 64:59:00 speed: 82,800 (72,548) K/sec


All hosts are attached to an UPS, the connection between both peers
is redundant.

What can I do to increase reliability? Am I missing some kernel patches?


Environment:

Debian 11, amd64

il06:~# ssh nasl006a drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ baaca8a080dc54652f57da4bafb2dce51dfe9f68\ reproducible\ build\,\ 2020-09-29\ 09:05:36
DRBDADM_API_VERSION=1
DRBD_KERNEL_VERSION_CODE=0x08040b
DRBDADM_VERSION_CODE=0x090f00
DRBDADM_VERSION=9.15.0

il06:~# ssh nasl006a uname -a
Linux nasl006a.example.com 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux



Regards

Harri