<div dir="ltr"><div>Dear DRBD-users,</div><div><br></div><div>after using successfully for many years the in-kernel module for two node setup, we've decided to pass to the version 9.x to use create clusters with more nodes, but we are facing a serious problem. <br></div><div><br></div><div>Our setup is pretty simple, a three node cluster (HV01, HV02 and HV03 each one with a 4 port ethernet controller cross-connecting each node with a dedicated port), 2x2TB SAS HDD in each node (RAID-1, /dev/sda). We are using an extendend partition (/dev/sda4) to create the partitions for backing the DRBD devices, as follows:</div><div><br></div><div><span style="font-family:monospace">/dev/sda4 210126848 3905945599 3695818752 1.7T 5 Extended<br>/dev/sda5 210128896 566644735 356515840 170G 83 Linux<br>/dev/sda6 566646784 1405507583 838860800 400G 83 Linux<br></span> </div><div>The OS is the latest Gentoo Linux with a vanilla 6.0.9 kernel (compiled from the original source code, not patches applied), while we are using the DRBD kernel module Version: 9.2.1 (api:2/proto:86-121), compiled from source without any patch/change. The three nodes are identical in hw and sw configuration (the whole / partition is cloned, just changed the network parameters)<br></div><div><br></div><div>We create a resource as follows (no other configs present, global.conf empty):</div><div><br></div><div><span style="font-family:monospace">resource "r01"<br>{<br> protocol C;<br> volume 0 {<br> device minor 1;<br> disk "/dev/sda5";<br> meta-disk internal;<br> }<br><br> volume 1 {<br> device minor 2;<br> disk "/dev/sda6";<br> meta-disk internal;<br> }<br><br> on HV01 {<br> node-id 0;<br> }<br><br> on HV02 {<br> node-id 1;<br> }<br><br> on HV03 {<br> node-id 2;<br> }<br><br> connection {<br> host HV01 address <a href="http://172.31.255.1:7012">172.31.255.1:7012</a>;<br> host HV02 address <a href="http://172.31.255.2:7021">172.31.255.2:7021</a>;<br> }<br><br> connection {<br> host HV01 address <a href="http://172.30.255.1:7013">172.30.255.1:7013</a>;<br> host HV03 address <a href="http://172.30.255.3:7031">172.30.255.3:7031</a>;<br> }<br><br> connection {<br> host HV02 address <a href="http://172.29.255.2:7023">172.29.255.2:7023</a>;<br> host HV03 address <a href="http://172.29.255.3:7032">172.29.255.3:7032</a>;<br> }<br>}</span></div><div>=====================================================</div><div><br></div><div><span style="font-family:monospace"># drbdadm create-md --force r01 (on each node)</span></div><div><span style="font-family:monospace">
# drbdadm up r01 (on each node) <br></span></div><div><span style="font-family:monospace">
# drbdadm create-md --force r01 (on each node)
</span></div><div><span style="font-family:monospace"># drbdadm new-current-uuid --clear-bitmap r01/0 (on the primary node HV01)<br></span></div><div><span style="font-family:monospace">
# drbdadm new-current-uuid --clear-bitmap r01/1
(on the primary node HV01)
</span></div><div><span style="font-family:monospace">
# drbdadm primary r01 (on the primary node) <br></span></div><div><span style="font-family:monospace">
<span style="font-family:monospace">
# mkfs.ext4 /dev/drbd1 (on the primary node) <br></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace">
<span style="font-family:monospace"><span style="font-family:monospace">
# mount /dev/drbd1 /mnt/r01 (on the primary node) <br></span></span></span></span></div><div><br></div><div><br></div><div><br>
</div><div>Everything is fine up to this point. The status is the following:</div><div><br></div><div><br></div><div><br></div><div><span style="font-family:monospace">HV01 ~ # drbdsetup status --verbose --statistics<br>r01 node-id:0 role:Primary suspended:no force-io-failures:no<br> write-ordering:flush<br> volume:0 minor:1 disk:UpToDate backing_dev:/dev/sda5 quorum:yes<br> size:178247004 read:0 written:0 al-writes:0 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no<br> volume:1 minor:2 disk:UpToDate backing_dev:/dev/sda6 quorum:yes<br> size:419404764 read:0 written:0 al-writes:0 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no<br> HV02 node-id:1 connection:Connected role:Secondary congested:no ap-in-flight:0 rs-in-flight:0<br> volume:0 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br> volume:1 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br> HV03 node-id:2 connection:Connected role:Secondary congested:no ap-in-flight:0 rs-in-flight:0<br> volume:0 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br> volume:1 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br><br>HV02 ~ # drbdsetup status --verbose --statistics<br>r01 node-id:1 role:Secondary suspended:no force-io-failures:no<br> write-ordering:flush<br> volume:0 minor:1 disk:UpToDate backing_dev:/dev/sda5 quorum:yes<br> size:178247004 read:0 written:0 al-writes:0 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no<br> volume:1 minor:2 disk:UpToDate backing_dev:/dev/sda6 quorum:yes<br> size:419404764 read:0 written:0 al-writes:0 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no<br> HV01 node-id:0 connection:Connected role:Primary congested:no ap-in-flight:0 rs-in-flight:0<br> volume:0 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br> volume:1 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br> HV03 node-id:2 connection:Connected role:Secondary congested:no ap-in-flight:0 rs-in-flight:0<br> volume:0 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br> volume:1 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">HV03 ~ # drbdsetup status --verbose --statistics<br>r01 node-id:2 role:Secondary suspended:no force-io-failures:no<br> write-ordering:flush<br> volume:0 minor:1 disk:UpToDate backing_dev:/dev/sda5 quorum:yes<br> size:178247004 read:0 written:0 al-writes:0 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no<br> volume:1 minor:2 disk:UpToDate backing_dev:/dev/sda6 quorum:yes<br> size:419404764 read:0 written:0 al-writes:0 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no<br> HV01 node-id:0 connection:Connected role:Primary congested:no ap-in-flight:0 rs-in-flight:0<br> volume:0 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br> volume:1 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br> HV02 node-id:1 connection:Connected role:Secondary congested:no ap-in-flight:0 rs-in-flight:0<br> volume:0 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0<br> volume:1 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:0 sent:0 out-of-sync:0 pending:0 unacked:0</span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace">
</span></span></span></span><div><br></div><div><br></div><div><br>
</div><div>But when starting a copy of (i.e.) a 10GB file, this error comes up immediately on both the secondary nodes:</div><div><br></div><div><span style="font-family:monospace"><b>[Tue Dec 6 16:18:12 2022] drbd r01/0 drbd1 HV01: ASSERTION d->bi_size <= DRBD_MAX_BIO_SIZE FAILED in read_in_block<br>[Tue Dec 6 16:18:12 2022] drbd r01 HV01: error receiving P_DATA, e: -5 l: 2093056!</b><br>[Tue Dec 6 16:18:12 2022] drbd r01 HV01: conn( Connected -> ProtocolError ) peer( Primary -> Unknown )<br>[Tue Dec 6 16:18:12 2022] drbd r01/0 drbd1: disk( UpToDate -> Consistent )<br>[Tue Dec 6 16:18:12 2022] drbd r01/0 drbd1 HV01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )<br>[Tue Dec 6 16:18:12 2022] drbd r01/1 drbd2: disk( UpToDate -> Consistent )<br>[Tue Dec 6 16:18:12 2022] drbd r01/1 drbd2 HV01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )<br>[Tue Dec 6 16:18:12 2022] drbd r01 HV01: Terminating sender thread<br>[Tue Dec 6 16:18:12 2022] drbd r01 HV01: Starting sender thread (from drbd_r_r01 [13658])<br>[Tue Dec 6 16:18:12 2022] drbd r01/1 drbd2 HV03: pdsk( UpToDate -> Consistent )<br>[Tue Dec 6 16:18:12 2022] drbd r01: Preparing cluster-wide state change 120227718 (1->-1 0/0)<br>[Tue Dec 6 16:18:12 2022] drbd r01 HV01: Connection closed<br>[Tue Dec 6 16:18:12 2022] drbd r01 HV01: helper command: /sbin/drbdadm disconnected<br>[Tue Dec 6 16:18:12 2022] drbd r01 HV01: helper command: /sbin/drbdadm disconnected exit code 0<br>[Tue Dec 6 16:18:12 2022] drbd r01 HV01: conn( ProtocolError -> Unconnected )<br>[Tue Dec 6 16:18:12 2022] drbd r01 HV01: Restarting receiver thread</span></div><div>
<span style="font-family:monospace">[...]</span>
</div><div><br></div>
</div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><b>[Tue Dec 6 16:18:13 2022] drbd r01/0 drbd1 HV01: ASSERTION d->bi_size <= DRBD_MAX_BIO_SIZE FAILED in read_in_block<br>[Tue Dec 6 16:18:13 2022] drbd r01 HV01: error receiving P_DATA, e: -5 l: 2093056!</b><br>[Tue Dec 6 16:18:13 2022] drbd r01 HV01: conn( Connected -> ProtocolError ) peer( Primary -> Unknown )<br>[Tue Dec 6 16:18:13 2022] drbd r01/0 drbd1: disk( UpToDate -> Consistent )<br>[Tue Dec 6 16:18:13 2022] drbd r01/0 drbd1 HV01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )<br>[Tue Dec 6 16:18:13 2022] drbd r01/1 drbd2: disk( UpToDate -> Consistent )<br>[Tue Dec 6 16:18:13 2022] drbd r01/1 drbd2 HV01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )<br>[Tue Dec 6 16:18:13 2022] drbd r01 HV01: Terminating sender thread<br>[Tue Dec 6 16:18:13 2022] drbd r01 HV01: Starting sender thread (from drbd_r_r01 [13627])<br>[Tue Dec 6 16:18:13 2022] drbd r01: Preparing cluster-wide state change 1546196417 (2->-1 0/0)<br>[Tue Dec 6 16:18:13 2022] drbd r01/0 drbd1 HV02: pdsk( UpToDate -> Consistent )<br>[Tue Dec 6 16:18:13 2022] drbd r01 HV01: Connection closed<br>[Tue Dec 6 16:18:13 2022] drbd r01 HV01: helper command: /sbin/drbdadm disconnected<br>[Tue Dec 6 16:18:13 2022] drbd r01 HV01: helper command: /sbin/drbdadm disconnected exit code 0<br>[Tue Dec 6 16:18:13 2022] drbd r01 HV01: conn( ProtocolError -> Unconnected )<br>[Tue Dec 6 16:18:13 2022] drbd r01 HV01: Restarting receiver thread<br>[...]<br></span></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></span></span></div><div>The problem is 100% replicable each time, the only thing that changes is the <span style="font-family:monospace">d->bi_size</span> value (it also changes in the other error messages during the copy). The <span style="font-family:monospace">DRBD_MAX_BIO_SIZE</span> is <span style="font-family:monospace">1048576</span>. <br></div><div>Copying the same file on the same net interface / partition with scp or other tools works perfectly, at maximum hw speed.</div><div><br></div><div>I've tried: <br></div><div>- changing the --peer-max-bio-size</div><div>- changing the primary node <br></div><div>- connecting only the primary node with a secondary node</div><div>- checking the equality of max_sectors_kb, max_hw_sectors_kb, nr_request for both the physical disk and the drbd device <br></div><div>- changing the MTU of the network interfaces<br></div><div><br></div><div>Nothing seems to have an effect. Everytime an intensive write operation starts, this error
immediately
shows on each secondary node. <br></div><div>Could someone give us some hint, we are totally stuck at the moment? <br></div><div><br></div><div>Thank you in advance and best regards,</div><div>Andrea<br></div></div>