drbdadm up: extent ..nnn.. beyond end of bitmap!
Sven Schlittermann
sr1 at drewag.de
Tue Jan 30 15:08:15 CET 2024
Hello,
I happened to see this error message and tried to reproduce it.
I did this:
drbdadm create-md --max-peers=10 r0
sysctl kernel.nmi_watchdog=0
drbdadm up r0
drbdadm primary --force r0
mke2fs -t ext4 /dev/drbd0
reboot
Adter reboot:
sysctl kernel.nmi_watchdog=0
drbdadm up r0
And here I get the messages:
...
extent 10064384 beyond end of bitmap!
extent 10064896 beyond end of bitmap!
extent 10065408 beyond end of bitmap!
extent 10065920 beyond end of bitmap!
Marked additional 232 GB as out-of-sync based on AL.
This problem does not appear when I choose --max-peers=4 instead of
--max-peers=10 .
This problem does not appear when I omit the above-shown mke2fs command .
Below more details.
My choosen parameters might be suboptimal or weird (hints welcome),
nevertheless above message irritates and seems to indicate a bug.
Best regards,
Sven
----
Linux 6.6.13
drbd-9.1.18
Debian drbd-utils 9.15.0-1
----
# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ baaca8a080dc54652f57da4bafb2dce51dfe9f68\ reproducible\ build\,\ 2020-09-29\ 09:05:36
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090112
DRBD_KERNEL_VERSION=9.1.18
DRBDADM_VERSION_CODE=0x090f00
DRBDADM_VERSION=9.15.0
----
# cat /etc/drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
----
# grep -v "^[[:space:]]*#" /etc/drbd.d/global_common.conf
global {
usage-count no;
udev-always-use-vnr; # treat implicit the same as explicit volumes
}
common {
protocol B;
handlers {
}
startup {
}
options {
}
disk {
}
net {
}
}
----
# cat /etc/drbd.d/r0.res
resource r0 {
startup {
wfc-timeout 60;
outdated-wfc-timeout 60;
degr-wfc-timeout 60;
}
net {
verify-alg sha1;
cram-hmac-alg "sha1";
shared-secret "..secret..";
max-buffers 20k;
max-epoch-size 16k;
transport "tcp";
sndbuf-size 5120k;
rcvbuf-size 5120k;
}
disk {
c-plan-ahead 5;
c-fill-target 1M;
c-max-rate 400M;
c-min-rate 4M;
al-extents 6433;
}
volume 0 {
device /dev/drbd0;
disk /dev/vg0/data;
meta-disk internal;
}
connection-mesh {
....
}
----
# lvdisplay -m /dev/vg0/data
--- Logical volume ---
LV Path /dev/vg0/data
LV Name data
VG Name vg0
LV UUID ..uuid
LV Write Access read/write
LV Creation host, time ..host.., 2022-05-03 17:18:24 +0200
LV Status available
# open 2
LV Size 50.00 TiB
Current LE 51200
Segments 5
Allocation inherit
Read ahead sectors auto
- currently set to 81920
Block device 252:0
--- Segments ---
Logical extents 0 to 11177:
Type linear
Physical volume /dev/md8
Physical extents 0 to 11177
Logical extents 11178 to 22355:
Type linear
Physical volume /dev/md126
Physical extents 0 to 11177
Logical extents 22356 to 33533:
Type linear
Physical volume /dev/md10
Physical extents 0 to 11177
Logical extents 33534 to 44711:
Type linear
Physical volume /dev/md11
Physical extents 0 to 11177
Logical extents 44712 to 51199:
Type linear
Physical volume /dev/md12
Physical extents 0 to 6487
----
(The other md devices are similarly constructed.)
# mdadm --detail /dev/md8
/dev/md8:
Version : 0.90
Creation Time : Tue May 3 17:02:22 2022
Raid Level : raid5
Array Size : 11721118720 (11178.13 GiB 12002.43 GB)
Used Dev Size : 1172111872 (1117.81 GiB 1200.24 GB)
Raid Devices : 11
Total Devices : 12
Preferred Minor : 8
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Jan 29 16:54:17 2024
State : clean
Active Devices : 11
Working Devices : 12
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
UUID : ..uuid..
Events : 0.3128
Number Major Minor RaidDevice State
0 67 49 0 active sync /dev/sdaz1
1 67 33 1 active sync /dev/sday1
2 67 97 2 active sync /dev/sdbc1
3 67 65 3 active sync /dev/sdba1
4 67 81 4 active sync /dev/sdbb1
5 67 113 5 active sync /dev/sdbd1
6 67 129 6 active sync /dev/sdbe1
7 67 145 7 active sync /dev/sdbf1
8 67 161 8 active sync /dev/sdbg1
9 67 177 9 active sync /dev/sdbh1
10 67 193 10 active sync /dev/sdbi1
11 67 209 - spare /dev/sdbj1
----
command sequence including output:
# drbdadm create-md --max-peers=10 r0
md_offset 54975581384704
al_offset 54975581351936
bm_offset 54958804135936
Found ext3 filesystem
2131099612 kB data area apparently used
53670707164 kB left usable by current configuration
Even though it looks like this would place the new meta data into
unused space, you still need to confirm, as this is only a guess.
Do you want to proceed?
[need to type 'yes' to confirm] yes
initializing activity log
initializing bitmap (16384000 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.
# sysctl kernel.nmi_watchdog=0
kernel.nmi_watchdog = 0
# drbdadm up r0
# # below the kernel log messages,
# # connection to DRBD peers is intentionally impossible
drbd r0: Starting worker thread (from drbdsetup [6182])
drbd r0 ..peer..: Starting sender thread (from drbdsetup [6186])
drbd r0 ..peer2..: Starting sender thread (from drbdsetup [6191])
drbd r0/0 drbd0: meta-data IO uses: blk-bio
drbd r0/0 drbd0: disk( Diskless -> Attaching ) [attach]
drbd r0/0 drbd0: Maximum number of peer devices = 10
drbd r0: Method to ensure write ordering: flush
drbd r0/0 drbd0: drbd_bm_resize called with capacity == 107341414328
drbd r0/0 drbd0: resync bitmap: bits=13417676791 words=2096512000 pages=4094750
drbd r0/0 drbd0: md_sync_timer expired! Worker calls drbd_md_sync().
drbd0: detected capacity change from 0 to 107341414328
drbd r0/0 drbd0: size = 50 TB (53670707164 KB)
drbd r0/0 drbd0: recounting of set bits took additional 15144ms
drbd r0/0 drbd0: disk( Attaching -> Inconsistent ) [attach]
drbd r0/0 drbd0: attached to current UUID: 0000000000000004
drbd r0 ..peer1..: conn( StandAlone -> Unconnected ) [connect]
drbd r0 ..peer2..: conn( StandAlone -> Unconnected ) [connect]
drbd r0 ..peer1..: Starting receiver thread (from drbd_w_r0 [6183])
drbd r0 ..peer2..: Starting receiver thread (from drbd_w_r0 [6183])
drbd r0 ..peer1..: conn( Unconnected -> Connecting ) [connecting]
drbd r0 ..peer2..: conn( Unconnected -> Connecting ) [connecting]
# drbdadm primary --force r0
# # below the kernel log messages
drbd r0: Preparing cluster-wide state change 3194273183 (7->-1 7683/4609)
drbd r0: Committing cluster-wide state change 3194273183 (7ms)
drbd r0: role( Secondary -> Primary ) [primary]
drbd r0/0 drbd0: disk( Inconsistent -> UpToDate ) [primary]
drbd r0/0 drbd0: size = 50 TB (53670707164 KB)
drbd r0: Forced to consider local data as UpToDate!
drbd r0/0 drbd0: new current UUID: 99EDA65846741BAB weak: FFFFFFFFFFFFFF7F
# mke2fs -t ext4 /dev/drbd0
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 13417676791 4k blocks and 838604800 inodes
Filesystem UUID: ..uuid..
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
2560000000, 3855122432, 5804752896, 12800000000
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done
# reboot
...
# # started a new shell on host
# sysctl kernel.nmi_watchdog=0
kernel.nmi_watchdog = 0
# drbdadm up r0
...
extent 10059264 beyond end of bitmap!
extent 10059776 beyond end of bitmap!
extent 10060288 beyond end of bitmap!
extent 10060800 beyond end of bitmap!
extent 10061312 beyond end of bitmap!
extent 10061824 beyond end of bitmap!
extent 10062336 beyond end of bitmap!
extent 10062848 beyond end of bitmap!
extent 10063360 beyond end of bitmap!
extent 10063872 beyond end of bitmap!
extent 10064384 beyond end of bitmap!
extent 10064896 beyond end of bitmap!
extent 10065408 beyond end of bitmap!
extent 10065920 beyond end of bitmap!
Marked additional 232 GB as out-of-sync based on AL.
# # below the kernel log messages
drbd r0: Starting worker thread (from drbdsetup [6122])
drbd r0 ..peer1..: Starting sender thread (from drbdsetup [6126])
drbd r0 ..peer2..: Starting sender thread (from drbdsetup [6134])
drbd r0/0 drbd0: meta-data IO uses: blk-bio
drbd r0/0 drbd0: disk( Diskless -> Attaching ) [attach]
drbd r0/0 drbd0: Maximum number of peer devices = 10
drbd r0: Method to ensure write ordering: flush
drbd r0/0 drbd0: drbd_bm_resize called with capacity == 107341414328
drbd r0/0 drbd0: md_sync_timer expired! Worker calls drbd_md_sync().
drbd r0/0 drbd0: resync bitmap: bits=13417676791 words=2096512000 pages=4094750
drbd0: detected capacity change from 0 to 107341414328
drbd r0/0 drbd0: size = 50 TB (53670707164 KB)
drbd r0/0 drbd0: size = 50 TB (53670707164 KB)
drbd r0/0 drbd0: recounting of set bits took additional 15755ms
drbd r0/0 drbd0: disk( Attaching -> UpToDate ) [attach]
drbd r0/0 drbd0: attached to current UUID: 99EDA65846741BAB
drbd r0/0 drbd0: Setting exposed data uuid: 99EDA65846741BAA
drbd r0 ..peer1..: conn( StandAlone -> Unconnected ) [connect]
drbd r0 ..peer2..: conn( StandAlone -> Unconnected ) [connect]
drbd r0 ..peer1..: Starting receiver thread (from drbd_w_r0 [6123])
drbd r0 ..peer2..: Starting receiver thread (from drbd_w_r0 [6123])
drbd r0 ..peer1..: conn( Unconnected -> Connecting ) [connecting]
drbd r0 ..peer2..: conn( Unconnected -> Connecting ) [connecting]
More information about the drbd-user
mailing list