drbdadm up: extent ..nnn.. beyond end of bitmap!
Sven Schlittermann
sr1 at drewag.de
Mon Mar 25 17:45:49 CET 2024
Hello,
I did not manage to reproduce the problem with Linux 6.6.22 an DRBD
9.1.19 .
I also tried with --max-peers=31 .
OTOH i didn't recognize the fix mentioned in the DRBD changelog.
Sven
Sven Schlittermann <sr1 at drewag.de> writes:
> Hello,
>
> I happened to see this error message and tried to reproduce it.
>
> I did this:
>
> drbdadm create-md --max-peers=10 r0
> sysctl kernel.nmi_watchdog=0
> drbdadm up r0
> drbdadm primary --force r0
> mke2fs -t ext4 /dev/drbd0
> reboot
>
> Adter reboot:
> sysctl kernel.nmi_watchdog=0
> drbdadm up r0
>
> And here I get the messages:
>
> ...
> extent 10064384 beyond end of bitmap!
> extent 10064896 beyond end of bitmap!
> extent 10065408 beyond end of bitmap!
> extent 10065920 beyond end of bitmap!
> Marked additional 232 GB as out-of-sync based on AL.
>
> This problem does not appear when I choose --max-peers=4 instead of
> --max-peers=10 .
>
> This problem does not appear when I omit the above-shown mke2fs command .
>
> Below more details.
>
> My choosen parameters might be suboptimal or weird (hints welcome),
> nevertheless above message irritates and seems to indicate a bug.
>
> Best regards,
>
> Sven
>
> ----
>
> Linux 6.6.13
> drbd-9.1.18
> Debian drbd-utils 9.15.0-1
>
> ----
>
> # drbdadm --version
> DRBDADM_BUILDTAG=GIT-hash:\ baaca8a080dc54652f57da4bafb2dce51dfe9f68\ reproducible\ build\,\ 2020-09-29\ 09:05:36
> DRBDADM_API_VERSION=2
> DRBD_KERNEL_VERSION_CODE=0x090112
> DRBD_KERNEL_VERSION=9.1.18
> DRBDADM_VERSION_CODE=0x090f00
> DRBDADM_VERSION=9.15.0
>
> ----
>
> # cat /etc/drbd.conf
> # You can find an example in /usr/share/doc/drbd.../drbd.conf.example
>
> include "drbd.d/global_common.conf";
> include "drbd.d/*.res";
>
> ----
>
> # grep -v "^[[:space:]]*#" /etc/drbd.d/global_common.conf
>
> global {
>
> usage-count no;
>
> udev-always-use-vnr; # treat implicit the same as explicit volumes
>
> }
>
> common {
>
> protocol B;
>
>
> handlers {
>
> }
>
> startup {
> }
>
> options {
>
> }
>
> disk {
> }
>
> net {
> }
> }
>
> ----
>
> # cat /etc/drbd.d/r0.res
> resource r0 {
> startup {
> wfc-timeout 60;
> outdated-wfc-timeout 60;
> degr-wfc-timeout 60;
> }
> net {
> verify-alg sha1;
> cram-hmac-alg "sha1";
> shared-secret "..secret..";
>
> max-buffers 20k;
> max-epoch-size 16k;
>
> transport "tcp";
> sndbuf-size 5120k;
> rcvbuf-size 5120k;
> }
>
> disk {
> c-plan-ahead 5;
> c-fill-target 1M;
> c-max-rate 400M;
> c-min-rate 4M;
>
> al-extents 6433;
>
> }
>
> volume 0 {
> device /dev/drbd0;
> disk /dev/vg0/data;
> meta-disk internal;
> }
>
> connection-mesh {
> ....
> }
> ----
> # lvdisplay -m /dev/vg0/data
> --- Logical volume ---
> LV Path /dev/vg0/data
> LV Name data
> VG Name vg0
> LV UUID ..uuid
> LV Write Access read/write
> LV Creation host, time ..host.., 2022-05-03 17:18:24 +0200
> LV Status available
> # open 2
> LV Size 50.00 TiB
> Current LE 51200
> Segments 5
> Allocation inherit
> Read ahead sectors auto
> - currently set to 81920
> Block device 252:0
>
> --- Segments ---
> Logical extents 0 to 11177:
> Type linear
> Physical volume /dev/md8
> Physical extents 0 to 11177
>
> Logical extents 11178 to 22355:
> Type linear
> Physical volume /dev/md126
> Physical extents 0 to 11177
>
> Logical extents 22356 to 33533:
> Type linear
> Physical volume /dev/md10
> Physical extents 0 to 11177
>
> Logical extents 33534 to 44711:
> Type linear
> Physical volume /dev/md11
> Physical extents 0 to 11177
>
> Logical extents 44712 to 51199:
> Type linear
> Physical volume /dev/md12
> Physical extents 0 to 6487
>
> ----
>
> (The other md devices are similarly constructed.)
>
> # mdadm --detail /dev/md8
> /dev/md8:
> Version : 0.90
> Creation Time : Tue May 3 17:02:22 2022
> Raid Level : raid5
> Array Size : 11721118720 (11178.13 GiB 12002.43 GB)
> Used Dev Size : 1172111872 (1117.81 GiB 1200.24 GB)
> Raid Devices : 11
> Total Devices : 12
> Preferred Minor : 8
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Mon Jan 29 16:54:17 2024
> State : clean
> Active Devices : 11
> Working Devices : 12
> Failed Devices : 0
> Spare Devices : 1
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Consistency Policy : bitmap
>
> UUID : ..uuid..
> Events : 0.3128
>
> Number Major Minor RaidDevice State
> 0 67 49 0 active sync /dev/sdaz1
> 1 67 33 1 active sync /dev/sday1
> 2 67 97 2 active sync /dev/sdbc1
> 3 67 65 3 active sync /dev/sdba1
> 4 67 81 4 active sync /dev/sdbb1
> 5 67 113 5 active sync /dev/sdbd1
> 6 67 129 6 active sync /dev/sdbe1
> 7 67 145 7 active sync /dev/sdbf1
> 8 67 161 8 active sync /dev/sdbg1
> 9 67 177 9 active sync /dev/sdbh1
> 10 67 193 10 active sync /dev/sdbi1
>
> 11 67 209 - spare /dev/sdbj1
>
> ----
>
> command sequence including output:
>
> # drbdadm create-md --max-peers=10 r0
> md_offset 54975581384704
> al_offset 54975581351936
> bm_offset 54958804135936
>
> Found ext3 filesystem
> 2131099612 kB data area apparently used
> 53670707164 kB left usable by current configuration
>
> Even though it looks like this would place the new meta data into
> unused space, you still need to confirm, as this is only a guess.
>
> Do you want to proceed?
> [need to type 'yes' to confirm] yes
>
> initializing activity log
> initializing bitmap (16384000 KB) to all zero
> Writing meta data...
> New drbd meta data block successfully created.
> # sysctl kernel.nmi_watchdog=0
> kernel.nmi_watchdog = 0
> # drbdadm up r0
> # # below the kernel log messages,
> # # connection to DRBD peers is intentionally impossible
>
> drbd r0: Starting worker thread (from drbdsetup [6182])
> drbd r0 ..peer..: Starting sender thread (from drbdsetup [6186])
> drbd r0 ..peer2..: Starting sender thread (from drbdsetup [6191])
> drbd r0/0 drbd0: meta-data IO uses: blk-bio
> drbd r0/0 drbd0: disk( Diskless -> Attaching ) [attach]
> drbd r0/0 drbd0: Maximum number of peer devices = 10
> drbd r0: Method to ensure write ordering: flush
> drbd r0/0 drbd0: drbd_bm_resize called with capacity == 107341414328
> drbd r0/0 drbd0: resync bitmap: bits=13417676791 words=2096512000 pages=4094750
> drbd r0/0 drbd0: md_sync_timer expired! Worker calls drbd_md_sync().
> drbd0: detected capacity change from 0 to 107341414328
> drbd r0/0 drbd0: size = 50 TB (53670707164 KB)
> drbd r0/0 drbd0: recounting of set bits took additional 15144ms
> drbd r0/0 drbd0: disk( Attaching -> Inconsistent ) [attach]
> drbd r0/0 drbd0: attached to current UUID: 0000000000000004
> drbd r0 ..peer1..: conn( StandAlone -> Unconnected ) [connect]
> drbd r0 ..peer2..: conn( StandAlone -> Unconnected ) [connect]
> drbd r0 ..peer1..: Starting receiver thread (from drbd_w_r0 [6183])
> drbd r0 ..peer2..: Starting receiver thread (from drbd_w_r0 [6183])
> drbd r0 ..peer1..: conn( Unconnected -> Connecting ) [connecting]
> drbd r0 ..peer2..: conn( Unconnected -> Connecting ) [connecting]
>
> # drbdadm primary --force r0
> # # below the kernel log messages
>
> drbd r0: Preparing cluster-wide state change 3194273183 (7->-1 7683/4609)
> drbd r0: Committing cluster-wide state change 3194273183 (7ms)
> drbd r0: role( Secondary -> Primary ) [primary]
> drbd r0/0 drbd0: disk( Inconsistent -> UpToDate ) [primary]
> drbd r0/0 drbd0: size = 50 TB (53670707164 KB)
> drbd r0: Forced to consider local data as UpToDate!
> drbd r0/0 drbd0: new current UUID: 99EDA65846741BAB weak: FFFFFFFFFFFFFF7F
>
> # mke2fs -t ext4 /dev/drbd0
> mke2fs 1.46.2 (28-Feb-2021)
> Creating filesystem with 13417676791 4k blocks and 838604800 inodes
> Filesystem UUID: ..uuid..
> Superblock backups stored on blocks:
> 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
> 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
> 102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
> 2560000000, 3855122432, 5804752896, 12800000000
>
> Allocating group tables: done
> Writing inode tables: done
> Creating journal (262144 blocks): done
> Writing superblocks and filesystem accounting information: done
>
> # reboot
> ...
>
> # # started a new shell on host
> # sysctl kernel.nmi_watchdog=0
> kernel.nmi_watchdog = 0
> # drbdadm up r0
> ...
> extent 10059264 beyond end of bitmap!
> extent 10059776 beyond end of bitmap!
> extent 10060288 beyond end of bitmap!
> extent 10060800 beyond end of bitmap!
> extent 10061312 beyond end of bitmap!
> extent 10061824 beyond end of bitmap!
> extent 10062336 beyond end of bitmap!
> extent 10062848 beyond end of bitmap!
> extent 10063360 beyond end of bitmap!
> extent 10063872 beyond end of bitmap!
> extent 10064384 beyond end of bitmap!
> extent 10064896 beyond end of bitmap!
> extent 10065408 beyond end of bitmap!
> extent 10065920 beyond end of bitmap!
> Marked additional 232 GB as out-of-sync based on AL.
>
> # # below the kernel log messages
>
> drbd r0: Starting worker thread (from drbdsetup [6122])
> drbd r0 ..peer1..: Starting sender thread (from drbdsetup [6126])
> drbd r0 ..peer2..: Starting sender thread (from drbdsetup [6134])
> drbd r0/0 drbd0: meta-data IO uses: blk-bio
> drbd r0/0 drbd0: disk( Diskless -> Attaching ) [attach]
> drbd r0/0 drbd0: Maximum number of peer devices = 10
> drbd r0: Method to ensure write ordering: flush
> drbd r0/0 drbd0: drbd_bm_resize called with capacity == 107341414328
> drbd r0/0 drbd0: md_sync_timer expired! Worker calls drbd_md_sync().
> drbd r0/0 drbd0: resync bitmap: bits=13417676791 words=2096512000 pages=4094750
> drbd0: detected capacity change from 0 to 107341414328
> drbd r0/0 drbd0: size = 50 TB (53670707164 KB)
> drbd r0/0 drbd0: size = 50 TB (53670707164 KB)
> drbd r0/0 drbd0: recounting of set bits took additional 15755ms
> drbd r0/0 drbd0: disk( Attaching -> UpToDate ) [attach]
> drbd r0/0 drbd0: attached to current UUID: 99EDA65846741BAB
> drbd r0/0 drbd0: Setting exposed data uuid: 99EDA65846741BAA
> drbd r0 ..peer1..: conn( StandAlone -> Unconnected ) [connect]
> drbd r0 ..peer2..: conn( StandAlone -> Unconnected ) [connect]
> drbd r0 ..peer1..: Starting receiver thread (from drbd_w_r0 [6123])
> drbd r0 ..peer2..: Starting receiver thread (from drbd_w_r0 [6123])
> drbd r0 ..peer1..: conn( Unconnected -> Connecting ) [connecting]
> drbd r0 ..peer2..: conn( Unconnected -> Connecting ) [connecting]
>
>
>
>
--
DREWAG - Stadtwerke Dresden GmbH
Sitz der Gesellschaft: Dresden
Handelsregister: Amtsgericht Dresden HRB Nr. 2626
Vorsitzender des Aufsichtsrates: Dirk Hilbert
Geschäftsführung: Dr. Frank Brinkmann (Sprecher)
Dr. Axel Cunow, Lars Seiffert
Informationen zur Verarbeitung personenbezogener Daten finden Sie unter
http://www.drewag.de/datenschutz.
More information about the drbd-user
mailing list