[DRBD-user] [CASE-16] WFBitMapS status is not ended and mount command is hung

김재헌 jhkim at mantech.co.kr
Fri Feb 12 08:04:15 CET 2016

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi,

WFBitMapS status is not ended and mount command is hung.

1. Test version
 - CentOS-7 Linux 3.10.0-229.7.2.el7.x86_64
 - Engine: V9.0.1-1
     --  GIT-hash: f57acfc22d29a95697e683fb6bbacd9a1ad4348e build by
root at drbd9-02, 2016-02-09 09:46:20


2. Test scenario

1) status

[root at drbd9-01 ~]# drbdadm status r0
r0 role:Secondary
  disk:Outdated
  drbd9-02 role:Secondary
    peer-disk:Outdated

2) try mount

[root at drbd9-01 ~]# mount /dev/drbd1 /mnt
mount: /dev/drbd1 is write-protected, mounting read-only
mount: mount /dev/drbd1 on /mnt failed: 잘못된 미디어 종류

[root at drbd9-01 ~]# df
Filesystem              1K-blocks    Used Available Use% Mounted on
/dev/mapper/centos-root  18307072 8820096   9486976  49% /
devtmpfs                   491892       0    491892   0% /dev
tmpfs                      501580     140    501440   1% /dev/shm
tmpfs                      501580    7156    494424   2% /run
tmpfs                      501580       0    501580   0% /sys/fs/cgroup
/dev/sda1                  508588  134148    374440  27% /boot

3) promote

[root at drbd9-01 ~]# drbdadm primary r0
r0: State change failed: (-2) Need access to UpToDate data
Command 'drbdsetup primary r0' terminated with exit code 17

[root at drbd9-01 ~]# drbdadm primary --force r0


4) try mount again

[root at drbd9-01 ~]# mount /dev/drbd1 /mnt

  *   -- mount command is pending.!!!!*

5)  check status at another command terminal.
    primary is WFBitMapS  and secondary is UpToDate normal.

[root at drbd9-01 ~]# drbdadm status r0
r0 role:Primary
  disk:UpToDate
  drbd9-02 role:Secondary
    replication:WFBitMapS peer-disk:Consistent



3. Logs

1) node1(primary)

Feb  9 14:19:10 drbd9-01 kernel: drbd r0: Failed: role( Secondary ->
Primary )
Feb  9 14:19:10 drbd9-01 kernel: drbd r0: Auto-promote failed: Need access
to UpToDate data
Feb  9 14:19:27 drbd9-01 kernel: drbd r0: State change failed: Need access
to UpToDate data
Feb  9 14:19:27 drbd9-01 kernel: drbd r0: Failed: role( Secondary ->
Primary )

Feb  9 14:19:31 drbd9-01 kernel: drbd r0: Preparing cluster-wide state
change 2512721173 (0->-1 7683/4609)
Feb  9 14:19:31 drbd9-01 kernel: drbd r0: State change 2512721173:
primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC
Feb  9 14:19:31 drbd9-01 kernel: drbd r0: Committing cluster-wide state
change 2512721173 (2ms)
Feb  9 14:19:31 drbd9-01 kernel: drbd r0: role( Secondary -> Primary )
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1: disk( Outdated ->
UpToDate )
Feb  9 14:19:31 drbd9-01 kernel: drbd r0: Forced to consider local data as
UpToDate!
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1: new current UUID:
B89DA2131EDE0311 weak: FFFFFFFFFFFFFFFE
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02:
drbd_sync_handshake:
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: self
B89DA2131EDE0311:C223F7CE3C9D7358:C223F7CE3C9D7358:3D7DF30CF04727A4 bits:0
flags:0
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: peer
C223F7CE3C9D7358:0000000000000000:C223F7CE3C9D7358:3D7DF30CF04727A4 bits:0
flags:120
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: uuid_compare()=2
by rule 70
*Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: pdsk( Outdated
-> Consistent ) repl( Established -> WFBitMapS )*
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: send bitmap
stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: unexpected
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: In UUIDs from
node 1 found equal UUID (3D7DF30CF04727A4) for nodes 2 3
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: I have
C223F7CE3C9D7358 for node_id=2
Feb  9 14:19:31 drbd9-01 kernel: drbd r0/0 drbd1 drbd9-02: I have
C223F7CE3C9D7358 for node_id=3

Feb  9 14:19:35 drbd9-01 kernel: EXT4-fs (drbd1): mounting ext3 file system
using the ext4 subsystem
Feb  9 14:20:01 drbd9-01 systemd: Starting Session 13 of user root.
Feb  9 14:20:01 drbd9-01 systemd: Started Session 13 of user root.

Feb  9 14:21:49 drbd9-01 kernel: INFO: task mount:4758 blocked for more
than 120 seconds.
Feb  9 14:21:49 drbd9-01 kernel: "echo 0 > /proc/sys/
*kernel/hung_task_timeout_secs*" disables this message.
Feb  9 14:21:49 drbd9-01 kernel: mount           D ffff88003d613680     0
4758   4685 0x00000080
Feb  9 14:21:49 drbd9-01 kernel: ffff88003aa93b38 0000000000000086
ffff88001bf5db00 ffff88003aa93fd8
Feb  9 14:21:49 drbd9-01 kernel: ffff88003aa93fd8 ffff88003aa93fd8
ffff88001bf5db00 ffff88003d613f48
Feb  9 14:21:49 drbd9-01 kernel: ffff88003ffaaa18 0000000000000002
ffffffff811f8500 ffff88003aa93bb0
Feb  9 14:21:49 drbd9-01 kernel: Call Trace:
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811f8500>] ?
generic_block_bmap+0x70/0x70
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff816099ad>] io_schedule+0x9d/0x130
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811f850e>]
sleep_on_buffer+0xe/0x20
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff81607780>]
__wait_on_bit+0x60/0x90
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811f8500>] ?
generic_block_bmap+0x70/0x70
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff81607837>]
out_of_line_wait_on_bit+0x87/0xb0
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff81098270>] ?
autoremove_wake_function+0x40/0x40
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811f99d0>] ?
_submit_bh+0x160/0x210
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811fa628>]
__sync_dirty_buffer+0xb8/0xe0
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811fa663>]
sync_dirty_buffer+0x13/0x20
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffffa050cdf0>]
ext4_commit_super+0x1b0/0x240 [ext4]
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffffa050f500>]
ext4_setup_super+0x120/0x190 [ext4]
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffffa0512163>]
ext4_fill_super+0x1d53/0x2970 [ext4]
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811c9ce0>] mount_bdev+0x1b0/0x1f0
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffffa0510410>] ?
ext4_calculate_overhead+0x3d0/0x3d0 [ext4]
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffffa0505d35>] ext4_mount+0x15/0x20
[ext4]
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811ca619>] mount_fs+0x39/0x1b0
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811784b0>] ?
__alloc_percpu+0x10/0x20
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811e5c5f>]
vfs_kern_mount+0x5f/0xf0
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811e81ae>] do_mount+0x24e/0xa40
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff8117320b>] ?
strndup_user+0x4b/0xf0
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff811e8a36>] SyS_mount+0x96/0xf0
Feb  9 14:21:49 drbd9-01 kernel: [<ffffffff81614209>]
system_call_fastpath+0x16/0x1b

Feb  9 14:22:56 drbd9-01 gnome-session: (gnome-settings-daemon:2063):
color-plugin-WARNING **: unable to get EDID for xrandr-Virtual1: unable to
get EDID for output
Feb  9 14:22:56 drbd9-01 gnome-session: (gnome-settings-daemon:2063):
color-plugin-WARNING **: unable to get EDID for xrandr-Virtual1: unable to
get EDID for output
Feb  9 14:22:56 drbd9-01 gnome-session: (gnome-settings-daemon:2063):
color-plugin-WARNING **: unable to get EDID for xrandr-Virtual1: unable to
get EDID for output
Feb  9 14:23:08 drbd9-01 gnome-session: console.error:
Feb  9 14:23:08 drbd9-01 gnome-session: [CustomizableUI]
Feb  9 14:23:08 drbd9-01 gnome-session: Custom widget with id loop-button
does not return a valid node
Feb  9 14:23:09 drbd9-01 gnome-session: console.error:
Feb  9 14:23:09 drbd9-01 gnome-session: [CustomizableUI]
Feb  9 14:23:09 drbd9-01 gnome-session: Custom widget with id loop-button
does not return a valid node
Feb  9 14:23:49 drbd9-01 kernel: INFO: task mount:4758 blocked for more
than 120 seconds.
Feb  9 14:23:49 drbd9-01 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb  9 14:23:49 drbd9-01 kernel: mount           D ffff88003d613680     0
4758   4685 0x00000080
Feb  9 14:23:49 drbd9-01 kernel: ffff88003aa93b38 0000000000000086
ffff88001bf5db00 ffff88003aa93fd8
Feb  9 14:23:49 drbd9-01 kernel: ffff88003aa93fd8 ffff88003aa93fd8
ffff88001bf5db00 ffff88003d613f48
Feb  9 14:23:49 drbd9-01 kernel: ffff88003ffaaa18 0000000000000002
ffffffff811f8500 ffff88003aa93bb0
Feb  9 14:23:49 drbd9-01 kernel: Call Trace:
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811f8500>] ?
generic_block_bmap+0x70/0x70
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff816099ad>] io_schedule+0x9d/0x130
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811f850e>]
sleep_on_buffer+0xe/0x20
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff81607780>]
__wait_on_bit+0x60/0x90
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811f8500>] ?
generic_block_bmap+0x70/0x70
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff81607837>]
out_of_line_wait_on_bit+0x87/0xb0
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff81098270>] ?
autoremove_wake_function+0x40/0x40
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811f99d0>] ?
_submit_bh+0x160/0x210
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811fa628>]
__sync_dirty_buffer+0xb8/0xe0
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811fa663>]
sync_dirty_buffer+0x13/0x20
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffffa050cdf0>]
ext4_commit_super+0x1b0/0x240 [ext4]
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffffa050f500>]
ext4_setup_super+0x120/0x190 [ext4]
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffffa0512163>]
ext4_fill_super+0x1d53/0x2970 [ext4]
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811c9ce0>] mount_bdev+0x1b0/0x1f0
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffffa0510410>] ?
ext4_calculate_overhead+0x3d0/0x3d0 [ext4]
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffffa0505d35>] ext4_mount+0x15/0x20
[ext4]
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811ca619>] mount_fs+0x39/0x1b0
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811784b0>] ?
__alloc_percpu+0x10/0x20
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811e5c5f>]
vfs_kern_mount+0x5f/0xf0
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811e81ae>] do_mount+0x24e/0xa40
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff8117320b>] ?
strndup_user+0x4b/0xf0
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff811e8a36>] SyS_mount+0x96/0xf0
Feb  9 14:23:49 drbd9-01 kernel: [<ffffffff81614209>]
system_call_fastpath+0x16/0x1b
Feb  9 14:24:10 drbd9-01 gnome-session: ** (firefox:4868): CRITICAL **:
gst_app_src_set_size: assertion 'GST_IS_APP_SRC (appsrc)' failed
Feb  9 14:24:10 drbd9-01 gnome-session: ** (firefox:4868): CRITICAL **:
gst_app_src_set_size: assertion 'GST_IS_APP_SRC (appsrc)' failed
Feb  9 14:24:10 drbd9-01 gnome-session: ** (firefox:4868): CRITICAL **:
gst_app_src_set_size: assertion 'GST_IS_APP_SRC (appsrc)' failed


4. Question
 - Why does the  WFBitMapS status pend?
 - Why does  the mount command hang?

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20160212/e0337e91/attachment.htm>


More information about the drbd-user mailing list