Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi all,
after upgrading DRBD-Dkms to 9.0.8+linbit-1 (on kernel 4.4.67-1-pve) I get those errors:
Software is: root at virt5:~# dpkg -l |grep drbd
ii drbd-dkms 9.0.8+linbit-1 all RAID 1 over TCP/IP for Linux module source
ii drbd-utils 9.0.0+linbit-1 amd64 RAID 1 over TCP/IP for Linux (user utilities)
ii drbdmanage-proxmox 1.0-1 all DRBD distributed resource management utility
ii python-drbdmanage 0.99.8-1 all DRBD distributed resource management utility
kernel:[233761.855451] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [drbd_s_vm-109-d:65197]
Aug 14 14:21:49 virt2 kernel: [233761.855451] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [drbd_s_vm-109-d:65197]
Aug 14 14:21:49 virt2 kernel: [233761.855544] Modules linked in: ipt_REJECT nf_reject_ipv4 drbd_transport_tcp(O) drbd(O) ip_set ip6table_filter ip6_tables xt_multiport binfmt_misc iptable_filter ip_tables x_tables softdog rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfnetlink_log nfnetlink zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel sb_edac aes_x86_64 snd_pcm lrw snd_timer edac_core gf128mul snd glue_helper shpchp soundcore joydev pcspkr input_leds ablk_helper cryptd i2c_i801 lpc_ich ipmi_si 8250_fintek mei_me ipmi_msghandler mei mac_hid acpi_power_meter wmi vhost_net vhost macvtap macvlan coretemp autofs4 hid_generic ixgbe(O) usbkbd usbmouse dca be2net usbhid vxlan ahci ip6_udp_tunnel ptp hid libahci udp_tunnel pps_core megaraid_sas fjes
Aug 14 14:21:49 virt2 kernel: [233761.855592] CPU: 1 PID: 65197 Comm: drbd_s_vm-109-d Tainted: P O L 4.4.67-1-pve #1
Aug 14 14:21:49 virt2 kernel: [233761.855593] Hardware name: FUJITSU PRIMERGY RX2530 M1/D3279-A1, BIOS V5.0.0.9 R1.28.0 for D3279-A1x 12/09/2015
Aug 14 14:21:49 virt2 kernel: [233761.855594] task: ffff883003707000 ti: ffff881dc5100000 task.ti: ffff881dc5100000
Aug 14 14:21:49 virt2 kernel: [233761.855595] RIP: 0010:[<ffffffffc0936ae2>] [<ffffffffc0936ae2>] wait_for_sender_todo+0x142/0x270 [drbd]
Aug 14 14:21:49 virt2 kernel: [233761.855606] RSP: 0018:ffff881dc5103db0 EFLAGS: 00000282
Aug 14 14:21:49 virt2 kernel: [233761.855607] RAX: 00000000002aef32 RBX: ffff88300dd94000 RCX: 8000000000000000
Aug 14 14:21:49 virt2 kernel: [233761.855608] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff881c71ab1998
Aug 14 14:21:49 virt2 kernel: [233761.855609] RBP: ffff881dc5103e10 R08: ffff88300dd944d8 R09: 0000000000000001
Aug 14 14:21:49 virt2 kernel: [233761.855610] R10: 000000002b300018 R11: 00000000fffffffb R12: ffff88300dd940b8
Aug 14 14:21:49 virt2 kernel: [233761.855610] R13: ffff88300dd944c0 R14: ffff883003707000 R15: 0000000000000001
Aug 14 14:21:49 virt2 kernel: [233761.855611] FS: 0000000000000000(0000) GS:ffff88181f640000(0000) knlGS:0000000000000000
Aug 14 14:21:49 virt2 kernel: [233761.855612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 14:21:49 virt2 kernel: [233761.855613] CR2: 0000000000000030 CR3: 0000000001e0b000 CR4: 00000000001426e0
Aug 14 14:21:49 virt2 kernel: [233761.855614] Stack:
Aug 14 14:21:49 virt2 kernel: [233761.855614] ffff88300dd94520 0000000000000000 ffff883003707000 ffffffff810c46e0
Aug 14 14:21:49 virt2 kernel: [233761.855616] ffff881dc5103dd0 ffff881dc5103dd0 00000000e2e71a4a ffff88300dd94000
Aug 14 14:21:49 virt2 kernel: [233761.855617] ffff88300dd944d8 ffff88300dd94528 ffff88300dd94520 ffff88300dd944d8
Aug 14 14:21:49 virt2 kernel: [233761.855618] Call Trace:
Aug 14 14:21:49 virt2 kernel: [233761.855623] [<ffffffff810c46e0>] ? wait_woken+0x90/0x90
Aug 14 14:21:49 virt2 kernel: [233761.855627] [<ffffffffc093c1fe>] drbd_sender+0x34e/0x400 [drbd]
Aug 14 14:21:49 virt2 kernel: [233761.855633] [<ffffffffc095ad60>] ? w_complete+0x20/0x20 [drbd]
Aug 14 14:21:49 virt2 kernel: [233761.855637] [<ffffffffc095adc0>] drbd_thread_setup+0x60/0x110 [drbd]
Aug 14 14:21:49 virt2 kernel: [233761.855641] [<ffffffffc095ad60>] ? w_complete+0x20/0x20 [drbd]
Aug 14 14:21:49 virt2 kernel: [233761.855643] [<ffffffff810a134a>] kthread+0xfa/0x110
Aug 14 14:21:49 virt2 kernel: [233761.855644] [<ffffffff810a1250>] ? kthread_park+0x60/0x60
Aug 14 14:21:49 virt2 kernel: [233761.855646] [<ffffffff81864a8f>] ret_from_fork+0x3f/0x70
Aug 14 14:21:49 virt2 kernel: [233761.855647] [<ffffffff810a1250>] ? kthread_park+0x60/0x60
Aug 14 14:21:49 virt2 kernel: [233761.855648] Code: 8b 7b 10 8b 87 a0 03 00 00 84 d2 74 09 3b 83 38 09 00 00 0f 95 c2 48 81 c7 98 01 00 00 c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <84> d2 0f 85 99 00 00 00 f0 0f ba b3 f8 00 00 00 0f 72 43 83 bb
Aug 14 14:21:51 virt2 kernel: [233763.964203] drbd_send_and_submit: 546 callbacks suppressed
Aug 14 14:21:51 virt2 kernel: [233763.964206] drbd vm-109-disk-1/0 drbd176: IO ERROR: neither local nor remote data, sector 38107320+8
Aug 14 14:21:51 virt2 kernel: [233763.964327] buffer_io_error: 546 callbacks suppressed
top shows processes using all of one cpu core
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
65197 root 20 0 0 0 0 R 100.0 0.0 92:01.19 drbd_s_vm-109-d
65229 root 20 0 0 0 0 R 100.0 0.0 91:54.35 kworker/u145:2
>drbdadm status
root at virt5:~# drbdadm status
.drbdctrl role:Secondary
volume:0 disk:UpToDate
volume:1 disk:UpToDate
fs1 role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate
fs2 role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate
virt1 role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate
virt2 role:Primary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate
virt3 role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate
virt4 role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate
backups role:Secondary
disk:UpToDate
fs1 role:Secondary
peer-disk:UpToDate
fs2 role:Secondary
peer-disk:UpToDate
vm-109-disk-1 role:Secondary
disk:UpToDate
virt2 connection:Connecting
all drbdmanage commands tell me that >>Error: Startup not successful (no quorum? not *both* nodes up in a 2 node cluster?)
Is there anything I can do about it? (reboot is no option on production machines…)
Best regards,
Frank