<div dir="ltr"><div>Another VM have the same problem, new errors appear:</div><div><br></div><div># dmesg | grep one-vm-8099-disk-0<br>[20158467.625882] drbd one-vm-8099-disk-0: susp-io( no -> user)<br>[20158469.308525] drbd one-vm-8099-disk-0: susp-io( user -> no)<br>[20158469.309400] drbd one-vm-8099-disk-0/0 drbd1545: drbd_req_complete: Logic BUG rq_state: 8000, completion_ref = -1<br>[20158469.309504] drbd one-vm-8099-disk-0/0 drbd1545: drbd_req_complete: Logic BUG rq_state: 8000, completion_ref = -1<br>[20158469.309553] drbd one-vm-8099-disk-0/0 drbd1545: drbd_req_complete: Logic BUG rq_state: 8000, completion_ref = -1<br>[20158469.309647] drbd one-vm-8099-disk-0/0 drbd1545: drbd_req_complete: Logic BUG rq_state: 8000, completion_ref = -1<br>[20158469.309989] drbd one-vm-8099-disk-0/0 drbd1545: drbd_req_complete: Logic BUG rq_state: 8000, completion_ref = -1<br>[20158469.310208] drbd one-vm-8099-disk-0/0 drbd1545: drbd_req_complete: Logic BUG rq_state: 8000, completion_ref = -1<br>[20158469.310762] drbd one-vm-8099-disk-0/0 drbd1545: drbd_req_destroy: Logic BUG rq_state: 8000, completion_ref = -1<br></div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Best Regards,</div><div>Andrei Kvapil<br></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 29, 2020 at 3:14 PM kvaps <<a href="mailto:kvapss@gmail.com">kvapss@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello,</div><div><br></div><div>I'm not sure if this bug was already fixed on the latest drbd version but better to report it.</div><div>I'm using 9.0.19-1 (6f5fa5d348a99e5eeb09d83c49853d72e614fd07) and kernel 4.15.18-18-pve</div><div><br></div><div>We're running weekly backup for all our resources, backups are made for each resource like described here:</div><div><br></div><div><a href="https://github.com/LINBIT/linstor-server/issues/150#issuecomment-635942823" target="_blank">https://github.com/LINBIT/linstor-server/issues/150#issuecomment-635942823</a></div><div><br></div><div>Thus for each resource we create new snapshot and new deploy the resource from this snapshot, then snapshot is removed, then perform the backup for created resource, then remove this resource created from the snapshot.<br></div><div><br></div><div>Time-to-time the VMs might stuck forever even resource is unsuspected after snapshot.</div><div>Such VMs can be killed only with -9 signal. But resource stay on primary mode and can't be shuted down:<br></div><div><br></div><div># drbdsetup status one-vm-7944-disk-0 --verbose --statistics<br>one-vm-7944-disk-0 node-id:3 role:Primary suspended:no<br> write-ordering:none<br> volume:0 minor:1509 disk:Diskless client:yes quorum:yes<br> size:272632908 read:0 written:0 al-writes:0 bm-writes:0 upper-pending:4 lower-pending:0 al-suspended:no blocked:no<br> m13c28 node-id:0 connection:Connected role:Secondary congested:no ap-in-flight:0 rs-in-flight:0<br> volume:0 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:2027182412 sent:1080354540 out-of-sync:0 pending:0 unacked:0<br> m14c10 node-id:1 connection:Connected role:Secondary congested:no ap-in-flight:0 rs-in-flight:0<br> volume:0 replication:Established peer-disk:UpToDate resync-suspended:no<br> received:2027184872 sent:1080354540 out-of-sync:0 pending:0 unacked:0</div><div><br></div><div># dmesg | grep one-vm-7944-disk-0<br>[20157532.568950] drbd one-vm-7944-disk-0: susp-io( no -> user)<br>[20157534.979777] drbd one-vm-7944-disk-0: susp-io( user -> no)</div><div><br></div><div># drbdsetup secondary one-vm-7944-disk-0</div><div><stuck forever><br></div><div><br></div><div>strace log:</div><div><br></div><div>execve("/usr/sbin/drbdsetup", ["drbdsetup", "secondary", "one-vm-7944-disk-0"], 0x7ffc6b833b10 /* 16 vars */) = 0<br>brk(NULL) = 0x56134ae0f000<br>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)<br>access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3<br>fstat(3, {st_mode=S_IFREG|0644, st_size=37110, ...}) = 0<br>mmap(NULL, 37110, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5155c12000<br>close(3) = 0<br>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3<br>read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\34\2\0\0\0\0\0"..., 832) = 832<br>fstat(3, {st_mode=S_IFREG|0755, st_size=2030544, ...}) = 0<br>mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5155c10000<br>mmap(NULL, 4131552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f5155604000<br>mprotect(0x7f51557eb000, 2097152, PROT_NONE) = 0<br>mmap(0x7f51559eb000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7f51559eb000<br>mmap(0x7f51559f1000, 15072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f51559f1000<br>close(3) = 0<br>arch_prctl(ARCH_SET_FS, 0x7f5155c11580) = 0<br>mprotect(0x7f51559eb000, 16384, PROT_READ) = 0<br>mprotect(0x561349fd2000, 4096, PROT_READ) = 0<br>mprotect(0x7f5155c1c000, 4096, PROT_READ) = 0<br>munmap(0x7f5155c12000, 37110) = 0<br>chdir("/") = 0<br>stat("/proc/drbd", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0<br>openat(AT_FDCWD, "/proc/drbd", O_RDONLY) = 3<br>brk(NULL) = 0x56134ae0f000<br>brk(0x56134ae30000) = 0x56134ae30000<br>read(3, "version: 9.0.19-1 (api:2/proto:8"..., 4095) = 170<br>close(3) = 0<br>socket(AF_NETLINK, SOCK_DGRAM, NETLINK_GENERIC) = 3<br>setsockopt(3, SOL_SOCKET, SO_SNDBUF, [1048576], 4) = 0<br>setsockopt(3, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0<br>bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0<br>getsockname(3, {sa_family=AF_NETLINK, nl_pid=8512, nl_groups=00000000}, [12]) = 0<br>write(3, " \0\0\0\20\0\1\0\340\t\321^@!\0\0\3\2\0\0\t\0\2\0drbd\0\0\0\0", 32) = 32<br>poll([{fd=3, events=POLLIN}], 1, 3000) = 1 ([{fd=3, revents=POLLIN}])<br>recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=816, type=nlctrl, flags=0, seq=1590757856, pid=8512}, "\x01\x02\x00\x00\x09\x00\x02\x00\x64\x72\x62\x64\x00\x00\x00\x00\x06\x00\x01\x00\x1f\x00\x00\x00\x08\x00\x03\x00\x02\x00\x00\x00"...}, iov_len=8192}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_PEEK) = 816<br>poll([{fd=3, events=POLLIN}], 1, 3000) = 1 ([{fd=3, revents=POLLIN}])<br>recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=816, type=nlctrl, flags=0, seq=1590757856, pid=8512}, "\x01\x02\x00\x00\x09\x00\x02\x00\x64\x72\x62\x64\x00\x00\x00\x00\x06\x00\x01\x00\x1f\x00\x00\x00\x08\x00\x03\x00\x02\x00\x00\x00"...}, iov_len=8192}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 816<br>write(3, "8\0\0\0\37\0\1\0\341\t\321^@!\0\0\17\2\0\0\377\377\377\377\0\0\0\0\34\0\2\0"..., 56<br></div><div><br></div><div><div dir="ltr"><div dir="ltr"><div>Best Regards,</div><div>Andrei Kvapil<br></div></div></div></div></div>
</blockquote></div>