<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>Hi Everyone,</p>
    <p><br>
    </p>
    <p>we're having the same issue described in here:
      <a class="moz-txt-link-freetext" href="http://lists.linbit.com/pipermail/drbd-user/2018-November/024667.html">http://lists.linbit.com/pipermail/drbd-user/2018-November/024667.html</a>
      in the exact</p>
    <p>same constellation. Sadly, no one aswered to that Thread so i'm
      opening a new one.</p>
    <p><br>
    </p>
    <p>Packages installed on both nodes:</p>
    <p><br>
    </p>
    <p>root@node2:~# dpkg -l | grep 'drbd\|linstor'<br>
      ii  drbd-dkms                           
      9.0.17-1                       all          RAID 1 over TCP/IP for
      Linux module source<br>
      ii  drbd-utils                          
      9.9.0-1                        amd64        RAID 1 over TCP/IP for
      Linux (user utilities)<br>
      ii  drbdtop                             
      0.2.1-1                        amd64        like top, but for drbd<br>
      ii  linstor-client                      
      0.9.5-1                        all          Linstor client command
      line tool<br>
      ii  linstor-common                      
      0.9.10-1                       all          DRBD distributed
      resource management utility<br>
      ii  linstor-proxmox                     
      3.0.3-1                        all          DRBD distributed
      resource management utility<br>
      ii  linstor-satellite                   
      0.9.10-1                       all          DRBD distributed
      resource management utility<br>
      ii  python-linstor                      
      0.9.5-1                        all          Linstor python api
      library<br>
      <br>
    </p>
    <pre class="moz-signature" cols="72">


</pre>
    <p>Proxmox Error Message: <br>
    </p>
    <p><br>
    </p>
    <p>()<br>
      SUCCESS:<br>
      Description:<br>
          New resource definition 'vm-102-state-test123' created.<br>
      Details:<br>
          Resource definition 'vm-102-state-test123' UUID is:
      7eab2e30-5a5a-4e57-a4e2-d5da7c58f770<br>
      SUCCESS:<br>
      Description:<br>
          Resource definition 'vm-102-state-test123' modified.<br>
      Details:<br>
          Resource definition 'vm-102-state-test123' UUID is:
      7eab2e30-5a5a-4e57-a4e2-d5da7c58f770<br>
      SUCCESS:<br>
          New volume definition with number '0' of resource definition
      'vm-102-state-test123' created.<br>
      SUCCESS:<br>
      Description:<br>
          Resource 'vm-102-state-test123' successfully autoplaced on 2
      nodes<br>
      Details:<br>
          Used storage pool: 'drbdpool'<br>
          Used nodes: 'node1', 'node2'<br>
      SUCCESS:<br>
          Created resource 'vm-102-state-test123' on 'node2'<br>
      SUCCESS:<br>
          Created resource 'vm-102-state-test123' on 'node1'<br>
      SUCCESS:<br>
      Description:<br>
          Resource 'vm-102-state-test123' on 'node2' ready<br>
      Details:<br>
          Auto-placing resource: vm-102-state-test123<br>
      SUCCESS:<br>
      Description:<br>
          Resource 'vm-102-state-test123' on 'node1' ready<br>
      Details:<br>
          Auto-placing resource: vm-102-state-test123<br>
      SUCCESS:<br>
      Description:<br>
          New snapshot 'snap_vm-102-disk-1_test123' of resource
      'vm-102-disk-1' registered.<br>
      Details:<br>
          Snapshot 'snap_vm-102-disk-1_test123' of resource
      'vm-102-disk-1' UUID is: 832dbdf2-b3dc-46f9-8f1d-ac9d20c1790c<br>
      SUCCESS:<br>
          Suspended IO of 'vm-102-disk-1' on 'node2' for snapshot<br>
      SUCCESS:<br>
          Suspended IO of 'vm-102-disk-1' on 'node1' for snapshot<br>
      ERROR:<br>
      Description:<br>
          (Node: 'node2') Preparing resources for layer StorageLayer
      failed<br>
      Cause:<br>
          External command timed out<br>
      Details:<br>
          External command: lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
      ERROR:<br>
      Description:<br>
          (Node: 'node1') Preparing resources for layer StorageLayer
      failed<br>
      Cause:<br>
          External command timed out<br>
      Details:<br>
          External command: lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
      ERROR:<br>
      Description:<br>
          (Node: 'node2') Preparing resources for layer StorageLayer
      failed<br>
      Cause:<br>
          External command timed out<br>
      Details:<br>
          External command: lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
      ERROR:<br>
      Description:<br>
          (Node: 'node1') Preparing resources for layer StorageLayer
      failed<br>
      Cause:<br>
          External command timed out<br>
      Details:<br>
          External command: lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
      VM 102 qmp command 'savevm-end' failed - unable to connect to VM
      102 qmp socket - timeout after 5992 retries<br>
      snapshot create failed: starting cleanup<br>
      error with cfs lock 'storage-drbdpool': Could not remove
      vm-102-state-test123: got lock timeout - aborting command<br>
      TASK ERROR: Could not create cluster wide snapshot for:
      vm-102-disk-1: exit code 10</p>
    <p><br>
    </p>
    <p><br>
    </p>
    <p><br>
    </p>
    <p>Haning Tasks:</p>
    <p><br>
    </p>
    <p>root@node2:~# systemctl status linstor-satellite.service <br>
      ● linstor-satellite.service - LINSTOR Satellite Service<br>
         Loaded: loaded (/lib/systemd/system/linstor-satellite.service;
      enabled; vendor preset: enabled)<br>
        Drop-In: /etc/systemd/system/linstor-satellite.service.d<br>
                 └─override.conf<br>
         Active: active (running) since Tue 2019-05-28 17:42:11 CEST;
      30min ago<br>
       Main PID: 4612 (java)<br>
          Tasks: 303 (limit: 14745)<br>
         Memory: 263.1M<br>
            CPU: 24.237s<br>
         CGroup: /system.slice/linstor-satellite.service<br>
                 ├─ 4612 java -Xms32M -XX:MinHeapFreeRatio=40
      -XX:MaxHeapFreeRatio=70 -classpath
/usr/share/linstor-server/lib/satellite-0.9.10.jar:/usr/share/linstor-server/lib/reactive-streams-1.0.2.jar:/usr/share/linstor-server/lib/logback-classic-1.2.3.ja<br>
                 ├─ 5406 drbdsetup events2 all<br>
                 ├─16954 lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
                 ├─17236 vgs -o vg_name --noheadings<br>
                 ├─17525 lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
                 ├─17788 vgs -o vg_name --noheadings<br>
                 ├─18075 vgs -o vg_name --noheadings<br>
                 ├─18323 vgs -o vg_name --noheadings<br>
                 ├─18612 vgs -o vg_name --noheadings<br>
                 ├─18889 vgs -o vg_name --noheadings<br>
                 ├─19162 vgs -o vg_name --noheadings<br>
                 ├─19442 vgs -o vg_name --noheadings<br>
                 ├─19700 vgs -o vg_name --noheadings<br>
                 ├─19984 vgs -o vg_name --noheadings<br>
                 ├─20242 vgs -o vg_name --noheadings<br>
                 ├─20523 lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
                 ├─20779 vgs -o vg_name --noheadings<br>
                 ├─21047 lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
                 ├─21310 vgs -o vg_name --noheadings<br>
                 ├─21572 lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
                 ├─21821 vgs -o vg_name --noheadings<br>
                 ├─22083 lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
                 ├─22346 vgs -o vg_name --noheadings<br>
                 ├─22612 lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
                 ├─22864 vgs -o vg_name --noheadings<br>
                 └─23135 lvs -o
      lv_name,lv_path,lv_size,vg_name,pool_lv,data_percent,lv_attr
      --separator ; --noheadings --units k --nosuffix drbdpool<br>
    </p>
    <p><br>
    </p>
    <p><br>
    </p>
    <p>Kernel Stack Trace: <br>
    </p>
    <p><br>
    </p>
    <p>[  967.562965] INFO: task kvm:18314 blocked for more than 120
      seconds.<br>
      [  967.563540]       Tainted: P           O     4.15.18-14-pve #1<br>
      [  967.563974] "echo 0 &gt;
      /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
      [  967.564414] kvm             D    0 18314      1 0x00000000<br>
      [  967.564419] Call Trace:<br>
      [  967.564435]  __schedule+0x3e0/0x870<br>
      [  967.564440]  ? touch_pmd+0x32/0x40<br>
      [  967.564442]  schedule+0x36/0x80<br>
      [  967.564460]  __drbd_make_request+0x27c/0x5b0 [drbd]<br>
      [  967.564465]  ? wait_woken+0x80/0x80<br>
      [  967.564472]  drbd_make_request+0x4f/0x80 [drbd]<br>
      [  967.564477]  generic_make_request+0x123/0x2f0<br>
      [  967.564479]  submit_bio+0x73/0x140<br>
      [  967.564480]  ? submit_bio+0x73/0x140<br>
      [  967.564483]  blkdev_direct_IO+0x3c0/0x400<br>
      [  967.564486]  ? do_io_getevents+0xd0/0xd0<br>
      [  967.564489]  ? generic_update_time+0x7b/0xd0<br>
      [  967.564490]  ? current_time+0x32/0x70<br>
      [  967.564494]  generic_file_read_iter+0xb1/0xbb0<br>
      [  967.564496]  ? generic_file_read_iter+0xb1/0xbb0<br>
      [  967.564498]  ? __check_object_size+0xb3/0x190<br>
      [  967.564500]  ? _copy_from_user+0x36/0x70<br>
      [  967.564502]  blkdev_read_iter+0x35/0x40<br>
      [  967.564503]  aio_read+0xfc/0x140<br>
      [  967.564505]  do_io_submit+0x530/0x6c0<br>
      [  967.564506]  ? do_io_submit+0x530/0x6c0<br>
      [  967.564508]  SyS_io_submit+0x10/0x20<br>
      [  967.564509]  ? SyS_io_submit+0x10/0x20<br>
      [  967.564512]  do_syscall_64+0x73/0x130<br>
      [  967.564514]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2<br>
      [  967.564517] RIP: 0033:0x7f1f217d6717<br>
      [  967.564517] RSP: 002b:00007f1de9999558 EFLAGS: 00000246
      ORIG_RAX: 00000000000000d1<br>
      [  967.564519] RAX: ffffffffffffffda RBX: 0000000000000001 RCX:
      00007f1f217d6717<br>
      [  967.564520] RDX: 00007f1de9999570 RSI: 0000000000000001 RDI:
      00007f1f23117000<br>
      [  967.564521] RBP: 00007f1de9999570 R08: 0000000000000008 R09:
      0000000000000000<br>
      [  967.564521] R10: 00007f1efc94ba60 R11: 0000000000000246 R12:
      00007f1de9999560<br>
      [  967.564522] R13: 00007f1efc94ba38 R14: 00007f1efc94ba10 R15:
      000000114f118a00<br>
      [  967.564531] INFO: task lvs:18730 blocked for more than 120
      seconds.<br>
      [  967.564975]       Tainted: P           O     4.15.18-14-pve #1<br>
      [  967.565425] "echo 0 &gt;
      /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
      [  967.565894] lvs             D    0 18730   5826 0x00000004<br>
      [  967.565896] Call Trace:<br>
      [  967.565898]  __schedule+0x3e0/0x870<br>
      [  967.565900]  schedule+0x36/0x80<br>
      [  967.565907]  __drbd_make_request+0x27c/0x5b0 [drbd]<br>
      [  967.565909]  ? wait_woken+0x80/0x80<br>
      [  967.565915]  drbd_make_request+0x4f/0x80 [drbd]<br>
      [  967.565917]  generic_make_request+0x123/0x2f0<br>
      [  967.565918]  submit_bio+0x73/0x140<br>
      [  967.565920]  ? submit_bio+0x73/0x140<br>
      [  967.565921]  __blkdev_direct_IO_simple+0x1b1/0x380<br>
      [  967.565922]  ? __blkdev_get+0x240/0x4d0<br>
      [  967.565924]  ? bdget+0x120/0x120<br>
      [  967.565925]  blkdev_direct_IO+0x3f1/0x400<br>
      [  967.565926]  ? blkdev_direct_IO+0x3f1/0x400<br>
      [  967.565929]  ? dput+0xc7/0x1f0<br>
      [  967.565931]  ? mntput+0x24/0x40<br>
      [  967.565932]  ? terminate_walk+0x91/0xf0<br>
      [  967.565934]  ? path_openat+0x6f7/0x14a0<br>
      [  967.565935]  ? putname+0x54/0x60<br>
      [  967.565936]  generic_file_read_iter+0xb1/0xbb0<br>
      [  967.565938]  ? generic_file_read_iter+0xb1/0xbb0<br>
      [  967.565944]  ? kref_debug_sub+0x3c/0x60 [drbd]<br>
      [  967.565946]  ? _copy_to_user+0x2a/0x40<br>
      [  967.565948]  ? cp_new_stat+0x152/0x180<br>
      [  967.565949]  blkdev_read_iter+0x35/0x40<br>
      [  967.565951]  new_sync_read+0xe4/0x130<br>
      [  967.565952]  __vfs_read+0x29/0x40<br>
      [  967.565954]  vfs_read+0x96/0x130<br>
      [  967.565955]  SyS_read+0x55/0xc0<br>
      [  967.565957]  do_syscall_64+0x73/0x130<br>
      [  967.565958]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2<br>
      [  967.565959] RIP: 0033:0x7f1d049a9910<br>
      [  967.565960] RSP: 002b:00007ffec8316918 EFLAGS: 00000246
      ORIG_RAX: 0000000000000000<br>
      [  967.565961] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
      00007f1d049a9910<br>
      [  967.565962] RDX: 0000000000001000 RSI: 0000556670ae3000 RDI:
      0000000000000004<br>
      [  967.565963] RBP: 00007ffec8316970 R08: 00007f1d04c68258 R09:
      0000000000001000<br>
      [  967.565963] R10: 0000000000000030 R11: 0000000000000246 R12:
      0000556670ae3000<br>
      [  967.565964] R13: 0000000000000000 R14: 0000000000000004 R15:
      00007ffec83169d0<br>
      [  967.565968] INFO: task lvs:18773 blocked for more than 120
      seconds.<br>
      [  967.566433]       Tainted: P           O     4.15.18-14-pve #1<br>
      [  967.566944] "echo 0 &gt;
      /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
      [  967.567435] lvs             D    0 18773  18766 0x00000004<br>
      [  967.567438] Call Trace:<br>
      [  967.567441]  __schedule+0x3e0/0x870<br>
      [  967.567443]  schedule+0x36/0x80<br>
      [  967.567450]  __drbd_make_request+0x27c/0x5b0 [drbd]<br>
      [  967.567452]  ? wait_woken+0x80/0x80<br>
      [  967.567458]  drbd_make_request+0x4f/0x80 [drbd]<br>
      [  967.567460]  generic_make_request+0x123/0x2f0<br>
      [  967.567462]  submit_bio+0x73/0x140<br>
      [  967.567464]  ? submit_bio+0x73/0x140<br>
      [  967.567465]  __blkdev_direct_IO_simple+0x1b1/0x380<br>
      [  967.567467]  ? __blkdev_get+0x240/0x4d0<br>
      [  967.567468]  ? bdget+0x120/0x120<br>
      [  967.567470]  blkdev_direct_IO+0x3f1/0x400<br>
      [  967.567471]  ? blkdev_direct_IO+0x3f1/0x400<br>
      [  967.567472]  ? dput+0xc7/0x1f0<br>
      [  967.567474]  ? mntput+0x24/0x40<br>
      [  967.567476]  ? terminate_walk+0x91/0xf0<br>
      [  967.567478]  ? path_openat+0x6f7/0x14a0<br>
      [  967.567479]  ? putname+0x54/0x60<br>
      [  967.567480]  generic_file_read_iter+0xb1/0xbb0<br>
      [  967.567482]  ? generic_file_read_iter+0xb1/0xbb0<br>
      [  967.567488]  ? kref_debug_sub+0x3c/0x60 [drbd]<br>
      [  967.567490]  ? _copy_to_user+0x2a/0x40<br>
      [  967.567492]  ? cp_new_stat+0x152/0x180<br>
      [  967.567493]  blkdev_read_iter+0x35/0x40<br>
      [  967.567494]  new_sync_read+0xe4/0x130<br>
      [  967.567497]  __vfs_read+0x29/0x40<br>
      [  967.567499]  vfs_read+0x96/0x130<br>
      [  967.567500]  SyS_read+0x55/0xc0<br>
      [  967.567502]  do_syscall_64+0x73/0x130<br>
      [  967.567504]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2<br>
      [  967.567505] RIP: 0033:0x7f313938e910<br>
      [  967.567506] RSP: 002b:00007fffa10d77c8 EFLAGS: 00000246
      ORIG_RAX: 0000000000000000<br>
      [  967.567507] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
      00007f313938e910<br>
      [  967.567508] RDX: 0000000000001000 RSI: 0000564dbe7da000 RDI:
      0000000000000004<br>
      [  967.567509] RBP: 00007fffa10d7820 R08: 00007f313964d248 R09:
      0000000000001000<br>
      [  967.567509] R10: 0000000000000030 R11: 0000000000000246 R12:
      0000564dbe7da000<br>
      [  967.567510] R13: 0000000000000000 R14: 0000000000000004 R15:
      00007fffa10d7880<br>
    </p>
    <p><br>
    </p>
    <p><br>
    </p>
    <p><br>
    </p>
    <p>We also have LVM_THIN Storage Pools.</p>
    <p><br>
    </p>
    <p>Maybe there's anybody that could help us and resolve this Thread
      and the one before.</p>
    <p><br>
    </p>
    <p><br>
    </p>
    <p>Kind Regards,</p>
    <p>Alex<br>
    </p>
    <p><br>
    </p>
    <p><br>
    </p>
    <div id="proxmoxLogView-7497-body" data-ref="body"
      class="x-panel-body x-panel-body-default x-panel-body-default
      x-docked-noborder-right x-docked-noborder-bottom
      x-docked-noborder-left" role="presentation" style="padding: 0px;
      overflow: auto; width: 794px; height: 287px; left: 0px; top:
      36px;">
      <div id="proxmoxLogView-7497-outerCt" data-ref="outerCt"
        class="x-autocontainer-outerCt" role="presentation"
        style="width: 100%; height: 100%;">
        <div id="proxmoxLogView-7497-innerCt" data-ref="innerCt"
          style="padding:5px 5px 5px 5px" role="presentation"
          class="x-autocontainer-innerCt">
          <div class="x-component x-component-default" style="font:normal 11px tahoma, arial, verdana, sans-serif;white-space:pre;line-height:16px;" id="component-7498">
</div>
        </div>
      </div>
    </div>
    <pre class="moz-signature" cols="72">
</pre>
  </body>
</html>