[DRBD-user] linstor-proxmox snapshots causes tainted kernel
Greb
opengreb at free.fr
Fri Nov 23 16:04:52 CET 2018
Hello,
I just tested again after installing the latest updates, unfortunately the problem persists:
kernel: Tainted: P O 4.15.18-9 -pve # 1
drbd-dkms 9.0.16-1
drbd-utils 9.6.0-1
drbdtop 0.2.1-1
linstor-client 0.7.2-1
linstor-common 0.7.3-1
linstor-controller 0.7.3-1
linstor-proxmox 3.0.2-2
linstor-satellite 0.7.3-1
python-linstor 0.7.2-1
pve-manager / 5.2-11 / 13c2da63 (running kernel: 4.15.18-9-pve)
Here are the details of the errors as requested:
Thank you,
TASK : VM 100 - Snapshot
SUCCESS:
Description:
New resource definition 'vm-100-state-test_snap_14_08' created.
Details:
Resource definition 'vm-100-state-test_snap_14_08' UUID is: aa2b8e23-ffeb-4e2e-810c-66c4e252f575
SUCCESS:
Description:
Resource definition 'vm-100-state-test_snap_14_08' modified.
Details:
Resource definition 'vm-100-state-test_snap_14_08' UUID is: aa2b8e23-ffeb-4e2e-810c-66c4e252f575
SUCCESS:
New volume definition with number '0' of resource definition 'vm-100-state-test_snap_14_08' created.
SUCCESS:
Description:
Resource 'vm-100-state-test_snap_14_08' successfully autoplaced on 2 nodes
Details:
Used storage pool: 'drbdpool'
Used nodes: 'vpx3-2', 'vpx3-1'
SUCCESS:
Created resource 'vm-100-state-test_snap_14_08' on 'vpx3-2'
SUCCESS:
Created resource 'vm-100-state-test_snap_14_08' on 'vpx3-1'
SUCCESS:
Description:
Resource 'vm-100-state-test_snap_14_08' on 'vpx3-2' ready
Details:
Self-placing resource: vm-100-state-test_snap_14_08
SUCCESS:
Description:
Resource 'vm-100-state-test_snap_14_08' on 'vpx3-1' ready
Details:
Self-placing resource: vm-100-state-test_snap_14_08
SUCCESS:
Description:
New snapshot 'snap_vm-100-disk-2_test_snap_14_08' of resource 'vm-100-disk-2' registered.
Details:
Snapshot 'snap_vm-100-disk-2_test_snap_14_08' of resource 'vm-100-disk-2' UUID is: 6b944397-6536-4fc7-85a1-05d69f580c30
SUCCESS:
Suspended IO of 'vm-100-disk-2' on 'vpx3-2' for snapshot
SUCCESS:
Suspended IO of 'vm-100-disk-2' on 'vpx3-1' for snapshot
ERROR:
(Node: 'vpx3-2') Storage configuration exception
Show reports:
linstor error-reports show 5BF6D8BA-12EE8-000000
ERROR:
(Node: 'vpx3-1') Storage configuration exception
Show reports:
linstor error-reports show 5BF6D8AF-6B93F-000000
ERROR:
(Node: 'vpx3-2') Storage configuration exception
Show reports:
linstor error-reports show 5BF6D8BA-12EE8-000001
ERROR:
(Node: 'vpx3-1') Storage configuration exception
Show reports:
linstor error-reports show 5BF6D8AF-6B93F-000001
snapshot create failed: starting cleanup
error with cfs lock 'storage-drbdstorage': Could not remove vm-100-state-test_snap_14_08: got lock timeout - aborting command
TASK ERROR: Could not create cluster wide snapshot for: vm-100-disk-2: exit code 10
╭────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ SnapshotName ┊ NodeNames ┊ Volumes ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ vm-100-disk-2 ┊ snap_vm-100-disk-2_test_snap_14_08 ┊ vpx3-1, vpx3-2 ┊ 0: 2.01 GiB ┊ Failed ┊
╰────────────────────────────────────────────────────────────────────────────────────────────╯
ERROR REPORT 5BF6D8BA-12EE8-000000
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.7.3
Build ID: 6e47dd2cb440e65727889f4c2f7423996153bdb3
Build time: 2018-11-22T10:24:37+00:00
Error time: 2018-11-23 14:10:13
Node: vpx3-2
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ensureStorageDriver', Source file 'DrbdDeviceHandler.java', Line #777
Error message: Storage configuration exception
Error context:
Storage configuration exception
Call backtrace:
Method Native Class:Line number
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:777
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Description:
Failed to verify volume group name
Cause:
External command timed out
Additional information:
External command: vgs -o vg_name --noheadings
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'checkVolumeGroupEntry', Source file 'LvmDriver.java', Line #479
Error message: Failed to verify volume group name
Call backtrace:
Method Native Class:Line number
checkVolumeGroupEntry N com.linbit.linstor.storage.LvmDriver:479
checkConfiguration N com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Category: Exception
Class name: ChildProcessTimeoutException
Class canonical name: com.linbit.ChildProcessTimeoutException
Generated at: Method 'waitFor', Source file 'ChildProcessHandler.java', Line #187
Call backtrace:
Method Native Class:Line number
waitFor N com.linbit.extproc.ChildProcessHandler:187
waitFor N com.linbit.extproc.ChildProcessHandler:124
syncProcess N com.linbit.extproc.ExtCmd:92
exec N com.linbit.extproc.ExtCmd:56
checkVolumeGroupEntry N com.linbit.linstor.storage.LvmDriver:446
checkConfiguration N com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
END OF ERROR REPORT.
ERROR REPORT 5BF6D8AF-6B93F-000000
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.7.3
Build ID: 6e47dd2cb440e65727889f4c2f7423996153bdb3
Build time: 2018-11-22T10:24:37+00:00
Error time: 2018-11-23 14:10:13
Node: vpx3-1
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ensureStorageDriver', Source file 'DrbdDeviceHandler.java', Line #777
Error message: Storage configuration exception
Error context:
Storage configuration exception
Call backtrace:
Method Native Class:Line number
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:777
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Description:
Failed to verify volume group name
Cause:
External command timed out
Additional information:
External command: vgs -o vg_name --noheadings
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'checkVolumeGroupEntry', Source file 'LvmDriver.java', Line #479
Error message: Failed to verify volume group name
Call backtrace:
Method Native Class:Line number
checkVolumeGroupEntry N com.linbit.linstor.storage.LvmDriver:479
checkConfiguration N com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Category: Exception
Class name: ChildProcessTimeoutException
Class canonical name: com.linbit.ChildProcessTimeoutException
Generated at: Method 'waitFor', Source file 'ChildProcessHandler.java', Line #187
Call backtrace:
Method Native Class:Line number
waitFor N com.linbit.extproc.ChildProcessHandler:187
waitFor N com.linbit.extproc.ChildProcessHandler:124
syncProcess N com.linbit.extproc.ExtCmd:92
exec N com.linbit.extproc.ExtCmd:56
checkVolumeGroupEntry N com.linbit.linstor.storage.LvmDriver:446
checkConfiguration N com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
END OF ERROR REPORT.
ERROR REPORT 5BF6D8BA-12EE8-000001
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.7.3
Build ID: 6e47dd2cb440e65727889f4c2f7423996153bdb3
Build time: 2018-11-22T10:24:37+00:00
Error time: 2018-11-23 14:11:19
Node: vpx3-2
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ensureStorageDriver', Source file 'DrbdDeviceHandler.java', Line #777
Error message: Storage configuration exception
Error context:
Storage configuration exception
Call backtrace:
Method Native Class:Line number
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:777
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Description:
Failed to verify volume group name
Cause:
External command timed out
Additional information:
External command: vgs -o vg_name --noheadings
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'checkVolumeGroupEntry', Source file 'LvmDriver.java', Line #479
Error message: Failed to verify volume group name
Call backtrace:
Method Native Class:Line number
checkVolumeGroupEntry N com.linbit.linstor.storage.LvmDriver:479
checkConfiguration N com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Category: Exception
Class name: ChildProcessTimeoutException
Class canonical name: com.linbit.ChildProcessTimeoutException
Generated at: Method 'waitFor', Source file 'ChildProcessHandler.java', Line #187
Call backtrace:
Method Native Class:Line number
waitFor N com.linbit.extproc.ChildProcessHandler:187
waitFor N com.linbit.extproc.ChildProcessHandler:124
syncProcess N com.linbit.extproc.ExtCmd:92
exec N com.linbit.extproc.ExtCmd:56
checkVolumeGroupEntry N com.linbit.linstor.storage.LvmDriver:446
checkConfiguration N com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
END OF ERROR REPORT.
ERROR REPORT 5BF6D8AF-6B93F-000001
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.7.3
Build ID: 6e47dd2cb440e65727889f4c2f7423996153bdb3
Build time: 2018-11-22T10:24:37+00:00
Error time: 2018-11-23 14:11:19
Node: vpx3-1
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ensureStorageDriver', Source file 'DrbdDeviceHandler.java', Line #777
Error message: Storage configuration exception
Error context:
Storage configuration exception
Call backtrace:
Method Native Class:Line number
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:777
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Description:
Failed to verify volume group name
Cause:
External command timed out
Additional information:
External command: vgs -o vg_name --noheadings
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'checkVolumeGroupEntry', Source file 'LvmDriver.java', Line #479
Error message: Failed to verify volume group name
Call backtrace:
Method Native Class:Line number
checkVolumeGroupEntry N com.linbit.linstor.storage.LvmDriver:479
checkConfiguration N com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Category: Exception
Class name: ChildProcessTimeoutException
Class canonical name: com.linbit.ChildProcessTimeoutException
Generated at: Method 'waitFor', Source file 'ChildProcessHandler.java', Line #187
Call backtrace:
Method Native Class:Line number
waitFor N com.linbit.extproc.ChildProcessHandler:187
waitFor N com.linbit.extproc.ChildProcessHandler:124
syncProcess N com.linbit.extproc.ExtCmd:92
exec N com.linbit.extproc.ExtCmd:56
checkVolumeGroupEntry N com.linbit.linstor.storage.LvmDriver:446
checkConfiguration N com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N com.linbit.linstor.core.DrbdDeviceHandler:364
run N com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N com.linbit.WorkerPool$WorkerThread:179
END OF ERROR REPORT.
Nov 23 14:08:35 vpx3-1 pvedaemon[17198]: <root at pam> snapshot VM 100: test_snap_14_08
Nov 23 14:08:35 vpx3-1 pvedaemon[13971]: <root at pam> starting task UPID:vpx3-1:0000432E:0071C1A2:5BF7FBD3:qmsnapshot:100:root at pam:
Nov 23 14:08:39 vpx3-1 Controller[1009]: 14:08:38.992 [MainWorkerPool-2] INFO LINSTOR/Controller - New volume definition with number '0' of resource definition 'vm-100-state-test_snap_14_08' created.
Nov 23 14:08:40 vpx3-1 Satellite[1021]: 14:08:40.502 [MainWorkerPool-2] INFO LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' created for node 'vpx3-1'.
Nov 23 14:08:40 vpx3-1 Satellite[1021]: 14:08:40.613 [MainWorkerPool-2] INFO LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' created for node 'vpx3-2'.
Nov 23 14:08:41 vpx3-1 Satellite[1021]: 14:08:41.174 [StltWorkerPool_0001] INFO LINSTOR/Satellite - Aligning size from 1560948 KiB to 1564672 KiB to be a multiple of extent size 4096 KiB
Nov 23 14:08:42 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Starting worker thread (from drbdsetup [17326])
Nov 23 14:08:42 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Starting sender thread (from drbdsetup [17333])
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: disk( Diskless -> Attaching )
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: Maximum number of peer devices = 7
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Method to ensure write ordering: flush
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: drbd_bm_resize called with capacity == 3128600
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: resync bitmap: bits=391075 words=42777 pages=84
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: size = 1528 MB (1564300 KB)
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: recounting of set bits took additional 0ms
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: disk( Attaching -> UpToDate )
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: attached to current UUID: 3B5634C1B39D3086
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: size = 1528 MB (1564300 KB)
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn( StandAlone -> Unconnected )
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Starting receiver thread (from drbd_w_vm-100-s [17327])
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn( Unconnected -> Connecting )
Nov 23 14:08:43 vpx3-1 Satellite[1021]: 14:08:43.140 [MainWorkerPool-2] INFO LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' updated for node 'vpx3-1'.
Nov 23 14:08:43 vpx3-1 Satellite[1021]: 14:08:43.140 [MainWorkerPool-2] INFO LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' updated for node 'vpx3-2'.
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Handshake to peer 0 successful: Agreed network protocol version 114
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Peer authenticated using 20 bytes HMAC
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Starting ack_recv thread (from drbd_r_vm-100-s [17349])
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Preparing remote state change 2055326401
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Committing remote state change 2055326401 (primary_nodes=0)
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004 vpx3-2: drbd_sync_handshake:
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004 vpx3-2: self 3B5634C1B39D3086:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004 vpx3-2: peer 3B5634C1B39D3086:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004 vpx3-2: uuid_compare()=0 by rule 40
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004 vpx3-2: pdsk( DUnknown -> UpToDate ) repl( Off -> Established )
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Preparing cluster-wide state change 1246189751 (1->-1 3/1)
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: State change 1246189751: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Committing cluster-wide state change 1246189751 (4ms)
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: role( Secondary -> Primary )
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: new current UUID: B89F98A90ABEF3CF weak: FFFFFFFFFFFFFFFC
Nov 23 14:09:01 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:09:03 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:09:05 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: role( Primary -> Secondary )
Nov 23 14:09:06 vpx3-1 pvestatd[1477]: status update time (14.130 seconds)
Nov 23 14:09:08 vpx3-1 Satellite[1021]: 14:09:08.181 [MainWorkerPool-1] INFO LINSTOR/Satellite - Snapshot 'snap_vm-100-disk-2_test_snap_14_08' of resource 'vm-100-disk-2' registered.
Nov 23 14:09:08 vpx3-1 kernel: drbd vm-100-disk-2: susp-io( no -> user)
Nov 23 14:09:08 vpx3-1 Satellite[1021]: 14:09:08.623 [MainWorkerPool-1] INFO LINSTOR/Satellite - Snapshot 'snap_vm-100-disk-2_test_snap_14_08' of resource 'vm-100-disk-2' registered.
Nov 23 14:10:00 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:10:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:10:14 vpx3-1 Satellite[1021]: 14:10:14.152 [StltWorkerPool_0002] ERROR LINSTOR/Satellite - Storage configuration exception [Report number 5BF6D8AF-6B93F-000000]
Nov 23 14:10:14 vpx3-1 Satellite[1021]: 14:10:14.371 [MainWorkerPool-1] INFO LINSTOR/Satellite - Snapshot 'snap_vm-100-disk-2_test_snap_14_08' ended.
Nov 23 14:11:00 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:11:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:11:19 vpx3-1 Satellite[1021]: 14:11:19.416 [StltWorkerPool_0001] ERROR LINSTOR/Satellite - Storage configuration exception [Report number 5BF6D8AF-6B93F-000001]
Nov 23 14:11:20 vpx3-1 pvedaemon[17198]: snapshot create failed: starting cleanup
Nov 23 14:11:21 vpx3-1 Satellite[1021]: 14:11:21.156 [MainWorkerPool-1] INFO LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' updated for node 'vpx3-1'.
Nov 23 14:11:21 vpx3-1 Satellite[1021]: 14:11:21.157 [MainWorkerPool-1] INFO LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' updated for node 'vpx3-2'.
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Preparing cluster-wide state change 1830748394 (1->0 496/16)
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: State change 1830748394: primary_nodes=0, weak_nodes=0
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Cluster is now split
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Committing cluster-wide state change 1830748394 (0ms)
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004 vpx3-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: ack_receiver terminated
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Terminating ack_recv thread
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Connection closed
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn( Disconnecting -> StandAlone )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Terminating receiver thread
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Terminating sender thread
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: disk( UpToDate -> Detaching )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: disk( Detaching -> Diskless )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004: drbd_bm_resize called with capacity == 0
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Terminating worker thread
Nov 23 14:12:00 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:12:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:12:20 vpx3-1 pvedaemon[17198]: error with cfs lock 'storage-drbdstorage': Could not remove vm-100-state-test_snap_14_08: got lock timeout - aborting command
Nov 23 14:12:20 vpx3-1 pvedaemon[17198]: Could not create cluster wide snapshot for: vm-100-disk-2: exit code 10
Nov 23 14:12:20 vpx3-1 pvedaemon[13971]: <root at pam> end task UPID:vpx3-1:0000432E:0071C1A2:5BF7FBD3:qmsnapshot:100:root at pam: Could not create cluster wide snapshot for: vm-100-disk-2: exit code 10
Nov 23 14:12:26 vpx3-1 Satellite[1021]: 14:12:26.295 [StltWorkerPool_0000] ERROR LINSTOR/Satellite - Failed to verify volume group name [Report number 5BF6D8AF-6B93F-000002]
Nov 23 14:12:40 vpx3-1 kernel: INFO: task kvm:17785 blocked for more than 120 seconds.
Nov 23 14:12:40 vpx3-1 kernel: Tainted: P O 4.15.18-9-pve #1
Nov 23 14:12:40 vpx3-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 14:12:40 vpx3-1 kernel: kvm D 0 17785 1 0x00000000
Nov 23 14:12:40 vpx3-1 kernel: Call Trace:
Nov 23 14:12:40 vpx3-1 kernel: __schedule+0x3e0/0x870
Nov 23 14:12:40 vpx3-1 kernel: schedule+0x36/0x80
Nov 23 14:12:40 vpx3-1 kernel: io_schedule+0x16/0x40
Nov 23 14:12:40 vpx3-1 kernel: __lock_page+0xff/0x140
Nov 23 14:12:40 vpx3-1 kernel: ? page_cache_tree_insert+0xe0/0xe0
Nov 23 14:12:40 vpx3-1 kernel: pagecache_get_page+0x195/0x2b0
Nov 23 14:12:40 vpx3-1 kernel: ? I_BDEV+0x20/0x20
Nov 23 14:12:40 vpx3-1 kernel: grab_cache_page_write_begin+0x23/0x40
Nov 23 14:12:40 vpx3-1 kernel: block_write_begin+0x2d/0xe0
Nov 23 14:12:40 vpx3-1 kernel: blkdev_write_begin+0x23/0x30
Nov 23 14:12:40 vpx3-1 kernel: generic_perform_write+0xb9/0x1b0
Nov 23 14:12:40 vpx3-1 kernel: __generic_file_write_iter+0x185/0x1c0
Nov 23 14:12:40 vpx3-1 kernel: blkdev_write_iter+0xa8/0x130
Nov 23 14:12:40 vpx3-1 kernel: new_sync_write+0xe7/0x140
Nov 23 14:12:40 vpx3-1 kernel: ? __blkdev_get+0x4d0/0x4d0
Nov 23 14:12:40 vpx3-1 kernel: ? new_sync_write+0xe7/0x140
Nov 23 14:12:40 vpx3-1 kernel: __vfs_write+0x29/0x40
Nov 23 14:12:40 vpx3-1 kernel: vfs_write+0xb5/0x1a0
Nov 23 14:12:40 vpx3-1 kernel: SyS_pwrite64+0x95/0xb0
Nov 23 14:12:40 vpx3-1 kernel: do_syscall_64+0x73/0x130
Nov 23 14:12:40 vpx3-1 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Nov 23 14:12:40 vpx3-1 kernel: RIP: 0033:0x7f309f856963
Nov 23 14:12:40 vpx3-1 kernel: RSP: 002b:00007f30653c55d0 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
Nov 23 14:12:40 vpx3-1 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f309f856963
Nov 23 14:12:40 vpx3-1 kernel: RDX: 0000000000001000 RSI: 00007f3087be2000 RDI: 0000000000000011
Nov 23 14:12:40 vpx3-1 kernel: RBP: 00007f306c694080 R08: 0000000000000000 R09: 00007a395bc07916
Nov 23 14:12:40 vpx3-1 kernel: R10: 0000000009f0d000 R11: 0000000000000293 R12: 00007f3087be2000
Nov 23 14:12:40 vpx3-1 kernel: R13: 00007f30918bcfd8 R14: 00007f306dfc11d0 R15: 00007f3061c0e400
Nov 23 14:13:00 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:13:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:14:01 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:14:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:14:34 vpx3-1 pveproxy[5479]: worker exit
Nov 23 14:14:34 vpx3-1 pveproxy[1559]: worker 5479 finished
Nov 23 14:14:34 vpx3-1 pveproxy[1559]: starting 1 worker(s)
Nov 23 14:14:34 vpx3-1 pveproxy[1559]: worker 18255 started
Nov 23 14:14:40 vpx3-1 kernel: INFO: task kvm:17593 blocked for more than 120 seconds.
Nov 23 14:14:40 vpx3-1 kernel: Tainted: P O 4.15.18-9-pve #1
Nov 23 14:14:40 vpx3-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 14:14:40 vpx3-1 kernel: kvm D 0 17593 1 0x00000000
Nov 23 14:14:40 vpx3-1 kernel: Call Trace:
Nov 23 14:14:40 vpx3-1 kernel: __schedule+0x3e0/0x870
Nov 23 14:14:40 vpx3-1 kernel: schedule+0x36/0x80
Nov 23 14:14:40 vpx3-1 kernel: __drbd_make_request+0x27c/0x5b0 [drbd]
Nov 23 14:14:40 vpx3-1 kernel: ? wait_woken+0x80/0x80
Nov 23 14:14:40 vpx3-1 kernel: drbd_make_request+0x4f/0x80 [drbd]
Nov 23 14:14:40 vpx3-1 kernel: generic_make_request+0x123/0x2f0
Nov 23 14:14:40 vpx3-1 kernel: submit_bio+0x73/0x140
Nov 23 14:14:40 vpx3-1 kernel: ? submit_bio+0x73/0x140
Nov 23 14:14:40 vpx3-1 kernel: ? guard_bio_eod+0x2c/0xf0
Nov 23 14:14:40 vpx3-1 kernel: submit_bh_wbc+0x180/0x1b0
Nov 23 14:14:40 vpx3-1 kernel: __block_write_full_page+0x197/0x3f0
Nov 23 14:14:40 vpx3-1 kernel: block_write_full_page+0x104/0x120
Nov 23 14:14:40 vpx3-1 kernel: blkdev_writepage+0x18/0x20
Nov 23 14:14:40 vpx3-1 kernel: __writepage+0x1d/0x50
Nov 23 14:14:40 vpx3-1 kernel: write_cache_pages+0x22e/0x4b0
Nov 23 14:14:40 vpx3-1 kernel: ? __wb_calc_thresh+0x140/0x140
Nov 23 14:14:40 vpx3-1 kernel: ? futex_wait+0x1f5/0x260
Nov 23 14:14:40 vpx3-1 kernel: generic_writepages+0x61/0xa0
Nov 23 14:14:40 vpx3-1 kernel: blkdev_writepages+0x2f/0x40
Nov 23 14:14:40 vpx3-1 kernel: ? blkdev_writepages+0x2f/0x40
Nov 23 14:14:40 vpx3-1 kernel: do_writepages+0x1f/0x70
Nov 23 14:14:40 vpx3-1 kernel: __filemap_fdatawrite_range+0xc6/0x100
Nov 23 14:14:40 vpx3-1 kernel: file_write_and_wait_range+0x5a/0xb0
Nov 23 14:14:40 vpx3-1 kernel: blkdev_fsync+0x1b/0x50
Nov 23 14:14:40 vpx3-1 kernel: vfs_fsync_range+0x51/0xb0
Nov 23 14:14:40 vpx3-1 kernel: do_fsync+0x3d/0x70
Nov 23 14:14:40 vpx3-1 kernel: ? exit_to_usermode_loop+0xa5/0xd0
Nov 23 14:14:40 vpx3-1 kernel: SyS_fdatasync+0x13/0x20
Nov 23 14:14:40 vpx3-1 kernel: do_syscall_64+0x73/0x130
Nov 23 14:14:40 vpx3-1 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Nov 23 14:14:40 vpx3-1 kernel: RIP: 0033:0x7f309f58860d
Nov 23 14:14:40 vpx3-1 kernel: RSP: 002b:00007f305d3f55f0 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Nov 23 14:14:40 vpx3-1 kernel: RAX: ffffffffffffffda RBX: 00000000fffffffb RCX: 00007f309f58860d
Nov 23 14:14:40 vpx3-1 kernel: RDX: 00007f30918bcf70 RSI: 000055cb42a026e0 RDI: 0000000000000011
Nov 23 14:14:40 vpx3-1 kernel: RBP: 00007f306c6941c0 R08: 0000000000000000 R09: 00000000ffffffff
Nov 23 14:14:40 vpx3-1 kernel: R10: 00007f305d3f5620 R11: 0000000000000293 R12: 00007f30919c4bc0
Nov 23 14:14:40 vpx3-1 kernel: R13: 00007f30918bcfd8 R14: 00007f306dfc10f0 R15: 00007f308f45e400
Nov 23 14:14:40 vpx3-1 kernel: INFO: task kvm:17785 blocked for more than 120 seconds.
╭───────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ first ┊ vpx3-1 ┊ 7002 ┊ Unused ┊ UpToDate ┊
┊ first ┊ vpx3-2 ┊ 7002 ┊ Unused ┊ UpToDate ┊
┊ vm-100-disk-1 ┊ vpx3-1 ┊ 7001 ┊ Unused ┊ UpToDate ┊
┊ vm-100-disk-1 ┊ vpx3-2 ┊ 7001 ┊ Unused ┊ UpToDate ┊
┊ vm-100-disk-2 ┊ vpx3-1 ┊ 7003 ┊ Unused ┊ UpToDate ┊
┊ vm-100-disk-2 ┊ vpx3-2 ┊ 7003 ┊ Unused ┊ UpToDate ┊
╰───────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ SupportsSnapshots ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ drbdpool ┊ vpx3-1 ┊ LvmThinDriver ┊ vg_drbd/drbdthinpool ┊ 59.27 GiB ┊ 63 GiB ┊ true ┊
┊ drbdpool ┊ vpx3-2 ┊ LvmThinDriver ┊ vg_drbd/drbdthinpool ┊ 59.27 GiB ┊ 63 GiB ┊ true ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────╮
┊ ResourceName ┊ VolumeNr ┊ VolumeMinor ┊ Size ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ first ┊ 0 ┊ 1002 ┊ 10 MiB ┊ ok ┊
┊ vm-100-disk-1 ┊ 0 ┊ 1001 ┊ 2.01 GiB ┊ ok ┊
┊ vm-100-disk-2 ┊ 0 ┊ 1003 ┊ 2.01 GiB ┊ ok ┊
╰───────────────────────────────────────────────────────────╯
More information about the drbd-user
mailing list