[DRBD-user] [proxmox 5.4]Cannot create VM, cannot restore : timeout linstor sp l...

Mon Jan 27 10:19:31 CET 2020

  Hi all,

Running a 2 node proxmox cluster :

pveversion -v
proxmox-ve: 5.4-2 (running kernel: 4.15.18-24-pve)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-12

linstor-client                       1.0.10-1
linstor-common                       1.4.1-1
linstor-controller                   1.4.1-1
linstor-proxmox                      4.1.1-1
linstor-satellite                    1.4.1-1
python-linstor                       1.0.10-1

Last update - upgrade : 24/01/2020.

Storage : drbd on top of lvm.

After extending drbd storage size from 2Tb to 5Tb, we can not restore /
create any VM.
A timeout occurs :
TASK ERROR: unable to create VM 8001 - error with cfs lock
'storage-drbdstorage': Could not set allow-two-primaries on resource
definition vm-8001-disk-2, because: got lock timeout - aborting command at
/usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 285.
PVE::Storage::Custom::LINSTORPlugin::alloc_image("PVE::Storage::Custom::LINSTORPlugin",
"drbdstorage", HASH(0x56188be2bb70), 8001, "raw", undef, 1048576) called at
/usr/share/perl5/PVE/Storage.pm line 738 eval {...} called at
/usr/share/perl5/PVE/Storage.pm line 738 PVE::Storage::__ANON__() called at
/usr/share/perl5/PVE/Cluster.pm line 915 eval {...} called at
/usr/share/perl5/PVE/Cluster.pm line 883
PVE::Cluster::__ANON__("storage-drbdstorage", undef, CODE(0x56188bc35a68))
called at /usr/share/perl5/PVE/Cluster.pm line 954
PVE::Cluster::cfs_lock_storage("drbdstorage", undef, CODE(0x56188bc35a68))
called at /usr/share/perl5/PVE/Storage/Plugin.pm line 390
PVE::Storage::Plugin::cluster_lock_storage("PVE::Storage::Custom::LINSTORPlugin",
"drbdstorage", 1, undef, CODE(0x56188bc35a68)) called at
/usr/share/perl5/PVE/Storage.pm line 743
PVE::Storage::vdisk_alloc(HASH(0x56188b409290), "drbdstorage", 8001, "raw",
undef, 1048576) called at /usr/share/perl5/PVE/API2/Qemu.pm line 181
PVE::API2::Qemu::__ANON__("scsi0", HASH(0x56188bc359d8)) called at
/usr/share/perl5/PVE/QemuServer.pm line 3288
PVE::QemuServer::foreach_drive(HASH(0x56188b405ab8), CODE(0x56188bc35a50))
called at /usr/share/perl5/PVE/API2/Qemu.pm line 216 eval {...} called at
/usr/share/perl5/PVE/API2/Qemu.pm line 216
PVE::API2::Qemu::__ANON__(PVE::RPCEnvironment=HASH(0x56188bc37128),
"root\@pam", HASH(0x56188b405ab8), "x86_64", HASH(0x56188b409290), 8001,
undef, HASH(0x56188b405ab8), ...) called at
/usr/share/perl5/PVE/API2/Qemu.pm line 604 eval {...} called at
/usr/share/perl5/PVE/API2/Qemu.pm line 602 PVE::API2::Qemu::__ANON__()
called at /usr/share/perl5/PVE/Tools.pm line 203 eval {...} called at
/usr/share/perl5/PVE/Tools.pm line 203
PVE::Tools::lock_file_full("/var/lock/qemu-server/lock-8001.conf", 1, 0,
CODE(0x56188bc365e8)) called at /usr/share/perl5/PVE/Tools.pm line 212
PVE::Tools::lock_file("/var/lock/qemu-server/lock-8001.conf", 1,
CODE(0x56188bc365e8)) called at /usr/share/perl5/PVE/AbstractConfig.pm line
78 PVE::AbstractConfig::lock_config_full("PVE::QemuConfig", 8001, 1,
CODE(0x56188bc365e8)) called at /usr/share/perl5/PVE/API2/Qemu.pm line 636
PVE::API2::Qemu::__ANON__() called at /usr/share/perl5/PVE/API2/Qemu.pm
line 659 eval {...} called at /usr/share/perl5/PVE/API2/Qemu.pm line 659
PVE::API2::Qemu::__ANON__("UPID:px1-administration:00005D93:000AA66C:5E2B1A0E:qmcreate:8"...)
called at /usr/share/perl5/PVE/RESTEnvironment.pm line 610 eval {...}
called at /usr/share/perl5/PVE/RESTEnvironment.pm line 601
PVE::RESTEnvironment::fork_worker(PVE::RPCEnvironment=HASH(0x56188bc37128),
"qmcreate", 8001, "root\@pam", CODE(0x56188be420e0)) called at
/usr/share/perl5/PVE/API2/Qemu.pm line 671
PVE::API2::Qemu::__ANON__(HASH(0x56188b405ab8)) called at
/usr/share/perl5/PVE/RESTHandler.pm line 446
PVE::RESTHandler::handle("PVE::API2::Qemu", HASH(0x5618890d5be8),
HASH(0x56188b405ab8)) called at /usr/share/perl5/PVE/HTTPServer.pm line 167
eval {...} called at /usr/share/perl5/PVE/HTTPServer.pm line 133
PVE::HTTPServer::rest_handler(PVE::HTTPServer=HASH(0x56188bc367e0),
"10.249.15.146", "POST", "/nodes/px1-administration/qemu",
HASH(0x56188be46af8), HASH(0x56188bc791f8), "extjs") called at
/usr/share/perl5/PVE/APIServer/AnyEvent.pm line 712 eval {...} called at
/usr/share/perl5/PVE/APIServer/AnyEvent.pm line 686
PVE::APIServer::AnyEvent::handle_api2_request(PVE::HTTPServer=HASH(0x56188bc367e0),
HASH(0x56188be25c50), HASH(0x56188be46af8), "POST",
"/api2/extjs/nodes/px1-administration/qemu") called at
/usr/share/perl5/PVE/APIServer/AnyEvent.pm line 939 eval {...} called at
/usr/share/perl5/PVE/APIServer/AnyEvent.pm line 931
PVE::APIServer::AnyEvent::handle_request(PVE::HTTPServer=HASH(0x56188bc367e0),
HASH(0x56188be25c50), HASH(0x56188be46af8), "POST",
"/api2/extjs/nodes/px1-administration/qemu") called at
/usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1325
PVE::APIServer::AnyEvent::__ANON__(AnyEvent::Handle=HASH(0x56188be1a520),
"numa=0\x{26}vmid=8001\x{26}sockets=1\x{26}cores=1\x{26}net0=virtio%2Cbridge%3Dvmb"...)
called at /usr/lib/x86_64-linux-gnu/perl5/5.24/AnyEvent/Handle.pm line 1522
AnyEvent::Handle::__ANON__(AnyEvent::Handle=HASH(0x56188be1a520)) called at
/usr/lib/x86_64-linux-gnu/perl5/5.24/AnyEvent/Handle.pm line 1332
AnyEvent::Handle::_drain_rbuf(AnyEvent::Handle=HASH(0x56188be1a520)) called
at /usr/lib/x86_64-linux-gnu/perl5/5.24/AnyEvent/Handle.pm line 2032
AnyEvent::Handle::__ANON__(EV::IO=SCALAR(0x56188be19860), 1) called at
/usr/lib/x86_64-linux-gnu/perl5/5.24/AnyEvent/Impl/EV.pm line 88 eval {...}
called at /usr/lib/x86_64-linux-gnu/perl5/5.24/AnyEvent/Impl/EV.pm line 88
AnyEvent::CondVar::Base::_wait(AnyEvent::CondVar=HASH(0x56188b405278))
called at /usr/lib/x86_64-linux-gnu/perl5/5.24/AnyEvent.pm line 2026
AnyEvent::CondVar::Base::recv(AnyEvent::CondVar=HASH(0x56188b405278))
called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1612
PVE::APIServer::AnyEvent::run(PVE::HTTPServer=HASH(0x56188bc367e0)) called
at /usr/share/perl5/PVE/Service/pvedaemon.pm line 52
PVE::Service::pvedaemon::run(PVE::Service::pvedaemon=HASH(0x561885644508))
called at /usr/share/perl5/PVE/Daemon.pm line 171 eval {...} called at
/usr/share/perl5/PVE/Daemon.pm line 171
PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x561885644508)) called
at /usr/share/perl5/PVE/Daemon.pm line 391 eval {...} called at
/usr/share/perl5/PVE/Daemon.pm line 380
PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x561885644508), undef)
called at /usr/share/perl5/PVE/Daemon.pm line 552 eval {...} called at
/usr/share/perl5/PVE/Daemon.pm line 550
PVE::Daemon::start(PVE::Service::pvedaemon=HASH(0x561885644508), undef)
called at /usr/share/perl5/PVE/Daemon.pm line 661
PVE::Daemon::__ANON__(HASH(0x56188562a410)) called at
/usr/share/perl5/PVE/RESTHandler.pm line 446
PVE::RESTHandler::handle("PVE::Service::pvedaemon", HASH(0x561885644820),
HASH(0x56188562a410)) called at /usr/share/perl5/PVE/RESTHandler.pm line
855 eval {...} called at /usr/share/perl5/PVE/RESTHandler.pm line 838
PVE::RESTHandler::cli_handler("PVE::Service::pvedaemon", "pvedaemon start",
"start", ARRAY(0x5618858e94d0), ARRAY(0x561885644e98), undef, undef, undef)
called at /usr/share/perl5/PVE/CLIHandler.pm line 580
PVE::CLIHandler::__ANON__(ARRAY(0x56188562a638), CODE(0x5618858aa2d8),
undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 657
PVE::CLIHandler::run_cli_handler("PVE::Service::pvedaemon", "prepare",
CODE(0x5618858aa2d8)) called at /usr/bin/pvedaemon line 27

DRBD resources are correctly created on both sides. VMID.conf appears
briefly, then disappears. And VM is NOT created.

We noticed long response time when :
linstor sp pl

An strace give a long wait (20s) to :
sendto(3, "GET /v1/view/storage-pools HTTP/"..., 153, 0, NULL, 0) = 153
poll([{fd=3, events=POLLIN}], 1, 300000

Live migration of running VMs works.

Any idea?

  Thank you,

    Christophe.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20200127/b370003b/attachment.htm>