[DRBD-user] Error when copying/downloading an image in opennebula datastore

Sun Oct 6 18:08:22 CEST 2019

Greetings DRBD users,

we have recently started using  drbd9 as a primary mean of storing opennebula user data.

We have followed every instruction in the plugin guide, but unfortunately we are facing some sort of an execution/permission issues. We have tried to give oneadmin full tmp and datastore permissions and tweaked datastore settings, but the problem is always the same, and has been reproduced at least 3 times with a clean installation of opennebula, and drbd9. 

It should also needs to be noted, that a similar case has already been posted recently https://lists.gt.net/drbd/users/30739. 

ERROR (taken form /var/log/oned.log)

Command execution failed (exit code: 1): /var/lib/one/remotes/datastore/linstor/cp PERTX0RSSVZFUl9BQ1RJT05fREFUQT48SU1BR0U+PElEPjIyPC9JRD48VUlEPjA8L1VJRD48R0lEPjA8L0dJRD48VU5BTUU+b25lYWRtaW48L1VOQU1FPjxHTkFNRT5vbmVhZG1pbjwvR05BTUU+PE5BTUU+RGViaWFuIDggLSBLVk08L05BTUU+PFBFUk1JU1NJT05TPjxPV05FUl9VPjE8L09XTkVSX1U+PE9XTkVSX00+MTwvT1dORVJfTT48T1dORVJfQT4wPC9PV05FUl9BPjxHUk9VUF9VPjA8L0dST1VQX1U+PEdST1VQX00+MDwvR1JPVVBfTT48R1JPVVBfQT4wPC9HUk9VUF9BPjxPVEhFUl9VPjA8L09USEVSX1U+PE9USEVSX00+MDwvT1RIRVJfTT48T1RIRVJfQT4wPC9PVEhFUl9BPjwvUEVSTUlTU0lPTlM+PFRZUEU+MDwvVFlQRT48RElTS19UWVBFPjI8L0RJU0tfVFlQRT48UEVSU0lTVEVOVD4wPC9QRVJTSVNURU5UPjxSRUdUSU1FPjE1NzAzNzU0NzA8L1JFR1RJTUU+PFNPVVJDRT48IVtDREFUQVtdXT48L1NPVVJDRT48UEFUSD48IVtDREFUQVtodHRwczovL21hcmtldHBsYWNlLm9wZW5uZWJ1bGEuc3lzdGVtcy8vYXBwbGlhbmNlL2U5MDJlOTFmLTRlODMtNDBhZi1iNDFlLWI4OTVhYmYwNDYxZC9kb3dubG9hZC8wXV0+PC9QQVRIPjxGU1RZUEU+PCFbQ0RBVEFbXV0+PC9GU1RZUEU+PFNJWkU+MjA0ODwvU0laRT48U1RBVEU+NDwvU1RBVEU+PFJVTk5JTkdfVk1TPjA8L1JVTk5JTkdfVk1TPjxDTE9OSU5HX09QUz4wPC9DTE9OSU5HX09QUz48Q0xPTklOR19JRD4tMTwvQ0xPTklOR19JRD48VEFSR0VUX1NOQVBTSE9UPi0xPC9UQVJHRVRfU05BUFNIT1Q+PERBVEFTVE9SRV9JRD4xMDU8L0RBVEFTVE9SRV9JRD48REFUQVNUT1JFPmxpbnN0b3JfYXV0b19wbGFjZTwvREFUQVNUT1JFPjxWTVM+PC9WTVM+PENMT05FUz48L0NMT05FUz48QVBQX0NMT05FUz48L0FQUF9DTE9ORVM+PFRFTVBMQVRFPjxERVZfUFJFRklYPjwhW0NEQVRBW3ZkXV0+PC9ERVZfUFJFRklYPjxEUklWRVI+PCFbQ0RBVEFbcWNvdzJdXT48L0RSSVZFUj48Rk9STUFUPjwhW0NEQVRBW3Fjb3cyXV0+PC9GT1JNQVQ+PEZST01fQVBQPjwhW0NEQVRBWzE4XV0+PC9GUk9NX0FQUD48RlJPTV9BUFBfTUQ1PjwhW0NEQVRBWzcxOTI3ZmQwOTUxNjA1OWJjZGI5ZTgzZjM2ZDc3OWI3XV0+PC9GUk9NX0FQUF9NRDU+PEZST01fQVBQX05BTUU+PCFbQ0RBVEFbRGViaWFuIDggLSBLVk1dXT48L0ZST01fQVBQX05BTUU+PC9URU1QTEFURT48U05BUFNIT1RTPjxBTExPV19PUlBIQU5TPjwhW0NEQVRBW05PXV0+PC9BTExPV19PUlBIQU5TPjxDVVJSRU5UX0JBU0U+PCFbQ0RBVEFbLTFdXT48L0NVUlJFTlRfQkFTRT48TkVYVF9TTkFQU0hPVD48IVtDREFUQVswXV0+PC9ORVhUX1NOQVBTSE9UPjwvU05BUFNIT1RTPjwvSU1BR0U+PERBVEFTVE9SRT48SUQ+MTA1PC9JRD48VUlEPjA8L1VJRD48R0lEPjA8L0dJRD48VU5BTUU+b25lYWRtaW48L1VOQU1FPjxHTkFNRT5vbmVhZG1pbjwvR05BTUU+PE5BTUU+bGluc3Rvcl9hdXRvX3BsYWNlPC9OQU1FPjxQRVJNSVNTSU9OUz48T1dORVJfVT4xPC9PV05FUl9VPjxPV05FUl9NPjE8L09XTkVSX00+PE9XTkVSX0E+MDwvT1dORVJfQT48R1JPVVBfVT4xPC9HUk9VUF9VPjxHUk9VUF9NPjA8L0dST1VQX00+PEdST1VQX0E+MDwvR1JPVVBfQT48T1RIRVJfVT4wPC9PVEhFUl9VPjxPVEhFUl9NPjA8L09USEVSX00+PE9USEVSX0E+MDwvT1RIRVJfQT48L1BFUk1JU1NJT05TPjxEU19NQUQ+PCFbQ0RBVEFbbGluc3Rvcl1dPjwvRFNfTUFEPjxUTV9NQUQ+PCFbQ0RBVEFbbGluc3Rvcl1dPjwvVE1fTUFEPjxCQVNFX1BBVEg+PCFbQ0RBVEFbL3Zhci9saWIvb25lLy9kYXRhc3RvcmVzLzEwNV1dPjwvQkFTRV9QQVRIPjxUWVBFPjA8L1RZUEU+PERJU0tfVFlQRT4yPC9ESVNLX1RZUEU+PFNUQVRFPjA8L1NUQVRFPjxDTFVTVEVSUz48SUQ+MDwvSUQ+PC9DTFVTVEVSUz48VE9UQUxfTUI+MzY0MzgwNDwvVE9UQUxfTUI+PEZSRUVfTUI+MzY0MzgwNDwvRlJFRV9NQj48VVNFRF9NQj4wPC9VU0VEX01CPjxJTUFHRVM+PC9JTUFHRVM+PFRFTVBMQVRFPjxBTExPV19PUlBIQU5TPjwhW0NEQVRBW3llc11dPjwvQUxMT1dfT1JQSEFOUz48QlJJREdFX0xJU1Q+PCFbQ0RBVEFbbnRzLWNsb3VkLW4xMDEgbnRzLWNsb3VkLTEwMCBudHMtY2xvdWQtbjEwMl1dPjwvQlJJREdFX0xJU1Q+PENMT05FX1RBUkdFVD48IVtDREFUQVtTRUxGXV0+PC9DTE9ORV9UQVJHRVQ+PENMT05FX1RBUkdFVF9TSEFSRUQ+PCFbQ0RBVEFbU0VMRl1dPjwvQ0xPTkVfVEFSR0VUX1NIQVJFRD48Q0xPTkVfVEFSR0VUX1NTSD48IVtDREFUQVtTRUxGXV0+PC9DTE9ORV9UQVJHRVRfU1NIPjxESVNLX1RZUEU+PCFbQ0RBVEFbQkxPQ0tdXT48L0RJU0tfVFlQRT48RElTS19UWVBFX1NIQVJFRD48IVtDREFUQVtCTE9DS11dPjwvRElTS19UWVBFX1NIQVJFRD48RElTS19UWVBFX1NTSD48IVtDREFUQVtCTE9DS11dPjwvRElTS19UWVBFX1NTSD48RFNfTUFEPjwhW0NEQVRBW2xpbnN0b3JdXT48L0RTX01BRD48TElOU1RPUl9BVVRPX1BMQUNFPjwhW0NEQVRBWzJdXT48L0xJTlNUT1JfQVVUT19QTEFDRT48TElOU1RPUl9DTE9ORV9NT0RFPjwhW0NEQVRBW3NuYXBzaG90XV0+PC9MSU5TVE9SX0NMT05FX01PREU+PExJTlNUT1JfU1RPUkFHRV9QT09MPjwhW0NEQVRBW2RhdGFzdG9yZS1zc2RdXT48L0xJTlNUT1JfU1RPUkFHRV9QT09MPjxMTl9UQVJHRVQ+PCFbQ0RBVEFbTk9ORV1dPjwvTE5fVEFSR0VUPjxMTl9UQVJHRVRfU0hBUkVEPjwhW0NEQVRBW05PTkVdXT48L0xOX1RBUkdFVF9TSEFSRUQ+PExOX1RBUkdFVF9TU0g+PCFbQ0RBVEFbTk9ORV1dPjwvTE5fVEFSR0VUX1NTSD48UkVTVFJJQ1RFRF9ESVJTPjwhW0NEQVRBWy9dXT48L1JFU1RSSUNURURfRElSUz48U0FGRV9ESVJTPjwhW0NEQVRBWy92YXIvdG1wXV0+PC9TQUZFX0RJUlM+PFRNX01BRD48IVtDREFUQVtsaW5zdG9yXV0+PC9UTV9NQUQ+PFRNX01BRF9TWVNURU0+PCFbQ0RBVEFbc3NoLHNoYXJlZF1dPjwvVE1fTUFEX1NZU1RFTT48VFlQRT48IVtDREFUQVtJTUFHRV9EU11dPjwvVFlQRT48L1RFTVBMQVRFPjwvREFUQVNUT1JFPjxNQVJLRVRQTEFDRT48SUQ+MDwvSUQ+PFVJRD4wPC9VSUQ+PEdJRD4wPC9HSUQ+PFVOQU1FPm9uZWFkbWluPC9VTkFNRT48R05BTUU+b25lYWRtaW48L0dOQU1FPjxOQU1FPk9wZW5OZWJ1bGEgUHVibGljPC9OQU1FPjxNQVJLRVRfTUFEPjwhW0NEQVRBW29uZV1dPjwvTUFSS0VUX01BRD48Wk9ORV9JRD48IVtDREFUQVswXV0+PC9aT05FX0lEPjxUT1RBTF9NQj4wPC9UT1RBTF9NQj48RlJFRV9NQj4wPC9GUkVFX01CPjxVU0VEX01CPjA8L1VTRURfTUI+PE1BUktFVFBMQUNFQVBQUz48SUQ+MDwvSUQ+PElEPjE8L0lEPjxJRD4yPC9JRD48SUQ+MzwvSUQ+PElEPjQ8L0lEPjxJRD41PC9JRD48SUQ+NjwvSUQ+PElEPjc8L0lEPjxJRD44PC9JRD48SUQ+OTwvSUQ+PElEPjEwPC9JRD48SUQ+MTE8L0lEPjxJRD4xMjwvSUQ+PElEPjEzPC9JRD48SUQ+MTQ8L0lEPjxJRD4xNTwvSUQ+PElEPjE2PC9JRD48SUQ+MTc8L0lEPjxJRD4xODwvSUQ+PElEPjE5PC9JRD48SUQ+MjA8L0lEPjxJRD4yMTwvSUQ+PElEPjIyPC9JRD48SUQ+MjM8L0lEPjxJRD4yNDwvSUQ+PElEPjI1PC9JRD48SUQ+MjY8L0lEPjxJRD4yNzwvSUQ+PElEPjI4PC9JRD48SUQ+Mjk8L0lEPjxJRD4zMDwvSUQ+PElEPjMxPC9JRD48SUQ+MzI8L0lEPjxJRD4zMzwvSUQ+PElEPjM0PC9JRD48SUQ+MzU8L0lEPjxJRD4zNjwvSUQ+PElEPjM3PC9JRD48L01BUktFVFBMQUNFQVBQUz48UEVSTUlTU0lPTlM+PE9XTkVSX1U+MTwvT1dORVJfVT48T1dORVJfTT4xPC9PV05FUl9NPjxPV05FUl9BPjE8L09XTkVSX0E+PEdST1VQX1U+MTwvR1JPVVBfVT48R1JPVVBfTT4wPC9HUk9VUF9NPjxHUk9VUF9BPjA8L0dST1VQX0E+PE9USEVSX1U+MTwvT1RIRVJfVT48T1RIRVJfTT4wPC9PVEhFUl9NPjxPVEhFUl9BPjA8L09USEVSX0E+PC9QRVJNSVNTSU9OUz48VEVNUExBVEU+PERFU0NSSVBUSU9OPjwhW0NEQVRBW09wZW5OZWJ1bGEgU3lzdGVtcyBNYXJrZXRQbGFjZV1dPjwvREVTQ1JJUFRJT04+PE1BUktFVF9NQUQ+PCFbQ0RBVEFbb25lXV0+PC9NQVJLRVRfTUFEPjwvVEVNUExBVEU+PC9NQVJLRVRQTEFDRT48L0RTX0RSSVZFUl9BQ1RJT05fREFUQT4= 22

Sun Oct  6 15:25:34 2019 [Z0][ImM][E]: Error copying image in the datastore

journalctl opennebula entry:

INFO Entering datastore cp 

Oct 06 15:24:30 nts-cloud-100 cp[16558]: INFO Deploying resource 'OpenNebula-Image-22'

Oct 06 15:24:32 nts-cloud-100 cp[16558]: INFO running shell command: bash -c source /var/lib/one/remotes//datastore/libfs.sh && set_up_datastore /var/lib/one//datastores/105 / /var/tmp

Oct 06 15:24:32 nts-cloud-100 cp[16558]: INFO running shell command: bash -c source /var/lib/one/remotes//datastore/libfs.sh && set_downloader_args "" "" "" "" https://marketplace.opennebula

Oct 06 15:24:32 nts-cloud-100 cp[16558]: INFO Downloading https://marketplace.opennebula.systems//appliance/e902e91f-4e83-40af-b41e-b895abf0461d/download/0 to the image repository

Oct 06 15:24:32 nts-cloud-100 cp[16558]: INFO running shell command: bash -c source /var/lib/one/remotes//datastore/libfs.sh && check_restricted https://marketplace.opennebula.systems//appli

Oct 06 15:24:32 nts-cloud-100 cp[16558]: INFO Copying local image https://marketplace.opennebula.systems//appliance/e902e91f-4e83-40af-b41e-b895abf0461d/download/0 to the image repository

Oct 06 15:24:33 nts-cloud-100 cp[16558]: INFO running shell command: bash -c /var/lib/one/remotes//datastore/downloader.sh   https://marketplace.opennebula.systems//appliance/e902e91f-4e83-4

Oct 06 15:24:50 nts-cloud-100 monitor[16860]: INFO Entering datastore monitor.

Oct 06 15:24:50 nts-cloud-100 monitor[16854]: INFO Entering datastore monitor.

Oct 06 15:24:50 nts-cloud-100 monitor[16877]: INFO Entering tm monitor.

Oct 06 15:24:50 nts-cloud-100 monitor[16860]: INFO Exiting datastore monitor[274/3643804/3643530].

Oct 06 15:24:50 nts-cloud-100 monitor[16854]: INFO Exiting datastore monitor[274/3643804/3643530].

Oct 06 15:24:51 nts-cloud-100 monitor[16877]: INFO Leaving tm monitor.

Oct 06 15:25:33 nts-cloud-100 cp[16558]: [334B blob data]

Oct 06 15:25:33 nts-cloud-100 cp[16558]: ERROR "Error copying data to OpenNebula-Image-22 on nts-cloud-100"

journalctl DRBD9 entry:

Oct 06 15:10:17 nts-cloud-100 Controller[1545]: 15:10:17.294 [grizzly-http-server-26] INFO  LINSTOR/Controller - SYSTEM - New volume definition with number '0' of resource definition 'OpenNe

Oct 06 15:24:30 nts-cloud-100 Controller[1545]: 15:24:30.825 [grizzly-http-server-27] INFO  LINSTOR/Controller - SYSTEM - New volume definition with number '0' of resource definition 'OpenNe

Oct 06 15:58:28 nts-cloud-100 Controller[1545]: 15:58:28.694 [grizzly-http-server-24] INFO  LINSTOR/Controller - SYSTEM - New volume definition with number '0' of resource definition 'OpenNe

Oct 06 15:59:43 nts-cloud-100 Controller[1545]: 15:59:43.348 [grizzly-http-server-7] INFO  LINSTOR/Controller - SYSTEM - New volume definition with number '0' of resource definition 'OpenNeb

The datastore configs in opennebula are following:

NAME="datastore-ssd-image"

TYPE="IMAGE_DS"

STORAGE_POOL="datastore-ssd"

AUTO_PLACE="2"

BRIDGE_LIST="n101 n100 n102"

DISK_TYPE="BLOCK"

DS_MAD="linstor_un"

TM_MAD="linstor_un"

NAME = linstor_auto_place233

TM_MAD = linstor

TYPE = SYSTEM_DS

LINSTOR_AUTO_PLACE = 2

LINSTOR_STORAGE_POOL = "datastore-ssd"

BRIDGE_LIST = "n101 n100 n102"

All the best,

Notespeak.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20191006/edfe9bc7/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oned.log
Type: application/octet-stream
Size: 125284 bytes
Desc: not available
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20191006/edfe9bc7/attachment-0001.obj>