[DRBD-user] Error copying image in the datastore
Luke Camilleri
luke.camilleri at zylacomputing.com
Thu Jun 6 14:51:08 CEST 2019
Dear Rene, I have once again hit this issue. I was uploading an ISO file into a linstor image datastore and failed.
In OpenNebula the error was:
Thu Jun 6 12:45:07 2019 [Z0][ImM][I]: linstor.errors.LinstorError: Error: Could not autoplace resource OpenNebula-Image-14: {u'error_report_ids': [u'5CF7F56F-00000-000040'], u
'message': u"Registration of auto-placing resource: 'OpenNebula-Image-14' failed due to an unknown exception.", u'obj_refs': {u'RscDfn': u'OpenNebula-Image-14'}, u'details': u'
Auto-placing resource: OpenNebula-Image-14', u'ret_code': -4611686018407201818}
Thu Jun 6 12:45:07 2019 [Z0][ImM][E]: Error copying image in the datastore
On the linstor-controller the error was:
ERROR REPORT 5CF7F56F-00000-000040
============================================================
Application: LINBIT® LINSTOR
Module: Controller
Version: 0.9.9
Build ID: 64b60e05c91b50097963c60f88830ce504dd4fd7
Build time: 2019-05-23T07:18:37+00:00
Error time: 2019-06-06 12:45:07
Node: drbd-kvm-1
Peer: RestClient(192.168.122.2)
============================================================
Reported error:
===============
Category: RuntimeException
Class name: AccessToDeletedDataException
Class canonical name: com.linbit.linstor.AccessToDeletedDataException
Generated at: Method 'checkDeleted', Source file 'VolumeDefinitionData.java', Line #353
Error message: Access to deleted volume definition
Error context:
Registration of auto-placing resource: 'OpenNebula-Image-14' failed due to an unknown exception.
Asynchronous stage backtrace:
Auto-place resource including thin pools
Auto-place resource
Call backtrace:
Method Native Class:Line number
checkDeleted N com.linbit.linstor.VolumeDefinitionData:353
getVolumeSize N com.linbit.linstor.VolumeDefinitionData:206
getReservedCapacity N com.linbit.linstor.FreeSpaceMgr:200
getReservedCapacityPrivileged N com.linbit.linstor.core.apicallhandler.controller.FreeCapacityAutoPoolSelectorUtils:142
getFreeCapacityCurrentEstimationPrivileged N com.linbit.linstor.core.apicallhandler.controller.FreeCapacityAutoPoolSelectorUtils:105
isStorPoolUsable N com.linbit.linstor.core.apicallhandler.controller.FreeCapacityAutoPoolSelectorUtils:47
lambda$filterUsableNodes$9 N com.linbit.linstor.core.apicallhandler.controller.CtrlRscAutoPlaceApiCallHandler:383
accept N java.util.stream.ReferencePipeline$2$1:174
forEachRemaining N java.util.ArrayList$ArrayListSpliterator:1382
copyInto N java.util.stream.AbstractPipeline:482
wrapAndCopyInto N java.util.stream.AbstractPipeline:472
evaluateSequential N java.util.stream.ReduceOps$ReduceOp:708
evaluate N java.util.stream.AbstractPipeline:234
collect N java.util.stream.ReferencePipeline:499
filterUsableNodes N com.linbit.linstor.core.apicallhandler.controller.CtrlRscAutoPlaceApiCallHandler:391
lambda$findBestCandidate$8 N com.linbit.linstor.core.apicallhandler.controller.CtrlRscAutoPlaceApiCallHandler:361
lambda$toMap$58 N java.util.stream.Collectors:1321
accept N java.util.stream.ReduceOps$3ReducingSink:169
forEachRemaining N java.util.HashMap$EntrySpliterator:1699
copyInto N java.util.stream.AbstractPipeline:482
wrapAndCopyInto N java.util.stream.AbstractPipeline:472
evaluateSequential N java.util.stream.ReduceOps$ReduceOp:708
evaluate N java.util.stream.AbstractPipeline:234
collect N java.util.stream.ReferencePipeline:499
findBestCandidate N com.linbit.linstor.core.apicallhandler.controller.CtrlRscAutoPlaceApiCallHandler:359
autoPlaceThinInTransaction N com.linbit.linstor.core.apicallhandler.controller.CtrlRscAutoPlaceApiCallHandler:318
lambda$null$5 N com.linbit.linstor.core.apicallhandler.controller.CtrlRscAutoPlaceApiCallHandler:292
doInScope N com.linbit.linstor.core.apicallhandler.ScopeRunner:141
lambda$null$0 N com.linbit.linstor.core.apicallhandler.ScopeRunner:71
call N reactor.core.publisher.MonoCallable:91
trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:126
subscribe N reactor.core.publisher.MonoFlatMapMany:46
subscribe N reactor.core.publisher.Flux:7734
onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:184
request N reactor.core.publisher.Operators$ScalarSubscription:2041
onSubscribe N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:131
subscribe N reactor.core.publisher.MonoCurrentContext:35
subscribe N reactor.core.publisher.MonoFlatMapMany:49
subscribe N reactor.core.publisher.FluxOnAssembly:164
subscribe N reactor.core.publisher.Flux:7734
onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:184
onNext N reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber:121
onNext N reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber:121
complete N reactor.core.publisher.Operators$MonoSubscriber:1476
onComplete N reactor.core.publisher.MonoStreamCollector$StreamCollectorSubscriber:163
onComplete N reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber:383
onComplete N reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner:248
checkTerminated N reactor.core.publisher.FluxFlatMap$FlatMapMain:794
drainLoop N reactor.core.publisher.FluxFlatMap$FlatMapMain:560
innerComplete N reactor.core.publisher.FluxFlatMap$FlatMapMain:863
onComplete N reactor.core.publisher.FluxFlatMap$FlatMapInner:957
onComplete N reactor.core.publisher.FluxMap$MapSubscriber:136
onComplete N reactor.core.publisher.Operators$MultiSubscriptionSubscriber:1713
onComplete N reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber:78
complete N reactor.core.publisher.FluxCreate$BaseSink:424
drain N reactor.core.publisher.FluxCreate$BufferAsyncSink:753
complete N reactor.core.publisher.FluxCreate$BufferAsyncSink:707
drainLoop N reactor.core.publisher.FluxCreate$SerializedSink:227
drain N reactor.core.publisher.FluxCreate$SerializedSink:197
complete N reactor.core.publisher.FluxCreate$SerializedSink:192
apiCallComplete N com.linbit.linstor.netcom.TcpConnectorPeer:450
handleComplete N com.linbit.linstor.proto.CommonMessageProcessor:361
handleDataMessage N com.linbit.linstor.proto.CommonMessageProcessor:285
doProcessInOrderMessage N com.linbit.linstor.proto.CommonMessageProcessor:233
lambda$doProcessMessage$3 N com.linbit.linstor.proto.CommonMessageProcessor:218
subscribe N reactor.core.publisher.FluxDefer:46
subscribe N reactor.core.publisher.Flux:7734
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:389
drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:395
drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:638
onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:242
drainFused N reactor.core.publisher.UnicastProcessor:277
drain N reactor.core.publisher.UnicastProcessor:310
onNext N reactor.core.publisher.UnicastProcessor:386
next N reactor.core.publisher.FluxCreate$IgnoreSink:593
next N reactor.core.publisher.FluxCreate$SerializedSink:151
processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:373
doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:216
lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:162
onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:177
runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:398
run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:484
call N reactor.core.scheduler.WorkerTask:84
call N reactor.core.scheduler.WorkerTask:37
run N java.util.concurrent.FutureTask:266
access$201 N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:180
run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:293
runWorker N java.util.concurrent.ThreadPoolExecutor:1149
run N java.util.concurrent.ThreadPoolExecutor$Worker:624
run N java.lang.Thread:748
END OF ERROR REPORT.
From: drbd-user-bounces at lists.linbit.com [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Luke Camilleri
Sent: 31 May 2019 17:22
To: Rene Peinthor <rene.peinthor at linbit.com>
Cc: drbd-user at lists.linbit.com
Subject: Re: [DRBD-user] Error copying image in the datastore
I have rebuilt the setup and do not have any issues with permissions.
I believe that the issue was due to the fact that I had 2 NIC’s in each node and 2 hostnames with different IP addresses in the hosts files.
I must have mixed the proper usage of the hostnames where the services were running
Thanks again Rene for your assistance
From: Luke Camilleri
Sent: 31 May 2019 14:18
To: 'Rene Peinthor' <rene.peinthor at linbit.com<mailto:rene.peinthor at linbit.com>>
Cc: 'drbd-user at lists.linbit.com' <drbd-user at lists.linbit.com<mailto:drbd-user at lists.linbit.com>>
Subject: RE: [DRBD-user] Error copying image in the datastore
This is from /var/log/messages of the combined node:
May 31 12:01:46 drbd-kvm-1 Satellite: 12:01:46.242 [DeviceManager] DEBUG LINSTOR/Satellite - Executing command: drbdadm -vvv --max-peers 7 -- --force create-md OpenNebula-Image-24/0
May 31 12:01:46 drbd-kvm-1 Satellite: 12:01:46.244 [Thread-3736] DEBUG LINSTOR/Satellite - 'OpenNebula-Image-24/0' not defined in your config (for this host).
May 31 12:01:46 drbd-kvm-1 Satellite: 12:01:46.244 [DeviceManager] TRACE LINSTOR/Satellite - External command finished in 1ms: drbdadm -vvv --max-peers 7 -- --force create-md OpenNebula-Image-24/0
May 31 12:01:46 drbd-kvm-1 Satellite: 12:01:46.246 [DeviceManager] ERROR LINSTOR/Satellite - Failed to create meta-data for DRBD volume OpenNebula-Image-24/0 [Report number 5CF0050A-049A1-000024]
Regards
From: Luke Camilleri
Sent: 31 May 2019 13:33
To: 'Rene Peinthor' <rene.peinthor at linbit.com<mailto:rene.peinthor at linbit.com>>
Cc: drbd-user at lists.linbit.com<mailto:drbd-user at lists.linbit.com>
Subject: RE: [DRBD-user] Error copying image in the datastore
I believe you meant ‘journalctl -u opennebula’ ☺ which in fact shows more info
May 31 12:28:48 frontend cp[12048]: INFO running shell command: bash -c source /var/lib/one/remotes//datastore/libfs.sh && set_up_datastore /var/lib/one//datastores/101 / /var/tmp
May 31 12:28:48 frontend cp[12048]: INFO running shell command: bash -c source /var/lib/one/remotes//datastore/libfs.sh && set_downloader_args "" "" "" "" /var/tmp/241172480-vyos-118-amd64iso -
May 31 12:28:48 frontend cp[12048]: INFO running shell command: bash -c /var/lib/one/remotes//datastore/downloader.sh /var/tmp/241172480-vyos-118-amd64iso - | ssh drbd-1 dd of=/dev/drbd1003 bs=2M
May 31 12:28:49 frontend cp[12048]: command ['bash', '-c', u'/var/lib/one/remotes//datastore/downloader.sh /var/tmp/241172480-vyos-118-amd64iso - | ssh drbd-1 dd of=/dev/drbd1003 bs=2M'] failed: dd: failed to open ‘/dev/drbd1003’: Permission denied
cat: write error: Broken pipe
tee: standard output: Broken pipe
tee: write error
Error copying
May 31 12:28:49 frontend cp[12048]: "Error copying data to OpenNebula-Image-21 on drbd-1"
Am I right in assuming that the permission denied error seems to be from the front-end to the drbd node?
Thanks in advance
From: Rene Peinthor [mailto:rene.peinthor at linbit.com]
Sent: 31 May 2019 13:05
To: Luke Camilleri <luke.camilleri at zylacomputing.com<mailto:luke.camilleri at zylacomputing.com>>
Cc: drbd-user at lists.linbit.com<mailto:drbd-user at lists.linbit.com>
Subject: Re: [DRBD-user] Error copying image in the datastore
Not much here to see a reason, please also check `systemctl -u opennebula`
On Fri, May 31, 2019 at 12:47 PM Luke Camilleri <luke.camilleri at zylacomputing.com<mailto:luke.camilleri at zylacomputing.com>> wrote:
I am receiving the below when uploading or downloading images to the image datastore. If I create a datablock in the same image datastore it completes successfully. Is this a permission issue or am I missing some configuration?
ERROR
Fri May 31 10:28:49 2019 : Error copying image in the datastore
This happens after the download has completed and in oned.log I can see the below entries:
Fri May 31 10:28:47 2019 [Z0][ImM][I]: Copying /var/tmp/241172480-vyos-118-amd64iso to repository for image 21
Fri May 31 10:28:47 2019 [Z0][ReM][D]: Req:2800 UID:0 one.image.allocate result SUCCESS, 21
Fri May 31 10:28:47 2019 [Z0][ReM][D]: Req:6432 UID:0 IP:127.0.0.1 one.image.info<http://one.image.info> invoked , 21
Fri May 31 10:28:47 2019 [Z0][ReM][D]: Req:6432 UID:0 one.image.info<http://one.image.info> result SUCCESS, "<IMAGE><ID>21</ID><U..."
Fri May 31 10:28:49 2019 [Z0][ImM][I]: Command execution failed (exit code: 1): /var/lib/one/remotes/datastore/linstor/cp PERTX0RSSVZFUl9BQ1RJT05fREFUQT48SU1BR0U+PElEPjIxPC9JRD4
8VUlEPjA8L1VJRD48R0lEPjA8L0dJRD48VU5BTUU+b25lYWRtaW48L1VOQU1FPjxHTkFNRT5vbmVhZG1pbjwvR05BTUU+PE5BTUU+dnlvcy1jZHJvbTwvTkFNRT48UEVSTUlTU0lPTlM+PE9XTkVSX1U+MTwvT1dORVJfVT48T1dORVJf
TT4xPC9PV05FUl9NPjxPV05FUl9BPjA8L09XTkVSX0E+PEdST1VQX1U+MDwvR1JPVVBfVT48R1JPVVBfTT4wPC9HUk9VUF9NPjxHUk9VUF9BPjA8L0dST1VQX0E+PE9USEVSX1U+MDwvT1RIRVJfVT48T1RIRVJfTT4wPC9PVEhFUl9NP
jxPVEhFUl9BPjA8L09USEVSX0E+PC9QRVJNSVNTSU9OUz48VFlQRT4xPC9UWVBFPjxESVNLX1RZUEU+MjwvRElTS19UWVBFPjxQRVJTSVNURU5UPjA8L1BFUlNJU1RFTlQ+PFJFR1RJTUU+MTU1OTI5ODUyNzwvUkVHVElNRT48U09VUk
NFPjwhW0NEQVRBW11dPjwvU09VUkNFPjxQQVRIPjwhW0NEQVRBWy92YXIvdG1wLzI0MTE3MjQ4MC12eW9zLTExOC1hbWQ2NGlzb11dPjwvUEFUSD48RlNUWVBFPjwhW0NEQVRBW11dPjwvRlNUWVBFPjxTSVpFPjIzMDwvU0laRT48U1R
BVEU+NDwvU1RBVEU+PFJVTk5JTkdfVk1TPjA8L1JVTk5JTkdfVk1TPjxDTE9OSU5HX09QUz4wPC9DTE9OSU5HX09QUz48Q0xPTklOR19JRD4tMTwvQ0xPTklOR19JRD48VEFSR0VUX1NOQVBTSE9UPi0xPC9UQVJHRVRfU05BUFNIT1Q+
PERBVEFTVE9SRV9JRD4xMDE8L0RBVEFTVE9SRV9JRD48REFUQVNUT1JFPmxpbnN0b3JfaW1hZ2VfZHM8L0RBVEFTVE9SRT48Vk1TPjwvVk1TPjxDTE9ORVM+PC9DTE9ORVM+PEFQUF9DTE9ORVM+PC9BUFBfQ0xPTkVTPjxURU1QTEFUR
T48REVWX1BSRUZJWD48IVtDREFUQVtoZF1dPjwvREVWX1BSRUZJWD48RFJJVkVSPjwhW0NEQVRBW3Jhd11dPjwvRFJJVkVSPjwvVEVNUExBVEU+PFNOQVBTSE9UUz48QUxMT1dfT1JQSEFOUz48IVtDREFUQVtOT11dPjwvQUxMT1dfT1
JQSEFOUz48Q1VSUkVOVF9CQVNFPjwhW0NEQVRBWy0xXV0+PC9DVVJSRU5UX0JBU0U+PE5FWFRfU05BUFNIT1Q+PCFbQ0RBVEFbMF1dPjwvTkVYVF9TTkFQU0hPVD48L1NOQVBTSE9UUz48L0lNQUdFPjxEQVRBU1RPUkU+PElEPjEwMTw
vSUQ+PFVJRD4wPC9VSUQ+PEdJRD4wPC9HSUQ+PFVOQU1FPm9uZWFkbWluPC9VTkFNRT48R05BTUU+b25lYWRtaW48L0dOQU1FPjxOQU1FPmxpbnN0b3JfaW1hZ2VfZHM8L05BTUU+PFBFUk1JU1NJT05TPjxPV05FUl9VPjE8L09XTkVS
X1U+PE9XTkVSX00+MTwvT1dORVJfTT48T1dORVJfQT4wPC9PV05FUl9BPjxHUk9VUF9VPjE8L0dST1VQX1U+PEdST1VQX00+MDwvR1JPVVBfTT48R1JPVVBfQT4wPC9HUk9VUF9BPjxPVEhFUl9VPjA8L09USEVSX1U+PE9USEVSX00+M
DwvT1RIRVJfTT48T1RIRVJfQT4wPC9PVEhFUl9BPjwvUEVSTUlTU0lPTlM+PERTX01BRD48IVtDREFUQVtsaW5zdG9yXV0+PC9EU19NQUQ+PFRNX01BRD48IVtDREFUQVtsaW5zdG9yXV0+PC9UTV9NQUQ+PEJBU0VfUEFUSD48IVtDRE
FUQVsvdmFyL2xpYi9vbmUvL2RhdGFzdG9yZXMvMTAxXV0+PC9CQVNFX1BBVEg+PFRZUEU+MDwvVFlQRT48RElTS19UWVBFPjI8L0RJU0tfVFlQRT48U1RBVEU+MDwvU1RBVEU+PENMVVNURVJTPjxJRD4wPC9JRD48L0NMVVNURVJTPjx
UT1RBTF9NQj4xMzgyMzE8L1RPVEFMX01CPjxGUkVFX01CPjEzODIxNjwvRlJFRV9NQj48VVNFRF9NQj4xNTwvVVNFRF9NQj48SU1BR0VTPjxJRD4xOTwvSUQ+PElEPjIwPC9JRD48L0lNQUdFUz48VEVNUExBVEU+PEFMTE9XX09SUEhB
TlM+PCFbQ0RBVEFbeWVzXV0+PC9BTExPV19PUlBIQU5TPjxCUklER0VfTElTVD48IVtDREFUQVtrdm0tMSBrdm0tMiBrdm0tM11dPjwvQlJJREdFX0xJU1Q+PENMT05FX1RBUkdFVD48IVtDREFUQVtTRUxGXV0+PC9DTE9ORV9UQVJHR
VQ+PERJU0tfVFlQRT48IVtDREFUQVtCTE9DS11dPjwvRElTS19UWVBFPjxEU19NQUQ+PCFbQ0RBVEFbbGluc3Rvcl1dPjwvRFNfTUFEPjxMSU5TVE9SX0FVVE9fUExBQ0U+PCFbQ0RBVEFbM11dPjwvTElOU1RPUl9BVVRPX1BMQUNFPj
xMSU5TVE9SX0NMT05FX01PREU+PCFbQ0RBVEFbY29weV1dPjwvTElOU1RPUl9DTE9ORV9NT0RFPjxMSU5TVE9SX0NPTlRST0xMRVJTPjwhW0NEQVRBWzE3Mi4xNy4wLjJdXT48L0xJTlNUT1JfQ09OVFJPTExFUlM+PExJTlNUT1JfU1R
PUkFHRV9QT09MPjwhW0NEQVRBW3RoaW4tbHZtLTFdXT48L0xJTlNUT1JfU1RPUkFHRV9QT09MPjxMTl9UQVJHRVQ+PCFbQ0RBVEFbTk9ORV1dPjwvTE5fVEFSR0VUPjxSRVNUUklDVEVEX0RJUlM+PCFbQ0RBVEFbL11dPjwvUkVTVFJJ
Q1RFRF9ESVJTPjxTQUZFX0RJUlM+PCFbQ0RBVEFbL3Zhci90bXBdXT48L1NBRkVfRElSUz48VE1fTUFEPjwhW0NEQVRBW2xpbnN0b3JdXT48L1RNX01BRD48VFlQRT48IVtDREFUQVtJTUFHRV9EU11dPjwvVFlQRT48L1RFTVBMQVRFP
jwvREFUQVNUT1JFPjwvRFNfRFJJVkVSX0FDVElPTl9EQVRBPg== 21
Fri May 31 10:28:49 2019 [Z0][ImM][E]: Error copying image in the datastore
_______________________________________________
drbd-user mailing list
drbd-user at lists.linbit.com<mailto:drbd-user at lists.linbit.com>
http://lists.linbit.com/mailman/listinfo/drbd-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20190606/10d554b1/attachment-0001.htm>
More information about the drbd-user
mailing list