[DRBD-user] Linstor hangs on deleting snapshots

[Michał Szamocki]

I now use: 
# linstor controller version 
linstor controller 1.12.0; GIT-hash: 8e15f3ceaa73a9217ddd644221bb8952403f7d84 

on 3 nodes. But my problem started before upgrade from 1.11.1-1 - see att1.txt . 

Now I see that during taking snapshot linstor-satellite on another node was restarted - see att2.txt 

I was able to remove all snapshots (with: linstor s d) so now list of snapshots returned now by linstor s l is empty. Those snapshots was taken by vzdump utility therefore they are taken and removed without creating any new resource. 

I would be grateful for your help in fixing our db. (separate mail in delivery ). 

BR, 
Michal Szamocki 
Cirrus 

> Od: "Gábor Hernádi" <gabor.hernadi at linbit.com>
> Do: "drbd-user" <drbd-user at lists.linbit.com>
> Wysłane: środa, 28 kwietnia, 2021 13:45:00
> Temat: Re: [DRBD-user] Linstor hangs on deleting snapshots

> Hello,

> can you please give us more details? for example the version of the linstor
> controller
> linstor controller version

> also what exactly happened before you tried to delete ... what exactly? You are
> talking about snapshots, but you are showing us a list of resources and
> resource-definitions, not snapshots and snapshot-definitions.
> Please describe what happened since you created the origin resource. Did you
> create a snapshot? And afterwards restored it in a new *resource* called
> "snap_vm-107-disk-1_vzdump"?
> Did something else happen which seems unrelated (restart of controller, other
> resources failing, or other resources were deleted or such...)

> My goal would be here to reproduce this issue, afterwards I am quite sure we can
> figure out what happens and come up with a proper fix.

> For the database, feel free to send me a direct email (not via mailing list)
> with the database file so I can fix it for you.
> Or if you want to try it yourself - make a BACKUP first of the database, just in
> case. Afterwards your goal should be to let Linstor delete the resource as
> there are quite a few tables that need to be cleaned up. As the exception
> states an entry in LDV (LAYER_DRBD_VOLUME) has still a foreign key to LRI
> (LAYER_RESOURCE_IDS), I'd look for entries in LDV and see if there are orphaned
> or duplicates (same "target resource" but multiple IDs per KIND) . After double
> checking that, you can try to delete the orphaned entries and see if Linstor
> manages to cleanly remove the rest of the resource.

> On Wed, Apr 28, 2021 at 12:01 PM Michał Szamocki < [ mailto:mszamocki at cirrus.pl
> | mszamocki at cirrus.pl ] > wrote:

>> Hello,

>> my linstor cluster failed to delete snapshots and now I have:
>> # linstor rd l | grep DELETING
>> | snap_vm-107-disk-1_vzdump | 7021 | DfltRscGrp | DELETING |
>> | snap_vm-108-disk-1_vzdump | 7019 | DfltRscGrp | DELETING |

>> # linstor r l | grep DELETING
>>| snap_vm-107-disk-1_vzdump | debra | 7021 | | Ok | DELETING | 2021-04-28 07:09:35
>> | |
>>| snap_vm-107-disk-1_vzdump | elsa | 7021 | | Ok | DELETING | 2021-04-28 07:09:36
>> | |
>>| snap_vm-108-disk-1_vzdump | debra | 7019 | | Ok | DELETING | 2021-04-28 07:07:23
>> | |
>>| snap_vm-108-disk-1_vzdump | elsa | 7019 | | Ok | DELETING | 2021-04-28 07:07:23
>> | |

>> Any operation fails error similar to this:
>> Caused by:
>> ==========

>> Category: Exception
>> Class name: JdbcSQLException
>> Class canonical name: org.h2.jdbc.JdbcSQLException
>> Generated at: Method 'getJdbcSQLException', Source file 'DbException.java', Line
>> #357

>> Error message: Naruszenie ograniczenia Klucza Głównego lub Indeksu Unikalnego:
>> "FK_LDV_LRI_INDEX_C ON LINSTOR.LAYER_RESOURCE_IDS(LAYER_RESOURCE_ID) VALUES
>> (21, 116935)"
>> Unique index or primary key violation: "FK_LDV_LRI_INDEX_C ON
>> LINSTOR.LAYER_RESOURCE_IDS(LAYER_RESOURCE_ID) VALUES (21, 116935)"; SQL
>> statement:
>> INSERT INTO LAYER_RESOURCE_IDS ( LAYER_RESOURCE_ID, NODE_NAME, RESOURCE_NAME,
>> SNAPSHOT_NAME, LAYER_RESOURCE_PARENT_ID, LAYER_RESOURCE_KIND,
>> LAYER_RESOURCE_SUFFIX, LAYER_RESOURCE_SUSPENDED ) VALUES ( ?, ?, ?, ?, ?, ?, ?,
>> ? ) [23505-197]

>> lvs and drbdadm status don't show any information about those snapshot.

>> How can I safetly remove information about those snapshots from
>> linstor-controller database?

>> BR,
>> Michal Szamocki
>> Cirrus
>> _______________________________________________
>> Star us on GITHUB: [ https://github.com/LINBIT | https://github.com/LINBIT ]
>> drbd-user mailing list
>> [ mailto:drbd-user at lists.linbit.com | drbd-user at lists.linbit.com ]
>> [ https://lists.linbit.com/mailman/listinfo/drbd-user |
>> https://lists.linbit.com/mailman/listinfo/drbd-user ]

> --
> Best regards,
> Gabor Hernadi

> _______________________________________________
> Star us on GITHUB: https://github.com/LINBIT
> drbd-user mailing list
> drbd-user at lists.linbit.com
> https://lists.linbit.com/mailman/listinfo/drbd-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20210428/ff519714/attachment-0001.htm>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: att2.txt
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20210428/ff519714/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: att1.txt
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20210428/ff519714/attachment-0003.txt>