<html><body><div style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000"><div><div>I now use:<br></div><div><span style="font-family: monospace;" data-mce-style="font-family: monospace;"><span style="color: #000000; background-color: #ffffff;" data-mce-style="color: #000000; background-color: #ffffff;"># linstor controller version </span><br>linstor controller 1.12.0; GIT-hash: 8e15f3ceaa73a9217ddd644221bb8952403f7d84<br></span></div><div><br></div><div>on 3 nodes. But my problem started before upgrade from 1.11.1-1 - see att1.txt . <br></div><div><br></div><div>Now I see that during taking snapshot linstor-satellite on another node was restarted - see att2.txt<br></div><div><br></div><div>I was able to remove all snapshots (with: linstor s d) <span class="Object" role="link" id="OBJ_PREFIX_DWT5134_com_zimbra_date"><span class="Object" role="link" id="OBJ_PREFIX_DWT5151_com_zimbra_date">so</span></span> now list of snapshots returned now by linstor s l is empty. Those snapshots was taken by vzdump utility therefore they are taken and removed without creating any new resource.<br></div><div><br></div><div>I would be grateful for your help in fixing our db. (separate mail in delivery <span style="height: 18; width: 18; padding: 9px 18px 9px 0; background: url('https://zimbra.cirrus.pl/service/zimlet/com_zimbra_ymemoticons/img/1.gif') no-repeat 0 50%;" title=":) - szczęśliwy" data-mce-style="height: 18; width: 18; padding: 9px 18px 9px 0; background: url('https://zimbra.cirrus.pl/service/zimlet/com_zimbra_ymemoticons/img/1.gif') no-repeat 0 50%;"><span style="visibility: hidden;" data-mce-style="visibility: hidden;"></span></span> ).<br></div><div><br></div><div>BR,<br></div><div>Michal Szamocki<br></div><div>Cirrus</div></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>Od: </b>"Gábor Hernádi" <gabor.hernadi@linbit.com><br><b>Do: </b>"drbd-user" <drbd-user@lists.linbit.com><br><b>Wysłane: </b>środa, 28 kwietnia, 2021 13:45:00<br><b>Temat: </b>Re: [DRBD-user] Linstor hangs on deleting snapshots<br></blockquote></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div>Hello,</div><br><div>can you please give us more details? for example the version of the linstor controller</div><div> linstor controller version</div><br><div>also what exactly happened before you tried to delete ... what exactly? You are talking about snapshots, but you are showing us a list of resources and resource-definitions, not snapshots and snapshot-definitions. <br></div><div>Please describe what happened since you created the origin resource. Did you create a snapshot? And afterwards restored it in a new *resource* called "snap_vm-107-disk-1_vzdump"? <br></div><div>Did something else happen which seems unrelated (restart of controller, other resources failing, or other resources were deleted or such...)</div><br><div>My goal would be here to reproduce this issue, afterwards I am quite sure we can figure out what happens and come up with a proper fix.</div><br><div>For the database, feel free to send me a direct email (not via mailing list) with the database file so I can fix it for you. <br></div><div>Or if you want to try it yourself - make a BACKUP first of the database, just in case. Afterwards your goal should be to let Linstor delete the resource as there are quite a few tables that need to be cleaned up. As the exception states an entry in LDV (LAYER_DRBD_VOLUME) has still a foreign key to LRI (LAYER_RESOURCE_IDS), I'd look for entries in LDV and see if there are orphaned or duplicates (same "target resource" but multiple IDs per KIND) . After double checking that, you can try to delete the orphaned entries and see if Linstor manages to cleanly remove the rest of the resource. <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 28, 2021 at 12:01 PM Michał Szamocki <<a href="mailto:mszamocki@cirrus.pl" target="_blank" rel="nofollow noopener noreferrer">mszamocki@cirrus.pl</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex">Hello,<br>
<br>
my linstor cluster failed to delete snapshots and now I have:<br>
# linstor rd l | grep DELETING<br>
| snap_vm-107-disk-1_vzdump | 7021 | DfltRscGrp | DELETING |<br>
| snap_vm-108-disk-1_vzdump | 7019 | DfltRscGrp | DELETING |<br>
<br>
# linstor r l | grep DELETING<br>
| snap_vm-107-disk-1_vzdump | debra | 7021 | | Ok | DELETING | 2021-04-28 07:09:35 |<br>
| snap_vm-107-disk-1_vzdump | elsa | 7021 | | Ok | DELETING | 2021-04-28 07:09:36 |<br>
| snap_vm-108-disk-1_vzdump | debra | 7019 | | Ok | DELETING | 2021-04-28 07:07:23 |<br>
| snap_vm-108-disk-1_vzdump | elsa | 7019 | | Ok | DELETING | 2021-04-28 07:07:23 |<br>
<br>
<br>
Any operation fails error similar to this:<br>
Caused by:<br>
==========<br>
<br>
Category: Exception<br>
Class name: JdbcSQLException<br>
Class canonical name: org.h2.jdbc.JdbcSQLException<br>
Generated at: Method 'getJdbcSQLException', Source file 'DbException.java', Line #357<br>
<br>
Error message: Naruszenie ograniczenia Klucza Głównego lub Indeksu Unikalnego: "FK_LDV_LRI_INDEX_C ON LINSTOR.LAYER_RESOURCE_IDS(LAYER_RESOURCE_ID) VALUES (21, 116935)"<br>
Unique index or primary key violation: "FK_LDV_LRI_INDEX_C ON LINSTOR.LAYER_RESOURCE_IDS(LAYER_RESOURCE_ID) VALUES (21, 116935)"; SQL statement:<br>
INSERT INTO LAYER_RESOURCE_IDS ( LAYER_RESOURCE_ID, NODE_NAME, RESOURCE_NAME, SNAPSHOT_NAME, LAYER_RESOURCE_PARENT_ID, LAYER_RESOURCE_KIND, LAYER_RESOURCE_SUFFIX, LAYER_RESOURCE_SUSPENDED ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ? ) [23505-197]<br>
<br>
lvs and drbdadm status don't show any information about those snapshot. <br>
<br>
How can I safetly remove information about those snapshots from linstor-controller database?<br>
<br>
BR,<br>
Michal Szamocki<br>
Cirrus<br>
_______________________________________________<br>
Star us on GITHUB: <a href="https://github.com/LINBIT" rel="noreferrer nofollow noopener noreferrer" target="_blank">https://github.com/LINBIT</a><br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank" rel="nofollow noopener noreferrer">drbd-user@lists.linbit.com</a><br>
<a href="https://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer nofollow noopener noreferrer" target="_blank">https://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div>Best regards,<br></div>Gabor Hernadi<br></div></div></div></div>
<br>_______________________________________________<br>Star us on GITHUB: https://github.com/LINBIT<br>drbd-user mailing list<br>drbd-user@lists.linbit.com<br>https://lists.linbit.com/mailman/listinfo/drbd-user<br></blockquote></div></div></body></html>