<div dir="ltr">hello everyone,<br><br>on every linstor controller i have setup on proxmox (3 different clusters on different locations, not related between them)<br><br>got the following error at the same time :<br><br><br>root@pve02:~# systemctl status linstor-controller<br>● linstor-controller.service - drbd-reactor controlled linstor-controller<br>     Loaded: loaded (/lib/systemd/system/linstor-controller.service; disabled; preset: enabled)<br>    Drop-In: /run/systemd/system/linstor-controller.service.d<br>             └─reactor.conf<br>     Active: active (running) since Sun 2023-09-24 21:41:01 EEST; 6 days ago<br>   Main PID: 7156 (java)<br>      Tasks: 58 (limit: 38227)<br>     Memory: 481.8M<br>        CPU: 21min 54.401s<br>     CGroup: /system.slice/linstor-controller.service<br>             └─7156 java -Xms32M -classpath &quot;/usr/share/linstor-server/lib/conf:/usr/share/linstor-server/lib/*&quot; com.linbit.linstor.core.Controller --logs=/var/log/linstor-controller --config-directory=/etc/linstor<br><br>Oct 01 01:12:02 pve02 Controller[7156]: 01:12:02.885 [grizzly-http-server-4] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.122; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:12:11 pve02 Controller[7156]: 01:12:11.810 [grizzly-http-server-7] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.121; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:12:33 pve02 Controller[7156]: 01:12:33.507 [grizzly-http-server-9] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.123; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:13:02 pve02 Controller[7156]: 01:13:02.844 [grizzly-http-server-11] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.122; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:13:11 pve02 Controller[7156]: 01:13:11.607 [grizzly-http-server-1] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.121; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:13:33 pve02 Controller[7156]: 01:13:33.955 [grizzly-http-server-2] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.123; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:14:02 pve02 Controller[7156]: 01:14:02.687 [grizzly-http-server-4] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.122; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:14:11 pve02 Controller[7156]: 01:14:11.690 [grizzly-http-server-7] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.121; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:14:43 pve02 Controller[7156]: 01:14:43.390 [grizzly-http-server-9] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.123; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:15:00 pve02 Controller[7156]: 01:15:00.043 [SpaceTrackingService] ERROR LINSTOR/Controller - SYSTEM - Uncaught exception in j [Report number 651082BA-00000-000000]<br><br><br>the reports shows : <br><br><br>root@pve02:~# linstor error-reports show 651082BA-00000-000000<br>ERROR REPORT 651082BA-00000-000000<br><br>============================================================<br><br>Application:                        LINBIT® LINSTOR<br>Module:                             Controller<br>Version:                            1.24.2<br>Build ID:                           adb19ca96a07039401023410c1ea116f09929295<br>Build time:                         2023-08-30T05:15:08+00:00<br>Error time:                         2023-10-01 01:15:00<br>Node:                               pve02<br><br>============================================================<br><br>Reported error:<br>===============<br><br>Category:                           Error<br>Class name:                         ImplementationError<br>Class canonical name:               com.linbit.ImplementationError<br>Generated at:                       Method &#39;run&#39;, Source file &#39;SpaceTrackingTask.java&#39;, Line #315<br><br>Error message:                      Uncaught exception in j<br><br>Call backtrace:<br><br>    Method                                   Native Class:Line number<br>    run                                      N      com.linbit.linstor.spacetracking.j:315<br>    run                                      N      java.lang.Thread:833<br><br>Caused by:<br>==========<br><br>Category:                           RuntimeException<br>Class name:                         ProvisionException<br>Class canonical name:               com.google.inject.ProvisionException<br>Generated at:                       Method &#39;toProvisionException&#39;, Source file &#39;InternalProvisionException.java&#39;, Line #251<br><br>Error message:                      Unable to provision, see the following errors:<br><br>1) [Guice/ErrorInCustomProvider]: OutOfScopeException: Cannot access Key[type=TransactionMgrSQL, annotation=[none]] outside of a scoping block<br>  at ApiModule.configure(ApiModule.java:79)<br>  while locating TransactionMgrSQL<br><br>Learn more:<br>  <a href="https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER">https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER</a><br><br>1 error<br><br>======================<br>Full classname legend:<br>======================<br>ApiModule:           &quot;com.linbit.linstor.api.ApiModule&quot;<br>OutOfScopeException: &quot;com.google.inject.OutOfScopeException&quot;<br>TransactionMgrSQL:   &quot;com.linbit.linstor.transaction.manager.TransactionMgrSQL&quot;<br>========================<br>End of classname legend:<br>========================<br><br><br>Call backtrace:<br><br>    Method                                   Native Class:Line number<br>    toProvisionException                     N      com.google.inject.internal.InternalProvisionException:251<br>    get                                      N      com.google.inject.internal.InjectorImpl$1:1104<br>    getConnection                            N      com.linbit.linstor.dbdrivers.sql.SQLEngine:689<br>    loadAll                                  N      com.linbit.linstor.dbdrivers.sql.SQLEngine:491<br>    loadAll                                  N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:170<br>    loadAllAsList                            N      com.linbit.linstor.dbdrivers.ControllerDatabaseDriver:33<br>    b                                        N      com.linbit.linstor.spacetracking.h:260<br>    e                                        N      com.linbit.linstor.spacetracking.j:737<br>    run                                      N      com.linbit.linstor.spacetracking.j:283<br>    run                                      N      java.lang.Thread:833<br><br>Caused by:<br>==========<br><br>Category:                           RuntimeException<br>Class name:                         OutOfScopeException<br>Class canonical name:               com.google.inject.OutOfScopeException<br>Generated at:                       Method &#39;getScopedObjectMap&#39;, Source file &#39;LinStorScope.java&#39;, Line #125<br><br>Error message:                      Cannot access Key[type=com.linbit.linstor.transaction.manager.TransactionMgrSQL, annotation=[none]] outside of a scoping block<br><br>Call backtrace:<br><br>    Method                                   Native Class:Line number<br>    getScopedObjectMap                       N      com.linbit.linstor.api.LinStorScope:125<br>    get                                      N      com.linbit.linstor.api.LinStorScope$2:91<br>    get                                      N      com.google.inject.internal.InternalFactoryToProviderAdapter:45<br>    get                                      N      com.google.inject.internal.InjectorImpl$1:1101<br>    getConnection                            N      com.linbit.linstor.dbdrivers.sql.SQLEngine:689<br>    loadAll                                  N      com.linbit.linstor.dbdrivers.sql.SQLEngine:491<br>    loadAll                                  N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:170<br>    loadAllAsList                            N      com.linbit.linstor.dbdrivers.ControllerDatabaseDriver:33<br>    b                                        N      com.linbit.linstor.spacetracking.h:260<br>    e                                        N      com.linbit.linstor.spacetracking.j:737<br>    run                                      N      com.linbit.linstor.spacetracking.j:283<br>    run                                      N      java.lang.Thread:833<br><br><br>END OF ERROR REPORT.<br><br><br>running a systemctl restart linstor-controller on the node that runs the controller seems to bring things back in order.<br><div><br></div><div>any thoughts ?</div></div>