<div dir="ltr">hello everyone,<br><br>on every linstor controller i have setup on proxmox (3 different clusters on different locations, not related between them)<br><br>got the following error at the same time :<br><br><br>root@pve02:~# systemctl status linstor-controller<br>● linstor-controller.service - drbd-reactor controlled linstor-controller<br>  Â  Â Loaded: loaded (/lib/systemd/system/linstor-controller.service; disabled; preset: enabled)<br>  Â  Drop-In: /run/systemd/system/linstor-controller.service.d<br>  Â  Â  Â  Â  Â  Â â””─reactor.conf<br>  Â  Â Active: active (running) since Sun 2023-09-24 21:41:01 EEST; 6 days ago<br>  Â Main PID: 7156 (java)<br>  Â  Â  Tasks: 58 (limit: 38227)<br>  Â  Â Memory: 481.8M<br>  Â  Â  Â  CPU: 21min 54.401s<br>  Â  Â CGroup: /system.slice/linstor-controller.service<br>  Â  Â  Â  Â  Â  Â â””─7156 java -Xms32M -classpath &quot;/usr/share/linstor-server/lib/conf:/usr/share/linstor-server/lib/*&quot; com.linbit.linstor.core.Controller --logs=/var/log/linstor-controller --config-directory=/etc/linstor<br><br>Oct 01 01:12:02 pve02 Controller[7156]: 01:12:02.885 [grizzly-http-server-4] INFO Â LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.122; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:12:11 pve02 Controller[7156]: 01:12:11.810 [grizzly-http-server-7] INFO Â LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.121; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:12:33 pve02 Controller[7156]: 01:12:33.507 [grizzly-http-server-9] INFO Â LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.123; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:13:02 pve02 Controller[7156]: 01:13:02.844 [grizzly-http-server-11] INFO Â LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.122; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:13:11 pve02 Controller[7156]: 01:13:11.607 [grizzly-http-server-1] INFO Â LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.121; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:13:33 pve02 Controller[7156]: 01:13:33.955 [grizzly-http-server-2] INFO Â LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.123; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:14:02 pve02 Controller[7156]: 01:14:02.687 [grizzly-http-server-4] INFO Â LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.122; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:14:11 pve02 Controller[7156]: 01:14:11.690 [grizzly-http-server-7] INFO Â LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.121; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:14:43 pve02 Controller[7156]: 01:14:43.390 [grizzly-http-server-9] INFO Â LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.123; &#39;linstor-proxmox/7.0.1&#39;)/QryAllSizeInfo<br>Oct 01 01:15:00 pve02 Controller[7156]: 01:15:00.043 [SpaceTrackingService] ERROR LINSTOR/Controller - SYSTEM - Uncaught exception in j [Report number 651082BA-00000-000000]<br><br><br>the reports shows : <br><br><br>root@pve02:~# linstor error-reports show 651082BA-00000-000000<br>ERROR REPORT 651082BA-00000-000000<br><br>============================================================<br><br>Application: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â LINBIT® LINSTOR<br>Module: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Controller<br>Version: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â 1.24.2<br>Build ID: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  adb19ca96a07039401023410c1ea116f09929295<br>Build time: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  2023-08-30T05:15:08+00:00<br>Error time: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  2023-10-01 01:15:00<br>Node: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  pve02<br><br>============================================================<br><br>Reported error:<br>===============<br><br>Category: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Error<br>Class name: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ImplementationError<br>Class canonical name: Â  Â  Â  Â  Â  Â  Â  com.linbit.ImplementationError<br>Generated at: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Method &#39;run&#39;, Source file &#39;SpaceTrackingTask.java&#39;, Line #315<br><br>Error message: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â Uncaught exception in j<br><br>Call backtrace:<br><br>  Â  Method Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Native Class:Line number<br>  Â  run Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.spacetracking.j:315<br>  Â  run Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â java.lang.Thread:833<br><br>Caused by:<br>==========<br><br>Category: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  RuntimeException<br>Class name: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ProvisionException<br>Class canonical name: Â  Â  Â  Â  Â  Â  Â  com.google.inject.ProvisionException<br>Generated at: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Method &#39;toProvisionException&#39;, Source file &#39;InternalProvisionException.java&#39;, Line #251<br><br>Error message: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â Unable to provision, see the following errors:<br><br>1) [Guice/ErrorInCustomProvider]: OutOfScopeException: Cannot access Key[type=TransactionMgrSQL, annotation=[none]] outside of a scoping block<br>  at ApiModule.configure(ApiModule.java:79)<br>  while locating TransactionMgrSQL<br><br>Learn more:<br>  <a href="https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER">https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER</a><br><br>1 error<br><br>======================<br>Full classname legend:<br>======================<br>ApiModule: Â  Â  Â  Â  Â  &quot;com.linbit.linstor.api.ApiModule&quot;<br>OutOfScopeException: &quot;com.google.inject.OutOfScopeException&quot;<br>TransactionMgrSQL: Â  &quot;com.linbit.linstor.transaction.manager.TransactionMgrSQL&quot;<br>========================<br>End of classname legend:<br>========================<br><br><br>Call backtrace:<br><br>  Â  Method Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Native Class:Line number<br>  Â  toProvisionException Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  N Â  Â  Â com.google.inject.internal.InternalProvisionException:251<br>  Â  get Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.google.inject.internal.InjectorImpl$1:1104<br>  Â  getConnection Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.dbdrivers.sql.SQLEngine:689<br>  Â  loadAll Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.dbdrivers.sql.SQLEngine:491<br>  Â  loadAll Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.dbdrivers.AbsDatabaseDriver:170<br>  Â  loadAllAsList Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.dbdrivers.ControllerDatabaseDriver:33<br>  Â  b Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.spacetracking.h:260<br>  Â  e Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.spacetracking.j:737<br>  Â  run Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.spacetracking.j:283<br>  Â  run Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â java.lang.Thread:833<br><br>Caused by:<br>==========<br><br>Category: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  RuntimeException<br>Class name: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  OutOfScopeException<br>Class canonical name: Â  Â  Â  Â  Â  Â  Â  com.google.inject.OutOfScopeException<br>Generated at: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Method &#39;getScopedObjectMap&#39;, Source file &#39;LinStorScope.java&#39;, Line #125<br><br>Error message: Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â Cannot access Key[type=com.linbit.linstor.transaction.manager.TransactionMgrSQL, annotation=[none]] outside of a scoping block<br><br>Call backtrace:<br><br>  Â  Method Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Native Class:Line number<br>  Â  getScopedObjectMap Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  N Â  Â  Â com.linbit.linstor.api.LinStorScope:125<br>  Â  get Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.api.LinStorScope$2:91<br>  Â  get Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.google.inject.internal.InternalFactoryToProviderAdapter:45<br>  Â  get Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.google.inject.internal.InjectorImpl$1:1101<br>  Â  getConnection Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.dbdrivers.sql.SQLEngine:689<br>  Â  loadAll Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.dbdrivers.sql.SQLEngine:491<br>  Â  loadAll Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.dbdrivers.AbsDatabaseDriver:170<br>  Â  loadAllAsList Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.dbdrivers.ControllerDatabaseDriver:33<br>  Â  b Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.spacetracking.h:260<br>  Â  e Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.spacetracking.j:737<br>  Â  run Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â com.linbit.linstor.spacetracking.j:283<br>  Â  run Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â N Â  Â  Â java.lang.Thread:833<br><br><br>END OF ERROR REPORT.<br><br><br>running a systemctl restart linstor-controller on the node that runs the controller seems to bring things back in order.<br><div><br></div><div>any thoughts ?</div></div>