<div dir="ltr">hello everyone,<br><br>on every linstor controller i have setup on proxmox (3 different clusters on different locations, not related between them)<br><br>got the following error at the same time :<br><br><br>root@pve02:~# systemctl status linstor-controller<br>◠linstor-controller.service - drbd-reactor controlled linstor-controller<br>   Loaded: loaded (/lib/systemd/system/linstor-controller.service; disabled; preset: enabled)<br>  Drop-In: /run/systemd/system/linstor-controller.service.d<br>       └─reactor.conf<br>   Active: active (running) since Sun 2023-09-24 21:41:01 EEST; 6 days ago<br>  Main PID: 7156 (java)<br>   Tasks: 58 (limit: 38227)<br>   Memory: 481.8M<br>    CPU: 21min 54.401s<br>   CGroup: /system.slice/linstor-controller.service<br>       └─7156 java -Xms32M -classpath "/usr/share/linstor-server/lib/conf:/usr/share/linstor-server/lib/*" com.linbit.linstor.core.Controller --logs=/var/log/linstor-controller --config-directory=/etc/linstor<br><br>Oct 01 01:12:02 pve02 Controller[7156]: 01:12:02.885 [grizzly-http-server-4] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.122; 'linstor-proxmox/7.0.1')/QryAllSizeInfo<br>Oct 01 01:12:11 pve02 Controller[7156]: 01:12:11.810 [grizzly-http-server-7] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.121; 'linstor-proxmox/7.0.1')/QryAllSizeInfo<br>Oct 01 01:12:33 pve02 Controller[7156]: 01:12:33.507 [grizzly-http-server-9] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.123; 'linstor-proxmox/7.0.1')/QryAllSizeInfo<br>Oct 01 01:13:02 pve02 Controller[7156]: 01:13:02.844 [grizzly-http-server-11] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.122; 'linstor-proxmox/7.0.1')/QryAllSizeInfo<br>Oct 01 01:13:11 pve02 Controller[7156]: 01:13:11.607 [grizzly-http-server-1] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.121; 'linstor-proxmox/7.0.1')/QryAllSizeInfo<br>Oct 01 01:13:33 pve02 Controller[7156]: 01:13:33.955 [grizzly-http-server-2] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.123; 'linstor-proxmox/7.0.1')/QryAllSizeInfo<br>Oct 01 01:14:02 pve02 Controller[7156]: 01:14:02.687 [grizzly-http-server-4] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.122; 'linstor-proxmox/7.0.1')/QryAllSizeInfo<br>Oct 01 01:14:11 pve02 Controller[7156]: 01:14:11.690 [grizzly-http-server-7] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.121; 'linstor-proxmox/7.0.1')/QryAllSizeInfo<br>Oct 01 01:14:43 pve02 Controller[7156]: 01:14:43.390 [grizzly-http-server-9] INFO  LINSTOR/Controller - SYSTEM - REST/API RestClient(10.0.3.123; 'linstor-proxmox/7.0.1')/QryAllSizeInfo<br>Oct 01 01:15:00 pve02 Controller[7156]: 01:15:00.043 [SpaceTrackingService] ERROR LINSTOR/Controller - SYSTEM - Uncaught exception in j [Report number 651082BA-00000-000000]<br><br><br>the reports shows : <br><br><br>root@pve02:~# linstor error-reports show 651082BA-00000-000000<br>ERROR REPORT 651082BA-00000-000000<br><br>============================================================<br><br>Application:             LINBIT® LINSTOR<br>Module:               Controller<br>Version:               1.24.2<br>Build ID:              adb19ca96a07039401023410c1ea116f09929295<br>Build time:             2023-08-30T05:15:08+00:00<br>Error time:             2023-10-01 01:15:00<br>Node:                pve02<br><br>============================================================<br><br>Reported error:<br>===============<br><br>Category:              Error<br>Class name:             ImplementationError<br>Class canonical name:        com.linbit.ImplementationError<br>Generated at:            Method 'run', Source file 'SpaceTrackingTask.java', Line #315<br><br>Error message:            Uncaught exception in j<br><br>Call backtrace:<br><br>  Method                  Native Class:Line number<br>  run                    N    com.linbit.linstor.spacetracking.j:315<br>  run                    N    java.lang.Thread:833<br><br>Caused by:<br>==========<br><br>Category:              RuntimeException<br>Class name:             ProvisionException<br>Class canonical name:        com.google.inject.ProvisionException<br>Generated at:            Method 'toProvisionException', Source file 'InternalProvisionException.java', Line #251<br><br>Error message:            Unable to provision, see the following errors:<br><br>1) [Guice/ErrorInCustomProvider]: OutOfScopeException: Cannot access Key[type=TransactionMgrSQL, annotation=[none]] outside of a scoping block<br> at ApiModule.configure(ApiModule.java:79)<br> while locating TransactionMgrSQL<br><br>Learn more:<br> <a href="https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER">https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER</a><br><br>1 error<br><br>======================<br>Full classname legend:<br>======================<br>ApiModule:      "com.linbit.linstor.api.ApiModule"<br>OutOfScopeException: "com.google.inject.OutOfScopeException"<br>TransactionMgrSQL:  "com.linbit.linstor.transaction.manager.TransactionMgrSQL"<br>========================<br>End of classname legend:<br>========================<br><br><br>Call backtrace:<br><br>  Method                  Native Class:Line number<br>  toProvisionException           N    com.google.inject.internal.InternalProvisionException:251<br>  get                    N    com.google.inject.internal.InjectorImpl$1:1104<br>  getConnection               N    com.linbit.linstor.dbdrivers.sql.SQLEngine:689<br>  loadAll                  N    com.linbit.linstor.dbdrivers.sql.SQLEngine:491<br>  loadAll                  N    com.linbit.linstor.dbdrivers.AbsDatabaseDriver:170<br>  loadAllAsList               N    com.linbit.linstor.dbdrivers.ControllerDatabaseDriver:33<br>  b                     N    com.linbit.linstor.spacetracking.h:260<br>  e                     N    com.linbit.linstor.spacetracking.j:737<br>  run                    N    com.linbit.linstor.spacetracking.j:283<br>  run                    N    java.lang.Thread:833<br><br>Caused by:<br>==========<br><br>Category:              RuntimeException<br>Class name:             OutOfScopeException<br>Class canonical name:        com.google.inject.OutOfScopeException<br>Generated at:            Method 'getScopedObjectMap', Source file 'LinStorScope.java', Line #125<br><br>Error message:            Cannot access Key[type=com.linbit.linstor.transaction.manager.TransactionMgrSQL, annotation=[none]] outside of a scoping block<br><br>Call backtrace:<br><br>  Method                  Native Class:Line number<br>  getScopedObjectMap            N    com.linbit.linstor.api.LinStorScope:125<br>  get                    N    com.linbit.linstor.api.LinStorScope$2:91<br>  get                    N    com.google.inject.internal.InternalFactoryToProviderAdapter:45<br>  get                    N    com.google.inject.internal.InjectorImpl$1:1101<br>  getConnection               N    com.linbit.linstor.dbdrivers.sql.SQLEngine:689<br>  loadAll                  N    com.linbit.linstor.dbdrivers.sql.SQLEngine:491<br>  loadAll                  N    com.linbit.linstor.dbdrivers.AbsDatabaseDriver:170<br>  loadAllAsList               N    com.linbit.linstor.dbdrivers.ControllerDatabaseDriver:33<br>  b                     N    com.linbit.linstor.spacetracking.h:260<br>  e                     N    com.linbit.linstor.spacetracking.j:737<br>  run                    N    com.linbit.linstor.spacetracking.j:283<br>  run                    N    java.lang.Thread:833<br><br><br>END OF ERROR REPORT.<br><br><br>running a systemctl restart linstor-controller on the node that runs the controller seems to bring things back in order.<br><div><br></div><div>any thoughts ?</div></div>