[DRBD-user] linstor-server 1.9.0 release

Rene Peinthor rene.peinthor at linbit.com
Thu Sep 24 15:48:11 CEST 2020


I can only recreate that error, if I actively change the password in the
linstor.toml to something incorrect.
I don't know how this would happen in normal operation, are you sure you
never had another password or changed the password?

if you can connect to the h2 database with the commandline tool, you might
be able to set a new password.

Cheers,
Rene

On Thu, Sep 24, 2020 at 2:48 PM Adam Goryachev <
mailinglists at websitemanagers.com.au> wrote:

>
> On 24/9/20 22:42, Rene Peinthor wrote:
>
> They are.
> And how do you start the controller?
>
> It should start automatically at boot from systemd, but I can start it
> manually with:
>
> systemctl restart linstor-controller
>
> In case it is your next question, here is the content of
> /etc/systemd/system/multi-user.target.wants/linstor-controller.service
>
> [Unit]
> Description=LINSTOR Controller Service
> Wants=network-online.target
> After=network-online.target
>
> [Service]
> Type=simple
> ExecStart=/usr/share/linstor-server/bin/Controller
> --logs=/var/log/linstor-controller --config-directory=/etc/linstor
> # if killed by signal 143 -> SIGTERM, 129 -> SIGHUP
> SuccessExitStatus=0 143 129
> User=root
> PrivateTmp=yes
>
> [Install]
> WantedBy=multi-user.target
>
> Regards,
> Adam
>
> On Thu, Sep 24, 2020 at 2:19 PM Adam Goryachev <
> mailinglists at websitemanagers.com.au> wrote:
>
>>
>> On 24/9/20 15:30, Rene Peinthor wrote:
>>
>> Sounds weird, I don't think we changed anything with the database
>> initialization.
>>
>> Can you post your /etc/linstor/linstor.toml?
>>
>> Cheers,
>> Rene
>>
>> Hi, please see the contents below (I assume these are default settings):
>>
>> # Basic linstor configuration toml file
>> # For more options check documentation
>>
>> [db]
>>   user = "linstor"
>>   password = "linstor"
>>   connection_url = "jdbc:h2:/var/lib/linstor/linstordb"
>>
>> Thanks,
>> Adam
>>
>>
>> On Wed, Sep 23, 2020 at 6:54 PM Adam Goryachev <
>> mailinglists at websitemanagers.com.au> wrote:
>>
>>> Hi,
>>>
>>> After upgrading on one of my test nodes I got some errors, so I thought
>>> to upgrade the controller, and now I am unable to start the controller.
>>>
>>> Here is what I get in the system log:
>>>
>>> Sep 24 02:33:40 castle systemd[1]: Started LINSTOR Controller Service.
>>> Sep 24 02:33:41 castle Controller[1093]: LINSTOR, Module Controller
>>> Sep 24 02:33:41 castle Controller[1093]: Version:            1.9.0
>>> (678acd24a8b9b73a735407cd79ca33a5e95eb2e2)
>>> Sep 24 02:33:41 castle Controller[1093]: Build time:
>>> 2020-09-23T10:27:49+00:00
>>> Sep 24 02:33:41 castle Controller[1093]: Java Version:       11
>>> Sep 24 02:33:41 castle Controller[1093]: Java VM:            Debian,
>>> Version 11.0.8+10-post-Debian-1deb10u1
>>> Sep 24 02:33:41 castle Controller[1093]: Operating system:   Linux,
>>> Version 4.19.0-10-amd64
>>> Sep 24 02:33:41 castle Controller[1093]: Environment:        amd64, 2
>>> processors, 1988 MiB memory reserved for allocations
>>> Sep 24 02:33:41 castle Controller[1093]: System components
>>> initialization in progress
>>> Sep 24 02:33:42 castle Controller[1093]: 02:33:42.355 [main] INFO
>>> LINSTOR/Controller - SYSTEM - ErrorReporter DB version 1 found.
>>> Sep 24 02:33:42 castle Controller[1093]: 02:33:42.359 [main] INFO
>>> LINSTOR/Controller - SYSTEM - Log directory set to:
>>> '/var/log/linstor-controller'
>>> Sep 24 02:33:42 castle Controller[1093]: 02:33:42.410 [main] WARN
>>> io.sentry.dsn.Dsn - *** Couldn't find a suitable DSN, Sentry operations
>>> will do nothing! See documentation: https://docs.sentry.io/clients/java/
>>> ***
>>> Sep 24 02:33:42 castle Controller[1093]: 02:33:42.434 [Main] INFO
>>> LINSTOR/Controller - SYSTEM - Loading API classes started.
>>> Sep 24 02:33:43 castle Controller[1093]: 02:33:43.046 [Main] INFO
>>> LINSTOR/Controller - SYSTEM - API classes loading finished: 611ms
>>> Sep 24 02:33:43 castle Controller[1093]: 02:33:43.047 [Main] INFO
>>> LINSTOR/Controller - SYSTEM - Dependency injection started.
>>> Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main] INFO
>>> LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module
>>> "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
>>> Sep 24 02:33:43 castle Controller[1093]: 02:33:43.065 [Main] INFO
>>> LINSTOR/Controller - SYSTEM - Extension module
>>> "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" is not
>>> installed
>>> Sep 24 02:33:43 castle Controller[1093]: WARNING: An illegal reflective
>>> access operation has occurred
>>> Sep 24 02:33:43 castle Controller[1093]: WARNING: Illegal reflective
>>> access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (
>>> file:/usr/share/linstor-server/lib/guice-4.2.2.jar) to method
>>> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
>>> Sep 24 02:33:43 castle Controller[1093]: WARNING: Please consider
>>> reporting this to the maintainers of
>>> com.google.inject.internal.cglib.core.$ReflectUtils$1
>>> Sep 24 02:33:43 castle Controller[1093]: WARNING: Use
>>> --illegal-access=warn to enable warnings of further illegal reflective
>>> access operations
>>> Sep 24 02:33:43 castle Controller[1093]: WARNING: All illegal access
>>> operations will be denied in a future release
>>> Sep 24 02:33:44 castle Controller[1093]: 02:33:44.962 [Main] INFO
>>> LINSTOR/Controller - SYSTEM - Dependency injection finished: 1915ms
>>> Sep 24 02:33:45 castle Controller[1093]: 02:33:45.533 [Main] INFO
>>> LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
>>> Sep 24 02:33:45 castle Controller[1093]: 02:33:45.736 [Main] INFO
>>> LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService'
>>> of type TimerEventService
>>> Sep 24 02:33:45 castle Controller[1093]: 02:33:45.740 [Main] INFO
>>> LINSTOR/Controller - SYSTEM - Initializing the database connection pool
>>> Sep 24 02:33:46 castle Controller[1093]: 02:33:46.303 [Main] ERROR
>>> LINSTOR/Controller - SYSTEM - Database initialization error [Report number
>>> 5F6B78E5-00000-000000]
>>> Sep 24 02:33:46 castle Controller[1093]: 02:33:46.343 [Thread-2] INFO
>>> LINSTOR/Controller - SYSTEM - Shutdown in progress
>>> Sep 24 02:33:46 castle Controller[1093]: 02:33:46.344 [Thread-2] INFO
>>> LINSTOR/Controller - SYSTEM - Shutting down service instance
>>> 'TaskScheduleService' of type TaskScheduleService
>>> Sep 24 02:33:46 castle Controller[1093]: 02:33:46.345 [Thread-2] INFO
>>> LINSTOR/Controller - SYSTEM - Waiting for service instance
>>> 'TaskScheduleService' to complete shutdown
>>> Sep 24 02:33:46 castle Controller[1093]: 02:33:46.345 [Thread-2] INFO
>>> LINSTOR/Controller - SYSTEM - Shutting down service instance
>>> 'DatabaseService' of type DatabaseService
>>> Sep 24 02:33:46 castle Controller[1093]: 02:33:46.349 [Thread-2] INFO
>>> LINSTOR/Controller - SYSTEM - Waiting for service instance
>>> 'DatabaseService' to complete shutdown
>>> Sep 24 02:33:46 castle Controller[1093]: 02:33:46.350 [Thread-2] INFO
>>> LINSTOR/Controller - SYSTEM - Shutting down service instance
>>> 'TimerEventService' of type TimerEventService
>>> Sep 24 02:33:46 castle Controller[1093]: 02:33:46.351 [Thread-2] INFO
>>> LINSTOR/Controller - SYSTEM - Waiting for service instance
>>> 'TimerEventService' to complete shutdown
>>> Sep 24 02:33:46 castle Controller[1093]: 02:33:46.351 [Thread-2] INFO
>>> LINSTOR/Controller - SYSTEM - Shutdown complete
>>> Sep 24 02:33:46 castle systemd[1]: linstor-controller.service: Main
>>> process exited, code=exited, status=20/n/a
>>> Sep 24 02:33:46 castle systemd[1]: linstor-controller.service: Failed
>>> with result 'exit-code'.
>>>
>>> Here is /var/log/linstor-controller/ErrorReport-5F6B78E5-00000-000000.log
>>>
>>> ERROR REPORT 5F6B78E5-00000-000000
>>>
>>> ============================================================
>>>
>>> Application:                        LINBIT® LINSTOR
>>> Module:                             Controller
>>> Version:                            1.9.0
>>> Build ID:
>>> 678acd24a8b9b73a735407cd79ca33a5e95eb2e2
>>> Build time:                         2020-09-23T10:27:49+00:00
>>> Error time:                         2020-09-24 02:33:46
>>> Node:                               castle
>>>
>>> ============================================================
>>>
>>> Reported error:
>>> ===============
>>>
>>> Description:
>>>     Database initialization error
>>>
>>> Category:                           LinStorException
>>> Class name:                         SystemServiceStartException
>>> Class canonical name:
>>> com.linbit.SystemServiceStartException
>>> Generated at:                       Method 'initialize', Source file
>>> 'DbConnectionPoolInitializer.java', Line #56
>>> Error message:                      Database initialization error
>>>
>>> Call backtrace:
>>>
>>>     Method                                   Native Class:Line number
>>>     initialize                               N
>>> com.linbit.linstor.dbcp.DbConnectionPoolInitializer:56
>>>     startSystemServices                      N
>>> com.linbit.linstor.core.ApplicationLifecycleManager:87
>>>     start                                    N
>>> com.linbit.linstor.core.Controller:337
>>>     main                                     N
>>> com.linbit.linstor.core.Controller:556
>>>
>>> Caused by:
>>> ==========
>>>
>>> Category:                           RuntimeException
>>> Class name:                         LinStorDBRuntimeException
>>> Class canonical name:
>>> com.linbit.linstor.LinStorDBRuntimeException
>>> Generated at:                       Method 'setTransactionIsolation',
>>> Source file 'DbConnectionPool.java', Line #485
>>>
>>> Error message:                      Failed to set transaction isolation
>>>
>>> Call backtrace:
>>>
>>>     Method                                   Native Class:Line number
>>>     setTransactionIsolation                  N
>>> com.linbit.linstor.dbcp.DbConnectionPool:485
>>>     migrate                                  N
>>> com.linbit.linstor.dbcp.DbConnectionPool:202
>>>     initialize                               N
>>> com.linbit.linstor.dbcp.DbConnectionPoolInitializer:50
>>>     startSystemServices                      N
>>> com.linbit.linstor.core.ApplicationLifecycleManager:87
>>>     start                                    N
>>> com.linbit.linstor.core.Controller:337
>>>     main                                     N
>>> com.linbit.linstor.core.Controller:556
>>>
>>> Caused by:
>>> ==========
>>>
>>> Category:                           Exception
>>> Class name:                         JdbcSQLException
>>> Class canonical name:               org.h2.jdbc.JdbcSQLException
>>> Generated at:                       Method 'getJdbcSQLException', Source
>>> file 'DbException.java', Line #357
>>>
>>> Error message:                      Wrong user name or password
>>> [28000-197]
>>>
>>> Call backtrace:
>>>
>>>     Method                                   Native Class:Line number
>>>     getJdbcSQLException                      N
>>> org.h2.message.DbException:357
>>>     get                                      N
>>> org.h2.message.DbException:179
>>>     get                                      N
>>> org.h2.message.DbException:155
>>>     get                                      N
>>> org.h2.message.DbException:144
>>>     validateUserAndPassword                  N
>>> org.h2.engine.Engine:341
>>>     createSessionAndValidate                 N
>>> org.h2.engine.Engine:165
>>>     createSession                            N
>>> org.h2.engine.Engine:140
>>>     createSession                            N
>>> org.h2.engine.Engine:28
>>>     connectEmbeddedOrServer                  N
>>> org.h2.engine.SessionRemote:351
>>>     <init>                                   N
>>> org.h2.jdbc.JdbcConnection:124
>>>     <init>                                   N
>>> org.h2.jdbc.JdbcConnection:103
>>>     connect                                  N      org.h2.Driver:69
>>>     getConnection                            N
>>> java.sql.DriverManager:677
>>>     getConnection                            N
>>> java.sql.DriverManager:189
>>>     createConnection                         N
>>> org.apache.commons.dbcp2.DriverManagerConnectionFactory:123
>>>     makeObject                               N
>>> org.apache.commons.dbcp2.PoolableConnectionFactory:355
>>>     create                                   N
>>> org.apache.commons.pool2.impl.GenericObjectPool:889
>>>     borrowObject                             N
>>> org.apache.commons.pool2.impl.GenericObjectPool:424
>>>     borrowObject                             N
>>> org.apache.commons.pool2.impl.GenericObjectPool:349
>>>     getConnection                            N
>>> org.apache.commons.dbcp2.PoolingDataSource:134
>>>     setTransactionIsolation                  N
>>> com.linbit.linstor.dbcp.DbConnectionPool:477
>>>     migrate                                  N
>>> com.linbit.linstor.dbcp.DbConnectionPool:202
>>>     initialize                               N
>>> com.linbit.linstor.dbcp.DbConnectionPoolInitializer:50
>>>     startSystemServices                      N
>>> com.linbit.linstor.core.ApplicationLifecycleManager:87
>>>     start                                    N
>>> com.linbit.linstor.core.Controller:337
>>>     main                                     N
>>> com.linbit.linstor.core.Controller:556
>>>
>>> Finally I found another log in /var/lib/linstor/linstordb.trace.db
>>>
>>> 2020-09-24 02:33:45 database: wrong user or password; user: "LINSTOR"
>>> org.h2.message.DbException: Wrong user name or password [28000-197]
>>>         at org.h2.message.DbException.get(DbException.java:179)
>>>         at org.h2.message.DbException.get(DbException.java:155)
>>>         at org.h2.message.DbException.get(DbException.java:144)
>>>         at org.h2.engine.Engine.openSession(Engine.java:107)
>>>         at org.h2.engine.Engine.openSession(Engine.java:179)
>>>         at org.h2.engine.Engine.createSessionAndValidate(Engine.java:157)
>>>         at org.h2.engine.Engine.createSession(Engine.java:140)
>>>         at org.h2.engine.Engine.createSession(Engine.java:28)
>>>         at
>>> org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:351)
>>>         at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:124)
>>>         at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:103)
>>>         at org.h2.Driver.connect(Driver.java:69)
>>>         at
>>> java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
>>>         at
>>> java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
>>>         at
>>> org.apache.commons.dbcp2.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:123)
>>>         at
>>> org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355)
>>>         at
>>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
>>>         at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
>>>         at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
>>>         at
>>> org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
>>>         at
>>> com.linbit.linstor.dbcp.DbConnectionPool.setTransactionIsolation(DbConnectionPool.java:477)
>>>         at
>>> com.linbit.linstor.dbcp.DbConnectionPool.migrate(DbConnectionPool.java:202)
>>>         at
>>> com.linbit.linstor.dbcp.DbConnectionPoolInitializer.initialize(DbConnectionPoolInitializer.java:50)
>>>         at
>>> com.linbit.linstor.core.ApplicationLifecycleManager.startSystemServices(ApplicationLifecycleManager.java:87)
>>>         at com.linbit.linstor.core.Controller.start(Controller.java:337)
>>>         at com.linbit.linstor.core.Controller.main(Controller.java:556)
>>> Caused by: org.h2.jdbc.JdbcSQLException: Wrong user name or password
>>> [28000-197]
>>>         at
>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
>>>         ... 26 more
>>>
>>>
>>> Luckily, this is just a test environment, but I'd like to know if I
>>> messed up in my upgrade process somehow or this is a bug in the new
>>> version. I haven't downgraded or anything yet.
>>>
>>> Regards,
>>> Adam
>>>
>>>
>>> On 23/9/20 22:31, Rene Peinthor wrote:
>>>
>>> Hi!
>>>
>>> Here is a new minor Linstor release, this release contains more bug
>>> fixes than new features.
>>> Notable features are a new auto-unplace option, to reduce the
>>> replica-count of resources
>>> to the value set in the resource-group after a certain time. And
>>> error-reports can now be deleted
>>> from the REST-API (client).
>>>
>>> linstor-server 1.9.0
>>> --------------------
>>>  * Added auto-"unplace", automatically removes resources if they exceed
>>> the replica count if `DrbdOptions/auto-unplace` is set
>>>  * Added error-report deletion API
>>>  * Expose DRBD promotion_score and may_promote via REST-API and server
>>> sent events
>>>  * Prometheus metrics are now mostly cached and much faster to query
>>>  * Send SD_NOTIFY READY to systemd if services are ready
>>>  * OpenFlex resource are now correctly limited to 32 characters
>>>  * Fix etcd modify net-interface (incorrect serialization)
>>>  * Fix autoplacer also toggling diskless resources
>>>  * Fix snapshot-shipping with more than 1 volume definitions
>>>  * Fix zfs create options handling
>>>  * Fixed recaching of lvm-filter
>>>  * Fix missing client answer if drbd device doesn't become ready in
>>> reasonable time
>>>  * REST-API v1.4.0
>>>
>>> https://www.linbit.com/downloads/linstor/linstor-server-1.9.0.tar.gz
>>>
>>> Linstor PPA:
>>> https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack
>>>
>>> Cheers,
>>> Rene
>>>
>>> _______________________________________________
>>> Star us on GITHUB: https://github.com/LINBIT
>>> drbd-user mailing listdrbd-user at lists.linbit.comhttps://lists.linbit.com/mailman/listinfo/drbd-user
>>>
>>> _______________________________________________
>>> Star us on GITHUB: https://github.com/LINBIT
>>> drbd-user mailing list
>>> drbd-user at lists.linbit.com
>>> https://lists.linbit.com/mailman/listinfo/drbd-user
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20200924/d68af92d/attachment-0001.htm>


More information about the drbd-user mailing list