Unsupported database file version, jbtimer.mv.db

[oxdrove]

Brief Summary

Update from 6.2023.8 to 6.2023.9, error reported:

Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: Connection could not be allocated because: Unsupported database file version or invalid file header in file "/.../payara/domain1/lib/databases/ejbtimer.mv.db" [90048-220]
Error Code: 0
        at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:319)
        at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:152)
        at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:174)
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:226)
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:809)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:259)
        at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:771)

File jbtimer.mv.db was created by the .8 version. I tried deleting the files inl ib/databases and let it remake but it gives another error, so that didn't help:

Internal Exception: java.sql.SQLException: java.lang.reflect.UndeclaredThrowableException
Error Code: 0
Call: SELECT "TIMERID", "APPLICATIONID", "BLOB", "CONTAINERID", "CREATIONTIMERAW", "INITIALEXPIRATIONRAW", "INTERVALDURATION", "LASTEXPIRATIONRAW", "OWNERID", "PKHASHCODE", "SCHEDULE", "STATE" FROM "EJB__TIMER__TBL" WHERE (("OWNERID" = ?) AND ("STATE" = ?))
        bind => [2 parameters bound]
Query: ReadAllQuery(name="findTimersByOwnerAndState" referenceClass=TimerState sql="SELECT "TIMERID", "APPLICATIONID", "BLOB", "CONTAINERID", "CREATIONTIMERAW", "INITIALEXPIRATIONRAW", "INTERVALDURATION", "LASTEXPIRATIONRAW", "OWNERID", "PKHASHCODE", "SCHEDULE", "STATE" FROM "EJB__TIMER__TBL" WHERE (("OWNERID" = ?) AND ("STATE" = ?))")
        at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:343)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:702)

Expected Outcome

Server starts with deployments.

Current Outcome

Payara starts but fails to deploy.

Reproducer

Install 6.2023.8
Deploy with a timer
Update to 6.2023.9

Operating System

OmniOS

JDK Version

openjdk version "17.0.8" 2023-07-18

Payara Distribution

Payara Server Full Profile

[kalinchan]

Hi @oxdrove,

Can you share more details on how to reproduce this issue?

[oxdrove]

How much detail do you need? I can repeat myself with more words - which might reveal something.

I have just repeated and have the same problem.

Install 6.2023.8
create the domain
$ asadmin create-domain --domaindir /..../payara
domain1
admin
*****
*****
$ ln -s /opt.../share/java/postgresqljdbc42.jar /.../payara/domain1/lib
# chown -R payara:webservd /.../payara/domain1
$ asadmin create-jvm-options "--add-opens=java.base/java.io=ALL-UNNAMED"
# svcadm enable payara
$ asadmin create-virtual-server --hosts ... --networklisteners http-listener-1 ...
$ asadmin create-jdbc-connection-pool
--datasourceclassname org.postgresql.ds.PGConnectionPoolDataSource
--restype javax.sql.ConnectionPoolDataSource
--property portNumber=5432:password=*****user=....:serverName=localhost:databaseName=....
--steadypoolsize=4
--poolresize=32
...-pool
$ asadmin create-jdbc-resource --connectionpoolid ....-pool jdbc/....
$ asadmin deploy --virtualservers ... --name ... target/....-0.1.war

Update to 6.2023.9, failure as above.

I have saved the timer db files and can check the version is someone tells me how.

Workaround: remove all files, create domain from zero, deploy. ie do a fresh install.

[github-actions]

Greetings,
It's been more than 5 days since we requested more information or an update from you on the details of this issue. Could you provide an update soon, please?
We're afraid that if we do not receive an update, we'll have to close this issue due to inactivity.

[oxdrove]

I gave more details the same day. Which of the three steps to reproduce this that I put in the original report do you not understand? At which step are you stuck? Please help me know what help you need.

[kalinchan]

Hi @oxdrove,

Apologies for the bot, I will try and reproduce this issue with the details you provided and will let you know the result.

[oxdrove]

Understood, no problem. I did very little special, hence little detail. If it works for you we can work to find out what might be different. I tried the fresh install .8, update to .9 several times. Unless I have missed something like an upgrade procedure that is supposed to run between versions.

[fturizo]

Greetings @oxdrove,

Following from what @kalinchan explained, it seems that the steps for your reproducer lack the configuration of the EJB timer service to use the new JDBC resource (that points to the PostgreSQL) database, like in this command:

asadmin set configs.config.server-config.ejb-container.ejb-timer-service.timer-datasource=jdbc/...

Once you have configured the timer service to use the correct data source, you should be able to move the domain from a previous version.

[oxdrove]

kalinchan has explained nothing [here] but let us move on. You are saying that if I configure an alternative timer service that the bug in the default timer database does not matter?

I can avoid this bug more easily by recreating the domain (as above) which is quicker than trying to do my bit in reporting a bug.

[fturizo]

@oxdrove, my apologies but it seems that this reproducer is not clear. In the steps you provided for your reproducer, you mentioned a step to configure a JDBC datasource for a PostgreSQL database so we assumed that this database was the one used to store EJB Timer data.

Coming back to your reproducer, this issue has been identified as a side effect of the upgraded version of H2 (2.2.0) introduced in release 6.2023.9:

org.h2.mvstore.MVStoreException: The write format 2 is smaller than the supported format 3 [2.2.220/5]

Sadly, this is unfortunately by design, as the database files from an older version of H2 aren't compatible with newer versions. This is intended as the correct practice is not to depend on the internal H2 database for persistent EJB timers but instead rely on a robust SQL database (like PostgreSQL) to move data around from older releases to new ones.

I'm afraid that to fully migrate from a previous release to 6.2023.9, a domain backup-and-restore is going to fail so it is better to re-create the domain as you mentioned it in your workaround.

[oxdrove]

Thank you for explaining. I will just recreate the domains for now, it is very easy to do as everything important is in the ears/wars or database and set on deployment. A line in the release notes would have helped! It does say the DB was updated but not that it is incompatible, I will look at moving the timers to postgresql, from what you suggest this is simple enough and better.

[oxdrove]

Hello, just to confirm what you already know. Putting the timers in the postgresql database works and the data survived the .9 to .10 update. I had to create the table manually; I don't know if there is some setting to auto create if not existing.

Thank you for your interest and support in this matter.