Java 21 deserialization of DateTime

[holgerfriedrich]

I just gave running openHAB on Java 21 another test but there seems to be some issue with date formats when deserializing timestamps on Java 21.

These errors show up when restarting openHAB:

2024-04-06 09:05:10.272 [ERROR] [re.storage.json.internal.JsonStorage] - Couldn't deserialize value 'org.openhab.core.storage.json.internal.StorageEntry@4a2246a9'. Root cause is: Failed parsing 'Apr 6, 2024, 8:53:04 AM' as Date; at path $.sessions[0].createdTime

09:01:06.164 [ERROR] [rnal.common.AbstractInvocationHandler] - An error occurred while calling method 'QueryablePersistenceService.query()' on 'org.openhab.persistence.mapdb.internal.MapDbPersistenceService@39aca3e6': Failed parsing 'Apr 6, 2024, 8:58:09 AM' as Date; at path $.timestamp
com.google.gson.JsonSyntaxException: Failed parsing 'Apr 6, 2024, 8:58:09 AM' as Date; at path $.timestamp
	at com.google.gson.internal.bind.DateTypeAdapter.deserializeToDate(DateTypeAdapter.java:90) ~[bundleFile:?]
	at com.google.gson.internal.bind.DateTypeAdapter.read(DateTypeAdapter.java:75) ~[bundleFile:?]
	at com.google.gson.internal.bind.DateTypeAdapter.read(DateTypeAdapter.java:46) ~[bundleFile:?]
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.readIntoField(ReflectiveTypeAdapterFactory.java:212) ~[bundleFile:?]
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$FieldReflectionAdapter.readField(ReflectiveTypeAdapterFactory.java:433) ~[bundleFile:?]
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:393) ~[bundleFile:?]
	at com.google.gson.Gson.fromJson(Gson.java:1227) ~[bundleFile:?]
	at com.google.gson.Gson.fromJson(Gson.java:1137) ~[bundleFile:?]
	at com.google.gson.Gson.fromJson(Gson.java:1047) ~[bundleFile:?]
	at com.google.gson.Gson.fromJson(Gson.java:982) ~[bundleFile:?]
	at org.openhab.persistence.mapdb.internal.MapDbPersistenceService.deserialize(MapDbPersistenceService.java:212) ~[?:?]
	at org.openhab.persistence.mapdb.internal.MapDbPersistenceService.query(MapDbPersistenceService.java:202) ~[?:?]
	at jdk.internal.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.openhab.core.internal.common.AbstractInvocationHandler.invokeDirect(AbstractInvocationHandler.java:147) [bundleFile:?]
	at org.openhab.core.internal.common.Invocation.call(Invocation.java:52) [bundleFile:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.text.ParseException: Failed to parse date ["Apr 6, 2024, 8:58:09 AM"]: Invalid number: Apr 
	at com.google.gson.internal.bind.util.ISO8601Utils.parse(ISO8601Utils.java:279) ~[bundleFile:?]
	at com.google.gson.internal.bind.DateTypeAdapter.deserializeToDate(DateTypeAdapter.java:88) ~[bundleFile:?]
	... 20 more
Caused by: java.lang.NumberFormatException: Invalid number: Apr 
	at com.google.gson.internal.bind.util.ISO8601Utils.parseInt(ISO8601Utils.java:316) ~[bundleFile:?]
	at com.google.gson.internal.bind.util.ISO8601Utils.parse(ISO8601Utils.java:133) ~[bundleFile:?]
	at com.google.gson.internal.bind.DateTypeAdapter.deserializeToDate(DateTypeAdapter.java:88) ~[bundleFile:?]
	... 20 more

Originally posted by @wborn in openhab/openhab-distro#1640 (comment)

[holgerfriedrich]

Could be caused by an update of CLDR data introduced in Java 20:
https://inside.java/2024/03/29/quality-heads-up/

The date format has changed, notably the narrow non-breakable space.

@wborn was the data to be deserialized created with Java 17 and failure on Java 21?
If yes, could you provide the parameter java.locale.providers=COMPAT to the JVM?

[holgerfriedrich]

I could reproduce the problem when switching from a fresh Java 21 installation back to Java 17. User database users.json could not be parsed.
Looking at the file, I see a UTF encoded character in the date:

[holgerfriedrich]

I have opened #4185 which fixes the loss of configuration when switching JDK versions.
It could be a viable approach to maintain the old DateTime format for serialization.
However, I have the feeling we need to apply this to all instances obtained from GsonBuilder.
There are a lot of occurrences in add-ons, so maybe we could use a custom implementation of GsonBuilder which sets the date format in the ctor.
WDYT?

[holgerfriedrich]

PRs for core #4185 and add-ons openhab/openhab-addons#16657 are ready for review.

[lolodomo]

I remember I got a similar issue when switching back to 4.1.1 from 4.2, keeping my JSON DB files.
Are you sure you have not already enabled new Java 21 format by default and by error?
I could try to reproduce the issue if necessary.

[holgerfriedrich]

@lolodomo thanks.
Pls. see the new tests in https://github.com/openhab/openhab-core/pull/4185/files#diff-35e8967f62965e1493090f9f94de7fec11e573479b32c3bd7491375edab44414

What seems to happen is that Date is serialized differently in Java 17 and Java 21 and cannot be deserialized in the other one.
The difference is visible when pasting the UTF8 string into a hex editor. The tests check for the serialization of 01-Jan-1980 without setting the format and with explicitly setting the format.

You can switch the Java versions and check that serialized data cannot be read. openhabian has an option to install Java 21.

[lolodomo]

My use case was when switching between 4.1 and 4.2 but remaining in Java 17. That may be a coincidence or a different problem with date deserialisation, that is just strange it just appeared at the same time you added java 21 stuff. I am just afraid you enabled the new format in 4.2 even with java 17 but that is just an hypothesis. I will have to reproduce the issue first.

[wborn]

Is there still something to do here or can it be closed?

[holgerfriedrich]

I did a quick test on S4052, issues with login and with mapdb seem do be gone.

[wborn]

Great! Then I'll also give it a try again soon. 😀