File saved in Julia 1.10.0-beta3 cannot be loaded in Julia 1.10.0-rc1 with Random.Xoshiro

[liuyxpp]

The JLD2 file is a Lux.jl model (ps + st) with many Random.Xoshiro instances, which is saved in Julia 1.10.0-beta3. When I tried to load it in Julia 1.10.0-rc1, the error mesasge is

julia> trained_models, trained_metrics = jldopen(jldfile) do f
               return f["trained_models"], f["trained_metrics"]
           end
┌ Warning: saved type Random.Xoshiro is missing field s4 in workspace type; reconstructing
└ @ JLD2 /home/lyx/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:207
ERROR: MethodError: Cannot `convert` an object of type JLD2.ReconstructedStatic{Symbol("Random.Xoshiro"), (:s0, :s1, :s2, :s3), NTuple{4, UInt64}} to an object of type Random.Xoshiro

Closest candidates are:
  convert(::Type{T}, ::T) where T
   @ Base Base.jl:84
  Random.Xoshiro(::Any)
   @ Random ~/.julia/juliaup/julia-1.10.0-rc1+0.x64.linux.gnu/share/julia/stdlib/v1.10/Random/src/Xoshiro.jl:55

Stacktrace:
  [1] rconvert(T::Type, x::JLD2.ReconstructedStatic{Symbol("Random.Xoshiro"), (:s0, :s1, :s2, :s3), NTuple{4, UInt64}})
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/custom_serialization.jl:9
  [2] jlconvert
    @ ~/.julia/packages/JLD2/u57Vt/src/data/writing_datatypes.jl:315 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:766 [inlined]
  [4] jlconvert(::JLD2.ReadRepresentation{…}, f::JLD2.JLDFile{…}, ptr::Ptr{…}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:685
  [5] macro expansion
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:685 [inlined]
  [6] jlconvert(::JLD2.ReadRepresentation{…}, f::JLD2.JLDFile{…}, ptr::Ptr{…}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:685
  [7] macro expansion
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:685 [inlined]
  [8] jlconvert(::JLD2.ReadRepresentation{…}, f::JLD2.JLDFile{…}, ptr::Ptr{…}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:685
  [9] macro expansion
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:685 [inlined]
 [10] jlconvert(::JLD2.ReadRepresentation{…}, f::JLD2.JLDFile{…}, ptr::Ptr{…}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:685
 [11] macro expansion
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:685 [inlined]
 [12] jlconvert
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:685 [inlined]
 [13] read_scalar(f::JLD2.JLDFile{…}, rr::JLD2.ReadRepresentation{…}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/dataio.jl:37
 [14] read_data(f::JLD2.JLDFile{…}, rr::Any, read_dataspace::Tuple{…}, attributes::Vector{…})
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/datasets.jl:238
 [15] read_data(f::JLD2.JLDFile{…}, dataspace::JLD2.ReadDataspace, datatype_class::UInt8, datatype_offset::Int64, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{…})
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/datasets.jl:194
 [16] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/datasets.jl:125
 [17] jlconvert
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/writing_datatypes.jl:314 [inlined]
 [18] macro expansion
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/dataio.jl:70 [inlined]
 [19] macro expansion
    @ JLD2 ./simdloop.jl:77 [inlined]
 [20] read_array!(v::Vector{Any}, f::JLD2.JLDFile{JLD2.MmapIO}, rr::JLD2.ReadRepresentation{Any, JLD2.RelOffset})
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/dataio.jl:68
 [21] read_array(f::JLD2.JLDFile{…}, dataspace::JLD2.ReadDataspace, rr::JLD2.ReadRepresentation{…}, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{…})
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/datasets.jl:410
 [22] read_data(f::JLD2.JLDFile{…}, rr::JLD2.ReadRepresentation{…}, read_dataspace::Tuple{…}, attributes::Vector{…})
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/datasets.jl:276
 [23] macro expansion
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/datasets.jl:224 [inlined]
 [24] macro expansion
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/datatypes.jl:105 [inlined]
 [25] read_data(f::JLD2.JLDFile{…}, dataspace::JLD2.ReadDataspace, datatype_class::UInt8, datatype_offset::Int64, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{…})
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/datasets.jl:211
 [26] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/datasets.jl:125
 [27] getindex(g::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, name::String)
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/groups.jl:109
 [28] getindex
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/JLD2.jl:483 [inlined]
 [29] (::var"#7#8")(f::JLD2.JLDFile{JLD2.MmapIO})
    @ Main ./REPL[6]:2
 [30] jldopen(f::Function, args::String; kws::@Kwargs{})
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/loadsave.jl:4
 [31] jldopen(f::Function, args::String)
    @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/loadsave.jl:1
 [32] top-level scope
    @ REPL[6]:1
Some type information was truncated. Use `show(err)` to see complete types.

[liuyxpp]

Just test that the file can be loaded in Julia 1.9.3. And resave it to jld2 file which still can not be loaded in Julia 1.10.0-rc1. There should be some tricky change of the Xoshiro in Julia 1.10.0-rc1.

[JonasIsensee]

I think this is due to

JuliaLang/julia#49110

[JonasIsensee]

The error message already suggests what to define to fix it, so I think I will close this for now.

[liuyxpp]

The error message already suggests what to define to fix it, so I think I will close this for now.

Can you explicitly explain how to do that?

[JonasIsensee]

You can define:

# Old xoshiro had 3 fields , new has 4 but has a constructor given only the first three fields.
JLD2.rconvert(::Type{Xoshiro}, nt) = Xoshiro(nt...)

In your case that should already be enough.
In my test case, I generated the file as jldsave("rng.jld2"; rng=Xoshiro(42) on julia 1.9

Then

julia> load("rng.jld2")
┌ Warning: saved type Xoshiro is missing field s4 in workspace type; reconstructing
└ @ JLD2 ~/.julia/dev/JLD2.jl/src/data/reconstructing_datatypes.jl:207
Dict{String, Any} with 1 entry:
  "rng" => Reconstruct@Xoshiro((0xa379de7eeeb2a4e8, 0x953dccb6b532b3af, 0xf597b8ff8cfd652a, 0xccd7337c571680d1))

julia> load("rng.jld2"; typemap=Dict("Random.Xoshiro"=>JLD2.Upgrade(Xoshiro)))
Dict{String, Any} with 1 entry:
  "rng" => Xoshiro(0xa379de7eeeb2a4e8, 0x953dccb6b532b3af, 0xf597b8ff8cfd652a, 0xccd7337c571680d1, 0xc90c4a0730db3f7e)