`rconvert` only called for reconstructed types in vectors; or, vector of reconstructed types is mistyped

[BioTurboNick]

There could be a couple ways to look at this issue, depending on what is intended and what the goals should be.

Here, I'm loading two items from a JLD2 file: a DataFrame, and a vector of DataFrames, using the @load x y macro. Both were saved in an earlier version of DataFrames lacking the metadata field.

The first DataFrame loads as a ReconstructedMutable. The vector of DataFrame errors because it is unable to convert the ReconstructedMutable to DataFrame:

MethodError: Cannot `convert` an object of type JLD2.ReconstructedMutable{Symbol("DataFrames.DataFrame"), (:columns, :colindex), Tuple{Any, DataFrames.Index}} to an object of type DataFrames.DataFrame

The function `convert` exists, but no method is defined for this combination of argument types.

rconvert(::Type, ::JLD2.ReconstructedMutable{Symbol("DataFrames.DataFrame"), (:columns, :colindex), Tuple{Any, DataFrames.Index}}) from custom_serialization.jl:11
lconvert(::JLD2.MappedRepr{DataFrames.DataFrame, JLD2.RelOffset}, ::JLD2.JLDFile{JLD2.MmapIO}, ::Ptr{Nothing}, ::JLD2.RelOffset) from writing_datatypes.jl:339
macro expansion from dataio.jl:125
macro expansion from simdloop.jl:77
read_array!(::Vector{DataFrames.DataFrame}, ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.MappedRepr{DataFrames.DataFrame, JLD2.RelOffset}) from dataio.jl:122
read_array(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation, ::JLD2.DataLayout, ::JLD2.Filters.FilterPipeline{Tuple{}}, ::JLD2.RelOffset, ::Vector{JLD2.ReadAttribute}) from datasets.jl:248
read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.MappedRepr{Any, JLD2.RelOffset}, ::Tuple{JLD2.ReadDataspace, JLD2.RelOffset, JLD2.DataLayout, JLD2.Filters.FilterPipeline{Tuple{}}}, ::Vector{JLD2.ReadAttribute}) from datasets.jl:136
read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.H5Datatype, ::JLD2.DataLayout, ::JLD2.Filters.FilterPipeline{Tuple{}}, ::JLD2.RelOffset, ::Vector{JLD2.ReadAttribute})from datasets.jl:84
load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) from datasets.jl:48

So one leg of this issue is that perhaps the array should be typed to hold the ReconstructedMutable instead, so that this does not error, and everything loads as reconstructed types.

However, it was actually quite nice that I could just define rconvert and get the vector and its elements to be converted automatically on loading without having to define the old DataFrame struct type and changing the loading code, and I wish I could have the same method be called to convert the DataFrame that isn't in the vector:

JLD2.rconvert(::Type{DataFrame}, x::JLD2.ReconstructedMutable{Symbol("DataFrames.DataFrame"), (:columns, :colindex), Tuple{Any, DataFrames.Index}}) = DataFrame(getfield(x, :fields)...)

[JonasIsensee]

For a solution to get your data without errors and also generally more consistenty checkout https://juliaio.github.io/JLD2.jl/dev/advanced/#Explicit-Type-Remapping (and without warnings..)

Conversion with rconvert only is a bit brittle in this case and apparently it is not called when reconstructing just a single element rather than a vector.
If you're interested, you could try figuring out where to add the rconvert by (somewhere in a read_data) function I think.

[BioTurboNick]

Thanks yeah, that's what I ended up doing. I don't want to change a hundred Pluto notebooks when I can just change a dependency 🙂.

If you all don't think it's desired behavior to officially support that, then the real bug is that the vector isn't typed to hold the reconstructed type.

[JonasIsensee]

Thanks yeah, that's what I ended up doing. I don't want to change a hundred Pluto notebooks when I can just change a dependency 🙂.

If you all don't think it's desired behavior to officially support that, then the real bug is that the vector isn't typed to hold the reconstructed type.

Well, I'm fairly convinced that we could add a single rconvert statement in the right place to also get the converted object in that case.

As for the eltype of the Vector: I fear that this is largely unfixable. For mutable objects like arrays and e.g. DataFrames, JLD2 must reconstruct objects in a top-down fashion. Here, the Vector{DataFrame} must be created before loading any of the elements because it might contain circular self-references.
At the time of instantiating Vector{DataFrame} it is in general unknowable whether the actual instances can be reconstructed at all - it only sees the type name DataFrames.DataFrame which it matches to the workspace DataFrames.DataFrame not knowing that the layout changed in the meantime.

Of course, I'm always open for clever ideas, but so far I have been unable to find a good solution for this.

A fun example that makes this even harder: What if it was Vector{AbstractDataFrame} ?
This abstract type has not changed at all.