ENH: Allow usage of custom library to serialize with to_json method.

[rmhowe425]

• closes ENH: Allow usage of custom library to serialize with to_json method. #46571
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[jbrockmendel]

My intuition is that passing a custom serializer is going to break a ton of stuff. e.g. would you expect the stdlib json.dumps to work in the existing tests? im pretty sure it wouldnt

[rmhowe425]

My intuition is that passing a custom serializer is going to break a ton of stuff. e.g. would you expect the stdlib json.dumps to work in the existing tests? im pretty sure it wouldnt

Yeah adding functionality to add json.dumps did break a lot of tests. That’s why I also added an additional ‘engine_kwargs’ parameter so that I could pass the appropriate parameters for the custom serializer.

A lot of the I/O methods already support ‘engine_kwargs’ so adding this would follow API standardization.

@jbrockmendel what do you think?

[jbrockmendel]

What is the actual use case here? Do you have a serializer that is more performant? Are there dtypes/scalars that are currently unsupported that you want to be supported?

[mroeschke]

Yeah I don't think completely sidestepping the existing implementation isn't nicely maintainable here.

[rmhowe425]

@mroeschke Do we want to close this PR and the corresponding issue then?

It looks like the Needs Triage label was removed for the issue roughly 2 years ago. It sounds like the acceptance criteria for the issue is no longer needed, or needs to be updated?

[mroeschke]

Yeah let's close this issue pending further discussion. Usually any enhancement request requires more buy in from the core team before a PR