[Core feature] Support both pydantic 1.x and 2.x. TypeEngine BaseModelTransformer

[Future-Outlier]

Motivation: Why do you think this is important?

Now, we can only support 1.x version in pydantic.
However, some users might want to use 2.x version.

Goal: What should the final outcome look like, ideally?

We should support both version with 2 type transformer.
Maybe we need to implement some kind of routing mechanism to recognize which transformer we should use.

Describe alternatives you've considered

When using pydantic version >= 2, only use pydantic.v1 BaseModel.

Propose: Link/Inline OR Additional context

• More details about pydantic: https://docs.pydantic.dev/latest/migration/#defining-custom-types
• Related PR: Support pydantic plugin in 2.xx version flytekit#2217

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[dosubot]

Hey @Future-Outlier! 🚀 Great to see you diving into another interesting challenge with us. How've you been since our last tech adventure? 🛠️

To support both 1.x and 2.x versions of Pydantic in the Flyte repository, we can implement a version-aware mechanism. This mechanism will determine the installed version of Pydantic at runtime and use the appropriate transformers accordingly. Here's a high-level overview of the steps involved:

1. Specify Pydantic as a Dependency with Version Ranges: In the pyproject.toml file, specify Pydantic in the dependencies section with a version range that includes both 1.x and 2.x versions. This can be done using the || operator. For example:

dependencies = [
    'googleapis-common-protos',
    'protoc-gen-openapiv2',
    'protobuf>=4.21.1,<5.0.0',
    'pydantic>=1.0,<3.0', # This line supports both 1.x and 2.x versions
]

2. Implement a Version-Aware Import Mechanism: In the Python code where Pydantic is used, implement a mechanism to dynamically import and use the correct version of Pydantic based on its installed version. Here's an example:

import pydantic

# Determine the major version of Pydantic
pydantic_version = int(pydantic.__version__.split('.')[0])

if pydantic_version == 1:
    # Define or import transformers and functionality specific to Pydantic 1.x
    from .transformers_v1 import MyTransformer
elif pydantic_version == 2:
    # Define or import transformers and functionality specific to Pydantic 2.x
    from .transformers_v2 import MyTransformer
else:
    raise ImportError("Unsupported version of Pydantic")

# Use MyTransformer as needed in your application

3. Testing: Ensure thorough testing with both versions of Pydantic. You may want to set up your continuous integration (CI) pipeline to run tests under both Pydantic 1.x and 2.x environments to catch any compatibility issues early.

4. Documentation: Update your project's documentation to inform users about the support for both Pydantic 1.x and 2.x versions, including any differences in behavior or functionality they should be aware of when using different versions.

Unfortunately, I couldn't find specific details about the current method of handling Pydantic in the Flyte codebase. If you could provide more information about this, it would be helpful in tailoring a more specific solution.

Sources

• flyteidl/pyproject.toml

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Future-Outlier]

update: we decided to deprecate pydantic v1 plugin.