Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage #22819

Closed
eyurtsev opened this issue Jun 12, 2024 · 1 comment
Closed

Prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage #22819

eyurtsev opened this issue Jun 12, 2024 · 1 comment
Labels
help wanted Good issue for contributors Ɑ: pydantic Related to pydantic or pydantic migration 🤖:refactor A large refactor of a feature(s) or restructuring of many files

Comments

@eyurtsev
Copy link
Collaborator

eyurtsev commented Jun 12, 2024 •
edited
Loading

Privileged issue

  • I am a LangChain maintainer, or was asked directly by a LangChain maintainer to create an issue here.

Issue Content

Context

Currently, LangChain supports Pydantic 2 only through the v1 namespace.

The plan is to transition for Pydantic 2 with release 0.3.0 of LangChain, and drop support for Pydantic 1.

LangChain has around ~1000 pydantic objects across different packages While LangChain uses a number of deprecated features, one of the harder things to update is the usage of a vanilla @root_validator() (which is used ~250 times across the code base).

The goal of this issue is to do as much preliminary work as possible to help prepare for the migration from pydantic v1 to pydantic 2.

To help prepare for the migration, we'll need to refactor each occurrence of a vanilla root_validator() to one of the following 3 variants (depending on what makes sense in the context of the model):

  1. root_validator(pre=True) -- pre initialization validator
  2. root_validator(pre=False, skip_on_failure=True) -- post initialization validator
  3. root_validator(pre=True) AND root_validator(pre=False, skip_on_failure=True) to include both pre initialization and post initialization validation.

Guidelines

  • Pre-initialization is most useful for creating defaults for values, especially when the defaults cannot be supplied per field individually.
  • Post-initialization is most useful for doing more complex validation, especially one that involves multiple fields.

What not to do

  • Do NOT upgrade to model_validator. We're trying to break the work into small chunks that can be done while we're still using Pydantic v1 functionality!
  • Do NOT create field_validators when doing the refactor.

Simple Example

class Foo(BaseModel):
    @root_validator()
    def validate_environment(cls, values: Dict) -> Dict:
        values["api_key"] = get_from_dict_or_env(
            values, "some_api_key", "SOME_API_KEY", default=""
        )

        if values["temperature"] is not None and not 0 <= values["temperature"] <= 1:
            raise ValueError("temperature must be in the range [0.0, 1.0]")
        return values

After refactor

class Foo(BaseModel):
    @root_validator(pre=True)
    def pre_init(cls, values):
        # Logic for setting defaults goes in the pre_init validator.
        # While in some cases, the logic could be pulled into the `Field` definition
        # directly, it's perfectly fine for this refactor to keep the changes minimal
        # and just move the logic into the pre_init validator.
        values["api_key"] = get_from_dict_or_env(
            values, "some_api_key", "SOME_API_KEY", default=""
        )
        return values

    @root_validator(pre=False, skip_on_failure=True)
    def post_init(self, values):
        # Post init validation works with an object that is already initialized
        # so it can access the fields and their values (e.g., temperature).
        # if this logic were part of the pre_init validator, it would raise
        # a KeyError exception since `temperature` does not exist in the values
        # dictionary at that point.
        if values["temperature"] is not None and not 0 <= values["temperature"] <= 1:
            raise ValueError("temperature must be in the range [0.0, 1.0]")
        return values

Example Refactors

Here are some actual for the refactors https://gist.github.com/eyurtsev/be30ddbc54dcdc02f98868eacb24b2a1

If you're feeling especially creative, you could try to use the example refactors, an LLM chain built with an appropriate prompt to attempt to automatically fix this code using LLMs!

Vanilla `root_validator
@dosubot dosubot bot added Ɑ: pydantic Related to pydantic or pydantic migration 🤖:refactor A large refactor of a feature(s) or restructuring of many files labels Jun 12, 2024
@eyurtsev eyurtsev changed the title Help prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage Prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage Jun 12, 2024
@eyurtsev eyurtsev changed the title Prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage Prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage (Not Open For Sign Yet) Jun 12, 2024
@eyurtsev eyurtsev changed the title Prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage (Not Open For Sign Yet) Prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage (Not Open For Sign-up Yet) Jun 12, 2024
@eyurtsev eyurtsev mentioned this issue Jun 12, 2024
Merged
eyurtsev added a commit that referenced this issue Jun 12, 2024
@eyurtsev
community[patch]: Update root_validators embeddings: llamacpp, jina, …
265e650 
…dashscope, mosaicml, huggingface_hub, Toolkits: Connery, ChatModels: PAI_EAS, (#22828)

This PR updates root validators for:

* Embeddings: llamacpp, jina, dashscope, mosaicml, huggingface_hub
* Toolkits: Connery
* ChatModels: PAI_EAS

Following this issue:
#22819
@eyurtsev eyurtsev changed the title Prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage (Not Open For Sign-up Yet) Prepare for pydantic 2 migration by refactoring vanilla @root_validator() usage Jun 12, 2024
@eyurtsev eyurtsev mentioned this issue Jun 12, 2024
Merged
@eyurtsev eyurtsev added the help wanted Good issue for contributors label Jun 12, 2024
@maang-h maang-h mentioned this issue Jun 13, 2024
Merged
eyurtsev added a commit that referenced this issue Jun 20, 2024
@maang-h @eyurtsev
community[patch]: Update root_validators ChatModels: ChatBaichuan, Qi…
bc4cd9c 
…anfanChatEndpoint, MiniMaxChat, ChatSparkLLM, ChatZhipuAI (#22853)

This PR updates root validators for:

- ChatModels: ChatBaichuan, QianfanChatEndpoint, MiniMaxChat,
ChatSparkLLM, ChatZhipuAI

Issues #22819

---------

Co-authored-by: Eugene Yurtsev <[email protected]>
hinthornw pushed a commit that referenced this issue Jun 20, 2024
@eyurtsev @hinthornw
community[patch]: Update root_validators embeddings: llamacpp, jina, …
83da58c 
…dashscope, mosaicml, huggingface_hub, Toolkits: Connery, ChatModels: PAI_EAS, (#22828)

This PR updates root validators for:

* Embeddings: llamacpp, jina, dashscope, mosaicml, huggingface_hub
* Toolkits: Connery
* ChatModels: PAI_EAS

Following this issue:
#22819
hinthornw pushed a commit that referenced this issue Jun 20, 2024
@maang-h @eyurtsev @hinthornw
community[patch]: Update root_validators ChatModels: ChatBaichuan, Qi…
9375705 
…anfanChatEndpoint, MiniMaxChat, ChatSparkLLM, ChatZhipuAI (#22853)

This PR updates root validators for:

- ChatModels: ChatBaichuan, QianfanChatEndpoint, MiniMaxChat,
ChatSparkLLM, ChatZhipuAI

Issues #22819

---------

Co-authored-by: Eugene Yurtsev <[email protected]>
@Zizo-Vi Zizo-Vi mentioned this issue Jun 21, 2024
Merged
@maang-h maang-h mentioned this issue Jun 22, 2024
Closed
@Zizo-Vi Zizo-Vi mentioned this issue Jun 23, 2024
Merged
eyurtsev added a commit that referenced this issue Jun 26, 2024
@Zizo-Vi @eyurtsev
langchain[patch]: update agent and chains modules root_validators (#2…
3bf1d98 
…3256)

Description: update agent and chains modules Pydantic root_validators.
Issue: the issue #22819

---------

Co-authored-by: gongwn1 <[email protected]>
Co-authored-by: Eugene Yurtsev <[email protected]>
Co-authored-by: Eugene Yurtsev <[email protected]>
This was referenced Jul 1, 2024
Merged
Closed
Merged
Merged
eyurtsev added a commit that referenced this issue Jul 2, 2024
@eyurtsev
community[patch]: root validator set explicit pre=False or pre=True (#…
d084172 
…23764)

See issue: #22819
eyurtsev pushed a commit that referenced this issue Jul 2, 2024
@Zizo-Vi
partners: support reading HuggingFace params from env (#23309)
ee5eedf 
Description: 
1. partners/HuggingFace module support reading params from env. Not
adjust langchain_community/.../huggingfaceXX modules since they are
deprecated.
  2. pydantic 2 @root_validator migration.

Issue: #22448 #22819

---------

Co-authored-by: gongwn1 <[email protected]>
@eyurtsev eyurtsev mentioned this issue Jul 2, 2024
Merged
eyurtsev added a commit that referenced this issue Jul 2, 2024
@eyurtsev
community[patch]: update @root_validator in utilities namespace (#23768)
338cef3 
Update all utilities to use `pre=True` or `pre=False`

#22819
@eyurtsev eyurtsev mentioned this issue Jul 2, 2024
Merged
@eyurtsev
Copy link
Collaborator Author

eyurtsev commented Jul 3, 2024

Maybe superseded by: #23841

This was referenced Jul 4, 2024
Closed
Merged
ccurme pushed a commit that referenced this issue Jul 5, 2024
@Zizo-Vi
community: add model_name param valid for GPT4AllEmbeddings (#23867)
b1e90b3 
Description: add model_name param valid for GPT4AllEmbeddings

Issue: #23863 #22819

---------

Co-authored-by: gongwn1 <[email protected]>
@eyurtsev eyurtsev closed this as completed Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Good issue for contributors Ɑ: pydantic Related to pydantic or pydantic migration 🤖:refactor A large refactor of a feature(s) or restructuring of many files
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@eyurtsev