Bug: Regression in BigQuery offline store caused by newer pydantic versions

[galen-ft]

Expected Behavior

Calls to store = FeatureStore(repo_path=repo_path) for a BigQuery offline store should just work.

Current Behavior

Running store = FeatureStore(repo_path=repo_path) for a BigQuery offline store causes:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: 'pydantic_core._pydantic_core.ValidationInfo' object is not subscriptable

This happens on line 108 in bigquery.py#L108.

Likely cause

pydantic>=2.0.0 changed the ValidationInfo object.

The setup.py for feast >= v0.36.0 requires that pydantic>=2.0.0.

Last known successful configuration:

• pydantic==1.10.16
• feast v.0.35.0 - the setup.py for feast v0.35.0 allows for pydantic>=1,<2.

Steps to reproduce

Example feature_store.yaml:

project: my_feature_repo
registry: gs://.../registry.db
# The provider AWS used for the online store.
# Mixing AWS and GCP should be okay provided that offline store type is specified.
provider: aws
offline_store:
    type: bigquery
    billing_project_id: some-project-id
    dataset: latest
    gcs_staging_location: gs://...
    location: EU
    project_id: some-project-id
online_store:
  type: redis
  connection_string: ...

entity_key_serialization_version: 2

import os
# Credentials for GCP
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="GCP_keyfile.json"
from feast import FeatureStore, RepoConfig


if __name__ == "__main__":
    # repo_path must contain the GCP_keyfile.json and feature_store.yaml files
    repo_path = "./"
    assert os.path.exists(repo_path)
    # Create a feature store object
    store = FeatureStore(repo_path=repo_path) # <--- Error occurs here

Specifications

• Version: feast v0.38.0
• Platform: x86-64 Ubuntu 22.04.3 LTS as well as arm64 Mac OS Sonoma 14.4.1
• Subsystem:

Possible Solution

On first try, using values.data["project_id"] instead of values["project_id"] in the pydantic.field_validator for billing_project_id in bigquery.py#L108 should work:

@field_validator("billing_project_id")
    def project_id_exists(cls, v, values, **kwargs):
        # if v and not values["project_id"]:
        if v and not values.data["project_id"]:
            raise ValueError(
                "please specify project_id if billing_project_id is specified"
            )
        return v

More testing is needed before I can make a PR.

[charlieviettq]

the same issue with feast v0.39.0

[tokoko]

@galen-ft Let me know if you're planning to open a PR for this

[jgesche]

Same issue with feast v0.40.1. I have opened a PR. @galen-ft I just used your proposed solution for the issue. Have you made any progress on your end, or do you have any other ideas on how to fix it better or differently?

[galen-ft]

Hi @tokoko and @jgesche, and apologies for my late reply. I've been quite busy during the summer so I haven't had the chance to investigate a better fix. I'll try to do it as soon as I get some time.