Unable to feast apply when using SQL Registry

[chasleslr]

Expected Behavior

Using a SQL registry, we should be able to run feast apply more than once for a given project, so that feast can identify the diff between the existing registry and project.

Current Behavior

When using a SQL registry, running feast apply for the first time works. However, when running it again, it returns the following error:

Traceback (most recent call last):
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/bin/feast", line 8, in <module>
    sys.exit(cli())
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/cli.py", line 492, in apply_total_command
    apply_total(repo_config, repo, skip_source_validation)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/usage.py", line 274, in wrapper
    return func(*args, **kwargs)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/repo_operations.py", line 305, in apply_total
    apply_total_with_repo_instance(
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/repo_operations.py", line 265, in apply_total_with_repo_instance
    registry_diff, infra_diff, new_infra = store.plan(repo)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/usage.py", line 285, in wrapper
    raise exc.with_traceback(traceback)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/usage.py", line 274, in wrapper
    return func(*args, **kwargs)
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/feature_store.py", line 708, in plan
    current_infra_proto = self._registry.proto().infra.__deepcopy__()
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/infra/registry_stores/sql.py", line 666, in proto
    r.infra.CopyFrom(self.get_infra(project).to_proto())
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/infra/registry_stores/sql.py", line 558, in get_infra
    return self._get_object(
  File "/Users/charleslariviere/.local/share/virtualenvs/data-feature-store-QM-Cj6mZ/lib/python3.8/site-packages/feast/infra/registry_stores/sql.py", line 774, in _get_object
    raise not_found_exception(name, project)
TypeError: 'NoneType' object is not callable

This error is raised because of what looks like a bug in SqlRegistry.get_infra(). Indeed, SqlRegistry.get_infra() calls SqlRegistry._get_object() but passing None for the not_found_exception argument.
feast/infra/registry_stores/sql.py

    ...
    def get_infra(self, project: str, allow_cache: bool = False) -> Infra:
        return self._get_object(
            managed_infra,
            "infra_obj",
            project,
            InfraProto,
            Infra,
            "infra_name",
            "infra_proto",
            >>>> None, <<<<
        )
    ...
    def _get_object(
        self,
        table,
        name,
        project,
        proto_class,
        python_class,
        id_field_name,
        proto_field_name,
        >>> not_found_exception <<<,
    ):
        self._maybe_init_project_metadata(project)

        with self.engine.connect() as conn:
            stmt = select(table).where(
                getattr(table.c, id_field_name) == name, table.c.project_id == project
            )
            row = conn.execute(stmt).first()
            if row:
                _proto = proto_class.FromString(row[proto_field_name])
                return python_class.from_proto(_proto)
        >>> raise not_found_exception(name, project) <<<

Since no rows are returned for the managed_infra table, _get_object attempts to raise an error with the not_found_exception callable provided, but since None was provided, it is unable to raise the error correctly.

This seems to be hiding the actual error, which is that no rows are returned from the managed_infra table, which I can confirm is indeed empty in the SQL backend. I've provided my feature_store.yaml configuration below -- is the expectation that something should be created in the managed_infra table?

Steps to reproduce

feature_store.yaml

project: feature_store
provider: local
registry:
    registry_type: sql
    path: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DATABASE}
offline_store:
    type: snowflake.offline
    account: ${SNOWFLAKE_ACCOUNT}
    user: ${SNOWFLAKE_USER}
    password: ${SNOWFLAKE_PASSWORD}
    role: ${SNOWFLAKE_ROLE}
    warehouse: ${SNOWFLAKE_WAREHOUSE}
    database: ${SNOWFLAKE_DATABASE}
    schema: ${SNOWFLAKE_SCHEMA}
online_store:
    type: redis
    redis_type: redis
    connection_string: "${REDIS_HOST}:${REDIS_PORT},password=${REDIS_PASSWORD}"

Run the following commands:

feast apply # works
feast apply # raises the error above

Specifications

• Version: 0.22.1
• Platform: macOS
• Subsystem: 12.4

Possible Solution

[adchia]

cc @felixwang9817 not sure if this is expected that the Infra objects aren't created for Redis?

[felixwang9817]

hm @adchia seems like no since Achal has a fix out

[adchia]

@felixwang9817 right but this is created from feast plan right. So might have just been expected that infra objects aren't created since we never finished that?

[felixwang9817]

@adchia good point, I'll take another look at this!