diff --git a/README.md b/README.md index 182637018f..d57247c60b 100644 --- a/README.md +++ b/README.md @@ -173,12 +173,12 @@ The list below contains the functionality that contributors are planning to deve * [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb) * [x] [Redis](https://docs.feast.dev/reference/online-stores/redis) * [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore) + * [x] [Bigtable](https://docs.feast.dev/reference/online-stores/bigtable) * [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite) * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure) * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres) + * [x] [Cassandra / AstraDB (contrib plugin)](https://docs.feast.dev/reference/online-stores/cassandra) * [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store) - * [x] [Cassandra / AstraDB](https://docs.feast.dev/reference/online-stores/cassandra) - * [ ] Bigtable (in progress) * **Feature Engineering** * [x] On-demand Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#)) * [x] Streaming Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1UzEyETHUaGpn0ap4G82DHluiCj7zEbrQLkJJkKSv4e8/edit)) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 2b58fb277e..cb4a2664cd 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -92,6 +92,7 @@ * [Redis](reference/online-stores/redis.md) * [Datastore](reference/online-stores/datastore.md) * [DynamoDB](reference/online-stores/dynamodb.md) + * [Bigtable](reference/online-stores/bigtable.md) * [PostgreSQL (contrib)](reference/online-stores/postgres.md) * [Cassandra + Astra DB (contrib)](reference/online-stores/cassandra.md) * [MySQL (contrib)](reference/online-stores/mysql.md) diff --git a/docs/getting-started/third-party-integrations.md b/docs/getting-started/third-party-integrations.md index 8e6a600aa0..ca974880ed 100644 --- a/docs/getting-started/third-party-integrations.md +++ b/docs/getting-started/third-party-integrations.md @@ -11,7 +11,7 @@ Don't see your offline store or online store of choice here? Check out our guide ## Integrations -See [Functionality and Roadmap](../../#-functionality-and-roadmap) +See [Functionality and Roadmap](../roadmap.md) ## Standards diff --git a/docs/reference/online-stores/README.md b/docs/reference/online-stores/README.md index 2ade4f0de9..e46fc28d16 100644 --- a/docs/reference/online-stores/README.md +++ b/docs/reference/online-stores/README.md @@ -26,6 +26,10 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli [dynamodb.md](dynamodb.md) {% endcontent-ref %} +{% content-ref url="bigtable.md" %} +[bigtable.md](mysql.md) +{% endcontent-ref %} + {% content-ref url="postgres.md" %} [postgres.md](postgres.md) {% endcontent-ref %} diff --git a/docs/reference/online-stores/bigtable.md b/docs/reference/online-stores/bigtable.md new file mode 100644 index 0000000000..0d6e7cfb13 --- /dev/null +++ b/docs/reference/online-stores/bigtable.md @@ -0,0 +1,56 @@ +# Bigtable online store + +## Description + +The [Bigtable](https://cloud.google.com/bigtable) online store provides support for +materializing feature values into Cloud Bigtable. The data model used to store feature +values in Bigtable is described in more detail +[here](../../specs/online_store_format.md#google-bigtable-online-store-format). + +## Getting started + +In order to use this online store, you'll need to run `pip install 'feast[gcp]'`. You +can then get started with the command `feast init REPO_NAME -t gcp`. + +## Example + +{% code title="feature_store.yaml" %} +```yaml +project: my_feature_repo +registry: data/registry.db +provider: gcp +online_store: + type: bigtable + project_id: my_gcp_project + instance: my_bigtable_instance +``` +{% endcode %} + +The full set of configuration options is available in +[BigtableOnlineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.bigtable.BigtableOnlineStoreConfig). + +## Functionality Matrix + +The set of functionality supported by online stores is described in detail [here](overview.md#functionality). +Below is a matrix indicating which functionality is supported by the Bigtable online store. + +| | Bigtable | +|-----------------------------------------------------------|----------| +| write feature values to the online store | yes | +| read feature values from the online store | yes | +| update infrastructure (e.g. tables) in the online store | yes | +| teardown infrastructure (e.g. tables) in the online store | yes | +| generate a plan of infrastructure changes | no | +| support for on-demand transforms | yes | +| readable by Python SDK | yes | +| readable by Java | no | +| readable by Go | no | +| support for entityless feature views | yes | +| support for concurrent writing to the same key | yes | +| support for ttl (time to live) at retrieval | no | +| support for deleting expired data | no | +| collocated by feature view | yes | +| collocated by feature service | no | +| collocated by entity key | yes | + +To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix). diff --git a/docs/roadmap.md b/docs/roadmap.md index 30f4317054..cea646a8f5 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -31,12 +31,12 @@ The list below contains the functionality that contributors are planning to deve * [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb) * [x] [Redis](https://docs.feast.dev/reference/online-stores/redis) * [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore) + * [x] [Bigtable](https://docs.feast.dev/reference/online-stores/bigtable) * [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite) * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure) * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres) + * [x] [Cassandra / AstraDB (contrib plugin)](https://docs.feast.dev/reference/online-stores/cassandra) * [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store) - * [x] [Cassandra / AstraDB](https://docs.feast.dev/reference/online-stores/cassandra) - * [ ] Bigtable (in progress) * **Feature Engineering** * [x] On-demand Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#)) * [x] Streaming Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1UzEyETHUaGpn0ap4G82DHluiCj7zEbrQLkJJkKSv4e8/edit)) diff --git a/docs/specs/online_store_format.md b/docs/specs/online_store_format.md index 5c3c545c8d..4022138e31 100644 --- a/docs/specs/online_store_format.md +++ b/docs/specs/online_store_format.md @@ -92,6 +92,29 @@ Other types of entity keys are not supported in this version of the specificatio ![Datastore Online Example](datastore_online_example.png) +## Google Bigtable Online Store Format + +[Bigtable storage model](https://cloud.google.com/bigtable/docs/overview#storage-model) +consists of massively scalable tables, with each row keyed by a "row key". The rows in a +table are stored lexicographically sorted by this row key. + +We use the following structure to store feature data in Bigtable: + +* All feature data for an entity or a specific group of entities is stored in the same + table. The table name is derived by concatenating the lexicographically sorted names + of entities. +* This implementation only uses one column family per table, named `features`. +* Each row key is created by concatenating a hash derived from the specific entity keys + and the name of the feature view. Each row only stores feature values for a specific + feature view. This arrangement also means that feature values for a given group of + entities are colocated. +* The columns used in each row are named after the features in the feature view. + Bigtable is perfectly content being sparsely populated. +* By default, we store 1 historical value of each feature value. This can be configured + using the `max_versions` setting in `BigtableOnlineStoreConfig`. This implementation + of the online store does not have the ability to revert any given value to its old + self. To use the historical version, you'll have to use custom code. + ## Cassandra/Astra DB Online Store Format ### Overview @@ -250,4 +273,4 @@ message BoolList { repeated bool val = 1; } -``` \ No newline at end of file +``` diff --git a/sdk/python/feast/infra/online_stores/bigtable.py b/sdk/python/feast/infra/online_stores/bigtable.py new file mode 100644 index 0000000000..e08bc44bdb --- /dev/null +++ b/sdk/python/feast/infra/online_stores/bigtable.py @@ -0,0 +1,338 @@ +import hashlib +import logging +from concurrent import futures +from datetime import datetime +from typing import Any, Callable, Dict, List, Optional, Sequence, Set, Tuple + +import google +from google.cloud import bigtable +from google.cloud.bigtable import row_filters +from pydantic import StrictStr +from pydantic.typing import Literal + +from feast import Entity, FeatureView, utils +from feast.feature_view import DUMMY_ENTITY_NAME +from feast.infra.online_stores.helpers import compute_entity_id +from feast.infra.online_stores.online_store import OnlineStore +from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto +from feast.protos.feast.types.Value_pb2 import Value as ValueProto +from feast.repo_config import FeastConfigBaseModel, RepoConfig +from feast.usage import log_exceptions_and_usage + +logger = logging.getLogger(__name__) + +# Number of mutations per Bigtable write operation we're aiming for. The official max is +# 100K; we're being conservative. +MUTATIONS_PER_OP = 50_000 +# The Bigtable client library limits the connection pool size to 10. This imposes a +# limitation to the concurrency we can get using a thread pool in each worker. +BIGTABLE_CLIENT_CONNECTION_POOL_SIZE = 10 + + +class BigtableOnlineStoreConfig(FeastConfigBaseModel): + """Online store config for GCP Bigtable""" + + type: Literal["bigtable"] = "bigtable" + """Online store typee selector""" + + project_id: Optional[StrictStr] = None + """(optional) GCP Project ID""" + + instance: StrictStr + """The Bigtable instance's ID""" + + max_versions: int = 2 + """The number of historical versions of data that will be kept around.""" + + +class BigtableOnlineStore(OnlineStore): + _client: Optional[bigtable.Client] = None + + feature_column_family: str = "features" + + @log_exceptions_and_usage(online_store="bigtable") + def online_read( + self, + config: RepoConfig, + table: FeatureView, + entity_keys: List[EntityKeyProto], + requested_features: Optional[List[str]] = None, + ) -> List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]]: + # Potential performance improvement opportunity described in + # https://github.com/feast-dev/feast/issues/3259 + feature_view = table + bt_table_name = self._get_table_name(config=config, feature_view=feature_view) + + client = self._get_client(online_config=config.online_store) + bt_instance = client.instance(instance_id=config.online_store.instance) + bt_table = bt_instance.table(bt_table_name) + row_keys = [ + self._compute_row_key( + entity_key=entity_key, + feature_view_name=feature_view.name, + config=config, + ) + for entity_key in entity_keys + ] + + row_set = bigtable.row_set.RowSet() + for row_key in row_keys: + row_set.add_row_key(row_key) + rows = bt_table.read_rows( + row_set=row_set, + filter_=( + row_filters.ColumnQualifierRegexFilter( + f"^({'|'.join(requested_features)}|event_ts)$".encode() + ) + if requested_features + else None + ), + ) + + # The BigTable client library only returns rows for keys that are found. This + # means that it's our responsibility to match the returned rows to the original + # `row_keys` and make sure that we're returning a list of the same length as + # `entity_keys`. + bt_rows_dict: Dict[bytes, bigtable.row.PartialRowData] = { + row.row_key: row for row in rows + } + return [self._process_bt_row(bt_rows_dict.get(row_key)) for row_key in row_keys] + + def _process_bt_row( + self, row: Optional[bigtable.row.PartialRowData] + ) -> Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]: + res = {} + + if row is None: + return (None, None) + + row_values = row.cells[self.feature_column_family] + event_ts = datetime.fromisoformat(row_values.pop(b"event_ts")[0].value.decode()) + for feature_name, feature_values in row_values.items(): + # We only want to retrieve the latest value for each feature + feature_value = feature_values[0] + val = ValueProto() + val.ParseFromString(feature_value.value) + res[feature_name.decode()] = val + + return (event_ts, res) + + @log_exceptions_and_usage(online_store="bigtable") + def online_write_batch( + self, + config: RepoConfig, + table: FeatureView, + data: List[ + Tuple[EntityKeyProto, Dict[str, ValueProto], datetime, Optional[datetime]] + ], + progress: Optional[Callable[[int], Any]], + ) -> None: + feature_view = table + bt_table_name = self._get_table_name(config=config, feature_view=feature_view) + + client = self._get_client(online_config=config.online_store) + bt_instance = client.instance(instance_id=config.online_store.instance) + bt_table = bt_instance.table(bt_table_name) + + # `columns_per_row` is used to calculate the number of rows we are allowed to + # mutate in one request. + columns_per_row = len(feature_view.features) + 1 # extra for event timestamp + rows_per_write = MUTATIONS_PER_OP // columns_per_row + + with futures.ThreadPoolExecutor( + max_workers=BIGTABLE_CLIENT_CONNECTION_POOL_SIZE + ) as executor: + fs = [] + while data: + rows_to_write, data = data[:rows_per_write], data[rows_per_write:] + fs.append( + executor.submit( + self._write_rows_to_bt, + rows_to_write=rows_to_write, + bt_table=bt_table, + feature_view_name=feature_view.name, + config=config, + progress=progress, + ) + ) + done_tasks, not_done_tasks = futures.wait(fs) + for task in done_tasks: + # If a task raised an exception, this will raise it here as well + task.result() + if not_done_tasks: + raise RuntimeError( + f"Not all batches were written to Bigtable: {not_done_tasks}" + ) + + def _write_rows_to_bt( + self, + rows_to_write: List[ + Tuple[EntityKeyProto, Dict[str, ValueProto], datetime, Optional[datetime]] + ], + bt_table: bigtable.table.Table, + feature_view_name: str, + config: RepoConfig, + progress: Optional[Callable[[int], Any]], + ): + rows = [] + for entity_key, features, timestamp, created_ts in rows_to_write: + bt_row = bt_table.direct_row( + self._compute_row_key( + entity_key=entity_key, + feature_view_name=feature_view_name, + config=config, + ) + ) + + for feature_name, feature_value in features.items(): + bt_row.set_cell( + self.feature_column_family, + feature_name.encode(), + feature_value.SerializeToString(), + ) + bt_row.set_cell( + self.feature_column_family, + b"event_ts", + utils.make_tzaware(timestamp).isoformat().encode(), + ) + rows.append(bt_row) + bt_table.mutate_rows(rows) + + if progress: + progress(len(rows)) + + def _compute_row_key( + self, entity_key: EntityKeyProto, feature_view_name: str, config: RepoConfig + ) -> bytes: + entity_id = compute_entity_id( + entity_key, + entity_key_serialization_version=config.entity_key_serialization_version, + ) + # Even though `entity_id` uniquely identifies an entity, we use the same table + # for multiple feature_views with the same set of entities. To uniquely identify + # the row for a feature_view, we suffix the name of the feature_view itself. + # This also ensures that features for entities from various feature_views are + # colocated. + return f"{entity_id}#{feature_view_name}".encode() + + def update( + self, + config: RepoConfig, + tables_to_delete: Sequence[FeatureView], + tables_to_keep: Sequence[FeatureView], + entities_to_delete: Sequence[Entity], + entities_to_keep: Sequence[Entity], + partial: bool, + ): + """Creates the appropriate tables and column family in Bigtable. + + We use a dedicated table for each entity combination. For example, if a + FeatureView uses the entities `shop` and `customer`, the resulting table would + be called `customer-shop` (entities are sorted lexicographically first). + """ + online_config = config.online_store + assert isinstance(online_config, BigtableOnlineStoreConfig) + client = self._get_client(online_config, admin=True) + bt_instance = client.instance(instance_id=online_config.instance) + max_versions_gc_rule = bigtable.column_family.MaxVersionsGCRule( + online_config.max_versions + ) + # The word "table" in the arguments refers to feature_views (this is for legacy + # reasons). To reduce confusion with bigtable tables, we use alternate variable + # names + feature_views_to_keep = tables_to_keep + feature_views_to_delete = tables_to_delete + # Multiple feature views can share the same tables. So just because a feature + # view has been deleted does not mean that we can just delete the table. We map + # feature views to Bigtable table names and figure out which ones to create + # and/or delete. + bt_tables_to_keep: Set[str] = { + self._get_table_name(config=config, feature_view=feature_view) + for feature_view in feature_views_to_keep + } + bt_tables_to_delete: Set[str] = { + self._get_table_name(config=config, feature_view=feature_view) + for feature_view in feature_views_to_delete + } - bt_tables_to_keep # we don't delete a table if it's in `bt_tables_to_keep` + + for bt_table_name in bt_tables_to_keep: + bt_table = bt_instance.table(bt_table_name) + if not bt_table.exists(): + logger.info(f"Creating table `{bt_table_name}` in Bigtable") + bt_table.create() + else: + logger.info(f"Table {bt_table_name} already exists in Bigtable") + + if self.feature_column_family not in bt_table.list_column_families(): + bt_table.column_family( + self.feature_column_family, gc_rule=max_versions_gc_rule + ).create() + + for bt_table_name in bt_tables_to_delete: + bt_table = bt_instance.table(bt_table_name) + logger.info(f"Deleting table {bt_table_name} in Bigtable") + bt_table.delete() + + @staticmethod + def _get_table_name(config: RepoConfig, feature_view: FeatureView) -> str: + entities_part = ( + "-".join(sorted(feature_view.entities)) + if feature_view.entities + else DUMMY_ENTITY_NAME + ) + BIGTABLE_TABLE_MAX_LENGTH = 50 + ENTITIES_PART_MAX_LENGTH = 24 + # Bigtable limits table names to 50 characters. We'll limit the max size of of + # the `entities_part` and if that's not enough, we'll just hash the + # entities_part. The remaining length is dedicated to the project name. This + # allows multiple projects to coexist in the same bigtable instance. This also + # allows multiple integration test executions to run simultaneously without + # conflicts. + if len(entities_part) > ENTITIES_PART_MAX_LENGTH: + entities_part = hashlib.md5(entities_part.encode()).hexdigest()[ + :ENTITIES_PART_MAX_LENGTH + ] + remaining_length = BIGTABLE_TABLE_MAX_LENGTH - len(entities_part) - 1 + if len(config.project) > remaining_length: + HUMAN_READABLE_PART_LENGTH = 10 + HASH_PART_LENGTH = remaining_length - HUMAN_READABLE_PART_LENGTH - 1 + project_part = ( + config.project[:HUMAN_READABLE_PART_LENGTH] + + "_" + + hashlib.md5(config.project.encode()).hexdigest()[:HASH_PART_LENGTH] + ) + else: + project_part = config.project + return f"{project_part}.{entities_part}" + + def teardown( + self, + config: RepoConfig, + tables: Sequence[FeatureView], + entities: Sequence[Entity], + ): + # Because of historical reasons, Feast calls them tables. We use this alias for + # readability. + feature_views = tables + + bt_tables = { + self._get_table_name(config=config, feature_view=fv) for fv in feature_views + } + + online_config = config.online_store + assert isinstance(online_config, BigtableOnlineStoreConfig) + client = self._get_client(online_config, admin=True) + bt_instance = client.instance(instance_id=online_config.instance) + for table_name in bt_tables: + try: + logger.info(f"Deleting Bigtable table `{table_name}`") + bt_instance.table(table_name).delete() + except google.api_core.exceptions.NotFound: + logger.warning( + f"Table `{table_name}` was not found. Skipping deletion." + ) + + def _get_client( + self, online_config: BigtableOnlineStoreConfig, admin: bool = False + ): + return bigtable.Client(project=online_config.project_id, admin=admin) diff --git a/sdk/python/feast/repo_config.py b/sdk/python/feast/repo_config.py index c11b25849e..673d039ff0 100644 --- a/sdk/python/feast/repo_config.py +++ b/sdk/python/feast/repo_config.py @@ -48,6 +48,7 @@ "redis": "feast.infra.online_stores.redis.RedisOnlineStore", "dynamodb": "feast.infra.online_stores.dynamodb.DynamoDBOnlineStore", "snowflake.online": "feast.infra.online_stores.snowflake.SnowflakeOnlineStore", + "bigtable": "feast.infra.online_stores.bigtable.BigtableOnlineStore", "postgres": "feast.infra.online_stores.contrib.postgres.PostgreSQLOnlineStore", "hbase": "feast.infra.online_stores.contrib.hbase_online_store.hbase.HbaseOnlineStore", "cassandra": "feast.infra.online_stores.contrib.cassandra_online_store.cassandra_online_store.CassandraOnlineStore", diff --git a/sdk/python/feast/templates/gcp/feature_repo/feature_store.yaml b/sdk/python/feast/templates/gcp/feature_repo/feature_store.yaml index 7d4096615a..e3d9d1d1e6 100644 --- a/sdk/python/feast/templates/gcp/feature_repo/feature_store.yaml +++ b/sdk/python/feast/templates/gcp/feature_repo/feature_store.yaml @@ -13,6 +13,11 @@ online_store: # type: datastore # project_id: my_gcp_project # namespace: my_datastore_namespace +# See https://docs.feast.dev/reference/online-stores/bigtable +#online_store: +# type: bigtable +# project_id: my_gcp_project +# instance: my_bigtable_instance # See https://docs.feast.dev/reference/online-stores/redis #online_store: # type: redis diff --git a/sdk/python/requirements/py3.10-ci-requirements.txt b/sdk/python/requirements/py3.10-ci-requirements.txt index c1656ee7e6..93729efc51 100644 --- a/sdk/python/requirements/py3.10-ci-requirements.txt +++ b/sdk/python/requirements/py3.10-ci-requirements.txt @@ -4,8 +4,6 @@ # # pip-compile --extra=ci --output-file=sdk/python/requirements/py3.10-ci-requirements.txt # -absl-py==1.2.0 - # via tensorflow-metadata adal==1.2.7 # via # azure-datalake-store @@ -14,13 +12,13 @@ adlfs==0.5.9 # via feast (setup.py) aiobotocore==2.1.2 # via s3fs -aiohttp==3.8.1 +aiohttp==3.8.3 # via # adlfs # aiobotocore # gcsfs # s3fs -aioitertools==0.10.0 +aioitertools==0.11.0 # via aiobotocore aiosignal==1.2.0 # via aiohttp @@ -56,7 +54,7 @@ attrs==22.1.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.25.0 +azure-core==1.25.1 # via # adlfs # azure-identity @@ -64,7 +62,7 @@ azure-core==1.25.0 # msrest azure-datalake-store==0.0.52 # via adlfs -azure-identity==1.10.0 +azure-identity==1.11.0 # via # adlfs # feast (setup.py) @@ -76,7 +74,7 @@ babel==2.10.3 # via sphinx backcall==0.2.0 # via ipython -black==22.6.0 +black==22.8.0 # via feast (setup.py) boto3==1.20.23 # via @@ -102,7 +100,7 @@ cachetools==5.2.0 # via google-auth cassandra-driver==3.25.0 # via feast (setup.py) -certifi==2022.6.15 +certifi==2022.9.24 # via # kubernetes # minio @@ -116,7 +114,7 @@ cffi==1.15.1 # snowflake-connector-python cfgv==3.3.1 # via pre-commit -charset-normalizer==2.0.12 +charset-normalizer==2.1.1 # via # aiohttp # requests @@ -131,13 +129,13 @@ click==8.1.3 # moreorless # pip-tools # uvicorn -cloudpickle==2.1.0 +cloudpickle==2.2.0 # via dask colorama==0.4.5 # via # feast (setup.py) # great-expectations -coverage[toml]==6.4.4 +coverage[toml]==6.5.0 # via pytest-cov cryptography==35.0.0 # via @@ -155,7 +153,7 @@ dask==2022.1.1 # via feast (setup.py) dataclasses==0.6 # via great-expectations -db-dtypes==1.0.3 +db-dtypes==1.0.4 # via google-cloud-bigquery decorator==5.1.1 # via @@ -169,7 +167,7 @@ dill==0.3.5.1 # via # feast (setup.py) # multiprocess -distlib==0.3.5 +distlib==0.3.6 # via virtualenv docker==6.0.0 # via @@ -183,19 +181,21 @@ entrypoints==0.4 # via altair execnet==1.9.0 # via pytest-xdist -executing==0.10.0 +executing==1.1.0 # via stack-data -fastapi==0.79.1 +fastapi==0.85.0 # via feast (setup.py) -fastavro==1.6.0 +fastavro==1.6.1 # via # feast (setup.py) # pandavro -fastjsonschema==2.16.1 +fastjsonschema==2.16.2 # via nbformat filelock==3.8.0 - # via virtualenv -firebase-admin==5.2.0 + # via + # snowflake-connector-python + # virtualenv +firebase-admin==5.4.0 # via feast (setup.py) fissix==21.11.13 # via bowler @@ -215,20 +215,21 @@ gcsfs==2022.1.0 # via feast (setup.py) geomet==0.2.1.post1 # via cassandra-driver -google-api-core[grpc]==2.8.2 +google-api-core[grpc]==2.10.1 # via # feast (setup.py) # firebase-admin # google-api-python-client # google-cloud-bigquery # google-cloud-bigquery-storage + # google-cloud-bigtable # google-cloud-core # google-cloud-datastore # google-cloud-firestore # google-cloud-storage -google-api-python-client==2.57.0 +google-api-python-client==2.63.0 # via firebase-admin -google-auth==2.10.0 +google-auth==2.12.0 # via # gcsfs # google-api-core @@ -240,63 +241,68 @@ google-auth==2.10.0 # kubernetes google-auth-httplib2==0.1.0 # via google-api-python-client -google-auth-oauthlib==0.5.2 +google-auth-oauthlib==0.5.3 # via gcsfs -google-cloud-bigquery[pandas]==3.3.2 +google-cloud-bigquery[pandas]==3.3.3 # via feast (setup.py) -google-cloud-bigquery-storage==2.14.2 +google-cloud-bigquery-storage==2.16.1 # via # feast (setup.py) # google-cloud-bigquery +google-cloud-bigtable==2.12.0 + # via feast (setup.py) google-cloud-core==2.3.2 # via # google-cloud-bigquery + # google-cloud-bigtable # google-cloud-datastore # google-cloud-firestore # google-cloud-storage google-cloud-datastore==2.8.1 # via feast (setup.py) -google-cloud-firestore==2.6.1 +google-cloud-firestore==2.7.0 # via firebase-admin google-cloud-storage==2.5.0 # via # feast (setup.py) # firebase-admin # gcsfs -google-crc32c==1.3.0 +google-crc32c==1.5.0 # via google-resumable-media -google-resumable-media==2.3.3 +google-resumable-media==2.4.0 # via # google-cloud-bigquery # google-cloud-storage -googleapis-common-protos==1.56.4 +googleapis-common-protos[grpc]==1.56.4 # via # feast (setup.py) # google-api-core + # grpc-google-iam-v1 # grpcio-status - # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -greenlet==1.1.2 - # via sqlalchemy -grpcio==1.47.0 +grpc-google-iam-v1==0.12.4 + # via google-cloud-bigtable +grpcio==1.49.1 # via # feast (setup.py) # google-api-core # google-cloud-bigquery + # googleapis-common-protos + # grpc-google-iam-v1 # grpcio-reflection # grpcio-status # grpcio-testing # grpcio-tools -grpcio-reflection==1.47.0 +grpcio-reflection==1.49.1 # via feast (setup.py) -grpcio-status==1.47.0 +grpcio-status==1.49.1 # via google-api-core -grpcio-testing==1.47.0 +grpcio-testing==1.49.1 # via feast (setup.py) -grpcio-tools==1.47.0 +grpcio-tools==1.49.1 # via feast (setup.py) -h11==0.13.0 +h11==0.14.0 # via uvicorn happybase==1.2.0 # via feast (setup.py) @@ -306,11 +312,11 @@ httplib2==0.20.4 # via # google-api-python-client # google-auth-httplib2 -httptools==0.4.0 +httptools==0.5.0 # via uvicorn -identify==2.5.3 +identify==2.5.5 # via pre-commit -idna==3.3 +idna==3.4 # via # anyio # requests @@ -322,7 +328,7 @@ importlib-metadata==4.12.0 # via great-expectations iniconfig==1.1.1 # via pytest -ipython==8.4.0 +ipython==8.5.0 # via great-expectations isodate==0.6.1 # via msrest @@ -345,7 +351,7 @@ jsonpatch==1.32 # via great-expectations jsonpointer==2.3 # via jsonpatch -jsonschema==4.13.0 +jsonschema==4.16.0 # via # altair # feast (setup.py) @@ -377,7 +383,7 @@ moreorless==0.4.0 # via bowler moto==3.1.18 # via feast (setup.py) -msal==1.18.0 +msal==1.19.0 # via # azure-identity # msal-extensions @@ -397,7 +403,7 @@ multidict==6.0.2 # yarl multiprocess==0.70.13 # via bytewax -mypy==0.971 +mypy==0.981 # via # feast (setup.py) # sqlalchemy @@ -409,11 +415,11 @@ mypy-protobuf==3.1 # via feast (setup.py) mysqlclient==2.1.1 # via feast (setup.py) -nbformat==5.4.0 +nbformat==5.6.1 # via great-expectations nodeenv==1.7.0 # via pre-commit -numpy==1.23.2 +numpy==1.23.3 # via # altair # db-dtypes @@ -439,7 +445,7 @@ packaging==21.3 # pytest # redis # sphinx -pandas==1.4.3 +pandas==1.4.4 # via # altair # db-dtypes @@ -454,7 +460,7 @@ parso==0.8.3 # via jedi partd==1.3.0 # via dask -pathspec==0.9.0 +pathspec==0.10.1 # via black pbr==5.10.0 # via mock @@ -478,21 +484,23 @@ portalocker==2.5.1 # via msal-extensions pre-commit==2.20.0 # via feast (setup.py) -prompt-toolkit==3.0.30 +prompt-toolkit==3.0.31 # via ipython -proto-plus==1.22.0 +proto-plus==1.22.1 # via # feast (setup.py) # google-cloud-bigquery # google-cloud-bigquery-storage + # google-cloud-bigtable # google-cloud-datastore # google-cloud-firestore -protobuf==3.20.2 +protobuf==4.21.7 # via # feast (setup.py) # google-api-core # google-cloud-bigquery # google-cloud-bigquery-storage + # google-cloud-bigtable # google-cloud-datastore # google-cloud-firestore # googleapis-common-protos @@ -502,7 +510,6 @@ protobuf==3.20.2 # grpcio-tools # mypy-protobuf # proto-plus - # tensorflow-metadata psutil==5.9.0 # via feast (setup.py) psycopg2-binary==2.9.3 @@ -519,7 +526,7 @@ py-cpuinfo==8.0.0 # via pytest-benchmark py4j==0.10.9.5 # via pyspark -pyarrow==6.0.1 +pyarrow==8.0.0 # via # db-dtypes # feast (setup.py) @@ -539,7 +546,7 @@ pycparser==2.21 # via cffi pycryptodomex==3.15.0 # via snowflake-connector-python -pydantic==1.9.2 +pydantic==1.10.2 # via # fastapi # feast (setup.py) @@ -550,7 +557,7 @@ pygments==2.13.0 # feast (setup.py) # ipython # sphinx -pyjwt[crypto]==2.4.0 +pyjwt[crypto]==2.5.0 # via # adal # msal @@ -562,7 +569,9 @@ pymysql==1.0.2 pyodbc==4.0.34 # via feast (setup.py) pyopenssl==22.0.0 - # via snowflake-connector-python + # via + # feast (setup.py) + # snowflake-connector-python pyparsing==2.4.7 # via # great-expectations @@ -572,7 +581,7 @@ pyrsistent==0.18.1 # via jsonschema pyspark==3.3.0 # via feast (setup.py) -pytest==7.1.2 +pytest==7.1.3 # via # feast (setup.py) # pytest-benchmark @@ -585,7 +594,7 @@ pytest==7.1.2 # pytest-xdist pytest-benchmark==3.4.1 # via feast (setup.py) -pytest-cov==3.0.0 +pytest-cov==4.0.0 # via feast (setup.py) pytest-forked==1.4.0 # via pytest-xdist @@ -608,7 +617,7 @@ python-dateutil==2.8.2 # kubernetes # moto # pandas -python-dotenv==0.20.0 +python-dotenv==0.21.0 # via uvicorn pytz==2022.2.1 # via @@ -666,7 +675,7 @@ s3fs==2022.1.0 # via feast (setup.py) s3transfer==0.5.2 # via boto3 -scipy==1.9.0 +scipy==1.9.1 # via great-expectations six==1.16.0 # via @@ -684,11 +693,11 @@ six==1.16.0 # msrestazure # pandavro # python-dateutil -sniffio==1.2.0 +sniffio==1.3.0 # via anyio snowballstemmer==2.2.0 # via sphinx -snowflake-connector-python[pandas]==2.7.8 +snowflake-connector-python[pandas]==2.8.0 # via feast (setup.py) sphinx==4.3.2 # via @@ -708,23 +717,21 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.5 # via sphinx -sqlalchemy[mypy]==1.4.40 +sqlalchemy[mypy]==1.4.41 # via feast (setup.py) -sqlalchemy2-stubs==0.0.2a25 +sqlalchemy2-stubs==0.0.2a27 # via sqlalchemy -stack-data==0.4.0 +stack-data==0.5.1 # via ipython -starlette==0.19.1 +starlette==0.20.4 # via fastapi tabulate==0.8.10 # via feast (setup.py) -tenacity==8.0.1 +tenacity==8.1.0 # via feast (setup.py) -tensorflow-metadata==1.9.0 - # via feast (setup.py) -termcolor==1.1.0 +termcolor==2.0.1 # via great-expectations -testcontainers==3.6.1 +testcontainers==3.7.0 # via feast (setup.py) thriftpy2==0.4.14 # via happybase @@ -745,21 +752,23 @@ toolz==0.12.0 # altair # dask # partd -tqdm==4.64.0 +tqdm==4.64.1 # via # feast (setup.py) # great-expectations -traitlets==5.3.0 +traitlets==5.4.0 # via # ipython # jupyter-core # matplotlib-inline # nbformat -trino==0.315.0 +trino==0.316.0 # via feast (setup.py) typeguard==2.13.3 # via feast (setup.py) -types-protobuf==3.19.22 +types-cryptography==3.3.23 + # via pyjwt +types-protobuf==3.20.4 # via # feast (setup.py) # mypy-protobuf @@ -769,17 +778,17 @@ types-python-dateutil==2.8.19 # via feast (setup.py) types-pytz==2022.2.1.0 # via feast (setup.py) -types-pyyaml==6.0.11 +types-pyyaml==6.0.12 # via feast (setup.py) -types-redis==4.3.14 +types-redis==4.3.21 # via feast (setup.py) -types-requests==2.28.9 +types-requests==2.28.11 # via feast (setup.py) -types-setuptools==65.1.0 +types-setuptools==65.4.0.0 # via feast (setup.py) types-tabulate==0.8.11 # via feast (setup.py) -types-urllib3==1.26.23 +types-urllib3==1.26.25 # via types-requests typing-extensions==4.3.0 # via @@ -787,14 +796,15 @@ typing-extensions==4.3.0 # great-expectations # mypy # pydantic + # snowflake-connector-python # sqlalchemy2-stubs -tzdata==2022.2 +tzdata==2022.4 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations uritemplate==4.1.1 # via google-api-python-client -urllib3==1.26.11 +urllib3==1.26.12 # via # botocore # docker @@ -804,19 +814,20 @@ urllib3==1.26.11 # minio # requests # responses -uvicorn[standard]==0.18.2 + # snowflake-connector-python +uvicorn[standard]==0.18.3 # via feast (setup.py) -uvloop==0.16.0 +uvloop==0.17.0 # via uvicorn -virtualenv==20.16.3 +virtualenv==20.16.5 # via pre-commit volatile==2.1.0 # via bowler -watchfiles==0.16.1 +watchfiles==0.17.0 # via uvicorn wcwidth==0.2.5 # via prompt-toolkit -websocket-client==1.3.3 +websocket-client==1.4.1 # via # docker # kubernetes diff --git a/sdk/python/requirements/py3.10-requirements.txt b/sdk/python/requirements/py3.10-requirements.txt index 9136930917..4770b765e5 100644 --- a/sdk/python/requirements/py3.10-requirements.txt +++ b/sdk/python/requirements/py3.10-requirements.txt @@ -4,8 +4,6 @@ # # pip-compile --output-file=sdk/python/requirements/py3.10-requirements.txt # -absl-py==1.2.0 - # via tensorflow-metadata anyio==3.6.1 # via # starlette @@ -20,9 +18,9 @@ bowler==0.9.0 # via feast (setup.py) cachetools==5.2.0 # via google-auth -certifi==2022.6.15 +certifi==2022.9.24 # via requests -charset-normalizer==2.1.0 +charset-normalizer==2.1.1 # via requests click==8.1.3 # via @@ -30,7 +28,7 @@ click==8.1.3 # feast (setup.py) # moreorless # uvicorn -cloudpickle==2.1.0 +cloudpickle==2.2.0 # via dask colorama==0.4.5 # via feast (setup.py) @@ -38,44 +36,41 @@ dask==2022.1.1 # via feast (setup.py) dill==0.3.5.1 # via feast (setup.py) -fastapi==0.79.1 +fastapi==0.85.0 # via feast (setup.py) -fastavro==1.6.0 +fastavro==1.6.1 # via # feast (setup.py) # pandavro fissix==21.11.13 # via bowler -fsspec==2022.7.1 +fsspec==2022.8.2 # via dask -google-api-core==2.8.2 +google-api-core==2.10.1 # via feast (setup.py) -google-auth==2.10.0 +google-auth==2.12.0 # via google-api-core googleapis-common-protos==1.56.4 # via # feast (setup.py) # google-api-core - # tensorflow-metadata -greenlet==1.1.2 - # via sqlalchemy -grpcio==1.47.0 +grpcio==1.49.1 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.47.0 +grpcio-reflection==1.49.1 # via feast (setup.py) -h11==0.13.0 +h11==0.14.0 # via uvicorn -httptools==0.4.0 +httptools==0.5.0 # via uvicorn -idna==3.3 +idna==3.4 # via # anyio # requests jinja2==3.1.2 # via feast (setup.py) -jsonschema==4.13.0 +jsonschema==4.16.0 # via feast (setup.py) locket==1.0.0 # via partd @@ -85,11 +80,11 @@ mmh3==3.0.0 # via feast (setup.py) moreorless==0.4.0 # via bowler -mypy==0.971 +mypy==0.981 # via sqlalchemy mypy-extensions==0.4.3 # via mypy -numpy==1.23.2 +numpy==1.23.3 # via # feast (setup.py) # pandas @@ -97,7 +92,7 @@ numpy==1.23.2 # pyarrow packaging==21.3 # via dask -pandas==1.4.3 +pandas==1.5.0 # via # feast (setup.py) # pandavro @@ -105,16 +100,15 @@ pandavro==1.5.2 # via feast (setup.py) partd==1.3.0 # via dask -proto-plus==1.22.0 +proto-plus==1.22.1 # via feast (setup.py) -protobuf==3.20.2 +protobuf==4.21.7 # via # feast (setup.py) # google-api-core # googleapis-common-protos # grpcio-reflection # proto-plus - # tensorflow-metadata pyarrow==8.0.0 # via feast (setup.py) pyasn1==0.4.8 @@ -123,7 +117,7 @@ pyasn1==0.4.8 # rsa pyasn1-modules==0.2.8 # via google-auth -pydantic==1.9.2 +pydantic==1.10.2 # via # fastapi # feast (setup.py) @@ -135,7 +129,7 @@ pyrsistent==0.18.1 # via jsonschema python-dateutil==2.8.2 # via pandas -python-dotenv==0.20.0 +python-dotenv==0.21.0 # via uvicorn pytz==2022.2.1 # via pandas @@ -154,19 +148,17 @@ six==1.16.0 # grpcio # pandavro # python-dateutil -sniffio==1.2.0 +sniffio==1.3.0 # via anyio -sqlalchemy[mypy]==1.4.40 +sqlalchemy[mypy]==1.4.41 # via feast (setup.py) -sqlalchemy2-stubs==0.0.2a25 +sqlalchemy2-stubs==0.0.2a27 # via sqlalchemy -starlette==0.19.1 +starlette==0.20.4 # via fastapi tabulate==0.8.10 # via feast (setup.py) -tenacity==8.0.1 - # via feast (setup.py) -tensorflow-metadata==1.9.0 +tenacity==8.1.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) @@ -176,7 +168,7 @@ toolz==0.12.0 # via # dask # partd -tqdm==4.64.0 +tqdm==4.64.1 # via feast (setup.py) typeguard==2.13.3 # via feast (setup.py) @@ -185,15 +177,15 @@ typing-extensions==4.3.0 # mypy # pydantic # sqlalchemy2-stubs -urllib3==1.26.11 +urllib3==1.26.12 # via requests -uvicorn[standard]==0.18.2 +uvicorn[standard]==0.18.3 # via feast (setup.py) -uvloop==0.16.0 +uvloop==0.17.0 # via uvicorn volatile==2.1.0 # via bowler -watchfiles==0.16.1 +watchfiles==0.17.0 # via uvicorn websockets==10.3 # via uvicorn diff --git a/sdk/python/requirements/py3.8-ci-requirements.txt b/sdk/python/requirements/py3.8-ci-requirements.txt index 84a8c7408e..e1b99210aa 100644 --- a/sdk/python/requirements/py3.8-ci-requirements.txt +++ b/sdk/python/requirements/py3.8-ci-requirements.txt @@ -4,8 +4,6 @@ # # pip-compile --extra=ci --output-file=sdk/python/requirements/py3.8-ci-requirements.txt # -absl-py==1.2.0 - # via tensorflow-metadata adal==1.2.7 # via # azure-datalake-store @@ -14,13 +12,13 @@ adlfs==0.5.9 # via feast (setup.py) aiobotocore==2.1.2 # via s3fs -aiohttp==3.8.1 +aiohttp==3.8.3 # via # adlfs # aiobotocore # gcsfs # s3fs -aioitertools==0.10.0 +aioitertools==0.11.0 # via aiobotocore aiosignal==1.2.0 # via aiohttp @@ -56,7 +54,7 @@ attrs==22.1.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.25.0 +azure-core==1.25.1 # via # adlfs # azure-identity @@ -64,7 +62,7 @@ azure-core==1.25.0 # msrest azure-datalake-store==0.0.52 # via adlfs -azure-identity==1.10.0 +azure-identity==1.11.0 # via # adlfs # feast (setup.py) @@ -80,7 +78,7 @@ backports-zoneinfo==0.2.1 # via # pytz-deprecation-shim # tzlocal -black==22.6.0 +black==22.8.0 # via feast (setup.py) boto3==1.20.23 # via @@ -106,7 +104,7 @@ cachetools==5.2.0 # via google-auth cassandra-driver==3.25.0 # via feast (setup.py) -certifi==2022.6.15 +certifi==2022.9.24 # via # kubernetes # minio @@ -120,7 +118,7 @@ cffi==1.15.1 # snowflake-connector-python cfgv==3.3.1 # via pre-commit -charset-normalizer==2.0.12 +charset-normalizer==2.1.1 # via # aiohttp # requests @@ -135,13 +133,13 @@ click==8.1.3 # moreorless # pip-tools # uvicorn -cloudpickle==2.1.0 +cloudpickle==2.2.0 # via dask colorama==0.4.5 # via # feast (setup.py) # great-expectations -coverage[toml]==6.4.4 +coverage[toml]==6.5.0 # via pytest-cov cryptography==35.0.0 # via @@ -159,7 +157,7 @@ dask==2022.1.1 # via feast (setup.py) dataclasses==0.6 # via great-expectations -db-dtypes==1.0.3 +db-dtypes==1.0.4 # via google-cloud-bigquery decorator==5.1.1 # via @@ -173,7 +171,7 @@ dill==0.3.5.1 # via # feast (setup.py) # multiprocess -distlib==0.3.5 +distlib==0.3.6 # via virtualenv docker==6.0.0 # via @@ -187,19 +185,21 @@ entrypoints==0.4 # via altair execnet==1.9.0 # via pytest-xdist -executing==0.10.0 +executing==1.1.0 # via stack-data -fastapi==0.79.1 +fastapi==0.85.0 # via feast (setup.py) -fastavro==1.6.0 +fastavro==1.6.1 # via # feast (setup.py) # pandavro -fastjsonschema==2.16.1 +fastjsonschema==2.16.2 # via nbformat filelock==3.8.0 - # via virtualenv -firebase-admin==5.2.0 + # via + # snowflake-connector-python + # virtualenv +firebase-admin==5.4.0 # via feast (setup.py) fissix==21.11.13 # via bowler @@ -219,20 +219,21 @@ gcsfs==2022.1.0 # via feast (setup.py) geomet==0.2.1.post1 # via cassandra-driver -google-api-core[grpc]==2.8.2 +google-api-core[grpc]==2.10.1 # via # feast (setup.py) # firebase-admin # google-api-python-client # google-cloud-bigquery # google-cloud-bigquery-storage + # google-cloud-bigtable # google-cloud-core # google-cloud-datastore # google-cloud-firestore # google-cloud-storage -google-api-python-client==2.57.0 +google-api-python-client==2.63.0 # via firebase-admin -google-auth==2.10.0 +google-auth==2.12.0 # via # gcsfs # google-api-core @@ -244,63 +245,68 @@ google-auth==2.10.0 # kubernetes google-auth-httplib2==0.1.0 # via google-api-python-client -google-auth-oauthlib==0.5.2 +google-auth-oauthlib==0.5.3 # via gcsfs -google-cloud-bigquery[pandas]==3.3.2 +google-cloud-bigquery[pandas]==3.3.3 # via feast (setup.py) -google-cloud-bigquery-storage==2.14.2 +google-cloud-bigquery-storage==2.16.1 # via # feast (setup.py) # google-cloud-bigquery +google-cloud-bigtable==2.12.0 + # via feast (setup.py) google-cloud-core==2.3.2 # via # google-cloud-bigquery + # google-cloud-bigtable # google-cloud-datastore # google-cloud-firestore # google-cloud-storage google-cloud-datastore==2.8.1 # via feast (setup.py) -google-cloud-firestore==2.6.1 +google-cloud-firestore==2.7.0 # via firebase-admin google-cloud-storage==2.5.0 # via # feast (setup.py) # firebase-admin # gcsfs -google-crc32c==1.3.0 +google-crc32c==1.5.0 # via google-resumable-media -google-resumable-media==2.3.3 +google-resumable-media==2.4.0 # via # google-cloud-bigquery # google-cloud-storage -googleapis-common-protos==1.56.4 +googleapis-common-protos[grpc]==1.56.4 # via # feast (setup.py) # google-api-core + # grpc-google-iam-v1 # grpcio-status - # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -greenlet==1.1.2 - # via sqlalchemy -grpcio==1.47.0 +grpc-google-iam-v1==0.12.4 + # via google-cloud-bigtable +grpcio==1.49.1 # via # feast (setup.py) # google-api-core # google-cloud-bigquery + # googleapis-common-protos + # grpc-google-iam-v1 # grpcio-reflection # grpcio-status # grpcio-testing # grpcio-tools -grpcio-reflection==1.47.0 +grpcio-reflection==1.49.1 # via feast (setup.py) -grpcio-status==1.47.0 +grpcio-status==1.49.1 # via google-api-core -grpcio-testing==1.47.0 +grpcio-testing==1.49.1 # via feast (setup.py) -grpcio-tools==1.47.0 +grpcio-tools==1.49.1 # via feast (setup.py) -h11==0.13.0 +h11==0.14.0 # via uvicorn happybase==1.2.0 # via feast (setup.py) @@ -310,11 +316,11 @@ httplib2==0.20.4 # via # google-api-python-client # google-auth-httplib2 -httptools==0.4.0 +httptools==0.5.0 # via uvicorn -identify==2.5.3 +identify==2.5.5 # via pre-commit -idna==3.3 +idna==3.4 # via # anyio # requests @@ -328,7 +334,7 @@ importlib-resources==5.9.0 # via jsonschema iniconfig==1.1.1 # via pytest -ipython==8.4.0 +ipython==8.5.0 # via great-expectations isodate==0.6.1 # via msrest @@ -351,7 +357,7 @@ jsonpatch==1.32 # via great-expectations jsonpointer==2.3 # via jsonpatch -jsonschema==4.13.0 +jsonschema==4.16.0 # via # altair # feast (setup.py) @@ -383,7 +389,7 @@ moreorless==0.4.0 # via bowler moto==3.1.18 # via feast (setup.py) -msal==1.18.0 +msal==1.19.0 # via # azure-identity # msal-extensions @@ -403,7 +409,7 @@ multidict==6.0.2 # yarl multiprocess==0.70.13 # via bytewax -mypy==0.971 +mypy==0.981 # via # feast (setup.py) # sqlalchemy @@ -415,11 +421,11 @@ mypy-protobuf==3.1 # via feast (setup.py) mysqlclient==2.1.1 # via feast (setup.py) -nbformat==5.4.0 +nbformat==5.6.1 # via great-expectations nodeenv==1.7.0 # via pre-commit -numpy==1.23.2 +numpy==1.23.3 # via # altair # db-dtypes @@ -445,7 +451,7 @@ packaging==21.3 # pytest # redis # sphinx -pandas==1.4.3 +pandas==1.4.4 # via # altair # db-dtypes @@ -460,7 +466,7 @@ parso==0.8.3 # via jedi partd==1.3.0 # via dask -pathspec==0.9.0 +pathspec==0.10.1 # via black pbr==5.10.0 # via mock @@ -486,21 +492,23 @@ portalocker==2.5.1 # via msal-extensions pre-commit==2.20.0 # via feast (setup.py) -prompt-toolkit==3.0.30 +prompt-toolkit==3.0.31 # via ipython -proto-plus==1.22.0 +proto-plus==1.22.1 # via # feast (setup.py) # google-cloud-bigquery # google-cloud-bigquery-storage + # google-cloud-bigtable # google-cloud-datastore # google-cloud-firestore -protobuf==3.20.2 +protobuf==4.21.7 # via # feast (setup.py) # google-api-core # google-cloud-bigquery # google-cloud-bigquery-storage + # google-cloud-bigtable # google-cloud-datastore # google-cloud-firestore # googleapis-common-protos @@ -510,7 +518,6 @@ protobuf==3.20.2 # grpcio-tools # mypy-protobuf # proto-plus - # tensorflow-metadata psutil==5.9.0 # via feast (setup.py) psycopg2-binary==2.9.3 @@ -527,7 +534,7 @@ py-cpuinfo==8.0.0 # via pytest-benchmark py4j==0.10.9.5 # via pyspark -pyarrow==6.0.1 +pyarrow==8.0.0 # via # db-dtypes # feast (setup.py) @@ -547,7 +554,7 @@ pycparser==2.21 # via cffi pycryptodomex==3.15.0 # via snowflake-connector-python -pydantic==1.9.2 +pydantic==1.10.2 # via # fastapi # feast (setup.py) @@ -558,7 +565,7 @@ pygments==2.13.0 # feast (setup.py) # ipython # sphinx -pyjwt[crypto]==2.4.0 +pyjwt[crypto]==2.5.0 # via # adal # msal @@ -570,7 +577,9 @@ pymysql==1.0.2 pyodbc==4.0.34 # via feast (setup.py) pyopenssl==22.0.0 - # via snowflake-connector-python + # via + # feast (setup.py) + # snowflake-connector-python pyparsing==2.4.7 # via # great-expectations @@ -580,7 +589,7 @@ pyrsistent==0.18.1 # via jsonschema pyspark==3.3.0 # via feast (setup.py) -pytest==7.1.2 +pytest==7.1.3 # via # feast (setup.py) # pytest-benchmark @@ -593,7 +602,7 @@ pytest==7.1.2 # pytest-xdist pytest-benchmark==3.4.1 # via feast (setup.py) -pytest-cov==3.0.0 +pytest-cov==4.0.0 # via feast (setup.py) pytest-forked==1.4.0 # via pytest-xdist @@ -616,7 +625,7 @@ python-dateutil==2.8.2 # kubernetes # moto # pandas -python-dotenv==0.20.0 +python-dotenv==0.21.0 # via uvicorn pytz==2022.2.1 # via @@ -676,7 +685,7 @@ s3fs==2022.1.0 # via feast (setup.py) s3transfer==0.5.2 # via boto3 -scipy==1.9.0 +scipy==1.9.1 # via great-expectations six==1.16.0 # via @@ -694,11 +703,11 @@ six==1.16.0 # msrestazure # pandavro # python-dateutil -sniffio==1.2.0 +sniffio==1.3.0 # via anyio snowballstemmer==2.2.0 # via sphinx -snowflake-connector-python[pandas]==2.7.8 +snowflake-connector-python[pandas]==2.8.0 # via feast (setup.py) sphinx==4.3.2 # via @@ -718,23 +727,21 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.5 # via sphinx -sqlalchemy[mypy]==1.4.40 +sqlalchemy[mypy]==1.4.41 # via feast (setup.py) -sqlalchemy2-stubs==0.0.2a25 +sqlalchemy2-stubs==0.0.2a27 # via sqlalchemy -stack-data==0.4.0 +stack-data==0.5.1 # via ipython -starlette==0.19.1 +starlette==0.20.4 # via fastapi tabulate==0.8.10 # via feast (setup.py) -tenacity==8.0.1 +tenacity==8.1.0 # via feast (setup.py) -tensorflow-metadata==1.9.0 - # via feast (setup.py) -termcolor==1.1.0 +termcolor==2.0.1 # via great-expectations -testcontainers==3.6.1 +testcontainers==3.7.0 # via feast (setup.py) thriftpy2==0.4.14 # via happybase @@ -755,21 +762,23 @@ toolz==0.12.0 # altair # dask # partd -tqdm==4.64.0 +tqdm==4.64.1 # via # feast (setup.py) # great-expectations -traitlets==5.3.0 +traitlets==5.4.0 # via # ipython # jupyter-core # matplotlib-inline # nbformat -trino==0.315.0 +trino==0.316.0 # via feast (setup.py) typeguard==2.13.3 # via feast (setup.py) -types-protobuf==3.19.22 +types-cryptography==3.3.23 + # via pyjwt +types-protobuf==3.20.4 # via # feast (setup.py) # mypy-protobuf @@ -779,17 +788,17 @@ types-python-dateutil==2.8.19 # via feast (setup.py) types-pytz==2022.2.1.0 # via feast (setup.py) -types-pyyaml==6.0.11 +types-pyyaml==6.0.12 # via feast (setup.py) -types-redis==4.3.14 +types-redis==4.3.21 # via feast (setup.py) -types-requests==2.28.9 +types-requests==2.28.11 # via feast (setup.py) -types-setuptools==65.1.0 +types-setuptools==65.4.0.0 # via feast (setup.py) types-tabulate==0.8.11 # via feast (setup.py) -types-urllib3==1.26.23 +types-urllib3==1.26.25 # via types-requests typing-extensions==4.3.0 # via @@ -799,15 +808,16 @@ typing-extensions==4.3.0 # great-expectations # mypy # pydantic + # snowflake-connector-python # sqlalchemy2-stubs # starlette -tzdata==2022.2 +tzdata==2022.4 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations uritemplate==4.1.1 # via google-api-python-client -urllib3==1.26.11 +urllib3==1.26.12 # via # botocore # docker @@ -817,19 +827,20 @@ urllib3==1.26.11 # minio # requests # responses -uvicorn[standard]==0.18.2 + # snowflake-connector-python +uvicorn[standard]==0.18.3 # via feast (setup.py) -uvloop==0.16.0 +uvloop==0.17.0 # via uvicorn -virtualenv==20.16.3 +virtualenv==20.16.5 # via pre-commit volatile==2.1.0 # via bowler -watchfiles==0.16.1 +watchfiles==0.17.0 # via uvicorn wcwidth==0.2.5 # via prompt-toolkit -websocket-client==1.3.3 +websocket-client==1.4.1 # via # docker # kubernetes diff --git a/sdk/python/requirements/py3.8-requirements.txt b/sdk/python/requirements/py3.8-requirements.txt index b992ad3fc0..9a321bf9a5 100644 --- a/sdk/python/requirements/py3.8-requirements.txt +++ b/sdk/python/requirements/py3.8-requirements.txt @@ -4,8 +4,6 @@ # # pip-compile --output-file=sdk/python/requirements/py3.8-requirements.txt # -absl-py==1.2.0 - # via tensorflow-metadata anyio==3.6.1 # via # starlette @@ -20,9 +18,9 @@ bowler==0.9.0 # via feast (setup.py) cachetools==5.2.0 # via google-auth -certifi==2022.6.15 +certifi==2022.9.24 # via requests -charset-normalizer==2.1.0 +charset-normalizer==2.1.1 # via requests click==8.1.3 # via @@ -30,7 +28,7 @@ click==8.1.3 # feast (setup.py) # moreorless # uvicorn -cloudpickle==2.1.0 +cloudpickle==2.2.0 # via dask colorama==0.4.5 # via feast (setup.py) @@ -38,38 +36,35 @@ dask==2022.1.1 # via feast (setup.py) dill==0.3.5.1 # via feast (setup.py) -fastapi==0.79.1 +fastapi==0.85.0 # via feast (setup.py) -fastavro==1.6.0 +fastavro==1.6.1 # via # feast (setup.py) # pandavro fissix==21.11.13 # via bowler -fsspec==2022.7.1 +fsspec==2022.8.2 # via dask -google-api-core==2.8.2 +google-api-core==2.10.1 # via feast (setup.py) -google-auth==2.10.0 +google-auth==2.12.0 # via google-api-core googleapis-common-protos==1.56.4 # via # feast (setup.py) # google-api-core - # tensorflow-metadata -greenlet==1.1.2 - # via sqlalchemy -grpcio==1.47.0 +grpcio==1.49.1 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.47.0 +grpcio-reflection==1.49.1 # via feast (setup.py) -h11==0.13.0 +h11==0.14.0 # via uvicorn -httptools==0.4.0 +httptools==0.5.0 # via uvicorn -idna==3.3 +idna==3.4 # via # anyio # requests @@ -77,7 +72,7 @@ importlib-resources==5.9.0 # via jsonschema jinja2==3.1.2 # via feast (setup.py) -jsonschema==4.13.0 +jsonschema==4.16.0 # via feast (setup.py) locket==1.0.0 # via partd @@ -87,11 +82,11 @@ mmh3==3.0.0 # via feast (setup.py) moreorless==0.4.0 # via bowler -mypy==0.971 +mypy==0.981 # via sqlalchemy mypy-extensions==0.4.3 # via mypy -numpy==1.23.2 +numpy==1.23.3 # via # feast (setup.py) # pandas @@ -99,7 +94,7 @@ numpy==1.23.2 # pyarrow packaging==21.3 # via dask -pandas==1.4.3 +pandas==1.5.0 # via # feast (setup.py) # pandavro @@ -109,16 +104,15 @@ partd==1.3.0 # via dask pkgutil-resolve-name==1.3.10 # via jsonschema -proto-plus==1.22.0 +proto-plus==1.22.1 # via feast (setup.py) -protobuf==3.20.2 +protobuf==4.21.7 # via # feast (setup.py) # google-api-core # googleapis-common-protos # grpcio-reflection # proto-plus - # tensorflow-metadata pyarrow==8.0.0 # via feast (setup.py) pyasn1==0.4.8 @@ -127,7 +121,7 @@ pyasn1==0.4.8 # rsa pyasn1-modules==0.2.8 # via google-auth -pydantic==1.9.2 +pydantic==1.10.2 # via # fastapi # feast (setup.py) @@ -139,7 +133,7 @@ pyrsistent==0.18.1 # via jsonschema python-dateutil==2.8.2 # via pandas -python-dotenv==0.20.0 +python-dotenv==0.21.0 # via uvicorn pytz==2022.2.1 # via pandas @@ -158,19 +152,17 @@ six==1.16.0 # grpcio # pandavro # python-dateutil -sniffio==1.2.0 +sniffio==1.3.0 # via anyio -sqlalchemy[mypy]==1.4.40 +sqlalchemy[mypy]==1.4.41 # via feast (setup.py) -sqlalchemy2-stubs==0.0.2a25 +sqlalchemy2-stubs==0.0.2a27 # via sqlalchemy -starlette==0.19.1 +starlette==0.20.4 # via fastapi tabulate==0.8.10 # via feast (setup.py) -tenacity==8.0.1 - # via feast (setup.py) -tensorflow-metadata==1.9.0 +tenacity==8.1.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) @@ -180,7 +172,7 @@ toolz==0.12.0 # via # dask # partd -tqdm==4.64.0 +tqdm==4.64.1 # via feast (setup.py) typeguard==2.13.3 # via feast (setup.py) @@ -190,15 +182,15 @@ typing-extensions==4.3.0 # pydantic # sqlalchemy2-stubs # starlette -urllib3==1.26.11 +urllib3==1.26.12 # via requests -uvicorn[standard]==0.18.2 +uvicorn[standard]==0.18.3 # via feast (setup.py) -uvloop==0.16.0 +uvloop==0.17.0 # via uvicorn volatile==2.1.0 # via bowler -watchfiles==0.16.1 +watchfiles==0.17.0 # via uvicorn websockets==10.3 # via uvicorn diff --git a/sdk/python/requirements/py3.9-ci-requirements.txt b/sdk/python/requirements/py3.9-ci-requirements.txt index 4893035a7e..af7b79ea22 100644 --- a/sdk/python/requirements/py3.9-ci-requirements.txt +++ b/sdk/python/requirements/py3.9-ci-requirements.txt @@ -4,8 +4,6 @@ # # pip-compile --extra=ci --output-file=sdk/python/requirements/py3.9-ci-requirements.txt # -absl-py==1.2.0 - # via tensorflow-metadata adal==1.2.7 # via # azure-datalake-store @@ -14,13 +12,13 @@ adlfs==0.5.9 # via feast (setup.py) aiobotocore==2.1.2 # via s3fs -aiohttp==3.8.1 +aiohttp==3.8.3 # via # adlfs # aiobotocore # gcsfs # s3fs -aioitertools==0.10.0 +aioitertools==0.11.0 # via aiobotocore aiosignal==1.2.0 # via aiohttp @@ -56,7 +54,7 @@ attrs==22.1.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.25.0 +azure-core==1.25.1 # via # adlfs # azure-identity @@ -64,7 +62,7 @@ azure-core==1.25.0 # msrest azure-datalake-store==0.0.52 # via adlfs -azure-identity==1.10.0 +azure-identity==1.11.0 # via # adlfs # feast (setup.py) @@ -76,7 +74,7 @@ babel==2.10.3 # via sphinx backcall==0.2.0 # via ipython -black==22.6.0 +black==22.8.0 # via feast (setup.py) boto3==1.20.23 # via @@ -102,7 +100,7 @@ cachetools==5.2.0 # via google-auth cassandra-driver==3.25.0 # via feast (setup.py) -certifi==2022.6.15 +certifi==2022.9.24 # via # kubernetes # minio @@ -116,7 +114,7 @@ cffi==1.15.1 # snowflake-connector-python cfgv==3.3.1 # via pre-commit -charset-normalizer==2.0.12 +charset-normalizer==2.1.1 # via # aiohttp # requests @@ -131,13 +129,13 @@ click==8.1.3 # moreorless # pip-tools # uvicorn -cloudpickle==2.1.0 +cloudpickle==2.2.0 # via dask colorama==0.4.5 # via # feast (setup.py) # great-expectations -coverage[toml]==6.4.4 +coverage[toml]==6.5.0 # via pytest-cov cryptography==35.0.0 # via @@ -155,7 +153,7 @@ dask==2022.1.1 # via feast (setup.py) dataclasses==0.6 # via great-expectations -db-dtypes==1.0.3 +db-dtypes==1.0.4 # via google-cloud-bigquery decorator==5.1.1 # via @@ -169,7 +167,7 @@ dill==0.3.5.1 # via # feast (setup.py) # multiprocess -distlib==0.3.5 +distlib==0.3.6 # via virtualenv docker==6.0.0 # via @@ -183,19 +181,21 @@ entrypoints==0.4 # via altair execnet==1.9.0 # via pytest-xdist -executing==0.10.0 +executing==1.1.0 # via stack-data -fastapi==0.79.1 +fastapi==0.85.0 # via feast (setup.py) -fastavro==1.6.0 +fastavro==1.6.1 # via # feast (setup.py) # pandavro -fastjsonschema==2.16.1 +fastjsonschema==2.16.2 # via nbformat filelock==3.8.0 - # via virtualenv -firebase-admin==5.2.0 + # via + # snowflake-connector-python + # virtualenv +firebase-admin==5.4.0 # via feast (setup.py) fissix==21.11.13 # via bowler @@ -215,20 +215,21 @@ gcsfs==2022.1.0 # via feast (setup.py) geomet==0.2.1.post1 # via cassandra-driver -google-api-core[grpc]==2.8.2 +google-api-core[grpc]==2.10.1 # via # feast (setup.py) # firebase-admin # google-api-python-client # google-cloud-bigquery # google-cloud-bigquery-storage + # google-cloud-bigtable # google-cloud-core # google-cloud-datastore # google-cloud-firestore # google-cloud-storage -google-api-python-client==2.57.0 +google-api-python-client==2.63.0 # via firebase-admin -google-auth==2.10.0 +google-auth==2.12.0 # via # gcsfs # google-api-core @@ -240,63 +241,68 @@ google-auth==2.10.0 # kubernetes google-auth-httplib2==0.1.0 # via google-api-python-client -google-auth-oauthlib==0.5.2 +google-auth-oauthlib==0.5.3 # via gcsfs -google-cloud-bigquery[pandas]==3.3.2 +google-cloud-bigquery[pandas]==3.3.3 # via feast (setup.py) -google-cloud-bigquery-storage==2.14.2 +google-cloud-bigquery-storage==2.16.1 # via # feast (setup.py) # google-cloud-bigquery +google-cloud-bigtable==2.12.0 + # via feast (setup.py) google-cloud-core==2.3.2 # via # google-cloud-bigquery + # google-cloud-bigtable # google-cloud-datastore # google-cloud-firestore # google-cloud-storage google-cloud-datastore==2.8.1 # via feast (setup.py) -google-cloud-firestore==2.6.1 +google-cloud-firestore==2.7.0 # via firebase-admin google-cloud-storage==2.5.0 # via # feast (setup.py) # firebase-admin # gcsfs -google-crc32c==1.3.0 +google-crc32c==1.5.0 # via google-resumable-media -google-resumable-media==2.3.3 +google-resumable-media==2.4.0 # via # google-cloud-bigquery # google-cloud-storage -googleapis-common-protos==1.56.4 +googleapis-common-protos[grpc]==1.56.4 # via # feast (setup.py) # google-api-core + # grpc-google-iam-v1 # grpcio-status - # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -greenlet==1.1.2 - # via sqlalchemy -grpcio==1.47.0 +grpc-google-iam-v1==0.12.4 + # via google-cloud-bigtable +grpcio==1.49.1 # via # feast (setup.py) # google-api-core # google-cloud-bigquery + # googleapis-common-protos + # grpc-google-iam-v1 # grpcio-reflection # grpcio-status # grpcio-testing # grpcio-tools -grpcio-reflection==1.47.0 +grpcio-reflection==1.49.1 # via feast (setup.py) -grpcio-status==1.47.0 +grpcio-status==1.49.1 # via google-api-core -grpcio-testing==1.47.0 +grpcio-testing==1.49.1 # via feast (setup.py) -grpcio-tools==1.47.0 +grpcio-tools==1.49.1 # via feast (setup.py) -h11==0.13.0 +h11==0.14.0 # via uvicorn happybase==1.2.0 # via feast (setup.py) @@ -306,11 +312,11 @@ httplib2==0.20.4 # via # google-api-python-client # google-auth-httplib2 -httptools==0.4.0 +httptools==0.5.0 # via uvicorn -identify==2.5.3 +identify==2.5.5 # via pre-commit -idna==3.3 +idna==3.4 # via # anyio # requests @@ -322,7 +328,7 @@ importlib-metadata==4.12.0 # via great-expectations iniconfig==1.1.1 # via pytest -ipython==8.4.0 +ipython==8.5.0 # via great-expectations isodate==0.6.1 # via msrest @@ -345,7 +351,7 @@ jsonpatch==1.32 # via great-expectations jsonpointer==2.3 # via jsonpatch -jsonschema==4.13.0 +jsonschema==4.16.0 # via # altair # feast (setup.py) @@ -377,7 +383,7 @@ moreorless==0.4.0 # via bowler moto==3.1.18 # via feast (setup.py) -msal==1.18.0 +msal==1.19.0 # via # azure-identity # msal-extensions @@ -397,7 +403,7 @@ multidict==6.0.2 # yarl multiprocess==0.70.13 # via bytewax -mypy==0.971 +mypy==0.981 # via # feast (setup.py) # sqlalchemy @@ -409,11 +415,11 @@ mypy-protobuf==3.1 # via feast (setup.py) mysqlclient==2.1.1 # via feast (setup.py) -nbformat==5.4.0 +nbformat==5.6.1 # via great-expectations nodeenv==1.7.0 # via pre-commit -numpy==1.23.2 +numpy==1.23.3 # via # altair # db-dtypes @@ -439,7 +445,7 @@ packaging==21.3 # pytest # redis # sphinx -pandas==1.4.3 +pandas==1.4.4 # via # altair # db-dtypes @@ -454,7 +460,7 @@ parso==0.8.3 # via jedi partd==1.3.0 # via dask -pathspec==0.9.0 +pathspec==0.10.1 # via black pbr==5.10.0 # via mock @@ -478,21 +484,23 @@ portalocker==2.5.1 # via msal-extensions pre-commit==2.20.0 # via feast (setup.py) -prompt-toolkit==3.0.30 +prompt-toolkit==3.0.31 # via ipython -proto-plus==1.22.0 +proto-plus==1.22.1 # via # feast (setup.py) # google-cloud-bigquery # google-cloud-bigquery-storage + # google-cloud-bigtable # google-cloud-datastore # google-cloud-firestore -protobuf==3.20.2 +protobuf==4.21.7 # via # feast (setup.py) # google-api-core # google-cloud-bigquery # google-cloud-bigquery-storage + # google-cloud-bigtable # google-cloud-datastore # google-cloud-firestore # googleapis-common-protos @@ -502,7 +510,6 @@ protobuf==3.20.2 # grpcio-tools # mypy-protobuf # proto-plus - # tensorflow-metadata psutil==5.9.0 # via feast (setup.py) psycopg2-binary==2.9.3 @@ -519,7 +526,7 @@ py-cpuinfo==8.0.0 # via pytest-benchmark py4j==0.10.9.5 # via pyspark -pyarrow==6.0.1 +pyarrow==8.0.0 # via # db-dtypes # feast (setup.py) @@ -539,7 +546,7 @@ pycparser==2.21 # via cffi pycryptodomex==3.15.0 # via snowflake-connector-python -pydantic==1.9.2 +pydantic==1.10.2 # via # fastapi # feast (setup.py) @@ -550,7 +557,7 @@ pygments==2.13.0 # feast (setup.py) # ipython # sphinx -pyjwt[crypto]==2.4.0 +pyjwt[crypto]==2.5.0 # via # adal # msal @@ -562,7 +569,9 @@ pymysql==1.0.2 pyodbc==4.0.34 # via feast (setup.py) pyopenssl==22.0.0 - # via snowflake-connector-python + # via + # feast (setup.py) + # snowflake-connector-python pyparsing==2.4.7 # via # great-expectations @@ -572,7 +581,7 @@ pyrsistent==0.18.1 # via jsonschema pyspark==3.3.0 # via feast (setup.py) -pytest==7.1.2 +pytest==7.1.3 # via # feast (setup.py) # pytest-benchmark @@ -585,7 +594,7 @@ pytest==7.1.2 # pytest-xdist pytest-benchmark==3.4.1 # via feast (setup.py) -pytest-cov==3.0.0 +pytest-cov==4.0.0 # via feast (setup.py) pytest-forked==1.4.0 # via pytest-xdist @@ -608,7 +617,7 @@ python-dateutil==2.8.2 # kubernetes # moto # pandas -python-dotenv==0.20.0 +python-dotenv==0.21.0 # via uvicorn pytz==2022.2.1 # via @@ -660,15 +669,15 @@ responses==0.21.0 # via moto rsa==4.9 # via google-auth -ruamel.yaml==0.17.17 +ruamel-yaml==0.17.17 # via great-expectations -ruamel.yaml.clib==0.2.6 - # via ruamel.yaml +ruamel-yaml-clib==0.2.6 + # via ruamel-yaml s3fs==2022.1.0 # via feast (setup.py) s3transfer==0.5.2 # via boto3 -scipy==1.9.0 +scipy==1.9.1 # via great-expectations six==1.16.0 # via @@ -686,11 +695,11 @@ six==1.16.0 # msrestazure # pandavro # python-dateutil -sniffio==1.2.0 +sniffio==1.3.0 # via anyio snowballstemmer==2.2.0 # via sphinx -snowflake-connector-python[pandas]==2.7.8 +snowflake-connector-python[pandas]==2.8.0 # via feast (setup.py) sphinx==4.3.2 # via @@ -710,23 +719,21 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.5 # via sphinx -sqlalchemy[mypy]==1.4.40 +sqlalchemy[mypy]==1.4.41 # via feast (setup.py) -sqlalchemy2-stubs==0.0.2a25 +sqlalchemy2-stubs==0.0.2a27 # via sqlalchemy -stack-data==0.4.0 +stack-data==0.5.1 # via ipython -starlette==0.19.1 +starlette==0.20.4 # via fastapi tabulate==0.8.10 # via feast (setup.py) -tenacity==8.0.1 +tenacity==8.1.0 # via feast (setup.py) -tensorflow-metadata==1.9.0 - # via feast (setup.py) -termcolor==1.1.0 +termcolor==2.0.1 # via great-expectations -testcontainers==3.6.1 +testcontainers==3.7.0 # via feast (setup.py) thriftpy2==0.4.14 # via happybase @@ -747,21 +754,23 @@ toolz==0.12.0 # altair # dask # partd -tqdm==4.64.0 +tqdm==4.64.1 # via # feast (setup.py) # great-expectations -traitlets==5.3.0 +traitlets==5.4.0 # via # ipython # jupyter-core # matplotlib-inline # nbformat -trino==0.315.0 +trino==0.316.0 # via feast (setup.py) typeguard==2.13.3 # via feast (setup.py) -types-protobuf==3.19.22 +types-cryptography==3.3.23 + # via pyjwt +types-protobuf==3.20.4 # via # feast (setup.py) # mypy-protobuf @@ -771,17 +780,17 @@ types-python-dateutil==2.8.19 # via feast (setup.py) types-pytz==2022.2.1.0 # via feast (setup.py) -types-pyyaml==6.0.11 +types-pyyaml==6.0.12 # via feast (setup.py) -types-redis==4.3.14 +types-redis==4.3.21 # via feast (setup.py) -types-requests==2.28.9 +types-requests==2.28.11 # via feast (setup.py) -types-setuptools==65.1.0 +types-setuptools==65.4.0.0 # via feast (setup.py) types-tabulate==0.8.11 # via feast (setup.py) -types-urllib3==1.26.23 +types-urllib3==1.26.25 # via types-requests typing-extensions==4.3.0 # via @@ -791,15 +800,16 @@ typing-extensions==4.3.0 # great-expectations # mypy # pydantic + # snowflake-connector-python # sqlalchemy2-stubs # starlette -tzdata==2022.2 +tzdata==2022.4 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations uritemplate==4.1.1 # via google-api-python-client -urllib3==1.26.11 +urllib3==1.26.12 # via # botocore # docker @@ -809,19 +819,20 @@ urllib3==1.26.11 # minio # requests # responses -uvicorn[standard]==0.18.2 + # snowflake-connector-python +uvicorn[standard]==0.18.3 # via feast (setup.py) -uvloop==0.16.0 +uvloop==0.17.0 # via uvicorn -virtualenv==20.16.3 +virtualenv==20.16.5 # via pre-commit volatile==2.1.0 # via bowler -watchfiles==0.16.1 +watchfiles==0.17.0 # via uvicorn wcwidth==0.2.5 # via prompt-toolkit -websocket-client==1.3.3 +websocket-client==1.4.1 # via # docker # kubernetes diff --git a/sdk/python/requirements/py3.9-requirements.txt b/sdk/python/requirements/py3.9-requirements.txt index 395302a9f9..3b5135b602 100644 --- a/sdk/python/requirements/py3.9-requirements.txt +++ b/sdk/python/requirements/py3.9-requirements.txt @@ -4,8 +4,6 @@ # # pip-compile --output-file=sdk/python/requirements/py3.9-requirements.txt # -absl-py==1.2.0 - # via tensorflow-metadata anyio==3.6.1 # via # starlette @@ -20,9 +18,9 @@ bowler==0.9.0 # via feast (setup.py) cachetools==5.2.0 # via google-auth -certifi==2022.6.15 +certifi==2022.9.24 # via requests -charset-normalizer==2.1.0 +charset-normalizer==2.1.1 # via requests click==8.1.3 # via @@ -30,7 +28,7 @@ click==8.1.3 # feast (setup.py) # moreorless # uvicorn -cloudpickle==2.1.0 +cloudpickle==2.2.0 # via dask colorama==0.4.5 # via feast (setup.py) @@ -38,44 +36,41 @@ dask==2022.1.1 # via feast (setup.py) dill==0.3.5.1 # via feast (setup.py) -fastapi==0.79.1 +fastapi==0.85.0 # via feast (setup.py) -fastavro==1.6.0 +fastavro==1.6.1 # via # feast (setup.py) # pandavro fissix==21.11.13 # via bowler -fsspec==2022.7.1 +fsspec==2022.8.2 # via dask -google-api-core==2.8.2 +google-api-core==2.10.1 # via feast (setup.py) -google-auth==2.10.0 +google-auth==2.12.0 # via google-api-core googleapis-common-protos==1.56.4 # via # feast (setup.py) # google-api-core - # tensorflow-metadata -greenlet==1.1.2 - # via sqlalchemy -grpcio==1.47.0 +grpcio==1.49.1 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.47.0 +grpcio-reflection==1.49.1 # via feast (setup.py) -h11==0.13.0 +h11==0.14.0 # via uvicorn -httptools==0.4.0 +httptools==0.5.0 # via uvicorn -idna==3.3 +idna==3.4 # via # anyio # requests jinja2==3.1.2 # via feast (setup.py) -jsonschema==4.13.0 +jsonschema==4.16.0 # via feast (setup.py) locket==1.0.0 # via partd @@ -85,11 +80,11 @@ mmh3==3.0.0 # via feast (setup.py) moreorless==0.4.0 # via bowler -mypy==0.971 +mypy==0.981 # via sqlalchemy mypy-extensions==0.4.3 # via mypy -numpy==1.23.2 +numpy==1.23.3 # via # feast (setup.py) # pandas @@ -97,7 +92,7 @@ numpy==1.23.2 # pyarrow packaging==21.3 # via dask -pandas==1.4.3 +pandas==1.5.0 # via # feast (setup.py) # pandavro @@ -105,16 +100,15 @@ pandavro==1.5.2 # via feast (setup.py) partd==1.3.0 # via dask -proto-plus==1.22.0 +proto-plus==1.22.1 # via feast (setup.py) -protobuf==3.20.2 +protobuf==4.21.7 # via # feast (setup.py) # google-api-core # googleapis-common-protos # grpcio-reflection # proto-plus - # tensorflow-metadata pyarrow==8.0.0 # via feast (setup.py) pyasn1==0.4.8 @@ -123,7 +117,7 @@ pyasn1==0.4.8 # rsa pyasn1-modules==0.2.8 # via google-auth -pydantic==1.9.2 +pydantic==1.10.2 # via # fastapi # feast (setup.py) @@ -135,7 +129,7 @@ pyrsistent==0.18.1 # via jsonschema python-dateutil==2.8.2 # via pandas -python-dotenv==0.20.0 +python-dotenv==0.21.0 # via uvicorn pytz==2022.2.1 # via pandas @@ -154,19 +148,17 @@ six==1.16.0 # grpcio # pandavro # python-dateutil -sniffio==1.2.0 +sniffio==1.3.0 # via anyio -sqlalchemy[mypy]==1.4.40 +sqlalchemy[mypy]==1.4.41 # via feast (setup.py) -sqlalchemy2-stubs==0.0.2a25 +sqlalchemy2-stubs==0.0.2a27 # via sqlalchemy -starlette==0.19.1 +starlette==0.20.4 # via fastapi tabulate==0.8.10 # via feast (setup.py) -tenacity==8.0.1 - # via feast (setup.py) -tensorflow-metadata==1.9.0 +tenacity==8.1.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) @@ -176,7 +168,7 @@ toolz==0.12.0 # via # dask # partd -tqdm==4.64.0 +tqdm==4.64.1 # via feast (setup.py) typeguard==2.13.3 # via feast (setup.py) @@ -186,15 +178,15 @@ typing-extensions==4.3.0 # pydantic # sqlalchemy2-stubs # starlette -urllib3==1.26.11 +urllib3==1.26.12 # via requests -uvicorn[standard]==0.18.2 +uvicorn[standard]==0.18.3 # via feast (setup.py) -uvloop==0.16.0 +uvloop==0.17.0 # via uvicorn volatile==2.1.0 # via bowler -watchfiles==0.16.1 +watchfiles==0.17.0 # via uvicorn websockets==10.3 # via uvicorn diff --git a/sdk/python/tests/integration/feature_repos/repo_configuration.py b/sdk/python/tests/integration/feature_repos/repo_configuration.py index 708d9c0a14..174b0b91ad 100644 --- a/sdk/python/tests/integration/feature_repos/repo_configuration.py +++ b/sdk/python/tests/integration/feature_repos/repo_configuration.py @@ -51,6 +51,9 @@ create_order_feature_view, create_pushable_feature_view, ) +from tests.integration.feature_repos.universal.online_store.bigtable import ( + BigtableOnlineStoreCreator, +) from tests.integration.feature_repos.universal.online_store.datastore import ( DatastoreOnlineStoreCreator, ) @@ -84,6 +87,12 @@ "schema": "ONLINE", } +BIGTABLE_CONFIG = { + "type": "bigtable", + "project_id": os.getenv("GCLOUD_PROJECT", "kf-feast"), + "instance": os.getenv("BIGTABLE_INSTANCE_ID", "feast-integration-tests"), +} + OFFLINE_STORE_TO_PROVIDER_CONFIG: Dict[str, DataSourceCreator] = { "file": ("local", FileDataSourceCreator), "bigquery": ("gcp", BigQueryDataSourceCreator), @@ -115,6 +124,7 @@ AVAILABLE_ONLINE_STORES["dynamodb"] = (DYNAMO_CONFIG, None) AVAILABLE_ONLINE_STORES["datastore"] = ("datastore", None) AVAILABLE_ONLINE_STORES["snowflake"] = (SNOWFLAKE_CONFIG, None) + AVAILABLE_ONLINE_STORES["bigtable"] = (BIGTABLE_CONFIG, None) full_repo_configs_module = os.environ.get(FULL_REPO_CONFIGS_MODULE_ENV_NAME) @@ -161,6 +171,7 @@ "redis": (REDIS_CONFIG, RedisOnlineStoreCreator), "dynamodb": (DYNAMO_CONFIG, DynamoDBOnlineStoreCreator), "datastore": ("datastore", DatastoreOnlineStoreCreator), + "bigtable": ("bigtable", BigtableOnlineStoreCreator), } for key, replacement in replacements.items(): diff --git a/sdk/python/tests/integration/feature_repos/universal/online_store/bigtable.py b/sdk/python/tests/integration/feature_repos/universal/online_store/bigtable.py new file mode 100644 index 0000000000..c06143e245 --- /dev/null +++ b/sdk/python/tests/integration/feature_repos/universal/online_store/bigtable.py @@ -0,0 +1,47 @@ +import os +from typing import Dict + +from google.cloud import bigtable +from testcontainers.core.container import DockerContainer +from testcontainers.core.waiting_utils import wait_for_logs + +from tests.integration.feature_repos.universal.online_store_creator import ( + OnlineStoreCreator, +) + + +class BigtableOnlineStoreCreator(OnlineStoreCreator): + gcp_project = "test-project" + host = "0.0.0.0" + port = "8086" + bt_instance = "test-instance" + + def __init__(self, project_name: str, **kwargs): + super().__init__(project_name) + self.container = ( + DockerContainer( + "gcr.io/google.com/cloudsdktool/cloud-sdk:380.0.0-emulators" + ) + .with_command( + f"gcloud beta emulators bigtable start --project {self.gcp_project} --host-port {self.host}:{self.port}" + ) + .with_exposed_ports(self.port) + ) + + def create_online_store(self) -> Dict[str, str]: + self.container.start() + log_string_to_wait_for = r"\[bigtable\] Cloud Bigtable emulator running" + wait_for_logs( + container=self.container, predicate=log_string_to_wait_for, timeout=10 + ) + exposed_port = self.container.get_exposed_port(self.port) + os.environ[bigtable.client.BIGTABLE_EMULATOR] = f"{self.host}:{exposed_port}" + return { + "type": "bigtable", + "project_id": self.gcp_project, + "instance": self.bt_instance, + } + + def teardown(self): + del os.environ[bigtable.client.BIGTABLE_EMULATOR] + self.container.stop() diff --git a/setup.py b/setup.py index 59c362ff9f..2b09bcbe94 100644 --- a/setup.py +++ b/setup.py @@ -59,7 +59,7 @@ "mmh3", "numpy>=1.22,<3", "pandas>=1.4.3,<2", - "pandavro==1.5.*", # For some reason pandavro higher than 1.5.* only support pandas less than 1.3. + "pandavro==1.5.*", # For some reason pandavro higher than 1.5.* only support pandas less than 1.3. "protobuf<5,>3", "proto-plus>=1.20.0,<2", "pyarrow>=4,<9", @@ -74,7 +74,6 @@ "typeguard", "fastapi>=0.68.0,<1", "uvicorn[standard]>=0.14.0,<1", - "tensorflow-metadata>=1.0.0,<2.0.0", "dask>=2021.*,<2022.02.0", "bowler", # Needed for automatic repo upgrades ] @@ -84,6 +83,7 @@ "google-cloud-bigquery-storage >= 2.0.0,<3", "google-cloud-datastore>=2.1.*,<3", "google-cloud-storage>=1.34.*,<3", + "google-cloud-bigtable>=2.11.*,<3", ] REDIS_REQUIRED = [ @@ -97,6 +97,10 @@ SNOWFLAKE_REQUIRED = [ "snowflake-connector-python[pandas]>=2.7.3,<3", + # `pyOpenSSL==22.1.0` requires `cryptography<39,>=38.0.0`, which is incompatible + # with `snowflake-connector-python[pandas]==2.8.0`, which depends on + # `cryptography<37.0.0,>=3.1.0`. + "pyOpenSSL<22.1.0", ] SPARK_REQUIRED = [ @@ -111,11 +115,7 @@ "psycopg2-binary>=2.8.3,<3", ] -MYSQL_REQUIRED = [ - "mysqlclient", - "pymysql", - "types-PyMySQL" -] +MYSQL_REQUIRED = ["mysqlclient", "pymysql", "types-PyMySQL"] HBASE_REQUIRED = [ "happybase>=1.2.0,<3", @@ -131,15 +131,13 @@ "cffi==1.15.*,<2", ] -AZURE_REQUIRED = ( - [ - "azure-storage-blob>=0.37.0", - "azure-identity>=1.6.1", - "SQLAlchemy>=1.4.19", - "pyodbc>=4.0.30", - "pymssql", - ] -) +AZURE_REQUIRED = [ + "azure-storage-blob>=0.37.0", + "azure-identity>=1.6.1", + "SQLAlchemy>=1.4.19", + "pyodbc>=4.0.30", + "pymssql", +] CI_REQUIRED = ( [ @@ -152,7 +150,7 @@ "grpcio-testing>=1.47.0", "minio==7.1.0", "mock==2.0.0", - "moto", + "moto<4", "mypy>=0.931", "mypy-protobuf==3.1", "avro==1.10.0",