Skip to content

Commit

Permalink
Merge branch 'kevinliu/test' into kevinjqliu/fix-schema-comparison
Browse files Browse the repository at this point in the history
  • Loading branch information
kevinjqliu committed Mar 1, 2024
2 parents 672efe2 + 10adb1c commit 18c6c53
Show file tree
Hide file tree
Showing 6 changed files with 87 additions and 358 deletions.
9 changes: 0 additions & 9 deletions mkdocs/docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,15 +265,6 @@ catalog:
The In-Memory catalog uses in-memory data-structures to store information.
This is useful for test, demo, and playground. Do not use in production as the data is not persisted.

While you can specify In-Memory catalog in the configuration file like this, it is not recommended since information is only persisted for the duration of the function call.

```yaml
catalog:
default:
type: in_memory
warehouse: /tmp/warehouse # default warehouse location
```

# Concurrency

PyIceberg uses multiple threads to parallelize operations. The number of workers can be configured by supplying a `max-workers` entry in the configuration file, or by setting the `PYICEBERG_MAX_WORKERS` environment variable. The default value depends on the system hardware and Python version. See [the Python documentation](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) for more details.
8 changes: 0 additions & 8 deletions pyiceberg/catalog/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,6 @@ class CatalogType(Enum):
GLUE = "glue"
DYNAMODB = "dynamodb"
SQL = "sql"
IN_MEMORY = "in_memory"


def load_rest(name: str, conf: Properties) -> Catalog:
Expand Down Expand Up @@ -144,19 +143,12 @@ def load_sql(name: str, conf: Properties) -> Catalog:
) from exc


def load_in_memory(name: str, conf: Properties) -> Catalog:
from pyiceberg.catalog.in_memory import InMemoryCatalog

return InMemoryCatalog(name, **conf)


AVAILABLE_CATALOGS: dict[CatalogType, Callable[[str, Properties], Catalog]] = {
CatalogType.REST: load_rest,
CatalogType.HIVE: load_hive,
CatalogType.GLUE: load_glue,
CatalogType.DYNAMODB: load_dynamodb,
CatalogType.SQL: load_sql,
CatalogType.IN_MEMORY: load_in_memory,
}


Expand Down
246 changes: 0 additions & 246 deletions pyiceberg/catalog/in_memory.py

This file was deleted.

2 changes: 1 addition & 1 deletion pyiceberg/cli/output.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ def describe_properties(self, properties: Properties) -> None:
Console().print(output_table)

def text(self, response: str) -> None:
Console().print(response)
Console(soft_wrap=True).print(response)

def schema(self, schema: Schema) -> None:
output_table = self._table
Expand Down
Loading

0 comments on commit 18c6c53

Please sign in to comment.