fix: allow AstraDB to prevent indexing on metadata columns with long text #3003
Conversation
potter-potter
changed the title
| @@ -55,7 +55,8 @@ PYTHONPATH=. ./unstructured/ingest/main.py \ | |||
| --token "$ASTRA_DB_TOKEN" \ | |||
| --api-endpoint "$ASTRA_DB_ENDPOINT" \ | |||
| --collection-name "$COLLECTION_NAME" \ | |||
| --embedding-dimension "$EMBEDDING_DIMENSION" | |||
| --embedding-dimension "$EMBEDDING_DIMENSION" \ | |||
| --requested-indexing-policy '{"deny": "metadata"}' | |||
There was a problem hiding this comment.
@erichare Can you shed some light on why this --requested-indexing-policy results in an error. Maybe there is better json to test this attribute.
There was a problem hiding this comment.
@erichare Switched this pr from draft to normal. Just in case you couldn't access the draft.
There was a problem hiding this comment.
@potter-potter apologies about the delay - i was out most of yesterday. Let me test that shortly - offhand that looks right, assuming that it gets converted into a dictionary correctly, but its kind of a deeply nested object on the Astra Data API side and so i may have missed a top level attribute. Checking it out shortly...
There was a problem hiding this comment.
@potter-potter i opened a small bugfix pr to merge into this branch (didn't have write access to this). It was my mistake - i was passing a single field, but the API expects a list of fields even if it's just a single one.
There was a problem hiding this comment.
@erichare your fix looks good.
Thanks to @erichare from AstraDB
Adds support for specifying the indexing options for various columns in Astra DB, allowing users to avoid a situation where long text columns are by-default indexed.
Changes to: test_unstructured_ingest/python/test-ingest-astra-output.py are forward looking from AstraDB