[ENH] Arrow-backed block #1820
Merged
[ENH] Arrow-backed block #1820
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
HammadB
commented
Mar 6, 2024
| let value = data.data.column(2); | ||
| // TODO: This should be binary search | ||
| for i in 0..prefix.len() { | ||
| if prefix |
There was a problem hiding this comment.
i'll clean up this unwrapping when i change this to binary search in a future PR
This was referenced Mar 6, 2024
Merged
Merged
Merged
HammadB
force-pushed
the
hammad/add_blockfile_provider
branch
from
3c8b0f4 to
cbfd563
Compare
HammadB
force-pushed
the
hammad/add_block
branch
from
f1159f4 to
c453d2b
Compare
beggers
approved these changes
Mar 7, 2024
HammadB
force-pushed
the
hammad/add_block
branch
from
c453d2b to
86c9e49
Compare
HammadB
force-pushed
the
hammad/add_block
branch
from
86c9e49 to
2ec00ad
Compare
This was referenced Mar 8, 2024
atroyn
added a commit
to csbasil/chroma
that referenced
this pull request
Apr 3, 2024
commit 193988d268f472f792fe27bfa3c0fb92b333791a Author: Mihir1003 <[email protected]> Date: Wed Apr 3 04:43:12 2024 +0530 [ENH] Support langchain embedding functions with chroma (#1880) ## Description of changes *Summarize the changes made by this PR.* - New functionality - Adding a function to create a chroma langchain embedding interface. This interface acts as a bridge between the langchain embedding function and the chroma custom embedding function. - Native Langchain multimodal support: The PR adds a Passthrough data loader that lets langchain users use OpenClip and other multi-modal embedding functions from langchain with chroma without having to handle storing images themselves. ## Test plan *How are these changes tested?* - installing chroma as an editable package locally and passing langchain integration tests - pytest test_api.py test_client.py succeeds ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* Co-authored-by: Anton Troynikov <[email protected]> commit fdfda56d2e83fea5df1343ef3df0d43528bd27a8 Author: Liquan Pei <[email protected]> Date: Tue Apr 2 10:02:41 2024 -0700 [CLN] Clean up unused imports (#1959) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR cleans up unused import and addresses some warnings by cargo build and cargo test. - Unused variables are not touched. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit d3f61b145eeb0f4108f1b1a1b5919a36c0e3b8b6 Author: James <[email protected]> Date: Tue Apr 2 04:08:43 2024 +0100 [ENH] Create `RoboflowEmbeddingFunction` (#1434) ## Description of changes This PR adds a new `RoboflowEmbeddingFunction` with which a user can calculate CLIP text and image embeddings using [Roboflow Inference](https://inference.roboflow.com). ## Test plan You can test the embedding function using the following code: ```python import chromadb import os from chromadb.utils.embedding_functions import RoboflowEmbeddingFunction import uuid from PIL import Image client = chromadb.PersistentClient(path="database") collection = client.create_collection(name="images", metadata={"hnsw:space": "cosine"}) # collection = client.get_collection(name="images") IMAGE_DIR = "images/train/images/" SERVER_URL = "https://infer.roboflow.com" API_KEY = "" results = [] ef = RoboflowEmbeddingFunction(API_KEY) documents = [os.path.join(IMAGE_DIR, img) for img in os.listdir(IMAGE_DIR)] embeddings = ef(images = [img for img in documents]) ids = [str(uuid.uuid4()) for _ in range(len(documents))] print(len(embeddings)) collection.add( embeddings=embeddings, documents=documents, ids=ids, ) query = ef(prompt = "baseball") results = collection.query( query_embeddings=query, n_results=3 ) top_result = results["documents"] for i in top_result: print(i) ``` You will need a [Roboflow API key](https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key) - [X] Tests pass locally with `pytest` for python, `yarn test` for js ## Documentation Changes I will file a PR to the `chroma-core/docs` repository with documentation. --------- Co-authored-by: Anton Troynikov <[email protected]> commit 1cd2ced9577369171c6a19ffa9e1a5de5b23ace1 Author: Liquan Pei <[email protected]> Date: Mon Apr 1 16:47:51 2024 -0700 [ENH] Add memberlist for compaction service (#1949) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR adds memberlist consumption for the compaction service and filter collections based on memberlist using Rendezvous assignment policy. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust - [ ] Manual local integration testing with Tilt ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 978547c32d52da81b6991827a4808144fc1166ea Author: Liquan Pei <[email protected]> Date: Mon Apr 1 12:35:04 2024 -0700 [BUG] Fix memberlist manager to only list memberlist with the provide… (#1956) …d memberlist name ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - This PR fixes memberlist manager to only list memberlist with the provided memberlist name. - New functionality - ... ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit ab05830948d6d459ac4b39c943e5ff81a9b5aced Author: Liquan Pei <[email protected]> Date: Mon Apr 1 11:37:49 2024 -0700 [CLN] Change Task to CompactionJob for higher level compaction work d… (#1955) …escription ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - This PR changes Task to CompactionJob for higher level compaction work description for better code readability. - New functionality - ... ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit d8c1055d6f6709bc7fa2e100c553b50d162cecf9 Author: Liquan Pei <[email protected]> Date: Mon Apr 1 10:25:15 2024 -0700 [CLN] Remove unused worker_entrypoint (#1954) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - This PR cleans up unused worker_entrypoint - New functionality - ... ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit b6e86a60a3484314c71c7612768830ffd58423e1 Author: Liquan Pei <[email protected]> Date: Fri Mar 29 11:16:47 2024 -0700 [ENH] Compaction service setup (#1945) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR adds the necessary artifacts to setup compaction service. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 6af795b059b9d606c6d96d2de20d1f1af1b774fe Author: Liquan Pei <[email protected]> Date: Fri Mar 29 07:45:18 2024 -0700 [ENH] Compaction manager (#1944) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR adds a compaction manager that gets a list of collections to compact and orchestrate the execution of compaction tasks. - It uses FuturesUnordered to await the compaction tasks and allow them to run in parallel. - We will add retry and timeout in follow up PRs. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 2c416aa655f4049ce41eea6bc1a12b77ed676f99 Author: Liquan Pei <[email protected]> Date: Thu Mar 28 23:09:22 2024 -0700 [ENH] Add compaction service memberlist propagation (#1946) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR adds memberlist manager for compaction service. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 99f05494f2c261028e1cf1f6f8741a7670a1b04c Author: Ben Eggers <[email protected]> Date: Thu Mar 28 17:04:08 2024 -0700 [CLN] Create public if not exists (#1943) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Usually local postgres has the schema already, but staging currently doesn't. We should just create if not exists. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit a78aedee895dd813ed7f8a372788346bd92f2f57 Author: Ben Eggers <[email protected]> Date: Thu Mar 28 14:43:15 2024 -0700 [CLN] Stop clearing the sysdb on each deploy (#1941) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - ... ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 4da1a6ccdbc6f249b3d1429af1320cf22b240892 Author: Hammad Bashir <[email protected]> Date: Thu Mar 28 14:22:20 2024 -0700 [ENH] Remove seq_id from protos, record and result types. Refactor rust to use LogRecord / OperationRecord (#1935) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Remove the seq_id concept from our record and result types. This is unused and not needed. The python segments still internally use this though. - Refactor rust to use LogRecord/OperationRecord - Remove Segment Manager and Ingestor from rust, as well as other pulsar based paraphenelia such as message_id. This resulted in the /ingest module being removed altogether. - New functionality - None ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 1d77f99edc43ade4d3490a4638b5e5d802050303 Author: Trayan Azarov <[email protected]> Date: Thu Mar 28 19:59:01 2024 +0200 [ENH]: Ollama embedding function (#1813) ## Description of changes *Summarize the changes made by this PR.* - New functionality - New Ollama embedding function (Python and JS) - Example of how to run Ollama with the embedding function ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js ## Documentation Changes chroma-core/docs#222 commit 5f3f141c55103ae949f726ccd88d1568e592dc69 Author: Hammad Bashir <[email protected]> Date: Wed Mar 27 17:51:57 2024 -0700 [CLN] Restructure EmbeddingRecord -> LogRecord. Use 'log_offset' as term instead of 'id' (#1934) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - This PR follows #1933 and restructures EmbeddingRecord to be a nested type around OperationRecord. We rename EmbeddingRecord to be LogRecord. A LogRecord is a model of an OperationRecord stored on a log, and just keeps a log_offset The log is assumed to be per-collection. - Rename the Logservices use of "ID" to instead be "log_offset" to standardize on the "log_offset" terminology. - Rename RecordLog -> LogRecord for conceptual clarity and to align with other types. - Fixes a proto style guide violation where we used camelcase for fieldnames - https://protobuf.dev/programming-guides/style/#message_and_field_names. - In GetColllectionCompactionInfo rename first_log_id to first_log_offset - which is much clearer. Also rename first_log_id_ts to first_log_offset. What is a id_ts? Confusing name. - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 1ce93c7e41344446dc18af7b27474b5d5bcd4a68 Author: Hammad Bashir <[email protected]> Date: Wed Mar 27 10:48:48 2024 -0700 [CLN] Rename SubmitEmbeddingRecord to OperationRecord. Remove Topic concept from Segment/Collection. (#1933) ## Description of changes This PR has two high-level aims - Clean up artifacts of our design that were implicitly coupled to pulsar. Namely topics and multiplexing. - Begin a rename of SubmitEmbeddingRecord, EmbeddingRecord to names that more accurately reflect their intents. I apologize for how large this PR is, but in practice, breaking something up like this is not really feasible AFAICT, unless we allow test-breaking stacked PRs... *Summarize the changes made by this PR.* - Improvements & Bug fixes - Renames SubmitEmbeddingRecord to OperationRecord in order to more correctly identify what it is - a record of an Operation (future PRs will rename EmbeddingRecord as well to LogRecord to make its intent clearer). - An OperationRecord does not need to store collection_id. This was an artifact of the pulsar log when we needed to demux data. We now improve the Ingest interface by presenting producers/consumers over logical log streams by collection. - Remove the concept of topic from the Producer/Consumer interfaces - it is no longer needed in a post pulsar-world. This also means Collection/Segment don't need to store Topic. - Removed the AssignmentPolicy concept. This only existed for multiplexing - which is not a concept without pulsar. - Update the Rust code with the topic field removed and with the OperationRecord naming. - Update Go code with the SysDB changes (No assignment policy + no log) no as well as the OperationRecord naming. - New functionality - None ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 739e942b165dd7fae7888ea34656d79e584e045f Author: Liquan Pei <[email protected]> Date: Tue Mar 26 17:01:24 2024 -0700 [ENH] DataChunk abstraction with PullLog, Group and BF (#1929) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR adds DataChunk abstraction - PullLog, Group and BF are reimplemented with DataChunk - Compaction orchestrator is implemented with DataChunk ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* --------- Co-authored-by: Hammad Bashir <[email protected]> commit b8dfc5b0f35a20f5049b1981d2885da2bae58242 Author: Hammad Bashir <[email protected]> Date: Tue Mar 26 15:44:39 2024 -0700 [CLN] Remove pulsar from python codebase (#1932) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Removes pulsar from the python codebase - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 6b68fd11950d0ab43cd5470851f31f26018b745e Author: Hammad Bashir <[email protected]> Date: Sun Mar 24 17:24:59 2024 -0700 [BUG] Fix as_nanos bad merge (#1928) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - #1927 accidentally reverted #1918 - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes none commit 53696308d918a21d3fd0796daac6d6e1256479a4 Author: Hammad Bashir <[email protected]> Date: Sun Mar 24 16:33:53 2024 -0700 [ENH] make hnsw query orchestrator use BF operator (#1927) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - This is a low-quality commit of wiring up the BF operator just to get things working e2e. We will rewrite a lot of this marginally when the chunk abstraction lands. - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit a7cf00d37fe7a25a769775a24095ac67cf1f45e0 Author: Hammad Bashir <[email protected]> Date: Sun Mar 24 16:25:31 2024 -0700 [BUG] BF operator should min(data, k) to prevent panic (#1926) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - The brute force operator should return min(data_size, k) - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 71865259f3fe8a335ad0bc987a7cf94c1724984a Author: Hammad Bashir <[email protected]> Date: Sun Mar 24 14:41:37 2024 -0700 [CLN] Clean up log error message (#1925) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - The error message here seems to have been copied from grpc sysdb and was mentioning segment which was confusing. - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 917fa0d14f07d75829d354ab8f6e7b9e6cb6af2c Author: Hammad Bashir <[email protected]> Date: Sun Mar 24 14:40:50 2024 -0700 [BUG] Rust server should wire up system (#1924) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Rust server boot code should wire up the system. - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 93b1f2a03de97f1f0c1c2313c0ab52e0c561940a Author: Hammad Bashir <[email protected]> Date: Sun Mar 24 14:39:05 2024 -0700 [BUG] Log service N+1 query (#1923) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - The rust log service makes an additional query when it doesn't need to if the timestamp is provided and we read less than a batch size. This means that we are done reading the log and don't need to fetch again. - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit ba7b52e00b164e0cdeb43bc42affbe5d10fd629f Author: Hammad Bashir <[email protected]> Date: Sun Mar 24 14:24:45 2024 -0700 [BUG] Log service incorrectly doesn't hydrate collection id (#1922) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Log service incorrectly did not return the collection ID of the record. I suspect that we meant to strip the collection id and rehydrate it into the proto for space reasons. I honored this intent and rehydrate the colleciton id. - Add some readme for running tests locally with PG - Fixed a bug in the log_service test where the input is mutated, which makes the source of truth have no collection id. This was passing when we incorrectly returned no collection id but was correctly failing now. I patched the test by cloning the records for a SOT. - For the test I fixed, the expected vs actual order was incorrect. - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 3a484552b2e1c69dd692c8f05cdc8b3a043fb68d Author: Liquan Pei <[email protected]> Date: Sun Mar 24 13:18:35 2024 -0700 [BUG] Use nano seconds as end_timestamp (#1918) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Match the timestamp with the LogService, which uses nanoseconds. - New functionality - ... ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 1c140bc536c4fe3438647adcff1d8dd9c6c58c69 Author: Ben Eggers <[email protected]> Date: Sat Mar 23 18:37:19 2024 -0700 build correctly in release mode commit b602cfc2998b4a01b885b8d9ef0a8b0a405e4813 Author: Ben Eggers <[email protected]> Date: Sat Mar 23 17:15:48 2024 -0700 [BUG] Point query service to the right logservice port (#1921) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Point the query service to port 50051 on the log service, not 50052. We only use 50052 for the external-facing service during testing since it conflicts with the sysdb on 50051. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 8c7cc1b537e380d3132982bfeeefb037e2c64897 Author: Ben Eggers <[email protected]> Date: Sat Mar 23 17:12:25 2024 -0700 Make tilt and docker faster (#1920) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Tighten up dockerfiles and tiltfile: Get rid of Alpine linux, make dockerfiles a little friendlier to cache and make the final images smaller, and tell tilt only to reload some of the dockerfiles on certain path reloads. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 650c09cd985dfe650cef81c7867174f3dabefb26 Author: Ben Eggers <[email protected]> Date: Fri Mar 22 17:30:57 2024 -0700 k8s commands override docker commands commit d66ff6b23a9134613b4c30c472550202f086e041 Author: Ben Eggers <[email protected]> Date: Fri Mar 22 15:55:40 2024 -0700 Rust worker now rebuilds in ~53s on my machine commit 08f23d209f600236e0c132ec4adf837753677760 Author: Ben Eggers <[email protected]> Date: Fri Mar 22 11:30:35 2024 -0700 run release query service commit 8f189d5de5197ba43f9bdab5d0da80aa9274e995 Author: Sumaiyah <[email protected]> Date: Fri Mar 22 16:28:00 2024 +0000 [ENH] Add optional kwargs when initialising SentenceTransformerEmbeddingFunction class (#1891) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Add optional kwargs for `SetenceTransformer` when initialising `SentenceTransformerEmbeddingFunction` class (Issue [#1857](https://github.com/chroma-core/chroma/issues/1857)) ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python - installing chroma as an editable package locally and testing with the code ```python import chromadb from chromadb.utils import embedding_functions sentence_transformer_ef = embedding_functions.SentenceTransformerEmbeddingFunction(prompts={"query": "query: ", "passage": "passage: "}) print(sentence_transformer_ef.models['all-MiniLM-L6-v2'].prompts) ``` returned ```bash {'query': 'query: ', 'passage': 'passage: '} ``` ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* I have added the documentation for `SentenceTransformerEmbeddingFunction` initialisation. Co-authored-by: sumaiyah <o##wJdTOvNkIC!1R@bQO> commit 468cb4cd04c3a5011716bedaae5a95519ef5a229 Author: Ben Eggers <[email protected]> Date: Fri Mar 22 09:22:36 2024 -0700 [BUG] remove query service build cache - breaks staging (#1917) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Remove the build cache for the query service. Right now it makes the stagnig query node recompile the binary. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit dd7707c62c7cc18e67d922fbcfb423f7cb513dda Author: Hammad Bashir <[email protected]> Date: Fri Mar 22 08:44:28 2024 -0700 [CLN] Add trace for memberlist update (#1912) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Add trace - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit e2ee34e63f574ca8052f8225be611accada54014 Author: Hammad Bashir <[email protected]> Date: Thu Mar 21 16:51:12 2024 -0700 [CHORE] Go Memberlist debug (#1913) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Add some debugs for memberlist in go - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 234d1902d83b1338d91b9447fdc5a7d5a1631918 Author: Hammad Bashir <[email protected]> Date: Thu Mar 21 16:03:18 2024 -0700 [BUG] Make memberlist name configurable (#1911) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Make memberlist label configurable - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 81375e82992b67d02d579d8bda1637975f0f0503 Author: Hammad Bashir <[email protected]> Date: Thu Mar 21 15:33:00 2024 -0700 [BUG] Rename memberlist to match new memberlist name (#1910) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - We renamed the memberlist to query-service-memberlist given the query/compactor split. - This aligns all the defaults with that name - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit f5e173d09bc0ec34bd9f1e90803c4fb57869643f Author: Hammad Bashir <[email protected]> Date: Thu Mar 21 14:49:37 2024 -0700 [ENH] Add brute force operator (#1907) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - / - New functionality - Adds a brute force operator type ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 7417f3de4841be16c3c4b878fee5700e4bc5d465 Author: Ben Eggers <[email protected]> Date: Thu Mar 21 13:49:38 2024 -0700 [ENH] Seed distributed sysdb (#1909) ## Description of changes *Summarize the changes made by this PR.* - New functionality - Seed distributed sysdb with default tenant and default db ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit e6f3aec9d37cc34f9b820e23737b0817a96a2029 Author: Liquan Pei <[email protected]> Date: Thu Mar 21 01:21:18 2024 -0700 [ENH] Implement the PullLog operator (#1906) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR implements the PullLog operator ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 3985032d6c7cb0b691aeff56f1367bb5e7bbfa1b Author: Hammad Bashir <[email protected]> Date: Wed Mar 20 15:43:33 2024 -0700 [ENH] Operator/Executor error handling (#1903) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Allows errors to be propagated from operators back to the original caller of the task. - New functionality - None ## Test plan *How are these changes tested?* Compile checks show that the error propagates as expected. - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 976aee3875993b6573f5f06eb9cd7c61b38f6f4a Author: Hammad Bashir <[email protected]> Date: Wed Mar 20 15:36:01 2024 -0700 [BLD] Auto rebuild protos if they change in rust (#1902) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Auto rebuild protos if they change in rust - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 51883f37a447ecfe9c1aa181ab3c3135aebb3a16 Author: Hammad Bashir <[email protected]> Date: Wed Mar 20 15:34:30 2024 -0700 [ENH] Add query-service server (#1899) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Add a new entrypoint for query-service. Delete the old worker entrypoint that did not have read/write decoupling. - Make dispatcher configurable - Wrapped hnsw orchestrator logic into run() so server is unaware of it - Make server struct {} have the resources it needs - New functionality - Add dynamic creation for log - Add dynamic creation for sysdb - Make server respond to query by using orchestrator ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 82e3db68896d8fe42091e3d1529cd93a4e3b6c10 Author: Liquan Pei <[email protected]> Date: Wed Mar 20 12:52:14 2024 -0700 [ENH] Add end_timestamp to PullLog API (#1897) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR adds end_timestamp to the PullLog API ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit af40bb11b4986052dfe551dc6dd8ed05a40eea7c Author: Hammad Bashir <[email protected]> Date: Tue Mar 19 22:10:34 2024 -0700 [TST] Add test that the default config loads and parses (#1901) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Adds a basic test to ensure that the default config loads and parses - New functionality - None ## Test plan *How are these changes tested?* This is a test - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit c04a31ad241e8e2641b0fdaf37852f33cac65d79 Author: Ben Eggers <[email protected]> Date: Tue Mar 19 20:39:19 2024 -0700 [BUG] Add logservice config to query service (#1900) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Add log service config to query service. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 9f165338cbe6ab5e416e38a529a350412f11adc9 Author: Ben Eggers <[email protected]> Date: Tue Mar 19 17:18:35 2024 -0700 [BUG] Wipe sysdb and reapply schema each time (#1898) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Correctly wipe the sysdb and re-apply schema each push. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit c07281fdaa8d98e6cd9ee97dfe6fc2db12dcafa3 Author: Ben Eggers <[email protected]> Date: Tue Mar 19 15:58:20 2024 -0700 [ENH] Clean DB before applying (for now) (#1896) ## Description of changes *Summarize the changes made by this PR.* - New functionality - Call `atlas schema clean` before applying migrations. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 93a659dff11f2b6f839596afc5efe3314dd92856 Author: Hammad Bashir <[email protected]> Date: Tue Mar 19 15:41:02 2024 -0700 [ENH] Add push based operators, centralized dispatch, hardcode query plan as state machine (#1888) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Change on_start to async - Move on_start into the executor. - New functionality - This PR adds operators to the query/compaction workers - Implements a PullLog operator - Adds a HnswQueryOrchestrator, which can be thought of as a passive state machine of a hardcoded query plan. - Adds a dispatcher with worker threads for scheduling tasks. Worker threads pull for tasks. Pending work I will address in following prs: - [ ] Make the callers of dispatch unaware of wrap() - [ ] PullLogs should poll until completion @Ishiihara is taking this - [ ] Error handling - [ ] Wrap the orchestrator - [ ] Add a server struct and have it create the aforementioned orchestrator wrapper and then push to it. - [ ] Impl configurable ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit db4cabaa0bf6c3cf5f62cdb18f323eabb92caf32 Author: Ben Eggers <[email protected]> Date: Tue Mar 19 13:38:39 2024 -0700 [BUG] install jq in migration, not sysdb (#1894) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - We use jq for some env var stuff. We need it in the migration, not the sysdb service (which knows how to correctly URL-escape args) ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit ab4f039e10ff94988d3ef3bafb9805b169cb83df Author: Ben Eggers <[email protected]> Date: Mon Mar 18 17:40:32 2024 -0700 [ENH] Run the sysdb migration in sh so we can do env var substitition (#1887) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Run the SysDB migration in `/bin/sh` so we can sub in env vars as needed. This requires installing atlas into a debian image instead of using the atlas image directly. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 2829c57d6c5a58c14e42ca10ad22537b5c2a20df Author: Ben Eggers <[email protected]> Date: Mon Mar 18 14:56:07 2024 -0700 [ENH] Let us specify sysdbMigration target (#1886) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Template out values in sysdb-migration.yaml so we can point it at databases besides localhost postgres. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit ea954bffaa435d0abdc2bbc0aaa88e12f2964225 Author: Ben Eggers <[email protected]> Date: Fri Mar 15 18:42:37 2024 -0700 [CLN] Clean up k8s, rename services and values (#1881) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Standardize k8s on the naming scheme we agreed on. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 41ad7ba1debc0db86f7885b17aa733504cdab396 Author: Weili Gu <[email protected]> Date: Fri Mar 15 16:26:06 2024 -0700 [ENH] Compactor flush (#1879) ### FlushCollectionCompaction https://github.com/chroma-core/chroma/pull/1869 https://linear.app/trychroma/issue/CHR-298/segment-compaction-flush - grpc in collection service FlushCollectionCompaction - table_catalog: FlushCollectionCompaction - in one transaction: - segment: RegisterFilePaths - collection: UpdateLogPositionAndVersion - tenant: UpdateTenantLastCompactionTime // optional, should add a config to disable ### Tests - test_utils to create tenant, database, collection, segments - change existing tests to use test suite: begin suite set up tenant, db (unless testing tenant and db in the test itself), end suite clean up, begin test set up sample objects (eg, collection and segments) - each tests should create objects that with different IDs and Names commit 47fdc612ee38042cf01d4f92a001b827021c5d7b Author: Ben Eggers <[email protected]> Date: Thu Mar 14 21:25:51 2024 -0700 [BUG] Fix Golang codebase test flakiness commit fa3a6f13530f43fc7651004e7053667bd5ed5dae Author: Ben Eggers <[email protected]> Date: Thu Mar 14 19:06:57 2024 -0700 Revert "[ENH] fush API for compactor" (#1874) Reverts chroma-core/chroma#1869 breaks tests commit c1835806183ec631760af3ae5ee9571c09952619 Author: Weili Gu <[email protected]> Date: Thu Mar 14 18:13:00 2024 -0700 [ENH] fush API for compactor (#1869) ### FlushCollectionCompaction https://linear.app/trychroma/issue/CHR-298/segment-compaction-flush - grpc in collection service FlushCollectionCompaction - table_catalog: FlushCollectionCompaction - in one transaction: - segment: RegisterFilePaths - collection: UpdateLogPositionAndVersion - tenant: UpdateTenantLastCompactionTime // optional, should add a config to disable ### Tests - test_utils to create tenant, database, collection, segments - change existing tests to use test suite: begin suite set up tenant, db (unless testing tenant and db in the test itself), end suite clean up, begin test set up sample objects (eg, collection and segments) - each tests should create objects that with different IDs and Names commit 21e5fcceefb6a9101e474f4d91a485734a550a80 Author: Hammad Bashir <[email protected]> Date: Wed Mar 13 17:30:11 2024 -0700 [ENH] Add uint blockfile key/val (#1854) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Lets the blockfile use uint vals and keys. - Some name cleanup / unification - New functionality - / ## Test plan *How are these changes tested?* Basic sanity tests. - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit b31a07cf112af78df0d9c2a7b92320dff5923c03 Author: Ben Eggers <[email protected]> Date: Wed Mar 13 17:15:54 2024 -0700 [ENH] Property testing for metadata indices (#1794) ## Description of changes *Summarize the changes made by this PR.* - New functionality - Make MetadataIndex and BlockfileMetadataIndex properly generic. Let the type system do the work! - Property testing for metadata indices. I learned a lot about the borrow checker :0) ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 79f0cd17cc6424a2d78e718ec5a255211855d935 Author: Hammad Bashir <[email protected]> Date: Wed Mar 13 17:11:05 2024 -0700 [ENH] Add Segment Interfaces (#1853) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Adds an interface type for segments. We can add flush etc here in the future. I will move hnsw to use this in a stacked PR. - New functionality - None. ## Test plan *How are these changes tested?* No logical changes. These are just interfaces. - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 97c7539bf1420f1293ab906eaaa8babe276ddd5b Author: Matthew Keller <[email protected]> Date: Wed Mar 13 19:58:05 2024 -0400 [CLN] Apply Prettier Config to JS Client Repo (#1831) ## Description of changes *Summarize the changes made by this PR.* Applies Prettier formatting to the JS Client Repo and add a presubmit rule to enforce that behavior. ## Test plan N/A just formatting changes. commit d425111fbf98d901bb4a416fd0a414bbeeddd324 Author: Hammad Bashir <[email protected]> Date: Wed Mar 13 16:56:03 2024 -0700 [ENH] Add arrow blockfile provider (#1847) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - / - New functionality - Adds a provider for ArrowBlockfiles ## Test plan *How are these changes tested?* Not tested for now. A future PR will contains tests. - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 69660b8f638ad40b3131e6e9cfbd11eb426ecbd6 Author: Hammad Bashir <[email protected]> Date: Wed Mar 13 16:43:15 2024 -0700 [ENH] Add Arrow-backed blockfile (#1846) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Adds roaring bitmap value types, builders etc - New functionality - Adds the base of the blockfile implementation for the arrow-backed blockfile. ## Test plan *How are these changes tested?* Basic - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit dd7331148bb0cc822069a85bc810e147d5e2aa9d Author: Ben Eggers <[email protected]> Date: Wed Mar 13 16:25:24 2024 -0700 [BUG] grace=None not 0 commit 3c4881740e2a1ba1ecafec8f2e7e1cf996cb29f2 Author: Ben Eggers <[email protected]> Date: Wed Mar 13 10:21:56 2024 -0700 [CLN] undo unecessary change in 0e2776c98a6637e1cbab5c1582214bfdbb861499 commit 0e2776c98a6637e1cbab5c1582214bfdbb861499 Author: Ben Eggers <[email protected]> Date: Wed Mar 13 10:20:05 2024 -0700 [BUG] Shut down test system to hopefully fix test flakiness commit 17d427bd15d2eca30561e1f048ceb0194c16e32c Author: Matthew Keller <[email protected]> Date: Tue Mar 12 18:52:05 2024 -0400 [CHORE] add .chrome_env file to the gitignore (#1834) ## Description of changes *Summarize the changes made by this PR.* The js tests automatically generate this file so it shouldn't be checked in. ## Test plan *How are these changes tested?* - [x] confirmed that this file is no longer included. ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 19512e6ca4ca599f93911aa28b704d3508cb13e6 Author: nicolasgere <[email protected]> Date: Tue Mar 12 13:22:08 2024 -0700 [BUG]: fix auth (#1864) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Hosted auth is not working due to a missing import commit e715cd2ac85281a415aa54750f67fc3240f6e419 Author: Hammad Bashir <[email protected]> Date: Tue Mar 12 11:19:41 2024 -0700 [ENH] Sparse index (#1825) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - / - New functionality - Adds a sparse index for blockfiles ## Test plan *How are these changes tested?* Basic unit tests for now - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit 73766c315b54a192f2ef241d82520443173fefdf Author: Hammad Bashir <[email protected]> Date: Mon Mar 11 22:30:35 2024 -0700 [ENH] Block delta (#1823) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - None - New functionality - Adds a BlockDelta which is used to mutate a block. It reads the contents of a block and then can allows changes to be applied directly to it. - Adds the ability to apply() a block delta - Add a BlockProvider, which is the interface for loading and managing blocks. ## Test plan *How are these changes tested?* - [x] Basic unit tests `cargo test` ## Documentation Changes None commit 9b607c492120e4a99019710c9758ae810396cbd9 Author: Hammad Bashir <[email protected]> Date: Mon Mar 11 11:45:11 2024 -0700 [BLD] Add docker arg to build in release mode (#1859) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Adds the ability to build dockerfile for rust worker in release mdoe - New functionality - / ## Test plan *How are these changes tested?* I manually modified the arg and made sure the expected control flow was followed. - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None required. commit 017138ae49b370a0cd460737e30662abb855851d Author: Weili Gu <[email protected]> Date: Mon Mar 11 09:11:46 2024 -0700 [ENH] last compaction time (#1856) ## Description of changes https://linear.app/trychroma/issue/CHR-366/add-api-to-get-tenant-last-compaction-time Add functionalities to set and get tenants' last compaction time. ## Test plan - [ ] tenant_database_service_test.go - [ ] tenant_test.go commit 1e5e024fa2984e06d1ab17ee3d9c4960c56400b4 Author: Weili Gu <[email protected]> Date: Sat Mar 9 13:38:30 2024 -0800 [TST] fix flacky coordinator tests (#1852) Fix flack coordinator tests, two root causes: since we replaced in memory map with real DB, tests become flacky because 1. we didn't clean up the state in apis_test.go 2. tests were using default names and running asynchronously fix: 1. making apis_test.go setup and teardown the db 2. disable async flag in go test in longer term: we can run tests asyn when we have individual tests create it's own objects in DB instead of sharing same ones. Unless in some special cases where we want to test very specific namespace, these tests need to be configured to run synchronously. We also need to complete delete function (including cascading deletions) for some objects before we can make tests async so that we don't need to clean the entire DB. one more fix to atlas set up file due to renaming the go packages. commit 186165dfbfd6eeb3db92c07d8ea638751b3f861d Author: Hammad Bashir <[email protected]> Date: Sat Mar 9 09:44:50 2024 -0800 [TST] Scheduler test fix (#1851) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Fixes an issue with the scheduler test where tiebreaks on 0 last compaction time are random. This uses contains instead of order. When @Ishiihara adds a the api for last compaction time to the sysdb this will be fixed and reverted! - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None commit cf46f7789fd4a6a66aa1ca762fac5d7216b44c04 Author: Liquan Pei <[email protected]> Date: Fri Mar 8 11:55:55 2024 -0800 [ENH] Compactor scheduler with system (#1844) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR adds compactor scheduler with system components and scheduler. ## Test plan *How are these changes tested?* - [ ] cargo test ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 34e2795216d610389e05cba52fcb781ba1b433bc Author: Hammad Bashir <[email protected]> Date: Fri Mar 8 09:10:48 2024 -0800 [DOC] Update and rename README to README.md (#1848) Doc cleanup commit 722a307c46c36d312747cfe99ec7fb48a0dde9f8 Author: Hammad Bashir <[email protected]> Date: Fri Mar 8 08:29:19 2024 -0800 [ENH] Block iterator (#1822) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - None - New functionality - Adds a basic block iterator to allow for easily cloning a block into a BlockDelta ## Test plan *How are these changes tested?* - [x] Tests pass locally with `cargo test` ## Documentation Changes None commit cb8f2bb76c0671b77349603b3ffe7977fe7f0ad7 Author: Hammad Bashir <[email protected]> Date: Fri Mar 8 07:54:47 2024 -0800 [ENH] Block builder (#1821) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - / - New functionality - Adds a block builder for creating BlockData. Blocks hold BlockData in order to serve queries. ## Test plan *How are these changes tested?* - [x] Tests pass locally with `cargo test` Basic unit tests were added ## Documentation Changes None commit 64b39b155c3efaea3e6d908371fa2e285a2b7a1f Author: Hammad Bashir <[email protected]> Date: Thu Mar 7 17:18:58 2024 -0800 [ENH] Arrow-backed block (#1820) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - / - New functionality - Adds an arrow-backed block, the code documentation covers what this is more in detail. ## Test plan *How are these changes tested?* - [x] Tests pass locally with `cargo test` Tests will be in stacked PR ## Documentation Changes None commit 369eddde01d94ad2f4af4ad5c6fd2733ce0a7ecd Author: Liquan Pei <[email protected]> Date: Thu Mar 7 16:45:26 2024 -0800 [ENH] Add system scheduler to allow tasks to run with a schedule (#1839) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - ... - New functionality - This PR adds the compatibility to allow components to schedule tasks ## Test plan *How are these changes tested?* - [ ] cargo test ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 2cbbb2cf3cabbcdcfc887c7dd09145eb95941541 Author: Weili Gu <[email protected]> Date: Thu Mar 7 15:58:15 2024 -0800 [ENH] log service staging (#1843) ## Description of changes connect log service in staging commit 123826f8661db2198ce77dc7e61f6daf600d9185 Author: Ben Eggers <[email protected]> Date: Thu Mar 7 14:56:33 2024 -0800 [CLN] Run coordinator in `/bin/sh` (#1816) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Run coordinator in a shell instead of `exec` environment ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 8087c22c512f6220112dda5288f48745ab915d0f Author: Hammad Bashir <[email protected]> Date: Thu Mar 7 11:54:04 2024 -0800 [ENH] Blockfile provider (#1819) ## Description of changes This PR intermingles one substantial change with two smaller ones, apologies. *Summarize the changes made by this PR.* - Improvements & Bug fixes - Adds sizing methods for Keys and Values and a Clone method that is aware of the idiosyncrasies of Arrow - New functionality - Adds a blockfile provider, which presents an open/create API for managing blockfiles. All callers of blockfile:new() were migrated to this way of managing blockfiles. ## Test plan *How are these changes tested?* - [x] Tests pass locally with `cargo test` ## Documentation Changes None required. commit 5f1263ba4c8c2749054f83065c583437d6408daf Author: Ben Eggers <[email protected]> Date: Thu Mar 7 11:00:51 2024 -0800 [ENH] Helm CD to hosted (#1840) ## Description of changes *Summarize the changes made by this PR.* - Changes to OSS k8s will trigger hosted-chroma to update its k8s to stay in sync. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit f7fff9a0f70edee70542ad59f8ee758037a2ef20 Author: Ben Eggers <[email protected]> Date: Thu Mar 7 10:53:39 2024 -0800 [ENH] helm chart tweaks (#1836) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Move jaeger to `test/` where it belongs (it shouldn't be deployed alongside prod distributed chroma) - indent extra env configs and strings correctly. ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* commit 0c44deb66637cf2e4dcf4fe502fa44d0c6b462ce Author: Weili Gu <[email protected]> Date: Wed Mar 6 15:57:18 2024 -0800 [ENH] remove sysdb cache (#1835) ## Description of changes https://linear.app/trychroma/issue/CHR-299/remove-cache-in-sysdb - remove sysdb cache - remove created field from collection object ## Test plan - [ ] passing existing changes commit b7e8b62e6444a779193f0afb06bcb47fb4f25a26 Author: nicolasgere <[email protected]> Date: Wed Mar 6 13:15:22 2024 -0800 [ENH]: add grpc client interceptor (#1818) ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Add grpc client interceptor ## Test plan *How are these changes tested?* - Tested locally with distributed tracing commit 401a7f0543ad259a16648c68d670661456a9818c Author: Matthew Keller <[email protected]> Date: Wed Mar 6 14:23:32 2024 -0500 [DOC] add a bit more detail to the js client DEVELOP.md (#1829) commit 362d3506a0f4de1764c566a22255ece334107a65 Author: Trayan Azarov <[email protected]> Date: Wed Mar 6 20:57:27 2024 +0200 [ENH]: Updates to validation errors (#1703) Refs: #1694 ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Shortened validation error printouts to improve log readability ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js ## Documentation Changes N/A commit 73a163067cb33d1eff75e193bd08ca4ec8202920 Author: Trayan Azarov <[email protected]> Date: Wed Mar 6 20:55:44 2024 +0200 [BUG]: Deleting FTS entries upon deletion of individual records (#1689) Refs: #1664 ## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Follow up on #1664 ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python ## Documentation Changes N/A commit 276226b82fe567f12c62ba7432f5b04cfb426ae0 Author: Trayan Azarov <[email protected]…
atroyn
pushed a commit
to csbasil/chroma
that referenced
this pull request
Apr 3, 2024
## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - / - New functionality - Adds an arrow-backed block, the code documentation covers what this is more in detail. ## Test plan *How are these changes tested?* - [x] Tests pass locally with `cargo test` Tests will be in stacked PR ## Documentation Changes None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Summarize the changes made by this PR.
Test plan
How are these changes tested?
cargo testTests will be in stacked PR
Documentation Changes
None