9.0.0
Features
- Introduce
Benchmark
andStudioBenchmark
Benchmark
allows you to evaluate and compare the performance of differentTask
s with a fixed evaluation logic, aggregation logic andDataset
.- Add
how_to_execute_a_benchmark.ipynb
to how-tos - Add
studio.ipynb
to notebooks to show how one can debug aTask
with Studio
- Introduce
BenchmarkRepository
andStudioBenchmarkRepository
- Add
create_project
bool toStudioClient.__init__()
to enable users to automatically create their Studio projects - Add progressbar to the
Runner
to be able to track theRun
- Add
StudioClient.submit_benchmark_lineages
function and include it inStudioClient.submit_benchmark_execution
DocumentIndexClient
- Add method
DocumentIndexClient.chunks()
for retrieving all text chunks of a document. - Add metadata filter
FilterOps.IS_NULL
, that allows to filter fields based on whether their value is null.
Fixes
- The Document Index
SearchQuery
now correctly allows searches with a negativemin_score
.
Deprecations
...
Breaking Changes
- The env variable
POSTGRES_HOST
is split intoPOSTGRES_HOST
andPOSTGRES_PORT
. This affects all classes interacting with Studio and theInstructionFinetuningDataRepository
. - The following env variables now need to be set (previously pointed to defaults)
CLIENT_URL
- URL of your inference stackDOCUMENT_INDEX_URL
- URL of the document index