Skip to content

v9.0.0

Latest
Compare
Choose a tag to compare
@NiklasKoehneckeAA NiklasKoehneckeAA released this 19 Dec 09:30
· 1 commit to main since this release
e320ce3

9.0.0

Features

  • Introduce Benchmark and StudioBenchmark
    • Benchmark allows you to evaluate and compare the performance of different Tasks with a fixed evaluation logic, aggregation logic and Dataset.
    • Add how_to_execute_a_benchmark.ipynb to how-tos
    • Add studio.ipynb to notebooks to show how one can debug a Task with Studio
  • Introduce BenchmarkRepositoryand StudioBenchmarkRepository
  • Add create_project bool to StudioClient.__init__() to enable users to automatically create their Studio projects
  • Add progressbar to the Runner to be able to track the Run
  • Add StudioClient.submit_benchmark_lineages function and include it in StudioClient.submit_benchmark_execution

DocumentIndexClient

  • Add method DocumentIndexClient.chunks() for retrieving all text chunks of a document.
  • Add metadata filter FilterOps.IS_NULL, that allows to filter fields based on whether their value is null.

Fixes

  • The Document Index SearchQuery now correctly allows searches with a negative min_score.

Deprecations

...

Breaking Changes

  • The env variable POSTGRES_HOST is split into POSTGRES_HOST and POSTGRES_PORT. This affects all classes interacting with Studio and the InstructionFinetuningDataRepository.
  • The following env variables now need to be set (previously pointed to defaults)
    • CLIENT_URL - URL of your inference stack
    • DOCUMENT_INDEX_URL - URL of the document index

Full changelog: v8.0.0...v9.0.0