chore(test): fix and re-activate integration and unit tests

[justinthelaw]

Description

See PR #994 for details on testing issues, and #564 for the overall integration testing issues. If the integration tests were run prior to committing, or in the pipeline, the file_ids issue fixed by PR #994 (related to Issue #1001) would have been caught earlier.

BREAKING CHANGES

• N/A

CHANGES

• Fixes the Makefile targets for executing integration tests - now works on all shells and scenarios
• Further improves README for local unit and integration test execution
• Enables integration and unit testing in local development and inside our CI pipelines
• Refactors the Repeater to better mock an actual in-cluster model, fixing issues with chat completion integration tests and streaming chat unit tests
• Refactors unit tests to test the behavior of the API and SDK, rather than the Repeater implementation code
• Bumps unstructured and nltk versions to allow for .pptx chunking, which was previously not working and failed integration tests

Related Issue

Fixes Issues #1001 #564
Related to PR #994

Checklist before merging

• Tests, documentation, ADR added or updated as needed
• Followed the Contributor Guide Steps

[netlify]

✅ Deploy Preview for leapfrogai-docs canceled.

Name	Link
🔨 Latest commit	926ba60
🔍 Latest deploy log	https://app.netlify.com/sites/leapfrogai-docs/deploys/66df5826f162fa00088192e4

[jalling97]

Having the integration tests be a part of the pipeline is fantastic!

Just pointing out though that the pytest workflow takes about 25 minutes now, which is about as long as it takes to test text-embeddings. So while we're not taking any longer to run all of the tests, we're further cementing that 25-30 minute timeframe. If we make changes in the future to futher optimize the E2E tests, addressing the integration tests will probably also be necessary.

[justinthelaw]

Having the integration tests be a part of the pipeline is fantastic!

Just pointing out though that the pytest workflow takes about 25 minutes now, which is about as long as it takes to test text-embeddings. So while we're not taking any longer to run all of the tests, we're further cementing that 25-30 minute timeframe. If we make changes in the future to futher optimize the E2E tests, addressing the integration tests will probably also be necessary.

@jalling97 totally agree! I will make an evergreen spike to investigate speeding up all e2e tests, including integration tests. The difficulty is definitely that each backend is very time-consuming to install and build. What we can do in the spike is investigate GH caching (like I do with the pip deps) and also more usage of a no/low-weight model like the repeater or T5, GPT2 etc., in tests-only.

EDIT: here is the aforementioned issue