[PERF]: Improved server-side serialization with orjson + async

[tazarov]

Future work: Add orjson serialization at client-side

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Added orjson serialization improving serialization performance especially on large batches 100+ docs 2x faster (tested with locust)
    • Added async body serialization further improving performance (tested with locust)
    • ⚠️ Fixed an issue with SQLite lack of case_sensitive_like for FastAPI Server thread pool connections

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js
• Locust performance tests

For performance tests, 5 concurrent clients were querying, and 1 was adding pre-computed OpenAI (ada-02) embeddings (runtime ~1m)

Batch size: 100 (1000 also attached in a zip)

Tests with existing main codebase:

Tests with orjson + async:

1m-100batch-5concur.zip
1m-1000batch-5concur.zip

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?

Refs

• https://showmax.engineering/articles/json-python-libraries-overview
• https://github.com/ijl/orjson
• https://catnotfoundnear.github.io/finding-the-fastest-python-json-library-on-all-python-versions-8-compared.html

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[tazarov]

@HammadB, can you rerun failed tests?

[HammadB]

@tazarov yep - reran!

[tazarov]

@HammadB, rebased and removed the to_thread offloading.

[tazarov]

@HammadB, I think this is ready now.

[tazarov]

@HammadB, I've started working through our discussion yesterday, and I think I'm making some assumptions as to the Auth (and other middlewares, e.g. rate-limit), which will not work well as they are right now. I think both Rate limiting and authz need to be made async which they are not rn.

The bottom line is that I can offload async body reads to a thread-pool (required as I want to still keep the methods sync) and then continue with the sync processing as it is now, until we have all parts ready to make FastAPI methods async.

[tazarov]

@HammadB, I think this is good to go, but we need to have a quick follow-up about the following:

• async telemetry
• async auth
• async rate-limit etc
• async anything IO-bound that is run in the main event loop

[HammadB]

can we stack those changes on top and we can merge it all down coherently

[tazarov]

@HammadB, closing this in favor of #1938 stack