Make query endpoint ready for production workloads

[julian-risch]

Problem Statement
As a developer running Haystack in a production environment, I want the /query endpoint to be scalable and reliable so that my system is stable.

User Tasks

• Pull a Docker image
• Customize the default Haystack setup
• Configure additional services
• Configure a pipeline
• Deploy the container in a pod
  • 🔴 The official Helm chart is not working with the new Haystack images
• Ensure the container is healthy
  • 🔴 Pain point: we don't know if there was a problem caching the model until we use it
• Receive query from user
• Ensure we have enough resources to respond fast
  • 🔴 No idea if there's enough GPU RAM left for larger batches
  • 🔴 Missing guidance on Autoscaling horizontally
  • 🔴 Concurrency is an issue
• Combine queries in batches
  • 🔴 Feature doesn't exist, look at other ML frameworks for inspiration
• Send queries to Haystack
• Run pipeline
  • 🟡  We suspect it can be more efficient
• Get the query result
• Handle result schema problems
  • 🔴 Schema change shouldn't go unnoticed
• Send results to user
• Change Haystack version
  • 🔴 update is risky as we don't know in advance if upgrade causes problems
• ensure the deployment is healthy
  • 🔴 very limited service observability

Tasks

Beta Give feedback

1. Update Helm chart to new Haystack Docker images haystack-helm#3
   P3 +1
2. Deploy explore the world demo using the Helm chart haystack-demos#6
   enhancement +1
3. Provide guideline on how to deploy Haystack for being used in production #3910
   P2 journey:advanced topic:rest_api type:documentation +4
4. Provide a Docker image blueprint for air-gapped environments #3618
   P3 journey:advanced topic:docker topic:installation type:enhancement +5
   
5. rest_api: worker timeout prevents bootstrapping the app #3870
   P1 topic:rest_api +2
   
6. Allow batches of queries in REST API's /query endpoint #3911
   topic:rest_api type:feature wontfix +3
7. Catch CUDA out-of-memory errors and enhance their error message #3912
   P2 topic:pipeline type:feature wontfix +4
8. Investigate alternatives to gunicorn settings in REST API #3913
   P1 journey:advanced topic:rest_api type:enhancement +4
   
9. Ensure that there are no race conditions in REST API /query endpoint #3914
   P2 topic:rest_api +2
10. REST API file-upload schema for meta #3790
    topic:rest_api type:enhancement wontfix +3
11. Add REST API endpoint exposing application metrics #3915
    topic:rest_api type:feature wontfix +3
12. https://github.com/deepset-ai/haystack-private/issues/46