Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

505 Gateway timeout #116

Open
dtrckd opened this issue Dec 15, 2024 · 2 comments
Open

505 Gateway timeout #116

dtrckd opened this issue Dec 15, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@dtrckd
Copy link

dtrckd commented Dec 15, 2024

Hi,

I struggle to use this model meta-llama/Meta-Llama-3.1-70B-Instruct with suspect error code with a Gateway respoonse timeout from nginx not the albert-api itself.

Here are the error I got

Error: 504 Server Error: Gateway Time-out for url: https://albert.api.etalab.gouv.fr/v1/chat/completions, retrying in 5 seconds...
Albert API error: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.27.0</center>
</body>
</html>

Does ther error migh be caused by

  • a GPU OOM ?
  • a HTTP timeout ?
@dtrckd dtrckd added the bug Something isn't working label Dec 15, 2024
@dtrckd
Copy link
Author

dtrckd commented Dec 15, 2024

I have the intuition that there is no timeout error catching in the API, which leads to this error. If some gpu/model is stuck an do not respond, the api just wait and the GPU memory might be unnecessary loaded ? (at least after a few minutes, its unlikely that we will get a response from the model...)

Maybe adding a TTL of a few minutes (5 minutes ?) in the api, and returning a timeout error and free the GPU memory used could help scaling the infra while improving and the user experience ?

@leoguillaume @ArtixJP

@leoguillaume
Copy link
Contributor

Il y a timeout de 2 min configuré, l'erreur ne doit pas être transmise en réponse je pense, je vais regarder ca

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dtrckd @leoguillaume