Investigate 504 Gateway Timeout errors

[jkfran]

Investigate 504 errors, see https://forum.snapcraft.io/t/many-504-gateway-timeout-errors/23152

Could https://sentry.is.canonical.com/canonical/snapcraft-io/issues/8351/ be related?

[rhys-the-davies]

Hey @jkfran I see this has been pushed to this iteration, just wondering if you had a chance to look at it at all and if you could maybe update the discourse thread with progress? 😊

[jkfran]

I consider these events to be related to the store API slowness or being down. We can't reproduce them. On the Discourse topic, someone is also mentioning issues with the CLI command, which would also mean that it is an issue with the API.

[jkfran]

I am going to close the issue for now. If the issues are still happening we can take a look again.

[popey]

It still happens. I don't notice it as much since I left Canonical I have few snaps to maintain. But when I do, I almost always get a 504 at some point.

[lucyllewy]

yeah, this isn't fixed at all.

[lucyllewy]

@jkfran please reopen this.

[jkfran]

Hello! In an effort to improve this situation we did an internal investigation. Here is a list of related changes we implemented:

#3832 Split publisher (Debugging purposes, it was reverted later on)
#53 Flask-base v0.9.3 - Include PID on talisker logs (Improve server logging)
#3836 Extra logging (Debugging purposes, some changes on this PR will be reverted soon)
#3865 Test with gevent.Timeout (Debugging purposes, we would like to keep it a bit longer)
#3866 Remove gunicorn max-requests (Actual fix of these issues)
#3867 Use two gunicorn workers (Configuration improvement)

I created a follow-up issue to revert unnecessary changes.

So far the cause of this issue disappear from our logs. We will keep an eye on it.

Please keep in mind that many parts of the website rely on the Snapcraft APIs, any downtime of these APIs will be reflected on snapcraft.io, the best way to find out if these APIs are down is status.snapcraft.io.

Thank you.