HTTP 504 response from charts.deliveryhero.io

[hkan]

Hello,

The ChartMuseum instance you're running on charts.deliveryhero.io has been responding with HTTP 504 for pretty much all endpoints since earlier today.

I'm not sure if this is the correct place to mention this, I didn't know where else to post.

[max-rocket-internet]

Hey @hkan,
I've applied a short term fix. Is the issue resolved for you?

[hkan]

Damn Max, that's some impressive response time. 🤩 I just rerun the CI jobs, and they passed. Thank you for the quick action!

[max-rocket-internet]

OK nice. One of our chart museum pods had some I/O errors. Seems to be OK now. Thanks for reporting it.

[bartlomiejkorupczynskivimn]

Hey @max-rocket-internet, the issue seems back

$ curl -v https://charts.deliveryhero.io/index.yaml
*   Trying 104.19.239.90:443...
* Connected to charts.deliveryhero.io (104.19.239.90) port 443 (#0)
* ALPN: offers h2,http/1.1
[...]
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: charts.deliveryhero.io]
* h2 [:path: /index.yaml]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7fc8f300a800)
> GET /index.yaml HTTP/2
> Host: charts.deliveryhero.io
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/2 504
< date: Wed, 13 Dec 2023 13:22:50 GMT
< content-type: text/plain; charset=UTF-8
< content-length: 15
< x-frame-options: SAMEORIGIN
< referrer-policy: same-origin
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< set-cookie: ***
< server: cloudflare
< cf-ray: 834e8334aedfbffd-WAW
< alt-svc: h3=":443"; ma=86400
<
* Connection #0 to host charts.deliveryhero.io left intact

[max-rocket-internet]

Hmmm I see we are periodically throwing 499s

[max-rocket-internet]

I've applied some changes but still not sure what the root cause is. I'll leave this open, let me know if there's further problems.

[pierluigilenoci]

@max-rocket-internet, my comment (and the linked PR) was relevant because it allows everyone to have a more resilient Chart Museum than Rocket's.

[max-rocket-internet]

@pierluigilenoci

was relevant because it allows everyone to have a more resilient Chart Museum

It doesn't, it just moves that single chart to github pages whereas in this repo there are many charts. Also I don't want to use github pages on my own account 🙂

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

For me, this is still an issue.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

This is still an issue.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

This is still an issue.

[max-rocket-internet]

@pierluigilenoci how often are you seeing this?

e.g. if I run this:

while true; do curl -w "%{response_code} %{time_total}\n" -s -o /dev/null https://charts.deliveryhero.io/index.yaml ; done

For me it's solid, left it running for about an hour, it's all like this:

200 0.297999
200 0.214413
200 0.177474
200 0.294827
200 0.310920

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

It still needs to be resolved.
It happened once in a while, and it could happen again.
What is DeliveryHero's Chart Registry SLA?

Possible solution? max-rocket-internet/k8s-event-logger#45

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

No action corresponds to no solution.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

No solution corresponds to the problem that is still present.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

[max-rocket-internet]

@pierluigilenoci as I mentioned here, it would be helpful to have more info from your side.

It happened once in a while

Is not helpful 🙂

Please run the command mentioned for a few hours or a day and post the results.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[jjesusglia]

This is happening again, is there a planned fix? Thanks!

[2/10/24 8:00:34] ➜  while true; do curl -w "%{response_code} %{time_total}\n" -s -o /dev/null https://charts.deliveryhero.io/index.yaml ; done
504 60.226101
504 59.951381
504 61.321052
...

[pierluigilenoci]

@max-rocket-internet As I wrote last spring, the problem has not been solved, and it happened again.
Ref: #606
The solution: #591