unexpected EOF on client connection with an open transaction #5641

woozhijun · 2021-11-08T07:30:12Z

When we upgrade v8.0.0 to v10.0.0, I have a error log
unexpected EOF on client connection with an open transaction with postgres for exec cmd docker logs -f xxxx.

Not sure if it will affect the service?

The text was updated successfully, but these errors were encountered:

songdeebuzzni · 2022-01-13T08:12:48Z

An error that occurs because the db client (sqlalchemy?) terminated abnormally.

The same occurs even if localhost and (committed internal) IPs are additionally set in "/var/lib/postgresql/data/pg_hba.conf".

It seems to happen right after the connection is closed or rollback failed.

silvaartur · 2022-06-10T00:58:45Z

Solved?

susodapop · 2022-07-04T03:25:16Z

Thanks for the ping. No solution so far. Because I've not been able to reproduce this in version 10.1 locally. Does anyone have steps to reproduce? Are specific requests occurring around which these errors are emitted?

KeycapCaper · 2022-07-21T19:02:26Z

I'm not sure about steps to reproduce, but it does seem to be related to the volume of records. Currently experiencing the issue on a couple queries. If i set a smaller date range of records to return, it doesnt hit the error. Once I walked up the date range, it started hitting this issue somewhere after 22,000 records returned. Also confirmed it wasn't a specific bad record by taking multiple sample datasets from various date ranges.

susodapop · 2022-07-21T21:26:51Z

Is this a regression from V8?

ribtoks · 2022-11-12T17:09:32Z

Also hitting this in v10

puttehi · 2023-02-02T14:12:42Z

Also seeing this in Redash 10.0.0 (9c928bd) with PostgreSQL 9.6.24

Logs seems to come in 15s or 30s intervals in batches of 1-3. The clear interval making it feel like some keepalive ping or such.

OnkarVO7 · 2023-03-13T09:01:06Z

Facing the same issue while migrating from v8 to v10. Is there any solution to fix this?

songdeebuzzni · 2023-03-13T09:54:27Z

@OnkarVO7 Copy and paste your docker-compose.yml file script.

george74greece · 2023-05-25T09:01:36Z

Hello, i have the same issue when i am trying to upgrade from 8V to 10V
postgres_1 | LOG: unexpected EOF on client connection with an open transaction

ubuntu@ip-10-233-135-229:/opt/redash$ cat docker-compose.yml
version: "2"
x-redash-service: &redash-service
  image: redash/redash:10.1.0.b50633
  depends_on:
    - postgres
    - redis
  env_file: /opt/redash/env
  restart: always
services:
  server:
    <<: *redash-service
    command: server
    ports:
      - "5000:5000"
    environment:
      REDASH_WEB_WORKERS: 4
  scheduler:
    <<: *redash-service
    command: scheduler
  scheduled_worker:
    <<: *redash-service
    command: worker
  adhoc_worker:
    <<: *redash-service
    command: worker
  redis:
    image: redis:5.0-alpine
    restart: always
  postgres:
    image: postgres:9.6-alpine
    env_file: /opt/redash/env
    volumes:
      - /opt/redash/postgres-data:/var/lib/postgresql/data
    restart: always
  nginx:
    image: redash/nginx:latest
    ports:
      - "80:80"
    depends_on:
      - server
    links:
      - server:redash
    restart: always
  worker:
    <<: *redash-service
    command: worker
    environment:
      QUEUES: "periodic emails default"
      WORKERS_COUNT: 1
ubuntu@ip-10-233-135-229:/opt/redash$

Samuel29 · 2023-06-07T14:47:33Z

Hi. same issue for me on a fresh install via Helm on a K8s cluster

JPGallo1510 · 2023-08-10T21:03:40Z

Facing the same issue while migrating from v8 to v10. Is there any solution to fix this?

@OnkarVO7 i have the same issue, did you solve it?

lpong · 2023-12-06T02:17:54Z

Has this problem been resolved so far? I am also troubled by this problem.

alavi-sorrek · 2024-01-24T00:44:02Z

Has this been resolved?

guidopetri · 2024-01-30T02:42:14Z

Hi folks, unless you give us instructions to reproduce the error, we can't resolve it.

alavi-sorrek · 2024-01-30T02:44:32Z

As stupid as this sounds, it's by following the walkthrough video for updating to v10 from v8. Have you or someone on the team tried to follow those instructions recently to see if they are still correct? I tried doing it 3 or 4 times, each time on a fresh AWS EC2 using the pre-baked AMI.

Here's the link: https://redash.io/help/open-source/setup

alavi-sorrek · 2024-01-30T02:46:35Z

To troubleshoot I tried several things - force closing any open processes that might be running postgres (which wasn't installed), I tried installing postgres, I made sure there weren't any active connections, I tried setting it up once by only logging in after the upgrade, etc.

guidopetri · 2024-01-30T03:03:07Z

Well; a couple of things.

Have you or someone on the team

Redash is community owned now, there is no team of dedicated developers.

using the pre-baked AMI
updating to v10

Both of these indicate that you're using a pretty old version of redash; we don't really maintain the AMIs anymore, and v10 is from several years ago (let alone v8). My recommendation would be to back up your database and set redash up using either the v10 docker image directly, or the redash:preview image, and wiping the AMI machine.

I don't personally have access to running any AMI things, so I'm unable to reproduce this bug if you're only using the AMIs. Does the same issue happen if you use the docker images?

ribtoks · 2024-01-30T07:56:00Z

unless you give us instructions to reproduce the error, we can't resolve it.

@guidopetri For me the reproduction is docker-compose up

guidopetri · 2024-01-30T12:50:04Z

@ribtoks are you starting from scratch? or an existing db?

ribtoks · 2024-01-30T14:17:10Z

@guidopetri An existing DB. But "from scratch" instruction also requires you to "init" the DB first so it's kind of always from existing DB in such sense.

guidopetri · 2024-02-03T01:21:14Z

I can repro! 🎉

I can also assert that it's definitely related to refreshing schemas. The screenshot below has the scheduled_worker (in my case, responsible for refreshing schemas even on-demand) logs at the top, and postgres logs at the bottom. Whenever I click the "refresh schema" button on the UI, I get a new log hit on the top asserting the schema got refreshed, and on the bottom that there's the EOF error.

From an initial look, it appears the postgres runner doesn't have the get_schema() method defined, and as a result it breaks the connection. I'm not sure how the schema gets refreshed without that method defined, so this needs more investigation, but maybe all we need is to define that method?

guidopetri · 2024-02-03T04:04:16Z

Some more investigation:

this is definitely related to refreshing schemas
this doesn't seem to really affect the service nor postgres
the postgres runner does have a get_schema definition, it's just OOP'd in
the connection that gets created is async and gets closed, but no rollback/commit gets created. I tried adding a rollback and it did not work to fix this. I also tried making the connection not-async and it also did not work (even with a rollback call).
the EOF log line comes up after RQ saves the result (in redis?), and this doesn't seem to be related to any of the statds tracking we have

I only have access to a postgres data source; can anyone confirm if something similar happens on different data sources?

ribtoks · 2024-02-03T12:04:19Z

@guidopetri nope, only using Postgres so far. Great progress btw!

guidopetri · 2024-02-03T12:29:37Z

@ribtoks @alavi-sorrek - to be clear - do you see this message on the postgres db backing redash, or on a postgres data source?

(again, in my case I only have one postgres server, so I can't tell)

alavi-sorrek · 2024-02-03T16:21:19Z

I don’t have a postgres server at all and I see it lol

…

On Sat, Feb 3, 2024 at 7:29 AM Guido Petri ***@***.***> wrote: @ribtoks <https://github.com/ribtoks> @alavi-sorrek <https://github.com/alavi-sorrek> - to be clear - do you see this message on the postgres db backing redash, or on a postgres data source? (again, in my case I only have one postgres server, so I can't tell) — Reply to this email directly, view it on GitHub <#5641 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2QBB56AGSFX4JA75DT3MWLYRYUUBAVCNFSM5HR6DVU2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJSGUZTAOBVG42Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

guidopetri · 2024-02-03T18:18:32Z

Hmm. What data source are you using then?

alavi-sorrek · 2024-02-03T18:41:03Z

I’m using Redshift, which I suppose could cause a similar message, but I received the error during upgrading to v10 before I even connected to a data source.

…

On Sat, Feb 3, 2024 at 1:18 PM Guido Petri ***@***.***> wrote: Hmm. What data source are you using then? — Reply to this email directly, view it on GitHub <#5641 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2QBB5YWNI2GZQJJF6C3JRTYRZ5QNAVCNFSM5HR6DVU2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJSGU2DEMBUGU4Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

wtfiwtz · 2024-03-19T22:41:06Z

I found a potentially related issue.
Seems that the version of gunicorn in Redash 10.1.0 is v20.0.4 where they were halfway between patching this Keep Alive behaviour. benoitc/gunicorn#2297

If you have hit the graceful shutdown request count, then the processing loop could be terminated early, closing the connection. I think this situation could also cause this log error if you've opened any database connections for running the query.

I'm about to test an upgrade of gunicorn to (I think) 20.1.0 to see if it fixes it for us in production. We only see this under load, and whilst it has nothing to do with the load balancer itself, our AWS ALB returns a HTTP 502 Bad Gateway because the connection has been dropped (you see "502 -" in the access logs). This happens repeatedly under load for bigger JSON requests with respect to this patch that we applied - #78 (comment).

NOTE: maybe only with async workers, not sync workers (for Keep Alive)
NOTE 2: On second thought, DB access would be on the 'rq' workers, not the HTTP request handlers. Maybe it's unrelated then!

See getredash#5641 (comment) and benoitc/gunicorn#2297

wtfiwtz · 2024-03-20T00:32:32Z

Seems to be stable in production with gunicorn v21.0.1 🎉

…#4) * Fix 502 Bad Gateway error from gunicorn keep-alive default setting of 2 seconds See https://www.ikw.cz/aws-alb-gunicorn-error-502-bad-gateway-fix * Log gunicorn errors * Ungrade gunicorn to latest See getredash#5641 (comment) and benoitc/gunicorn#2297

mackenzieclark · 2024-05-23T18:56:14Z

As stupid as this sounds, it's by following the walkthrough video for updating to v10 from v8. Have you or someone on the team tried to follow those instructions recently to see if they are still correct? I tried doing it 3 or 4 times, each time on a fresh AWS EC2 using the pre-baked AMI.

Here's the link: https://redash.io/help/open-source/setup

I was having the same problem and resolved it by using the Bitnami AMI. https://bitnami.com/stack/redash

mashanz · 2024-11-13T22:32:28Z

wdyt if we migrate Gunicorn to Granian for better performance and stability?
https://github.com/emmett-framework/granian

And from poetry to UV for faster dependency resolver?
https://github.com/astral-sh/uv

mashanz · 2024-11-13T22:37:50Z

Seems to be stable in production with gunicorn v21.0.1 🎉

Let's see, I will try it

wtfiwtz added a commit to orchestrated-io/redash that referenced this issue Mar 19, 2024

Ungrade gunicorn to latest

2effe8c

See getredash#5641 (comment) and benoitc/gunicorn#2297

wtfiwtz mentioned this issue Mar 20, 2024

Better support for large query results #78

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unexpected EOF on client connection with an open transaction #5641

unexpected EOF on client connection with an open transaction #5641

woozhijun commented Nov 8, 2021

songdeebuzzni commented Jan 13, 2022 •

edited

Loading

silvaartur commented Jun 10, 2022

susodapop commented Jul 4, 2022

KeycapCaper commented Jul 21, 2022

susodapop commented Jul 21, 2022

ribtoks commented Nov 12, 2022

puttehi commented Feb 2, 2023

OnkarVO7 commented Mar 13, 2023

songdeebuzzni commented Mar 13, 2023

george74greece commented May 25, 2023 •

edited by justinclift

Loading

Samuel29 commented Jun 7, 2023

JPGallo1510 commented Aug 10, 2023

lpong commented Dec 6, 2023

alavi-sorrek commented Jan 24, 2024

guidopetri commented Jan 30, 2024

alavi-sorrek commented Jan 30, 2024

alavi-sorrek commented Jan 30, 2024

guidopetri commented Jan 30, 2024 •

edited

Loading

ribtoks commented Jan 30, 2024 •

edited

Loading

guidopetri commented Jan 30, 2024

ribtoks commented Jan 30, 2024

guidopetri commented Feb 3, 2024 •

edited

Loading

guidopetri commented Feb 3, 2024

ribtoks commented Feb 3, 2024

guidopetri commented Feb 3, 2024

alavi-sorrek commented Feb 3, 2024 via email

guidopetri commented Feb 3, 2024

alavi-sorrek commented Feb 3, 2024 via email

wtfiwtz commented Mar 19, 2024 •

edited

Loading

wtfiwtz commented Mar 20, 2024 •

edited

Loading

mackenzieclark commented May 23, 2024

mashanz commented Nov 13, 2024

mashanz commented Nov 13, 2024

unexpected EOF on client connection with an open transaction #5641

unexpected EOF on client connection with an open transaction #5641

Comments

woozhijun commented Nov 8, 2021

songdeebuzzni commented Jan 13, 2022 • edited Loading

silvaartur commented Jun 10, 2022

susodapop commented Jul 4, 2022

KeycapCaper commented Jul 21, 2022

susodapop commented Jul 21, 2022

ribtoks commented Nov 12, 2022

puttehi commented Feb 2, 2023

OnkarVO7 commented Mar 13, 2023

songdeebuzzni commented Mar 13, 2023

george74greece commented May 25, 2023 • edited by justinclift Loading

Samuel29 commented Jun 7, 2023

JPGallo1510 commented Aug 10, 2023

lpong commented Dec 6, 2023

alavi-sorrek commented Jan 24, 2024

guidopetri commented Jan 30, 2024

alavi-sorrek commented Jan 30, 2024

alavi-sorrek commented Jan 30, 2024

guidopetri commented Jan 30, 2024 • edited Loading

ribtoks commented Jan 30, 2024 • edited Loading

guidopetri commented Jan 30, 2024

ribtoks commented Jan 30, 2024

guidopetri commented Feb 3, 2024 • edited Loading

guidopetri commented Feb 3, 2024

ribtoks commented Feb 3, 2024

guidopetri commented Feb 3, 2024

alavi-sorrek commented Feb 3, 2024 via email

guidopetri commented Feb 3, 2024

alavi-sorrek commented Feb 3, 2024 via email

wtfiwtz commented Mar 19, 2024 • edited Loading

wtfiwtz commented Mar 20, 2024 • edited Loading

mackenzieclark commented May 23, 2024

mashanz commented Nov 13, 2024

mashanz commented Nov 13, 2024

songdeebuzzni commented Jan 13, 2022 •

edited

Loading

george74greece commented May 25, 2023 •

edited by justinclift

Loading

guidopetri commented Jan 30, 2024 •

edited

Loading

ribtoks commented Jan 30, 2024 •

edited

Loading

guidopetri commented Feb 3, 2024 •

edited

Loading

wtfiwtz commented Mar 19, 2024 •

edited

Loading

wtfiwtz commented Mar 20, 2024 •

edited

Loading