odbc-bridge stuck

[johnskopis]

We use external dictionary in postgres via odbc-bridge. The odbc-bridge connects to a pgbouncer proxy on postgres. both pgbouncer and postgres seem healthy but it's possible some query times out.

After a recent upgrade:
v18.14.11-stable...v18.14.17-stable

We are seeing instability in odbc-bridge. The bridge gets "stuck". Here is what I observe:
https://gist.github.com/johnskopis/c2f661dee3b1e5a79b636ca038d254f9

There is no activity on postgres socket (verified with tcpdump, strace).

After killing odbc-bridge it seems to be OK. As a workaround maybe a periodic task can check odbc-bridge health (/ping) and restart when unhealthy?

I am not too familiar with poco but looks like a coroutine bug. Maybe it gets confused and never switches back and gets stuck between read/write? I mention this because I noticed in the changeset:
The upgrade included some process lifecycle management changes for odbc-bridge
The upgrade also included a change in AIO eof detection.

Any help would be greatly appreciated.

Also, is it possible to run odbc-bridge outside of clickhouse? Maybe a workaround is to run using a scheduler and restart unhealthy odbc-bridge.

Thanks

[iFoton]

Same problem. Begins since 18.14.17

[ztlpn]

@alesapin please take a look

[alesapin]

Looks same #3829.

[alesapin]

The bug fixed in #3857. As workaround you can set logger settings for odbc-bridge in server config.xml (it will not hang in current version):

    <logger>
        <!-- Possible levels: https://github.com/pocoproject/poco/blob/develop/Foundation/include/Poco/Logger.h#L105 -->
        <level>trace</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <odbc_bridge_log>/path/for/log/file</odbc_bridge_log>
        <odbc_bridge_errlog>/path/for/err/logfile</odbc_bridge_errlog>
        <odbc_bridge_level>trace</odbc_bridge_level>
        <size>1000M</size>
        <count>10</count>
        <!-- <console>1</console> --> <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
    </logger>

[alesapin]

v18.14.19-stable released, please update and try to disable settings.