Failed data export message does not provide any details

[szymond]

I am trying exporting website's data. It fails.

Ui says: Something went wrong when preparing your download. Please try again.

Mail says: Your Plausible CE export for [...] has encountered an error and was unsuccessful. Sorry for the trouble this may have caused.

There are no more details. I'd love to know what's the error about. How can I find it out?

[ruslandoga]

👋 @szymond

What CE version you are using? There were some issues with exports in v2.1.2 and v2.1.3.

[ruslandoga]

You can try increasing the timeout even more.

[szymond]

With configuration:
CLICKHOUSE_DATABASE_URL=http://plausible_events_db:8123/plausible_events_db?timeout=9900000000
at 23:29 I started an export, without success.

Logs look like this:

plausible-1            | Loading plausible..
plausible-1            | Starting dependencies..
plausible-1            | Starting repos..
plausible-1            | Plausible.Repo database already exists
plausible-1            | Plausible.IngestRepo database already exists
plausible-1            | Creation of Db successful!
plausible-1            | Loading plausible..
plausible-1            | Starting dependencies..
plausible-1            | Starting repos..
plausible-1            | 23:44:20.597 [error] Background job (%{args: %{"email_to" => "[email protected]", "local_path" => "/var/lib/plausible/plausible-exports/1", "site_id" => 1, "storage" => "local"}, id: 99088, meta: %{}, queue: "analytics_exports", worker: "Plausible.Workers.ExportAnalytics", duration: 900023198806, queue_time: 15295000}) failed:
plausible-1            |
plausible-1            |   ** (Oban.TimeoutError) Plausible.Workers.ExportAnalytics timed out after 900000ms
plausible-1            | 23:55:36.678 [error] Ch.Connection (#PID<0.4282.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.678 [error] Ch.Connection (#PID<0.4277.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.678 [error] Ch.Connection (#PID<0.4288.0>) disconnected: ** (Mint.TransportError) socket closed
plausible-1            | 23:55:36.687 [error] Ch.Connection (#PID<0.4280.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4281.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.688 [error] Ch.Connection (#PID<0.4298.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4289.0>) disconnected: ** (Mint.TransportError) socket closed
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4294.0>) disconnected: ** (Mint.TransportError) socket closed
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4278.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4274.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4290.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4276.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4273.0>) disconnected: ** (Mint.TransportError) socket closed
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4287.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4279.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4275.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.7319.0>) disconnected: ** (Mint.TransportError) socket closed
plausible-1            | 23:55:36.689 [error] Ch.Connection (#PID<0.4286.0>) disconnected: ** (Mint.TransportError) connection reset by peer
plausible-1            | 23:55:36.723 [error] Ch.Connection (#PID<0.4280.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.725 [error] Ch.Connection (#PID<0.4288.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.725 [error] Ch.Connection (#PID<0.4286.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.724 [error] Ch.Connection (#PID<0.4298.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.724 [error] Ch.Connection (#PID<0.7319.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.724 [error] Ch.Connection (#PID<0.4281.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.726 [error] Ch.Connection (#PID<0.4282.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.727 [error] Ch.Connection (#PID<0.4279.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.729 [error] Ch.Connection (#PID<0.4290.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.729 [error] Ch.Connection (#PID<0.4275.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.730 [error] Ch.Connection (#PID<0.4276.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.731 [error] Ch.Connection (#PID<0.4273.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.730 [error] Ch.Connection (#PID<0.4277.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.731 [error] Ch.Connection (#PID<0.4274.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.732 [error] Ch.Connection (#PID<0.4289.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.731 [error] Ch.Connection (#PID<0.4287.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.732 [error] Ch.Connection (#PID<0.4294.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.733 [error] Ch.Connection (#PID<0.4278.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:36.741 [error] Background job (%{args: %{"email_to" => "[email protected]", "local_path" => "/var/lib/plausible/plausible-exports/1", "site_id" => 1, "storage" => "local"}, id: 99088, meta: %{}, queue: "analytics_exports", worker: "Plausible.Workers.ExportAnalytics", duration: 657565921789, queue_time: 469714000}) failed:
plausible-1            |
plausible-1            |   ** (Mint.TransportError) socket closed
plausible-1            |     (db_connection 2.7.0) lib/db_connection.ex:1269: anonymous fn/4 in DBConnection.reduce/3
plausible-1            |     (elixir 1.17.1) lib/stream.ex:1675: anonymous fn/5 in Stream.resource/3
plausible-1            |     (elixir 1.17.1) lib/stream.ex:1891: Enumerable.Stream.do_each/4
plausible-1            |     (elixir 1.17.1) lib/stream.ex:1052: Stream.do_transform_inner_enum/7
plausible-1            |     (elixir 1.17.1) lib/stream.ex:943: Stream.do_transform/5
plausible-1            |     (elixir 1.17.1) lib/stream.ex:585: Stream.do_into/4
plausible-1            |     (elixir 1.17.1) lib/stream.ex:690: Stream.run/1
plausible-1            |     (db_connection 2.7.0) lib/db_connection.ex:959: DBConnection.run/3
plausible-1            |
plausible_events_db-1 exited with code 0
plausible-1            | 23:55:37.986 [error] Ch.Connection (#PID<0.4287.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:38.125 [error] Ch.Connection (#PID<0.4282.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:38.138 [error] Ch.Connection (#PID<0.4298.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:38.171 [error] Ch.Connection (#PID<0.4279.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:38.213 [error] Ch.Connection (#PID<0.4277.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible_events_db-1  | ClickHouse Database directory appears to contain a database; Skipping initialization
plausible_events_db-1  | Processing configuration file '/etc/clickhouse-server/config.xml'.
plausible_events_db-1  | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
plausible_events_db-1  | Merging configuration file '/etc/clickhouse-server/config.d/logging.xml'.
plausible_events_db-1  | Logging warning to /var/log/clickhouse-server/clickhouse-server.log
plausible_events_db-1  | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
plausible_events_db-1  | 2024.11.19 23:55:38.463255 [ 1 ] {} <Warning> Context: Linux transparent hugepages are set to "always". Check /sys/kernel/mm/transparent_hugepage/enabled
plausible_events_db-1  | 2024.11.19 23:55:38.465399 [ 1 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
plausible-1            | 23:55:38.575 [error] Ch.Connection (#PID<0.4276.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:38.669 [error] Ch.Connection (#PID<0.4274.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:38.677 [error] Ch.Connection (#PID<0.4278.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:38.697 [error] Ch.Connection (#PID<0.4288.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:38.791 [error] Ch.Connection (#PID<0.4289.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:38.834 [error] Ch.Connection (#PID<0.4294.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible_events_db-1  | 2024.11.19 23:55:38.942560 [ 1 ] {} <Error> Application: Disabling cgroup memory observer because of an error during initialization: Code: 107. DB::Exception: Cannot find cgroups v1 or v2 current memory file. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):
plausible_events_db-1  |
plausible_events_db-1  | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000cbc851b
plausible_events_db-1  | 1. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x00000000076614c3
plausible_events_db-1  | 2. DB::CgroupsMemoryUsageObserver::CgroupsMemoryUsageObserver(std::chrono::duration<long long, std::ratio<1l, 1l>>) @ 0x000000000cce452e
plausible_events_db-1  | 3. DB::Server::main(std::vector<String, std::allocator<String>> const&) @ 0x000000000cd562bd
plausible_events_db-1  | 4. Poco::Util::Application::run() @ 0x0000000014ca8606
plausible_events_db-1  | 5. DB::Server::run() @ 0x000000000cd46671
plausible_events_db-1  | 6. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000014cb14b9
plausible_events_db-1  | 7. mainEntryClickHouseServer(int, char**) @ 0x000000000cd4246a
plausible_events_db-1  | 8. main @ 0x0000000007651578
plausible_events_db-1  | 9. ? @ 0x00007f03c9324083
plausible_events_db-1  | 10. _start @ 0x0000000005dee46e
plausible_events_db-1  |  (version 24.3.3.102 (official build))
plausible_events_db-1  | 2024.11.19 23:55:39.058252 [ 1 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.3.102 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
plausible_events_db-1  | 2024.11.19 23:55:39.120860 [ 686 ] {} <Warning> system.asynchronous_insert_log (b3aa4271-0ec9-438e-acfd-f0a0894fb636): Removing temporary directory /var/lib/clickhouse/store/b3a/b3aa4271-0ec9-438e-acfd-f0a0894fb636/tmp_merge_20241119_26681_35223_6877/
plausible-1            | 23:55:39.125 [error] Ch.Connection (#PID<0.4290.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:39.164 [error] Ch.Connection (#PID<0.4286.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:39.198 [error] Ch.Connection (#PID<0.4273.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:39.356 [error] Ch.Connection (#PID<0.4275.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible-1            | 23:55:39.362 [error] Ch.Connection (#PID<0.4281.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible_events_db-1  | 2024.11.19 23:55:39.537268 [ 745 ] {} <Warning> plausible_events_db.sessions_v2 (f5d2dc37-e0a5-49af-bb2a-de3ad73728eb): Removing temporary directory /var/lib/clickhouse/store/f5d/f5d2dc37-e0a5-49af-bb2a-de3ad73728eb/delete_tmp_202411_63022_63022_0/
plausible_events_db-1  | 2024.11.19 23:55:39.538684 [ 748 ] {} <Warning> plausible_events_db.ingest_counters (b670f59f-e2f0-4259-888e-682a2f7dade7): Removing temporary directory /var/lib/clickhouse/store/b67/b670f59f-e2f0-4259-888e-682a2f7dade7/tmp_merge_all_3263349_3305753_33667/
plausible-1            | 23:55:39.547 [error] Ch.Connection (#PID<0.4280.0>) failed to connect: ** (Mint.TransportError) connection refused
plausible_events_db-1  | 2024.11.19 23:55:39.554155 [ 1 ] {} <Warning> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.3.102 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
plausible_events_db-1  | 2024.11.19 23:55:39.554636 [ 1 ] {} <Warning> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.3.102 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
plausible_events_db-1  | 2024.11.19 23:55:39.554920 [ 1 ] {} <Warning> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.3.102 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
plausible_events_db-1  | 2024.11.19 23:55:39.555258 [ 1 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.3.102 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
plausible-1            | 23:55:40.286 [error] GenServer Plausible.Event.WriteBuffer terminating
plausible-1            | ** (Mint.TransportError) socket closed
plausible-1            |     (ecto_sql 3.12.0) lib/ecto/adapters/sql.ex:1078: Ecto.Adapters.SQL.raise_sql_call_error/1
plausible-1            |     (plausible 0.0.1) lib/plausible/ingestion/write_buffer.ex:64: Plausible.Ingestion.WriteBuffer.handle_info/2
plausible-1            |     (stdlib 6.0) gen_server.erl:2173: :gen_server.try_handle_info/3
plausible-1            |     (stdlib 6.0) gen_server.erl:2261: :gen_server.handle_msg/6
plausible-1            |     (stdlib 6.0) proc_lib.erl:329: :proc_lib.init_p_do_apply/3
plausible-1            | Last message: :tick
plausible-1            | 23:55:40.302 [error] GenServer Plausible.Session.WriteBuffer terminating
plausible-1            | ** (Mint.TransportError) socket closed
plausible-1            |     (ecto_sql 3.12.0) lib/ecto/adapters/sql.ex:1078: Ecto.Adapters.SQL.raise_sql_call_error/1
plausible-1            |     (plausible 0.0.1) lib/plausible/ingestion/write_buffer.ex:64: Plausible.Ingestion.WriteBuffer.handle_info/2
plausible-1            |     (stdlib 6.0) gen_server.erl:2173: :gen_server.try_handle_info/3
plausible-1            |     (stdlib 6.0) gen_server.erl:2261: :gen_server.handle_msg/6
plausible-1            |     (stdlib 6.0) proc_lib.erl:329: :proc_lib.init_p_do_apply/3
plausible-1            | Last message: :tick
plausible-1            |     get-location-data-table-comment Done!
plausible-1            |
plausible-1            | ------------------------------------------------------------------------------
plausible-1            | 00:10:57.762 [error] Background job (%{args: %{"email_to" => "[email protected]", "local_path" => "/var/lib/plausible/plausible-exports/1", "site_id" => 1, "storage" => "local"}, id: 99088, meta: %{}, queue: "analytics_exports", worker: "Plausible.Workers.ExportAnalytics", duration: 900020581082, queue_time: 983195000}) failed:
plausible-1            |
plausible-1            |   ** (Oban.TimeoutError) Plausible.Workers.ExportAnalytics timed out after 900000ms
plausible_events_db-1  | 2024.11.20 00:17:50.365848 [ 775 ] {e114afcc-ceb2-40ac-8f47-0a338eb7a99c} <Error> executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket (172.19.0.3:8123 -> 172.19.0.5:40614): While executing ParallelFormattingOutputFormat. (NETWORK_ERROR) (version 24.3.3.102 (official build)) (from 172.19.0.5:40614) (in query: SELECT toDate(e0.timestamp, 'Europe/Budapest') AS date, any(e0.hostname) AS hostname, e0.pathname AS page, toUInt64(round(uniq(e0.session_id) * any(_sample_factor))) AS visits, toUInt64(round(uniq(e0.user_id) * any(_sample_factor))) AS visitors, toUInt64(round(count() * any(_sample_factor))) AS pageviews FROM events_v2 AS e0 WHERE (((e0.site_id = _CAST(1, 'Int64')) AND (date >= _CAST(19418, 'Date'))) AND (date <= _CAST(20047, 'Date'))) AND (e0.name = 'pageview') GROUP BY date, e0.pathname ORDER BY date ASC), Stack trace (when copying this message, always include the lines below):
plausible_events_db-1  |
plausible_events_db-1  | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000cbc851b
plausible_events_db-1  | 1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000cd02621
plausible_events_db-1  | 2. DB::WriteBufferFromPocoSocket::socketSendBytes(char const*, unsigned long) @ 0x000000000cd02357
plausible_events_db-1  | 3. DB::WriteBufferFromHTTPServerResponse::writeHeaderProgressImpl(char const*) @ 0x000000001237aa72
plausible_events_db-1  | 4. DB::WriteBufferFromHTTPServerResponse::finishSendHeaders() @ 0x000000001237ac4a
plausible_events_db-1  | 5. DB::WriteBufferFromHTTPServerResponse::nextImpl() @ 0x000000001237aec3
plausible_events_db-1  | 6. DB::ParallelFormattingOutputFormat::collectorThreadFunction(std::shared_ptr<DB::ThreadGroup> const&) @ 0x0000000012550065
plausible_events_db-1  | 7. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::ParallelFormattingOutputFormat::ParallelFormattingOutputFormat(DB::ParallelFormattingOutputFormat::Params)::'lambda'()>(DB::ParallelFormattingOutputFormat::ParallelFormattingOutputFormat(DB::ParallelFormattingOutputFormat::Params)::'lambda'()&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000012389f5a
plausible_events_db-1  | 8. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000cc7714d
plausible_events_db-1  | 9. ? @ 0x00007f03c94fa609
plausible_events_db-1  | 10. ? @ 0x00007f03c941f353
plausible_events_db-1  |
plausible_events_db-1  | 2024.11.20 00:17:50.985843 [ 775 ] {e114afcc-ceb2-40ac-8f47-0a338eb7a99c} <Error> DynamicQueryHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket (172.19.0.3:8123 -> 172.19.0.5:40614): While executing ParallelFormattingOutputFormat. (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):
plausible_events_db-1  |
plausible_events_db-1  | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000cbc851b
plausible_events_db-1  | 1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000cd02621
plausible_events_db-1  | 2. DB::WriteBufferFromPocoSocket::socketSendBytes(char const*, unsigned long) @ 0x000000000cd02357
plausible_events_db-1  | 3. DB::WriteBufferFromHTTPServerResponse::writeHeaderProgressImpl(char const*) @ 0x000000001237aa72
plausible_events_db-1  | 4. DB::WriteBufferFromHTTPServerResponse::finishSendHeaders() @ 0x000000001237ac4a
plausible_events_db-1  | 5. DB::WriteBufferFromHTTPServerResponse::nextImpl() @ 0x000000001237aec3
plausible_events_db-1  | 6. DB::ParallelFormattingOutputFormat::collectorThreadFunction(std::shared_ptr<DB::ThreadGroup> const&) @ 0x0000000012550065
plausible_events_db-1  | 7. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::ParallelFormattingOutputFormat::ParallelFormattingOutputFormat(DB::ParallelFormattingOutputFormat::Params)::'lambda'()>(DB::ParallelFormattingOutputFormat::ParallelFormattingOutputFormat(DB::ParallelFormattingOutputFormat::Params)::'lambda'()&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000012389f5a
plausible_events_db-1  | 8. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000cc7714d
plausible_events_db-1  | 9. ? @ 0x00007f03c94fa609
plausible_events_db-1  | 10. ? @ 0x00007f03c941f353
plausible_events_db-1  |  (version 24.3.3.102 (official build))
plausible_events_db-1  | 2024.11.20 00:17:50.999947 [ 775 ] {e114afcc-ceb2-40ac-8f47-0a338eb7a99c} <Error> DynamicQueryHandler: Cannot send exception to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket (172.19.0.3:8123 -> 172.19.0.5:40614). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):
plausible_events_db-1  |
plausible_events_db-1  | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000cbc851b
plausible_events_db-1  | 1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000cd02621
plausible_events_db-1  | 2. DB::WriteBufferFromPocoSocket::socketSendBytes(char const*, unsigned long) @ 0x000000000cd02357
plausible_events_db-1  | 3. DB::WriteBufferFromHTTPServerResponse::nextImpl() @ 0x000000001237afdb
plausible_events_db-1  | 4. DB::HTTPHandler::trySendExceptionToClient(String const&, int, DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x00000000122f2c71
plausible_events_db-1  | 5. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000122f60fb
plausible_events_db-1  | 6. DB::HTTPServerConnection::run() @ 0x00000000123740da
plausible_events_db-1  | 7. Poco::Net::TCPServerConnection::start() @ 0x0000000014c9b8d2
plausible_events_db-1  | 8. Poco::Net::TCPServerDispatcher::run() @ 0x0000000014c9c719
plausible_events_db-1  | 9. Poco::PooledThread::run() @ 0x0000000014d94e81
plausible_events_db-1  | 10. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000014d9341d
plausible_events_db-1  | 11. ? @ 0x00007f03c94fa609
plausible_events_db-1  | 12. ? @ 0x00007f03c941f353
plausible_events_db-1  |  (version 24.3.3.102 (official build))
plausible_events_db-1  | 2024.11.20 00:17:51.006577 [ 775 ] {e114afcc-ceb2-40ac-8f47-0a338eb7a99c} <Error> DynamicQueryHandler: Cannot flush data to client (after sending exception): Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket (172.19.0.3:8123 -> 172.19.0.5:40614). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):
plausible_events_db-1  |
plausible_events_db-1  | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000cbc851b
plausible_events_db-1  | 1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000cd02621
plausible_events_db-1  | 2. DB::WriteBufferFromPocoSocket::socketSendBytes(char const*, unsigned long) @ 0x000000000cd02357
plausible_events_db-1  | 3. DB::WriteBufferFromHTTPServerResponse::writeHeaderProgressImpl(char const*) @ 0x000000001237aa72
plausible_events_db-1  | 4. DB::WriteBufferFromHTTPServerResponse::finishSendHeaders() @ 0x000000001237ac4a
plausible_events_db-1  | 5. DB::WriteBufferFromHTTPServerResponse::finalizeImpl() @ 0x000000001237b986
plausible_events_db-1  | 6. DB::WriteBuffer::finalize() @ 0x00000000076613a8
plausible_events_db-1  | 7. DB::HTTPHandler::trySendExceptionToClient(String const&, int, DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x00000000122f2df3
plausible_events_db-1  | 8. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000122f60fb
plausible_events_db-1  | 9. DB::HTTPServerConnection::run() @ 0x00000000123740da
plausible_events_db-1  | 10. Poco::Net::TCPServerConnection::start() @ 0x0000000014c9b8d2
plausible_events_db-1  | 11. Poco::Net::TCPServerDispatcher::run() @ 0x0000000014c9c719
plausible_events_db-1  | 12. Poco::PooledThread::run() @ 0x0000000014d94e81
plausible_events_db-1  | 13. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000014d9341d
plausible_events_db-1  | 14. ? @ 0x00007f03c94fa609
plausible_events_db-1  | 15. ? @ 0x00007f03c941f353
plausible_events_db-1  |  (version 24.3.3.102 (official build))
plausible_events_db-1  | 2024.11.20 00:17:51.010421 [ 775 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
plausible_events_db-1  |
plausible_events_db-1  | 0. Poco::Net::SocketImpl::error(int, String const&) @ 0x0000000014c97192
plausible_events_db-1  | 1. Poco::Net::SocketImpl::peerAddress() @ 0x0000000014c997f6
plausible_events_db-1  | 2. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptr<DB::IHTTPContext>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000012375309
plausible_events_db-1  | 3. DB::HTTPServerConnection::run() @ 0x0000000012373a68
plausible_events_db-1  | 4. Poco::Net::TCPServerConnection::start() @ 0x0000000014c9b8d2
plausible_events_db-1  | 5. Poco::Net::TCPServerDispatcher::run() @ 0x0000000014c9c719
plausible_events_db-1  | 6. Poco::PooledThread::run() @ 0x0000000014d94e81
plausible_events_db-1  | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000014d9341d
plausible_events_db-1  | 8. ? @ 0x00007f03c94fa609
plausible_events_db-1  | 9. ? @ 0x00007f03c941f353
plausible_events_db-1  |  (version 24.3.3.102 (official build))

It seems to me, maybe problem with executing query partially is caused by ClickHouse restarting itself? But why would it do that, I don't know.

[ruslandoga]

No, the errors mean that the client (us) closed the connection prematurely because the job timed out after 15 minutes.

[ruslandoga]

And the ingesters (from the WriteBuffer errors) might be failing because ClickHouse and / or your whole VM gets "overwhelmed" with the export. This is just a guess though since I don't see the actual error message there, just a closed socket.

[ruslandoga]

Btw, please use fenced code snippets for the logs :) It would make reading them easier.

I've edited your previous messages to improve formatting.