Client dropped the connection

[makorne]

Hi!
Thank you for great crate!

Once a second in the logs such an error:

. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xaf62837 in /usr/bin/clickhouse
6. ? @ 0xaf662fd in /usr/bin/clickhouse
7. start_thread @ 0x7ea7 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
8. clone @ 0xfca2f in /usr/lib/x86_64-linux-gnu/libc-2.31.so
 (version 22.2.2.1)
2023.05.18 03:07:48.260050 [ 2943751 ] {9b6a5d14-e020-4294-8661-b9b5ee959924} <Error> DynamicQueryHandler: Cannot send exception to client: Code: 24. DB::Exception: Cannot write to ostream at offset 108473. (CANNOT_WRITE_TO_OSTREAM), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xaebed1a in /usr/bin/clickhouse
1. DB::WriteBufferFromOStream::nextImpl() @ 0xafc9992 in /usr/bin/clickhouse
2. DB::WriteBufferFromHTTPServerResponse::nextImpl() @ 0x15a14191 in /usr/bin/clickhouse
3. DB::HTTPHandler::trySendExceptionToClient(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x157a7e67 in /usr/bin/clickhouse
4. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x157a9735 in /usr/bin/clickhouse
5. DB::HTTPServerConnection::run() @ 0x15a0d67b in /usr/bin/clickhouse
6. Poco::Net::TCPServerConnection::start() @ 0x18667a0f in /usr/bin/clickhouse
7. Poco::Net::TCPServerDispatcher::run() @ 0x18669e61 in /usr/bin/clickhouse
8. Poco::PooledThread::run() @ 0x1881a549 in /usr/bin/clickhouse
9. Poco::ThreadImpl::runnableEntry(void*) @ 0x18817c40 in /usr/bin/clickhouse
10. start_thread @ 0x7ea7 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
11. clone @ 0xfca2f in /usr/lib/x86_64-linux-gnu/libc-2.31.so
 (version 22.2.2.1)
2023.05.18 03:07:48.260320 [ 2943751 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):

0. Poco::IOException::IOException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x187b79ec in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x18660c5b in /usr/bin/clickhouse
2. Poco::Net::SocketImpl::sendBytes(void const*, int, int) @ 0x18661dba in /usr/bin/clickhouse
3. Poco::Net::StreamSocketImpl::sendBytes(void const*, int, int) @ 0x18666896 in /usr/bin/clickhouse
4. Poco::Net::HTTPSession::write(char const*, long) @ 0x1863dd73 in /usr/bin/clickhouse
5. Poco::Net::HTTPChunkedStreamBuf::writeToDevice(char const*, long) @ 0x1862bfd5 in /usr/bin/clickhouse
6. Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::sync() @ 0x1862d0a9 in /usr/bin/clickhouse
7. Poco::Net::HTTPChunkedIOS::~HTTPChunkedIOS() @ 0x1862c1e9 in /usr/bin/clickhouse
8. Poco::Net::HTTPChunkedOutputStream::~HTTPChunkedOutputStream() @ 0x1862cd62 in /usr/bin/clickhouse
9. DB::HTTPServerConnection::run() @ 0x15a0d806 in /usr/bin/clickhouse
10. Poco::Net::TCPServerConnection::start() @ 0x18667a0f in /usr/bin/clickhouse
11. Poco::Net::TCPServerDispatcher::run() @ 0x18669e61 in /usr/bin/clickhouse
12. Poco::PooledThread::run() @ 0x1881a549 in /usr/bin/clickhouse
13. Poco::ThreadImpl::runnableEntry(void*) @ 0x18817c40 in /usr/bin/clickhouse
14. start_thread @ 0x7ea7 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
15. clone @ 0xfca2f in /usr/lib/x86_64-linux-gnu/libc-2.31.so
 (version 22.2.2.1)

Here alexey-milovidov wrote: It happens when client dropped the connection and server cannot send the result to the client.

http connect, over 1Gb network between servers, load average: 0.95, 1.15, 1.20 on 16 cpu server.

What would you recommend to do on the client?

[makorne]

The same errors after upgrade clickhouse-server to version 23.4.2.11:

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0xe3bb175 in /usr/bin/clickhouse
1. ? @ 0x8d30aec in /usr/bin/clickhouse
2. ? @ 0x8d30a05 in /usr/bin/clickhouse
3. DB::SerializationString::deserializeBinary(DB::IColumn&, DB::ReadBuffer&, DB::FormatSettings const&) const @ 0x12e21b89 in /usr/bin/clickhouse
4. DB::BinaryFormatReader::readField(DB::IColumn&, std::shared_ptr<DB::IDataType const> const&, std::shared_ptr<DB::ISerialization const> const&, bool, String const&) @ 0x14dcc1f6 in /usr/bin/clickhouse
5. DB::RowInputFormatWithNamesAndTypes::readRow(std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, DB::RowReadExtension&) @ 0x14dcf258 in /usr/bin/clickhouse
6. DB::IRowInputFormat::generate() @ 0x14da16ee in /usr/bin/clickhouse
7. DB::ISource::tryGenerate() @ 0x14d22235 in /usr/bin/clickhouse
8. DB::ISource::work() @ 0x14d21da6 in /usr/bin/clickhouse
9. DB::ExecutionThreadContext::executeTask() @ 0x14d3aa6a in /usr/bin/clickhouse
10. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x14d3073b in /usr/bin/clickhouse
11. DB::PipelineExecutor::executeImpl(unsigned long) @ 0x14d2fab6 in /usr/bin/clickhouse
12. DB::PipelineExecutor::execute(unsigned long) @ 0x14d2f751 in /usr/bin/clickhouse
13. DB::CompletedPipelineExecutor::execute() @ 0x14d2dd7d in /usr/bin/clickhouse
14. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, std::optional<DB::FormatSettings> const&) @ 0x13f073d1 in /usr/bin/clickhouse
15. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&) @ 0x14c7f9fa in /usr/bin/clickhouse
16. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x14c83c6e in /usr/bin/clickhouse
17. DB::HTTPServerConnection::run() @ 0x14cf2e1d in /usr/bin/clickhouse
18. Poco::Net::TCPServerConnection::start() @ 0x17c5cab4 in /usr/bin/clickhouse
19. Poco::Net::TCPServerDispatcher::run() @ 0x17c5dcdb in /usr/bin/clickhouse
20. Poco::PooledThread::run() @ 0x17ddb087 in /usr/bin/clickhouse
21. Poco::ThreadImpl::runnableEntry(void*) @ 0x17dd8abd in /usr/bin/clickhouse
22. start_thread @ 0x7ea7 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
23. clone @ 0xfca2f in /usr/lib/x86_64-linux-gnu/libc-2.31.so
 (version 23.4.2.11 (official build))

[loyd]

Hello, do you have some minimal example for this?

[makorne]

That's most likely clickhouse bug:

1195508094
34218