upstream_ha: upstream_node: process verify hostname on HA settings

[cosmo0920]

Reported in #9152 (comment).

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

Certificates are used from here: #8072 (comment)

Client side:

[SERVICE]
    flush 1
    log_level debug

[INPUT]
    name  dummy
    dummy {"message": "custom dummy message"}

[OUTPUT]
    name   forward
    match  *
    Upstream      upstream-ha.conf

upstream-ha.conf

[UPSTREAM]
    name        forward-balancing
[NODE]
    name          node-1
    host          other.fluent-backoffice.de
    port          24224
    tls           on
    tls.verify    on
    tls.verify_hostname on
    tls.ca_file   /path/to/necessary-certs/fluent-root.crt
    tls.crt_file  /path/to/necessary-certs/fluent-client.crt
    tls.key_file  /path/to/necessary-certs/fluent-client.key

Server side:

[SERVICE]
    flush 1
    log_level debug

[INPUT]
    name   forward
    listen 0.0.0.0
    port   24224
    tls             on
    tls.verify      on
    tls.debug       4
    tls.ca_file      /path/to/necessary-certs/fluent-root.crt
    tls.crt_file     /path/to/necessary-certs/fluent-backoffice.crt
    tls.key_file     /path/to/necessary-certs/fluent-backoffice.key

[OUTPUT]
    name stdout
    match *

• Debug log output from testing the change

With this change, upstream_ha refuses to connect with invalid certificates which have a wrong server CN.

Fluent Bit v3.1.5
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __  
|  ___| |                | |   | ___ (_) |         |____ |/  | 
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| | 
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | | 
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/08/13 17:17:11] [ info] Configuration:
[2024/08/13 17:17:11] [ info]  flush time     | 1.000000 seconds
[2024/08/13 17:17:11] [ info]  grace          | 5 seconds
[2024/08/13 17:17:11] [ info]  daemon         | 0
[2024/08/13 17:17:11] [ info] ___________
[2024/08/13 17:17:11] [ info]  inputs:
[2024/08/13 17:17:11] [ info]      dummy
[2024/08/13 17:17:11] [ info] ___________
[2024/08/13 17:17:11] [ info]  filters:
[2024/08/13 17:17:11] [ info] ___________
[2024/08/13 17:17:11] [ info]  outputs:
[2024/08/13 17:17:11] [ info]      forward.0
[2024/08/13 17:17:11] [ info] ___________
[2024/08/13 17:17:11] [ info]  collectors:
[2024/08/13 17:17:11] [ info] [fluent bit] version=3.1.5, commit=725640616f, pid=1982141
[2024/08/13 17:17:11] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/08/13 17:17:11] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/08/13 17:17:11] [ info] [cmetrics] version=0.9.3
[2024/08/13 17:17:11] [ info] [ctraces ] version=0.5.3
[2024/08/13 17:17:12] [ info] [input:dummy:dummy.0] initializing
[2024/08/13 17:17:12] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/08/13 17:17:12] [debug] [dummy:dummy.0] created event channels: read=21 write=22
[2024/08/13 17:17:12] [debug] [forward:forward.0] created event channels: read=23 write=24
[2024/08/13 17:17:12] [debug] [upstream_ha] opening file upstream-ha.conf
[2024/08/13 17:17:12] [ info] [output:forward:forward.0] worker #1 started
[2024/08/13 17:17:12] [ info] [sp] stream processor started
[2024/08/13 17:17:12] [ info] [output:forward:forward.0] worker #0 started
[2024/08/13 17:17:13] [debug] [task] created task=0x6182060 id=0 OK
[2024/08/13 17:17:13] [debug] [output:forward:forward.0] task_id=0 assigned to thread #0
[2024/08/13 17:17:13] [debug] [output:forward:forward.0] request 51 bytes to flush
[2024/08/13 17:17:13] [debug] [tls] connection #49 SSL_connect: before SSL initialization
[2024/08/13 17:17:13] [debug] [tls] connection #49 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:13] [debug] [tls] connection #49 WANT_READ
[2024/08/13 17:17:14] [debug] [tls] connection #49 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:14] [debug] [tls] connection #49 SSL_connect: SSLv3/TLS read server hello
[2024/08/13 17:17:14] [debug] [retry] new retry created for task_id=0 attempts=1
[2024/08/13 17:17:14] [debug] [tls] connection #49 SSL_connect: TLSv1.3 read encrypted extensions
[2024/08/13 17:17:14] [ warn] [engine] failed to flush chunk '1982141-1723537032.793889074.flb', retry in 6 seconds: task_id=0, input=dummy.0 > output=forward.0 (out_id=0)
[2024/08/13 17:17:14] [debug] [tls] connection #49 SSL_connect: SSLv3/TLS read server certificate request
[2024/08/13 17:17:14] [debug] [tls] connection #49 SSL3 alert write:fatal:bad certificate
[2024/08/13 17:17:14] [error] [tls] connection #49 SSL_connect: error in error
[2024/08/13 17:17:14] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2024/08/13 17:17:14] [debug] [upstream] connection #49 failed to other.fluent-backoffice.de:24224
[2024/08/13 17:17:14] [error] [output:forward:forward.0] no upstream connections available
[2024/08/13 17:17:14] [debug] [out flush] cb_destroy coro_id=0
[2024/08/13 17:17:14] [debug] [task] created task=0x8250af0 id=1 OK
[2024/08/13 17:17:14] [debug] [output:forward:forward.0] task_id=1 assigned to thread #1
[2024/08/13 17:17:14] [debug] [output:forward:forward.0] request 51 bytes to flush
[2024/08/13 17:17:14] [debug] [tls] connection #50 SSL_connect: before SSL initialization
[2024/08/13 17:17:14] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:14] [debug] [tls] connection #50 WANT_READ
[2024/08/13 17:17:14] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:14] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS read server hello
[2024/08/13 17:17:14] [debug] [tls] connection #50 SSL_connect: TLSv1.3 read encrypted extensions
[2024/08/13 17:17:14] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS read server certificate request
[2024/08/13 17:17:14] [debug] [tls] connection #50 SSL3 alert write:fatal:bad certificate
[2024/08/13 17:17:14] [error] [tls] connection #50 SSL_connect: error in error
[2024/08/13 17:17:14] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2024/08/13 17:17:14] [debug] [upstream] connection #50 failed to other.fluent-backoffice.de:24224
[2024/08/13 17:17:14] [error] [output:forward:forward.0] no upstream connections available
[2024/08/13 17:17:14] [debug] [out flush] cb_destroy coro_id=0
[2024/08/13 17:17:14] [debug] [retry] new retry created for task_id=1 attempts=1
[2024/08/13 17:17:14] [ warn] [engine] failed to flush chunk '1982141-1723537033.792273399.flb', retry in 8 seconds: task_id=1, input=dummy.0 > output=forward.0 (out_id=0)
^C[2024/08/13 17:17:15] [engine] caught signal (SIGINT)
[2024/08/13 17:17:15] [debug] [task] created task=0x82f9970 id=2 OK
[2024/08/13 17:17:15] [debug] [output:forward:forward.0] task_id=2 assigned to thread #0
[2024/08/13 17:17:15] [debug] [output:forward:forward.0] request 51 bytes to flush
[2024/08/13 17:17:15] [debug] [output:forward:forward.0] request 51 bytes to flush
[2024/08/13 17:17:15] [ warn] [engine] service will shutdown in max 5 seconds
[2024/08/13 17:17:15] [debug] [engine] re-scheduled retry=0x824a870 for task 0
[2024/08/13 17:17:15] [debug] [engine] re-scheduled retry=0x82f9760 for task 1
[2024/08/13 17:17:15] [ info] [input] pausing dummy.0
[2024/08/13 17:17:15] [debug] [output:forward:forward.0] task_id=0 assigned to thread #1
[2024/08/13 17:17:15] [debug] [output:forward:forward.0] task_id=1 assigned to thread #0
[2024/08/13 17:17:15] [debug] [output:forward:forward.0] request 51 bytes to flush
[2024/08/13 17:17:15] [debug] [tls] connection #50 SSL_connect: before SSL initialization
[2024/08/13 17:17:15] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:15] [debug] [tls] connection #50 WANT_READ
[2024/08/13 17:17:15] [debug] [tls] connection #51 SSL_connect: before SSL initialization
[2024/08/13 17:17:15] [debug] [tls] connection #49 SSL_connect: before SSL initialization
[2024/08/13 17:17:15] [debug] [tls] connection #51 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:15] [debug] [tls] connection #49 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:15] [debug] [tls] connection #51 WANT_READ
[2024/08/13 17:17:15] [debug] [tls] connection #49 WANT_READ
[2024/08/13 17:17:15] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:15] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS read server hello
[2024/08/13 17:17:15] [debug] [tls] connection #50 SSL_connect: TLSv1.3 read encrypted extensions
[2024/08/13 17:17:15] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS read server certificate request
[2024/08/13 17:17:15] [debug] [tls] connection #50 SSL3 alert write:fatal:bad certificate
[2024/08/13 17:17:15] [error] [tls] connection #50 SSL_connect: error in error
[2024/08/13 17:17:15] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2024/08/13 17:17:15] [debug] [upstream] connection #50 failed to other.fluent-backoffice.de:24224
[2024/08/13 17:17:15] [error] [output:forward:forward.0] no upstream connections available
[2024/08/13 17:17:15] [debug] [out flush] cb_destroy coro_id=1
[2024/08/13 17:17:15] [debug] [retry] new retry created for task_id=2 attempts=1
[2024/08/13 17:17:15] [ warn] [engine] failed to flush chunk '1982141-1723537034.773227003.flb', retry in 1 seconds: task_id=2, input=dummy.0 > output=forward.0 (out_id=0)
[2024/08/13 17:17:15] [ info] [task] dummy/dummy.0 has 3 pending task(s):
[2024/08/13 17:17:15] [ info] [task]   task_id=0 still running on route(s): forward/forward.0 
[2024/08/13 17:17:15] [debug] [output:forward:forward.0] request 51 bytes to flush
[2024/08/13 17:17:15] [ info] [task]   task_id=1 still running on route(s): forward/forward.0 
[2024/08/13 17:17:15] [ info] [task]   task_id=2 still running on route(s): forward/forward.0 
[2024/08/13 17:17:15] [debug] [output:forward:forward.0] task_id=2 assigned to thread #1
[2024/08/13 17:17:15] [ info] [input] pausing dummy.0
[2024/08/13 17:17:15] [debug] [tls] connection #50 SSL_connect: before SSL initialization
[2024/08/13 17:17:15] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:15] [debug] [tls] connection #50 WANT_READ
[2024/08/13 17:17:15] [debug] [tls] connection #49 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:15] [debug] [tls] connection #49 SSL_connect: SSLv3/TLS read server hello
[2024/08/13 17:17:15] [debug] [tls] connection #49 SSL_connect: TLSv1.3 read encrypted extensions
[2024/08/13 17:17:15] [debug] [tls] connection #49 SSL_connect: SSLv3/TLS read server certificate request
[2024/08/13 17:17:15] [debug] [tls] connection #49 SSL3 alert write:fatal:bad certificate
[2024/08/13 17:17:15] [error] [tls] connection #49 SSL_connect: error in error
[2024/08/13 17:17:15] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2024/08/13 17:17:15] [debug] [upstream] connection #49 failed to other.fluent-backoffice.de:24224
[2024/08/13 17:17:15] [error] [output:forward:forward.0] no upstream connections available
[2024/08/13 17:17:15] [debug] [out flush] cb_destroy coro_id=1
[2024/08/13 17:17:15] [debug] [task] task_id=0 reached retry-attempts limit 1/1
[2024/08/13 17:17:15] [error] [engine] chunk '1982141-1723537032.793889074.flb' cannot be retried: task_id=0, input=dummy.0 > output=forward.0
[2024/08/13 17:17:15] [debug] [task] destroy task=0x6182060 (task_id=0)
[2024/08/13 17:17:16] [debug] [tls] connection #51 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:16] [debug] [tls] connection #51 SSL_connect: SSLv3/TLS read server hello
[2024/08/13 17:17:16] [debug] [tls] connection #51 SSL_connect: TLSv1.3 read encrypted extensions
[2024/08/13 17:17:16] [debug] [tls] connection #51 SSL_connect: SSLv3/TLS read server certificate request
[2024/08/13 17:17:16] [debug] [tls] connection #51 SSL3 alert write:fatal:bad certificate
[2024/08/13 17:17:16] [error] [tls] connection #51 SSL_connect: error in error
[2024/08/13 17:17:16] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2024/08/13 17:17:16] [debug] [upstream] connection #51 failed to other.fluent-backoffice.de:24224
[2024/08/13 17:17:16] [error] [output:forward:forward.0] no upstream connections available
[2024/08/13 17:17:16] [debug] [out flush] cb_destroy coro_id=2
[2024/08/13 17:17:16] [debug] [task] task_id=1 reached retry-attempts limit 1/1
[2024/08/13 17:17:16] [error] [engine] chunk '1982141-1723537033.792273399.flb' cannot be retried: task_id=1, input=dummy.0 > output=forward.0
[2024/08/13 17:17:16] [debug] [task] destroy task=0x8250af0 (task_id=1)
[2024/08/13 17:17:16] [ info] [input] pausing dummy.0
[2024/08/13 17:17:16] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS write client hello
[2024/08/13 17:17:16] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS read server hello
[2024/08/13 17:17:16] [debug] [tls] connection #50 SSL_connect: TLSv1.3 read encrypted extensions
[2024/08/13 17:17:16] [debug] [tls] connection #50 SSL_connect: SSLv3/TLS read server certificate request
[2024/08/13 17:17:16] [debug] [tls] connection #50 SSL3 alert write:fatal:bad certificate
[2024/08/13 17:17:16] [error] [tls] connection #50 SSL_connect: error in error
[2024/08/13 17:17:16] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2024/08/13 17:17:16] [debug] [upstream] connection #50 failed to other.fluent-backoffice.de:24224
[2024/08/13 17:17:16] [error] [output:forward:forward.0] no upstream connections available
[2024/08/13 17:17:16] [debug] [out flush] cb_destroy coro_id=2
[2024/08/13 17:17:16] [debug] [task] task_id=2 reached retry-attempts limit 1/1
[2024/08/13 17:17:16] [error] [engine] chunk '1982141-1723537034.773227003.flb' cannot be retried: task_id=2, input=dummy.0 > output=forward.0
[2024/08/13 17:17:16] [debug] [task] destroy task=0x82f9970 (task_id=2)
[2024/08/13 17:17:17] [ info] [engine] service has stopped (0 pending tasks)
[2024/08/13 17:17:17] [ info] [input] pausing dummy.0
[2024/08/13 17:17:17] [ info] [output:forward:forward.0] thread worker #0 stopping...
[2024/08/13 17:17:17] [ info] [output:forward:forward.0] thread worker #1 stopping...

• Attached Valgrind output that shows no leaks or memory corruption was found

==1982141== 
==1982141== HEAP SUMMARY:
==1982141==     in use at exit: 0 bytes in 0 blocks
==1982141==   total heap usage: 32,508 allocs, 32,508 frees, 4,428,213 bytes allocated
==1982141== 
==1982141== All heap blocks were freed -- no leaks are possible
==1982141== 
==1982141== For lists of detected and suppressed errors, rerun with: -s
==1982141== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.