Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transform is sent to sink Clickhouse in which this transform is not specified #18625

Closed
owirtifo opened this issue Sep 21, 2023 · 7 comments
Closed

Transform is sent to sink Clickhouse in which this transform is not specified #18625

owirtifo opened this issue Sep 21, 2023 · 7 comments
Labels
type: bug A code related bug.

Comments

@owirtifo
Copy link

owirtifo commented Sep 21, 2023 •
edited by bruceg
Loading

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Transform is sent to sink Clickhouse in which this transform is not specified. Entries appear in 2 different tables at the same time.

Configuration

[sources.test]
type = "socket"
address = "0.0.0.0:9009"
mode = "udp"

[transforms.cisco_parser]
    inputs       = ["test"]
    type         = "remap"
    source = '''
    parsed, err = parse_grok(.message, "^<%{POSINT:facility}>%{NUMBER:seq}: %{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")
    if err == null {
      . |= parsed
    }
    .severity = to_string((to_syslog_level!(to_int!(.severity))))
    .timestamp = now()
    '''

[transforms.syslog_parser]
    inputs       = ["test"]
    type         = "remap"
    source = ". |= parse_syslog!(.message)"

[sinks.click_netdev]
type = "clickhouse"
inputs = ["cisco_parser"]
endpoint = "http://clickhouse:8123"
database = "testdb"
table = "netdev_table"
skip_unknown_fields = true
compression = "gzip"
#encoding.timestamp_format = "unix"
date_time_best_effort = true
batch.max_events = 10
batch.timeout_secs = 10

[sinks.click_syslog]
type = "clickhouse"
inputs = ["syslog_parser"]
endpoint = "http://clickhouse:8123"
database = "testdb"
table = "syslog_table"
skip_unknown_fields = true
compression = "gzip"
encoding.timestamp_format = "unix"
date_time_best_effort = true
batch.max_events = 10
batch.timeout_secs = 10

Version

vector 0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)

Debug Output

2023-09-21T11:28:59.383319Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-09-21T11:28:59.383424Z  INFO vector::app: Log level is enabled. level="vector=trace,codec=trace,vrl=trace,file_source=trace,tower_limit=trace,rdkafka=trace,buffers=trace,lapin=trace,kube=trace"
2023-09-21T11:28:59.383472Z  WARN vector::app: DEPRECATED The openssl legacy provider provides algorithms and key sizes no longer recommended for use. Set `--openssl-legacy-provider=false` or `VECTOR_OPENSSL_LEGACY_PROVIDER=false` to disable. See https://vector.dev/highlights/2023-08-15-0-32-0-upgrade-guide/#legacy-openssl for details.
2023-09-21T11:28:59.384732Z  INFO vector::app: Loaded openssl provider. provider="legacy"
2023-09-21T11:28:59.384825Z  INFO vector::app: Loaded openssl provider. provider="default"
2023-09-21T11:28:59.387471Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-09-21T11:28:59.391773Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-09-21T11:28:59.408400Z DEBUG vector::topology::builder: Building new source. component=test
2023-09-21T11:28:59.409090Z DEBUG vector::topology::builder: Building new transform. component=cisco_parser
2023-09-21T11:28:59.422752Z DEBUG vector::topology::builder: Building new transform. component=syslog_parser
2023-09-21T11:28:59.423341Z DEBUG vector::topology::builder: Building new sink. component=click_netdev
2023-09-21T11:28:59.520091Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_core::tls::settings: Fetching system root certs.
2023-09-21T11:28:59.610344Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_core::tls::settings: Fetching system root certs.
2023-09-21T11:28:59.662802Z DEBUG vector::topology::builder: Building new sink. component=click_syslog
2023-09-21T11:28:59.706536Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_core::tls::settings: Fetching system root certs.
2023-09-21T11:28:59.763118Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_core::tls::settings: Fetching system root certs.
2023-09-21T11:28:59.811029Z  INFO vector::topology::running: Running healthchecks.
2023-09-21T11:28:59.811488Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=http://clickhouse:8123//?query=SELECT%201 method=GET version=HTTP/1.1 headers={"authorization": Sensitive, "user-agent": "Vector/0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)", "accept-encoding": "identity"} body=[empty]
2023-09-21T11:28:59.811958Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=http://clickhouse:8123//?query=SELECT%201 method=GET version=HTTP/1.1 headers={"authorization": Sensitive, "user-agent": "Vector/0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)", "accept-encoding": "identity"} body=[empty]
2023-09-21T11:28:59.812299Z DEBUG vector::topology::running: Connecting changed/added component(s).
2023-09-21T11:28:59.812496Z DEBUG vector::topology::running: Configuring outputs for source. component=test
2023-09-21T11:28:59.812671Z DEBUG vector::topology::running: Configuring output for component. component=test output_id=None
2023-09-21T11:28:59.812771Z DEBUG vector::topology::running: Configuring outputs for transform. component=cisco_parser
2023-09-21T11:28:59.812860Z DEBUG vector::topology::running: Configuring output for component. component=cisco_parser output_id=None
2023-09-21T11:28:59.813006Z DEBUG vector::topology::running: Configuring outputs for transform. component=syslog_parser
2023-09-21T11:28:59.813106Z DEBUG vector::topology::running: Configuring output for component. component=syslog_parser output_id=None
2023-09-21T11:28:59.813196Z DEBUG vector::topology::running: Connecting inputs for transform. component=cisco_parser
2023-09-21T11:28:59.813296Z DEBUG vector::topology::running: Adding component input to fanout. component=cisco_parser fanout_id=test
2023-09-21T11:28:59.813392Z DEBUG vector::topology::running: Connecting inputs for transform. component=syslog_parser
2023-09-21T11:28:59.813491Z DEBUG vector::topology::running: Adding component input to fanout. component=syslog_parser fanout_id=test
2023-09-21T11:28:59.813607Z DEBUG vector::topology::running: Connecting inputs for sink. component=click_netdev
2023-09-21T11:28:59.813701Z DEBUG vector::topology::running: Adding component input to fanout. component=click_netdev fanout_id=cisco_parser
2023-09-21T11:28:59.813802Z DEBUG vector::topology::running: Connecting inputs for sink. component=click_syslog
2023-09-21T11:28:59.813894Z DEBUG vector::topology::running: Adding component input to fanout. component=click_syslog fanout_id=syslog_parser
2023-09-21T11:28:59.814009Z DEBUG vector::topology::running: Spawning new source. key=test
2023-09-21T11:28:59.814117Z DEBUG vector::topology::running: Registered new allocation group. component_kind="source" component_type="socket" component_id="test" group_id="2"
2023-09-21T11:28:59.814255Z DEBUG source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::topology::builder: Source pump supervisor starting.
2023-09-21T11:28:59.814335Z DEBUG source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::topology::builder: Source pump starting.
2023-09-21T11:28:59.814477Z DEBUG source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::topology::builder: Source starting.
2023-09-21T11:28:59.814578Z  INFO source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::sources::socket::udp: Listening. address=0.0.0.0:9009
2023-09-21T11:28:59.814837Z DEBUG vector::topology::running: Spawning new transform. key=cisco_parser
2023-09-21T11:28:59.815119Z DEBUG vector::topology::running: Registered new allocation group. component_kind="transform" component_type="remap" component_id="cisco_parser" group_id="3"
2023-09-21T11:28:59.815230Z DEBUG vector::topology::running: Spawning new transform. key=syslog_parser
2023-09-21T11:28:59.815337Z DEBUG vector::topology::running: Registered new allocation group. component_kind="transform" component_type="remap" component_id="syslog_parser" group_id="4"
2023-09-21T11:28:59.815905Z DEBUG transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector::topology::builder: Synchronous transform starting.
2023-09-21T11:28:59.815940Z DEBUG transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector::topology::builder: Synchronous transform starting.
2023-09-21T11:28:59.816051Z TRACE vector::topology::running: Spawning new sink. key=click_netdev
2023-09-21T11:28:59.816195Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="clickhouse" component_id="click_netdev" group_id="5"
2023-09-21T11:28:59.816315Z TRACE vector::topology::running: Spawning new sink. key=click_syslog
2023-09-21T11:28:59.816420Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="clickhouse" component_id="click_syslog" group_id="6"
2023-09-21T11:28:59.816595Z  INFO vector: Vector has started. debug="false" version="0.32.1" arch="x86_64" revision="9965884 2023-08-21 14:52:38.330227446"
2023-09-21T11:28:59.816698Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-09-21T11:28:59.817100Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector::topology::builder: Sink starting.
2023-09-21T11:28:59.817112Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector::topology::builder: Sink starting.
2023-09-21T11:28:59.817490Z DEBUG http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"date": "Thu, 21 Sep 2023 11:28:59 GMT", "connection": "Keep-Alive", "content-type": "text/tab-separated-values; charset=UTF-8", "x-clickhouse-server-display-name": "clickhouse", "transfer-encoding": "chunked", "keep-alive": "timeout=3"} body=[unknown]
2023-09-21T11:28:59.817615Z  INFO vector::topology::builder: Healthcheck passed.
2023-09-21T11:28:59.818188Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector::utilization: utilization=0.16123565263857365
2023-09-21T11:28:59.818380Z TRACE vector: Beep.
2023-09-21T11:28:59.818414Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector::utilization: utilization=0.09485695730367016
2023-09-21T11:28:59.821326Z DEBUG http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"date": "Thu, 21 Sep 2023 11:28:59 GMT", "connection": "Keep-Alive", "content-type": "text/tab-separated-values; charset=UTF-8", "x-clickhouse-server-display-name": "clickhouse", "transfer-encoding": "chunked", "keep-alive": "timeout=3"} body=[unknown]
2023-09-21T11:28:59.821397Z  INFO vector::topology::builder: Healthcheck passed.
2023-09-21T11:29:00.818690Z TRACE vector: Beep.
2023-09-21T11:29:01.798197Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::bytes_received: Bytes received. byte_size=392 protocol=udp
2023-09-21T11:29:01.798371Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::internal_events::socket: Events received. count=1 byte_size=406 mode=udp
2023-09-21T11:29:01.798519Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:01.798552Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=510 output=_default
2023-09-21T11:29:01.798866Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Processing control message outside of send: ControlMessage::Add(ComponentKey { id: "cisco_parser" })
2023-09-21T11:29:01.798907Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Processing control message outside of send: ControlMessage::Add(ComponentKey { id: "syslog_parser" })
2023-09-21T11:29:01.798967Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:01.799030Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:01.799070Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:01.799110Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=510
2023-09-21T11:29:01.799108Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=510
2023-09-21T11:29:01.799329Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_core::fanout: Processing control message outside of send: ControlMessage::Add(ComponentKey { id: "click_netdev" })
2023-09-21T11:29:01.799370Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:01.799400Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:01.799440Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=529 output=_default
2023-09-21T11:29:01.799499Z TRACE sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_common::internal_event::events_received: Events received. count=1 byte_size=529
2023-09-21T11:29:01.799396Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_core::fanout: Processing control message outside of send: ControlMessage::Add(ComponentKey { id: "click_syslog" })
2023-09-21T11:29:01.799606Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:01.799651Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:01.799685Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=546 output=_default
2023-09-21T11:29:01.799738Z TRACE sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_common::internal_event::events_received: Events received. count=1 byte_size=546
2023-09-21T11:29:01.818797Z TRACE vector: Beep.
2023-09-21T11:29:02.818779Z TRACE vector: Beep.
2023-09-21T11:29:03.818857Z TRACE vector: Beep.
2023-09-21T11:29:04.819094Z TRACE vector: Beep.
2023-09-21T11:29:04.819174Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector::utilization: utilization=0.0094992971667434
2023-09-21T11:29:04.819178Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector::utilization: utilization=0.016138560854673842
2023-09-21T11:29:04.908771Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::bytes_received: Bytes received. byte_size=392 protocol=udp
2023-09-21T11:29:04.908863Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::internal_events::socket: Events received. count=1 byte_size=406 mode=udp
2023-09-21T11:29:04.908948Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:04.908978Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=510 output=_default
2023-09-21T11:29:04.909046Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:04.909134Z DEBUG transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector::utilization: utilization=0.07173022356886494
2023-09-21T11:29:04.909220Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=510
2023-09-21T11:29:04.909447Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:04.909486Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:04.909514Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=529 output=_default
2023-09-21T11:29:04.909568Z TRACE sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_common::internal_event::events_received: Events received. count=1 byte_size=529
2023-09-21T11:29:04.909677Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:04.909711Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:04.909747Z DEBUG transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector::utilization: utilization=0.07162284160167964
2023-09-21T11:29:04.909783Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=510
2023-09-21T11:29:04.909932Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:04.909965Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:04.909993Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=546 output=_default
2023-09-21T11:29:04.910039Z TRACE sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_common::internal_event::events_received: Events received. count=1 byte_size=546
2023-09-21T11:29:05.819160Z TRACE vector: Beep.
2023-09-21T11:29:06.818532Z TRACE vector: Beep.
2023-09-21T11:29:07.818351Z TRACE vector: Beep.
2023-09-21T11:29:08.029268Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::bytes_received: Bytes received. byte_size=393 protocol=udp
2023-09-21T11:29:08.029394Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::internal_events::socket: Events received. count=1 byte_size=407 mode=udp
2023-09-21T11:29:08.029483Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:08.029512Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=511 output=_default
2023-09-21T11:29:08.029578Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:08.029621Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:08.029662Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:08.029681Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=511
2023-09-21T11:29:08.029703Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=511
2023-09-21T11:29:08.030098Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:08.030140Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:08.030174Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=547 output=_default
2023-09-21T11:29:08.030229Z TRACE sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_common::internal_event::events_received: Events received. count=1 byte_size=547
2023-09-21T11:29:08.030541Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:08.030581Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:08.030639Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=530 output=_default
2023-09-21T11:29:08.030686Z TRACE sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_common::internal_event::events_received: Events received. count=1 byte_size=530
2023-09-21T11:29:08.818894Z TRACE vector: Beep.
2023-09-21T11:29:09.564287Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::bytes_received: Bytes received. byte_size=196 protocol=udp
2023-09-21T11:29:09.564394Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::internal_events::socket: Events received. count=1 byte_size=210 mode=udp
2023-09-21T11:29:09.564474Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.564503Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=314 output=_default
2023-09-21T11:29:09.564567Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.564637Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.564669Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.564711Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=314
2023-09-21T11:29:09.564863Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=314
2023-09-21T11:29:09.565029Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.565062Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.565091Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=350 output=_default
2023-09-21T11:29:09.565140Z TRACE sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_common::internal_event::events_received: Events received. count=1 byte_size=350
2023-09-21T11:29:09.565193Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.565222Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.565250Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=333 output=_default
2023-09-21T11:29:09.565290Z TRACE sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_common::internal_event::events_received: Events received. count=1 byte_size=333
2023-09-21T11:29:09.566226Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::bytes_received: Bytes received. byte_size=190 protocol=udp
2023-09-21T11:29:09.566299Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::internal_events::socket: Events received. count=1 byte_size=204 mode=udp
2023-09-21T11:29:09.566365Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.566398Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=308 output=_default
2023-09-21T11:29:09.566456Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.566522Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.566560Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.566654Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=308
2023-09-21T11:29:09.566789Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=308
2023-09-21T11:29:09.566908Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.566955Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.566988Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=344 output=_default
2023-09-21T11:29:09.567036Z TRACE sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_common::internal_event::events_received: Events received. count=1 byte_size=344
2023-09-21T11:29:09.567088Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.567123Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.567150Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=327 output=_default
2023-09-21T11:29:09.567193Z TRACE sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_common::internal_event::events_received: Events received. count=1 byte_size=327
2023-09-21T11:29:09.648922Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::bytes_received: Bytes received. byte_size=196 protocol=udp
2023-09-21T11:29:09.649018Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::internal_events::socket: Events received. count=1 byte_size=210 mode=udp
2023-09-21T11:29:09.649100Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.649130Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=314 output=_default
2023-09-21T11:29:09.649195Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.649241Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.649273Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.649306Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=314
2023-09-21T11:29:09.649322Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=314
2023-09-21T11:29:09.649510Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.649561Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.649613Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=333 output=_default
2023-09-21T11:29:09.649682Z TRACE sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_common::internal_event::events_received: Events received. count=1 byte_size=333
2023-09-21T11:29:09.650186Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::bytes_received: Bytes received. byte_size=190 protocol=udp
2023-09-21T11:29:09.650242Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::internal_events::socket: Events received. count=1 byte_size=204 mode=udp
2023-09-21T11:29:09.650302Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.650330Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=308 output=_default
2023-09-21T11:29:09.650385Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.650415Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.650445Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.650480Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=308
2023-09-21T11:29:09.650723Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.650761Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.650788Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=327 output=_default
2023-09-21T11:29:09.650837Z TRACE sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_common::internal_event::events_received: Events received. count=1 byte_size=327
2023-09-21T11:29:09.649533Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.650926Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.650954Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=350 output=_default
2023-09-21T11:29:09.650992Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=308
2023-09-21T11:29:09.651082Z TRACE sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_common::internal_event::events_received: Events received. count=1 byte_size=350
2023-09-21T11:29:09.651133Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:09.651163Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:09.651190Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=344 output=_default
2023-09-21T11:29:09.651228Z TRACE sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_common::internal_event::events_received: Events received. count=1 byte_size=344
2023-09-21T11:29:09.818988Z TRACE vector: Beep.
2023-09-21T11:29:09.819058Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector::utilization: utilization=0.0009943732936318933
2023-09-21T11:29:09.819118Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector::utilization: utilization=0.0016643787727392762
2023-09-21T11:29:10.818366Z TRACE vector: Beep.
2023-09-21T11:29:11.151492Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::bytes_received: Bytes received. byte_size=393 protocol=udp
2023-09-21T11:29:11.151574Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector::internal_events::socket: Events received. count=1 byte_size=407 mode=udp
2023-09-21T11:29:11.151651Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:11.151682Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=511 output=_default
2023-09-21T11:29:11.151747Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:11.151793Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:11.151834Z TRACE source{component_kind="source" component_id=test component_type=socket component_name=test}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:11.151877Z DEBUG transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector::utilization: utilization=0.007493456026416777
2023-09-21T11:29:11.151914Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=511
2023-09-21T11:29:11.152059Z DEBUG transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector::utilization: utilization=0.007372685067318217
2023-09-21T11:29:11.152098Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_received: Events received. count=1 byte_size=511
2023-09-21T11:29:11.152244Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:11.152277Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:11.152305Z TRACE transform{component_kind="transform" component_id=cisco_parser component_type=remap component_name=cisco_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=530 output=_default
2023-09-21T11:29:11.152354Z TRACE sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}: vector_common::internal_event::events_received: Events received. count=1 byte_size=530
2023-09-21T11:29:11.152407Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_buffers::topology::channel::limited_queue: Sent item.
2023-09-21T11:29:11.152436Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_core::fanout: Sent item to fanout.
2023-09-21T11:29:11.152464Z TRACE transform{component_kind="transform" component_id=syslog_parser component_type=remap component_name=syslog_parser}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=547 output=_default
2023-09-21T11:29:11.152503Z TRACE sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_common::internal_event::events_received: Events received. count=1 byte_size=547
2023-09-21T11:29:11.803562Z TRACE sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}: vector_core::stream::driver: Submitting service request. in_flight_requests=0 request_id=1

2023-09-21T11:27:49.479852Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://clickhouse:8123/?input_format_import_nested_json=1&input_format_skip_unknown_fields=1&date_time_input_format=best_effort&query=INSERT+INTO+%22testdb%22.%22netdev_table%22+FORMAT+JSONEachRow method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "content-length": "412", "content-encoding": "gzip", "authorization": Sensitive, "user-agent": "Vector/0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)", "accept-encoding": "identity"} body=[412 bytes]
2023-09-21T11:27:49.479845Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://clickhouse:8123/?input_format_import_nested_json=1&input_format_skip_unknown_fields=1&date_time_input_format=best_effort&query=INSERT+INTO+%22testdb%22.%22syslog_table%22+FORMAT+JSONEachRow method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "content-length": "348", "content-encoding": "gzip", "authorization": Sensitive, "user-agent": "Vector/0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)", "accept-encoding": "identity"} body=[348 bytes]
2023-09-21T11:27:49.489461Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"date": "Thu, 21 Sep 2023 11:27:49 GMT", "connection": "Keep-Alive", "content-type": "text/plain; charset=UTF-8", "x-clickhouse-server-display-name": "clickhouse", "transfer-encoding": "chunked", "keep-alive": "timeout=3"} body=[unknown]
2023-09-21T11:27:49.489896Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"date": "Thu, 21 Sep 2023 11:27:49 GMT", "connection": "Keep-Alive", "content-type": "text/plain; charset=UTF-8", "x-clickhouse-server-display-name": "clickhouse", "transfer-encoding": "chunked", "keep-alive": "timeout=3"} body=[unknown]

Example Data

2023-09-21T11:27:49.479852Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://clickhouse:8123/?input_format_import_nested_json=1&input_format_skip_unknown_fields=1&date_time_input_format=best_effort&query=INSERT+INTO+%22testdb%22.%22netdev_table%22+FORMAT+JSONEachRow method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "content-length": "412", "content-encoding": "gzip", "authorization": Sensitive, "user-agent": "Vector/0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)", "accept-encoding": "identity"} body=[412 bytes]
2023-09-21T11:27:49.479845Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://clickhouse:8123/?input_format_import_nested_json=1&input_format_skip_unknown_fields=1&date_time_input_format=best_effort&query=INSERT+INTO+%22testdb%22.%22syslog_table%22+FORMAT+JSONEachRow method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "content-length": "348", "content-encoding": "gzip", "authorization": Sensitive, "user-agent": "Vector/0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)", "accept-encoding": "identity"} body=[348 bytes]
2023-09-21T11:27:49.489461Z DEBUG sink{component_kind="sink" component_id=click_netdev component_type=clickhouse component_name=click_netdev}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"date": "Thu, 21 Sep 2023 11:27:49 GMT", "connection": "Keep-Alive", "content-type": "text/plain; charset=UTF-8", "x-clickhouse-server-display-name": "clickhouse", "transfer-encoding": "chunked", "keep-alive": "timeout=3"} body=[unknown]
2023-09-21T11:27:49.489896Z DEBUG sink{component_kind="sink" component_id=click_syslog component_type=clickhouse component_name=click_syslog}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"date": "Thu, 21 Sep 2023 11:27:49 GMT", "connection": "Keep-Alive", "content-type": "text/plain; charset=UTF-8", "x-clickhouse-server-display-name": "clickhouse", "transfer-encoding": "chunked", "keep-alive": "timeout=3"} body=[unknown]

Additional Context

No response

References

No response
@owirtifo owirtifo added the type: bug A code related bug. label Sep 21, 2023
@jszwedko
Copy link
Member

Hi @owirtifo !

Looking at your config, you are fanning out the udp source to two transforms which feed into two Clickhouse sinks so all events will be sent to both. It seems like you might want to use a route transform to split up the event streams rather than fanning out.

I'll close this out, but please leave a comment if I am misunderstanding.

@jszwedko jszwedko closed this as not planned Won't fix, can't repro, duplicate, stale Sep 21, 2023
@owirtifo
Copy link
Author

owirtifo commented Sep 22, 2023 •
edited
Loading

Hi! @jszwedko

I would like to implement a scheme in which transforms.cisco_parser messages were sent to sinks.click_netdev, and transforms.syslog_parser messages were sent to sinks.click_syslog. Not both at the same time.
With the following configuration, it works fine for me:

[sources.sys]
type = "socket"
address = "0.0.0.0:9009"
mode = "udp"

[sources.netdev]
type = "socket"
address = "0.0.0.0:8008"
mode = "udp"

[transforms.cisco_parser]
    inputs       = ["netdev"]
    type         = "remap"
    source = '''
    parsed, err = parse_grok(.message, "^<%{POSINT:facility}>%{NUMBER:seq}: %{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")
    if err == null {
      . |= parsed
    }
    .severity = to_string((to_syslog_level!(to_int!(.severity))))
    .timestamp = now()
    '''

[transforms.syslog_parser]
    inputs       = ["sys"]
    type         = "remap"
    source = ". |= parse_syslog!(.message)"
	
[sinks.click_netdev]
type = "clickhouse"
inputs = ["cisco_parser"]
endpoint = "http://clickhouse:8123"
database = "testdb"
table = "netdev_table"
skip_unknown_fields = true
compression = "gzip"
date_time_best_effort = true
batch.max_events = 10
batch.timeout_secs = 10

[sinks.click_syslog]
type = "clickhouse"
inputs = ["sys"]
endpoint = "http://clickhouse:8123"
database = "testdb"
table = "syslog_table"
skip_unknown_fields = true
compression = "gzip"
encoding.timestamp_format = "unix"
date_time_best_effort = true
batch.max_events = 10
batch.timeout_secs = 10

@jszwedko
Copy link
Member

Hi @owirtifo ,

Gotcha, I see that in the update configuration you used two different sources to receive the data to route them differently. This works too rather than using a route transform.

@owirtifo
Copy link
Author

Hi @jszwedko!
I would like to use transform route instead of using two sources, but I can't understand, in my case, by what condition can I filter to redirect syslog_parser to sinks.click_syslog, and cisco_parser to click_netdev?

@jszwedko
Copy link
Member

You would need to identify some pattern in the logs that you could use to differentiate the two sources.

@owirtifo
Copy link
Author

owirtifo commented Nov 23, 2023 •
edited
Loading

Hi @owirtifo !

Looking at your config, you are fanning out the udp source to two transforms which feed into two Clickhouse sinks so all events will be sent to both. It seems like you might want to use a route transform to split up the event streams rather than fanning out.

I'll close this out, but please leave a comment if I am misunderstanding.

Hi @jszwedko!
Please explain why this is happening? After all, this is misleading. I'm not telling Vector to send all the transforms to one sink, but I'm telling Vector to send each transforms to a specific sink.

[sinks.click_netdev]
type = "clickhouse"
inputs = ["cisco_parser"]

[sinks.click_syslog]
type = "clickhouse"
inputs = ["syslog_parser"]

@bruceg
Copy link
Member

bruceg commented Nov 23, 2023

Please explain why this is happening? After all, this is misleading. I'm not telling Vector to send all the transforms to one sink, but I'm telling Vector to send each transforms to a specific sink.

You are right that each transform is going to a specific sink. However, the output of the source is going to both transforms, which is why a route before it is suggested. In the above config, the cisco_parser will never abort, so anything that can be parsed by the syslog_parser will be reported to both sinks. From what I am reading of the grok pattern, it does look at least similar enough to a syslog pattern that it seems likely that both can parse the message, hence the doubled output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: bug A code related bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jszwedko @bruceg @owirtifo