message line lenght max 1024 characters, after that the messages are truncated

[StefanSa]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hi there,
i have the problem that i can receive with the source socket (syslog) at most one message in the length of 1024 characters.
Larger than 1024 characters the message will be truncated at this point. Is there a limitation in Vector or is this already a problem at the sender, in this case a firewall?

thx for any help
StefanSa

Configuration

[sources.syslog_sophos_utm]
type = "socket"
address = "0.0.0.0:9010"
mode = "tcp"

[transforms.remap_grok_sophos_utm]
type = "remap"
inputs = ["syslog_sophos_utm"]
source = '''
. = parse_grok!(.message, "%{SYSLOG5424PRI}%{DATA:timestamp} %{DATA:device} %{SYSLOGPROG}: %{GREEDYDATA:message}")
'''

[transforms.split_sophos_utm]
type = "remap"
inputs = ["remap_grok_sophos_utm"]
source = '''
. = set!(., path: ["sophos", "utm"], data: parse_key_value!(.message,
   field_delimiter: " ",
        key_value_delimiter: "="
))
#flatten(.)
.sophos.utm.program = del(.program)
.sophos.utm.pid = del(.pid)
.sophos.utm.device = del(.device)
.event.ingested = now()
.event.original = .message
'''

Version

vector 0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)

Debug Output

No response

Example Data

<30>2023:09:22-13:04:15 xxx-utm-01-1 ulogd[10853]: id="0299" srcip="95.xx.211.35" localip="172.xx.22.10" size="414" user="-" host="95.xx.211.35" method="POST" statuscode="200" reason="-" extra="-" exceptions="SkipBlacklistDNSRBL, SkipBlacklistGeoIP, SkipAntiVirus, SkipTFT, SkipURLHardening, SkipFormHardening, SkipCookieSigning, SkipThreatsFilter" time="80243" url="/mapi/emsmdb/" server="webmail.xxxx.com" port="443" query="?MailboxId=[email protected]" referer="-" cookie="-" set-cookie="MapiRouting=UlVNOjkwYzZlMGVmLTdiMTctNDA0ZC05YWEyLTE3ZWU0MmY2ZjBlODoJChPcbLzbCA==; path=/mapi/; secure; HttpOnly, MapiContext=MAPIAAAAAOin9NmQxOm/8qqbrYCwgaKQoJKhjLyFqJmggLCFv4++hLCA2vnI/M76zfzK8sr+XdEAAAAAAAA=; path=/mapi/emsmdb; secure; HttpOnly, MapiSequence=0-QSoXNA==; path=/mapi/emsmdb; secure; HttpOnly, X-BackEndCookie=9a8183e3-33aa-4edd-9a00-9253666983eb=u56Lnp2ejJqBysnOzceZmsfSms7IydLLyZvI0sbNmszSy53Hzp3GyJnJzprMgYHNz83M0s7P0s3Mq87GxczGxcrM; expires=Mon, 23-Oct-2023 19:39:53 GMT; path=/mapi; secure; HttpOnly" websocket_scheme="-" websocket_pr

Additional Context

i have read that syslog by default has a 1KB message limit. This can be increased with fragmentation feature.
i already tested "new line" and "octet", both did not help.

References

No response

[StefanSa]

Currently i help myself with this.
This helps insofar that there is no fragmentation or broken messages with single keys.
But this is not a real solution.

if strlen(string!(.message)) >= 1000  {
      del(.message)
      abort
   }

[pront]

Hello, this is probably a sender issue.

I ran the following test:

[sources.src0]
type = "socket"
address = "0.0.0.0:9010"
mode = "tcp"

[transforms.t0]
type = "remap"
inputs = ["src0"]
source = '''
.l = length!(.message)
  '''

[sinks.sink0]
inputs = ["src0"]
target = "stdout"
type = "console"

[sinks.sink0.encoding]
codec = "json"

And then I successfully sent to the socket and processed a 4096 B message.

Please feel free to re-open if you have more data to add.