Hybrid EKS cluster with windows and linux worker nodes

[vinitmasaun]

Hello,

We have an eks cluster which includes both windows and linux worker nodes. I have the following questions:

1. Can I enable both Fluentd log engine for Linux worker nodes and Otel log engine for windows worker nodes? If this is not supported, how can I index logs from both linux and windows worker nodes in the same EKS cluster?
2. I tried deploying the 0.41.1 version of the chart with the following values.yaml overrides:

logsEngine: otel
cloudProvider: "aws"
distribution: "eks"
autodetect:
prometheus: false
istio: false
agent:
enabled: true
securityContext:
windowsOptions:
runAsUserName: "ContainerUser"
fluentd:
securityContext:
runAsUser: 0
clusterReceiver:
enabled: true
logsCollection:
containers:
enabled: true
checkpointPath: "/var/addon/splunk/otel_pos"
image:
fluentd:
repository: <private_cache_repo>/splunk/fluentd-hec
otelcol:
repository: <private_cache_repo>/signalfx/splunk-otel-collector-windows
isWindows: true
nodeSelector:
kubernetes.io/os: windows

With the above overrides, The Splunk-otel container on windows fails to start with the following error:

failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.
failed to try resolving symlinks in path "\var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log": CreateFile \var\log\pods\observability_splunk-logging-splunk-otel-collector-agent-hwwjb_c71ea1ae-4066-4d0b-80f5-0bfec70478e3\copy-old-checkpoint\8.log: The system cannot find the file specified.

[matthewmodestino]

Hi @vinitmasaun! I believe you need to have 2 deployments. One with isWindows: true and one with isWindows: false. Also, make sure to set clusterReceiver.enabled: false for one of the deployments. I would leave out fluentd for the otel native logging, personally.

[dmitryax]

@matthewmodestino provided the correct way to deploy the chart in the mixed environment, but it'll work for metrics and traces only. We don't support collection logs from Windows nodes at this point, this is coming soon.

[dmitryax]

We have docs for Windows here (should be probably moved to docs/advanced-configuration.md): https://github.com/signalfx/splunk-otel-collector-chart/blob/main/examples/README.md#deploy-for-k8s-cluster-with-windows-worker-node

[vinitmasaun]

Thank you Dmitrii. Is there a timeline for when windows logs will be supported? Is there an experimental branch that I could try out that has this support? From: Dmitrii Anoshin ***@***.***> Date: Tuesday, January 11, 2022 at 8:53 AM To: signalfx/splunk-otel-collector-chart ***@***.***> Cc: Masaun, Vinit (Associated) ***@***.***>, Mention ***@***.***> Subject: Re: [signalfx/splunk-otel-collector-chart] Hybrid EKS cluster with windows and linux worker nodes (Issue #351) EXTERNAL MESSAGE @matthewmodestino<https://github.com/matthewmodestino> provided the correct way to deploy the chart in the mixed environment, but it'll work for metrics and traces only. We don't support collection logs from Windows nodes at this point, this is coming soon. — Reply to this email directly, view it on GitHub<#351 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGNC5H5BJFIYOAVBWLXYW5TUVROBRANCNFSM5LVGTSEA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

…

________________________________ The information contained in this message is intended only for the recipient, and may be a confidential attorney-client communication or may otherwise be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, please be aware that any dissemination or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify us by replying to the message and deleting it from your computer. S&P Global Inc. reserves the right, subject to applicable local law, to monitor, review and process the content of any electronic message or information sent to or from S&P Global Inc. e-mail addresses without informing the sender or recipient of the message. By sending electronic message or information to S&P Global Inc. e-mail addresses you, as the sender, are consenting to S&P Global Inc. processing any of your personal data therein.

[dmitryax]

@vinitmasaun it's roughly 1 month away

[vinitmasaun]

@vinitmasaun it's roughly 1 month away

thank you! Anxiously looking forward to it!

[dmitryax]

Hey, the Windows logs collection support is added in this PR and will be available in the next release