YAML configuration files for Windows #6937

mp3monster · 2023-03-01T21:04:25Z

Is your feature request related to a problem? Please describe.
If YAML is to become the future format for configuration files it needs to be available for Windows deployments. Currently this isn't the case. The documentation about YAML doesn't indicate this. It would be worth including a note with the description of the configuration that it isn't a Windows option today

Describe the solution you'd like
The recommendation is to make YAML available for Windows. The min is to indicate it isn't a Windows option

Describe alternatives you've considered
documentation

Additional context
Development of book on Fluentbit - aim was to proof config using Windows before final examples being demonstrated, screenshots etc on Linux / Docker / K8s

patrick-stephens · 2023-03-02T12:29:19Z

Need libyaml and static linkage for it.

patrick-stephens · 2023-05-03T08:37:32Z

See #7296

patrick-stephens · 2023-05-25T12:27:55Z

@mp3monster I think 2.1.3+ should have YAML support now.

github-actions · 2023-08-24T01:48:44Z

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

ameya-rescale · 2023-08-24T01:50:08Z

Using a YAML config does not work with the latest Windows build zip from https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win64.zip. I'm testing with the sample config from https://github.com/fluent/fluent-bit/blob/master/packaging/testing/smoke/container/fluent-bit.yaml, but it returns the error [error] [config] error in .\conf\test.yaml:1: undefined value - check config is in valid classic format. Is using a YAML config only supported for the Windows container image?

patrick-stephens · 2023-08-24T05:34:33Z

Unless libyaml is installed, it is disabled:

fluent-bit/cmake/windows-setup.cmake

Line 24 in a2105c2

set(FLB_CONFIG_YAML No)

Currently we do not install it during Windows builds: https://github.com/fluent/fluent-bit/blob/master/.github/workflows/call-build-windows.yaml

A PR to resolve that would be appreciated however it needs to ensure static linkage to libyaml as well, otherwise complaints about extra dependencies will be generated.

CircuitGuy · 2023-11-14T02:08:34Z

This one just bit me as a new user and it was very confusing given the docs

[YAML] is production ready since Fluent Bit 2.0.

https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml

Given that statement, and being a new user, I was very confused when I downloaded a v2.1.x Windows bin and repeatedly failed to read a YAML file. This was made worse by an unhelpful error message.

mp3monster · 2023-11-14T07:52:15Z

Can you share an example of what didn't work for you?

ng-bsy · 2023-12-05T11:31:43Z

Unless libyaml is installed, it is disabled:

fluent-bit/cmake/windows-setup.cmake

Line 24 in a2105c2

set(FLB_CONFIG_YAML No)

Currently we do not install it during Windows builds: https://github.com/fluent/fluent-bit/blob/master/.github/workflows/call-build-windows.yaml

A PR to resolve that would be appreciated however it needs to ensure static linkage to libyaml as well, otherwise complaints about extra dependencies will be generated.

If YAML ain't supported on Windows, maybe the documentation should be corrected accordingly?
As it claims

[YAML] is production ready since Fluent Bit 2.0.

Yet

Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/12/05 12:12:15] [error] [config] error in C:\Program Files\fluent-bit\conf\fluent-bit.yaml:1: undefined value - check config is in valid classic format
[2023/12/05 12:12:15] [error] configuration file contains errors, aborting.

patrick-stephens · 2023-12-05T11:54:58Z

Is this an official Windows package because we do install libyaml when building those? If that's not working then it's a bug:

fluent-bit/.github/workflows/call-build-windows.yaml

Lines 143 to 146 in 323f343

    
                 - name: Build libyaml with vcpkg 
        
                   run: | 
        
                     C:\vcpkg\vcpkg install --recurse libyaml --triplet ${{ matrix.config.vcpkg_triplet }} 
        
                   shell: cmd

If it's a custom build then you need to ensure you provide libyaml as well. YAML config should work for Windows is my understanding.

patrick-stephens · 2023-12-05T11:56:20Z

If it's the Windows containers then that might be the issue as they need a bit of love and libyaml adding so a PR would be appreciated there: https://github.com/fluent/fluent-bit/blob/master/dockerfiles/Dockerfile.windows

ng-bsy · 2023-12-05T12:43:06Z

Is this an official Windows package because we do install libyaml when building those? If that's not working then it's a bug:

fluent-bit/.github/workflows/call-build-windows.yaml

Lines 143 to 146 in 323f343

- name: Build libyaml with vcpkg

run: |

C:\vcpkg\vcpkg install --recurse libyaml --triplet ${{ matrix.config.vcpkg_triplet }}

shell: cmd

If it's a custom build then you need to ensure you provide libyaml as well. YAML config should work for Windows is my understanding.

I installed via https://releases.fluentbit.io/2.2/fluent-bit-2.2.0-win64.exe
Where should I find the libyaml if it's being installed? It's nowhere to be found in the install folder.

patrick-stephens · 2023-12-05T13:58:23Z

It is statically linked, we cannot do anything else. At least that's the intent.

@cosmo0920 it looks like we may need to revisit this.

cosmo0920 · 2023-12-05T15:29:40Z

Sure, I'll revisit on this tomorrow.

patrick-stephens · 2023-12-05T16:04:10Z

Is this an official Windows package because we do install libyaml when building those? If that's not working then it's a bug:

fluent-bit/.github/workflows/call-build-windows.yaml

Lines 143 to 146 in 323f343

- name: Build libyaml with vcpkg

run: |

C:\vcpkg\vcpkg install --recurse libyaml --triplet ${{ matrix.config.vcpkg_triplet }}

shell: cmd

If it's a custom build then you need to ensure you provide libyaml as well. YAML config should work for Windows is my understanding.

I installed via https://releases.fluentbit.io/2.2/fluent-bit-2.2.0-win64.exe Where should I find the libyaml if it's being installed? It's nowhere to be found in the install folder.

Ah, this is the AppVeyor builds I think which are deprecated and use their own CI config to build.

@ng-bsy Could you try the Github ones from https://packages.fluentbit.io/windows/?

ng-bsy · 2023-12-05T16:53:39Z

Is this an official Windows package because we do install libyaml when building those? If that's not working then it's a bug:

fluent-bit/.github/workflows/call-build-windows.yaml

Lines 143 to 146 in 323f343

- name: Build libyaml with vcpkg

run: |

C:\vcpkg\vcpkg install --recurse libyaml --triplet ${{ matrix.config.vcpkg_triplet }}

shell: cmd

If it's a custom build then you need to ensure you provide libyaml as well. YAML config should work for Windows is my understanding.

I installed via https://releases.fluentbit.io/2.2/fluent-bit-2.2.0-win64.exe Where should I find the libyaml if it's being installed? It's nowhere to be found in the install folder.

Ah, this is the AppVeyor builds I think which are deprecated and use their own CI config to build.

@ng-bsy Could you try the Github ones from https://packages.fluentbit.io/windows/?

Will give it a try 👍
If it works, this doc should be updated:
https://docs.fluentbit.io/manual/installation/windows#installation-packages
It points to https://releases.fluentbit.io/2.2/

ng-bsy · 2023-12-05T17:03:38Z

Is this an official Windows package because we do install libyaml when building those? If that's not working then it's a bug:

fluent-bit/.github/workflows/call-build-windows.yaml

Lines 143 to 146 in 323f343

- name: Build libyaml with vcpkg

run: |

C:\vcpkg\vcpkg install --recurse libyaml --triplet ${{ matrix.config.vcpkg_triplet }}

shell: cmd

If it's a custom build then you need to ensure you provide libyaml as well. YAML config should work for Windows is my understanding.

I installed via https://releases.fluentbit.io/2.2/fluent-bit-2.2.0-win64.exe Where should I find the libyaml if it's being installed? It's nowhere to be found in the install folder.

Ah, this is the AppVeyor builds I think which are deprecated and use their own CI config to build.

@ng-bsy Could you try the Github ones from https://packages.fluentbit.io/windows/?

Sadly, the same problem...

cosmo0920 · 2023-12-06T06:34:32Z

Really? I tested fluent-bit 2.2.0 win64 zip and I obtained the following result.
I extracted the zip file into my home folder directly and using yaml configuration which contains processor stack:

C:\Users\cosmo PS> cd .\fluent-bit-2.2.0-win64\fluent-bit-2.2.0-win64\
C:\Users\cosmo\fluent-bit-2.2.0-win64\fluent-bit-2.2.0-win64 PS> bin\fluent-bit -c C:\Users\cosmo\Documents\processors.yaml
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io


[2023/12/06 15:30:55] [ info] [fluent bit] version=2.2.0, commit=db8487d123, pid=8416
[2023/12/06 15:30:55] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/12/06 15:30:55] [ info] [cmetrics] version=0.6.4
[2023/12/06 15:30:55] [ info] [ctraces ] version=0.3.1
[2023/12/06 15:30:55] [ info] [input:dummy:dummy.0] initializing
[2023/12/06 15:30:55] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/12/06 15:30:55] [ warn] [filter:rewrite_tag:rewrite_tag.0] 'Match' may cause infinite loop.
[2023/12/06 15:30:55] [ info] [input:emitter:re_emitted] initializing
[2023/12/06 15:30:55] [ info] [input:emitter:re_emitted] storage_strategy='memory' (memory only)
[2023/12/06 15:30:55] [ info] [sp] stream processor started
[2023/12/06 15:30:55] [ info] [output:stdout:stdout.0] worker #0 started
[0] from.test-tag.raw.new.fluent.bit.out: [[1701844256.091933250, {}], {"tag"=>"from.test-tag.raw.new.fluent.bit.out", "hostname"=>"monox", "output"=>"new data", "tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[0] from.test-tag.raw.new.fluent.bit.out: [[1701844257.098164081, {}], {"tag"=>"from.test-tag.raw.new.fluent.bit.out", "hostname"=>"monox", "output"=>"new data", "tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[0] from.test-tag.raw.new.fluent.bit.out: [[1701844258.092186450, {}], {"tag"=>"from.test-tag.raw.new.fluent.bit.out", "hostname"=>"monox", "output"=>"new data", "tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[0] from.test-tag.raw.new.fluent.bit.out: [[1701844259.093399524, {}], {"tag"=>"from.test-tag.raw.new.fluent.bit.out", "hostname"=>"monox", "output"=>"new data", "tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[2023/12/06 15:31:00] [engine] caught signal (SIGINT)
[0] from.test-tag.raw.new.fluent.bit.out: [[1701844260.098779201, {}], {"tag"=>"from.test-tag.raw.new.fluent.bit.out", "hostname"=>"monox", "output"=>"new data", "tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[0] from.test-tag.raw.new.fluent.bit.out: [[[2023/12/06 15:31:01] [ warn] [engine] service will shutdown in max 5 seconds
[2023/12/06 15:31:01] [ info] [input] pausing dummy.0
1701844261.099363327, {}], {"tag"=>"from.test-tag.raw.new.fluent.bit.out", "hostname"=>"monox", "output"=>"new data", "tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[2023/12/06 15:31:02] [ info] [engine] service has stopped (0 pending tasks)
[2023/12/06 15:31:02] [ info] [input] pausing dummy.0
[2023/12/06 15:31:02] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/12/06 15:31:02] [ info] [output:stdout:stdout.0] thread worker #0 stopped

PS> type ~\Documents\processors.yaml

pipeline:
    inputs:
      - name: dummy
        tag: test-tag.raw
        dummy: '{"tool": "fluent", "sub": {"s1": {"s2": "bit"}}}'

        processors:
          logs:
            - name: modify
              match: test-tag.*
              add: hostname monox
            - name: rewrite_tag
              match_regex: ^test-tag.*
              rule: $tool ^(fluent)$  from.$TAG.new.$tool.$sub['s1']['s2'].out false
              emitter_name: re_emitted
            - name: lua
              match: from.test-tag.raw.*
              call: append_tag
              code: |
                function append_tag(tag, timestamp, record)
                  new_record = record
                  new_record["tag"] = tag
                  return 1, timestamp, new_record
                end

    outputs:
      - name: stdout
        match: '*'

        processors:
          logs:
            - name: lua
              call: add_field
              code: |
                  function add_field(tag, timestamp, record)
                     new_record = record
                     new_record["output"] = "new data"
                     return 1, timestamp, new_record
                  end

ng-bsy · 2023-12-06T09:01:24Z

Really? I tested fluent-bit 2.2.0 win64 zip and I obtained the following result. I extracted the zip file into my home folder directly and using yaml configuration which contains processor stack:
...

Yep, sorry that one's on me. I reinstalled with the new package, while the fluent-bit executable was still running (service).
I installed /silent so there was no error that the exe could not be replaced.
Ensuring, the service was not running and reinstalling with https://packages.fluentbit.io/windows/fluent-bit-2.2.0-win64.exe solved it. Thanks a lot.

In this case the docs, as pointed out earlier, should definitely be updated

Will give it a try 👍 If it works, this doc should be updated: https://docs.fluentbit.io/manual/installation/windows#installation-packages It points to https://releases.fluentbit.io/2.2/

patrick-stephens · 2023-12-06T10:58:43Z

Yeah we should update the docs, we switched over a while back but left the old docs there to keep compatible.

We can do on next release potentially - I don't want to change checksum values in the docs without also changing version to make it obvious (and not stuck in a cache somewhere). I need to tweak the CI workflow slightly I think as well to inject the checksums from the Github builds (rather than AppVeyor ones) into the docs automatically too.

@ng-bsy just to confirm: the packages from releases.fluentbit.io did not work but the ones from packages.fluentbit.io did?

ng-bsy · 2023-12-06T13:24:39Z

@ng-bsy just to confirm: the packages from releases.fluentbit.io did not work but the ones from packages.fluentbit.io did?

@patrick-stephens Exactly. Though I only tested the -win64.exe

cosmo0920 · 2023-12-06T14:02:19Z

I also tested with -win32.exe and it works.

cosmo0920 · 2023-12-06T14:07:00Z

winarm64.exe is also working as well.
I confirmed it with Windows 11 on ARM64 (Parallels Desktop on Apple Silicon macOS).

PS C:\Users\cosmo\fluent-bit-2.2.0-winarm64> .\bin\fluent-bit.exe -c ..\processors.yaml
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/12/06 23:05:51] [error] [processor] error creating native processor instance lua
[2023/12/06 23:05:51] [error] cannot create 'lua' processor unit
[2023/12/06 23:05:51] [error] failed to load 'logs' processors
[2023/12/06 23:05:51] [error] [processor] error creating native processor instance lua
[2023/12/06 23:05:51] [error] cannot create 'lua' processor unit
[2023/12/06 23:05:51] [error] failed to load 'logs' processors
[2023/12/06 23:05:51] [ info] [fluent bit] version=2.2.0, commit=db8487d123, pid=5384
[2023/12/06 23:05:51] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/12/06 23:05:51] [ info] [cmetrics] version=0.6.4
[2023/12/06 23:05:51] [ info] [ctraces ] version=0.3.1
[2023/12/06 23:05:51] [ info] [input:dummy:dummy.0] initializing
[2023/12/06 23:05:51] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/12/06 23:05:51] [ warn] [filter:rewrite_tag:rewrite_tag.0] 'Match' may cause infinite loop.
[2023/12/06 23:05:51] [ info] [input:emitter:re_emitted] initializing
[2023/12/06 23:05:51] [ info] [input:emitter:re_emitted] storage_strategy='memory' (memory only)
[2023/12/06 23:05:51] [ info] [sp] stream processor started
[2023/12/06 23:05:51] [ info] [output:stdout:stdout.0] worker #0 started
[0] from.test-tag.raw.new.fluent.bit.out: [[1701871552.468503000, {}], {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}, "hostname"=>"monox"}]
[0] from.test-tag.raw.new.fluent.bit.out: [[1701871553.475230400, {}], {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}, "hostname"=>"monox"}]
[2023/12/06 23:05:55] [engine] caught signal (SIGINT)
[0] from.test-tag.raw.new.fluent.bit.out: [[1701871554.462450000, {}], {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}, "hostname"=>"monox"}]
[0] from.test-tag.raw.new.fluent.bit.out: [[[2023/12/06 23:05:55] [ warn] [engine] service will shutdown in max 5 seconds
1701871555.472658300, {}], {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}, "hostname"=>"monox"}]
[2023/12/06 23:05:55] [ info] [input] pausing dummy.0
[2023/12/06 23:05:56] [ info] [engine] service has stopped (0 pending tasks)
[2023/12/06 23:05:56] [ info] [input] pausing dummy.0
[2023/12/06 23:05:56] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/12/06 23:05:56] [ info] [output:stdout:stdout.0] thread worker #0 stopped

patrick-stephens · 2023-12-07T13:51:45Z

Thanks @cosmo0920

patrick-stephens · 2023-12-07T14:13:37Z

I've created fluent/fluent-bit-docs#1270 to update the docs but it needs a CI update here as well to pass in the right hashes on next release: #8260

patrick-stephens self-assigned this Mar 2, 2023

patrick-stephens added the packaging label Mar 2, 2023

patrick-stephens added this to the Fluent Bit v2.1.0 milestone Mar 2, 2023

lecaros modified the milestones: Fluent Bit v2.1.0, Fluent Bit v2.1.2 Apr 24, 2023

github-actions bot added the Stale label Aug 24, 2023

github-actions bot removed the Stale label Aug 25, 2023

patrick-stephens mentioned this issue Dec 7, 2023

windows: update to use new Github builds fluent/fluent-bit-docs#1270

Merged

patrick-stephens mentioned this issue Dec 7, 2023

workflows: update docs generation process for releases #8260

Merged

7 tasks

patrick-stephens closed this as completed in fluent/fluent-bit-docs#1270 Dec 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YAML configuration files for Windows #6937

YAML configuration files for Windows #6937

mp3monster commented Mar 1, 2023

patrick-stephens commented Mar 2, 2023

patrick-stephens commented May 3, 2023

patrick-stephens commented May 25, 2023

github-actions bot commented Aug 24, 2023

ameya-rescale commented Aug 24, 2023

patrick-stephens commented Aug 24, 2023

CircuitGuy commented Nov 14, 2023

mp3monster commented Nov 14, 2023

ng-bsy commented Dec 5, 2023 •

edited

Loading

patrick-stephens commented Dec 5, 2023

patrick-stephens commented Dec 5, 2023

ng-bsy commented Dec 5, 2023

patrick-stephens commented Dec 5, 2023

cosmo0920 commented Dec 5, 2023

patrick-stephens commented Dec 5, 2023

ng-bsy commented Dec 5, 2023

ng-bsy commented Dec 5, 2023

cosmo0920 commented Dec 6, 2023 •

edited

Loading

ng-bsy commented Dec 6, 2023 •

edited

Loading

patrick-stephens commented Dec 6, 2023

ng-bsy commented Dec 6, 2023

cosmo0920 commented Dec 6, 2023

cosmo0920 commented Dec 6, 2023 •

edited

Loading

patrick-stephens commented Dec 7, 2023

patrick-stephens commented Dec 7, 2023 •

edited

Loading

YAML configuration files for Windows #6937

YAML configuration files for Windows #6937

Comments

mp3monster commented Mar 1, 2023

patrick-stephens commented Mar 2, 2023

patrick-stephens commented May 3, 2023

patrick-stephens commented May 25, 2023

github-actions bot commented Aug 24, 2023

ameya-rescale commented Aug 24, 2023

patrick-stephens commented Aug 24, 2023

CircuitGuy commented Nov 14, 2023

mp3monster commented Nov 14, 2023

ng-bsy commented Dec 5, 2023 • edited Loading

patrick-stephens commented Dec 5, 2023

patrick-stephens commented Dec 5, 2023

ng-bsy commented Dec 5, 2023

patrick-stephens commented Dec 5, 2023

cosmo0920 commented Dec 5, 2023

patrick-stephens commented Dec 5, 2023

ng-bsy commented Dec 5, 2023

ng-bsy commented Dec 5, 2023

cosmo0920 commented Dec 6, 2023 • edited Loading

ng-bsy commented Dec 6, 2023 • edited Loading

patrick-stephens commented Dec 6, 2023

ng-bsy commented Dec 6, 2023

cosmo0920 commented Dec 6, 2023

cosmo0920 commented Dec 6, 2023 • edited Loading

patrick-stephens commented Dec 7, 2023

patrick-stephens commented Dec 7, 2023 • edited Loading

ng-bsy commented Dec 5, 2023 •

edited

Loading

cosmo0920 commented Dec 6, 2023 •

edited

Loading

ng-bsy commented Dec 6, 2023 •

edited

Loading

cosmo0920 commented Dec 6, 2023 •

edited

Loading

patrick-stephens commented Dec 7, 2023 •

edited

Loading