dockerfiles: default multiarch containers #4691
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jan 31, 2022
Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Patrick Stephens <[email protected]>
* packaging: update Windows build to detain fluent-bit artefacts Signed-off-by: Patrick Stephens <[email protected]> * packaging: build td-agent-bit too Signed-off-by: Patrick Stephens <[email protected]> * workflows: skip unit tests for this type of update Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
The following patch introduces a new interface called 'config_format'. This new interface implements a new layer to read and compose a configuration 'context' for Fluent Bit. The idea started on discussion #4331 where initially looking for feedback to extend the format and support 'groups' (or sub section) many needs were expressed like other formats. In the actual implementation which comes from lib/monkey, is not flexible enough and adds complexity when trying to extend it. The new 'config_format' is a 'layer' that generate configuration contexts: +-------------+ +---------------+ +----------------------------+ | Config File | <-- | Config Format | --> | Fluent Bit Readers & Setup | +-------------+ +---------------+ +----------------------------+ The advantage of this is that 'config_format' supports different 'backends' to read data from: this implementation supports 'fluentbit' (current) format but also 'YAML' format. +---------------+ | Config Format | +------+-+------+ | | +------------------+ +-------------+ | fluentbit format | | YAML format | +------------------+ +-------------+ 'fluentbit' format is a backport format from lib/monkey with API changes, the YAML format is parsed by using libyaml when available. The concepts of configuration keeps being the same: - sections: define a block - properties: key/value pairs that belongs to a section The following examples are identical configurations in different formats: --- fluentbit format --- [SERVICE] flush 1 log_level info [INPUT] name tail path /var/log/containers/*.log parser docker [INPUT] name forward listen 0.0.0.0 [OUTPUT] name stdout match * --- eof --- Now the identical representation in YAML: --- YAML format --- service: flush: 1 log_level: info inputs: tail: path: "/var/log/containers/*.log" parser: docker forward: listen: 0.0.0.0 outputs: stdout: match: "*" --- eof --- == Notes == - Fluent Bit config reader is still using the old mechanism, the next patches will migrate to this interface. - 'Groups' are not implemented/supported yet. Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
- adjust all files to "2015-2022 The Fluent Bit Authors" Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
The 'customs' plugins are not part of a pipeline, actually they are
generic plugins that can do anything, so being inside 'pipeline' scope
is wrong.
This patch moves the 'customs' plugins definition to the top left level
of the YAML file, e.g:
-- example --
service:
flush: 1
customs:
calyptia:
api_key: rceyJUb2tlbklEIjoiYads2ZlM
pipeline:
inputs:
tail:
path: ./test.log
read_from_head: true
-- end --
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Patrick Stephens <[email protected]>
The following patch implements two enhancements to the YAML config
support:
1. Move 'customs' plugins definition to the top level instead of inside
the 'pipeline'. A 'custom' plugin is not part of a pipeline, actually
it can do anything at the engine level when starting but not runtime
changes.
2. YAML add support to include files through the new 'includes' definition.
The new config schema will looks like:
--- YAML example ---
env:
flush_interval: 1
my_api_key: abcdefghijk
includes:
- dummy_pipeline.yaml
- opensearch_pipeline.yaml
service:
flush_interval: ${flush_interval}
log_level: info
customs:
calyptia:
api_key: ${my_api_key}
pipeline:
inputs:
tail:
path: ./test.log
parser: json
read_from_head: true
filters:
record_modifier:
match: "*"
record: powered_by calyptia
outputs:
stdout:
match: "*"
--- end of example ---
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Bumps [azure/setup-helm](https://github.com/azure/setup-helm) from 1 to 2.0. - [Release notes](https://github.com/azure/setup-helm/releases) - [Commits](Azure/setup-helm@v1...v2.0) --- updated-dependencies: - dependency-name: azure/setup-helm dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Patrick Stephens <[email protected]>
* workflows: remove Azure action usage Signed-off-by: Patrick Stephens <[email protected]> * workflows: update default version Signed-off-by: Patrick Stephens <[email protected]> * workflows: fix missing info Signed-off-by: Patrick Stephens <[email protected]> * workflows: trigger PR image tests on workflow change as well Signed-off-by: Patrick Stephens <[email protected]> * workflows: test this branch - must revert before merge Signed-off-by: Patrick Stephens <[email protected]> * workflows: revert to master branch Signed-off-by: Patrick Stephens <[email protected]> * workflows: ensure Dockle is for info only Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Patrick Stephens <[email protected]>
* installation: add script to install in one liner Signed-off-by: Patrick Stephens <[email protected]> * installation: ignore unbound variables Signed-off-by: Patrick Stephens <[email protected]> * installation: handle centos Signed-off-by: Patrick Stephens <[email protected]> * installation: minor tweaks Signed-off-by: Patrick Stephens <[email protected]> * installation: fall back to td-agent-bit Signed-off-by: Patrick Stephens <[email protected]> * installation: properly escape in nested HEREDOC Signed-off-by: Patrick Stephens <[email protected]> * installation: switch to ID checks Signed-off-by: Patrick Stephens <[email protected]> * installation: switch to sh compliant checks Signed-off-by: Patrick Stephens <[email protected]> * installation: switch to sh compliant checks Signed-off-by: Patrick Stephens <[email protected]> * workflows: disable unit tests for PRs with scripts Signed-off-by: Patrick Stephens <[email protected]>
Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 1.0.2 to 1.0.3. - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Commits](ossf/scorecard-action@v1.0.2...v1.0.3) --- updated-dependencies: - dependency-name: ossf/scorecard-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Stephens <[email protected]>
in_stdin_collect tests !ctx->parser to decide whether a parser is associated with the context or not. The problem with that check is ctx->parser is not explictily initialized in in_stdin_init and the malloc allocation does not guarantee that the memory assigned to ctx, and ctx->parser is zero initialized. This then will lead to undefined behavior where sometimes the ctx->parser will not be 0 and a non existing parser used. Errors like #4544 will then pop up randomly. This fix was validated with valgrind and the example provided in #4544 Signed-off-by: Paulo Neves <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
checking 'mkdir' and 'delimiter' and 'template' Signed-off-by: Takahiro Yamashita <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Resolves #4617. It will also warn if the both `GOOGLE_APPLCIATION_CREDENTIALS` and `GOOGLE_SERVICE_CREDENTIALS` are being used. Signed-off-by: Ridwan Sharif <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
The current implementation of the code uses a linked-list to keep the state/context of each file. When the number of files is high in the scanning path this generate high performance degradation, the tests were performed with > 50k files. The current patch implements hash tables to keep references of each file context, avoiding the linear scanning of files and improving the performance more than 20x. This design scales times better when high number of files exists. Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]> Signed-off-by: Patrick Stephens <[email protected]>
* workflows: add GPG passphrase support Signed-off-by: Patrick Stephens <[email protected]> * workflows: refactor for PR forks Signed-off-by: Patrick Stephens <[email protected]> * workflows: refactor for PR forks Signed-off-by: Patrick Stephens <[email protected]> * workflows: added labels to README Signed-off-by: Patrick Stephens <[email protected]>
Signed-off-by: Patrick Stephens <[email protected]>
patrick-stephens
force-pushed
the
default_multiarch_containers
branch
from
4a70a77 to
4610d2f
Compare
|
Merging went a little wonky so restarting here: #4778 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.