Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump fluent-plugin-azurestorage-gen2 version #344

Closed
wants to merge 1 commit into from

Conversation

matt-carr
Copy link

New plugin version includes support for Azure/Entra Workload ID.

Not sure what sort of proof you want but here's the pertinent parts of a test run with a locally built image (potentially sensitive data ped:

2024-08-29 14:17:29 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-08-29 14:17:29 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluentd.conf"
2024-08-29 14:17:30 +0000 [info]: gem 'fluentd' version '1.17.1'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-azure-loganalytics' version '0.7.0'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-azurestorage-gen2' version '0.3.7'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.4'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.4.3'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.3'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.4'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.2.0'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.2.0'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-route' version '1.0.0'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2024-08-29 14:17:30 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2024-08-29 14:17:34 +0000 [warn]: many chunk keys specified, and it may cause too many chunks on your system.
2024-08-29 14:17:35 +0000 [info]: using configuration file: <ROOT>
[...]
  <label @AZURE>
    <match **>
      @type azurestorage_gen2
      azure_storage_account "<snip>"
      azure_container "<snip>"
      azure_object_key_format "%{path}-%{index}.%{file_extension}"
      azure_use_workload_id "true"
      azure_oauth_tenant_id "<snip>"
      azure_oauth_app_id <snip>
      azure_oauth_refresh_interval 3600
      path "<snip>"
      store_as "json"
      auto_create_container true
      enable_retry true
      format json
      <inject>
        time_key "time"
        tag_key "tag"
        time_type string
        time_format "%Y-%m-%dT%H:%M:%S,%L%z"
        localtime false
        utc true
      </inject>
      <buffer tag, time, <snip>>
        @type "file"
        flush_thread_count 2
        path "/tmp/azure-buffer"
        timekey 1m
        timekey_wait 10s
        timekey_use_utc true
        chunk_limit_size 64MB
        total_limit_size 4GB
        flush_at_shutdown true
        retry_max_times 10
        retry_max_interval 5
        overflow_action drop_oldest_chunk
      </buffer>
      <format>
        @type json
      </format>
    </match>
  </label>
 [...]
2024-08-29 14:17:35 +0000 [info]: starting fluentd-1.17.1 pid=7 ruby="3.3.4"
2024-08-29 14:17:36 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "--config", "/fluentd/etc/fluentd.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
[...]
2024-08-29 14:17:48 +0000 [info]: adding match in @AZURE pattern="**" type="azurestorage_gen2"
[...]
2024-08-29 14:17:52 +0000 [info]: #0 starting fluentd worker pid=20 ppid=7 worker=0
2024-08-29 14:17:53 +0000 [info]: #0 azurestorage_gen2: Start getting access token every 3600 seconds.
2024-08-29 14:17:54 +0000 [info]: #0 azurestorage_gen2: Container '<snip>' exists.
2024-08-29 14:17:54 +0000 [info]: #0 [input] listening port port=24224 bind="0.0.0.0"
2024-08-29 14:17:54 +0000 [info]: #0 fluentd worker is now running worker=0
[...]
2024-08-29 14:17:53.595141281 +0000 fluent.info: {"message":"azurestorage_gen2: Start getting access token every 3600 seconds."}
2024-08-29 14:17:54.110813536 +0000 fluent.info: {"message":"azurestorage_gen2: Container '<snip>' exists."}
2024-08-29 14:17:54.153382887 +0000 fluent.info: {"port":24224,"bind":"0.0.0.0","message":"[input] listening port port=24224 bind=\"0.0.0.0\""}
2024-08-29 14:17:54.259700062 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}

@stevehipwell
Copy link
Owner

Thanks for the heads up @matt-carr. Dependabot runs weekly on this repo but I've triggered it to run now as it should pick this up automatically.

@stevehipwell
Copy link
Owner

I've noticed that the version doesn't look correct (oleewere/fluent-plugin-azurestorage-gen2#24)?

@matt-carr
Copy link
Author

Thanks for the heads up @matt-carr. Dependabot runs weekly on this repo but I've triggered it to run now as it should pick this up automatically.

wicked, thanks - closing in favour of #345

@matt-carr matt-carr closed this Aug 29, 2024
@matt-carr
Copy link
Author

@stevehipwell I'd be much obliged if you'd trigger another dependabot run, didn't manage to get the upstream PR merged before your scheduled one ;P after this I ought to be out of your hair

@stevehipwell
Copy link
Owner

@matt-carr does #349 capture the changes you wanted?

@matt-carr
Copy link
Author

Sure does, much appreciated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants