Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Doc]Add example for exclude pattern #204

Merged
merged 5 commits into from
Apr 16, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
## 3.5.1
- [DOC]Added example for `exclude_pattern` and reordered option descriptions [#204](https://github.com/logstash-plugins/logstash-input-s3/issues/204)

## 3.5.0
- Added support for including objects restored from Glacier or Glacier Deep [#199](https://github.com/logstash-plugins/logstash-input-s3/issues/199)
- Added `gzip_pattern` option, enabling more flexible determination of whether a file is gzipped [#165](https://github.com/logstash-plugins/logstash-input-s3/issues/165)
Expand Down
60 changes: 37 additions & 23 deletions docs/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,29 @@ This plugin uses the AWS SDK and supports several ways to get credentials, which
4. Environment variables `AMAZON_ACCESS_KEY_ID` and `AMAZON_SECRET_ACCESS_KEY`
5. IAM Instance Profile (available when running inside EC2)


[id="plugins-{type}s-{plugin}-additional_settings"]
===== `additional_settings`

* Value type is <<hash,hash>>
* Default value is `{}`

Key-value pairs of settings and corresponding values used to parametrize
the connection to s3. See full list in https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html[the AWS SDK documentation]. Example:

[source,ruby]
input {
s3 {
"access_key_id" => "1234"
"secret_access_key" => "secret"
"bucket" => "logstash-test"
"additional_settings" => {
"force_path_style" => true
"follow_redirects" => false
}
}
}

[id="plugins-{type}s-{plugin}-aws_credentials_file"]
===== `aws_credentials_file`

Expand Down Expand Up @@ -157,7 +180,20 @@ guaranteed to work correctly with the AWS SDK.
* Value type is <<string,string>>
* Default value is `nil`

Ruby style regexp of keys to exclude from the bucket
Ruby style regexp of keys to exclude from the bucket.
karenzone marked this conversation as resolved.
Show resolved Hide resolved

Note that files matching the pattern are skipped _after_ they have been listed.
Consider using <<plugins-{type}s-{plugin}-prefix>> instead where possible.

Example:

[source,ruby]
-----
"exclude_pattern" => "\/2020\/04\/"
-----

This pattern excludes all logs containing "/2020/04/" in the path.


[id="plugins-{type}s-{plugin}-gzip_pattern"]
===== `gzip_pattern`
Expand All @@ -167,28 +203,6 @@ Ruby style regexp of keys to exclude from the bucket

Regular expression used to determine whether an input file is in gzip format.

[id="plugins-{type}s-{plugin}-additional_settings"]
===== `additional_settings`

* Value type is <<hash,hash>>
* Default value is `{}`

Key-value pairs of settings and corresponding values used to parametrize
the connection to s3. See full list in https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html[the AWS SDK documentation]. Example:

[source,ruby]
input {
s3 {
"access_key_id" => "1234"
"secret_access_key" => "secret"
"bucket" => "logstash-test"
"additional_settings" => {
"force_path_style" => true
"follow_redirects" => false
}
}
}

[id="plugins-{type}s-{plugin}-include_object_properties"]
===== `include_object_properties`

Expand Down
2 changes: 1 addition & 1 deletion logstash-input-s3.gemspec
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Gem::Specification.new do |s|

s.name = 'logstash-input-s3'
s.version = '3.5.0'
s.version = '3.5.1'
s.licenses = ['Apache-2.0']
s.summary = "Streams events from files in a S3 bucket"
s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
Expand Down