Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it possible to send data to aws cloudwatch loggroup? #26

Open
JackJiaJJ opened this issue May 16, 2023 · 2 comments
Open

Is it possible to send data to aws cloudwatch loggroup? #26

JackJiaJJ opened this issue May 16, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@JackJiaJJ
Copy link

I tried to send logs from kafka to aws cloudwatch log groups, I searched some document, it mentioned the configuration something like below, but can NOT work well

output {
cloudwatch_logs {
log_group_name => "my-log-group"
log_stream_name => "my-log-stream"
region => "us-east-1"
access_key_id => "my-access-key-id"
secret_access_key => "my-secret-access-key"
message => "%{message}"
timestamp => "%{timestamp}"
}
}

I got below error:

[2023-05-15T02:23:24,356][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"output", :name=>"cloudwatch_logs"}
[2023-05-15T02:23:24,370][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'cloudwatch_logs'. Are you sure this is correct? Trying to load the cloudwatch_logs output plugin resulted in this error: Unable to load the requested plugin named cloudwatch_logs of type output. The plugin is not installed."

@JackJiaJJ JackJiaJJ added the enhancement New feature or request label May 16, 2023
@JackJiaJJ
Copy link
Author

I tried I logstash 8.7.0 and 8.5.3, neither can work

@Anarhyst266
Copy link

Anarhyst266 commented Jun 8, 2023

@JackJiaJJ I can propose you to use my https://github.com/Anarhyst266/logstash-output-awslogs
It's not well-documented, but there is an example:

output {
  awslogs {
    region => "us-east-2"
    access_key_id => "PLACEHOLDER"
    secret_access_key => "PLACEHOLDER"
    log_group_name => "/test_log_group"
    log_stream_name => "test_stream"
  }
}

It was created for my own needs, so you can't specify timestamp or message fields, it will use @timestamp as timestamp and the whole event in JSON format as message

To install it you can simply use logstash-plugin install logstash-output-awslogs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants