Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configure AWS SDK logging configuration #20313

Merged
merged 1 commit into from
Sep 3, 2016
Merged

Configure AWS SDK logging configuration #20313

merged 1 commit into from
Sep 3, 2016

Conversation

jasontedor
Copy link
Member

Because of security permissions that we do not grant to the AWS SDK (for
use in discovery-ec2 and repository-s3 plugins), certain calls in the
AWS SDK will lead to security exceptions that are logged at the warning
level. These warnings are noise and we should suppress them. This commit
adds plugin log configurations for discovery-ec2 and repository-s3 to
ship with default Log4j 2 configurations that suppress these log
warnings.

Closes #20294

@jasontedor
Configure AWS SDK logging configuration
cff9ab7 
Because of security permissions that we do not grant to the AWS SDK (for
use in discovery-ec2 and repository-s3 plugins), certain calls in the
AWS SDK will lead to security exceptions that are logged at the warning
level. These warnings are noise and we should suppress them. This commit
adds plugin log configurations for discovery-ec2 and repository-s3 to
ship with default Log4j 2 configurations that suppress these log
warnings.
@jasontedor jasontedor mentioned this pull request Sep 2, 2016
Closed
@dadoonet
Copy link
Member

dadoonet commented Sep 3, 2016

LGTM. Thanks!

@jasontedor jasontedor merged commit 981e4f5 into elastic:master Sep 3, 2016
@jasontedor jasontedor deleted the aws-sdk-log-configuration branch September 3, 2016 10:41
@jasontedor
Copy link
Member Author

Thanks @dadoonet.

@ppf2
Copy link
Member

ppf2 commented Oct 14, 2016

btw, did this make it to 5.0 RC1? I am still seeing it:

[2016-10-13T18:05:48,456][WARN ][c.a.m.AwsSdkMetrics      ] 
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_65]
    at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_65]
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_65]
    at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413) ~[?:1.8.0_65]
    at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361) ~[?:1.8.0_65]
    at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111) ~[?:?]
    at com.amazonaws.jmx.MBeans.isRegistered(MBeans.java:98) ~[?:?]
    at com.amazonaws.jmx.SdkMBeanRegistrySupport.isMBeanRegistered(SdkMBeanRegistrySupport.java:46) ~[?:?]
    at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:383) ~[?:?]
    at com.amazonaws.metrics.AwsSdkMetrics.<clinit>(AwsSdkMetrics.java:338) ~[?:?]
    at com.amazonaws.services.s3.AmazonS3Client.<clinit>(AmazonS3Client.java:196) ~[?:?]
    at org.elasticsearch.cloud.aws.InternalAwsS3Service.client(InternalAwsS3Service.java:72) ~[?:?]
    at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:304) ~[?:?]
    at org.elasticsearch.plugin.repository.s3.S3RepositoryPlugin.lambda$getRepositories$0(S3RepositoryPlugin.java:73) ~[?:?]
    at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:383) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
    at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:303) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
    at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:708) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
    at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:894) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:444) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:237) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:200) [elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

name    component     version
p9eEwZW repository-s3 5.0.0-rc1
p9eEwZW x-pack        5.0.0-rc1

{
  "name" : "p9eEwZW",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_9CAliR-QlGPpTcBHjIWtw",
  "version" : {
    "number" : "5.0.0-rc1",
    "build_hash" : "13e62e1",
    "build_date" : "2016-10-07T16:52:58.798Z",
    "build_snapshot" : false,
    "lucene_version" : "6.2.0"
  },
  "tagline" : "You Know, for Search"
}

@jasontedor
Copy link
Member Author

@ppf2 Yes; can you verify that there is a log4j2.properities in config/repository-s3?

@ppf2
Copy link
Member

ppf2 commented Oct 14, 2016

Yah it's there:

repository-s3.zip

jasontedor added a commit that referenced this pull request Oct 15, 2016
@jasontedor
Fix logging configuration for AwsSdkMetrics logger
f23ae90 
This commit fixes an issue with the configuration for the AwsSdkMetrics
logger; the issue is that the logging configuration had used underscores
instead of periods for the settings key (the perils of lenient settings
parsing).

Relates #20313
jasontedor added a commit that referenced this pull request Oct 15, 2016
@jasontedor
Fix logging configuration for AwsSdkMetrics logger
ad65b9f 
This commit fixes an issue with the configuration for the AwsSdkMetrics
logger; the issue is that the logging configuration had used underscores
instead of periods for the settings key (the perils of lenient settings
parsing).

Relates #20313
jasontedor added a commit that referenced this pull request Oct 15, 2016
@jasontedor
Fix logging configuration for AwsSdkMetrics logger
b2feefa 
This commit fixes an issue with the configuration for the AwsSdkMetrics
logger; the issue is that the logging configuration had used underscores
instead of periods for the settings key (the perils of lenient settings
parsing).

Relates #20313
@jasontedor
Copy link
Member Author

jasontedor commented Oct 15, 2016 •
edited
Loading

@ppf2 I pushed ad65b9f which will fix the issue.

@ppf2
Copy link
Member

ppf2 commented Oct 16, 2016

Thanks @jasontedor !

@clintongormley clintongormley added :Distributed Coordination/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Plugin Discovery EC2 labels Feb 13, 2018
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this pull request May 7, 2020
@DaveCTurner
Hide c.a.a.p.i.BasicProfileConfigFileLoader noise
c47783d 
A recent AWS SDK upgrade has introduced a new source of spurious `WARN` logs
when the security manager prevents access to the user's home directory and
therefore to their shared client configuration. This is actually the behaviour
we want, and it's harmless and handled by the SDK as if the profile config
doesn't exist, so this log message is unnecessary noise. This commit suppresses
this noisy logging by default.

Relates elastic#20313
Closes elastic#56333
@DaveCTurner DaveCTurner mentioned this pull request May 7, 2020
Merged
DaveCTurner added a commit that referenced this pull request May 7, 2020
@DaveCTurner
Hide c.a.a.p.i.BasicProfileConfigFileLoader noise (#56346)
806e2d2 
A recent AWS SDK upgrade has introduced a new source of spurious `WARN` logs
when the security manager prevents access to the user's home directory and
therefore to their shared client configuration. This is actually the behaviour
we want, and it's harmless and handled by the SDK as if the profile config
doesn't exist, so this log message is unnecessary noise. This commit suppresses
this noisy logging by default.

Relates #20313
Closes #56333
DaveCTurner added a commit that referenced this pull request May 7, 2020
@DaveCTurner
Hide c.a.a.p.i.BasicProfileConfigFileLoader noise (#56346)
8f4af29 
A recent AWS SDK upgrade has introduced a new source of spurious `WARN` logs
when the security manager prevents access to the user's home directory and
therefore to their shared client configuration. This is actually the behaviour
we want, and it's harmless and handled by the SDK as if the profile config
doesn't exist, so this log message is unnecessary noise. This commit suppresses
this noisy logging by default.

Relates #20313
Closes #56333
DaveCTurner added a commit that referenced this pull request May 7, 2020
@DaveCTurner
Hide c.a.a.p.i.BasicProfileConfigFileLoader noise (#56346)
61abf9d 
A recent AWS SDK upgrade has introduced a new source of spurious `WARN` logs
when the security manager prevents access to the user's home directory and
therefore to their shared client configuration. This is actually the behaviour
we want, and it's harmless and handled by the SDK as if the profile config
doesn't exist, so this log message is unnecessary noise. This commit suppresses
this noisy logging by default.

Relates #20313
Closes #56333
DaveCTurner added a commit that referenced this pull request May 7, 2020
@DaveCTurner
Hide c.a.a.p.i.BasicProfileConfigFileLoader noise (#56346)
9c1a112 
A recent AWS SDK upgrade has introduced a new source of spurious `WARN` logs
when the security manager prevents access to the user's home directory and
therefore to their shared client configuration. This is actually the behaviour
we want, and it's harmless and handled by the SDK as if the profile config
doesn't exist, so this log message is unnecessary noise. This commit suppresses
this noisy logging by default.

Relates #20313
Closes #56333
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this pull request Sep 17, 2020
@DaveCTurner
Hide c.a.s.s.i.UseArnRegionResolver noise
e3d6b28 
A recent AWS SDK upgrade has introduced a new source of spurious `WARN`
logs when the security manager prevents access to the user's home
directory and therefore to `$HOME/.aws/config`. This is the behaviour we
want, and it's harmless and handled by the SDK as if the config doesn't
exist, so this log message is unnecessary noise.  This commit suppresses
this noisy logging by default.

Relates elastic#20313, elastic#56346, elastic#53962
Closes elastic#62493
@DaveCTurner DaveCTurner mentioned this pull request Sep 17, 2020
Merged
DaveCTurner added a commit that referenced this pull request Sep 18, 2020
@DaveCTurner
Hide c.a.s.s.i.UseArnRegionResolver noise (#62522)
2ec7260 
A recent AWS SDK upgrade has introduced a new source of spurious `WARN`
logs when the security manager prevents access to the user's home
directory and therefore to `$HOME/.aws/config`. This is the behaviour we
want, and it's harmless and handled by the SDK as if the config doesn't
exist, so this log message is unnecessary noise.  This commit suppresses
this noisy logging by default.

Relates #20313, #56346, #53962
Closes #62493
DaveCTurner added a commit that referenced this pull request Sep 18, 2020
@DaveCTurner
Hide c.a.s.s.i.UseArnRegionResolver noise (#62522)
0a3f2c4 
A recent AWS SDK upgrade has introduced a new source of spurious `WARN`
logs when the security manager prevents access to the user's home
directory and therefore to `$HOME/.aws/config`. This is the behaviour we
want, and it's harmless and handled by the SDK as if the config doesn't
exist, so this log message is unnecessary noise.  This commit suppresses
this noisy logging by default.

Relates #20313, #56346, #53962
Closes #62493
@tlrx tlrx mentioned this pull request Jun 28, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
:Core/Infra/Logging Log management and logging utilities :Distributed Coordination/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement v5.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jasontedor @dadoonet @ppf2 @clintongormley