Getting security exception due to access denied "java.lang.RuntimePermission" "accessDeclaredMembers" when trying to get snapshots

[tomchlee]

Hi,

After upgrading our opensearch cluster from v1.2.4 to v2.2.0 and configuring to use IRSA via repository-s3 plugin for s3 access, we're getting security exception due to access denied "java.lang.RuntimePermission" "accessDeclaredMembers" when trying to get snapshots:

curl -s 'http://localhost:9200/_snapshot/s3_repository/_all?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")"
      }
    ],
    "type" : "security_exception",
    "reason" : "access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")"
  },
  "status" : 500
}

and stacktrace in opensearch log:

[2022-08-19T18:24:22,899][WARN ][r.suppressed             ] [coordinating-node] path: /_snapshot/s3_repository/_all, params: {repository=s3_repository, snapshot=_all}
org.opensearch.transport.RemoteTransportException: [cluster-manager-node][192.168.1.20:9300][cluster:admin/snapshot/get]
Caused by: java.lang.SecurityException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
        at java.security.AccessControlContext.checkPermission(Unknown Source) ~[?:?]
        at java.security.AccessController.checkPermission(Unknown Source) ~[?:?]
        at java.lang.SecurityManager.checkPermission(Unknown Source) ~[?:?]
        at java.lang.Class.checkMemberAccess(Unknown Source) ~[?:?]
        at java.lang.Class.getDeclaredConstructors(Unknown Source) ~[?:?]
        at com.fasterxml.jackson.databind.util.ClassUtil.getConstructors(ClassUtil.java:1280) ~[?:?]
        at com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector._findPotentialConstructors(AnnotatedCreatorCollector.java:115) ~[?:?]
        at com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collect(AnnotatedCreatorCollector.java:70) ~[?:?]
        at com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collectCreators(AnnotatedCreatorCollector.java:61) ~[?:?]
        at com.fasterxml.jackson.databind.introspect.AnnotatedClass._creators(AnnotatedClass.java:403) ~[?:?]
        at com.fasterxml.jackson.databind.introspect.AnnotatedClass.getFactoryMethods(AnnotatedClass.java:315) ~[?:?]
        at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.getFactoryMethods(BasicBeanDescription.java:572) ~[?:?]
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitFactoryCreators(BasicDeserializerFactory.java:646) ~[?:?]
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:279) ~[?:?]
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:223) ~[?:?]
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:1407) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:403) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:350) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) ~[?:?]
        at com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:632) ~[?:?]
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) ~[?:?]
        at com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:632) ~[?:?]
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) ~[?:?]
        at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:609) ~[?:?]
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:188) ~[?:?]
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:28) ~[?:?]
        at com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:825) ~[?:?]
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:550) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[?:?]
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) ~[?:?]
        at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:642) ~[?:?]
        at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4805) ~[?:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4675) ~[?:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3666) ~[?:?]
        at com.amazonaws.partitions.PartitionsLoader.loadPartitionFromStream(PartitionsLoader.java:92) ~[?:?]
        at com.amazonaws.partitions.PartitionsLoader.build(PartitionsLoader.java:84) ~[?:?]
        at com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30) ~[?:?]
        at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64) ~[?:?]
        at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:52) ~[?:?]
        at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:106) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.getRegionObject(AwsClientBuilder.java:256) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:460) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424) ~[?:?]
        at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46) ~[?:?]
        at com.amazonaws.auth.STSAssumeRoleWithWebIdentitySessionCredentialsProvider.buildStsClient(STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java:125) ~[?:?]
        at com.amazonaws.auth.STSAssumeRoleWithWebIdentitySessionCredentialsProvider.<init>(STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java:97) ~[?:?]
        at com.amazonaws.auth.STSAssumeRoleWithWebIdentitySessionCredentialsProvider.<init>(STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java:40) ~[?:?]
        at com.amazonaws.auth.STSAssumeRoleWithWebIdentitySessionCredentialsProvider$Builder.build(STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java:226) ~[?:?]
        at org.opensearch.repositories.s3.S3Service.buildCredentials(S3Service.java:321) ~[?:?]
        at org.opensearch.repositories.s3.S3Service.buildClient(S3Service.java:182) ~[?:?]
        at org.opensearch.repositories.s3.S3Service.client(S3Service.java:136) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobStore.clientReference(S3BlobStore.java:142) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:281) ~[?:?]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2306) ~[opensearch-2.2.0.jar:2.2.0]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2288) ~[opensearch-2.2.0.jar:2.2.0]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1668) ~[opensearch-2.2.0.jar:2.2.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.2.0.jar:2.2.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.2.0.jar:2.2.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.2.0.jar:2.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

We've followed the steps for Amazon S3 Step 6 in https://opensearch.org/docs/latest/opensearch/snapshots/snapshot-restore/.

Please advise. Thanks!

[dblock]

Do you have a short list of steps to reproduce this? I found aws/aws-sdk-java#788 that seems similar, we should narrow this issue down to a jackson-databind update or something like that.

I see https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/resources/org/opensearch/bootstrap/security.policy#L64 that explicitly grants that permission to lucene core.

So I would start by adding that to the "everything else" part of security.policy to see if it fixes the problem:

//// Everything else:

grant {
    permission java.lang.RuntimePermission "accessDeclaredMembers";
}

If this works, we may need to add it to a more narrow scope, and debug why and how this was introduced, why we didn't catch it earlier, etc.

[tomchlee]

We deploy opensearch cluster via docker image (with repository-s3 plugin installed) in a kubernetes cluster in aws and set the settings as specified for Amazon S3 Step 6 in https://opensearch.org/docs/latest/opensearch/snapshots/snapshot-restore/:

1. Set the location for s3.client.default.identity_token_file setting in opensearch.yml.
2. Set AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_SESSION_NAME via container environment variables.
3. Deploy the opensearch container.
4. Register a S3 bucket as snapshot repository.
5. Run the curl command to get snapshots in the repository.

As for the "everything else" part of security.policy, do you mean to update "org/opensearch/bootstrap/security.policy" in opensearch jar under lib directory?

[dblock]

As for the "everything else" part of security.policy, do you mean to update "org/opensearch/bootstrap/security.policy" in opensearch jar under lib directory?

Yes. Could you please give it a try? Restart the node, see if it changes anything?

[aabukhalil]

while trying to reproducing this, this message came during building custom docker file to install the s3 security plugin

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

[tomchlee]

@dblock I replaced opensearch jar with an updated "org/opensearch/bootstrap/security.policy" with the changes you indicated, restarted the node and get snapshot list request successfully returned the info without any permission exception.

Interestingly, repository-s3 plugin's plugin-security.policy (https://github.com/opensearch-project/OpenSearch/blob/main/plugins/repository-s3/src/main/plugin-metadata/plugin-security.policy#L36) already grants the same permission but probably limited to the plugin and its dependencies.

[reta]

@dblock @tomchlee sorry for jumping late, I somehow missed this ticket, I think I know what is the problem, will send the fix shortly.

[dblock]

Thanks for the fix in #4469 @reta !

[tomchlee]

Thanks for the fix! What is the ETA for the release with the fix?

[dblock]

@tomchlee See https://github.com/orgs/opensearch-project/projects/1 for the release roadmap. I guess it will make it into 2.4.0 (November)?

[tomchlee]

@dblock got it, thanks!