Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] ConcurrentModificationException in security package #3404

Closed
HoffmannTom opened this issue Sep 25, 2023 · 5 comments · Fixed by #3945
Closed

[BUG] ConcurrentModificationException in security package #3404

HoffmannTom opened this issue Sep 25, 2023 · 5 comments · Fixed by #3945
Labels
bug Something isn't working triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable.

Comments

@HoffmannTom
Copy link

HoffmannTom commented Sep 25, 2023 •
edited
Loading

Describe the bug
An update call via REST-API failed with an internal error.
Looking at the log-file, we see a stack trace with a ConcurrentModificationException of a HashMap.
The errors happen very rarely, less than 1x per month. The stacktrace should hopefully help out to narrow down the cause.

To Reproduce
Steps to reproduce the behavior:

  1. Happens rarely, not reliably reproducible

Expected behavior
Updating the index should work reliably

Plugins

  • opensearch-alerting
  • opensearch-anomaly-detection
  • opensearch-asynchronous-search
  • opensearch-cross-cluster-replication
  • opensearch-geospatial
  • opensearch-index-management
  • opensearch-job-scheduler
  • opensearch-knn
  • opensearch-ml
  • opensearch-neural-search
  • opensearch-notifications
  • opensearch-notifications-core
  • opensearch-observability
  • opensearch-performance-analyzer
  • opensearch-reports-scheduler
  • opensearch-security
  • opensearch-security-analytics
  • opensearch-sql

Screenshots
not available

Host/Environment (please complete the following information):

  • OS: Ubuntu 22 LTS
  • Version: 22 LTS
  • OpenSearch version 2.9.0

Additional context
Stacktrace:

[2023-09-22T18:12:41,188][ERROR][o.o.s.f.SecurityFilter   ] [srv-xx.yyy.com] Unexpected exception java.util.ConcurrentModificationException
java.util.ConcurrentModificationException: null
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?]
	at java.util.HashMap$KeyIterator.next(HashMap.java:1620) ~[?:?]
	at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1054) ~[?:?]
	at java.util.AbstractCollection.addAll(AbstractCollection.java:335) ~[?:?]
	at java.util.HashSet.<init>(HashSet.java:121) ~[?:?]
	at org.opensearch.security.securityconf.ConfigModelV7$RoleMappingHolder.map(ConfigModelV7.java:1268) ~[opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.security.securityconf.ConfigModelV7.mapSecurityRoles(ConfigModelV7.java:1332) ~[opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.security.privileges.PrivilegesEvaluator.mapRoles(PrivilegesEvaluator.java:558) ~[opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:246) ~[opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:377) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:57) [opensearch-performance-analyzer-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:552) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:127) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:134) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:320) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:411) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:249) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:68) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.9.0.jar:2.9.0]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
@HoffmannTom HoffmannTom added bug Something isn't working untriaged Require the attention of the repository maintainers and may need to be prioritized labels Sep 25, 2023
@HoffmannTom HoffmannTom mentioned this issue Sep 25, 2023
Closed
@HoffmannTom
Copy link
Author

The problem seems to be related to copying the HashMap in
https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/securityconf/ConfigModelV7.java
line:
final Set<String> abar = new HashSet<>(roleMapValue.getAnd_backend_roles());

@cwperks
Copy link
Member

cwperks commented Sep 25, 2023

Thank you @HoffmannTom . There was a change released in 2.9 that made the ConcurrentModificationException occur less frequently, but it was pointed out here that this call could (in rare circumstances) lead to a ConcurrentModificationException.

Thank you for filing this issue!

@HoffmannTom
Copy link
Author

@cwperks Thanks for the lightning-fast reply :)
I only checked the open issues and didn't stumble upon this PR. Thanks for pointing out!
Maybe you can use this issue to proceed with this topic. Seems that there were several thoughts already about it.

@davidlago davidlago added triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable. and removed untriaged Require the attention of the repository maintainers and may need to be prioritized labels Oct 2, 2023
@peternied peternied self-assigned this Jan 12, 2024
peternied added a commit to peternied/security that referenced this issue Jan 12, 2024
@peternied
Protect config object from concurrent modification issues
0c7fe39 
We've had rare reports of modification exceptions from customers around
the config objects.  Using a lock object to protect the internal
collection from modification on other threads and changing the behavoir
of the getter to pass a copy of the collection instead of direct
collection references.

- Resolves opensearch-project#3404

Signed-off-by: Peter Nied <[email protected]>
@peternied peternied mentioned this issue Jan 12, 2024
Merged
3 tasks
peternied added a commit to peternied/security that referenced this issue Jan 12, 2024
@peternied
Protect config object from concurrent modification issues
efdf2a0 
We've had rare reports of modification exceptions from customers around
the config objects.  Using a lock object to protect the internal
collection from modification on other threads and changing the behavoir
of the getter to pass a copy of the collection instead of direct
collection references.

- Resolves opensearch-project#3404

Signed-off-by: Peter Nied <[email protected]>
@javad87
Copy link

javad87 commented Jun 11, 2024

It seems this issue has not been resolved. I'm using OS v2.12.0 and still encountering the same exception. For more details, please refer to this post:

https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652

@peternied peternied removed their assignment Jun 11, 2024
@peternied
Copy link
Member

@javad87 Could you open up a new issue with the call stack that you encountered this issue on and any other details?

@javad87 javad87 mentioned this issue Jun 11, 2024
Open
@cwperks cwperks mentioned this issue Aug 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Protect config object from concurrent modification issues peternied/security
5 participants
@peternied @davidlago @cwperks @HoffmannTom @javad87