[DOC] Getting exception while indexing document with user having role with document level security enabled

[swapnilsvaidya]

What is the bug?
A clear and concise description of the bug.
I have created a user which is mapped with role having document level security enabled.
If I try to index the document through Bulkrequest using this user I get following exception:

 [ElasticsearchException[Elasticsearch exception [type=security_exception, reason=_opendistro_security_dls_query does not match (SG 900D)]]]

In OpenSearch logs, following is the exception logged.
resolved: Resolved [aliases=[], allIndices=[test_index], types=[*], originalRequested=[test_index], remoteIndices=[]]
mode: ADAPTIVE
[2022-06-19T05:44:05,351][DEBUG][o.o.s.c.DlsQueryParser   ] [node-1] containsTermLookupQuery() returns false
queries: [{"term": { "pminstanceid.keyword": "instance1"}}]
[2022-06-19T05:44:05,352][DEBUG][o.o.s.c.DlsFlsValveImpl  ] [node-1] Doing lucene-level DLS because the query does not contain a TLQ
[2022-06-19T05:44:05,352][DEBUG][o.o.s.f.SecurityFilter   ] [node-1] Failed to apply filter. Task id: 224 (requests[1], index[test_index][0]). Action: indices:data/write/bulk[s]
org.opensearch.OpenSearchSecurityException: _opendistro_security_dls_query does not match (SG 900D)
	at org.opensearch.security.configuration.DlsFlsValveImpl.setDlsHeaders(DlsFlsValveImpl.java:390) ~[opensearch-security-2.0.1.0.jar:2.0.1.0]
	at org.opensearch.security.configuration.DlsFlsValveImpl.invoke(DlsFlsValveImpl.java:157) ~[opensearch-security-2.0.1.0.jar:2.0.1.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:320) [opensearch-security-2.0.1.0.jar:2.0.1.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:157) [opensearch-security-2.0.1.0.jar:2.0.1.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:202) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:174) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:102) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:632) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:774) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.action.bulk.TransportBulkAction$1$1.doRun(TransportBulkAction.java:321) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:798) [opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.0.1.jar:2.0.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]

After looking into the code, it seems DlsFlsValveImpl is trying to read the dls query related header from the request.
------Code snipper from DlsFlsValveImpl -----

if (threadContext.getHeader(ConfigConstants.OPENDISTRO_SECURITY_DLS_QUERY_HEADER) != null) {
                    Object deserializedDlsQueries = Base64Helper.deserializeObject(threadContext.getHeader(ConfigConstants.OPENDISTRO_SECURITY_DLS_QUERY_HEADER));
                    if (!dlsQueries.equals(deserializedDlsQueries)) {                        
                        throw new OpenSearchSecurityException(ConfigConstants.OPENDISTRO_SECURITY_DLS_QUERY_HEADER + " does not match (SG 900D)");
                    }
                }

---

However I think this header should not come into picture while indexing the document.
This seems a bug.

How can one reproduce the bug?
Steps to reproduce the behavior:
Create a Role with DLS enabled, associate this role with user. Index the document using bulk request and using this user.
Verify bulkresponse

What is the expected behavior?
While indexing DLS should be ignored

What is your host/environment?

• OS: [e.g. iOS] All OS
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
NA

Do you have any additional context?
NA

[swapnilsvaidya]

@peternied Hi Peter, Could you please confirm if you are looking into this issue?

[peternied]

Hi @swapnilsvaidya I am not currently looking into this issue, and it has been added to our work queue. If you would like to create a pull request the maintainers and I would gladly review it.

[rursprung]

i haven't seen the corresponding comment in the OpenSearch Security documentation, but e.g. Search Guard explicitly declares that roles with DLS enabled mustn't have write access: https://docs.search-guard.com/latest/document-level-security#dls-and-write-access

i presume the same applies for OpenSearch Security and needs to be documented (and maybe enforced by a check in a new major release?)

[cliu123]

@opensearch-project/transfer-request This looks a documentation issue. Could you transfer to documentation repo? Thanks!

[rursprung]

@cliu123: seems that your transfer request hasn't happened yet. would it make sense to instead raise a new, well described, issue in the documentation repo and then close this one here with a reference to the new one?

[davidlago]

In the right repo now.

[hdhalter]

Hi Chris, can you please follow up with the security team to see what needs to be documented here? Thanks.

[peternied]

@cwillum A little more context, when users have a DLS/FLS rule assigned to them, this is in affect saying "This user XYZ should never see documents with these value(s) or be able to see these fields". If that same user has permissions to add/modify documents in an index with DLS/FLS rules it directly contradicts the cluster security configuration.

[hdhalter]

Hi @peternied , can you please help make the update to the file, and is the appropriate file https://github.com/opensearch-project/documentation-website/blob/main/_security/access-control/document-level-security.md? Thanks!

[AntonEliatra]

@hdhalter I can look at this one

[AntonEliatra]

@hdhalter not able to reproduce this issue with the OS 2.13, seems this was updated in code, as user is able to index with role which has DLS and upon query only receive relevant results based on the DLS configured

[hdhalter]

Thanks, @AntonEliatra ! I am closing this issue.

[rursprung]

if this behaviour changed (@peternied: was that intentional?) it should nevertheless be documented what needs to be looked out for and what can/cannot be done now.

[AntonEliatra]

Same behaviour is seen in OS 1.3, using DLS and FLS, user able to index any information, but is only able to retrieve the details that are permitted via DLS/FLS.
I can add this to the documentation for additional clarity if necessary

[rursprung]

then i don't think that anything changed and that the rule still is that there must be no DLS if you're indexing data (but this has to be confirmed by the security plugin team).

@hdhalter: please re-open the issue

[AntonEliatra]

@rursprung but the user is able to index with DLS/FLS enabled, I don't follow

[hdhalter]

@peternied , @scrawfor99

[AntonEliatra]

By design, there should be nothing preventing the user from indexing anything, the DLS/FLS is only applied during search/retrieval of data

[stephen-crawford]

Based on the documentation, I think @AntonEliatra is correct. If someone wants to share the configuration they were able to reproduce the issue with then I can try to help them debug the issue. Otherwise, I think things are correct as is and suspect this was a configuration issue at the time of its filing.

[rursprung]

note that we never had an exception (it wasn't us raising the issue) but we were very clearly told by floragunn (CC @nibix) that we should never use a user with a DLS role to index data (and it's still in their Search Guard documentation and since SG & OS-security are closely related the reasoning & behaviour should be the same?).

[nibix]

@rursprung

but we were very clearly told by floragunn (CC @nibix) that we should never use a user with a DLS role to index data (and it's still in their Search Guard documentation and since SG & OS-security are closely related the reasoning & behaviour should be the same?).

So, IMHO, the situation is like this:

• There's indeed no code which would prevent you from creating a user with DLS restrictions on an index, which is at the same time able to write into that index. Neither in OpenSearch security, nor in Search Guard.
• If one creates such a user, that user will be able to write new documents which they won't be able to retrieve (due to the DLS rule). Additionally, if they know the IDs of existing documents, they also can overwrite documents they are not allowed to see (disclaimer: untested)
• This is at least a contradiction to common security intuitions
• To keep things simple, the Search Guard docs recommend not to give write privileges to users with DLS restrictions on the same index: https://docs.search-guard.com/latest/document-level-security#dls-and-write-access
• I don't want to say that there is no use-case at all for using the combination of DLS with write privileges. Yet, I have yet to see one. I would also guess that this will be a 0.01% niche case - thus, in order to keep the docs simple for the 99.99% of users, I do believe that this plain recommendation is sufficient.
• Additionally, as this is such a niche case, the test coverage for this combination is likely extremely low. Thus, it is likely to encounter bugs like the one above.
• BTW, just my 5ct to the originally mentioned exception: I think that's a race condition - thus it will be difficult to reproduce.