Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Attribute Ranges with Active Directory #98

Closed
someknowledge opened this issue Feb 2, 2021 · 30 comments
Closed

Support Attribute Ranges with Active Directory #98

someknowledge opened this issue Feb 2, 2021 · 30 comments
Assignees

Comments

@someknowledge
Copy link

Hi I am new to MyVD and just setup simple configuration to proxy our ADLDS directory. However, when I search on a group with large amount of entries (15000), I am getting pages each contains 1500 entries. Is there a way to configure MyVD to return all existing entries in the group by interrogating ADLDS?

P.S. ADLDS default setting is 1500 entries per page and we cannot increase it.

Thank you.

@mlbiam mlbiam self-assigned this Feb 3, 2021
@mlbiam mlbiam added the bug label Feb 3, 2021
@mlbiam
Copy link
Contributor

mlbiam commented Feb 3, 2021

just realizing it's not documented. Yes, you can add the following to your ldap insert to enable paging:

server.ns.ldap.config.usePaging=true
server.ns.ldap.config.pageSize=300

where ns is your namespace and ldap is your LDAPInterceptor insert. MyVD will do the paging for you on large queries

@someknowledge
Copy link
Author

This is perfect!!! Works as you mentioned. Thank you.

@mlbiam mlbiam reopened this Feb 3, 2021
@mlbiam
Copy link
Contributor

mlbiam commented Feb 3, 2021

re-opening so i remember to document in the next release

@someknowledge
Copy link
Author

someknowledge commented Feb 3, 2021

Actually, I just re-run the test and looks like MyVD still getting pages from Active Directory. I even tried two scenarios:

  1. Set server.BaseServer.LDAPBaseServer.config.usePaging=false
  2. Set server.BaseServer.LDAPBaseServer.config.usePaging=true and server.BaseServer.LDAPBaseServer.config.pageSize=3000
    Both scenarios returned Active Directory Page of 1500.
    image

Here is my Config:
server.listener.port=389

#No global chain
server.globalChain=

#Setup a single proxy
server.nameSpaces=Root,Schema,BaseServer

#Define RootDSE
server.Root.chain=RootDSE
server.Root.nameSpace=
server.Root.weight=0
server.Root.RootDSE.className=net.sourceforge.myvd.inserts.RootDSE
server.Root.RootDSE.config.namingContexts=DC=pxed,DC=pfizer,DC=com
server.Root.RootDSE.config.supportedControls=1.2.840.113556.1.4.319,1.2.840.113556.1.4.801,1.2.840.113556.1.4.473,1.2.840.113556.1.4.528,1.2.840.113556.1.4.417,1.2.840.113556.1.4.619,1.2.840.113556.1.4.841,1.2.840.113556.1.4.529,1.2.840.113556.1.4.805,1.2.840.113556.1.4.521,1.2.840.113556.1.4.970,1.2.840.113556.1.4.1338,1.2.840.113556.1.4.474,1.2.840.113556.1.4.1339,1.2.840.113556.1.4.1340,1.2.840.113556.1.4.1413,2.16.840.1.113730.3.4.9,2.16.840.1.113730.3.4.10,1.2.840.113556.1.4.1504,1.2.840.113556.1.4.1852,1.2.840.113556.1.4.802,1.2.840.113556.1.4.1907,1.2.840.113556.1.4.1948,1.2.840.113556.1.4.1974,1.2.840.113556.1.4.1341,1.2.840.113556.1.4.2026,1.2.840.113556.1.4.2064,1.2.840.113556.1.4.2065,1.2.840.113556.1.4.2066
server.Root.RootDSE.config.supportedLDAPVersions=3,2

server.Schema.chain=schema
server.Schema.nameSpace=cn=schema
server.Schema.weight=50
server.Schema.schema.className=net.sourceforge.myvd.inserts.SchemaInsert
server.Schema.schema.config.schemaLDIF=C:\myvd\conf\test.ldif

#Simple LDAP insert
server.BaseServer.chain=LDAPBaseServer
server.BaseServer.nameSpace=DC=pxed,DC=pfizer,DC=com
server.BaseServer.weight=100
server.BaseServer.LDAPBaseServer.className=net.sourceforge.myvd.inserts.ldap.LDAPInterceptor
server.BaseServer.LDAPBaseServer.config.host=adLDAP.pfizer.com
server.BaseServer.LDAPBaseServer.config.port=3890
server.Root.RootDSE.config.supportedLDAPVersions=3,2
server.BaseServer.LDAPBaseServer.config.remoteBase=DC=pxed,DC=pfizer,DC=com
server.BaseServer.LDAPBaseServer.config.usePaging=false

@mlbiam
Copy link
Contributor

mlbiam commented Feb 4, 2021

server.BaseServer.LDAPBaseServer.config.pageSize=3000

If you're setting the page size to 3000 but the max is 1500 you'll still get the paging error. If you're max is 1500 I'd set pageSize to 1400

@someknowledge
Copy link
Author

I am looking for MyVD to return all attributes without pages. So, if Active Directory contains 5 pages of 1500 members each, I am looking for a setting which returns all 7500 members in one time.

@mlbiam
Copy link
Contributor

mlbiam commented Feb 4, 2021

I am looking for MyVD to return all attributes without pages.

right, MyVD handles the paging logic for you with AD/ADLDS. The usePaging tells MyVD to include the paged search control in the search request to AD. pageSize tells MyVD how many entries to include in each page. This is all transparent to the application making LDAP calls to MyVD.

I am looking for a setting which returns all 7500 members in one time.

Exactly. Setting the pageSize to 1400 means that MyVD will request pages of 1400 entries each. But the client to MyVD doesn't need any knowledge of paging on its end.

@someknowledge
Copy link
Author

ADLDS can only handle 1500 entries at the time. So, when I set 7500, I am still getting 1500 based on the limit of ADLDS setting. MyVD should have some setting to interrogate pages if ADLDS returns less members in the single page compare to preset setting. I would be ok if I can use "server.BaseServer.LDAPBaseServer.config.usePaging=false" and MyVD would interrogate all the pages from ADLDS and returns every member without pages.

@mlbiam
Copy link
Contributor

mlbiam commented Feb 4, 2021

I think there's a disconnect on how the paged search feature in MyVD works. When your client does a search against MyVD with usePaging is true MyVD will send the search request to ADLDS with paged search control with the maximum number of entries per page from pageSize (ie 1400). As results come back from ADLDS MyVD looks for the returned page indicator that ADLDS sends to say "you've hit your limit, request the next page". MyVD then requests the next page returning results back to your client transparently. So with 7500 entries MyVD will generate all 6 page requests for your behind the scenes. This process is transparent to the ldap client talking to MyVD.

The pageSize isn't reflective of the maximum number of results you could get from a query (7500), it's based on the limit of the maximum number of entries per page allowed by your directory (1500). By injecting the paging process for you into the LDAP request it makes it so it doesn't matter how many results will eventually come back from the request.

@someknowledge
Copy link
Author

So, if ADLDS limit is 1500 members per page, MyVD will not able to show to client larger amount. Unless, I will write custom adapter to interrogate ADLDS pages and return single blob with all 7500 entries. Is this correct?

@mlbiam
Copy link
Contributor

mlbiam commented Feb 4, 2021

Is this correct?

No. ADLDS has a limit of 1500 entries per page. Setting the MyVD pageSize to 1400 means that MyVD will ask ADLDS for all 7500 entries of your search results in 1400 entry increments. Your client will see 7500 results without having to know anything about ldap pages.

@someknowledge
Copy link
Author

I understand, by my teams is looking for results without any pages. So, if group has 11110 users, it would look like this:
image

@mlbiam
Copy link
Contributor

mlbiam commented Feb 4, 2021

i think the image got cropped. You have to use pages with ADLDS and AD. That's why MyVD implements it this way so you can work with large search results that exceed the maximum search size for your directory without your client having to understand how to work with pages. What happens when you set the pageSize in MyVD to 1400?

@someknowledge
Copy link
Author

1400 setting works well. Also, the image was not cropped, I used MyVD to proxy Oracle Virtual Directory which interrogated pages from ADLDS and returned 11110 count without paging feature. This use case is necessary because some legacy apps relying on retrieval of all users and not capable to support pages.

@mlbiam
Copy link
Contributor

mlbiam commented Feb 4, 2021

You're going client --> MyVD --> OVD --> ADLDS? Why? MyVD will do that paging for you. Unless you have a custom plugin written for OVD there's nothing OVD can do that MyVD can't.

@someknowledge
Copy link
Author

Correct, I want to remove OVD and do client --> MyVD -->ADLDS but show to client all members without paging.

@mlbiam
Copy link
Contributor

mlbiam commented Feb 4, 2021

So you're saying that with usePaging set to true, and pageSize set to 1400 client --> MyVD --> ADLDS is not working as expected?

@someknowledge
Copy link
Author

someknowledge commented Feb 4, 2021

I am saying that usePaging set to false do not return all members but pages per ADLDS page limit, while I am looking for MyVD to return all 11110 members.

@mlbiam
Copy link
Contributor

mlbiam commented Feb 4, 2021

I am saying that usePaging set to false do not return all members but pages per ADLDS page limit, while I am looking for MyVD to return all 11110 members.

That's expected. to keep your clients from having to use paging for large result sets you set usePaging to true, and pageSize to 1400.

client (no paging) --> MyVD --> paging --> ADLDS

once you set usePaging to true, and pageSize to 1400 a search from your client to myvd will return all 11110 members without your client having to know anything about paging. MyVD does all of the paging for you.

usePaging is "Tell MyVD to use pageing when interacting with ADLDS so your client doesn't have to"

@someknowledge
Copy link
Author

Looks like 1400 did not work:
image
image

@mlbiam
Copy link
Contributor

mlbiam commented Feb 4, 2021

  1. What version of ADLDS are you on?
  2. What's your LDAP Search look like?

@someknowledge
Copy link
Author

We are on 2016 ADLDS and I am using ApacheDS client as quick search.

@mlbiam
Copy link
Contributor

mlbiam commented Feb 5, 2021

Can you add the DumpTransactions insert? Add it to the global chain. It would look like:

server.globalChain=LogAllTransactions, other-global-inserts
server.globalChain.LogAllTransactions.className=net.sourceforge.myvd.inserts.DumpTransaction
server.globalChain.LogAllTransactions.config.logLevel=info
server.globalChain.LogAllTransactions.config.label=Global

this will tell me the ldap search that's being performed. Also, can you confirm that the logs say usePaging='true'?

@someknowledge
Copy link
Author

[2021-02-04 20:25:53,887][main] INFO LDAPInterceptor - usePaging - 'true'
[2021-02-04 20:25:53,887][main] INFO LDAPInterceptor - pageSize - '1200'
[2021-02-04 20:29:25,295][pool-4-thread-1] INFO DumpTransaction - [Global] Begin Post Search Complete - Filter=(objectClass=);Base=DC=pxed,DC=pfizer,DC=com;Scope=0;Attributes=[LDAPAttribute: {type=''}]
[2021-02-04 20:29:25,295][pool-4-thread-1] INFO DumpTransaction - [Global] Post Search Complete Complete
[2021-02-04 20:29:34,084][pool-4-thread-1] INFO DumpTransaction - [Global] Begin Seach - Filter=(cn=GBL-Colleagues);Base=DC=pxed,DC=pfizer,DC=com;Scope=2;Attributes=objectClass
[2021-02-04 20:29:34,086][pool-4-thread-1] INFO DumpTransaction - [Global] Seach submitted
[2021-02-04 20:29:34,151][pool-4-thread-1] INFO DumpTransaction - [Global] Begin Post Search Entry - Filter=(cn=GBL-Colleagues);Base=DC=pxed,DC=pfizer,DC=com;Scope=2;Attributes=[LDAPAttribute: {type='objectClass'}]
dn : CN=GBL-Colleagues,OU=GBLGroups,OU=Applications,OU=Delegated,OU=Groups,DC=pxed,DC=pfizer,DC=com
objectClass : top
objectClass : groupOfNames
objectClass : pfizerGroup
myVdReturnEntry: true
[2021-02-04 20:29:34,151][pool-4-thread-1] INFO DumpTransaction - [Global] Post Seach Entry Complete
dn : CN=GBL-Colleagues,OU=GBLGroups,OU=Applications,OU=Delegated,OU=Groups,DC=pxed,DC=pfizer,DC=com
objectClass : top
objectClass : groupOfNames
objectClass : pfizerGroup
myVdReturnEntry: true
[2021-02-04 20:29:34,152][pool-4-thread-1] INFO DumpTransaction - [Global] Begin Post Search Complete - Filter=(cn=GBL-Colleagues);Base=DC=pxed,DC=pfizer,DC=com;Scope=2;Attributes=[LDAPAttribute: {type='objectClass'}]
[2021-02-04 20:29:34,152][pool-4-thread-1] INFO DumpTransaction - [Global] Post Search Complete Complete
[2021-02-04 20:29:37,100][pool-4-thread-1] INFO DumpTransaction - [Global] Begin Seach - Filter=(objectClass=);Base=CN=GBL-Colleagues,OU=GBLGroups,OU=Applications,OU=Delegated,OU=Groups,DC=pxed,DC=pfizer,DC=com;Scope=0;Attributes=
[2021-02-04 20:29:37,102][pool-4-thread-1] INFO DumpTransaction - [Global] Seach submitted
[2021-02-04 20:29:37,778][pool-4-thread-1] INFO DumpTransaction - [Global] Begin Post Search Entry - Filter=(objectClass=);Base=CN=GBL-Colleagues,OU=GBLGroups,OU=Applications,OU=Delegated,OU=Groups,DC=pxed,DC=pfizer,DC=com;Scope=0;Attributes=[LDAPAttribute: {type=''}]

@mlbiam
Copy link
Contributor

mlbiam commented Feb 5, 2021

Ok, now i see where the confusion is. I thought we were talking about entry limits. You're talking about attribute limits (apparently AD calls this Range Retrieval). MyVD doesn't support this right now but let me take a look at what it would take to support.

@mlbiam mlbiam removed the bug label Feb 5, 2021
@mlbiam mlbiam changed the title Proxying Active Directory without paging Support Attribute Ranges with Active Directory Feb 5, 2021
@mlbiam
Copy link
Contributor

mlbiam commented Feb 5, 2021

implementation note - https://ldapwiki.com/wiki/LDAP_SERVER_RANGE_OPTION_OID

@someknowledge
Copy link
Author

Thank you for helping out!!!
Do you have approximate timeframe of such enhancement?

@mlbiam
Copy link
Contributor

mlbiam commented Feb 5, 2021

We can get this into the 1.0.10 release. I'm hoping to get that released with openunison 1.0.21 at end of february. i'll see if we can get this implemented for testing late next week.

@mlbiam
Copy link
Contributor

mlbiam commented Feb 9, 2021

@someknowledge I have an initial version working.

  1. Download https://nexus.tremolo.io/repository/files/myvd-server-1.0.10.jar
  2. copy it into your myvd libs folder
  3. remove libs/myvd-server-1.0.9.jar
  4. restart myvd

After restart you should get all your members. In the logs you should see something like:

[2021-02-09 14:15:22,285][main] INFO  LDAPEntrySet - attribute : member;range=0-1499 is a range
[2021-02-09 14:15:22,293][main] INFO  LDAPEntrySet - 0-1499
[2021-02-09 14:15:22,293][main] INFO  LDAPEntrySet - total : 1500
[2021-02-09 14:15:22,441][main] INFO  LDAPEntrySet - 1500-2999
[2021-02-09 14:15:22,602][main] INFO  LDAPEntrySet - 3000-4499
[2021-02-09 14:15:22,756][main] INFO  LDAPEntrySet - 4500-5999
[2021-02-09 14:15:22,938][main] INFO  LDAPEntrySet - 6000-7499
[2021-02-09 14:15:23,134][main] INFO  LDAPEntrySet - 7500-8999

mlbiam added a commit that referenced this issue Feb 11, 2021
@someknowledge
Copy link
Author

someknowledge commented Feb 17, 2021 via email

mlbiam added a commit that referenced this issue Feb 26, 2021
@mlbiam mlbiam closed this as completed Mar 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants