Performance issues when working with large number of users/policies

[aramhovsepyan]

Hi DJ Enriques,

First of all, kudos on the Vault UI. Great work.
We are currently somewhat struggling with Vault UI crashing when there are 50000 users/policies in the vault. What's the complexity of creating a pagination feature on that one?

Thanks a lot.
Regards,
Aram.

PS: I've looked through the list of issues and couldn't find this of being a duplicate.

[djenriquez]

Thanks for reporting this @aramhovsepyan and appreciate the kudos. Pagination should definitely be everywhere. Its easy enough to add it in to each module, but I think it would benefit the project greater to design this properly to have it be its own default module thats implemented for every list.

We'll look into getting this in.

In the meanwhile, can you confirm that listing is not a problem for you using the Vault CLI? Our bottleneck on performance will be Vault's API, which at this point, I don't believe has the ability to partition results.

[aramhovsepyan]

Hey @djenriquez,
Thanks for the prompt response.
Vault API is indeed slow and does not support pagination (it takes 1min to process the full request), however it does not crash. Vault UI crashes.
Regards,
Aram.

[djenriquez]

@aramhovsepyan If you can, please test out the image djenriquez/vault-ui:550de88 which has the changes needed to fix this issue.

[aramhovsepyan]

Hi @djenriquez
Any chance you are secretly working for our firm? You are faster that some of my colleagues ;).
The user listing seems to be fixed with this change. Policies are still crashing (we have >50k policies).
Thanks.
Regards,
Aram.
PS: The reason we have so many policies/users is the lack of dynamic policies in Vault, however they are not planning in fixing this soon.

[djenriquez]

Don't get used to it @aramhovsepyan! I work on this project in waves, when time between life and my main job allows 😝. I'll think about the policies some more, the problem with policies is that vault-ui must store them all in memory in the browser because of the nature of the call. It may the a browser limit that's being hit.

What operating system are you using? I can give you a build for the desktop client that may handle things better.

Open for other suggestions also.

[aramhovsepyan]

Thanks @djenriquez . We very much appreciate it even if it's in waves :).
We are using OSX. I guess at least having the search feature separately for both users/privacies is already a good enough feature for us. Right now even to search VaultUI tries to load all users and crashes at this point.
Thanks for the help.

[djenriquez]

Sorry @aramhovsepyan, a little confused. When you say right now, do you mean with the change or without? Is the issue resolved for you with this change?

[aramhovsepyan]

My bad, sorry. With the changes the issues for the users are indeed resolved! Vault UI still crashes when I try to list ~50k policies (each user has a policy). If you have time for a similar fix there as well we would highly appreciate it.
Thanks in any case.
Regards,

[aramhovsepyan]

Hi @djenriquez,
Any news on this one? The fix for the users is fine. We would appreciate a similar fix for the policy listing and we are good to go.
Thanks.
Regards,

[djenriquez]

Hi @aramhovsepyan.

The refactor actually did include pagination at the policy level. By default, the UI will only show 7 policies. I think, however, it may be having issues loading all 50K+ policies into the UI's memory on API call, causing the UI to be unresponsive.

I'm not really quite sure how to handle this since we're limited by Vault's API. We could possible truncate the results, but that doesn't do you any good if what you need was cut off...

Hmm.

[aramhovsepyan]

Hi @djenriquez
Okay then we should be fine. Do you plan on releasing a binary version of this update any soon?
Thanks a lot.
Regards,
Aram.

[aramhovsepyan]

Hi @djenriquez
Okay, we've got a build. The user view is great. We have no slow downs. It works like a charm.
However what I meant by policy pagination is not within the user detail view, but in the view with the policies. Have a look at the screenshot here. This view just freezes the VaultUI.
Thanks a lot.
Regards,
Aram.

[djenriquez]

aha! it looks like i missed that one!!! Great catch @aramhovsepyan, i'll get that in soon.

[djenriquez]

@aramhovsepyan apologies for the major delay. The fix is included in #187 and deployed to image djenriquez/vault-ui:dev if you, or anyone else, want to give that a try first before we merge in. If no responses by EOD Monday, I'll merge in.

[aramhovsepyan]

Hi @djenriquez ,
I can confirm that it all seems to work perfectly. Thanks for this update.
Do you accept donations?
Regards,
Aram.

[djenriquez]

Awesome! Glad it is working. Merged in.

About donations, I suppose I could try it out! http://ko-fi.com/djenriquez

Many thanks, @aramhovsepyan. Feel free to close this issue if you believe it to be resolved.