[Story] Implement cross namespace discovery

[misastovicek]

Currently web UI shows only endpoints from the Kubernetes namespace where the Cryostat is deployed.
It would be great to have an option to discover services across many namespaces.

[andrewazores]

Thanks for the report. This is certainly on our radar for the future. It is not technically very difficult to enable cross-namespace discovery and I believe communicating with pods across namespaces should be seamless as well. The major hurdle to overcome here is the security concerns around the possibilities of multi-tenant clusters, where not every namespace has the same set of authorized users. We need to be able to account for this possibility and lock down cross-namespace accesses by default, requiring the deploying end user to add some configurations (preferably leveraging the existing Kubernetes/OpenShift RBAC when available) to explicitly allow certain users to access JFR data from pods in specific namespaces. Otherwise, an authorized user in one namespace may be able to pull potentially sensitive JFR data from pods in another unrelated namespace belonging to other users or organizations.

[postol]

Hi, is there ETA for this? It would be very useful to have support for cross namespace discovery ...

[andrewazores]

Hi @postol , there is currently no ETA on delivering this feature. Cryostat 2.1 will be released in the coming weeks and this feature is something we will be looking into deeper in the next iteration, but I cannot say when it might land.

[postol]

Hi, sorry for asking again but is this something that will be implemented soon? I see that 2.1 was already released so would like to know if this is something that will be covered with next release. Thank you.

[andrewazores]

@postol no worries, I understand that this is a highly desirable feature. The team has had discussions on this topic recently and we think we have a good handle on how we want to implement this. Release planning for 2.2 (due in ~November this year) did not include this feature request, but if we hit our other targets early then this would be a high priority to take up next. Otherwise, I think we will prioritize it for 2.3 (~April next year).

[petr-postulka]

Hi @andrewazores just wanted to check how it goes with 2.2 release and whether there is still chance this feature will get into the release or not. Btw do you now have exact date for 2.2 release?

[andrewazores]

Hi @petr-postulka , 2.2 is targeted for November 15th but unfortunately cross-namespace support is not implemented yet. It's still a priority item for 2.3 and I don't think the remaining technical hurdles are too large so I would expect we'll have it done by then.

[petr-postulka]

Thank you @andrewazores for the swift response

[andrewazores]

Well, actually, let me revise that statement. Cryostat doesn't yet query the k8s API server across namespaces, and data like archived recordings would be readable by any user with access to the Cryostat instance (not only users who have access to the namespace the data came from). So it wouldn't be safe to use for multi-tenant reasons for cross-namespace.

If you know that all of your namespaces are trusted and all of your users have permissions across all namespaces then you can actually make this work. Cryostat still would not know how to discover target JVMs outside of its own namespace, but with 2.2's Discovery Plugin API you could implement a Discovery Plugin that you deploy and manage on your own within each of your namespaces, and that would publish information to Cryostat about what targets the Plugin can see. As long as the URLs you publish are reachable then Cryostat can perform all of its usual actions against those targets.

So at this point I hesitate to say we "support" cross-namespace discovery, because in my eyes we don't support that until we support it for a multi-tenant system where user permissions and authorization are covered including for stored data (archived recordings). But if you're willing and able to do a little bit of legwork on your own and understand the security caveats, you could make it work.

Furthermore... another high priority item for 2.3, which almost made it into 2.2, is the cryostat-agent. This already contains a mostly-complete implementation of a Discovery Plugin as a JVM TI agent that you can attach to your target JVM application, set some environment variables, and have your target tell Cryostat how to find and reach it, bypassing Cryostat's k8s API discovery mechanism.

[petr-postulka]

Interesting - the cryostat agent sounds really great. No chance to still include into 2.2? :). If not any chance to release as minor release without waiting multiple months for 2.3?

[andrewazores]

I was hoping to do something like that, but the team decided it would make sense to have a more complete -agent before publishing and publicizing it. We also plan for the agent to support locally collecting JFR files and pushing them to Cryostat, inverting the data flow from Cryostat pulling over JMX to the Agent pushing over HTTP. That work is also partially completed but not in a satisfactory way - again the permissions system for that pushed data is not implemented, but also currently all of that pushed data is all stored in a single location in the archives, not correlated to the specific agent instance at all.

Our upstream and downstream release work for 2.2 is already well underway and it's a bit too late now to also add in productizing the agent for a downstream release, but I do think it's feasible to get an early upstream release of the agent done on a sooner timeframe - one that probably only implements the Discovery Plugin half to begin with, at least. The later agent release to accompany 2.3 would be both an upstream and downstream release and also implement the JFR push.

[petr-postulka]

Nice, looking forward to it. Thanks a lot for the details.

[andrewazores]

Hi everyone, sorry it took so long but this will be in 2.3 finally, releasing in mid-May. There is a new k8s CR to use with the Operator for creating multi-namespace Cryostat instances, and Cryostat can do its usual Endpoints querying/watching discovery of target JVMs across a provided list of namespaces. There is also the existing Discovery Plugin API that allows clients to hook up to Cryostat and publish information about target JVMs they can observe, which may be in any namespace, so this can be used to build bridges between other kinds of discovery mechanisms and what Cryostat is aware of. Finally, the Cryostat Agent will be delivered as part of the 2.3 release, and it implements a minimal Discovery Plugin API client so that JVMs it is attached to can be easily configured (just a few environment variables) to talk to Cryostat and publish information about themselves, too. I hope that some combination of these features satisfies your many-namespaces needs :-)