Alternative Mechanisms for Real-Time Policy Distribution

[shumbashi]

Hello All;

First, please excuse my ignorance of all things OPA. I'm working on my Master's Thesis in Computer Science and looking for an interesting problem to tackle.

I'm interested in how policies are distributed to multiple OPAs. From my understanding, OPA supports using the "Bundle Service API" to periodically download bundles of policy and data. While functional, this pull mechanism inherently limits the immediacy of policy updates due to its reliance on pull frequencies.

This begs the question: Are there vulnerabilities in this approach? Could a real-time policy distribution mechanism be a valuable enhancement?

I'm eager to hear your insights on this. Specifically, do you believe there's merit in exploring alternative methods, such as leveraging a message broker like Kafka? Imagine OPA agents subscribing to designated channels (potentially via labels and the Discovery API) and receiving policy updates in real-time pushed by the message broker. Could this potentially address the limitations of the current pull-based system?

I'm keen to learn if you think this is a worthwhile endeavor, and if it aligns with the community's needs and interests.

Best Regards,

[anderseknert]

Hi there @shumbashi 👋

There are several alternatives to the pull based approach available for policy and data distribution, but first — the pull based approach is not limited to polling at given intervals. You can configure long polling as well as (optionally) delta bundles to have updates distributed as soon as they're made available.

OPA also allows for a push based approach via it's REST API. This is leveraged by external data providers like kube-mgmt and OPAL.

Real-time updates directly from systems like Kafka (and many others) is supported via Enterprise OPA.

Finally, there's also the http.send built-in function which is commonly used to fetch data at policy evaluation time.

All things considered, I'd say there's a good amount of options for fetching policy and data, and which one you'll want depends on many factors, like the expected size of the data set, frequency of updates, complexity concerns, and more.

[shumbashi]

Thank you very much for the informative answer, so many things I didn't know 😄. It does sound like this problem is well-tackled in many ways.