question: suggestion on effecting changes in external system

[newly12]

OPA is really great!

I started evaluating OPA recently and I would like to ask for some suggestions on how to make changes in external system during a policy execution phase. As per the doc, both http.send and custom built-in functions are mentioned not to be guaranteed to be executed. In my case, the decision making depends on historical data, for example, when a request is allowed, I would like to persist the request along with some additional data in a DB, and such historical data will be fetched from remote to evaluate the next request. Some workaround that I can think of

• a proxy service that intercepts the result, and check what data needs to be persisted, the cons might be additional latency by http calls
• a decision log plugin that basically does the similar thing

and both have to follow the contract that a specific field in the result states whether and what to be persisted.

default allow = true

persist = doc {
  allow
  doc = input.request
}

Not sure what is right way of doing so, any suggestion is appreciated. Thanks.

[anderseknert]

Hi Peter! That's a peculiar use case 😃

My first thought was that the caller (i.e. the client) would be the right place to collect the result and persist it elsewhere.. but I take it you have considered that already and dismissed the idea?

How fast are these requests expected to come in? I mean, would request #2 come just milliseconds after request #1, or how much time do you have to process the first decision? I could imagine building a custom service to process decision logs, parse out whatever data is of interest and then feed that back into a bundle or via OPA's REST API... but it would obviously not be instantaneous.

I think we can exclude using http.send for this purpose.

[newly12]

Hi Anders, thanks for your reply!

My first thought was that the caller (i.e. the client) would be the right place to collect the result and persist it elsewhere.. but I take it you have considered that already and dismissed the idea?

I would think to decouple the persisting responsibility from the client(requester), and move it to the whole decision making service as a whole considering historical data is the context for the evaluation, that being said, if it has to be done in this way, I will have a proxy service takes care of this to decouple it from the original client. :)

How fast are these requests expected to come in?

This is a good question! I haven't considered the "race" condition, considering similar requests usually do not happen that frequently(milliseconds).

or how much time do you have to process the first decision?

As eventually historical data will become large, they will be persisted in a remote storage(DB or NoSQL) and probably not suite in the bundle(and also considering the freshness), an evaluation could take milliseconds or less than a second, but it's acceptable.

So.. say if the persist is wrapped into a built-in function, which seems to be more intuitive, as it's part of the evaluation process, and user can see how it is done as a whole by looking at the rego file, is there a way to guarantee the call will not be omitted by OPA?

[anderseknert]

I would think to decouple the persisting responsibility from the client(requester), and move it to the whole decision making service as a whole

OPA is, like you said, responsible for the decision making. It is however not responsible for enforcement, or any other type of side effect that may be produced as a result of its decisions. Moving that responsibility into OPA is not decoupling (from an OPA POV) — it's introducing it.

As eventually historical data will become large, they will be persisted in a remote storage(DB or NoSQL) and probably not suite in the bundle

If that's so, I'd suggest using http.send for querying (but not persisting) that data rather than pulling it into OPA, given some latency was tolerable.

So.. say if the persist is wrapped into a built-in function, which seems to be more intuitive, as it's part of the evaluation process, and user can see how it is done as a whole by looking at the rego file

From OPA's perspective, this isn't intuitive at all. Again I think you're conflating the "evaluation process" with enforcement and side effects. OPA is built entirely with the assumption that isn't handled in policy.

is there a way to guarantee the call will not be omitted by OPA?

Sure, I don't think it'll get omitted if it's in a path required for the policy decision.

[newly12]

I think you are making valid points, querying data is part of the evaluation, but persist is actually not. Thanks for your time!