OathKeeper Default Helm Chart Issue | Pod throwing 503.

[sabinayakc]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• This issue affects my Ory Network project.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Describe the bug

Installed helm chart using helm install oathkeeper ory/oathkeeper.

oathkeeper pod has an error with 503 code.

Reproducing the bug

• Install helm chart helm install oathkeeper ory/oathkeeper

Relevant log output

time=2023-04-21T17:39:53Z level=info msg=TLS has not been configured for api, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:53Z level=info msg=Listening on http://:4456 audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:53Z level=info msg=TLS has not been configured for proxy, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:53Z level=info msg=Listening on http://:4455 audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:55Z level=error msg=An error occurred while handling a request audience=application error=map[message:The requested resource could not be found] http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.23+] host:127.0.0.1 method:GET path:/health/ready query:<nil> remote:172.26.6.14:33658 scheme:http] http_response=map[status_code:503] service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:56Z level=error msg=An error occurred while handling a request audience=application error=map[message:The requested resource could not be found] http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.23+] host:127.0.0.1 method:GET path:/health/ready query:<nil> remote:172.26.6.14:33662 scheme:http] http_response=map[status_code:503] service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:57Z level=error msg=An error occurred while handling a request audience=application error=map[message:The requested resource could not be found] http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.23+] host:127.0.0.1 method:GET path:/health/ready query:<nil> remote:172.26.6.14:42292 scheme:http] http_response=map[status_code:503] service_name=ORY Oathkeeper service_version=v0.40.2

Relevant configuration

- Default Helm Values

Version

Helm Version: 0.31.0

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Kubernetes with Helm

Additional Context

• AWS EKS 1.24
• When I set demo: true it works
• I also tried creating a Rule using CRDs.

[Demonsthere]

Hello there,
This behaviour is due to the k8s readiness probe failing and restarting the pod. The probe is failing because no rules are present in the system, that is why the demo mode works, as it deploys sample data.
As Oathkeeper is not k8s native, it expects the rules to be present on start, and treats an an empty rule array as an error state.

[sabinayakc]

How can I instantiate it with a basic rule? Do I have to provide it a basic rule via Helm Values always?

The default helm has the following access rule which might be blank.

  config:
    access_rules:
      repositories:
        - file:///etc/rules/access-rules.json

I also created a rule using the CRD to see if it picks it up.

apiVersion: oathkeeper.ory.sh/v1alpha1
kind: Rule
metadata:
  name: test-rule
  namespace: default
spec:
  match:
    url: http://http-bin.example/echo
    methods:
    - GET
  authenticators:
  - handler: anonymous
  authorizer:
    handler: allow
  mutators:
  - handler: noop

[Demonsthere]

To clear up some confusion :)

• The first code snippet you posted configures where Oathkeeper will look for rules, as that can be defined as a local file, or an url. Please refer to https://www.ory.sh/docs/oathkeeper/reference/configuration for the full config :)
• Rules can be supplied to the chart during installation using this parameter: https://github.com/ory/k8s/blob/master/hacks/values/oathkeeper.yaml#L50 The values in hacks are used for CI testing and represent ready to use examples
• The CRDs will be picked up, but you need to enable the k8s controller to do that. The extra controller is configurable via this option https://github.com/ory/k8s/blob/master/hacks/values/oathkeeper.yaml#L3-L4
  Once the CRD is picked up, it will be transformed into an updated cm for the oathkeeper to read

[TartanLeGrand]

I see your version is 0.40.2 try to downgrade in 0.40.1

[Demonsthere]

Closing as this is an user error, please reopen if you need more guidance :)

[TartanLeGrand]

I think the bug is not fixed in the actual version (0.40.6)

👋 @Demonsthere

[Jorgagu]

This is a problem created by PR ory/oathkeeper#1061 and so Release v0.40.2, we should have a flag or option to enable the check performed on PR 1061.

@zepatrik @hperl In our case, if we use Oathkeeper Maester, we get a 503 error when launching Oathkeeper Readiness Probe because there are no rules, although we create them later from the controller.

👋 @Demonsthere can you please reopen the issue ?

[Demonsthere]

I see, so this is a upstream issue from oathkeeper itself 🤔 I will talk with the devs, maybe adding a --allow-empty-rules flag could be added to disable that check, which would be the default option for maester enabled charts

[zepatrik]

Would be fine by me.

[ognyvrac]

Is there any progress on this issue?

[jeffmccune]

This issue is unrelated to rules, the pod starts fine with no rules.

Please disregard, I was checking /health/alive instead of /health/ready

[MohammadBnei]

Any news on this issue ? I'm facing the exact same challenge, and it's quite frustrating.

If i put a dummy access rule via file, then use the CRD for more control, will there be any issues ?

[Demonsthere]

yes, using a dummy placeholder rule is a valid workaround, as it's a design in oathkeeper that it should start with the access rules supplied. We need to open a issue in oathkeeper to allow for such situation