beyla is unable to resolve dns of internal kubernetes service

[kirankumar-grootan]

I deployed Beyla using the below values.yaml file. in Beyla logs it shows 2024/05/30 14:25:34 failed to upload metrics: Post "http://monitoring-alloy:4318/v1/metrics": dial tcp: lookup monitoring-alloy on 8.8.8.8:53: no such host

service:
  enabled: true

env:
  BEYLA_KUBE_CLUSTER_NAME: test

## More configuration options available at https://grafana.com/docs/beyla/latest/configure/options/
config:
  create: true
  name: ""
  data:
    log_level: info
    print_traces: false

    network:
      enable: true
      allowed_attributes:
        - src.port
        - dst.port
        - k8s.src.name
        - k8s.dst.name
        - k8s.src.namespace
        - k8s.dst.namespace
        - k8s.src.type
        - k8s.dst.type
        - k8s.src.node.name
        - k8s.dst.node.name
        - k8s.cluster.name

    routes:
      patterns: []
        # - /user/{id}
        # - /user/{id}/basket/{product}
      ignored_patterns:
        - /auth/health
        - /auth/live
        - /auth/ready
        - /auth/metrics
        - /metrics
        - /health
        - /live
        - /readyz
        - /minio/v2/metrics/*
      ignore_mode: all
      # unmatched: heuristic

    otel_traces_export:
      endpoint: "http://monitoring-alloy:4318"
      protocol: http/protobuf
      insecure_skip_verify: false
      sampler:
        name: "parentbased_always_on"
        # arg: "0.1"

    otel_metrics_export:
      endpoint: "http://monitoring-alloy:4318"
      protocol: http/protobuf
      insecure_skip_verify: false
      interval: 5s
      report_target: true
      report_peer: true
      features: ["application", "network", "application_span", "application_service_graph"]

    attributes:
      select:
        beyla_network_flow_bytes:
          include: ["*"]
          exclude:
            - dst.cidr
            - src.cidr
            - k8s.cluster.name
            - k8s.dst.owner.name
            - k8s.src.owner.name
      kubernetes:
        enable: true
      instance_id:
        dns: false

    ebpf:
      track_request_headers: false
      http_request_timeout: 0s

    discovery:
      skip_go_specific_tracers: false
      services:
        - open_ports: 80,443,3000,4000,5000,6379,8070,8080,8443,9000,9200
          namespace: test

    filter:
      application: {}
      network: {}

    internal_metrics:
      prometheus:
        port: 6060
        path: /internal/metrics

    prometheus_export:
      port: 9090
      path: /metrics

preset: application

serviceMonitor:
  enabled: false

but the DNS resolution for k8s svc endpoints is just fine. other services have no issue in resolving only Beyla is not able to

[grcevski]

Hi @kirankumar-grootan, it looks like the DNS that beyla is connecting too is an external DNS, e.g. 8.8.8.8 looks like Google's DNS and it makes sense that monitoring-alloy won't be resolved.

From k8s standpoint, Beyla runs with the setting of hostNetwork:true, in which case the DNS will need something like this:

    spec:
      serviceAccount: beyla
      hostPID: true           # <-- Important. Required in Daemonset mode so Beyla can discover all monitored processes 
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet # <-- Important for DNS to work.
      containers:
      ...

[kirankumar-grootan]

thanks @grcevski for looking into this. but chart doesnt adding dnsPolicy. I need to patch it manually.

[kirankumar-grootan]

would be great if this is handled in helm https://github.com/grafana/beyla/blob/main/charts/beyla/templates/daemon-set.yaml#L39

{{- if or (eq .Values.preset "network") .Values.config.data.network }}
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
{{- end }}

happy to raise PR 🙂

[grcevski]

Yes please, it would be awesome if you can make a PR for this! Thanks @kirankumar-grootan!

[kirankumar-grootan]

Hi @grcevski I don't have access to push the code to a new branch and which target branch should I branch out from?

[mariomac]

Hi @kirankumar-grootan , thank you for your willing to help.

Te way we all PR to Beyla is:

1. Fork the Beyla repository.
2. Commit your changes into a branch of your fork.
3. Submit a PR across forks

[kirankumar-grootan]

@mariomac @grcevski #898

[kirankumar-grootan]

#898 merged. Thanks for accepting the PR.