Create a KubernetesTarget which pulls from the pod/logs api, not files

[justinmchase]

Is your feature request related to a problem? Please describe.
My companies kubernetes cluster is managed and I do not have permissions to mount hostPath volumes. Therefore I cannot access the log files via file systems for nodes.

kind: Deployment
apiVersion: apps/v1
spec:
  template:
    spec:
      volumes:
      - name: logs
        hostPath:
          path: HOST_PATH # <-- permission denied

I can however clearly see the logs via the kubernetes api for the entire cluster from my terminal with:

kubectl logs -f -l "name=my-app" -l "name=another-app"

Describe the solution you'd like
I would like any solution that allows me to solve the problem without having to mount to the node's file system via hostPath volumes.

I imagine this solution would be to just simply call the kubernetes logging apis instead of relying on files and volume mounting. I would just like to use the same config that I am using for prometheus ideally:

scrape_configs:
- job_name: my-app-pods
  pipeline_stages: ...
  kubernetes_sd_configs:
  - role: pod
    selectors:
    - role: pod
      label: app.kubernetes.io/name=my-app

I feel like the above should be all you need, no access to log files needed. It should enumerate the pods based on the selector then it should monitor pod events and call out to the pod/logs api to tail their logs instead of trying to map them to a volume mounted file and then forward the logs to the database from there.

Describe alternatives you've considered

• I've considered using fluentd
• I've considered making my own code to pull from the kubernetes api
• I've considered using a totally different approach which would put the burden of logging directly to the database from the apps...

Additional context
I am using prometheus and already have that working, it is able to use the pod discover and poll the pods endpoint.

[justinmchase]

I did end up getting this to work by using the file system and sidecar but I ahve to admit it was super confusing to me, knowing that the log apis already exist in kubernetes but then finding out that the kubernetes_sd_configs is only using the pod detection api.