fix: prometheus input plugin run outside k8s cluster error

[yahaa]

Required for all PRs

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

resolves #

The first one, when we run telegra in outside of a kubernetes cluster the line config, err := rest.InClusterConfig() will be always get an error.

The second one, the method of getting rest.Config instance in loadClient function will be cause kubernetes client has error materURL, and this error can be reproduced by using the following UT:

// add this UT into plugins/inputs/prometheus/kubernetes_test.go
func TestLoadClient(t *testing.T) {
	kubeconfig := "/path_to_your_kubeconfig"
	client, err := loadClient(kubeconfig)
	if err != nil {
		t.Errorf("got err: %v", err)
		return
	}

	watcher, err := client.CoreV1().Pods("kube-system").Watch(context.TODO(), metav1.ListOptions{
		LabelSelector: "app=debug",
	})
	if err != nil {
		t.Errorf("watch pod got err: %v", err)
		return
	}
	defer watcher.Stop()
}

this UT can got watch pod got err: Get "http://localhost/api/v1/namespaces/kube-system/pods?labelSelector=app%3Ddebug&watch=true": dial tcp 127.0.0.1:80: connect: connection refused error.

[yahaa]

Hi @sspaink @powersj I was a little busy some time ago, so I closed this pr #9960,

I make a new pr again, please help me to review it.

[powersj]

Hi,

Can you remind me if you filed a bug for this issue?

Also, I understand the only way to try to start monitoring a K8s cluster is if monitor_kubernetes_pods is set to true. If you set that to false, shouldn't that avoid this?

Thanks

[yahaa]

Hi,

Can you remind me if you filed a bug for this issue?

Also, I understand the only way to try to start monitoring a K8s cluster is if monitor_kubernetes_pods is set to true. If you set that to false, shouldn't that avoid this?

Thanks

In this case such as following input config，the telegraf process is not running properly

[[inputs.prometheus]]
  monitor_kubernetes_pods = true
  kube_config = "/path/to/kubernetes.config"

[yahaa]

Hi,
Can you remind me if you filed a bug for this issue?
Also, I understand the only way to try to start monitoring a K8s cluster is if monitor_kubernetes_pods is set to true. If you set that to false, shouldn't that avoid this?
Thanks

In this case such as following input config，the telegraf process is not running properly

[[inputs.prometheus]]
  monitor_kubernetes_pods = true
  kube_config = "/path/to/kubernetes.config"

In my environment, telegraf is running outside the k8s cluster, so I need to specify kubeconfig to run telegraf, and then I get an error.

After analyzing the prometheus input plugin code, I found that the problem occurs in the following two points:

1. https://github.com/influxdata/telegraf/blob/master/plugins/inputs/prometheus/kubernetes.go#L59 always try to run inCluster mode.
2. https://github.com/influxdata/telegraf/blob/master/plugins/inputs/prometheus/kubernetes.go#L52 using this way to get kubeconfig will cause this k8s client watch pod error (watch pod got err: Get "http://localhost/api/v1/namespaces/kube-system/pods?labelSelector=app%3Ddebug&watch=true": dial tcp 127.0.0.1:80: connect: connection refused error.)

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[yahaa]

Hi @powersj would you please help me to review this pr once again?