Jaeger Agent Not Connecting to Collector

[gituser1379]

I have deployed an application to container along with Jaeger agent as side car deployment

Problem

Kubernetes used to startup both the containers in the pod with out any problem, from last week the jaeger agent seems to be having problems starting up within the same pod. When I looked at the container logs in the pod I see the following error

{"level":"warn","ts":1551640126.4296134,"caller":"tchannel/flags.go:72","msg":"Using deprecated configuration","option":"collector.host-port"}
{"level":"fatal","ts":1551640126.4301643,"caller":"agent/main.go:78","msg":"Could not create collector proxy","error":"could not create collector proxy, address is missing","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:121\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201"}

I am deploying Jaeger components to Kubernetes using the following command

kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml

My application yaml file looks like this

apiVersion: apps/v1
kind: Deployment
metadata:
  name: testproject-deployment
spec:
   selector:
    matchLabels:
      app: bwce-testproject
   replicas: 2
   template:
     metadata:
       labels:
         app: bwce-testproject
     spec:
       containers:
       - image: docker.io/syadavalli/bwce-testproject:latest
         name: bwce-testproject
         ports:
         - containerPort: 18000
         env:
         - name: BW_JAVA_OPTS
           value: "-Dbw.engine.opentracing.enable=true -Dbw.monitor.batchsize=1 -Dbw.monitor.publishtimer=1500"
         - name: BW_LOGLEVEL
           value: ERROR
         - name: BW_APP_MONITORING_CONFIG
           value: "{\"url\":\"http://100.98.34.21:80\"}"
       - image: jaegertracing/jaeger-agent
         name: jaeger-agent
         ports:
         - containerPort: 5775
           protocol: UDP
         - containerPort: 6831
           protocol: UDP
         - containerPort: 6832
           protocol: UDP
         - containerPort: 5778
           protocol: TCP
         args: ["--collector.host-port=100.69.87.41:14267"]
---
apiVersion: v1
kind: Service
metadata:
  name: my-testproject
spec:
  selector:
   app: bwce-testproject
  ports:
  - name: http
    port: 80
    targetPort: 18000
    protocol: "TCP"
  type: LoadBalancer

The collector host port is the correct one. I have manually replaced the --collector.host-port value by looking at the Kubernetes service

Proposal

I don't have a suggestion as to how to solve the problem.

Looks like there was a version upgrade from 1.9 to 1.10. Not sure if the issue I am experiencing has to do with the version upgrade. Please let me know why I am expriencing the above problem.

[jpkrohling]

Could you please the value for the image property in your jaeger deployment? We set it without an explicit version, which defaults to latest, but you should always set one when running it in production.

The option collector.host-port has been renamed as of 1.8.0, but it should still work.

In any case, could you please try changing this:

         args: ["--collector.host-port=100.69.87.41:14267"]

to this:

         args: ["--reporter.tchannel.host-port=100.69.87.41:14267"]

[pavolloffay]

Can you confirm what jaeger version are you using? The images in the template are not tagged so it seems you are running latest jaeger version.

The problem is related to reporter changes in the agent. The latest image uses by default grpc reporter which we want to make default in jaeger 1.11 #1343. The --collector.host-port instructs agent to use thrift reporter. This has been deprecated for some time.

So now you have two options:

• change agent args to --reporter.type=thrift --reporter.tchannel.host-port=100.69.87.41:14267 to use thrift
• use grpc reporter by --reporter.type=grpc --collector.host-port=100.69.87.41:14250 - you might have to check that grpc port is open.

We should upgrade our templates and expose grpc and use it. By making grpc default in Jaeger 1.11 we are going to break some deployments. Maybe we could still choose thrift if --collector.host-port is not empty && reporter.type=grpc && reporter.gprc.address is empty.

[objectiser]

Maybe we could still choose thrift if --collector.host-port is not empty && reporter.type=grpc && reporter.gprc.address is empty.

Might be a good approach, and produce a warning to say that grpc is now the default so should change.

[gituser1379]

After I changed the arguments to args: ["--reporter.type=grpc", "--collector.host-port=100.69.87.41:14250"] I see the following error in the Pod/container logs

{"level":"fatal","ts":1551700647.2718337,"caller":"agent/main.go:78","msg":"Could not create collector proxy","error":"unknown reporter type grpc, --collector.host-port=100.68.163.229:14250","errorVerbose":"unknown reporter type grpc, --collector.host-port=100.68.163.229:14250\nmain.createCollectorProxy\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:141\nmain.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:76\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:121\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:121\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201"}

And when I change arguments to args: ["--reporter.type=thrift", "--reporter.tchannel.host-port=100.69.87.41:14267"] I see the following error in the Pod/container logs

{"level":"fatal","ts":1551701176.9590647,"caller":"agent/main.go:78","msg":"Could not create collector proxy","error":"unknown reporter type thrift","errorVerbose":"unknown reporter type thrift\nmain.createCollectorProxy\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:141\nmain.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:76\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:121\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/agent/main.go:121\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201"}

[pavolloffay]

Sorry it's not thrift it should be tchannel

[pavolloffay]

The grpc reporter should be recognized. What version of jaeger are you using?

[gituser1379]

Not sure how I can tell which version is being pulled from git because I am using the following command to deploy Jaeger components to K8s
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml

Though changing to args: ["--reporter.type=tchannel", "--reporter.tchannel.host-port=100.69.87.41:14267"] seems to work.

[pavolloffay]

The version can be found on /version endpoint registered on healthcheck endpoint.