Deploy individual jaeger components(agent/collector/query) instead of all-in-one

[wentao-zh]

Describe the feature request
The deployment of Jaeger is all-in-one mode. That means all of the 3 components(jaeger agent/collector/query UI) are in one container and data is in memory. This deployment can not be used in production. A better way is to provide individual deployment for each jeager component.

Describe alternatives you've considered
I modified the jaeger deployment part of istio yaml like the snippet below. And specify an ELK as it's backend storage. It works for me.

spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jaeger
      annotations:
        sidecar.istio.io/inject: "false"
        scheduler.alpha.kubernetes.io/critical-pod: ""
    spec:
      containers:
        - name: jaeger-agent
          image: jaegertracing/jaeger-agent
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5778
              protocol: TCP
            - containerPort: 5775
              protocol: UDP
            - containerPort: 6831
              protocol: UDP
            - containerPort: 6832
              protocol: UDP
          env:
            - name: COLLECTOR_ZIPKIN_HTTP_PORT
              value: "jaeger-collector.istio-system.svc.cluster.local:14267"         
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-tracing-query
  namespace: istio-system
  labels:
    app: istio-tracing-query
    chart: tracing-query-0.1.0
    release: istio
    heritage: Tiller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jaeger
      annotations:
        sidecar.istio.io/inject: "false"
        scheduler.alpha.kubernetes.io/critical-pod: ""
    spec:
      containers:
        - name: jaeger-query
          image: jaegertracing/jaeger-query
          imagePullPolicy: IfNotPresent
          args:
          - "--es.server-urls=http://$ELASTIC_SEARCH:9200"
          ports:
            - containerPort: 16686
              protocol: TCP
            - containerPort: 16687
              protocol: TCP
          env:
          - name: SPAN_STORAGE_TYPE
            value: "elasticsearch"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-tracing-collector
  namespace: istio-system
  labels:
    app: istio-tracing-collector
    chart: tracing-collector-0.1.0
    release: istio
    heritage: Tiller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jaeger
      annotations:
        sidecar.istio.io/inject: "false"
        scheduler.alpha.kubernetes.io/critical-pod: ""
    spec:
      containers:
        - name: jaeger-collector
          image: jaegertracing/jaeger-collector
          imagePullPolicy: IfNotPresent
          args:
          - "--es.server-urls=http://$ELASTIC_SEARCH:9200"
          - "--es.num-shards=1"
          ports:
            - containerPort: 14269
              protocol: TCP
            - containerPort: 14268
              protocol: TCP
            - containerPort: 14267
              protocol: TCP
            - containerPort: 9411
              protocol: TCP
          env:
          - name: COLLECTOR_ZIPKIN_HTTP_PORT
             value: "9411"
          - name: SPAN_STORAGE_TYPE
             value: "elasticsearch"

Additional context
I'd like to put the snippet into the Helm Chart template of istio and summit a PR

[gyliu513]

This can help Jaeger for production use, thanks @wentao-zh .

/cc @linsun @sdake @costinm

[jpkrohling]

Perhaps a better approach would be to make use of the Jaeger Operator, which allows for a simplified configuration of the Jaeger instance, while still maintaining the flexibility about using all-in-one (in-memory or persistent storage) and production (persistent storage only).

https://github.com/jaegertracing/jaeger-operator

If there's anything in the operator that would be needed for this, open an issue there and we can certainly work on it. For instance, I believe you might need support for custom metadata (labels? annotations?).

[objectiser]

As discussed in #8893, there is the alternative of not including tracing.enabled=true and installing a production version of Jaeger separately - either via the upstream community helm chart, or using the new jaeger-operator as mentioned by @jpkrohling.

The issue with the production version is that it requires more resources - not least Elasticsearch or Cassandra storage. So I don't think this should be automatically installed when tracing.enabled=true.

So first decision is whether the Istio helm chart should provide an option to include the production configuration, or whether we should just document how a production environment can be installed?

[gyliu513]

/cc @douglas-reid

[douglas-reid]

@objectiser my instinct is that we should provide guidance on how to productionize, instead of trying to add all of the options into helm.

I think this is a question worthy of discussion in the WG. I'm adding it to the agenda for the next meeting (two weeks from today).

[gyliu513]

+1 to discuss this in WG meeting. But my thinking is we still can update the helm chart to add those options for easy configuration and installation, and the guidance for production can actually refer to the helm chart options, this maybe more simple.

@douglas-reid , as @wentao-zh is located in China, not sure if the time is China friendly. Just check the meeting notes at https://docs.google.com/document/d/1pn9QdRcoyT_nxOwzklsiYpt7OQraaSDfmtN14XTOrN0/edit#heading=h.vsumuoszb5i1 and did not found the time for the meeting, can you please update the time in the document?

[clyang82]

@douglas-reid Any decisions or suggestions from WG meeting about this topic? Thanks.

[objectiser]

@clyang82 Originally two options were on the table for discussion, (1) update the istio helm chart to include the production config, or (2) provide documentation on how to use the production approach.

My concern with (1) in the context of Jaeger, is that it would introduce yet another place where configuration needs to be maintained - along with upstream helm chart, jaeger operator, etc.

So the suggestion was to explore ways in which an upstream configuration could be "imported" into the Istio helm chart to avoid duplication.

I've done some initial experiments just copying the upstream helm charts for Jaeger and Elasticsearch into the Istio helm chart but ran into some issues with the way namespaces are defined. If this issue was resolved, then we would need to work out a way to sync the upstream helm charts somehow.

The other approach I am exploring is using the jaeger-operator. This is promising although may need some refinement - had to fix a couple of issues.

[clyang82]

@objectiser Thanks to share your idea. I am trying to use jaeger-operator on top of istio. Just fix a problem to support es client certificate - jaegertracing/jaeger#1139. After this fix, it works in my environment right now. But may need some enhancement is jaeger-operator, I will continue to try tomorrow.

[objectiser]

@clyang82 It would be great if you can identify areas in the jaeger-operator that need improvement to enable it to be used in the Istio helm chart.

If/when you find issues, if you could create an issue in the jaeger-operator repo, and cross reference it in issue #9508, that would be appreciated.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[insider89]

Please don't close this one, still actual.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[objectiser]

It was decided that the Istio installer would only support a simple installation of Jaeger, and if more complex configurations were required they would be installed separately (e.g. via the jaeger operator) with Istio being configured to point to the external instance. See https://istio.io/docs/tasks/telemetry/distributed-tracing/jaeger/#before-you-begin