elasticsearchexporter-v0.64 is incompatible with ES server 7.10.2

[canhnt]

Component(s)

exporter/elasticsearch

What happened?

Description

The elasticsearch exporter v0.64.0 that uses go-elasticsearch v8.4. Because of compatibility issue, it cannot deliver logs to elasticsearch-server old version 7.10.2

Steps to Reproduce

1. Run otelcontrib with the following config:

receivers:
  filelog:
    include: [/var/log/test/*.json]
    include_file_name: false
    include_file_path: true
    max_log_size: 5MiB
    operators:
      - type: json_parser
        # on_error: drop
        timestamp:
          parse_from: attributes.timestamp
          layout: "%Y-%m-%d %H:%M:%S"
        severity:
          parse_from: attributes.severity
processors:
  batch:
  memory_limiter:
    limit_percentage: 75
    spike_limit_percentage: 15
    check_interval: 1s
extensions:
  health_check: {}
  zpages: {}
  memory_ballast:
    size_mib: 128
exporters:
  logging:
    logLevel: debug
  elasticsearch:
    endpoints: ['http://elasticsearch-master:9200']
    logs_index: "otel-log-demo-01"
service:
  extensions: [health_check, zpages, memory_ballast]
  pipelines:
    logs:
      receivers:
        - filelog
      processors:
        - memory_limiter
        - batch
      exporters:
        - logging
        - elasticsearch

2. Deploy es-server 7.10.2 via helm chart:

    helm upgrade --install elasticsearch elastic/elasticsearch \
        --version=7.10.2 \
        --values ./elasticsearch-values.yaml \

with elasticsearch-values.yaml file:

replicas: 3
minimumMasterNodes: 2

sysctlInitContainer:
  enabled: false
persistence:
  enabled: false

Create the index:

curl -X PUT http://elasticsearch-master:9200/otel-log-demo-01

3. Generate logs to /var/log/test folder

printf "{\"timestamp\": \"$(date +'%Y-%m-%d %H:%M:%S')\", \"key\": 2, \"severity\": \"info\", \"body\":\"hello world\"}\n" >> ./test_logs/2.json

Expected Result

Logs are delivered to elasticsearch server

Actual Result

Exception thrown by otel.

When we test with elasticsearch server 7.17.2, the exception does not happen.

Collector version

0.64.0

Environment information

Environment

client OS:

Darwin N6P47FW00N 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000 arm64

K8s environment: v1.20.11

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: [/var/log/test/*.json]
    include_file_name: false
    include_file_path: true
    max_log_size: 5MiB
    operators:
      - type: json_parser
        # on_error: drop
        timestamp:
          parse_from: attributes.timestamp
          layout: "%Y-%m-%d %H:%M:%S"
        severity:
          parse_from: attributes.severity
processors:
  batch:
  memory_limiter:
    limit_percentage: 75
    spike_limit_percentage: 15
    check_interval: 1s
extensions:
  health_check: {}
  zpages: {}
  memory_ballast:
    size_mib: 128
exporters:
  logging:
    logLevel: debug
  elasticsearch:
    endpoints: ['http://elasticsearch-master:9200']
    logs_index: "otel-log-demo-01"
service:
  extensions: [health_check, zpages, memory_ballast]
  pipelines:
    logs:
      receivers:
        - filelog
      processors:
        - memory_limiter
        - batch
      exporters:
        - logging
        - elasticsearch

Log output

2022-11-15T10:42:43.080Z	error	[email protected]/elasticsearch_bulk.go:183	Bulk indexer error: flush: the client noticed that the server is not Elasticsearch and we do not support this unknown product	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch/payapplogs"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/elasticsearch_bulk.go:183
github.com/elastic/go-elasticsearch/v8/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/[email protected]/esutil/bulk_indexer.go:322

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[MovieStoreGuy]

Are you aware if there is an open issue for github.com/elastic/go-elasticsearch that related to this?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearch: @JaredTan95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmitryax]

Should be fixed by #16353