Migrate elasticsearch client to 8.* (or latest 7.*)

[potiuk]

Body

The eleasticsearch version we use is old (from 2021). It is eleasticsearch 7.* and since we are using elasticsearch-dsl, we cannot used elasticsearch 8 currently (first version of that was also released in 2021).

We are currently limited to < 7.15.0 additionally to being limited to 7, because 7.15.0 breaks our tests as we found out when we released some of the dependencies as part of #30067

We should migrate to a newer version of elasticsearch. There are two options:

1. Fix the tests and migrate to lates 7.* version - and bump to latest released 7* by fixing our tests (possible for short term).
2. Migreate to 8 version (better long term)

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[Owen-CH-Leung]

Since elasticsearch-dsl doesn't support elasticsearch 8, if we are to migrate to the official elasticsearch 8, does it mean we need to deprecate elasticsearch-dsl also ?

[potiuk]

If we do yes. But that also might be reason why we don't do it.

[Owen-CH-Leung]

Ok. Please assign this issue to me. I'll take some time to study the current usage of this package and propose whether to upgrade to 8.* or latest 7.*

[potiuk]

Done.

[Owen-CH-Leung]

I saw elasticsearch_dsl was only used in 3 files as follows (one usage in taskhandler & 2 in testing) :

airflow/airflow/providers/elasticsearch/log/es_task_handler.py

Line 291 in fc36d00

Search(index=self.index_patterns, using=self.client)

airflow/docker_tests/test_prod_image.py

Line 130 in fc36d00

"elasticsearch": ["elasticsearch", "es.elastic", "elasticsearch_dsl"],

airflow/tests/providers/elasticsearch/log/test_es_task_handler.py

Line 344 in fc36d00

with mock.patch("elasticsearch_dsl.Search.execute") as mock_execute:

Given the utilization of elasticsearch_dsl is fairly limited, I'll try to upgrade to elasticsearch 8.* and deprecate elasticsearch_dsl

[Owen-CH-Leung]

hmmmm - I just realize that we also use elasticsearch-dbapi, which requires elasticsearch==7.13.4

https://github.com/apache/airflow/blob/main/airflow/providers/elasticsearch/provider.yaml#L60
https://github.com/preset-io/elasticsearch-dbapi/blob/master/requirements.txt#L9

[Owen-CH-Leung]

I feel like if we are to upgrade to elasticsearch 8, we need to divide it into chunk of works (coz we will be deprecating 2 packages). Since elasticsearch-dbapi is used to construct the ElasticsearchSQLHook, we might first need to refactor it using the existing official elasticsearch library. Then we can go further to refactor the taskhandler that uses elasticsearch_dsl to use the existing elasticsearch library. Finally we can attempt to upgrade to 8, remove those 2 libraries, and fix the failing tests. What do you think ? @potiuk

[potiuk]

sure