AIRFLOW__ELASTICSEARCH__OFFSET_FIELD is not working

[PatrykKlimowicz]

After update to airflow 2.3.3 I cannot obtain logs from ES due to possible conflict of offset keys.

Basically, we added a offset key with help of Fluent Bit named 'abc' and specify AIRFLOW__ELASTICSEARCH__OFFSET_FIELD="abc", but for some reason this block the creation of offset field (This works well for v2.1.3). When the default is used for that config I can see that both abc and offset keys are created correctly as JSON fields and logs are visible on UI - I assume that the logs retrieval works based on the default airflow offset field and not via our custom one.

Is there a chance to have this working with the custom offset field only? I need to take care of older logs as well - they must be visible on UI for reference and dropping the custom offset field is not an option here...

[potiuk]

It it is quite likely that some new version of Elasticsearch provider is a problem. If so then you can downgrade to one of the previous provider versions: https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/stable/index.html#changelog - actually doing so might be able to determine what caused the problem because you might try to downgrade one-by-one and see which one will fix the problem.

Once you find out if specific provider fixes your problem, it would be great if you could create an issue and describe it in detail (and mention which downgrade fixes the problem).

You can see more about upgrading/downgrading providers separately from the core here: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installing-upgrading-downgrading-providers-separately-from-airflow-core

[PatrykKlimowicz]

I'll follow your suggestion and create an issue with findings. Thanks!