Adds example showing the ES_hook #17944

frankcash · 2021-08-31T14:19:50Z

Adds an example DAG showing how to utilize the elasticsearch_hook

Will produce output along the lines of:

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

github-actions · 2021-09-01T19:27:43Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

frankcash · 2021-09-01T19:49:05Z

@potiuk it appears CI is failing due to unrelated things with my pr

  + echo Installing all packages with constraints and upgrade if needed
  + echo
  + pip install --editable '.[devel_ci]' --constraint https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.6.txt
  Obtaining file:///opt/airflow
  Requirement already satisfied: alembic<2.0,>=1.2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.6.5)
  Requirement already satisfied: argcomplete~=1.10 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.12.3)
  Requirement already satisfied: attrs<21.0,>=20.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (20.3.0)
  Requirement already satisfied: blinker in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.4)
  Requirement already satisfied: clickclick>=1.2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (20.10.2)
  Requirement already satisfied: colorlog>=4.0.2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (4.8.0)
  Requirement already satisfied: croniter<1.1,>=0.3.17 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.0.15)
  Requirement already satisfied: cryptography>=0.9.3 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (3.4.8)
  Requirement already satisfied: dill<0.4,>=0.2.2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.3.1.1)
  Requirement already satisfied: docutils<0.17 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.16)
  Requirement already satisfied: flask<2.0,>=1.1.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.1.4)
  Requirement already satisfied: flask-appbuilder<4.0.0,>=3.3.2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (3.3.2)
  Requirement already satisfied: flask-caching<2.0.0,>=1.5.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.10.1)
  Requirement already satisfied: flask-login<0.5,>=0.3 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.4.1)
  Requirement already satisfied: flask-wtf<0.15,>=0.14.3 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.14.3)
  Requirement already satisfied: graphviz>=0.12 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.17)
  Requirement already satisfied: gunicorn>=19.5.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (20.1.0)
  Requirement already satisfied: httpx in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.19.0)
  Requirement already satisfied: importlib_resources~=1.4 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.5.0)
  Requirement already satisfied: inflection>=0.3.1 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.5.1)
  Requirement already satisfied: iso8601>=0.1.12 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.1.16)
  Requirement already satisfied: itsdangerous<2.0,>=1.1.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.1.0)
  Requirement already satisfied: jinja2<4,>=2.10.1 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (2.11.3)
  Requirement already satisfied: jsonschema~=3.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (3.2.0)
  Requirement already satisfied: lazy-object-proxy in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.4.3)
  Requirement already satisfied: lockfile>=0.12.2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.12.2)
  Requirement already satisfied: markdown<4.0,>=2.5.2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (3.3.4)
  Requirement already satisfied: markupsafe<2.0,>=1.1.1 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.1.1)
  Requirement already satisfied: marshmallow-oneofschema>=2.0.1 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (3.0.1)
  Requirement already satisfied: openapi-spec-validator>=0.2.4 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.3.1)
  Requirement already satisfied: pendulum~=2.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (2.1.2)
  Requirement already satisfied: psutil<6.0.0,>=4.2.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (5.8.0)
  Requirement already satisfied: pygments<3.0,>=2.0.1 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (2.10.0)
  Requirement already satisfied: pyjwt<2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.7.1)
  Requirement already satisfied: python-daemon>=2.2.4 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (2.3.0)
  Requirement already satisfied: python-dateutil<3,>=2.3 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (2.8.2)
  Requirement already satisfied: python-nvd3~=0.15.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.15.0)
  Requirement already satisfied: python-slugify<5.0,>=3.0.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (4.0.1)
  Requirement already satisfied: python3-openid~=3.2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (3.2.0)
  Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (5.4.1)
  Requirement already satisfied: rich>=9.2.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (10.7.0)
  Requirement already satisfied: setproctitle<2,>=1.1.8 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.2.2)
  Requirement already satisfied: sqlalchemy>=1.3.18 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.3.24)
  Requirement already satisfied: sqlalchemy_jsonfield~=1.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (1.0.0)
  Requirement already satisfied: swagger-ui-bundle>=0.0.2 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.0.8)
  Requirement already satisfied: tabulate<0.9,>=0.7.5 in /usr/local/lib/python3.6/site-packages (from apache-airflow==2.2.0.dev0) (0.8.9)
  ERROR: Cannot install apache-airflow[devel-ci]==2.2.0.dev0 because these package versions have conflicting dependencies.
  
  The conflict is caused by:
      apache-airflow[devel-ci] 2.2.0.dev0 depends on tenacity~=6.2.0
      The user requested (constraint) tenacity==8.0.1
  
  To fix this you could try to:
  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict
  
  ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
  WARNING: You are using pip version 21.2.2; however, version 21.2.4 is available.
  You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
  The command '/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then         bash /scripts/docker/install_airflow.sh;     fi' returned a non-zero code: 1

Anything I need to do on my end?

adds comment

frankcash · 2021-09-02T14:42:03Z

Just rebased

potiuk · 2021-09-02T19:37:55Z

🤞

frankcash · 2021-09-07T11:56:20Z

🎉

…

On Sat, Sep 4, 2021 at 3:48 PM Jarek Potiuk ***@***.***> wrote: Merged #17944 <#17944> into main. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#17944 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABN32R7KJM2T4DSLWPWNHZLUAJZZXANCNFSM5DEHNTHQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Charles Frank Cash FOSS Programmer ***@***.*** https://github.com/frankcash https://keybase.io/frankcash <https://github.com/frankcash>

softstar1990 · 2022-05-25T12:07:59Z

Are there any examples other than "show tables"?
tables = es_conn.execute('SHOW TABLES')
shall i use SQL or EQL or DSL or something else?

boring-cyborg bot added the area:providers label Aug 31, 2021

vishnuravi approved these changes Sep 1, 2021

View reviewed changes

potiuk approved these changes Sep 1, 2021

View reviewed changes

potiuk closed this Sep 1, 2021

potiuk reopened this Sep 1, 2021

github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Sep 1, 2021

Adds example showing the ES_hook

7226297

adds comment

reformats code

f0e9374

frankcash closed this Sep 2, 2021

frankcash reopened this Sep 2, 2021

frankcash added 3 commits September 2, 2021 13:03

Black formatting

ef51845

airflow.operators.python_operator -> airflow.operators.python

d6de398

Updates docs to reference example DAGs

9a4c3a8

frankcash requested review from mik-laj and vikramkoka as code owners September 2, 2021 19:31

potiuk merged commit a0a05ff into apache:main Sep 4, 2021

potiuk mentioned this pull request Oct 30, 2021

Status of testing Providers that were prepared on October 30, 2021 #19328

Closed

54 tasks

potiuk mentioned this pull request Dec 31, 2021

Status of testing Providers that were prepared on December 31, 2021 #20615

Closed

91 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds example showing the ES_hook #17944

Adds example showing the ES_hook #17944

frankcash commented Aug 31, 2021 •

edited

Loading

github-actions bot commented Sep 1, 2021

frankcash commented Sep 1, 2021

frankcash commented Sep 2, 2021

potiuk commented Sep 2, 2021

frankcash commented Sep 7, 2021 via email

softstar1990 commented May 25, 2022

Adds example showing the ES_hook #17944

Adds example showing the ES_hook #17944

Conversation

frankcash commented Aug 31, 2021 • edited Loading

github-actions bot commented Sep 1, 2021

frankcash commented Sep 1, 2021

frankcash commented Sep 2, 2021

potiuk commented Sep 2, 2021

frankcash commented Sep 7, 2021 via email

softstar1990 commented May 25, 2022

frankcash commented Aug 31, 2021 •

edited

Loading