[AIRFLOW-1695] Add RedshiftHook using boto3 #2717

andyxhadji · 2017-10-22T23:44:55Z

Dear Airflow maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1695

Description

Here are some details about my PR, including screenshots of any UI changes:
Added RedshiftHook using boto3, and tests.

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:
Added test_redshift_hook.py - includes unit tests for methods in class

Commits

My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

codecov-io · 2017-10-23T00:11:25Z

Codecov Report

Merging #2717 into master will increase coverage by 0.08%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2717      +/-   ##
==========================================
+ Coverage   72.48%   72.56%   +0.08%     
==========================================
  Files         154      154              
  Lines       11843    11843              
==========================================
+ Hits         8584     8594      +10     
+ Misses       3259     3249      -10

Impacted Files	Coverage Δ
airflow/hooks/__init__.py	`58.82% <ø> (ø)`	⬆️
airflow/models.py	`87.09% <0%> (-0.05%)`	⬇️
airflow/jobs.py	`79.66% <0%> (+0.44%)`	⬆️
airflow/utils/helpers.py	`56.32% <0%> (+2.87%)`	⬆️
airflow/task_runner/bash_task_runner.py	`100% <0%> (+6.66%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b3c247d...85d7e2d. Read the comment docs.

bolkedebruin · 2017-10-23T08:16:08Z

Is this reviewable? If so, please fix your commit message to make it readable. See commit guidelines. Furthermore please make sure the tests actually can run, include the right packages for ci.

sid88in · 2017-10-23T15:39:40Z

thanks @andyxhadji more power to #airflow and #aws

andyxhadji · 2017-10-23T22:07:19Z

@bolkedebruin This is now reviewable, all tests are passing. Please take a look!

andyxhadji · 2017-10-24T14:29:38Z

This is one of the follow-ups to #2532

bolkedebruin

Small nits. I don't think there is a need to touch aws_hook.py

bolkedebruin · 2017-10-28T13:20:09Z

airflow/contrib/hooks/aws_hook.py

        self.aws_conn_id = aws_conn_id
+        self.region_name = region_name


This is obtained from the extra connectin parameters, why have it in the signature?

bolkedebruin · 2017-10-28T13:22:38Z

Please also make sure your commit message adheres to the layout (descr lines are too long)

Adds RedshiftHook class, allowing for management of AWS Redshift clusters and snapshots using boto3 library. Also adds new test file and unit tests for class methods.

andyxhadji · 2017-10-28T16:09:46Z

@bolkedebruin Fixed - re-running tests because of an unrelated failure (https://travis-ci.org/apache/incubator-airflow/builds/294151528).

andyxhadji · 2017-10-28T17:39:02Z

@bolkedebruin Tests passing!

bolkedebruin · 2017-10-30T19:19:40Z

Your tests are not running as they are not prepended by "test_". Please verify in the travis that your tests run.

andyxhadji · 2017-10-30T19:32:43Z

@bolkedebruin Hmm, I'm not sure if I'm looking in the right place - the tests are prepended with "test_" and the tests run (screenshot from changed files & apache TravisCI below)

Am I missing something in a file? I really appreciate the help.

bolkedebruin · 2017-10-30T19:35:51Z

Apologies! I did do a search but I probably misspelled.

Adds RedshiftHook class, allowing for management of AWS Redshift clusters and snapshots using boto3 library. Also adds new test file and unit tests for class methods. Closes #2717 from andyxhadji/1695 (cherry picked from commit 4fb7a90) Signed-off-by: Bolke de Bruin <[email protected]>

andyxhadji · 2017-10-30T19:38:21Z

Not a problem! Thanks for the review!

…

On Mon, Oct 30, 2017 at 3:36 PM, bolkedebruin ***@***.***> wrote: Apologies! I did do a search but I probably misspelled. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#2717 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADcMX-3XcYDbwjRSiW4LLMz9K8H455kGks5sxiUjgaJpZM4QCJxv> .

Adds RedshiftHook class, allowing for management of AWS Redshift clusters and snapshots using boto3 library. Also adds new test file and unit tests for class methods. Closes apache#2717 from andyxhadji/1695

andyxhadji changed the title ~~[AIRFLOW-1695] RedshiftHook using boto3~~ [AIRFLOW-1695] Build RedshiftHook using boto3 Oct 23, 2017

andyxhadji force-pushed the 1695 branch from c24d9f9 to d257956 Compare October 23, 2017 13:00

andyxhadji changed the title ~~[AIRFLOW-1695] Build RedshiftHook using boto3~~ [AIRFLOW-1695] Add RedshiftHook using boto3 Oct 23, 2017

andyxhadji mentioned this pull request Oct 23, 2017

[AIRFLOW-1571] Add AWS Lambda Hook #2718

Closed

4 tasks

bolkedebruin requested changes Oct 28, 2017

View reviewed changes

[AIRFLOW-1695] Add RedshiftHook using boto3

85d7e2d

Adds RedshiftHook class, allowing for management of AWS Redshift clusters and snapshots using boto3 library. Also adds new test file and unit tests for class methods.

andyxhadji force-pushed the 1695 branch from d257956 to 85d7e2d Compare October 28, 2017 15:39

andyxhadji closed this Oct 28, 2017

andyxhadji reopened this Oct 28, 2017

asfgit closed this in 4fb7a90 Oct 30, 2017

andyxhadji deleted the 1695 branch October 30, 2017 19:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-1695] Add RedshiftHook using boto3 #2717

[AIRFLOW-1695] Add RedshiftHook using boto3 #2717

andyxhadji commented Oct 22, 2017 •

edited

Loading

codecov-io commented Oct 23, 2017 •

edited

Loading

bolkedebruin commented Oct 23, 2017

sid88in commented Oct 23, 2017

andyxhadji commented Oct 23, 2017

andyxhadji commented Oct 24, 2017

bolkedebruin left a comment

bolkedebruin Oct 28, 2017

bolkedebruin commented Oct 28, 2017

andyxhadji commented Oct 28, 2017

andyxhadji commented Oct 28, 2017

bolkedebruin commented Oct 30, 2017

andyxhadji commented Oct 30, 2017

bolkedebruin commented Oct 30, 2017

andyxhadji commented Oct 30, 2017 via email

		self.aws_conn_id = aws_conn_id
		self.region_name = region_name

[AIRFLOW-1695] Add RedshiftHook using boto3 #2717

[AIRFLOW-1695] Add RedshiftHook using boto3 #2717

Conversation

andyxhadji commented Oct 22, 2017 • edited Loading

JIRA

Description

Tests

Commits

codecov-io commented Oct 23, 2017 • edited Loading

Codecov Report

bolkedebruin commented Oct 23, 2017

sid88in commented Oct 23, 2017

andyxhadji commented Oct 23, 2017

andyxhadji commented Oct 24, 2017

bolkedebruin left a comment

Choose a reason for hiding this comment

bolkedebruin Oct 28, 2017

Choose a reason for hiding this comment

bolkedebruin commented Oct 28, 2017

andyxhadji commented Oct 28, 2017

andyxhadji commented Oct 28, 2017

bolkedebruin commented Oct 30, 2017

andyxhadji commented Oct 30, 2017

bolkedebruin commented Oct 30, 2017

andyxhadji commented Oct 30, 2017 via email

andyxhadji commented Oct 22, 2017 •

edited

Loading

codecov-io commented Oct 23, 2017 •

edited

Loading