Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EC2CreateInstanceOperator, EC2TerminateInstanceOperator #29548

Merged

Conversation

syedahsn
Copy link
Contributor

This PR creates two new operators EC2CreateInstanceOperator and EC2TerminateInstanceOperator which will be used to create and terminate EC2 instances respectively. The system test has been updated to use the new operators.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Copy link
Contributor

@josh-fell josh-fell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty solid, just some small comments but one that should be addressed related to unpacking an arg.

airflow/providers/amazon/aws/operators/ec2.py Outdated Show resolved Hide resolved
airflow/providers/amazon/aws/operators/ec2.py Outdated Show resolved Hide resolved
airflow/providers/amazon/aws/operators/ec2.py Show resolved Hide resolved
airflow/providers/amazon/aws/operators/ec2.py Outdated Show resolved Hide resolved
@syedahsn syedahsn force-pushed the syedahsn/ec2-create-terminate-operators branch from 6e4650d to ccf8984 Compare February 17, 2023 23:36
@syedahsn syedahsn force-pushed the syedahsn/ec2-create-terminate-operators branch from 879922e to 2be47c1 Compare February 21, 2023 00:25
@eladkal eladkal changed the title Syedahsn/ec2 create terminate operators Add EC2CreateInstanceOperator, EC2TerminateInstanceOperator Feb 22, 2023
Copy link
Contributor

@josh-fell josh-fell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small docstring update, but LGTM!

airflow/providers/amazon/aws/operators/ec2.py Outdated Show resolved Hide resolved
@syedahsn syedahsn force-pushed the syedahsn/ec2-create-terminate-operators branch 2 times, most recently from c34ff0b to e5a84e0 Compare February 27, 2023 18:09
Copy link
Contributor

@eladkal eladkal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
will merge when CI is green

@josh-fell
Copy link
Contributor

Seems like the "Wait for CI images" step failure could be related?

ERROR tests/system/providers/amazon/aws/example_ec2.py - botocore.exceptions.NoCredentialsError: Unable to locate credentials
______ ERROR collecting tests/system/providers/amazon/aws/example_ec2.py _______
tests/system/providers/amazon/aws/example_ec2.py:91: in <module>
    image_id = _get_latest_ami_id()
tests/system/providers/amazon/aws/example_ec2.py:52: in _get_latest_ami_id
    Filters=[{"Name": "description", "Values": [image_prefix]}], Owners=["amazon"]
/usr/local/lib/python3.7/site-packages/botocore/client.py:530: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/local/lib/python3.7/site-packages/botocore/client.py:944: in _make_api_call
    operation_model, request_dict, request_context
/usr/local/lib/python3.7/site-packages/botocore/client.py:966: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
/usr/local/lib/python3.7/site-packages/botocore/endpoint.py:119: in make_request
    return self._send_request(request_dict, operation_model)
/usr/local/lib/python3.7/site-packages/botocore/endpoint.py:198: in _send_request
    request = self.create_request(request_dict, operation_model)
/usr/local/lib/python3.7/site-packages/botocore/endpoint.py:137: in create_request
    operation_name=operation_model.name,
/usr/local/lib/python3.7/site-packages/botocore/hooks.py:412: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
/usr/local/lib/python3.7/site-packages/botocore/hooks.py:256: in emit
    return self._emit(event_name, kwargs)
/usr/local/lib/python3.7/site-packages/botocore/hooks.py:239: in _emit
    response = handler(**kwargs)
/usr/local/lib/python3.7/site-packages/botocore/signers.py:105: in handler
    return self.sign(operation_name, request)
/usr/local/lib/python3.7/site-packages/botocore/signers.py:189: in sign
    auth.add_auth(request)
/usr/local/lib/python3.7/site-packages/botocore/auth.py:418: in add_auth
    raise NoCredentialsError()
E   botocore.exceptions.NoCredentialsError: Unable to locate credentials

@syedahsn syedahsn force-pushed the syedahsn/ec2-create-terminate-operators branch 2 times, most recently from 1190b8a to 41c1751 Compare March 7, 2023 17:52
syedahsn and others added 5 commits March 7, 2023 11:13
Change system test to use the new operators
Add unit tests for new operators
Change system test to terminate without stopping instances
…tiple instances

Add tests for creating/terminating multiple instances
Fix doc string on EC2TerminateInstanceOperator
@syedahsn syedahsn force-pushed the syedahsn/ec2-create-terminate-operators branch from 41c1751 to 1a4e537 Compare March 7, 2023 19:13
@josh-fell josh-fell merged commit d2cc9df into apache:main Mar 7, 2023
@vandonr-amz vandonr-amz deleted the syedahsn/ec2-create-terminate-operators branch May 24, 2023 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants