Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws-stepfunctions-tasks for starting Glue job run #5266

Closed
2 tasks
sarchila opened this issue Dec 1, 2019 · 0 comments · Fixed by #6258
Closed
2 tasks

aws-stepfunctions-tasks for starting Glue job run #5266

sarchila opened this issue Dec 1, 2019 · 0 comments · Fixed by #6258
Assignees
Labels
@aws-cdk/aws-stepfunctions Related to AWS StepFunctions effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on.

Comments

@sarchila
Copy link

sarchila commented Dec 1, 2019

As mentioned in https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html, one can trigger an AWS Glue job run and monitor its state synchronously from Step Functions using a Step Functions state, as shown in the example:

    "Glue StartJobRun": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "GlueJob-JTrRO5l98qMG"
      },
      "Next": "ValidateOutput"
    },

This request is for a feature, similar to @aws-cdk/aws-stepfunctions-tasks.RunLambdaTask, @aws-cdk/aws-stepfunctions-tasks.RunEcsFargateTask, etc, but for an AWS Glue job - perhaps called @aws-cdk_aws-stepfunctions-tasks.RunGlueJobTask

Use Case

For use in creation of Step Functions definitions with Glue job states.

Proposed Solution

Implementation would likely be very similar to https://github.com/aws/aws-cdk/blob/v1.18.0/packages/%40aws-cdk/aws-stepfunctions-tasks/lib/run-lambda-task.ts, but with tweaks to have the "Resource" arn always point to "arn:aws:states:::glue:startJobRun.sync" with the proposed @aws-cdk_aws-stepfunctions-tasks.RunGlueJobTask interface only requiring the Glue "JobName".

Other

  • 👋 I may be able to implement this feature request
  • ⚠️ This feature might incur a breaking change

This is a 🚀 Feature Request

@sarchila sarchila added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Dec 1, 2019
@SomayaB SomayaB added @aws-cdk/aws-glue Related to AWS Glue @aws-cdk/aws-stepfunctions Related to AWS StepFunctions labels Dec 2, 2019
@rix0rrr rix0rrr assigned nija-at and unassigned rix0rrr Jan 23, 2020
@SomayaB SomayaB added the in-progress This issue is being actively worked on. label Feb 12, 2020
@nija-at nija-at added effort/medium Medium work item – several days of effort and removed @aws-cdk/aws-glue Related to AWS Glue needs-triage This issue or PR still needs to be triaged. labels Feb 25, 2020
@mergify mergify bot closed this as completed in #6258 Mar 2, 2020
mergify bot added a commit that referenced this issue Mar 2, 2020
…6258)

* feat(stepfunctions-tasks): add step functions task to run glue job

* add new task RunGlueJobTask and associated unit tests
* since Job construct does not yet exist, uses job name as required parameter

closes #5266

* cleanup constructor properties, add integration test

* remove job run ID from props, update default prop descriptions

* add s3 assets package to module

* fix linting errors

* clean up documentation, add links to docs and glue task example

* add verification step to integration step, ensure job succeeds

* update expected integration test stack (asset names)

* add integ test verification comment about glue cold start

* cleaned up the note around cold start

* specify glue job ARN in state machine role permissions

* change state machine role permissions based on service integration pattern

Co-authored-by: Niranjan Jayakar <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
eladb pushed a commit that referenced this issue Mar 9, 2020
…6258)

* feat(stepfunctions-tasks): add step functions task to run glue job

* add new task RunGlueJobTask and associated unit tests
* since Job construct does not yet exist, uses job name as required parameter

closes #5266

* cleanup constructor properties, add integration test

* remove job run ID from props, update default prop descriptions

* add s3 assets package to module

* fix linting errors

* clean up documentation, add links to docs and glue task example

* add verification step to integration step, ensure job succeeds

* update expected integration test stack (asset names)

* add integ test verification comment about glue cold start

* cleaned up the note around cold start

* specify glue job ARN in state machine role permissions

* change state machine role permissions based on service integration pattern

Co-authored-by: Niranjan Jayakar <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-stepfunctions Related to AWS StepFunctions effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants