Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor: Separate Pipeline Management into Serverless Application #424

Merged
merged 29 commits into from
Aug 26, 2022
Merged

Refactor: Separate Pipeline Management into Serverless Application #424

merged 29 commits into from
Aug 26, 2022

Conversation

StewartW
Copy link
Contributor

@StewartW StewartW commented Jan 10, 2022 •
edited
Loading

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@StewartW StewartW force-pushed the refactor/pipeline-management branch from 26a70ce to 26f2819 Compare April 12, 2022 10:10
@StewartW
Refactor: Pipeline Management State Machine
305ca55 
Seperate out Pipeline Mananagement away from CodePipeline and remove
the major bottlenecks of building each deployment map sequentially.

Starts to isolate the CodeCOmmit repository and moves the source of
deployment maps into an S3 Bucket.

Sets the groundwork for future refactoring of Pipeline Management and
moves towards enabling decentralised deployment maps.
@StewartW StewartW force-pushed the refactor/pipeline-management branch from 26f2819 to 305ca55 Compare April 12, 2022 11:27
@StewartW StewartW changed the title Refactor pipeline management Refactor: Separate Pipeline Management into Serverless Application Apr 12, 2022
@StewartW StewartW marked this pull request as ready for review April 25, 2022 14:05
@StewartW StewartW requested review from sbkok and dsudduth April 25, 2022 14:06
@StewartW StewartW added the enhancement New feature or request label May 20, 2022
sbkok
sbkok requested changes Aug 16, 2022
View reviewed changes
Copy link
Collaborator

@sbkok sbkok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for refactoring this logic! Very valuable!

I noticed a few changes were missing in the logic. For example wave generation and another PR that I linked in the review comments.
Could you make sure the changes introduced by those PRs are also included in this PR?

docs/images/technical-diagrams.drawio Outdated Show resolved Hide resolved
docs/technical-guide.md Outdated Show resolved Hide resolved
docs/technical-guide.md Outdated Show resolved Hide resolved
docs/technical-guide.md Outdated Show resolved Hide resolved
...repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/create_repository.py Outdated Show resolved Hide resolved
...ambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/deployment_map.py Outdated Show resolved Hide resolved
src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/repo.py Outdated Show resolved Hide resolved
src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/rule.py Outdated Show resolved Hide resolved
src/template.yml Outdated Show resolved Hide resolved
src/template.yml Outdated Show resolved Hide resolved
@StewartW StewartW force-pushed the refactor/pipeline-management branch 2 times, most recently from 5f27224 to 17767af Compare August 18, 2022 08:40
@StewartW StewartW force-pushed the refactor/pipeline-management branch from 17767af to 29a4fee Compare August 18, 2022 09:33
@sbkok sbkok mentioned this pull request Aug 19, 2022
Closed
sbkok
sbkok requested changes Aug 25, 2022
View reviewed changes
Copy link
Collaborator

@sbkok sbkok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvements, thanks for making those.
I've got a number of minor suggested changes, so we are close to merge this too!

...repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/create_repository.py Outdated Show resolved Hide resolved
...repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/create_repository.py Outdated Show resolved Hide resolved
...ory/adf-bootstrap/deployment/lambda_codebase/pipeline_management/generate_pipeline_inputs.py Outdated Show resolved Hide resolved
...f-bootstrap/deployment/lambda_codebase/pipeline_management/identify_out_of_date_pipelines.py Outdated Show resolved Hide resolved
...f-bootstrap/deployment/lambda_codebase/pipeline_management/identify_out_of_date_pipelines.py Outdated Show resolved Hide resolved
...odebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/pipeline_management.yml Outdated Show resolved Hide resolved
...odebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/pipeline_management.yml Outdated Show resolved Hide resolved
...odebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/pipeline_management.yml Outdated Show resolved Hide resolved
...ambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/deployment_map.py Outdated Show resolved Hide resolved
...ase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_deployment_map.py Show resolved Hide resolved
@StewartW @sbkok
Updating Function names
612855e 
Adding ADF Prefix to make it clearer that they are part of the ADF ecosystem

Co-authored-by: Simon Kok <[email protected]>
sbkok
sbkok approved these changes Aug 26, 2022
View reviewed changes
Copy link
Collaborator

@sbkok sbkok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making all these changes @StewartW!
This will help speed up ADF a lot, excellent work!

@sbkok sbkok requested a review from javydekoning August 26, 2022 14:58
javydekoning
javydekoning approved these changes Aug 26, 2022
View reviewed changes
Copy link
Contributor

@javydekoning javydekoning left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me!

I suggest also storing the .drawio originals to the repo so the diagram can be easily updated when there are changes to the statemachine later.

@sbkok
Copy link
Collaborator

sbkok commented Aug 26, 2022

For reference, the .drawio.png files that were added by the PR also include the source of the image.
You can edit them by opening the png file directly with the DrawIO tool.

@sbkok sbkok merged commit 0032fab into awslabs:master Aug 26, 2022
@javydekoning javydekoning mentioned this pull request May 15, 2023
Merged
pergardebrink added a commit to pergardebrink/aws-deployment-framework that referenced this pull request May 15, 2023
@pergardebrink
Remove cdk package as it's no longer used
20f0395 
According to @javydekoning, it's probably left over from awslabs#424
sbkok added a commit that referenced this pull request Jun 1, 2023
@pergardebrink @sbkok @javydekoning
Update CDK from v1 to v2 (#619)
b32326c 
* Update ADF pipeline generation to CDKv2

* Add trailing commas to imports

Co-authored-by: Simon Kok <[email protected]>

* Apply suggestions from code review

* Fix merge

* Upgrade to latest CDK + SAM CLI, Python 3.10 (Py3.11 in CodeBuild)

**Why?**

As this commit will update the new default CodeBuild container image to use,
we will pick the most recent Standard CodeBuild image that is available at the
time of this commit. Which is STANDARD_7_0.

This allows us to continue to support the version for the foreseeable future.
As we rather not need to change this frequently, as that would introduce a
breaking change.

* Remove cdk package as it's no longer used

According to @javydekoning, it's probably left over from #424

* Use aws-cdk package instead of cdk

---------

Co-authored-by: Simon Kok <[email protected]>
Co-authored-by: Javy de Koning <[email protected]>
Co-authored-by: Simon Kok <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@javydekoning javydekoning javydekoning approved these changes

@sbkok sbkok sbkok approved these changes

@dsudduth dsudduth Awaiting requested review from dsudduth

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@StewartW @sbkok @javydekoning