Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add rfc 007 static nonprod platform environment #159

Open
wants to merge 142 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
142 commits
Select commit Hold shift + click to select a range
5441610
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 17, 2024
2b85d46
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 17, 2024
5a01872
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 17, 2024
121937b
docs: update Terraform docs
github-actions[bot] Jun 17, 2024
43a6718
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 17, 2024
0c4b08b
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 17, 2024
175fdaa
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 18, 2024
46510c4
docs: update Terraform docs
github-actions[bot] Jun 18, 2024
9915316
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
fe4eeec
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
7248068
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
656c85a
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
4bfbf53
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
fd41436
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 20, 2024
a9a15d2
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
57f0bce
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
762e601
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 20, 2024
dfd518e
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
271aa4a
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
d565190
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
df89a0a
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
7eb23f7
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
211cc89
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
441eb6c
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 20, 2024
91925ee
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
6a2881f
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
8ddffe5
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 20, 2024
957da96
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
5b228a5
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
aea89ae
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
4604948
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
3b64b38
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
af3898e
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
85ab629
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 20, 2024
52b3b09
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
1b61aef
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
17ccd2a
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
9f4acf7
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
19ec94c
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
00d924d
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
a0c4a1c
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 20, 2024
8a98f58
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
1450a6f
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
7064cea
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
65539d8
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 20, 2024
0bc3997
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
9f350c8
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
6b89685
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
7a108ba
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
adbafab
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
d768aa1
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
6a80885
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
fc0c3ad
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
1caf3d4
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
4c0f074
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 20, 2024
b8de449
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
b814eb5
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
13ef98f
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
636013b
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 20, 2024
e188c3a
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
aee2f98
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
f522bb7
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
685e7fd
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 20, 2024
d0e0e53
docs: update Terraform docs
github-actions[bot] Jun 20, 2024
a2ea5f3
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 21, 2024
9c6c986
docs: update Terraform docs
github-actions[bot] Jun 21, 2024
393a9b2
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 21, 2024
64164e7
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 21, 2024
7fd73cb
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 21, 2024
4aded81
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 21, 2024
3097968
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 21, 2024
bae9162
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 21, 2024
7ca21c4
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 21, 2024
c05391e
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 25, 2024
af27d08
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 25, 2024
5f980d6
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 26, 2024
924aeeb
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 26, 2024
1ebfa31
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 26, 2024
38da013
docs: update Terraform docs
github-actions[bot] Jun 26, 2024
9e62e49
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 27, 2024
1579ccd
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jun 27, 2024
fe2ab3c
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 27, 2024
5323fc0
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 27, 2024
ad534d7
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 27, 2024
bc0060f
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 27, 2024
bd012b8
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 27, 2024
226f8a3
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 27, 2024
601b65c
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 27, 2024
b364b1a
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 28, 2024
47532ef
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 28, 2024
acce8aa
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 28, 2024
1e21979
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 28, 2024
b828631
feat(terraform): vol5504 add single batch job
cmarstondvsa Jun 28, 2024
9980da3
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 1, 2024
45018e2
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
0a59e25
docs: update Terraform docs
github-actions[bot] Jul 2, 2024
a6f7827
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
274b7ee
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jul 2, 2024
9954bd6
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
78005b6
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
f741337
docs: update Terraform docs
github-actions[bot] Jul 2, 2024
9677748
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
4a0d43b
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jul 2, 2024
510426a
docs: update Terraform docs
github-actions[bot] Jul 2, 2024
5e145d1
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
ff45e36
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jul 2, 2024
15319e6
docs: update Terraform docs
github-actions[bot] Jul 2, 2024
aba78ed
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
87b1a8f
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jul 2, 2024
4b896b6
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
f5f29a3
docs: update Terraform docs
github-actions[bot] Jul 2, 2024
cc52b05
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
aec40c3
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jul 2, 2024
598829a
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
87b0ff0
docs: update Terraform docs
github-actions[bot] Jul 2, 2024
a13f54d
formatting altered TF files using tf fmt
barkerl Jul 2, 2024
c27ac35
docs: update Terraform docs
github-actions[bot] Jul 2, 2024
03306c4
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
ccf3fd8
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
0279171
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
9ed82c5
docs: update Terraform docs
github-actions[bot] Jul 2, 2024
29f099b
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 2, 2024
805d28c
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jul 2, 2024
e27b2c1
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 3, 2024
02e279c
docs: update Terraform docs
github-actions[bot] Jul 3, 2024
23dbe3f
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 3, 2024
ec40a06
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jul 3, 2024
3acc511
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 3, 2024
1718150
docs: update Terraform docs
github-actions[bot] Jul 3, 2024
e117f0c
addressing terraform fmt errors
barkerl Jul 3, 2024
39b0301
docs: update Terraform docs
github-actions[bot] Jul 3, 2024
08949d2
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 3, 2024
7f17f6f
Merge branch 'vol5504-batch-cli-module' of https://github.com/dvsa/vo…
cmarstondvsa Jul 3, 2024
60d6b0d
feat(terraform): vol5504 add single batch job
cmarstondvsa Jul 3, 2024
219bbe0
Merge remote-tracking branch 'origin/main' into HEAD
barkerl Jul 3, 2024
3130922
add-rfc-007-initial-commit
barkerl Jul 3, 2024
a0a40c1
fixing typo
barkerl Jul 4, 2024
b464042
rebased
barkerl Jul 5, 2024
8a3cc82
docs: update Terraform docs
github-actions[bot] Jul 5, 2024
96de3fb
missed empty lines
barkerl Jul 5, 2024
165315d
Update docs/rfc/rfc-007-static-nonprod-platform-environment.md
barkerl Jul 10, 2024
a32455c
Update docs/rfc/rfc-007-static-nonprod-platform-environment.md
barkerl Jul 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions docs/rfc/rfc-007-static-nonprod-platform-environment.md
Copy link
Contributor

@JoshuaLicense JoshuaLicense Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I think the theory behind this is logical and makes sense. I believe there are more appropriate alternatives to solve this problem without adding what I believe is complexity to the code and development process.

The dev environment is a playground for the team, the team includes developers and platform engineers.

Any breaking change that can't be fixed within a certain amount of time (which could vary, but we're the same team so can be raised) should be reverted, git and GitHub give us this functionality with a command/click. Once reverted, the environment will be back working.

Feedback:

  • The proposed environment is not typical for a continuous deployment strategy. Not that this is a particular problem, but we need to add documentation for its proposed use case.
  • Large platform changes are infrequent and would mean the environment is sitting around and not being utilised. While the hosting cost may be low, this is still an environment that needs to be maintained. An extra environment adds ~25% extra maintenance (4 -> 5 environments).
  • I believe the ephemeral environment aligns with this use case more appropriately. The plan is to add functionality to deploy branches, wouldn't add extra maintenance and would be even lower cost.
  • The only platform changes this will be appropriate for are changes just in the vol-terraform repository, as the vol-app will always be applying main to this environment and using main app version. This increases the cognitive load on a platform engineer while making changes in VOL as the test process will change depending on the nature of the change.
    • To test changes in vol-app the team would need to test in dev.
    • To test changes in vol-app and vol-terraform the team would need to test in dev.
    • To test changes in vol-terraform the team would need to test in the newly proposed environment.

All said, I'm not a platform engineer so if the above doesn't persuade then I'll not stand in the way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my thoughts on this are that there is not foolproof pattern for continuous delivery and i think we have to understand what works for the service (otherwise there would be one pattern and tool set that everyone would use and that certainly isn't the case). The main platform changes going forward will be in the vol-terraform repo, as per rfc-005. To test changes to vol-terraform there will need to be a pipeline for that anyway as it isn't covered in the existing ones.

the test changes aren't quite correct, only if a change was made purely in the app component would we test in Dev. Anything else we would need to test in the new environment and then in dev/ephemeral.

Fundamental changes to the foundational elements of the platform (networks etc) that are changed semi regularly are potentially across all environments. This isn't about the effort but rather what these resources support and that is the important factor here. This is why the current ephemeral environment design may not have that use case, i would rather not have a pattern of applying and potentially breaking small well defined changes with the containers proving the case for a higher release cadence.

If we find that with the rationalisation that is continuing in VOL we are in a place where some of the concerns are not longer valid I will be the first to support tearing it down.

Copy link
Contributor

@JoshuaLicense JoshuaLicense Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the test changes aren't quite correct, only if a change was made purely in the app component would we test in Dev. Anything else we would need to test in the new environment and then in dev/ephemeral.

If you make a change in the vol-app and vol-terraform, for the Terraform or app changes in vol-app to take effect (applied) they will need to be merged and in dev first for it to make it to the new environment. Without merging changes in vol-app they will not be deployed anywhere.

If we find that with the rationalisation that is continuing in VOL we are in a place where some of the concerns are not longer valid I will be the first to support tearing it down.

As nothing is in place yet this is anticipating a theoretical problem in a theoretical solution, would a better way be to wait until we try the existing planned solutions (dev/ephemeral) to identify a practical problem and then iterate upon the solution we have? Rather than fixing the anticipated problem, and then tearing it down once (and if) it's proved to not be a problem?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As nothing is in place yet this is anticipating a theoretical problem in a theoretical solution, would a better way be to wait until we try the existing planned solutions (dev/ephemeral) to identify a practical problem and then iterate upon the solution we have? Rather than fixing the anticipated problem, and then tearing it down once (and if) it's proved to not be a problem?<

Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# RFC-007: Static non-prod platform environments

## Summary

This RFC proposes having a static platform environment (possibly QA) to be used by the infrastructure release pipeline. This is to provide an alternative route for any substantial infrastructure changes to attempt to prevent breaking or impeding the container path to live while being proved.

## Problems

### Application container release pipeline

As per [rfc-005](./rfc-005-add-terraform-to-mono-repository.md) the scope of terraform code deployed to the application repo was limited to prevent further Terraform dependency complexity. As the time of writing this has not been superseded by any future request for comment that amends this stance.

The existing pipeline fundamentally guides the deployment of containers to ECS clusters that sit upon the entirety of the VOL cloud infrastructure. Any change to just the container image (which as per current understanding will be a significant amount of the change requests going forward) will be efficient as per design. This means that container change will be ready for rapid promotion through the remaining three environment leading to decrease in release cadence.

### Platform based changes and potential scope for disruption

Infrastructure change will in most cases be broader and may involve a greater degree of proving. As such any breaking change will break the path to live for all other change being promoted to production. This has the potential to impede the progress we have made towards decreasing the release cadence.

An additional static environment using containers will be cheap and will give the longer riskier changes a pathway to live, ensuring proving on the container based solution removing some of the risk of impeding container deployment and creating feature bottlenecks.

## Proposal

During the CI/CD workflow, once a change is merged to main we will deploy to an additional static non-prod environment in parallel to Int after proving in Dev.

This will improve our deployment resilience when working on long-running potentially breaking changes while still allowing us to test on the existing containerised solution (currently this is only deployed to one non-prod environment).

If this does not achieve the stated aim of testing long-running cloud changes on a representative environment then we will remove and apply directly to the Dev environment as per the container pipeline. This is accepting in the vol-terraform pipelines and accept the potential for blocking the promotion route to live for all changes. As part of this we should also monitor the usage.

## Other options considered

### Deploying changes directly to Dev for any functional testing

This would allow us to test any functional capability along with the infrastructure change. It comes at the risk of delaying container change and preventing value being delivered as early as it could be from a delivery perspective. Given the reduced cost and complexity this container design is going to provide us it doesn't seem worth it.

### Deploying to ephemeral environments

Ephemeral environments won't allow us to effectively test the scope of a platform change in a single ephemeral environment. A network change (security group, Nacl etc.) break all ephemeral environments impeding all other container based change.