ADR - Infrastructure-as-Code Sharing Approach #104
Conversation
There was a problem hiding this comment.
It feels like this is a good point to dovetail into the work @idg10 and @jongeorge1 are doing on "what is a Marain service".
And this may have implications for what our goals are in various of these areas.
- tenancy
- management plane
- onboarding
- provisioning
- configuration
- identity
- operations
- observability
- notifications
- health
|
|
||
| Given this context and goals, the challenge is how to combine the use of shared Bicep modules (to de-duplicate the infrastructure-as-code) whilst still having a low friction means of deploying the Marain stack. The absence of a defacto public Bicep module repository that can host the Marain-specific modules means there isn't an obvious way to share these modules without additional work. | ||
|
|
||
| ***NOTE**: A more general, but related, Bicep reuse [ADR](https://github.com/endjin/Endjin.RecommendedPractices.Bicep/blob/main/docs/adr/0002-sharing-bicep-modules.md) was written before it became clear that the public Bicep registry would not be 'open to all' in the same way that nuget.org or Docker Hub are.* |
There was a problem hiding this comment.
Minor editing note: GitHub has interpretted the lone '*' at the end of this line as meaning that you wanted the whole of the rest of the document to be in bold!
There was a problem hiding this comment.
That's odd, it's not showing like that for me in GH, is the paragraph italicised for you?
|
|
||
| ### Hosting a public Bicep registry | ||
|
|
||
| This option involves us hosting an Azure Container Registry that we make publicly available and publishing the Marain Bicep modules to it as part of the overall Marain release process. |
There was a problem hiding this comment.
Is this an officially supported use of an ACR?
I don't know what Bicep's relationship with a registry looks like. Here are some things I don't know:
- is Bicep designed with the intention that it will be used with a mixture of "the public Bicep registry" referred to earlier and other registries (such as a private one, or a non-Microsoft public one)? E.g., does its configuration fully support an idea of "here are all the registries we're using" or do you end up doing something more hacky like pointing it to a registry which somehow defers back to the public registry (e.g., as you can do with Azure DevOps 'artifact feeds')?
- is it just making simple HTTP requests based on a presumed URL structure? (E.g., could you in principle use a blob store as a registry?) Or are there particular things it requires of a repository?
- what are the expectations around auth when the bicep tools talk to a registry? might these change over time?
In particular, I'm thinking about the fact that when NuGet first launched, it really did treat the repo as just a file store, so you could use pretty much any web server as a package server, but that's not exactly true any more.
So I'm wondering: does an ACR work as a Bicep store just because it happens to resemble a generic static HTTP service enough to keep bicep happy? (Bicep is not Docker. A Bicep module is not a container image. So it's not obvious to me that a container registry should necessarily work as a Bicep registry.) And if so, might Bicep's expectations change in the future in a way that means this won't always be true.
There was a problem hiding this comment.
Good questions:
- Bicep modules use the OCI package format, so a container registry is the intended store for them (Proposal: Distribute Bicep providers using OCI registry Azure/bicep#10662)
- The current Bicep tooling only supports ACR (i.e. you can't use GitHub or Docker registries), due to the way that it leans on the Azure CLI for authentication (Support OCI-compliant alternative container registry for bicep publish Azure/bicep#4884)
- You can refer to multiple registries when consuming modules as each reference has to contain the FQDN (e.g.
br:endjin.azurecr.io/bicep/general/mymodule:1.0.0) - You can create aliases for these registry references, which gives the flexibility to retarget them if necessary (e.g.
br/marain/bicep/general/mymodule:1.0.0)
| #### Advantages | ||
| * Simplest and lowest friction option for the Marain consumer. | ||
| * Easy to integrate with our existing release process. | ||
| * Some flexibility for where the source for the shared Bicep modules is stored, as they won't be primarily consumed as 'source' artefacts. |
There was a problem hiding this comment.
they won't be primarily consumed as 'source' artefacts.
I don't understand what this means
There was a problem hiding this comment.
I've pushed-up some changes to, hopefully, better describe this
| * The Marain deployment process becomes dependent on network connectivity to the public registry - although this is arguably no different to the current requirement for access to GitHub (needed to download release artefacts). | ||
|
|
||
|
|
||
| ### Supporting internal Bicep registries |
There was a problem hiding this comment.
Has any thought been given to the kinds of deployment topologies in which everything that can be locked off on a restrictive VNET has been?
There at least are a couple of dimensions to that:
- bicep registries accessible only via a VNET
- deployment mechanisms running on agents that are on a VNET in which outbound requests are restricted to prevent exfiltration attacks
I don't know enough about how multi-module bicep deployments work to know what's viable here. (I believe that with nested classic ARM deployments, Azure needs to be able to access the storage account containing the child templates, which imposes some limitations on the extent to which you can lock things down. I don't know if that also applies to Bicep; I believe Bicep has a mode in which you can pre-compile to an ARM template, so I can imagine it might be possible to run tools on a build agent does have access to a locked-down Bicep store and which fetches all the modules and builds them into one huge monolithic ARM template, which would mean Azure itself wouldn't need access to the stores containing the modules. But while that seems conceivable, I have no idea whether you can actually do it in practice.)
There was a problem hiding this comment.
In either scenario it's always the system invoking the deployment that needs to have access to any module registries - the ARM runtime still only deals with JSON-based templates.
As per your suggestion, what we typically do now is generate the ARM template as part of the build process and then treat it as an artefact to feed into the deployment.
As we migrate the Marain deployment tooling to use Bicep and introduce re-use of shared modules, we need to decide how we balance the desire to re-use and reduce duplication with the need for providing Marain consumers with a low-friction deployment experience.
This ADR describes the challenges and compares possible solutions.