[feature] DevOps/GitHub Actions Release Pipeline Guidance

[klawrawkz]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

There are sporadic mentions of benefits of the CAF rover in DevOps, but there is no guidance on configuration of such in Azure DevOps or in GitHub via "actions."

Describe the solution you'd like
A clear and concise description of what you want to happen.

I'm more than happy to contribute to this effort, as I believe this project has the potential to dramatically improve our ability to produce, release, maintain standards-based IaC. Perhaps I can be involved in the capacity of guinea pig, and apply instructions and guidance in DevOps implementation approach to validate the concept. I can create accompanying documentation for other developers/implementers of the project. I'll gladly produce the technical documentation for publication here in return for assistance with the DevOps pipelines.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

I've considered "hacking it together" and getting it to run in my DevOps environment. I think going through the process, and documenting the approach for the general public is a better more valuable option.

Additional context
Add any other context or screenshots about the feature request here.

[arnaudlh]

Good feedback @klawrawkz we will add this for 2004 release.

[Masahigo]

@arnaudlh Hi, I was testing running the Rover image from Azure DevOps container job using a Hosted agent. I was basically trying to mimic what works in Github action example: https://github.com/aztfmod/rover/blob/master/.github/workflows/ci-branches.yml#L63

This is what my azure-pipelines.yml looks like so far

variables:
  location: 'westeurope'

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

container: aztfmod/rover:2004.1606

steps:
- script: |
    az login --service-principal -u '$(ARM_CLIENT_ID)' -p '$(ARM_CLIENT_SECRET)' --tenant '$(ARM_TENANT_ID)'
    az account set -s  $(ARM_SUBSCRIPTION_ID)
    echo "local user: $(whoami)"
  displayName: 'Login to Azure'

The hosted agent OS (vm image) should be same as in Github action example (ubuntu-18.04).

However, the pipeline fails already in container initialization phase:

Are you planning to support ADO Hosted agents (container jobs) in the future or is it possible to get it working somehow like this? Was hoping to survive without self-hosted agents at this point but would be valuable to know if they're the only way to go if using Azure DevOps for CI/CD.

[arnaudlh]

hi folks, we are going to publish guidance on that in a couple of days, with a brand new version of rover to support those scenarios. Bear with us :)

[Masahigo]

Are you planning to support ADO Hosted agents (container jobs) in the future or is it possible to get it working somehow like this?

@arnaudlh Seems to work after all, was just missing this additional options from the container job

container:
  image: aztfmod/rover:2004.1606
  options: --user 0

But.. there is another issue, related to the Docker user in ADO container job: vsts_azpcontainer

I added another step for testing launchpad plan command

- script: |
    id=$(az storage account list --query "[?tags.tfstate=='level0']" -o json | jq -r .[0].id)
      if [ "${id}" == "null" ]; then
        /tf/rover/launchpad.sh /tf/launchpads/launchpad_opensource_light plan -var location='$(location)'
    fi
  displayName: 'Locate launchpad (light)'

The step fails with .. rm: cannot remove .. Permission denied.

2020-04-28T13:27:33.8588503Z ##[section]Starting: Locate launchpad (light)
2020-04-28T13:27:33.8597381Z ==============================================================================
2020-04-28T13:27:33.8597709Z Task         : Command line
2020-04-28T13:27:33.8598041Z Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2020-04-28T13:27:33.8598421Z Version      : 2.164.0
2020-04-28T13:27:33.8598655Z Author       : Microsoft Corporation
2020-04-28T13:27:33.8599043Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2020-04-28T13:27:33.8599427Z ==============================================================================
2020-04-28T13:27:34.1923422Z Generating script.
2020-04-28T13:27:34.1939829Z ========================== Starting Command Output ===========================
2020-04-28T13:27:34.1973142Z [command]/usr/bin/bash --noprofile --norc /__w/_temp/3587e928-28ca-4c43-a51b-b70e8bd98e40.sh
2020-04-28T13:27:35.7170604Z else -var
2020-04-28T13:27:35.7171033Z else location=westeurope
2020-04-28T13:27:35.7188092Z Launchpad management tool started with:
2020-04-28T13:27:35.7193303Z   tf_action   is : 'plan'
2020-04-28T13:27:35.7199101Z   tf_command  is : '-var location=westeurope'
2020-04-28T13:27:35.7206616Z   landingzone is : '/tf/launchpads/launchpad_opensource_light'
2020-04-28T13:27:35.7215550Z   workspace   is : 'level0'
2020-04-28T13:27:35.7219508Z 
2020-04-28T13:27:35.7239357Z Checking existing Azure session
2020-04-28T13:27:38.0545430Z No launchpad found.
2020-04-28T13:27:38.0551757Z rm: cannot remove '/home/vscode/.terraform.cache/tfstates/level0': Permission denied
2020-04-28T13:27:38.0568095Z �[41mError on or near line 58; exiting with status 1�[0m
2020-04-28T13:27:38.0586311Z 
2020-04-28T13:27:38.0586951Z cleanup variables
2020-04-28T13:27:38.0595722Z 
2020-04-28T13:27:38.0669912Z ##[error]Bash exited with code '1'.
2020-04-28T13:27:38.0989512Z ##[section]Finishing: Locate launchpad (light)

I noticed that in Github actions this user is root so this differs in ADO.

Possible solution could be to include sudo in the Rover image:
https://developercommunityapi.westus.cloudapp.azure.com/idea/780246/docker-container-root-user.html

[Masahigo]

I managed to get around this limitation with ADO hosted agents, by installing sudo to the Rover container in following way

..
container:
  image: aztfmod/rover:2004.1606
  options: --user 0 --name rover-container -v /usr/bin/docker:/tmp/docker:ro
..

steps:
- script: |
    /tmp/docker exec -t -u 0 rover-container \
    sh -c "yum install -y sudo"
  displayName: 'Set up sudo'
- script: |
    ls -la
    sudo chmod -R 777 /home/vscode
    sudo chmod -R 777 /tf/launchpads
  displayName: 'File permissions'
..

I had to elevate some file permissions this way. This is MVP but worked for me.

[arnaudlh]

hi @Masahigo and @klawrawkz, added basic guidance on ADO and GHA (check at the commits) will be ready for next merge. dont hesitate to further comment and contribute on that area. In next release, will come up with some more doc and automation for pipelines creation. Let us know any feedback :)

[Masahigo]

@arnaudlh I did a small example as well:
https://github.com/Masahigo/caf-terraform-landingzone-example

[Masahigo]

hi @Masahigo and @klawrawkz, added basic guidance on ADO and GHA (check at the commits) will be ready for next merge. dont hesitate to further comment and contribute on that area. In next release, will come up with some more doc and automation for pipelines creation. Let us know any feedback :)

I was missing proper instructions for creating the Service Principal to be used via CI/CD. It's not enough for the SP to have Contributor role on the subscription level, it also requires permissions to create new SPs (ie it needs the AAD Application Developer role) and some other RBAC permissions - for which I created another custom role:

https://github.com/Masahigo/caf-terraform-landingzone-example/tree/master/pipelines#create-the-service-principal-for-cicd

[carthikannayan]

Hi @arnaudlh May I please request you to provide a starter CI File/ Yaml file to run the newer version of the Landing zones? Any pointers are much appreciated. I m not able to go past the login in the pipeline