Pair to complete BIP Rollout for Prod

[nelsestu]

This Issue is a continuation/completion of #2242

User Story

As a consumer of the BIP API, I want to ensure that the svc-bip-api container can be built and deployed using accurate configuration options. After stabilizing the BIP integration, we discovered additional ops related issues in #2242. Compounded by unexpected sick time drawing down the team's capacity, we were only able to get Dev deployed in sprint I. This issue will continue the deployment process, seeing the completion of rollout to the rest of VRO environments in LHDI.

• Complete a new end-to-end test that will help validate the BIP integration as we roll it out to additional environments. This script will help drive a new directive focused on more thorough testing practices.
• Prepare environments for incoming BIP deployments by ensuring that the secrets configured in the environments are accurate and functioning properly
• Confirm the current state of BIP, resolving the mutual tls issue that it is presently experiencing
  • with some brief evaluation, we can see that the deployed svc-bip-api is in a CrashLoopBackOff state
  • we know it is somehow related to the mutual TLS handshake
  • we know that the BIP team already have a public key, so we need to identify the certificate, private key and password that will generate a valid ssl cert to use on the pod.
• Validate our cross environment API mapping

Before this ticket can start
#2298 will ensure the following dev validation is completed

• svc-bip-api is deployable and gets deployed for all VRO LHDI environments:
  • End to end test script is functional in DEV, and used to validate future deployments.
  • Redeploy certs/passwords as Verify BIP service secrets in dev environment #2311 indicates
  • validate the BIP connection with some real live BIP calls
  • The svc-bip-api pod should remain in a Running state.
  • Notes from this process in dev have been consolidated and shared here

Acceptance Criteria
Note Please pair with another engineer on this effort

1. After validating dev, and confirming its viable state there, we'll plan to repeat the steps from AC1 for all other non-prod environments including:
   • QA Deployment - namespace: va-abd-rrd-qa in nonprod context
   • Sandbox Deployment - namespace: va-abd-rrd-sandbox in nonprod context
   • ProdTest Deployment - namespace: va-abd-rrd-prodtest in nonprod context
2. Document secrets and how they are set up

Additional Validation Work

This work item includes an additional validation effort that was started in Sprint I. Erik is proposing that we raise the bar on the validation required to deploy a new service. Absence of errors is not sufficient validation and despite that BIP is not a new service, we haven't had validation scripts capable of demonstrating success in LHDI environments. We are merely representing the validation effort that has been lacking from past deployments, and we do hope to walk away with higher confidence and n't been done in the past.

Notes about work

• (links to documentation, resources, code, etc that might inform the person doing the work)

[nelsestu]

As of Jan 29 2024 the prod cluster deployment was in progress throughout the day. As it turns out, we don't have the JWT signing secret for the prod-test BIP environment, thus the bip deployment in prod-test is not working. Prod deployment on the other hand has been successfully deployed and validated.

In order to complete this issue, we will likely need to revert the config change that was made to application.prod-test.yaml and reuse one of the BIP environments that we do have signing secrets for.

[nelsestu]

By thursday evening we had a new release version which included the config change required in prod-test, so Friday morning I promoted that to prod-test. With all environments deployed this is now complete.