EP Merge: Fix health check / app startup based on health of rabbitmq and postgres connections

[dfitchett]

What was the problem?

This PR addresses the following:

• Application health check was not dependent on downstream services. Instead it was rather the availability of the application.
• Application startup is blocked by the task of starting connections to rabbitMQ.

Associated tickets or Slack threads:

Discovered while testing EP Merge App in dev:

• Test EP Merge App in dev #2449
  Related to:
• Update EP Merge App to address hanging health check #2498

How does this fix it?¹

• Allows the application to start up immediately and then proceeds to connect to rabbitMQ.
• Allows health check to respond instantly after checking the health of rabbitMQ and postgres.

How to test this PR

• pull code
• Unit Test:
  • from /domain-ee/ee-ep-merge-app run pytest
• Integration Tests:
  • run base platform
  • from /domain-ee/ee-ep-merge-app run pytest ./integration
• End to End tests:
  • run this CI action to verify successful integration test
  • run end to end tests locally:
    • run bgs and bip services and mocks:
      • COMPOSE_PROFILES="bip,bgs" ./gradlew :app:dockerComposeUp
      • COMPOSE_PROFILES="bip,bgs" ./gradlew -p mocks :dockerComposeUp
    • ./gradlew :domain-ee:ee-ep-merge-app:endToEndTest

Footnotes

1. Pull-Requests guidelines. If PR is significant, update Current Software State wiki page. ↩

[github-actions]

Test Results

148 tests  ±0   148 ✅ ±0   46s ⏱️ -1s
 39 suites ±0     0 💤 ±0 
 39 files   ±0     0 ❌ ±0 

Results for commit 11ead6c. ± Comparison against base commit d7f83a9.

♻️ This comment has been updated with latest results.

[github-actions]

JaCoCo Test Coverage

There is no coverage information present for the Files changed

Total Project Coverage	76.38%	❌

[raudabaugh]

Nice work on this! The unit and integration tests look great. However for some reason, I'm having trouble getting the E2E tests to pass locally. Example error:

ERROR root:ep_merge_machine.py:275 event=jobCompletedWithError trigger=resume_processing_from_running_cancel_ep400_claim job_id=2fb9cfb5-0004-4eea-8883-23db909e7b06 pending_claim_id=10000 ep400_claim_id=5002 state=COMPLETED_ERROR errorState=RUNNING_GET_PENDING_CLAIM_FAILED_REMOVE_SPECIAL_ISSUE error="["client=getClaimClient error='Unknown Downstream Error' status=500 status_message=INTERNAL_SERVER_ERROR", "client=getClaimContentionsClient error='Unknown Downstream Error' status=500 status_message=INTERNAL_SERVER_ERROR"]"

Any ideas?

[dfitchett]

error="["client=getClaimClient error='Unknown Downstream Error' status=500 status_message=INTERNAL_SERVER_ERROR", "client=getClaimContentionsClient error='Unknown Downstream Error' status=500 status_message=INTERNAL_SERVER_ERROR"]"
Hmmm.. I'm not sure what might cause that error! Do you have BIP/BGS and their mocks also running?

[raudabaugh]

Hmmm.. I'm not sure what might cause that error! Do you have BIP/BGS and their mocks also running?

Yes, after looking at the logs it seems that the issue is with the BIP_CLAIM_URL variable using HTTPS protocol which my local machine isn't set up with. Going to try changing to HTTP and see if that works!

[raudabaugh]

HTTPS>HTTP made things worse (integration tests failed) so I reverted that. Here's the error:

svc-bip-api logs

2024-01-29T22:48:04.355Z  INFO 1 --- [ntContainer#0-5] gov.va.vro.bip.service.BipApiService     : event=requestMade url=https://localhost:20300/claims/2/contentions method=GET
2024-01-29T22:48:04.356Z  INFO 1 --- [ntContainer#0-5] gov.va.vro.bip.service.BipApiService     : event=requestSent url=https://localhost:20300/claims/2/contentions method=GET
2024-01-29T22:48:04.358Z ERROR 1 --- [ntContainer#0-5] gov.va.vro.bip.service.BipApiService     : event=requestFailed url=https://localhost:20300/claims/2/contentions method=GET status=500 error=I/O error on GET request for "https://localhost:20300/claims/2/contentions": Connect to https://localhost:20300 [localhost/127.0.0.1] failed: Connection refused

mock-bip-claims-api logs

2024-01-29T20:50:48.889Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 20300 (https) 20306 (http) with context path ''
2024-01-29T20:50:49.332Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 20301 (http)
2024-01-29T20:50:49.334Z  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-01-29T20:50:49.335Z  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.18]
2024-01-29T20:50:49.412Z  INFO 1 --- [           main] o.a.c.c.C.[Tomcat-1].[localhost].[/]     : Initializing Spring embedded WebApplicationContext
2024-01-29T20:50:49.412Z  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 475 ms
2024-01-29T20:50:49.545Z  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2024-01-29T20:50:49.736Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 20301 (http) with context path ''
2024-01-29T20:50:49.821Z  INFO 1 --- [           main] g.v.v.m.MockBipClaimsApplication         : Started MockBipClaimsApplication in 36.923 seconds (process running for 42.356)
2024-01-29T20:51:08.691Z  INFO 1 --- [io-20301-exec-1] o.a.c.c.C.[Tomcat-1].[localhost].[/]     : Initializing Spring DispatcherServlet 'dispatcherServletRegistration'
2024-01-29T20:51:08.693Z  INFO 1 --- [io-20301-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServletRegistration'
2024-01-29T20:51:08.694Z  INFO 1 --- [io-20301-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms

[raudabaugh]

Good to go on my end. The issue with my environment was twofold: wrong value for BIP_CLAIM_URL (should be mock-bip-claims-api instead of localhost and obsolete docker images which a quick ./gradlew docker sorted it out. TY Derek for helping to troubleshoot in Slack chat