#2530 Enable SASL for bie kafka #2624

agile-josiah · 2024-02-15T22:11:05Z

What was the problem?

The VRO Certs to access Kafka are expiring and we need to decide whether VRO will provision their own Certs and provide the Public Key to BIP Solutions to renew the ACL access or attempt to complete the RBAC migration making the Certs a non-issue.

Associated tickets or Slack threads:

How does this fix it?¹

This enables SASL_SSL for bie-kafka comms

How to test this PR

run source ./scripts/kafka-service.sh
Watch the connection between mock-bie-kafka and svc-bie-kafka

Pull-Requests guidelines. If PR is significant, update Current Software State wiki page. ↩

github-actions · 2024-02-15T22:16:58Z

Test Results

151 tests ±0 151 ✅ ±0 46s ⏱️ -1s
46 suites ±0 0 💤 ±0
46 files ±0 0 ❌ ±0

Results for commit d8d2eea. ± Comparison against base commit e2fcd53.

This pull request removes 10 and adds 10 tests. Note that renamed tests count towards both.

gov.va.vro.services.bie.config.ContentionEventTest ‑ [1] inputTopic=TST_CONTENTION_BIE_CONTENTION_ASSOCIATED_TO_CLAIM_V02, bieChannel=bie-events-contention-associated-to-claim
gov.va.vro.services.bie.config.ContentionEventTest ‑ [1] inputTopic=TST_CONTENTION_BIE_CONTENTION_ASSOCIATED_TO_CLAIM_V02, expectedEvent=CONTENTION_ASSOCIATED_TO_CLAIM
gov.va.vro.services.bie.config.ContentionEventTest ‑ [2] inputTopic=TST_CONTENTION_BIE_CONTENTION_UPDATED_V02, bieChannel=bie-events-contention-updated
gov.va.vro.services.bie.config.ContentionEventTest ‑ [2] inputTopic=TST_CONTENTION_BIE_CONTENTION_UPDATED_V02, expectedEvent=CONTENTION_UPDATED
gov.va.vro.services.bie.config.ContentionEventTest ‑ [3] inputTopic=TST_CONTENTION_BIE_CONTENTION_CLASSIFIED_V02, bieChannel=bie-events-contention-classified
gov.va.vro.services.bie.config.ContentionEventTest ‑ [3] inputTopic=TST_CONTENTION_BIE_CONTENTION_CLASSIFIED_V02, expectedEvent=CONTENTION_CLASSIFIED
gov.va.vro.services.bie.config.ContentionEventTest ‑ [4] inputTopic=TST_CONTENTION_BIE_CONTENTION_COMPLETED_V02, bieChannel=bie-events-contention-completed
gov.va.vro.services.bie.config.ContentionEventTest ‑ [4] inputTopic=TST_CONTENTION_BIE_CONTENTION_COMPLETED_V02, expectedEvent=CONTENTION_COMPLETED
gov.va.vro.services.bie.config.ContentionEventTest ‑ [5] inputTopic=TST_CONTENTION_BIE_CONTENTION_DELETED_V02, bieChannel=bie-events-contention-deleted
gov.va.vro.services.bie.config.ContentionEventTest ‑ [5] inputTopic=TST_CONTENTION_BIE_CONTENTION_DELETED_V02, expectedEvent=CONTENTION_DELETED

gov.va.vro.services.bie.config.ContentionEventTest ‑ [1] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_ASSOCIATED_TO_CLAIM_V02, bieChannel=bie-events-contention-associated-to-claim
gov.va.vro.services.bie.config.ContentionEventTest ‑ [1] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_ASSOCIATED_TO_CLAIM_V02, expectedEvent=CONTENTION_ASSOCIATED_TO_CLAIM
gov.va.vro.services.bie.config.ContentionEventTest ‑ [2] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_UPDATED_V02, bieChannel=bie-events-contention-updated
gov.va.vro.services.bie.config.ContentionEventTest ‑ [2] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_UPDATED_V02, expectedEvent=CONTENTION_UPDATED
gov.va.vro.services.bie.config.ContentionEventTest ‑ [3] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_CLASSIFIED_V02, bieChannel=bie-events-contention-classified
gov.va.vro.services.bie.config.ContentionEventTest ‑ [3] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_CLASSIFIED_V02, expectedEvent=CONTENTION_CLASSIFIED
gov.va.vro.services.bie.config.ContentionEventTest ‑ [4] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_COMPLETED_V02, bieChannel=bie-events-contention-completed
gov.va.vro.services.bie.config.ContentionEventTest ‑ [4] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_COMPLETED_V02, expectedEvent=CONTENTION_COMPLETED
gov.va.vro.services.bie.config.ContentionEventTest ‑ [5] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_DELETED_V02, bieChannel=bie-events-contention-deleted
gov.va.vro.services.bie.config.ContentionEventTest ‑ [5] inputTopic=BIA_SERVICES_BIE_CATALOG_TST_CONTENTION_DELETED_V02, expectedEvent=CONTENTION_DELETED

♻️ This comment has been updated with latest results.

github-actions · 2024-02-15T22:17:23Z

JaCoCo Test Coverage

Overall Project	76.64%	❌
Files changed	100%	🍏

File	Coverage
ContentionEvent.java	100%	🍏
BieProperties.java	100%	🍏

…-sasl

svc-bie-kafka/src/main/resources/application.yaml

tejans24 · 2024-02-16T16:04:20Z

svc-bie-kafka/src/main/resources/application-integration-test.yaml

+        username: USERNAME
+        password: PASSWORD


we'll need to add the password to the Vault/kube secrets and inject it into the containers per env

msnwatson

Some comments

mocks/docker-compose.yml

svc-bie-kafka/src/main/resources/application-integration-test.yaml

…-sasl

… 3.2.3 (#2659) Bump org.springframework.boot:spring-boot-autoconfigure Bumps [org.springframework.boot:spring-boot-autoconfigure](https://github.com/spring-projects/spring-boot) from 3.2.2 to 3.2.3. - [Release notes](https://github.com/spring-projects/spring-boot/releases) - [Commits](spring-projects/spring-boot@v3.2.2...v3.2.3) --- updated-dependencies: - dependency-name: org.springframework.boot:spring-boot-autoconfigure dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cheng <[email protected]>

Made special issue code an environment variable, so it can be switched without code change.

…-sasl

…munication.

shared/lib-bie-kafka/src/main/java/gov/va/vro/model/biekafka/ContentionEvent.java

svc-bie-kafka/src/main/resources/application-dev.yaml

…cation

…-sasl

msnwatson · 2024-04-08T21:55:37Z

svc-bie-kafka/src/main/resources/application-dev.yaml

+        trust-store-type: "PKCS12"
+
+bie:
+  kafka-topic-env: "TST"


Nit: newline at EOF

…-sasl

nelsestu

The BIE team is now going to be the BIA team, and as auth is now using the BIA RBAC authentication, it simplfies some of the certificate generation and handling. As Teja mentioned, we'll need to include the username and password in our reimplementation of secret automation.

Initial attempt currently failing on sasl handshake

114b31d

agile-josiah marked this pull request as ready for review February 15, 2024 22:36

agile-josiah requested a review from a team as a code owner February 15, 2024 22:36

agile-josiah requested review from nelsestu and msnwatson February 15, 2024 22:36

Enables sasl_ssl for our kafka broker and clients

7f53b79

agile-josiah changed the title ~~WIP #2530 Enable SASL for bie kafka~~ #2530 Enable SASL for bie kafka Feb 15, 2024

agile-josiah added 3 commits February 15, 2024 14:42

More changes

a6646b1

Merge remote-tracking branch 'origin/develop' into josiah/#2530/kafka…

bbd1d23

…-sasl

Fix for integration test

348190b

tejans24 reviewed Feb 16, 2024

View reviewed changes

svc-bie-kafka/src/main/resources/application.yaml Outdated Show resolved Hide resolved

tejans24 reviewed Feb 16, 2024

View reviewed changes

msnwatson approved these changes Feb 17, 2024

View reviewed changes

mocks/docker-compose.yml Outdated Show resolved Hide resolved

mocks/docker-compose.yml Outdated Show resolved Hide resolved

svc-bie-kafka/src/main/resources/application-integration-test.yaml Outdated Show resolved Hide resolved

agile-josiah and others added 14 commits February 20, 2024 10:53

Remove comments and unnecessary duplication of ssl

1e3f44b

Merge remote-tracking branch 'origin/develop' into josiah/#2530/kafka…

a3fb2b3

…-sasl

Merge branch 'develop' into josiah/#2530/kafka-sasl

41e0bcc

Automated commit: Unpin versions of updated images

6c99c6f

EP Merge: Special Issue Code workaround for testing (#2645)

cda1eb7

Made special issue code an environment variable, so it can be switched without code change.

Update QA BGS URL (#2622)

3c61b66

Merge remote-tracking branch 'origin/develop' into josiah/#2530/kafka…

8fa6684

…-sasl

Update to use vault values, added to dev and integration test yamls

a789218

Merge remote-tracking branch 'origin/develop' into josiah/#2530/kafka…

834c0f6

…-sasl

Update the remaining environments

72f6975

Merge branch 'develop' into josiah/#2530/kafka-sasl

fd0d605

Updating the names of the topics and add the tls certs needed for com…

0e473b8

…munication.

Add the var initialization to setenv.sh

f22ff11

nelsestu approved these changes Mar 8, 2024

View reviewed changes

shared/lib-bie-kafka/src/main/java/gov/va/vro/model/biekafka/ContentionEvent.java Outdated Show resolved Hide resolved

svc-bie-kafka/src/main/resources/application-dev.yaml Outdated Show resolved Hide resolved

agile-josiah added 11 commits April 3, 2024 10:46

Changes to bie kafka docker vars

0a8a671

Add truststore back

23e76ad

Edits to ssl/sasl connection

58d8003

Updates to docker-entryprep.sh to ignore the keystore creation/verifi…

4a2d126

…cation

Remove references to keystore file

0be6866

Add a ssl piece to the properties file

0ae5791

Add ssl truststore everywhere.

38ab6c6

Change shape of schema.registry.truststore.*

14a2989

Add in USER_INFO basic auth for schema registry

4065cfc

Revert higher environments for RBAC to match develop

c408ab7

Merge remote-tracking branch 'origin/develop' into josiah/#2530/kafka…

b608c50

…-sasl

agile-josiah requested review from tejans24, msnwatson, chengjie8 and nelsestu April 8, 2024 21:19

msnwatson approved these changes Apr 8, 2024

View reviewed changes

agile-josiah added 7 commits April 9, 2024 08:58

Small nits and update to infix from env

fed6802

Merge remote-tracking branch 'origin/develop' into josiah/#2530/kafka…

b187a30

…-sasl

rm unnecessary Kafka cert gen

781f8c0

Add qa and sandbox as well

3b4e831

Revert mock BIE kafka to existing state

3c60d98

Merge remote-tracking branch 'origin/develop' into josiah/#2530/kafka…

4e4b2d8

…-sasl

Revert local config

268577e

This was referenced Apr 10, 2024

Implement RBAC for svc-bie-kafka in our in sandbox environment #2846

Closed

Implement RBAC for svc-bie-kafka in our in QA environment #2847

Closed

nelsestu approved these changes Apr 11, 2024

View reviewed changes

chengjie8 added 2 commits April 15, 2024 16:49

RBAC user id password are not needed for local test

fa020b8

Merge branch 'develop' into josiah/#2530/kafka-sasl

d8d2eea

chengjie8 merged commit 97b535f into develop Apr 17, 2024
1 check passed

chengjie8 deleted the josiah/#2530/kafka-sasl branch April 17, 2024 15:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#2530 Enable SASL for bie kafka #2624

#2530 Enable SASL for bie kafka #2624

agile-josiah commented Feb 15, 2024 •

edited

Loading

github-actions bot commented Feb 15, 2024 •

edited

Loading

github-actions bot commented Feb 15, 2024 •

edited

Loading

tejans24 Feb 16, 2024

msnwatson left a comment

msnwatson Apr 8, 2024

agile-josiah Apr 9, 2024

nelsestu left a comment

#2530 Enable SASL for bie kafka #2624

#2530 Enable SASL for bie kafka #2624

Conversation

agile-josiah commented Feb 15, 2024 • edited Loading

What was the problem?

How does this fix it?1

How to test this PR

Footnotes

github-actions bot commented Feb 15, 2024 • edited Loading

Test Results

github-actions bot commented Feb 15, 2024 • edited Loading

JaCoCo Test Coverage

tejans24 Feb 16, 2024

Choose a reason for hiding this comment

msnwatson left a comment

Choose a reason for hiding this comment

msnwatson Apr 8, 2024

Choose a reason for hiding this comment

agile-josiah Apr 9, 2024

Choose a reason for hiding this comment

nelsestu left a comment

Choose a reason for hiding this comment

agile-josiah commented Feb 15, 2024 •

edited

Loading

How does this fix it?¹

github-actions bot commented Feb 15, 2024 •

edited

Loading

github-actions bot commented Feb 15, 2024 •

edited

Loading