Multi cell database/mq adoption #746

bogdando · 2024-11-25T12:48:19Z

Fix OSPDo specifics for single vs multi cellls

Declare RUN_OVERRIDES before it is used.
Use env vars instead of docs generation conditions to reuse the same
code in tests:

Add MARIADB_RUN_OVERRIDES to cover all overrides and client annotations
Add missing definitions for rhoso/ospd namespace specific vars
Use env TRIPLEO_PASSWORDS for all cases as OSPDo still deploys
tripleo
Define and use NAMESPACE (default openstack) instead of
RHOSO18_NAMESPACE or OSPDO_NAMESPACE. Remove unused rhoso18 ns value
(only in these guide).

Refactor comments in commands into asciidocs native

Illustrate how commands in scripts could have comments
that become (almost as is) native ascii docs foot-notes.

When copying code into docs, the minimal adjustments will
be needed, like adding '$' prefix (or '>' for multiline commands).

Provide a static multi-cell config for databases and messaging
for adoption guide and tests, which comprises a 3 cells.

Keep renaming 'default' cell consistent for single and multi cells:

Default becomes cellX (or it can be imported as is, for a multi-cell
case only)
cell1 becomes mapped to openstack-cell1 osdp node set
cell2 becomes mapped to openstack-cell2 osdp node set, etc.
cellX (X=3 here) becomes mapped to openstack-cell3. Alternatively,
default cell retains its name for the openstack-default osdpns
mapping
Evaluate podified MariaDB passwords for cells from osp-secret
to align the tests with documented commands. Remove no longer
needed podified DB password variable.

Make ansible and shell variables compute cells aware.
Rework vars and secrets YAML values for the source and edpm
nodes to not confuse its different naming schemes for cells
in OSP/TripleO and RHOSO.
Remove cached fact for pulled OSP configuration as it can no longer
be generated in a multi-cell setup, where related shell variables
become bash arrays.
Simplify ENV headers management by collecting in a single place.
Adjust storage/storageRequests values to make it better fitting
a multi-cell test scenarios. Also provide values in docs and
add a comment to adjust them as needed.
Remove source_db_root_password as it is directly evaluated from
tripleo passwords into an env var.
Run mysql commands in individual pods.
Finished pods take time to terminate, avoid errors where
consequent mysql commands failing because the old and new pod use the
same name.
Rename nodesets to openstack-cell1, which is needed for adoption of
remaining multi-cell aware services in a follow up.
Make edpm_nodes input multi-cell aware.

Assume a single cell1 yet.

Remove edpm_computes and computes env var
from tests as it is not multi-cell aware, and should be no longer
needed. The docs still use that env var, it will be removed in
multi-cell adoption follow up, where we also cover EDPM multi-cell
adoption.

This is required as rhe rdo-jobs dependency introduces that
change for edpm_nodes and provides a common base for this and future
multi-cell follow ups.

Partial: #184
Depends-On: https://review.rdoproject.org/r/c/rdo-jobs/+/53192

Jira: #OSPRH-6548

bogdando · 2024-11-25T12:54:34Z

testproject https://review.rdoproject.org/r/c/testproject/+/55584

softwarefactory-project-zuul · 2024-11-25T14:30:31Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ff2fa0079d2d4d0f9d107db215021a87

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph FAILURE in 1h 34m 52s
❌ adoption-standalone-to-crc-no-ceph FAILURE in 1h 40m 01s
✔️ adoption-docs-preview SUCCESS in 1m 17s

docs_user/modules/proc_migrating-databases-to-mariadb-instances.adoc

tests/roles/backend_services/tasks/ospdo_backend_services.yaml

softwarefactory-project-zuul · 2024-11-26T15:43:07Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/2bf5c26d9ce042488b038c87505a0f82

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph FAILURE in 1h 38m 37s
❌ adoption-standalone-to-crc-no-ceph FAILURE in 1h 44m 16s
✔️ adoption-docs-preview SUCCESS in 1m 31s

softwarefactory-project-zuul · 2024-11-27T15:03:27Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ef997d9bd22a422fa0a81831c10ce76f

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph POST_FAILURE in 1h 40m 25s
❌ adoption-standalone-to-crc-no-ceph RETRY_LIMIT in 10m 21s
✔️ adoption-docs-preview SUCCESS in 1m 21s

softwarefactory-project-zuul · 2024-11-29T17:54:35Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/221b8eaa480445db948ba454fc1dc6d9

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph FAILURE in 1h 51m 13s
❌ adoption-standalone-to-crc-no-ceph FAILURE in 53m 45s
❌ adoption-docs-preview FAILURE in 1m 15s

bogdando · 2024-12-02T12:17:30Z

recheck

softwarefactory-project-zuul · 2024-12-02T13:59:45Z

This change depends on a change that failed to merge.

Change https://review.rdoproject.org/r/c/rdo-jobs/+/53192 is needed.

softwarefactory-project-zuul · 2024-12-04T13:55:54Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/28720608befb44c49d0f76f95466684b

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph RETRY_LIMIT in 48m 27s
❌ adoption-standalone-to-crc-no-ceph RETRY_LIMIT in 48m 06s
❌ adoption-docs-preview FAILURE in 1m 15s

tests/roles/backend_services/templates/openstack_control_plane.j2

bogdando · 2024-12-06T14:06:57Z

docs_user/modules/proc_retrieving-topology-specific-service-configuration.adoc

-MARIADB_IMAGE=registry.redhat.io/rhosp-dev-preview/openstack-mariadb-rhel9:18.0
-endif::[]
-SOURCE_MARIADB_IP=172.17.0.2
+$ PASSWORD_FILE="$HOME/overcloud-passwords.yaml"


see how we match this procedure's commands in docs with those in tests https://github.com/openstack-k8s-operators/data-plane-adoption/pull/746/files#diff-50c1d11b8d50eb791a57420a1cb81ce7bce10dee4ec137c139146ade1ca9dc57R27

softwarefactory-project-zuul · 2024-12-06T16:23:41Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/513df93417f943f8bc8b6d3849192150

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph FAILURE in 1h 42m 09s
❌ adoption-standalone-to-crc-no-ceph FAILURE in 1h 44m 20s
✔️ adoption-docs-preview SUCCESS in 1m 28s

jistr · 2024-12-11T08:39:30Z

Did this get expanded accidentally to include all the multi-cell changes and not just the DB part?

bogdando · 2024-12-11T14:22:55Z

Did this get expanded accidentally to include all the multi-cell changes and not just the DB part?

some parts of it did... I will fix that

docs_user/modules/proc_deploying-backend-services.adoc

docs_user/modules/proc_stopping-infrastructure-management-and-compute-services.adoc

docs_user/modules/proc_stopping-openstack-services.adoc

tests/roles/backend_services/tasks/main.yaml

tests/roles/backend_services/templates/openstack_control_plane.j2

tests/roles/mariadb_copy/tasks/main.yaml

softwarefactory-project-zuul · 2024-12-11T16:50:13Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/0599a819c87e4b3587b8f26fcb1bde3d

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph FAILURE in 2h 05m 40s
❌ adoption-standalone-to-crc-no-ceph FAILURE in 2h 05m 30s
✔️ adoption-docs-preview SUCCESS in 1m 21s

docs_user/modules/proc_deploying-backend-services.adoc

softwarefactory-project-zuul · 2024-12-12T18:06:05Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/bb124871501c4f128f85c5ba04239944

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph FAILURE in 2h 22m 42s
❌ adoption-standalone-to-crc-no-ceph FAILURE in 2h 13m 27s
✔️ adoption-docs-preview SUCCESS in 1m 16s

klgill

Please review my comments and suggestions.
It would really help me to look at the downstream preview to check for any formatting issues.

docs_user/modules/proc_migrating-databases-to-mariadb-instances.adoc

klgill · 2024-12-12T18:32:46Z

docs_user/modules/proc_retrieving-topology-specific-service-configuration.adoc

+$ chmod 0600 ~/.source_cloud_exported_variables*
+----
+
+<1> If `neutron-sriov-nic-agent` agents are running in your {OpenStackShort} deployment, get the configuration to use for the data plane adoption


Suggested change

<1> If `neutron-sriov-nic-agent` agents are running in your {OpenStackShort} deployment, get the configuration to use for the data plane adoption

<1> If `neutron-sriov-nic-agent` agents are running in your {OpenStackShort} deployment, get the configuration to use for the data plane adoption.

Where do you get this configuration?

This is probably a formatting change only, I was not introducing this statement in the 1st place, so I cannot answer to that question (@karelyatin perchance? we could address that improvement to SR-IOV adoption guide in follow up)

docs_user/modules/proc_retrieving-topology-specific-service-configuration.adoc

docs_user/modules/proc_stopping-infrastructure-management-and-compute-services.adoc

docs_user/modules/proc_stopping-openstack-services.adoc

klgill · 2024-12-12T18:44:47Z

docs_user/modules/proc_stopping-openstack-services.adoc

+----
+ifeval::["{build}" == "downstream"]
+<1> Replace `<path_to_SSH_key>` with the path to your SSH key.
+<2> Replace `<controller-X IP>` with IP addresses of all controllers.


Suggested change

<2> Replace `<controller-X IP>` with IP addresses of all controllers.

<2> Replace `<controller-1 IP>` with the IP addresses of all Controllers.

The value to replace here should match the line of code.

In code there is

ifeval::["{build}" == "downstream"] CONTROLLER1_SSH="ssh -i *<path to SSH key>* root@*<controller-1 IP>*" <2> CONTROLLER2_SSH="ssh -i *<path to SSH key>* root@*<controller-2 IP>*" CONTROLLER3_SSH="ssh -i *<path to SSH key>* root@*<node3 IP>*" # ... endif::[]

isn't that matches to <controller-X IP> (or shall we fix to node3?

@klgill sorry, I meant this renders OK for me

Ah, I see. You used as a "catch-all" for each value.
Typically, our style is to list each env var. But you could probably simplify it by using only one bullet point, for example:

Replace <controller-1 IP>, <controller-2 IP>, and <node3 IP> with the IP addresses of each Controller node.

Provide a static multi-cell config for databases and messaging for adoption guide and tests, which comprises a 3 cells. Keep renaming 'default' cell consistent for single and multi cells: Default becomes cellX (or it can be imported as is, for a multi-cell case only) cell1 becomes mapped to openstack-cell1 osdp node set cell2 becomes mapped to openstack-cell2 osdp node set, etc. cellX (X=3 here) becomes mapped to openstack-cell3. Alternatively, default cell retains its name for the openstack-default osdpns mapping Evaluate podified MariaDB passwords for cells from osp-secret to align the tests with documented commands. Remove no longer needed podified DB password variable. Make ansible and shell variables compute cells aware. Rework vars and secrets YAML values for the source and edpm nodes to not confuse its different naming schemes for cells in OSP/TripleO and RHOSO. Remove cached fact for pulled OSP configuration as it can no longer be generated in a multi-cell setup, where related shell variables become bash arrays. Simplify ENV headers management by collecting in a single place. Adjust storage/storageRequests values to make it better fitting a multi-cell test scenarios. Also provide values in docs and add a comment to adjust them as needed. Remove source_db_root_password as it is directly evaluated from tripleo passwords into an env var. Run mysql commands in individual pods. Finished pods take time to terminate, avoid errors where consequent mysql commands failing because the old and new pod use the same name. Rename nodesets to openstack-cell1, which is needed for adoption of remaining multi-cell aware services in a follow up. Signed-off-by: Bohdan Dobrelia <[email protected]> Fix Signed-off-by: Bohdan Dobrelia <[email protected]>

Declare RUN_OVERRIDES before it is used. Use env vars instead of docs generation conditions to reuse the same code in tests: * Add MARIADB_RUN_OVERRIDES to cover all overrides and client annotations * Add missing definitions for rhoso/ospd namespace specific vars * Use env TRIPLEO_PASSWORDS for all cases as OSPDo still deploys tripleo * Define and use NAMESPACE (default openstack) instead of RHOSO18_NAMESPACE or OSPDO_NAMESPACE. Remove unused rhoso18 ns value (only in these guide). Signed-off-by: Bohdan Dobrelia <[email protected]>

Illustrate how commands in scripts could have comments that become (almost as is) native ascii docs foot-notes. When copying code into docs, the minimal adjustments will be needed, like adding '$' prefix (or '>' for multiline commands). Signed-off-by: Bohdan Dobrelia <[email protected]>

Those will be added back in a follow up, which completes the guide and tests for extra cell2 and cell3. Signed-off-by: Bohdan Dobrelia <[email protected]>

softwarefactory-project-zuul · 2024-12-16T14:03:45Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/813ed9ca7fb3499e9c8fedf30ef0f316

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph NODE_FAILURE Node request 100-0007705163 failed in 0s
❌ adoption-standalone-to-crc-no-ceph NODE_FAILURE Node request 100-0007705164 failed in 0s
✔️ adoption-docs-preview SUCCESS in 1m 15s

Assume a single cell1 yet. Remove edpm_computes and computes env var from tests as it is not multi-cell aware, and should be no longer needed. The docs still use that env var, it will be removed in multi-cell adoption follow up, where we also cover EDPM multi-cell adoption. This is required as rhe rdo-jobs dependency introduces that change for edpm_nodes and provides a common base for this and future multi-cell follow ups. Signed-off-by: Bohdan Dobrelia <[email protected]>

Co-authored-by: Katie Gilligan <[email protected]>

softwarefactory-project-zuul · 2024-12-17T20:35:19Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ac1f2c5cea9b4e63a9759e67b769b052

✔️ noop SUCCESS in 0s
❌ adoption-standalone-to-crc-ceph FAILURE in 2h 57m 12s
❌ adoption-standalone-to-crc-no-ceph FAILURE in 2h 13m 26s
✔️ adoption-docs-preview SUCCESS in 1m 16s

bogdando requested a review from jistr November 25, 2024 12:48

bogdando force-pushed the multi_cell_database branch from 3323bbf to a1207eb Compare November 25, 2024 12:49

bogdando changed the title ~~Multi cell database~~ Multi cell database/mq adoption Nov 25, 2024

bogdando mentioned this pull request Nov 25, 2024

Multi-cell adoption #517

Open

bogdando added the check-before-merge/depends-on Don't forget to check depends-on before merging label Nov 25, 2024

bogdando mentioned this pull request Nov 25, 2024

Remove patching OSP as the package is released now openstack-k8s-operators/install_yamls#953

Merged

jistr approved these changes Nov 26, 2024

View reviewed changes

docs_user/modules/proc_migrating-databases-to-mariadb-instances.adoc Outdated Show resolved Hide resolved

tests/roles/backend_services/tasks/ospdo_backend_services.yaml Show resolved Hide resolved

bogdando force-pushed the multi_cell_database branch from 0ba8ce7 to cd5b8f9 Compare November 26, 2024 12:54

bogdando force-pushed the multi_cell_database branch 3 times, most recently from c392e11 to 006e247 Compare November 27, 2024 13:21

bogdando force-pushed the multi_cell_database branch from 006e247 to 55aec9f Compare November 29, 2024 15:36

bogdando force-pushed the multi_cell_database branch from 55aec9f to ee3defd Compare December 2, 2024 12:27

bogdando force-pushed the multi_cell_database branch from ee3defd to 03ce37c Compare December 4, 2024 13:04

bogdando force-pushed the multi_cell_database branch 3 times, most recently from 505e4e8 to 8d81b48 Compare December 6, 2024 13:59

bogdando commented Dec 6, 2024

View reviewed changes

tests/roles/backend_services/templates/openstack_control_plane.j2 Outdated Show resolved Hide resolved

bogdando commented Dec 6, 2024

View reviewed changes

tests/roles/backend_services/templates/openstack_control_plane.j2 Outdated Show resolved Hide resolved

bogdando commented Dec 6, 2024

View reviewed changes

bogdando force-pushed the multi_cell_database branch from 8d81b48 to cdff544 Compare December 6, 2024 14:20

bogdando force-pushed the multi_cell_database branch from 6a5c854 to a766ddc Compare December 11, 2024 14:10

bogdando commented Dec 11, 2024

View reviewed changes

docs_user/modules/proc_deploying-backend-services.adoc Outdated Show resolved Hide resolved

bogdando commented Dec 11, 2024

View reviewed changes

docs_user/modules/proc_stopping-infrastructure-management-and-compute-services.adoc Outdated Show resolved Hide resolved

bogdando commented Dec 11, 2024

View reviewed changes

docs_user/modules/proc_stopping-openstack-services.adoc Outdated Show resolved Hide resolved

bogdando commented Dec 11, 2024

View reviewed changes

tests/roles/backend_services/tasks/main.yaml Show resolved Hide resolved

bogdando commented Dec 11, 2024

View reviewed changes

tests/roles/backend_services/templates/openstack_control_plane.j2 Outdated Show resolved Hide resolved

bogdando commented Dec 11, 2024

View reviewed changes

tests/roles/mariadb_copy/tasks/main.yaml Show resolved Hide resolved

bogdando force-pushed the multi_cell_database branch from d5d083d to 6864b79 Compare December 12, 2024 11:54

klgill reviewed Dec 12, 2024

View reviewed changes

docs_user/modules/proc_deploying-backend-services.adoc Show resolved Hide resolved

klgill reviewed Dec 12, 2024

View reviewed changes

docs_user/modules/proc_deploying-backend-services.adoc Outdated Show resolved Hide resolved

klgill reviewed Dec 12, 2024

View reviewed changes

docs_user/modules/proc_deploying-backend-services.adoc Outdated Show resolved Hide resolved

bogdando mentioned this pull request Dec 12, 2024

Multi-cell adoption support openstack-k8s-operators/ci-framework#2601

Draft

klgill reviewed Dec 12, 2024

View reviewed changes

docs_user/modules/proc_deploying-backend-services.adoc Outdated Show resolved Hide resolved

bogdando force-pushed the multi_cell_database branch from c1a9693 to 3b6c0c3 Compare December 12, 2024 15:21

klgill requested changes Dec 12, 2024

View reviewed changes

bogdando added 4 commits December 16, 2024 14:49

Remove mentions of extra cells yet

f4848a2

Those will be added back in a follow up, which completes the guide and tests for extra cell2 and cell3. Signed-off-by: Bohdan Dobrelia <[email protected]>

bogdando force-pushed the multi_cell_database branch from 3b6c0c3 to 658849b Compare December 16, 2024 13:50

bogdando force-pushed the multi_cell_database branch from 7c2ea44 to 7e46a13 Compare December 17, 2024 16:47

bogdando and others added 2 commits December 17, 2024 18:10

Update docs_user/modules/proc_deploying-backend-services.adoc

8416c99

Co-authored-by: Katie Gilligan <[email protected]>

bogdando force-pushed the multi_cell_database branch from 7e46a13 to 8416c99 Compare December 17, 2024 17:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi cell database/mq adoption #746

Multi cell database/mq adoption #746

bogdando commented Nov 25, 2024 •

edited

Loading

bogdando commented Nov 25, 2024

softwarefactory-project-zuul bot commented Nov 25, 2024

softwarefactory-project-zuul bot commented Nov 26, 2024

softwarefactory-project-zuul bot commented Nov 27, 2024

softwarefactory-project-zuul bot commented Nov 29, 2024

bogdando commented Dec 2, 2024

softwarefactory-project-zuul bot commented Dec 2, 2024

softwarefactory-project-zuul bot commented Dec 4, 2024

bogdando Dec 6, 2024

softwarefactory-project-zuul bot commented Dec 6, 2024

jistr commented Dec 11, 2024

bogdando commented Dec 11, 2024 •

edited

Loading

softwarefactory-project-zuul bot commented Dec 11, 2024

softwarefactory-project-zuul bot commented Dec 12, 2024

klgill left a comment •

edited

Loading

klgill Dec 12, 2024

bogdando Dec 17, 2024

klgill Dec 12, 2024

bogdando Dec 17, 2024 •

edited

Loading

bogdando Dec 17, 2024

klgill Dec 17, 2024 •

edited

Loading

softwarefactory-project-zuul bot commented Dec 16, 2024

softwarefactory-project-zuul bot commented Dec 17, 2024

	<1> If `neutron-sriov-nic-agent` agents are running in your {OpenStackShort} deployment, get the configuration to use for the data plane adoption
	<1> If `neutron-sriov-nic-agent` agents are running in your {OpenStackShort} deployment, get the configuration to use for the data plane adoption.

	<2> Replace `<controller-X IP>` with IP addresses of all controllers.
	<2> Replace `<controller-1 IP>` with the IP addresses of all Controllers.

Multi cell database/mq adoption #746

Are you sure you want to change the base?

Multi cell database/mq adoption #746

Conversation

bogdando commented Nov 25, 2024 • edited Loading

bogdando commented Nov 25, 2024

softwarefactory-project-zuul bot commented Nov 25, 2024

softwarefactory-project-zuul bot commented Nov 26, 2024

softwarefactory-project-zuul bot commented Nov 27, 2024

softwarefactory-project-zuul bot commented Nov 29, 2024

bogdando commented Dec 2, 2024

softwarefactory-project-zuul bot commented Dec 2, 2024

softwarefactory-project-zuul bot commented Dec 4, 2024

bogdando Dec 6, 2024

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Dec 6, 2024

jistr commented Dec 11, 2024

bogdando commented Dec 11, 2024 • edited Loading

softwarefactory-project-zuul bot commented Dec 11, 2024

softwarefactory-project-zuul bot commented Dec 12, 2024

klgill left a comment • edited Loading

Choose a reason for hiding this comment

klgill Dec 12, 2024

Choose a reason for hiding this comment

bogdando Dec 17, 2024

Choose a reason for hiding this comment

klgill Dec 12, 2024

Choose a reason for hiding this comment

bogdando Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

bogdando Dec 17, 2024

Choose a reason for hiding this comment

klgill Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Dec 16, 2024

softwarefactory-project-zuul bot commented Dec 17, 2024

bogdando commented Nov 25, 2024 •

edited

Loading

bogdando commented Dec 11, 2024 •

edited

Loading

klgill left a comment •

edited

Loading

bogdando Dec 17, 2024 •

edited

Loading

klgill Dec 17, 2024 •

edited

Loading