Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix make ci-clean and runlocal-rp #3806

Merged
merged 2 commits into from
Sep 3, 2024

Conversation

SudoBrendan
Copy link
Collaborator

@SudoBrendan SudoBrendan commented Aug 30, 2024

Which issue this PR addresses:

Fixes local make errors

What this PR does / why we need it:

Problem 1: make runlocal-rp

Typo in image tag.

Problem 2: make ci-clean

Fix an error for orphaned running work containers by buildah that prevents prune from working. This can occasionally happen when you kill a build with ctrl-c (which can lead to some really wonky build states, like RUN commands missing the cache randomly/etc - make ci-clean needs to be capable of fixing these states):

$ make ci-clean
podman image prune --all --filter="label=aro-*=true"
WARNING! This command removes all images without at least one container associated with them.
Are you sure you want to continue? [y/N] y
Error: image used by fe1476ec4e2c451b5cdf0788df13f9006a64b4c7efa8cde43b72aec1f673c261: image is in use by a container: consider listing external containers and force-removing image
make: *** [Makefile:222: ci-clean] Error 125

Listing external containers, you can see the buildah working container artifacts that prevent image deletion:

$ podman ps --external
CONTAINER ID  IMAGE                                                                                          COMMAND     CREATED     STATUS      PORTS       NAMES
81b98b6b041d  registry.access.redhat.com/ubi8/nodejs-16:latest                                               buildah     4 days ago  Storage                 nodejs-16-working-container
3c1b2bf5c347  docker.io/library/46d89be108e7fa5cac82f7e68c543539b7c13e558ed1cdf78498b51916586f80-tmp:latest  buildah     4 days ago  Storage                 3d52b3bb3d8a-working-container
3efa745fabf8  docker.io/library/2aa66078bf34e95141549305672566212ddb4f66ba2fcce4a856e2bc1e199c4a-tmp:latest  buildah     4 days ago  Storage                 327bc823b740-working-container
3610dc395a7a  docker.io/library/cea1b1cccd979795295a974097fa8db114ac9ec5f125eafe5252865bf87ea16a-tmp:latest  buildah     4 days ago  Storage                 7b178c9a4c1f-working-container
eed2a06abda2  docker.io/library/11672cccafff0e2355b71f5458990cda5ad64035201a45e3923a3ae0ca924f6c-tmp:latest  buildah     4 days ago  Storage                 8a45bc60bb06-working-container
7fa4dba5db49  registry.access.redhat.com/ubi8/nodejs-16:latest                                               buildah     4 days ago  Storage                 nodejs-16-working-container-1

Unfortunately, I couldn't take "standard" routes:

  1. podman system prune --external does not fix the issue
  2. buildah rm -a would be nice, but buildah is strictly Linux and will not port to MacOs as part of make - so I had to build my own version of this command.

Test plan for issue:

Run make ci-clean locally, validate buildah containers are also removed with podman ps --external.

Is there any documentation that needs to be updated for this PR?

N/A

How do you know this will function as expected in production?

Does not affect production, only local image cache and containers created by buildah.

- fix an error for running work containers by buildah that prevents prune from working
@@ -232,6 +232,7 @@ ci-tunnel: fix-macos-vendor

.PHONY: ci-clean
ci-clean:
$(shell podman ps --external --format "{{.Command}} {{.ID}}" | grep buildah | cut -d " " -f 2 | xargs podman rm -f > /dev/null)
Copy link
Collaborator Author

@SudoBrendan SudoBrendan Aug 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I seriously wish this command could be prettier - I went through many iterations before settling here. If podman rm --filter supported COMMAND, this would be trivial, but it doesn't, and /dev/null is required because make does weird things with STDOUT on $(shell) stuff (you get errors like <Deleted Container SHA>: command not found) :/

Copy link
Collaborator

@kevinobriendotca kevinobriendotca Aug 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could shorten it a bit with command substitution.

podman rm -f $(podman ps | grep buildah | awk '{print $1}')

Also, podman rm has the -i flag that will ignore errors for non-existent containers and return 0 regardless of whether a container was deleted or not. Could possibly use that instead of the redirect to /dev/null.

Or, could we add a name to the build container so we can use the rm filter?

@SudoBrendan SudoBrendan changed the title Fix make ci-clean Fix make ci-clean and runlocal-rp Aug 30, 2024
@SudoBrendan
Copy link
Collaborator Author

/azp run ci

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@rajdeepc2792
Copy link
Collaborator

I was not able to reproduce the issue with the orphaned running work containers at my side. But the make ci-clean works correctly.
MacOS.
LGTM.

@SudoBrendan SudoBrendan merged commit ddff7a2 into master Sep 3, 2024
23 checks passed
gouthamMN pushed a commit that referenced this pull request Sep 5, 2024
* Fix make ci-clean error for running work containers by buildah that prevents prune from working
* Fix make runlocal-rp image syntax
gouthamMN pushed a commit that referenced this pull request Sep 10, 2024
update makfile to use go 1.23

update docs

bump gotestsum to 1.12.0

bump golangci-lint to 1.59.1

use the fips compliant golang image

generate a secret for the operator from workload identity

Update pkg/operator/deploy/deploy.go

Co-authored-by: Ayato Tokubi <[email protected]>

get subscription info from the subscription doc rather than env

test the operator identity secret generation code properly

Fixed  to correctly reference the local  image, preventing unauthorized Docker Hub pulls.

Align docs hierarchy

Indent bullet points

Copy fluentbit image from arointsvc ACR to your ACR

It is needed since it is compared against a default image (and digest) from const file

ARO-9570: Add a controller to the ARO operator to lay down etchosts machine config

ARO-9570: Update controller to watch MCP and ARO Cluster object

ARO-9750: Add a controller to create the etchosts machineconfigs if they dont exist

Fix linting

Add licenses

bump golangci-lint to v1.60.3 and exclude printf, SA1006 and S1009 from lint

update golangci-lint version

use non fips golang image

use go 1.22

bump go in ci

add git to dockerignore

set buildvcs to false

upgrade to go 1.22.6

update docs

fix go lint

address comments

remove commented code from onebranch pipelines file

change var to const

fix unit-tests and api cloudError

This is the new CI-RP stage for the pipline (#3768)

* This is the new CI-RP stage for the pipline (#3753)

* Ensure Podman Service is Started and PODMAN_REMOTE_ARGS is Configured Automatically

Ensure Podman Service is Started and PODMAN_REMOTE_ARGS is Configured Automatically

Ensure Podman Service is Started and PODMAN_REMOTE_ARGS is Configured Automatically

removed the tag

Add Podman service start and remote args setup for seamless operation

Add sudo to start Podman service for elevated permissions and fix permission errors

Add sudo to start Podman service for elevated permissions and fix permission errors

Refactor Makefile: Update Podman service handling with sudo and remove default PODMAN_REMOTE_ARGS to improve flexibility and ensure proper permissions.

Add sudo to start Podman service for elevated permissions and fix permission errors

* Added Podman service target and set PODMAN_REMOTE_ARGS for seamless builds.

* fix the makefile

* added the port to fix the Makefile

Add smoke test for alerts from Alertmanager (#3801)

Move ARM swagger to subfolder (#3805)

To add new HCP RP, the ARO RP is moved into the subfolder openshiftclusters.

There are no additional changes, no impact on the SDK and clients.

Add the old make runlocal-rp as an alternative to containerization (#3789)

Add smoke test documents (#3813)

Adding Ayato to CODEOWNERS

Fix make ci-clean and runlocal-rp (#3806)

* Fix make ci-clean error for running work containers by buildah that prevents prune from working
* Fix make runlocal-rp image syntax

Upgrade to Podman 5 to fix the vuln

Install required binary for Podman 5 in ci

Switch back to OneBranch build image

Install crun

Install more OCI packages

Change home dir to /tmp for podman

see containers/podman#23818
for more details.

Use sudo for tdnf

bump golangci-lint version in dockerfile ci-rp

add go flags

update go ver in ci.yml

update test

Correct testing/time issues in pkg/deploy (#3808)

- Percolate up the time to wait for LB healthcheck probes, test @ 0 sec
- Correct a context timeout test case, test @ 0 sec timeout

Fix slow tests in /pkg/backend (#3809)

Fix slow tests in /pkg/frontend (#3810)

* Clarifying etcd cert renew test

- Updated the test to make it clear it is passing because timeout is being reached
- Updated the timeout from 10s -> 0s to pass faster

* Fix slow changefeed tests

Generate smaller OIDC keys for unit tests (#3811)

- significantly increases unit test performance by moving from 4096 -> 256 bit keys
- preserves 4096 bit keys for all non-testing scenarios

Make CI-RP Improvements (#3791)

- Remove linting from ci-rp
- Remove generate from ci-rp

Set CGO_ENABLED

update test command in ci-rp dockerfile

Separate Makefile targets for local vs containers (#3816)

- reverts changes to runlocal-rp
- updates old run-portal to runlocal-portal since it uses local bins
- adds new targets for containerized run of RP and Portal; opt-in
- fixes docs and pipelines to use updated targets

Drop some unneccessary dependencies by moving to `bingo` for tooling (#3719)

* Move to using bingo for tools
* go mod vendor

[MIMO] Move cluster certificate functionality to ClientHelper (#3736)

* move over TLS applying, as well as some clienthelper work

bump go in bingo

merge makefile changes from Master

more Makefile updates

add GO var in toplevel Makefile
edisonLcardenas pushed a commit that referenced this pull request Sep 16, 2024
* Fix make ci-clean error for running work containers by buildah that prevents prune from working
* Fix make runlocal-rp image syntax
edisonLcardenas pushed a commit that referenced this pull request Sep 17, 2024
* Fix make ci-clean error for running work containers by buildah that prevents prune from working
* Fix make runlocal-rp image syntax
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants