container build is very slow on arm/arm64 platforms

[sp-yduck]

/kind feature

Describe the solution you'd like
somehow arm64 and arm/v7 container builds took about 30mins while amd64 builds were finished within 4mins. they should not take so much time comparing to amd64.
amd64 : 230s = 3m50s
arm64 : 1737s = 28m57s
arm/v7 : 1875s = 31m15s

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-proxmox version:
• Proxmox VE version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[sp-yduck]

it looks using docker/build-push-action is causing this slow because the other build done by make docker-buildx finishes with about 85s for both amd64 and arm64 . job : https://github.com/sp-yduck/cluster-api-provider-proxmox/actions/runs/6832135564/job/18582489629
let's use make docker-buildx for this image publishing job as well