[CI][Java] java-nightly cache has 8.6 GB

[kou]

Describe the enhancement requested

https://github.com/apache/arrow/actions/caches

java-nightly-6371112382
8.6 GB cached hours ago

We can use 10 GB in apache/arrow for cache. If the java-nightly cache uses 8.6 GB, other caches will be expired soon.

The java-nightly cache was introduced by GH-13839.

Component(s)

Continuous Integration, Java

[kou]

@davisusanibar @assignUser @raulcd Can we just remove the java-nightly cache?

diff --git a/.github/workflows/java_nightly.yml b/.github/workflows/java_nightly.yml
index 4440d36d1..41843d663 100644
--- a/.github/workflows/java_nightly.yml
+++ b/.github/workflows/java_nightly.yml
@@ -73,12 +73,6 @@ jobs:
           fi
           echo $PREFIX
           archery crossbow download-artifacts -f java-jars -t binaries  $PREFIX
-      - name: Cache Repo
-        uses: actions/cache@v3
-        with:
-          path: repo
-          key: java-nightly-${{ github.run_id }}
-          restore-keys: java-nightly
       - name: Sync from Remote
         uses: ./arrow/.github/actions/sync-nightlies
         with:

[assignUser]

Huh, looks like they are enforcing that limit now... for a long time we had around 60G of caches ^^ well we will have to download the full repo from nightlies.apache.org everytime... maybe we can find a better way to remove the old artifacts?

[kou]

How about removing the actions/cache as the first step? GH-37980
Because it slows down all other CI jobs.

We can improve our uploading/pruning nightlies algorithm as the next step (with a separated issue).
For example:

• We can upload new nightly artifacts without downloading the existing nightly artifacts.
• We can determine old nightly artifacts to be pruned only with directories. (We don't need to download each .jar files.)
• We can prune the target nightly artifacts by rsync --delete with local empty directory.