chore: Install python dependencies with uv in workflows

[tokoko]

What this PR does / why we need it:

• Upgrades setup-python action to latest
• Disables caching, caches are bound to branches and unit tests run on different PR source branches cannot reuse each other's cache. Only multiple runs on the same PR can.
• Installs python dependencies with uv

[tmihalac]

• Disables caching, caches are bound to branches and unit tests run on different PR source branches cannot reuse each other's cache. Only multiple runs on the same PR can.

Maybe this helps somehow with the caches:
Have a look here in the documentation Cache not found for input keys

"The cache action first searches for cache hits for key and the cache version in the branch containing the workflow run. If there is no hit, it searches for restore-keys and the version. If there are still no hits in the current branch, the cache action retries same steps on the default branch"

I think it solves the problem with the caches because in a pull request the merging is usually into master in which the artifacts should already be in, don't you think ?

[tokoko]

@tmihalac that might be a good idea, unit tests are never run on master, but integration tests are and unit test jobs might be able to reuse caches written by integration tests. Not sure what will happen when two jobs try to update the cache with the same key at the same time though.

[tokoko]

@tmihalac One concern I still have is that looks like some people's experience with caching and uv on github is that often it might actually slow things down (see here). I have reenabled caches for all workflows except the linter one. We can use that to compare the install runtimes and then make a decision whether we should leave caching in or not... does that sound okay?

[tmihalac]

Sounds good to me.

To me it seems the tests take shorter time, do you think the same ?

[tokoko]

I'm not too sure actually, just from a sample size of 1, Install Dependencies step seems to take about 30 seconds without a cache hit (https://github.com/feast-dev/feast/actions/runs/8652609326/job/23725916068?pr=4086). When there is a cache hit, the combined runtime of cache retrieval and install for 3.9 is 23 seconds and for 3.10 it is 32 seconds. On top of that there was an initial cache upload step (that's no longer visible now) for the first run which took somewhere around 45 seconds.

P.S. For some reason everything's slower on mac vms 😄

[tokoko]

@jeremyary I think this is safe to merge

[franciscojavierarceo]

Seems like a big change... CC @HaoXuAI

[tokoko]

fwiw, even though uv is brand new, airflow has already switched to using it in their ci, so it's somewhat battle-tested already.

[HaoXuAI]

Is the cache totally not making any benefits for build?

[tokoko]

@HaoXuAI sorry, PR description is misleading now, I have added cache back in to monitor that. tldr version is that, right now it's probably a net loss actually, we are using an older version of cache action, it's slow to upload and sometimes just fails to find the cache even if it exists. With upgraded cache action and pip, it would definitely be of value, no question about it. The question is whether that still is the case with uv instead of pip. uv is much much faster than pip and to me it looks like cache often brings no real benefit, in other words time spent on cache upload/download is roughly the same as a fresh uv install. The current version of this PR disables caching for a linter job only and we can use that to make a final decision later.

[tokoko]

This run is a good example. cache step seems to work at first, but decides the cache is invalid for some reason. pip install step takes 2.5 minutes without cache and then the job proceeds to waste almost 6 minutes to upload the directory again. For comparison uv install (without cache) is basically always under a minute.

[HaoXuAI]

i'm not familiar with uv. but using it to install in the CI, will it have a different result with direct pip install?

[tokoko]

For all intents and purposes, no, it's intended to be a drop-in replacement. There's of course some minor differences, pip tends to be more lenient towards packages with invalid metadata for example, version resolution (which we aren't changing in this pr, but I'll do in a follow-up) might also be negligibly different because they are using different resolvers with different heuristics. But at the end of the day, it shouldn't matter for us.

[HaoXuAI]

LGTM.
like the idea of upgrading the python env tool.