Time for a new release?

[vit-zikmund]

Hello Datopian folks!

I just stumbled upon this great project and hit a blocker in the 0.5.0 release, that's already fixed.
I need to target a private AWS-like object storage, but the endpoint setting is missing from the docker image. This year even boto3 got upgraded to read the AWS_ENDPOINT_URL env var to override it, but the one included is too old.

I see there's been a couple fixes here and there in the last two years from the 0.5.0 release. Would that make sense to carve out a new one? 🙂 I sure can build the stuff myself (thank you for that - making this nice tool opensource), but wanted to ask, before I start the plumbing process.

Thank you so much! 😉

[rufuspollock]

@vit-zikmund good point and sorry for the slow reply. We definitely should do a new release. If you also have patches or contributions you would like to make PRs are welcome.

@anuveyatsu can we push out a new release of this soon?

[rufuspollock]

@anuveyatsu @demenech tagging you here if we can get a new release out (and/or enable @athornton to push releases).

[demenech]

hello @rufuspollock

I do not have the necessary permissions.

@anuveyatsu perhaps you could give me and @athornton higher privilege?

[athornton]

The problem isn't on the GitHub side, it's on the uploading to PyPi side, if I remember correctly.

Someone who's an admin on the PyPi Datopian account could add me and @vit-zikmund , or (I think) they could go through the process to link the GH repository with PyPi as a trusted publisher:

https://docs.pypi.org/trusted-publishers/adding-a-publisher/

That's probably both easier and more secure since no one would have to grant additional PyPi privileges. Then we could just bake the PyPi upload into the Github Actions for doing a release.

[demenech]

hello @athornton

letting you know that I have been given the privilege, so I believe I can unblock this now, will try to do it between the end of my day today and tomorrow

[vit-zikmund]

Oh this is sweet! Could I get to squeeze in one little, but rather important security improvement for the github auth?

[demenech]

hello @athornton

The trusted publisher has been added. Can you try it out, please? And is there anything else that I have to do?

[athornton]

Thanks. I'll see what I can do with the CI and whether I can get it to push.

[athornton]

Turns out 3.13 support is going to be a little harder than I expected, so I won't be adding that right now. I am adding uv as a pip replacement, because it's much, much faster, and that matches what we did at Rubin.

We should think about what branches should build ghcr containers, too. Currently it's "tickets/*" and releases, which match what we do at Rubin, but the naming of ticket branches is a very specific-to-Rubin convention. If there's something else you prefer we can easily change that.

One thing I think you could do, @demenech , is give @vit-zikmund whatever permissions on this repo I've got. He's been doing most of the most recent work.

[athornton]

Would folks who aren't me, using Docker containers, please try out the image ghcr.io/datopian/giftless:tickets-DM-42598 and ensure it's working as expected? I am going to put it in my dev environment for the afternoon, and then try it in prod.

Assuming it looks good, I'll merge it up to main and then cut a 0.6.0 release early next week. That release should trigger the PyPi upload, and hopefully everything works and then we can announce Version 0.6.0 for real. If not, well, there will be some fast follower releases until we DO get it pushing smoothly to PyPi.

If Datopian wants to keep hosting at Docker Hub, we'll need to put a push token in the secrets for the repository or the organization and change the CI slightly. Myself, I feel like GitHub is likely to survive longer and in better shape than Docker-the-company and it's probably just appropriate for Datopian, when they announce the new release, to point out that the host for the docker containers has moved.

[anuveyatsu]

Hi @athornton @vit-zikmund thank you for the effort into making the new release possible 🙏🏼 we just invited @vit-zikmund as maintainer of this repo 👍🏼

I don't think we have any constraints regarding the location of the container registry, so a GitHub-based option should work perfectly fine.

[vit-zikmund]

Thank you, @anuveyatsu (and naturally @athornton for the nomination)! It's a big honor.

[athornton]

The upload did not trigger.

The only thing I see wrong is that my GHA environment name is set to pypi and you've got "(Any)" there. @anuveyatsu I don't have a lot of hope that it's going to help, but can you try deleting and recreating the trusted publisher with environment name set to pypi instead?

EDIT belay that...I think I missed putting the release in the "on" section at the top of CI.

[demenech]

hello @athornton

I'm around if anything has to change in the trusted publisher.

[athornton]

OK, getting closer:

WARNING  Error during upload. Retry with the --verbose option for more details. 
[532](https://github.com/datopian/giftless/actions/runs/12358922232/job/34490573261#step:4:538)
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
[533](https://github.com/datopian/giftless/actions/runs/12358922232/job/34490573261#step:4:539)
         'License :: OSI Approved:: MIT License' is not a valid classifier. See 
[534](https://github.com/datopian/giftless/actions/runs/12358922232/job/34490573261#step:4:540)
         https://packaging.python.org/specifications/core-metadata for more     
[535](https://github.com/datopian/giftless/actions/runs/12358922232/job/34490573261#step:4:541)
         information.

[athornton]

ghcr.io/datopian/giftless:0.6.2 is now available, matching PyPi giftless 0.6.2.

I'm going to have to add a setting to our build-and-push action to also tag with latest and push, and that may take a bit (or stop using composite actions and just tackle building and pushing separately, but likewise that will take a bit).

In the interim, though, doing a new release will both push to pypi and ghcr.io.

[athornton]

I'll also write up some docs about how to do a release through GHA, and then we probably should clean up the no-longer-used make targets, and additionally include "how to use scriv" which we've been using for changelog management. It plays nicely with GHA.

But the good news is: there's a new Giftless at PyPi, and a corresponding container at ghcr.io.

If someone from Datopian wants to upload ghcr.io/datopian/giftless:0.6.2 to docker.io and then tag both those as latest then we'd have a much more modern Giftless out there without the userbase having to change much about how they work.

Datopian should figure out a messaging strategy around "stop using hub.docker.com / docker.io and start using ghcr.io" to tell the users, as well as whatever sort of statement they want to make about a newer giftless.

After that it is, alas, almost time for another round of modernization...but it's smaller this time, and the thing I really wanted to do (Python 3.13 support) seems like it's gated behind grpc support, which is probably what's also holding gcloud back from being 3.13 ready, and I'd expect an eventual Google update that fixes all that at once, at which point Giftless is unblocked.

[demenech]

hey @athornton

This sounds great.

I have access to Dockerhub. I think it makes sense to push the latest version, I'll do that.

Regarding the deprecation of the docker hub repo, I'm thinking about tweaking the README and metadata at https://hub.docker.com/repository/docker/datopian/giftless and maybe echoing some sort of deprecation message at the latest version? Wdyt?

[athornton]

I think that's a great idea. I should have an updated GitHub Action that will let us tag releases with "latest" as well pretty soon. That can go into the Giftless build that refactors the Makefile.

[demenech]

hello @athornton

Thanks, and sorry for the late reply. I'm working on adding the warnings, but I don't think I'll be able to finish it today.

Hoping to have it done tomorrow.

CC: @rufuspollock

[demenech]

hello @athornton and @vit-zikmund

Merry Christimas!

When you are available, please take a look at #181

Thanks.

CC: @rufuspollock

[rufuspollock]

@demenech amazing job 👏