Pub publish over CI should be supported

[jaumard]

Would be amazing to be able to do something like

pub publish --auth=MY_PUB_TOKEN

Because if you want to setup automatic publishing (when you create a tag for example) it's currently quite ugly (see https://github.com/rushio-consulting/rc_router/blob/master/.github/workflows/on_tag.yml).

Having to put the path like run: echo ${{secrets.PUB_CREDENTIALS}} > /opt/hostedtoolcache/flutter/1.9.1-hotfix.2-stable/x64/.pub-cache/credentials.json is not really great to maintain.

I don't mind trying to make the PR for this if your telling me it's the way to go

[jonasfj]

I'm not convinced this is the way to go. Your personal credentials.json is not well suited for use by automation (it works, but I'm not convinced this is a good solution for automation).

Ideally, we could somehow allow service accounts to be member of a publisher or uploader on a package on pub.dev, and then go from there. But there is still a few unknowns here.

[jaumard]

I'm not convinced this is the way to go. Your personal credentials.json is not well suited for use by automation (it works, but I'm not convinced this is a good solution for automation).

I agree but that's the only way we have for now so people are using it, might be a good idea to have a quick win and make it easy to use for now no ?

Ideally, we could somehow allow service accounts to be member of a publisher or uploader on a package on pub.dev, and then go from there. But there is still a few unknowns here.

Not sure what device accounts mean in this context.
What about being able to create "publish token" for a publisher ?
It will allow anyone to create a token that can be use later for automation only. But the api of pub publish should still allow the usage of a token (by param or env for example).

[jonasfj]

I agree but that's the only way we have for now so people are using it, might be a good idea to have a quick win and make it easy to use for now no ?

I'm not sure we should add this to the pub tool, then we'll have to maintain it long-term.
This seems like the kind of thing that would be nice to have in a commnuity tool like pubx.

What about being able to create "publish token" for a publisher ?

That might be the solution, but then you're forced to manage a token which can be leaked.
I haven't had time to experiment with it, but I think we could link a GCP service account, it is possible to export keys from these. But you can also make a service directly available in something like Cloud Build.
That way, when using Cloud Build there would be no secrets that can be leaked, GCP rotates keys automatically behind the scenes.

But as I said, this is not fully conceived yet, I have some digging to do to find out how pub with authenticate using a service account.

[AKushWarrior]

@jonasfj has this progressed at all?

[sigurdm]

No - not much has happened here recently

[jonasfj]

We had some ideas, but nothing has shipped yet.

I think we're still not entirely sure how to best do this.. Simple tokens is easy, but also awfully easy to leak :/

[AKushWarrior]

@jonasfj I agree that simple tokens is too naive. Perhaps there might be an optional integration (from pub.dev) which scrapes GitHub/GitLab/whatever and tries to publish whenever a commit is pushed.

Of course, this means introducing an interface to these APIs, dealing with Oauth, etc. I know Heroku has a feature like this, but I don't know how much complexity is added by doing it.

EDIT: this comment might be out of scope here, since this is more for https://github.com/dart-lang/pub-dev. If you think this would be worth discussion, I can refile there.

[jonasfj]

@AKushWarrior yeah, options I'm thinking of is:

1. Allow service accounts to be added as members on pub.dev, then enhance pub publish to support impersonating a service account.
   • This allows publishing from Google Cloud Build without credentials
   • Allows publishing from anywhere using exported service account keys
   • This is very complicated 😞
2. Allow creation of simple static tokens for publishing, allow pub publish to read $PUB_SECRET_TOKEN.
   • This is easy to use on Github Actions
   • This is easy to use on travis
   • This is easy to use for authentication when publishing to 3rd party pub servers.
3. Write a Github App, and integration specifically with Github.
   • This is complicated to write, debug and test (maintenance cost will be non-zero).
   • This is specific to Github, a slightly different integration would be needed for Gitlab, etc.
   • You can publish from your own system in this setup.

So far that's my incomplete thoughts.. Maybe there are other options. Maybe simple tokens isn't so bad. Maybe we should do both (1) and (2). And maybe this is not the most urgent feature :)

[AKushWarrior]

@jonasfj thanks for taking the time to write that out.

RE Simple tokens: As long as you can regenerate them easily, I don't think there is too much risk there. But user-centric security errors will see a spike.

RE 1: I agree that it can coincide with 2. On the other hand, it is more of a supplementary than a full solution.

RE 3: I didn't know it would be that significant an overhead. Obviously CI publish is a fairly small feature, so GitHub integration should probably be way down the list.

Maybe, since simple tokens would be relatively easy, you could try rolling that out. In the case that it fails spectacularly (breaches and leaks everywhere), it should also be simple to roll back.

Of course, all this is not the most urgent thing. Still, it's a nice feature to have. As a package dev, I can say firsthand that pushing to both GitHub and pub is an annoying hindrance, as opposed to CI.

[aeneasr]

Is there any guideline on publishing a package over CI at the moment? Using an expiring access token does not seem like a permanent solution. I could not find anything in the docs or on Google (except some attempts using the access token from credentials.json which I assume expires after a couple of hours).

[jonasfj]

@aeneasr, the credentials.json file contains a refresh token too, but I wouldn't recommend this approach. This credentials are linked your personal account and not suitable for publishing from CI.

It's recommended that packages be published manually until a solution have been found.

[aeneasr]

Thank you for the info! Would the refresh token not be invalidated once used? Or is it valid until forever? Unfortunately, using non-automation/manual is not an option for us except for not supporting Dart, which is also not an option :)

[jonasfj]

Would the refresh token not be invalidated once used? Or is it valid until forever?

It's an oauth token refresh token, if you sign-in to pub many places then eventually it'll be invalidated -- I think. But I've been using pub on my laptop for more than a year without being asked to sign-in again, so I think it's reasonably stable.

[aeneasr]

Thank you for the clarification. I assume pub update the local access and refresh token in credentials.json once expired, which isn't an option for CI, as the filesystem is ephemeral. But I'll try it out and see how it goes :)

[jonasfj]

pub only changes the refresh_token when it has been expired, and that requires user interaction. In my experience it's very rare, I have seen someone people using credentials.json as an encrypted file on travis. They often have a dedicated Google account for this purpose, so that it's not sign-in to pub anywhere else.

But yes, this is not designed for CI publishing, even if it's technically possible.

[aeneasr]

Awesome, thank you again for the clarification and super quick turnover!

[dotdoom]

I'm coming in late, but I read the thread and I wonder if we could be a bit more precise here. So far what I understood is:

1. There's no recommended way to publish packages to pub.dev from CI (there's no officially documented way, and the current community-built workaround requires exposing personal account OAuth token)
2. This request is about making this possible (and not just easier)

Does this summary sound reasonable? Should we rename to simply "Pub publish should support CI" to carry out the sentiment?

[sigurdm]

Should we rename to simply "Pub publish should support CI" to carry out the sentiment?

I edited the title to be a bit more precise.

[jonasfj]

Token authentication have been introduced, this is really just a matter of supporting it for pub.dev.

[jonasfj]

Refiled as -> dart-lang/pub-dev#5388