Support all available endpoints for Github App with preview request

[tkbky]

Not able to access all available endpoints with Github App. Only certain endpoints that are accessible because they have this custom media type set internally.

According to this doc, accessing the Github API with Github App requires custom media type application/vnd.github.machine-man-preview+json. Currently, there isn't a way to set this.

[tkbky]

cc/ @bitwiseman

[bitwiseman]

@tkbky
What endpoints are you trying to access and what methods in this library are you calling to do so?

This has been added recently, but not released yet: #522

[tkbky]

I'm trying to create a pull request using installation token, which is expose via a GHRepository instance. It does not work because GET /:repos/:owner/:repo has to be a user-to-server request, while the endpoint to create a pull request works fine with installation token.

My current workaround is have a method in GitHub to set the authorization token with installation token, and expose a method in GHPullRequest that creates a pull request.

[bitwiseman]

@tkbky
By installation token, do you mean like this? https://github.com/github-api/github-api/pull/522/files#diff-f03746f715e382c39138f17b1dfa8ce8R141

[tkbky]

@bitwiseman
Not the JWT token, it's the GHAppInstallationToken.

[PauloMigAlmeida]

Hi @tkbky,

I am not entirely sure if I understood what your use case is and what is causing the issue. Would you care to share with us some code snippets (stacktraces too if applicable) of how you're trying to do it?

[PauloMigAlmeida]

@bitwiseman Please feel free to cc me on any Github App-related issue...I will be more than happy to help anyone with this feature of the SDK.

@tkbky Okay, I think I finally got what is going on....but we need to address a meta-consideration or two before we can discuss a solution.

I'm trying to create a pull request using installation token, which is expose via a GHRepository instance. It does not work because GET /:repos/:owner/:repo has to be a user-to-server request, while the endpoint to create a pull request works fine with installation token.

IMHO, I think you may have misunderstood the docs. GET /:repos/:owner/:repo is GithubApp ready which means it plays nice with GitHub App's functionality

I use this on an internal application at work..and this is how it's defined (curated function) as shown below:

void postComments(String oAuthToken, AnalysisRequestModel model, String... comments) throws *****Exception {
    var gitHub = new GitHubBuilder().withOAuthToken(oAuthToken).build();
    var repository = gitHub.getRepository(model.getRepositoryFullname());
    var pullRequest = repository.getPullRequest(model.getPullRequestNumber());
    for (String comment : comments) {
        pullRequest.comment(comment);
    }
}

Again, If you could share how you're doing it, app's permissions and log we may be able to better assist you.

My current workaround is have a method in GitHub to set the authorization token with installation token, and expose a method in GHPullRequest that creates a pull request.

We need to unpack this:

1 :: Although Github isn't much upfront about it, the InstallationToken is essentially an OAuth 2.0 token as it does follow the same pre-reqs as defined at the RFC6749... and will eventually be sent as the HTTP Header Authorization: token 123412341234.
Having said that, using the GitHubBuilder().withOAuthToken can't be classified as a workaround.

2 :: Creating a method in GHPullRequest class that creates a pull request sounds a bit funny. Given the foundation structure that this sdk has been built upon, it would (again, IMHO) make more sense to use the GHRepository to do it.
I confess that (using GHApp) I haven't had the need to create a PR that way but it should work just fine. If you have any difficulties with that share what you have and let's debug it together. :)

Not able to access all available endpoints with Github App. Only certain endpoints that are accessible because they have this custom media type set internally.
According to this doc, accessing the Github API with Github App requires custom media type application/vnd.github.machine-man-preview+json. Currently, there isn't a way to set this.

My interpretation of this is that "for accessing the GitHub APP-related API" you need to set the media-type to application/vnd.github.machine-man-preview+json.

For instance, the code-snippet I shared with you shouldn't work as GHIssue.comment doesn't have any media-type set. The same applies to many other interactions such as getRepository, getPullRequest and so on... and they all work.

In case you have an example where this media-type is required just because you're accessing it via GitHub App (which would potentially prove me wrong), I would happily work on a PR to expand this feature's integration with the current SDK's classes.

[tkbky]

@PauloMigAlmeida

Thanks for breaking this down with thorough explanation. Appreciate that. 😃

IMHO, I think you may have misunderstood the docs. GET /:repos/:owner/:repo is GithubApp ready which means it plays nice with GitHub App's functionality

Yes. While it's enabled for Github App, it has to be a user-to-server request when accessing non-organization repository.

Although Github isn't much upfront about it, the InstallationToken is essentially an OAuth 2.0 token as it does follow the same pre-reqs as defined at the RFC6749... and will eventually be sent as the HTTP Header Authorization: token 123412341234.
Having said that, using the GitHubBuilder().withOAuthToken can't be classified as a workaround.

While it's not considered as a workaround, A clear separation between the two would be good, because they are behaving differently in some cases, i.e. the server-to-server, and user-to-server request.

Creating a method in GHPullRequest class that creates a pull request sounds a bit funny. Given the foundation structure that this sdk has been built upon, it would (again, IMHO) make more sense to use the GHRepository to do it.
I confess that (using GHApp) I haven't had the need to create a PR that way but it should work just fine. If you have any difficulties with that share what you have and let's debug it together. :)

Do agree that my workaround sounds weird 😄. What I'm trying to do is make creating a pull request not depending on a GHRepository, because my need of getting a repository instance using an installation token doesn't work.

i.e.

var github = new GitHubBuilder().withLogin("unused").withOAuthToken(installationToken).build(); # Note the forceful non-null for `login` is to skip `getMyself().getLogin();` which would result in Error: "Resource not accessible by integration"
var repository = github.getRepository(repositoryName); # Error: "Resource not accessible by integration"
repository.createPullRequest(...) # without repository, this does not work too.

My interpretation of this is that "for accessing the GitHub APP-related API" you need to set the media-type to application/vnd.github.machine-man-preview+json.

That's just my interpretation of the Github API doc, but it doesn't seems like it needs to. 😅

--

After all, the problem that I have becomes: How do I create a pull request using a Github App installation token?

[PauloMigAlmeida]

Hi @tkbky I will take a look at it when I get home today.

[PauloMigAlmeida]

@tkbky Thanks for provided code-snippets, I could reproduce it locally now.

I'm trying to create a pull request using installation token, which is exposed via a GHRepository instance. It does not work because GET /:repos/:owner/:repo has to be a user-to-server request, while the endpoint to create a pull request works fine with installation token.

IMHO, I think you may have misunderstood the docs. GET /:repos/:owner/:repo is GithubApp ready which means it plays nice with GitHub App's functionality

Yes. While it's enabled for Github App, it has to be a user-to-server request when accessing non-organization repository.

I got now where the source of confusion is. Here is what you should take into consideration:

GET /:repos/:owner/:repo is not a user-to-server request

When we install a GithubApp on a user account or organisation (Let's call it GHPerson), we have to specify the repositorySelection attribute. The possible values are ALL or a list of repositories.

It's paramount that you understand that repositorySelection refers to private repositories that in the case of a user will be the private repos created on that account, or in the case of an organisation will be the private repos within that organisation.

We can infer two important things out of that:

1. Any private repository outside of the specified on repositorySelection is out-of-scope of the GitHub App, hence you should get an error when trying to access it. This is, after all, its value proposition.
2. Public repositories are accessible via GitHub App as long as they part of the same user/org. Public repositories that are from other users/orgs fall into the same security execution flow as item 1. This is due to the Github API design rather than an issue with the github-api sdk for java.

This is an example of how to create a PR on a repository which is part of the permission scope of the GithubApp:

String jwtToken = createJWT("44435", 600000); //sdk-github-api-app-test
GitHub gitHubApp = new GitHubBuilder().withJwtToken(jwtToken).build();

GHApp app = gitHubApp.getApp();
for (GHAppInstallation appInstallation : app.listInstallations()) {

    Map<String, GHPermissionType> permissions = new HashMap<>();
    permissions.put("pull_requests", GHPermissionType.WRITE);

    GHAppInstallationToken appInstallationToken = appInstallation
                    .createToken(permissions)
                    .create();

    GitHub githubAuthAsInst = new GitHubBuilder()
                    .withOAuthToken(appInstallationToken.getToken(), "")
                    .build();

    GHRepository repository = githubAuthAsInst.getRepository("PauloMigAlmeida/github-api");
    repository.createPullRequest(
                    "Test PR", "PauloMigAlmeida-patch-1", 
                    "master", "Test create PR via Github App");
}

PR created via GithubApp: PauloMigAlmeida#1

Any out-of-scope repository should return an HTTP status code 403 with the message:

Exception in thread "main" org.kohsuke.github.HttpException: 
{"message":"Resource not accessible by integration","documentation_url":"https://developer.github.com/v3/pulls/#create-a-pull-request"}

Having said that, we can safely come to the conclusion that GET /:repos/:owner/:repo is not a user-to-server request and any error obtained while playing with it is related to either a permission issue or it's out-of-scope of the Github App due to its security design.

While it's not considered as a workaround, A clear separation between the two would be good, because they are behaving differently in some cases, i.e. the server-to-server, and user-to-server request.

I see. While a "clear separation" is always a good thing to bear in mind, it would be beneficial if you could consider evaluating a couple of other points too.

SDK design: the github-api sdk was designed based on the idea that classes like GHRepository are somewhat a wrapper of the Github API endpoints. Those classes receive an instance of GitHub class which is supposed to be configured with an authentication method meant to access that wrapper. For instance:

GitHub gitHubAnonymous = GitHub.connectAnonymously();
GitHub gitHubApp = new GitHubBuilder().withJwtToken(jwtToken).build();
GitHub gitHubAsUser = new GitHubBuilder().withPassword("my_user","my_password").build();
GitHub githubAuthAsInst = new GitHubBuilder().withOAuthToken("appInstallationToken", "").build();
 
// Should work because we can anonymously get a public repository
GHRepository repository = gitHubAnonymous.getRepository("PauloMigAlmeida/github-api");
// Should work because I installed my github app on my user and allowed it to be accessed
GHRepository repository = githubAuthAsInst.getRepository("PauloMigAlmeida/github-api");
// Should not work because my github app isn't installed on the nfscan org even though this repo is public
GHRepository repository = githubAuthAsInst.getRepository("nfscan/nfscan");
// Should not work as GitHub app (with JWT token) isn't meant to access it
GHRepository repository = gitHubApp.getRepository("PauloMigAlmeida/github-api");
// Should work becuase the repository is public and because I have access to it too
GHRepository repository = gitHubAsUser.getRepository("PauloMigAlmeida/github-api");

So it is up to the developer to diligently pass the right GitHub object containing sufficient permissions so that the org.kohsuke.github.Requester class send it to Github API.

backwards-compatibility: This sdk has the peculiarity of being reasonably old and yet still being relevant to the java community. However, it was conceived when GitHub didn't have many of the features it has today which implies that certain decisions in the past affect our implementation freedom if we want to maintain backwards-compatibility with previous versions of the SDK.

Deprecating classes/methods are a way of doing it up to a certain point only. Don't forget about 2 things:
1 - We are still on version 1.xx which means that according to semantic versioning, we must keep the API contract until we decide to release a major version. That implies that for the time being multiple versions of classes/methods would have to be maintained which isn't desirable and it can make maintainers/contributors/developers grumpy.
2 - Some vendors usually create API contract sunset mechanisms which can get quite tricky.

My point is given all the design/maintainability principles applied to this sdk so far the easiest way would be to write getting started/usage docs so that other developers can grasp it rapidly rather than redesigning/reimplementing this sdk. I must confess I would love to sit with many of you guys and discuss the 2.0 version of this sdk but this is definitely outside the scope of this PR. Let's catch up on it when you feel like doing it.

i.e.

 var github = new GitHubBuilder().withLogin("unused").withOAuthToken(installationToken).build(); > # Note the forceful non-null for `login` is to skip `getMyself().getLogin();` which would result in Error: "Resource not accessible by integration"
var repository = github.getRepository(repositoryName); # Error: "Resource not accessible by integration"
repository.createPullRequest(...) # without repository, this does not work too.

I'm preassuming that you created the withLogin() method locally. You can obtain the same result by calling .withOAuthToken("appInstallationToken", "");

Last but not least,

@bitwiseman Even though this is not impacting anyone using the SDK currently, what do think about creating a method on GithubBuilder that makes people wrap their heads around how to use the GithubApp right away? .... something like:

new GitHubBuilder().withAppToken("myToken).build();

The internal implementation would be something akin to:

public GitHubBuilder withAppToken(String token){
   return withOAuthToken(token, "");
}

While this is just for readability/cosmetic purposes, this can also reduce future misunderstandings with the usage the sdk. I will defer the final decision to you and in case you are in favour of that, I will happily propose a more elaborated PR on that.

@tkbky Happy coding guys :)

[tkbky]

@PauloMigAlmeida Thanks for taking time for all these detailed clarifications and suggestions, appreciate that.

[bitwiseman]

@PauloMigAlmeida
I'm completely open to what you describe. The more the API helps people and guides people in a good direction, the better. The same with getting started and user guides.

I've been focusing on getting CI working and clearing the backlog of PRs.

Open to discussions of ideas for 2.x API but my time is limited.

[timja]

Is there any documentation for this @PauloMigAlmeida ?

I'm specifically looking for if the SDK has any classes for handling the initial GitHub app authentication to get the installation token:
https://developer.github.com/apps/building-github-apps/authenticating-with-github-apps/#authenticating-as-an-installation

[PauloMigAlmeida]

@tkbky any time :)

BItwiseman says -> I'm completely open to what you describe. The more the API helps people and guides people in a good direction, the better. The same with getting started and user guides.
I've been focusing on getting CI working and clearing the backlog of PRs.

Open to discussions of ideas for 2.x API but my time is limited.

@bitwiseman Great, I'm gonna work on a PR later today which will add that convenience method and also some docs around its usage.
You've been doing a great job, I can tell that ;)
Regarding the discussion of ideas for 2.x API... let me know which format/cadence works for you and we can make that work. I'm flexible with time ... (my timezone is NZDT)

@timja I'm working on the documentation piece for this feature to make it easier for future developers. However, answering your question. Yes, the sdk has classes for dealing with GitHub app auth to get the installation token.

I had posted this piece of code in this thread but to save you from reading it all, I will post it again :)

String jwtToken = createJWT("44435", 600000); //sdk-github-api-app-test
GitHub gitHubApp = new GitHubBuilder().withJwtToken(jwtToken).build();

GHApp app = gitHubApp.getApp();
for (GHAppInstallation appInstallation : app.listInstallations()) {

    Map<String, GHPermissionType> permissions = new HashMap<>();
    permissions.put("pull_requests", GHPermissionType.WRITE);

    GHAppInstallationToken appInstallationToken = appInstallation
                    .createToken(permissions)
                    .create();

    GitHub githubAuthAsInst = new GitHubBuilder()
                    .withOAuthToken(appInstallationToken.getToken(), "")
                    .build();

    GHRepository repository = githubAuthAsInst.getRepository("PauloMigAlmeida/github-api");
    repository.createPullRequest(
                    "Test PR", "PauloMigAlmeida-patch-1", 
                    "master", "Test create PR via Github App");
}

We decided not to create methods for generating the JWT in the SDK as we felt that this isn't the sdk's value-proposition and everyone should use the JWT generation libraries/methods they see fit.

Having said that, createJWT is a method defined in my application's source code (originally created here) that takes the Github Account Id and the TTL of the JWT token. This is its definition in case that helps you:

<dependency>
  <groupId>io.jsonwebtoken</groupId>
  <artifactId>jjwt-api</artifactId>
  <version>0.10.5</version>
</dependency>
<dependency>
  <groupId>io.jsonwebtoken</groupId>
  <artifactId>jjwt-impl</artifactId>
  <version>0.10.5</version>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>io.jsonwebtoken</groupId>
  <artifactId>jjwt-jackson</artifactId>
  <version>0.10.5</version>
  <scope>runtime</scope>
</dependency>

static PrivateKey get(String filename) throws Exception {
    byte[] keyBytes = Files.toByteArray(new File(filename));

    PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(keyBytes);
    KeyFactory kf = KeyFactory.getInstance("RSA");
    return kf.generatePrivate(spec);
}

static String createJWT(String githubAppId, long ttlMillis) throws Exception {
    //The JWT signature algorithm we will be using to sign the token
    SignatureAlgorithm signatureAlgorithm = SignatureAlgorithm.RS256;

    long nowMillis = System.currentTimeMillis();
    Date now = new Date(nowMillis);

    //We will sign our JWT with our ApiKey secret
    Key signingKey = get("sdk-github-api-app-test.2019-10-23.private-key.der");

    //Let's set the JWT Claims
    JwtBuilder builder = Jwts.builder()
            .setIssuedAt(now)
            .setIssuer(githubAppId)
            .signWith(signingKey, signatureAlgorithm);

    //if it has been specified, let's add the expiration
    if (ttlMillis > 0) {
        long expMillis = nowMillis + ttlMillis;
        Date exp = new Date(expMillis);
        builder.setExpiration(exp);
    }

    //Builds the JWT and serializes it to a compact, URL-safe string
    return builder.compact();
}

PS.: In order to generate a JWT you must use the signing key that your Github app gives you. The problem is that Java doesn't play nice with pem keys, so you have 2 options.: Either you convert it to der (my preferred approach) or to use the bouncycastle library to do the dirty conversion work.

To convert the Github App pem key I did:

openssl pkcs8 -topk8 -inform PEM -outform DER -in sdk-github-api-app-test.2019-10-23.private-key.pem -out sdk-github-api-app-test.2019-10-23.private-key.der -nocrypt

Hope that this helps you.

[timja]

Thanks a lot!
I’m looking at adding GitHub app support to the github-branch-source-plugin. I have it working hardcoding the Oauth token so this will be very helpful.

Another question, there’s nothing in the SDK for automatically generating a new installation token when the old one expires right? I couldn’t see something like a supplier or function that could be passed to generate the oauth token

[PauloMigAlmeida]

Thanks a lot!
I’m looking at adding GitHub app support to the github-branch-source-plugin. I have it working hardcoding the Oauth token so this will be very helpful.

That's pretty cool! Looking forward to seeing this implemented :)

Another question, there’s nothing in the SDK for automatically generating a new installation token when the old one expires right? I couldn’t see something like a supplier or function that could be passed to generate the oauth token

Not really. This SDK works as a stateless wrapper of the GitHub endpoints which means that we don't store any of those generated tokens. It's up to the developer to decide when to call createToken method when the same is expired or about to expire.

We do give developers access to the expireAt property so that you can implement this logic as you see fit.
https://github.com/github-api/github-api/blob/master/src/main/java/org/kohsuke/github/GHAppInstallationToken.java#L74-L76

[ojacques]

I’m looking at adding GitHub app support to the github-branch-source-plugin. I have it working hardcoding the Oauth token so this will be very helpful.

Sorry to hijack the thread, but I'm very interested by this @timja (and have Jenkins act as a GitHub app to read repos / update status). Let me know if / how I can help.

[timja]

I’m looking at adding GitHub app support to the github-branch-source-plugin. I have it working hardcoding the Oauth token so this will be very helpful.

Sorry to hijack the thread, but I'm very interested by this @timja (and have Jenkins act as a GitHub app to read repos / update status). Let me know if / how I can help.

I got distracted from this and haven't got back to it recently, I'll try get back to it next week otherwise there's a branch here:
https://github.com/jenkinsci/github-branch-source-plugin/compare/master...timja:github-app-support?expand=1

It needs docs, tests and adjusting the api it calls to check whether the token is valid, currently it tries to retrieve the current user's details which doesn't work for a github app. It's quite a nice check though as you can show in the UI "authenticated as timja-bot"

But the code does work as is, (or at least it worked before I just merged master and bumped to the latest release of github-api)

[timja]

I’m looking at adding GitHub app support to the github-branch-source-plugin. I have it working hardcoding the Oauth token so this will be very helpful.

Sorry to hijack the thread, but I'm very interested by this @timja (and have Jenkins act as a GitHub app to read repos / update status). Let me know if / how I can help.

@ojacques I finally got back to this, there's a functional PR here:
jenkinsci/github-branch-source-plugin#269
Still needs a bit more work and a github-api-plugin release

[blacelle]

A side-note to this thread which has been very helpful to my own case:

if you want to open a PR over a private repository, in order to get the GHRepository instance, you need as permissions in the installationToken:

Map.of(
    "pull_requests", GHPermissionType.WRITE,
    "metadata", GHPermissionType.READ
)

Quite interestingly, without the READ permission over "metadata", you can still access the GHRepository instance through github.getOrganization(someOrganization).listRepositories()