Retest PR cause Github API call failed with 404 #441

richard1122 · 2016-11-28T12:09:42Z

After comments on a Github PR "retest this please". Jenkins could not rebuild, and the following exception is logged in Jenkins log page.

It might be https://api.github.com/repos/<UserName>/<RepoName>/pulls/4/commits .

Plugin version: 1.33.2

A thread (Thread-9/80) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.Error: java.io.FileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}
	at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:433)
	at org.kohsuke.github.Requester$PagingIterator.hasNext(Requester.java:400)
	at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:44)
	at org.kohsuke.github.PagedIterator.hasNext(PagedIterator.java:32)
	at org.jenkinsci.plugins.ghprb.GhprbPullRequest.tryBuild(GhprbPullRequest.java:365)
	at org.jenkinsci.plugins.ghprb.GhprbPullRequest.check(GhprbPullRequest.java:214)
	at org.jenkinsci.plugins.ghprb.GhprbRepository.onIssueCommentHook(GhprbRepository.java:346)
	at org.jenkinsci.plugins.ghprb.GhprbTrigger.handleComment(GhprbTrigger.java:618)
	at org.jenkinsci.plugins.ghprb.GhprbRootAction$1.run(GhprbRootAction.java:233)
Caused by: java.io.FileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}
	at org.kohsuke.github.Requester.handleApiError(Requester.java:592)
	at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:429)
	... 8 more
Caused by: java.io.FileNotFoundException: https://api.github.com/repos/<UserName>/<RepoName>/issues/4/commits
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
	at org.kohsuke.github.Requester.parse(Requester.java:524)
	at org.kohsuke.github.Requester.access$200(Requester.java:68)
	at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:424)
	... 8 more
Caused by: java.io.FileNotFoundException: https://api.github.com/repos/<UserName>/<RepoName>/issues/4/commits
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1872)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
	at org.kohsuke.github.Requester.parse(Requester.java:514)
	... 10 more

The text was updated successfully, but these errors were encountered:

benpatterson · 2016-11-28T13:02:46Z

Richard * What version of the plugin are you using? * Did you check your Github access for the user/token you are using for the plugin? (Check the global configuration and the job) * How about the connectivity between your Jenkins instance and Github? In a vacuum this looks like either a permissions issue or a network issue.

richard1122 · 2016-11-28T13:50:40Z

Hi,

Plugin version: 1.33.2
I've tested the token with curl, the https://api.github.com/repos/<UserName>/<RepoName>/issues/4/commits returns 404, and https://api.github.com/repos/<UserName>/<RepoName>/pulls/4/commits returns fine. (change issues to pulls).
I've also tested in jenkins configuration page with the same repo:
Connectivity is not very well, but just slow.

benpatterson · 2016-11-28T14:14:28Z

Thanks @richard1122

richard1122 · 2016-11-28T14:21:06Z

It's a private repo inside an organization. And I'm the owner of that organization.

richard1122 · 2016-11-28T14:24:39Z

Here are all the scopes used.

sjstyle · 2016-11-29T15:42:08Z

Hi, I am also suffering from this issue.
I'm using Github Enterprise 2.8 with Github Pull Request Builder plugin 1.33.2
(but, this also occurs at the master branch)

In my opinion, this is more complex issue because it related to github-api.
This error is occurs at this point ( https://github.com/jenkinsci/ghprb-plugin/blob/master/src/main/java/org/jenkinsci/plugins/ghprb/GhprbPullRequest.java#L365 )
by calling listCommits.

But, github's hook returns api url /repos/:user/:repo/issues/:pr_number not /repos/:user/:repo/pulls/:pr_number.

github-api project's GHPullRequest implements are ( https://github.com/kohsuke/github-api/blob/master/src/main/java/org/kohsuke/github/GHPullRequest.java#L250 ).
At this point, we neither could modify PR's api url nor call user defined url.

aboivin · 2016-11-30T17:07:59Z

+1 having same issue

benpatterson · 2016-12-01T11:21:25Z

@richard1122 if you have a cycle, could you see if this was a problem in 1.33.1? We upgraded the github-api dependency with the last release. However, grepping the commit history on the api library, nothing is sticking out to me. So, I/m trying to determine the depth of this problem.

Also, is it true that everyone on this list is using GH Enterprise? If someone is using GH public, please let me know? (I will verify on GH public when I have time, but I'm not seeing that happen for a few days at least, so thought I'd ask.)

nMustaki · 2016-12-01T12:22:42Z

@benpatterson I am using github public on a private repo and I have this very issue. Tell me if I can do anything, I am very motivated on this one :-)

sjstyle · 2016-12-01T12:25:02Z

@benpatterson
I just solved this issue. In my case, it occurred when I set the label in Github PR Builder.
I will comment more detailed source trace and pull request soon.

benpatterson · 2016-12-01T13:02:16Z

Great news, Sang. We added label capability with this release so your conclusion is certainly plausible. I'll hold tight on a deeper dive, then.

nMustaki · 2016-12-01T13:42:57Z

I have no label on my side. Currently:

PR are built when a new commit is pushed
PR are not built when there are created or when we request a build through a comment

sjstyle · 2016-12-01T13:59:59Z

@nMustaki
Did you mean you leaving blank on List of github labels for which the build should not be triggered. to ghprb's advanced option?
I think it may be checking building phase, or webhooks has permission on issue_comment.

nMustaki · 2016-12-01T14:18:31Z

@sjstyle yes it's blank, not even a space, on both jenkins/configuration and myproject/configuration

sjstyle · 2016-12-01T14:36:56Z

@nMustaki
I think you better check Issue Comment in Webhook option on your GitHub Repository.

On this issue, FileNotFoundException occurs in Jenkins log. But I doubt your case just didn't receive any comment event on pull request.

nMustaki · 2016-12-01T14:54:23Z

Issue comment is checked, I have the same issue as you (sadly)

Pull request #807 was updated/initialized on ORG/REPO at 12/1/16 1:21 PM by Nathan Mustaki (comment)

Dec 01, 2016 1:21:31 PM SEVERE hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException

A thread (Thread-72/1006) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.Error: java.io.FileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}
    at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:433)
    at org.kohsuke.github.Requester$PagingIterator.hasNext(Requester.java:400)
    at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:44)
    at org.kohsuke.github.PagedIterator.hasNext(PagedIterator.java:32)
    at org.jenkinsci.plugins.ghprb.GhprbPullRequest.tryBuild(GhprbPullRequest.java:365)
    at org.jenkinsci.plugins.ghprb.GhprbPullRequest.check(GhprbPullRequest.java:214)
    at org.jenkinsci.plugins.ghprb.GhprbRepository.onIssueCommentHook(GhprbRepository.java:346)
    at org.jenkinsci.plugins.ghprb.GhprbTrigger.handleComment(GhprbTrigger.java:618)
    at org.jenkinsci.plugins.ghprb.GhprbRootAction$1.run(GhprbRootAction.java:233)
Caused by: java.io.FileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}
    at org.kohsuke.github.Requester.handleApiError(Requester.java:592)
    at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:429)
    ... 8 more
Caused by: java.io.FileNotFoundException: https://api.github.com/repos/ORG/REPO/issues/807/commits
    at sun.reflect.GeneratedConstructorAccessor153.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at org.kohsuke.github.Requester.parse(Requester.java:524)
    at org.kohsuke.github.Requester.access$200(Requester.java:68)
    at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:424)
    ... 8 more
Caused by: java.io.FileNotFoundException: https://api.github.com/repos/ORG/REPO/issues/807/commits
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1872)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
    at org.kohsuke.github.Requester.parse(Requester.java:514)
    ... 10 more

mattsch · 2016-12-01T17:35:46Z

Got the same behavior as @nMustaki, no labels anywhere but ended up with the same stack trace. A downgrade to 1.33.1 fixed it for me.

richard1122 · 2016-12-01T17:47:32Z

Downgrade to 1.33.1 works fine for me too.
And I'm using Github Public version.

pascalw · 2016-12-01T20:51:48Z

Just chiming in to say I'm experiencing the same issue. In my case:

A newly created PR with a single commit was not built
Adding a trigger comment didn't trigger a build either

benpatterson · 2016-12-01T21:28:45Z

I will look at this once I'm free, though it might take a couple of days. We'll see if @sjstyle beats me to it, in that case.

benpatterson · 2016-12-02T01:40:10Z

@richard1122 , et al. I believe #443 addresses this.

Thanks for doing the initial analysis; I was able to reproduce the problem, and rolled back github-api versions. The problem appears to be introduced in 1.76 so this rolls us back to 1.75. I haven't isolated the precise change, but again this should solve the problem.

Looks like I did have some cycles to look tonight. #UnderPromiseOverDeliver

Pls let me know if you have any feedback & cheers.

benpatterson · 2016-12-02T02:00:33Z

Ah and I should point out you may need to manually install the older github-api plugin when pulling in the fix.

sjstyle · 2016-12-02T02:41:52Z

@benpatterson
I solved and traced this issue by #444.

benpatterson · 2016-12-03T01:26:32Z

I see @sjstyle 's PR and will continue reviewing that. In the meantime, I've released 1.33.3 with the downgraded github-api dependency. I was able to resolve the error described in this ticket with the downgraded github-api plugin, and via reproducing the problem, narrowed down the failures to be in the 1.76 (and beyond) release of that dependency. That said, I wasn't able to narrow down to the exact offending commit on that other plugin, so I release ghprb 1.33.3 with only partial confidence that the problem is resolved. So, I'll look at the other PR that @sjstyle has referenced, when I have time in the coming days.

Please take the following steps if you're still seeing this error:

Install the 1.33.3 (or greater)
Downgrade github-api plugin manually to 1.75. You will only need to do this if you had 1.33.2 installed.
Restart the Jenkins service and re-save your job.

I'll leave this ticket open for now; ping back if the new release has had effect. Cheers.

AudriusButkevicius · 2016-12-04T12:22:04Z

So I've:
Updated GHPRB to 1.33.3
Downgraded Github API to 1.75
Was forced to downgrade Github Authentication Plugin to 0.24 (as it requires github api 1.77 otherwise)
Github plugin is still at 1.24.0 (not sure if it matters)

Yet I am still getting:

A thread (Thread-18/121) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.Error: java.io.FileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}
	at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:425)
	at org.kohsuke.github.Requester$PagingIterator.hasNext(Requester.java:392)
	at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:44)
	at org.kohsuke.github.PagedIterator.hasNext(PagedIterator.java:32)
	at org.jenkinsci.plugins.ghprb.GhprbPullRequest.tryBuild(GhprbPullRequest.java:365)
	at org.jenkinsci.plugins.ghprb.GhprbPullRequest.check(GhprbPullRequest.java:214)
	at org.jenkinsci.plugins.ghprb.GhprbRepository.onIssueCommentHook(GhprbRepository.java:346)
	at org.jenkinsci.plugins.ghprb.GhprbTrigger.handleComment(GhprbTrigger.java:618)
	at org.jenkinsci.plugins.ghprb.GhprbRootAction$1.run(GhprbRootAction.java:233)
Caused by: java.io.FileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}
	at org.kohsuke.github.Requester.handleApiError(Requester.java:553)
	at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:421)
	... 8 more
Caused by: java.io.FileNotFoundException: https://api.github.com/repos/syncthing/syncthing/issues/3765/commits
	at sun.reflect.GeneratedConstructorAccessor89.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at sun.net.www.protocol.http.HttpURLConnection$7.run(HttpURLConnection.java:1725)
	at sun.net.www.protocol.http.HttpURLConnection$7.run(HttpURLConnection.java:1723)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1721)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1292)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
	at org.kohsuke.github.Requester.parse(Requester.java:492)
	at org.kohsuke.github.Requester.access$200(Requester.java:69)
	at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:416)
	... 8 more
Caused by: java.io.FileNotFoundException: https://api.github.com/repos/syncthing/syncthing/issues/3765/commits
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1674)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
	at org.kohsuke.github.Requester.parse(Requester.java:486)
	... 10 more

Any workarounds?

benpatterson · 2016-12-04T12:44:51Z

I'm assuming you also restarted Jenkins? No other options at the moment. I've not had a chance to debug in depth. I'm hoping to have time for that this week.

AudriusButkevicius · 2016-12-04T13:01:19Z

Yeah I have restarted jenkins and resaved all the jobs.

benpatterson · 2016-12-04T13:21:36Z

OK. Thanks for pinging back.

sjstyle · 2016-12-04T13:28:37Z

@AudriusButkevicius
Can you verify my solution works?
If you installed Git, JDK 7 (or above) and maven 3, you can compile and run my solution at #444 .

git clone https://github.com/sjstyle/ghprb-plugin.git
cd ghprb-plugin
git checkout add/whitelist-label
mvn compile package -Dmaven.test.skip=true

Go to jenkins plugin manager, 'Advanced' option and upload ./target/ghprb.hpi in directory and restart jenkins.

You can roll back to previous version by jenkin's plugin manager.

AudriusButkevicius · 2016-12-04T13:45:03Z

That has a lot of things I'd rather not pull in, if there is a branch with just the fix, let me know.

benpatterson · 2016-12-05T03:33:40Z

I've narrowed down the problem to checkLabels(). Labels are only available in the github issues API.

In terms of implementation, as a way of obtaining labels through the Jenkins github-api plugin, the pr object is being rewritten with the fetchIssue call. The result is pr.url has issues in the path rather than pulls, as other folks on this list have observed. For now I'm going to plan on reverting the labels PR. (Looks like the @sjstyle fix will not resolve that issue since getLabels is called in that version of code as well.)

sjstyle · 2016-12-05T04:01:23Z

@benpatterson
My solution works because checkDirtyPullRequest() method.

This checks whether checkLabels() method calls. If it called, it calls github-api to get original pull request object and destroy rewritten issue object.

So, after checkDirtyPullRequest it verify that pull request object's path is /pulls/

benpatterson · 2016-12-05T12:31:55Z

Thanks for pointing that out, Sang. I want to keep changes discrete in order to reduce any noise and get a working plugin out. I'm still planning on reverting the previous PR. That way, when looking at your changes, we won't be under pressure and can focus on the implementation details. Don't worry, your PR will come soon.

benpatterson · 2016-12-05T14:14:48Z

@richard1122 version 1.33.4 should solve this problem. Pls ping back in either case. Again I noticed that some data is cached (I found that the issues URL was among the items cached, and it wouldn't be cleared until a new commit arrived.) So if you still see the problem, I'd ask questions about the state of your instance, etc.

But I'm optimistic this will solve the problem.

ilovezfs · 2016-12-06T10:42:46Z

Looks like @BrewTestBot test this please is still failing after the 1.33.4 upgrade.

AudriusButkevicius · 2016-12-06T10:46:50Z

In my case the issue is fixed.

ilovezfs · 2016-12-06T10:48:01Z

@AudriusButkevicius where is your test phrase set? In the main Jenkins settings or in the particular job's settings or both?

AudriusButkevicius · 2016-12-06T10:52:29Z

I think its default whatever the plugin ships. I don't think it matters.

ilovezfs · 2016-12-06T11:18:41Z

Downgrading to 1.33.1 gets it working again for us.

benpatterson · 2016-12-06T12:34:21Z

Hi ilovezfs... Do you have a stacktrace?

ilovezfs · 2016-12-06T13:10:39Z

@benpatterson I'm not sure it's crashing as opposed to just deciding the comment doesn't warrant triggering the build after considering it.

benpatterson · 2016-12-06T13:36:45Z

Ok thanks @ilovezfs .

This sounds like a distinct issue from the one associated with this ticket. Could you open a new issue and share your configuration there? I'm particularly curious about the various phrases you've set globally and within the job configuration.

Based on @AudriusButkevicius's response, I'm going to close this ticket. Everyone's feedback along the way was extremely helpful as I dove into the issue.

AudriusButkevicius · 2016-12-06T14:14:07Z

Thanks for the prompt fix.

ilovezfs · 2016-12-06T14:14:59Z

@benpatterson voilà #450

benpatterson added bug needs verification labels Nov 28, 2016

pascalw mentioned this issue Dec 1, 2016

1.33.2: builds are not triggered #442

Closed

benpatterson added the high priority label Dec 1, 2016

benpatterson removed the needs verification label Dec 2, 2016

sjstyle mentioned this issue Dec 2, 2016

Add whitelist label option. #444

Merged

martenson mentioned this issue Dec 3, 2016

Fix bug with detection of Jenkins BUILD_NUMBER. galaxyproject/galaxy#3261

Closed

AudriusButkevicius mentioned this issue Dec 4, 2016

lib/model: little cleanups in model.go syncthing/syncthing#3765

Closed

This was referenced Dec 5, 2016

Add support for github labels ignore list #431

Merged

getLabels() call for a pull request results in downstream 404s hub4j/github-api#319

Closed

benpatterson closed this as completed Dec 6, 2016

Retest PR cause Github API call failed with 404 #441

Retest PR cause Github API call failed with 404 #441

Comments

richard1122 commented Nov 28, 2016 • edited Loading

benpatterson commented Nov 28, 2016 via email

richard1122 commented Nov 28, 2016 • edited Loading

benpatterson commented Nov 28, 2016

richard1122 commented Nov 28, 2016 • edited Loading

richard1122 commented Nov 28, 2016

sjstyle commented Nov 29, 2016

aboivin commented Nov 30, 2016

benpatterson commented Dec 1, 2016

nMustaki commented Dec 1, 2016

sjstyle commented Dec 1, 2016

benpatterson commented Dec 1, 2016 via email

nMustaki commented Dec 1, 2016

sjstyle commented Dec 1, 2016 • edited Loading

nMustaki commented Dec 1, 2016

sjstyle commented Dec 1, 2016

nMustaki commented Dec 1, 2016

mattsch commented Dec 1, 2016

richard1122 commented Dec 1, 2016

pascalw commented Dec 1, 2016

benpatterson commented Dec 1, 2016

benpatterson commented Dec 2, 2016

benpatterson commented Dec 2, 2016

sjstyle commented Dec 2, 2016

benpatterson commented Dec 3, 2016 • edited Loading

AudriusButkevicius commented Dec 4, 2016 • edited Loading

benpatterson commented Dec 4, 2016 via email

AudriusButkevicius commented Dec 4, 2016 • edited Loading

benpatterson commented Dec 4, 2016 via email

sjstyle commented Dec 4, 2016

AudriusButkevicius commented Dec 4, 2016

benpatterson commented Dec 5, 2016 • edited Loading

sjstyle commented Dec 5, 2016

benpatterson commented Dec 5, 2016 via email

benpatterson commented Dec 5, 2016

ilovezfs commented Dec 6, 2016

AudriusButkevicius commented Dec 6, 2016

ilovezfs commented Dec 6, 2016

AudriusButkevicius commented Dec 6, 2016

ilovezfs commented Dec 6, 2016

benpatterson commented Dec 6, 2016 via email • edited Loading

ilovezfs commented Dec 6, 2016

benpatterson commented Dec 6, 2016

AudriusButkevicius commented Dec 6, 2016

ilovezfs commented Dec 6, 2016

richard1122 commented Nov 28, 2016 •

edited

Loading

richard1122 commented Nov 28, 2016 •

edited

Loading

richard1122 commented Nov 28, 2016 •

edited

Loading

sjstyle commented Dec 1, 2016 •

edited

Loading

benpatterson commented Dec 3, 2016 •

edited

Loading

AudriusButkevicius commented Dec 4, 2016 •

edited

Loading

AudriusButkevicius commented Dec 4, 2016 •

edited

Loading

benpatterson commented Dec 5, 2016 •

edited

Loading

benpatterson commented Dec 6, 2016 via email •

edited

Loading