400 from OKHttp getInputStream

[sbhaktha]

Hello,

I have built a tool using your API, which has been working for many months now, but suddenly fails due to a FileNotFoundException thrown by getInputStrem in OKHttp. When I paste the url it is trying to download, into my browser, it works perfectly fine. I can see the file contents. This is very strange. Do you have any clue about this?

Stack trace:

tables [ERROR] [07/12/2016 12:15:41.086] [TableService-akka.actor.default-dispatcher-8] [akka://TableService/user/simple-service-actor] Error during processing of request HttpRequest(POST,http://localhost:8084/api/metadata/list,List(Host: localhost:8084, Connection: keep-alive, Content-Length: 59, Origin: http://localhost:8084, X-Requested-With: XMLHttpRequest, User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36 OPR/38.0.2220.41, Content-Type: application/json, Referer: http://localhost:8084/, Accept-Encoding: gzip, deflate, lzma, Accept-Language: en-US, en;q=0.8, Cookie: tablestore-data-git=eyJhY2Nlc3NUb2tlbiI6ImYyZDI1OTViNGQ4OTMwMjIwMGI2YjE2MjNhNzQ3ZjQ3MzI1ZWJkOWMifQ==; tablestore-data-git2=eyJhY2Nlc3NUb2tlbiI6ImZlYmMxYTRlYjA1MmI4OWJlNDQwOWFmODk3OGNkYWFhYWY1OWZlNWEifQ==),HttpEntity(application/json,{"repo":"aristo-tables","fork":"allenai","branch":"master"}),HTTP/1.1)
tables java.io.FileNotFoundException: 400: Invalid request
tables 
tables  at org.kohsuke.github.Requester.handleApiError(Requester.java:527)
tables  at org.kohsuke.github.Requester.asStream(Requester.java:293)
tables  at org.kohsuke.github.GHContent.read(GHContent.java:118)
tables  at org.allenai.ari.tables.GitHubUtil$$anonfun$getCsvFileLocal$1.apply(GitHubUtil.scala:420)
tables  at org.allenai.ari.tables.GitHubUtil$$anonfun$getCsvFileLocal$1.apply(GitHubUtil.scala:417)
tables  at scala.Option.foreach(Option.scala:257)
tables  at org.allenai.ari.tables.GitHubUtil.getCsvFileLocal(GitHubUtil.scala:417)
tables  at org.allenai.ari.tables.GitHubUtil$$anonfun$7.apply(GitHubUtil.scala:239)
tables  at org.allenai.ari.tables.GitHubUtil$$anonfun$7.apply(GitHubUtil.scala:236)
tables  at scala.collection.parallel.mutable.ParArray$ParArrayIterator.flatmap2combiner(ParArray.scala:417)
tables  at scala.collection.parallel.ParIterableLike$FlatMap.leaf(ParIterableLike.scala:1072)
tables  at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)
tables  at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
tables  at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
tables  at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51)
tables  at scala.collection.parallel.ParIterableLike$FlatMap.tryLeaf(ParIterableLike.scala:1068)
tables  at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.internal(Tasks.scala:159)
tables  at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:443)
tables  at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:149)
tables  at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)
tables  at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
tables  at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
tables  at scala.concurrent.forkjoin.ForkJoinTask.doJoin(ForkJoinTask.java:341)
tables  at scala.concurrent.forkjoin.ForkJoinTask.join(ForkJoinTask.java:673)
tables  at scala.collection.parallel.ForkJoinTasks$WrappedTask$class.sync(Tasks.scala:378)
tables  at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:443)
tables  at scala.collection.parallel.ForkJoinTasks$class.executeAndWaitResult(Tasks.scala:426)
tables  at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:56)
tables  at scala.collection.parallel.ExecutionContextTasks$class.executeAndWaitResult(Tasks.scala:558)
tables  at scala.collection.parallel.ExecutionContextTaskSupport.executeAndWaitResult(TaskSupport.scala:80)
tables  at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:958)
tables  at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)
tables  at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
tables  at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
tables  at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51)
tables  at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:953)
tables  at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:152)
tables  at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)
tables  at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
tables  at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
tables  at scala.concurrent.forkjoin.ForkJoinTask.doJoin(ForkJoinTask.java:341)
tables  at scala.concurrent.forkjoin.ForkJoinTask.join(ForkJoinTask.java:673)
tables  at scala.collection.parallel.ForkJoinTasks$WrappedTask$class.sync(Tasks.scala:378)
tables  at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:443)
tables  at scala.collection.parallel.ForkJoinTasks$class.executeAndWaitResult(Tasks.scala:426)
tables  at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:56)
tables  at scala.collection.parallel.ExecutionContextTasks$class.executeAndWaitResult(Tasks.scala:558)
tables  at scala.collection.parallel.ExecutionContextTaskSupport.executeAndWaitResult(TaskSupport.scala:80)
tables  at scala.collection.parallel.ParIterableLike$class.flatMap(ParIterableLike.scala:513)
tables  at scala.collection.parallel.mutable.ParArray.flatMap(ParArray.scala:56)
tables  at org.allenai.ari.tables.GitHubUtil.org$allenai$ari$tables$GitHubUtil$$getTableMetadata(GitHubUtil.scala:236)
tables  at org.allenai.ari.tables.GitHubUtil$$anonfun$getTableMetadata$2.apply(GitHubUtil.scala:229)
tables  at org.allenai.ari.tables.GitHubUtil$$anonfun$getTableMetadata$2.apply(GitHubUtil.scala:229)
tables  at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
tables  at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:249)
tables  at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
tables  at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
tables  at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
tables  at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
tables  at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
tables  at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
tables  at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
tables  at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
tables  at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:399)
tables  at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
tables  at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
tables  at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
tables  at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
tables Caused by: java.io.FileNotFoundException: https://raw.githubusercontent.com/allenai/aristo-tables/master/tables/abstract_concrete/abstract_concrete.csv?token=AFXsrivOTIoDru9Htw9aNTJbprmE5m0Gks5XhUHfwA%3D%3D
tables  at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:228)
tables  at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getInputStream(DelegatingHttpsURLConnection.java:210)
tables  at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:25)
tables  at org.kohsuke.github.Requester.asStream(Requester.java:291)
tables  ... 66 more

Your help is greatly appreciated,
Sumithra

[Shredder121]

You might want to make sure that token is revoked, it might expire on its own though.

But I don't see that repository anywhere.
https://github.com/allenai?utf8=✓&query=table
https://github.com/allenai?utf8=✓&query=aristo

If it's a private repository, make sure that in the flow you ask for the right permissions, so that the token you receive from GitHub can actually be used for your usecase. (I believe that would be repo)

[sbhaktha]

@Shredder121 : Thanks for your response.
Yes, it automatically expires. When I paste the url of the file it's trying to access programmatically into the browser, it works, but only for some time, and then expires.

The repo I am accessing is a private repo and I am using OAuth to access it. I have permissions to access it and outside of this tool, do not have issues using the repo in any way.

[Shredder121]

Could you, for science, build in a small delay between obtaining the token and using it?
Maybe you're too quick for GitHub.

And maybe that's why manually browsing to it works, because that has a long enough delay.

[sbhaktha]

The way my program operates is - at the start of the session it authenticates to GitHub using OAuth and obtains the token. Then, it gets all directories in the repo, using the token. So far so good. Then, it iterates over these directories and gets files from each directory (again using the token). At this point it fails to obtain that file although the GHContent object corresponding to each directory came back containing the files that are supposed to be in there. for kicks I tried to add a delay anyway, in the loop where I iterate over the directories, to make sure we're not too quick for GitHub, but that did not fix it. :-(

[sbhaktha]

Any other ideas?

[Shredder121]

Just wondering, what version of okhttp-urlconnection do you use?

[sbhaktha]

I actually meant to close this later last night. I was on version 1.72. I switched to the latest (1.76) and that resolved the issue. It's a little weird because.this code has been untouched for several months and it used to work (even with 1.72). Maybe GitHub made some breaking change underneath?

I noticed this issue: #215, which could've led to the symptom I was seeing, however I confirmed that version 1.72 had the fix for it. So it has to be some later commit that changed something. Not sure which one but glad that updating the version fixes this issue, for now.

Thank you for your responses!

[Shredder121]

Maybe it was the issue I reported here?
It was fixed again in dbc79f8, which was 1.74.

Anyway, indeed, glad that updating the version fixed it for you.

[sbhaktha]

Yes, possible! Thanks. Still a mystery why it was working earlier but suddenly stopped.

[Shredder121]

I can confirm that GitHub changed the way they handled (POST) requests.
The issue I had was that I didn't query raw.githubusercontent.com, but instead used cdn.rawgit.com.
RawGitOkHttpConnector.
This made me realize faster that OkHttp switched to POST, since rawgit doesn't allow POSTs.
So, right now I changed my version back to 1.72, and removed RawGitOkHttpConnector, and the tests don't pass anymore for me as well.

All in all, that means that was the issue, so it's fixed for you, and not just 'for now' but for always.