Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File write errors when writing small files #2527

Closed
vipulnaik opened this issue Oct 14, 2017 · 2 comments
Closed

File write errors when writing small files #2527

vipulnaik opened this issue Oct 14, 2017 · 2 comments
Assignees
Labels
api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@vipulnaik
Copy link

vipulnaik commented Oct 14, 2017

Similar to #723.

Version of GCloud running: default versions provided with dataproc on Google Compute Engine. Looks like these are:

Google Cloud SDK 174.0.0
alpha 2017.10.02
beta 2017.10.02
bq 2.0.27
core 2017.10.02
gsutil 4.27

Here is the code we are using to write files:

  def writeBytes(bytes: Array[Byte], dst: String): Unit = {
    val blob = blobOf(dst)
    val blobTry = storage.get(blob) // THIS APPEARS TO BE THE LINE THAT THROWS THE ERROR
    val writer =
      if(blobTry != null) blobTry.writer()
      else {
        val blobInfoBuilder = BlobInfo.newBuilder(blob)
        blobInfoBuilder.setContentType("text/plain")
        blobInfoBuilder.setContentEncoding("utf-8")
        storage.writer(blobInfoBuilder.build)
      }

      writer.write(java.nio.ByteBuffer.wrap(bytes))
      writer.close
  }

Here is the error that sometimes gets thrown when writing short files (about 550 bytes) when running Spark jobs via dataproc.

Exception in thread "main" com.google.cloud.storage.StorageException: java.lang.RuntimeException: java.lang.NullPointerException: ssl == null
        at com.google.cloud.storage.StorageException.translateAndThrow(StorageException.java:71)
        at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:196)
        at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:202)
        at com.petametrics.api.util.GSFile$.writeBytes(GSFile.scala:84)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException: ssl == null
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1488)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
        at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3018)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
        at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:94)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:333)
        at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:191)
        at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:188)
        at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:93)
        at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:49)
        at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:188)
        ... 20 more
Caused by: java.lang.NullPointerException: ssl == null
        at org.conscrypt.NativeCrypto.SSL_write(Native Method)
        at org.conscrypt.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:806)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at sun.net.www.MessageHeader.print(MessageHeader.java:301)
        at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:644)
        at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:655)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:693)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1567)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        ... 33 more
@vam-google vam-google added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Oct 23, 2017
@yihanzhen yihanzhen assigned yihanzhen and unassigned andreamlin Nov 29, 2017
@yihanzhen
Copy link
Contributor

@vipulnaik Sorry but I cannot reproduce this bug. Can you give me the full stacktrace and probably more context about how this writeBytes() method is called? Thanks.

@yihanzhen
Copy link
Contributor

Close it for now. Please reopen if this is still an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

4 participants