Skip to content

Commit

Permalink
Merge pull request #512 from mziccard/fix-bigquery-example
Browse files Browse the repository at this point in the history 
Fix bigquery example and related issues
  • Loading branch information
@ajkannan
ajkannan committed Dec 30, 2015
2 parents 45f5661 + 6af6db4 commit 823f0de
Show file tree
Hide file tree
Showing 8 changed files with 212 additions and 141 deletions.
156 changes: 78 additions & 78 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ Java idiomatic client for [Google Cloud Platform][cloud-platform] services.

This client supports the following Google Cloud Platform services:

- [Google Cloud BigQuery] (#google-cloud-bigquery)
- [Google Cloud Datastore] (#google-cloud-datastore)
- [Google Cloud Storage] (#google-cloud-storage)
- [Google Cloud Resource Manager] (#google-cloud-resource-manager)
- [Google Cloud BigQuery] (#google-cloud-bigquery)
- [Google Cloud Storage] (#google-cloud-storage)

> Note: This client is a work-in-progress, and may occasionally
> make backwards-incompatible changes.
Expand All @@ -42,14 +42,14 @@ libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.1"
Example Applications
--------------------

- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html).
- [`DatastoreExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java) - A simple command line interface for the Cloud Datastore
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/DatastoreExample.html).
- [`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html).
- [`ResourceManagerExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java) - A simple command line interface providing some of Cloud Resource Manager's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/ResourceManagerExample.html).
- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html).
- [`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html).

Specifying a Project ID
-----------------------
Expand Down Expand Up @@ -109,6 +109,51 @@ Next, choose a method for authenticating API requests from within your project:
4. Google Cloud SDK credentials
5. Compute Engine credentials
Google Cloud BigQuery
----------------------
- [API Documentation][bigquery-api]
- [Official Documentation][cloud-bigquery-docs]
#### Preview
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you
must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
```java
import com.google.gcloud.bigquery.BaseTableInfo;
import com.google.gcloud.bigquery.BigQuery;
import com.google.gcloud.bigquery.BigQueryOptions;
import com.google.gcloud.bigquery.Field;
import com.google.gcloud.bigquery.JobStatus;
import com.google.gcloud.bigquery.LoadJobInfo;
import com.google.gcloud.bigquery.Schema;
import com.google.gcloud.bigquery.TableId;
import com.google.gcloud.bigquery.TableInfo;
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
TableId tableId = TableId.of("dataset", "table");
BaseTableInfo info = bigquery.getTable(tableId);
if (info == null) {
System.out.println("Creating table " + tableId);
Field integerField = Field.of("fieldName", Field.Type.integer());
bigquery.create(TableInfo.of(tableId, Schema.of(integerField)));
} else {
System.out.println("Loading data into table " + tableId);
LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path");
loadJob = bigquery.create(loadJob);
while (loadJob.status().state() != JobStatus.State.DONE) {
Thread.sleep(1000L);
loadJob = bigquery.getJob(loadJob.jobId());
}
if (loadJob.status().error() != null) {
System.out.println("Job completed with errors");
} else {
System.out.println("Job succeeded");
}
}
```
Google Cloud Datastore
----------------------
Expand Down Expand Up @@ -149,45 +194,6 @@ if (entity == null) {
}
```
Google Cloud Storage
----------------------
- [API Documentation][storage-api]
- [Official Documentation][cloud-storage-docs]
*Follow the [activation instructions][cloud-storage-activation] to use the Google Cloud Storage API with your project.*
#### Preview
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
```java
import static java.nio.charset.StandardCharsets.UTF_8;
import com.google.gcloud.storage.Blob;
import com.google.gcloud.storage.BlobId;
import com.google.gcloud.storage.Storage;
import com.google.gcloud.storage.StorageOptions;
import java.nio.ByteBuffer;
import java.nio.channels.WritableByteChannel;
Storage storage = StorageOptions.defaultInstance().service();
BlobId blobId = BlobId.of("bucket", "blob_name");
Blob blob = Blob.load(storage, blobId);
if (blob == null) {
BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
} else {
System.out.println("Updating content for " + blobId.name());
byte[] prevContent = blob.content();
System.out.println(new String(prevContent, UTF_8));
WritableByteChannel channel = blob.writer();
channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
channel.close();
}
```
Google Cloud Resource Manager
----------------------
Expand Down Expand Up @@ -219,48 +225,42 @@ while (projectIterator.hasNext()) {
}
```
Google Cloud BigQuery
Google Cloud Storage
----------------------
- [API Documentation][bigquery-api]
- [Official Documentation][cloud-bigquery-docs]
- [API Documentation][storage-api]
- [Official Documentation][cloud-storage-docs]
*Follow the [activation instructions][cloud-storage-activation] to use the Google Cloud Storage API with your project.*
#### Preview
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you
must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
```java
import com.google.gcloud.bigquery.BaseTableInfo;
import com.google.gcloud.bigquery.BigQuery;
import com.google.gcloud.bigquery.BigQueryOptions;
import com.google.gcloud.bigquery.Field;
import com.google.gcloud.bigquery.JobStatus;
import com.google.gcloud.bigquery.LoadJobInfo;
import com.google.gcloud.bigquery.Schema;
import com.google.gcloud.bigquery.TableId;
import com.google.gcloud.bigquery.TableInfo;
import static java.nio.charset.StandardCharsets.UTF_8;
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
TableId tableId = TableId.of("dataset", "table");
BaseTableInfo info = bigquery.getTable(tableId);
if (info == null) {
System.out.println("Creating table " + tableId);
Field integerField = Field.of("fieldName", Field.Type.integer());
bigquery.create(TableInfo.of(tableId, Schema.of(integerField)));
import com.google.gcloud.storage.Blob;
import com.google.gcloud.storage.BlobId;
import com.google.gcloud.storage.Storage;
import com.google.gcloud.storage.StorageOptions;
import java.nio.ByteBuffer;
import java.nio.channels.WritableByteChannel;
Storage storage = StorageOptions.defaultInstance().service();
BlobId blobId = BlobId.of("bucket", "blob_name");
Blob blob = Blob.load(storage, blobId);
if (blob == null) {
BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
} else {
System.out.println("Loading data into table " + tableId);
LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path");
loadJob = bigquery.create(loadJob);
while (loadJob.status().state() != JobStatus.State.DONE) {
Thread.sleep(1000L);
loadJob = bigquery.getJob(loadJob.jobId());
}
if (loadJob.status().error() != null) {
System.out.println("Job completed with errors");
} else {
System.out.println("Job succeeded");
}
System.out.println("Updating content for " + blobId.name());
byte[] prevContent = blob.content();
System.out.println(new String(prevContent, UTF_8));
WritableByteChannel channel = blob.writer();
channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
channel.close();
}
```
Expand Down
4 changes: 3 additions & 1 deletion gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FormatOptions.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@

package com.google.gcloud.bigquery;

import static com.google.common.base.Preconditions.checkNotNull;

import com.google.common.base.MoreObjects;

import java.io.Serializable;
Expand Down Expand Up @@ -85,7 +87,7 @@ public static FormatOptions datastoreBackup() {
* Default options for the provided format.
*/
public static FormatOptions of(String format) {
if (format.equals(CSV)) {
if (checkNotNull(format).equals(CSV)) {
return csv();
}
return new FormatOptions(format);
Expand Down
43 changes: 27 additions & 16 deletions gcloud-java-examples/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,27 @@ To run examples from your command line:

4. Run an example using Maven from command line.

Here's an example run of `BigQueryExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure
that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket
(replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS
[web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to
load data into a BigQuery table and should look something like:
```csv
value1
value2
value3
```
Then you are ready to run the following example:
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
```

Here's an example run of `DatastoreExample`.

Note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
Expand All @@ -48,16 +69,6 @@ To run examples from your command line:
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name delete"
```

Here's an example run of `StorageExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
```

Here's an example run of `ResourceManagerExample`.

Be sure to change the placeholder project ID "my-project-id" with your own globally unique project ID.
Expand All @@ -67,14 +78,14 @@ To run examples from your command line:
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get my-project-id"
```

Here's an example run of `BigQueryExample`.
Here's an example run of `StorageExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud BigQuery API is enabled.
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
```

Troubleshooting
Expand Down
Loading

0 comments on commit 823f0de

Please sign in to comment.