Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bigquery example and related issues #512

Merged
merged 4 commits into from
Dec 30, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 78 additions & 78 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ Java idiomatic client for [Google Cloud Platform][cloud-platform] services.

This client supports the following Google Cloud Platform services:

- [Google Cloud BigQuery] (#google-cloud-bigquery)
- [Google Cloud Datastore] (#google-cloud-datastore)
- [Google Cloud Storage] (#google-cloud-storage)
- [Google Cloud Resource Manager] (#google-cloud-resource-manager)
- [Google Cloud BigQuery] (#google-cloud-bigquery)
- [Google Cloud Storage] (#google-cloud-storage)

> Note: This client is a work-in-progress, and may occasionally
> make backwards-incompatible changes.
Expand All @@ -42,14 +42,14 @@ libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.1"
Example Applications
--------------------

- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html).
- [`DatastoreExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java) - A simple command line interface for the Cloud Datastore
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/DatastoreExample.html).
- [`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html).
- [`ResourceManagerExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java) - A simple command line interface providing some of Cloud Resource Manager's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/ResourceManagerExample.html).
- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html).
- [`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html).

Specifying a Project ID
-----------------------
Expand Down Expand Up @@ -109,6 +109,51 @@ Next, choose a method for authenticating API requests from within your project:
4. Google Cloud SDK credentials
5. Compute Engine credentials

Google Cloud BigQuery
----------------------

- [API Documentation][bigquery-api]
- [Official Documentation][cloud-bigquery-docs]

#### Preview

Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you
must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.

```java
import com.google.gcloud.bigquery.BaseTableInfo;
import com.google.gcloud.bigquery.BigQuery;
import com.google.gcloud.bigquery.BigQueryOptions;
import com.google.gcloud.bigquery.Field;
import com.google.gcloud.bigquery.JobStatus;
import com.google.gcloud.bigquery.LoadJobInfo;
import com.google.gcloud.bigquery.Schema;
import com.google.gcloud.bigquery.TableId;
import com.google.gcloud.bigquery.TableInfo;

BigQuery bigquery = BigQueryOptions.defaultInstance().service();
TableId tableId = TableId.of("dataset", "table");
BaseTableInfo info = bigquery.getTable(tableId);
if (info == null) {
System.out.println("Creating table " + tableId);
Field integerField = Field.of("fieldName", Field.Type.integer());
bigquery.create(TableInfo.of(tableId, Schema.of(integerField)));
} else {
System.out.println("Loading data into table " + tableId);
LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path");
loadJob = bigquery.create(loadJob);
while (loadJob.status().state() != JobStatus.State.DONE) {
Thread.sleep(1000L);
loadJob = bigquery.getJob(loadJob.jobId());
}
if (loadJob.status().error() != null) {
System.out.println("Job completed with errors");
} else {
System.out.println("Job succeeded");
}
}
```

Google Cloud Datastore
----------------------

Expand Down Expand Up @@ -149,45 +194,6 @@ if (entity == null) {
}
```

Google Cloud Storage
----------------------

- [API Documentation][storage-api]
- [Official Documentation][cloud-storage-docs]

*Follow the [activation instructions][cloud-storage-activation] to use the Google Cloud Storage API with your project.*

#### Preview

Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.

```java
import static java.nio.charset.StandardCharsets.UTF_8;

import com.google.gcloud.storage.Blob;
import com.google.gcloud.storage.BlobId;
import com.google.gcloud.storage.Storage;
import com.google.gcloud.storage.StorageOptions;

import java.nio.ByteBuffer;
import java.nio.channels.WritableByteChannel;

Storage storage = StorageOptions.defaultInstance().service();
BlobId blobId = BlobId.of("bucket", "blob_name");
Blob blob = Blob.load(storage, blobId);
if (blob == null) {
BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
} else {
System.out.println("Updating content for " + blobId.name());
byte[] prevContent = blob.content();
System.out.println(new String(prevContent, UTF_8));
WritableByteChannel channel = blob.writer();
channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
channel.close();
}
```

Google Cloud Resource Manager
----------------------

Expand Down Expand Up @@ -219,48 +225,42 @@ while (projectIterator.hasNext()) {
}
```

Google Cloud BigQuery
Google Cloud Storage
----------------------

- [API Documentation][bigquery-api]
- [Official Documentation][cloud-bigquery-docs]
- [API Documentation][storage-api]
- [Official Documentation][cloud-storage-docs]

*Follow the [activation instructions][cloud-storage-activation] to use the Google Cloud Storage API with your project.*

#### Preview

Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you
must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.

```java
import com.google.gcloud.bigquery.BaseTableInfo;
import com.google.gcloud.bigquery.BigQuery;
import com.google.gcloud.bigquery.BigQueryOptions;
import com.google.gcloud.bigquery.Field;
import com.google.gcloud.bigquery.JobStatus;
import com.google.gcloud.bigquery.LoadJobInfo;
import com.google.gcloud.bigquery.Schema;
import com.google.gcloud.bigquery.TableId;
import com.google.gcloud.bigquery.TableInfo;
import static java.nio.charset.StandardCharsets.UTF_8;

BigQuery bigquery = BigQueryOptions.defaultInstance().service();
TableId tableId = TableId.of("dataset", "table");
BaseTableInfo info = bigquery.getTable(tableId);
if (info == null) {
System.out.println("Creating table " + tableId);
Field integerField = Field.of("fieldName", Field.Type.integer());
bigquery.create(TableInfo.of(tableId, Schema.of(integerField)));
import com.google.gcloud.storage.Blob;
import com.google.gcloud.storage.BlobId;
import com.google.gcloud.storage.Storage;
import com.google.gcloud.storage.StorageOptions;

import java.nio.ByteBuffer;
import java.nio.channels.WritableByteChannel;

Storage storage = StorageOptions.defaultInstance().service();
BlobId blobId = BlobId.of("bucket", "blob_name");
Blob blob = Blob.load(storage, blobId);
if (blob == null) {
BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
} else {
System.out.println("Loading data into table " + tableId);
LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path");
loadJob = bigquery.create(loadJob);
while (loadJob.status().state() != JobStatus.State.DONE) {
Thread.sleep(1000L);
loadJob = bigquery.getJob(loadJob.jobId());
}
if (loadJob.status().error() != null) {
System.out.println("Job completed with errors");
} else {
System.out.println("Job succeeded");
}
System.out.println("Updating content for " + blobId.name());
byte[] prevContent = blob.content();
System.out.println(new String(prevContent, UTF_8));
WritableByteChannel channel = blob.writer();
channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
channel.close();
}
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@

package com.google.gcloud.bigquery;

import static com.google.common.base.Preconditions.checkNotNull;

import com.google.common.base.MoreObjects;

import java.io.Serializable;
Expand Down Expand Up @@ -85,7 +87,7 @@ public static FormatOptions datastoreBackup() {
* Default options for the provided format.
*/
public static FormatOptions of(String format) {
if (format.equals(CSV)) {
if (checkNotNull(format).equals(CSV)) {
return csv();
}
return new FormatOptions(format);
Expand Down
43 changes: 27 additions & 16 deletions gcloud-java-examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,27 @@ To run examples from your command line:

4. Run an example using Maven from command line.

Here's an example run of `BigQueryExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure
that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket
(replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS
[web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to
load data into a BigQuery table and should look something like:
```csv
value1
value2
value3
```
Then you are ready to run the following example:
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
```

Here's an example run of `DatastoreExample`.

Note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
Expand All @@ -48,16 +69,6 @@ To run examples from your command line:
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name delete"
```

Here's an example run of `StorageExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
```

Here's an example run of `ResourceManagerExample`.

Be sure to change the placeholder project ID "my-project-id" with your own globally unique project ID.
Expand All @@ -67,14 +78,14 @@ To run examples from your command line:
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get my-project-id"
```

Here's an example run of `BigQueryExample`.
Here's an example run of `StorageExample`.

Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud BigQuery API is enabled.
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
```
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
```

Troubleshooting
Expand Down
Loading