diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bf87d471e34a..3d93f2d032c7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,6 +32,8 @@ When changes are made to authentication and project ID-related code, authenticat Known issue: If you have installed the Google Cloud SDK, be sure to log in (using `gcloud auth login`) before running tests. Though the Datastore tests use a local Datastore emulator that doesn't require authentication, they will not run if you have the Google Cloud SDK installed but aren't authenticated. +**Please, do not use your production projects for executing integration tests.** While we do our best to make our tests independent of your project's state and content, they do perform create, modify and deletes, and you do not want to have your production data accidentally modified. + Adding Features --------------- In order to add a feature to gcloud-java: diff --git a/README.md b/README.md index 520bf8b62c55..32a0f539425e 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Platform][cloud-platform] services. [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs) @@ -27,33 +29,33 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java - 0.1.2 + 0.1.4 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java:0.1.2' +compile 'com.google.gcloud:gcloud-java:0.1.4' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.2" +libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.4" ``` Example Applications -------------------- -- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality - - Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html). +- [`BigQueryExample`](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality + - Read more about using this application on the [`BigQueryExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/bigquery/BigQueryExample.html). - [`Bookshelf`](https://github.com/GoogleCloudPlatform/getting-started-java/tree/master/bookshelf) - An App Engine app that manages a virtual bookshelf. - This app uses `gcloud-java` to interface with Cloud Datastore and Cloud Storage. It also uses Cloud SQL, another Google Cloud Platform service. -- [`DatastoreExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java) - A simple command line interface for the Cloud Datastore - - Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/DatastoreExample.html). -- [`ResourceManagerExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java) - A simple command line interface providing some of Cloud Resource Manager's functionality - - Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/ResourceManagerExample.html). -- [`SparkDemo`](https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/managedvms/sparkjava) - An example of using gcloud-java-datastore from within the SparkJava and App Engine Managed VM frameworks. - - Read about how it works on the example's [README page](https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/managedvms/sparkjava#how-does-it-work). -- [`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality - - Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html). +- [`DatastoreExample`](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java) - A simple command line interface for the Cloud Datastore + - Read more about using this application on the [`DatastoreExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/datastore/DatastoreExample.html). +- [`ResourceManagerExample`](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java) - A simple command line interface providing some of Cloud Resource Manager's functionality + - Read more about using this application on the [`ResourceManagerExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/resourcemanager/ResourceManagerExample.html). +- [`SparkDemo`](https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/managed_vms/sparkjava) - An example of using gcloud-java-datastore from within the SparkJava and App Engine Managed VM frameworks. + - Read about how it works on the example's [README page](https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/managed_vms/sparkjava#how-does-it-work). +- [`StorageExample`](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality + - Read more about using this application on the [`StorageExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/storage/StorageExample.html). Specifying a Project ID ----------------------- @@ -123,38 +125,39 @@ Google Cloud BigQuery (Alpha) Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. +Complete source code can be found at +[CreateTableAndLoadData.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/CreateTableAndLoadData.java). ```java -import com.google.gcloud.bigquery.BaseTableInfo; import com.google.gcloud.bigquery.BigQuery; import com.google.gcloud.bigquery.BigQueryOptions; import com.google.gcloud.bigquery.Field; -import com.google.gcloud.bigquery.JobStatus; -import com.google.gcloud.bigquery.LoadJobInfo; +import com.google.gcloud.bigquery.FormatOptions; +import com.google.gcloud.bigquery.Job; import com.google.gcloud.bigquery.Schema; +import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; import com.google.gcloud.bigquery.TableId; import com.google.gcloud.bigquery.TableInfo; BigQuery bigquery = BigQueryOptions.defaultInstance().service(); TableId tableId = TableId.of("dataset", "table"); -BaseTableInfo info = bigquery.getTable(tableId); -if (info == null) { +Table table = bigquery.getTable(tableId); +if (table == null) { System.out.println("Creating table " + tableId); Field integerField = Field.of("fieldName", Field.Type.integer()); - bigquery.create(TableInfo.of(tableId, Schema.of(integerField))); + Schema schema = Schema.of(integerField); + table = bigquery.create(TableInfo.of(tableId, StandardTableDefinition.of(schema))); +} +System.out.println("Loading data into table " + tableId); +Job loadJob = table.load(FormatOptions.csv(), "gs://bucket/path"); +while (!loadJob.isDone()) { + Thread.sleep(1000L); +} +if (loadJob.status().error() != null) { + System.out.println("Job completed with errors"); } else { - System.out.println("Loading data into table " + tableId); - LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path"); - loadJob = bigquery.create(loadJob); - while (loadJob.status().state() != JobStatus.State.DONE) { - Thread.sleep(1000L); - loadJob = bigquery.getJob(loadJob.jobId()); - } - if (loadJob.status().error() != null) { - System.out.println("Job completed with errors"); - } else { - System.out.println("Job succeeded"); - } + System.out.println("Job succeeded"); } ``` @@ -168,8 +171,32 @@ Google Cloud Datastore #### Preview -Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. +Here are two code snippets showing simple usage examples from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. + +The first snippet shows how to create a Datastore entity. Complete source code can be found at +[CreateEntity.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/CreateEntity.java). + +```java +import com.google.gcloud.datastore.Datastore; +import com.google.gcloud.datastore.DatastoreOptions; +import com.google.gcloud.datastore.DateTime; +import com.google.gcloud.datastore.Entity; +import com.google.gcloud.datastore.Key; +import com.google.gcloud.datastore.KeyFactory; +Datastore datastore = DatastoreOptions.defaultInstance().service(); +KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind"); +Key key = keyFactory.newKey("keyName"); +Entity entity = Entity.builder(key) + .set("name", "John Doe") + .set("age", 30) + .set("access_time", DateTime.now()) + .build(); +datastore.put(entity); +``` +The second snippet shows how to update a Datastore entity if it exists. Complete source code can be +found at +[UpdateEntity.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/UpdateEntity.java). ```java import com.google.gcloud.datastore.Datastore; import com.google.gcloud.datastore.DatastoreOptions; @@ -179,17 +206,10 @@ import com.google.gcloud.datastore.Key; import com.google.gcloud.datastore.KeyFactory; Datastore datastore = DatastoreOptions.defaultInstance().service(); -KeyFactory keyFactory = datastore.newKeyFactory().kind(KIND); -Key key = keyFactory.newKey(keyName); +KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind"); +Key key = keyFactory.newKey("keyName"); Entity entity = datastore.get(key); -if (entity == null) { - entity = Entity.builder(key) - .set("name", "John Do") - .set("age", 30) - .set("access_time", DateTime.now()) - .build(); - datastore.put(entity); -} else { +if (entity != null) { System.out.println("Updating access_time for " + entity.getString("name")); entity = Entity.builder(entity) .set("access_time", DateTime.now()) @@ -207,22 +227,26 @@ Google Cloud Resource Manager (Alpha) #### Preview Here is a code snippet showing a simple usage example. Note that you must supply Google SDK credentials for this service, not other forms of authentication listed in the [Authentication section](#authentication). - +Complete source code can be found at +[UpdateAndListProjects.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java). ```java -import com.google.gcloud.resourcemanager.ProjectInfo; +import com.google.gcloud.resourcemanager.Project; import com.google.gcloud.resourcemanager.ResourceManager; import com.google.gcloud.resourcemanager.ResourceManagerOptions; import java.util.Iterator; ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service(); -ProjectInfo myProject = resourceManager.get("some-project-id"); // Use an existing project's ID -ProjectInfo newProjectInfo = resourceManager.replace(myProject.toBuilder() - .addLabel("launch-status", "in-development").build()); -System.out.println("Updated the labels of project " + newProjectInfo.projectId() - + " to be " + newProjectInfo.labels()); -// List all the projects you have permission to view. -Iterator projectIterator = resourceManager.list().iterateAll(); +Project project = resourceManager.get("some-project-id"); // Use an existing project's ID +if (project != null) { + Project newProject = project.toBuilder() + .addLabel("launch-status", "in-development") + .build() + .replace(); + System.out.println("Updated the labels of project " + newProject.projectId() + + " to be " + newProject.labels()); +} +Iterator projectIterator = resourceManager.list().iterateAll(); System.out.println("Projects I can view:"); while (projectIterator.hasNext()) { System.out.println(projectIterator.next().projectId()); @@ -239,8 +263,28 @@ Google Cloud Storage #### Preview -Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. +Here are two code snippets showing simple usage examples from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. + +The first snippet shows how to create a Storage blob. Complete source code can be found at +[CreateBlob.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateBlob.java). + +```java +import static java.nio.charset.StandardCharsets.UTF_8; + +import com.google.gcloud.storage.Blob; +import com.google.gcloud.storage.BlobId; +import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.Storage; +import com.google.gcloud.storage.StorageOptions; +Storage storage = StorageOptions.defaultInstance().service(); +BlobId blobId = BlobId.of("bucket", "blob_name"); +BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build(); +Blob blob = storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8)); +``` +The second snippet shows how to update a Storage blob if it exists. Complete source code can be +found at +[UpdateBlob.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/UpdateBlob.java). ```java import static java.nio.charset.StandardCharsets.UTF_8; @@ -254,12 +298,8 @@ import java.nio.channels.WritableByteChannel; Storage storage = StorageOptions.defaultInstance().service(); BlobId blobId = BlobId.of("bucket", "blob_name"); -Blob blob = Blob.get(storage, blobId); -if (blob == null) { - BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build(); - storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8)); -} else { - System.out.println("Updating content for " + blobId.name()); +Blob blob = storage.get(blobId); +if (blob != null) { byte[] prevContent = blob.content(); System.out.println(new String(prevContent, UTF_8)); WritableByteChannel channel = blob.writer(); diff --git a/RELEASING.md b/RELEASING.md index 586ebeb9b53b..5e2d6202062e 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -26,6 +26,8 @@ As mentioned before, there is an optional version argument. By default, the scr 6. Create and merge in another PR to reflect the updated project version. For an example of what this PR should look like, see [#227](https://github.com/GoogleCloudPlatform/gcloud-java/pull/227). +7. Be sure to update App Engine documentation and [java-docs-samples](https://github.com/GoogleCloudPlatform/java-docs-samples) code as necessary. See directions [here](https://docs.google.com/a/google.com/document/d/1SS3xNn2v0qW7EadGUPBUAPIQAH5VY6WSFmT17ZjjUVE/edit?usp=sharing). + ### To push a snapshot version Pushing a snapshot is completely automated. If "-SNAPSHOT" is included in the version denoted by the base directory's pom.xml, then an updated artifact will be pushed to the snapshot repository when Travis CI successfully completes a non-PR build. diff --git a/codacy-conf.json b/codacy-conf.json new file mode 100644 index 000000000000..e8c819684c9c --- /dev/null +++ b/codacy-conf.json @@ -0,0 +1 @@ +{"patterns":[{"patternId":"Custom_Javascript_Scopes","enabled":true},{"patternId":"Custom_Javascript_EvalWith","enabled":true},{"patternId":"Custom_Javascript_TryCatch","enabled":true},{"patternId":"Custom_Scala_NonFatal","enabled":true},{"patternId":"bitwise","enabled":true},{"patternId":"maxparams","enabled":true},{"patternId":"CSSLint_universal_selector","enabled":true},{"patternId":"CSSLint_unqualified_attributes","enabled":true},{"patternId":"CSSLint_zero_units","enabled":true},{"patternId":"CSSLint_overqualified_elements","enabled":true},{"patternId":"CSSLint_shorthand","enabled":true},{"patternId":"CSSLint_duplicate_background_images","enabled":true},{"patternId":"CSSLint_box_model","enabled":true},{"patternId":"CSSLint_compatible_vendor_prefixes","enabled":true},{"patternId":"CSSLint_display_property_grouping","enabled":true},{"patternId":"CSSLint_duplicate_properties","enabled":true},{"patternId":"CSSLint_empty_rules","enabled":true},{"patternId":"CSSLint_errors","enabled":true},{"patternId":"CSSLint_gradients","enabled":true},{"patternId":"CSSLint_important","enabled":true},{"patternId":"CSSLint_known_properties","enabled":true},{"patternId":"CSSLint_text_indent","enabled":true},{"patternId":"CSSLint_unique_headings","enabled":true},{"patternId":"PyLint_E0100","enabled":true},{"patternId":"PyLint_E0101","enabled":true},{"patternId":"PyLint_E0102","enabled":true},{"patternId":"PyLint_E0103","enabled":true},{"patternId":"PyLint_E0104","enabled":true},{"patternId":"PyLint_E0105","enabled":true},{"patternId":"PyLint_E0106","enabled":true},{"patternId":"PyLint_E0107","enabled":true},{"patternId":"PyLint_E0108","enabled":true},{"patternId":"PyLint_E0202","enabled":true},{"patternId":"PyLint_E0203","enabled":true},{"patternId":"PyLint_E0211","enabled":true},{"patternId":"PyLint_E0601","enabled":true},{"patternId":"PyLint_E0603","enabled":true},{"patternId":"PyLint_E0604","enabled":true},{"patternId":"PyLint_E0701","enabled":true},{"patternId":"PyLint_E0702","enabled":true},{"patternId":"PyLint_E0710","enabled":true},{"patternId":"PyLint_E0711","enabled":true},{"patternId":"PyLint_E0712","enabled":true},{"patternId":"PyLint_E1003","enabled":true},{"patternId":"PyLint_E1102","enabled":true},{"patternId":"PyLint_E1111","enabled":true},{"patternId":"PyLint_E1120","enabled":true},{"patternId":"PyLint_E1121","enabled":true},{"patternId":"PyLint_E1123","enabled":true},{"patternId":"PyLint_E1124","enabled":true},{"patternId":"PyLint_E1200","enabled":true},{"patternId":"PyLint_E1201","enabled":true},{"patternId":"PyLint_E1205","enabled":true},{"patternId":"PyLint_E1206","enabled":true},{"patternId":"PyLint_E1300","enabled":true},{"patternId":"PyLint_E1301","enabled":true},{"patternId":"PyLint_E1302","enabled":true},{"patternId":"PyLint_E1303","enabled":true},{"patternId":"PyLint_E1304","enabled":true},{"patternId":"PyLint_E1305","enabled":true},{"patternId":"PyLint_E1306","enabled":true},{"patternId":"rulesets-codesize.xml-CyclomaticComplexity","enabled":true},{"patternId":"rulesets-codesize.xml-NPathComplexity","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessiveMethodLength","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessiveClassLength","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessiveParameterList","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessivePublicCount","enabled":true},{"patternId":"rulesets-codesize.xml-TooManyFields","enabled":true},{"patternId":"rulesets-codesize.xml-TooManyMethods","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessiveClassComplexity","enabled":true},{"patternId":"rulesets-controversial.xml-Superglobals","enabled":true},{"patternId":"rulesets-design.xml-ExitExpression","enabled":true},{"patternId":"rulesets-design.xml-EvalExpression","enabled":true},{"patternId":"rulesets-design.xml-GotoStatement","enabled":true},{"patternId":"rulesets-design.xml-NumberOfChildren","enabled":true},{"patternId":"rulesets-design.xml-DepthOfInheritance","enabled":true},{"patternId":"rulesets-unusedcode.xml-UnusedPrivateField","enabled":true},{"patternId":"rulesets-unusedcode.xml-UnusedLocalVariable","enabled":true},{"patternId":"rulesets-unusedcode.xml-UnusedPrivateMethod","enabled":true},{"patternId":"rulesets-unusedcode.xml-UnusedFormalParameter","enabled":true},{"patternId":"PyLint_C0303","enabled":true},{"patternId":"PyLint_C1001","enabled":true},{"patternId":"rulesets-naming.xml-ShortVariable","enabled":true},{"patternId":"rulesets-naming.xml-LongVariable","enabled":true},{"patternId":"rulesets-naming.xml-ShortMethodName","enabled":true},{"patternId":"rulesets-naming.xml-ConstantNamingConventions","enabled":true},{"patternId":"rulesets-naming.xml-BooleanGetMethodName","enabled":true},{"patternId":"PyLint_W0101","enabled":true},{"patternId":"PyLint_W0102","enabled":true},{"patternId":"PyLint_W0104","enabled":true},{"patternId":"PyLint_W0105","enabled":true},{"patternId":"Custom_Scala_GetCalls","enabled":true},{"patternId":"ScalaStyle_EqualsHashCodeChecker","enabled":true},{"patternId":"ScalaStyle_ParameterNumberChecker","enabled":true},{"patternId":"ScalaStyle_ReturnChecker","enabled":true},{"patternId":"ScalaStyle_NullChecker","enabled":true},{"patternId":"ScalaStyle_NoCloneChecker","enabled":true},{"patternId":"ScalaStyle_NoFinalizeChecker","enabled":true},{"patternId":"ScalaStyle_CovariantEqualsChecker","enabled":true},{"patternId":"ScalaStyle_StructuralTypeChecker","enabled":true},{"patternId":"ScalaStyle_MethodLengthChecker","enabled":true},{"patternId":"ScalaStyle_NumberOfMethodsInTypeChecker","enabled":true},{"patternId":"ScalaStyle_WhileChecker","enabled":true},{"patternId":"ScalaStyle_VarFieldChecker","enabled":true},{"patternId":"ScalaStyle_VarLocalChecker","enabled":true},{"patternId":"ScalaStyle_RedundantIfChecker","enabled":true},{"patternId":"ScalaStyle_DeprecatedJavaChecker","enabled":true},{"patternId":"ScalaStyle_EmptyClassChecker","enabled":true},{"patternId":"ScalaStyle_NotImplementedErrorUsage","enabled":true},{"patternId":"Custom_Scala_GroupImports","enabled":true},{"patternId":"Custom_Scala_ReservedKeywords","enabled":true},{"patternId":"Custom_Scala_ElseIf","enabled":true},{"patternId":"Custom_Scala_CallByNameAsLastArguments","enabled":true},{"patternId":"Custom_Scala_WildcardImportOnMany","enabled":true},{"patternId":"Custom_Scala_UtilTryForTryCatch","enabled":true},{"patternId":"Custom_Scala_ProhibitObjectName","enabled":true},{"patternId":"Custom_Scala_ImportsAtBeginningOfPackage","enabled":true},{"patternId":"Custom_Scala_NameResultsAndParameters","enabled":true},{"patternId":"Custom_Scala_IncompletePatternMatching","enabled":true},{"patternId":"Custom_Scala_UsefulTypeAlias","enabled":true},{"patternId":"Custom_Scala_JavaThreads","enabled":true},{"patternId":"Custom_Scala_DirectPromiseCreation","enabled":true},{"patternId":"Custom_Scala_StructuralTypes","enabled":true},{"patternId":"Custom_Scala_CollectionLastHead","enabled":true},{"patternId":"PyLint_W0106","enabled":true},{"patternId":"PyLint_W0107","enabled":true},{"patternId":"PyLint_W0108","enabled":true},{"patternId":"PyLint_W0109","enabled":true},{"patternId":"PyLint_W0110","enabled":true},{"patternId":"PyLint_W0120","enabled":true},{"patternId":"PyLint_W0122","enabled":true},{"patternId":"PyLint_W0150","enabled":true},{"patternId":"PyLint_W0199","enabled":true},{"patternId":"rulesets-cleancode.xml-ElseExpression","enabled":true},{"patternId":"rulesets-cleancode.xml-StaticAccess","enabled":true},{"patternId":"ScalaStyle_NonASCIICharacterChecker","enabled":true},{"patternId":"ScalaStyle_FieldNamesChecker","enabled":true},{"patternId":"Custom_Scala_WithNameCalls","enabled":true},{"patternId":"strictexception_AvoidRethrowingException","enabled":true},{"patternId":"strings_AppendCharacterWithChar","enabled":true},{"patternId":"braces_IfElseStmtsMustUseBraces","enabled":true},{"patternId":"basic_AvoidDecimalLiteralsInBigDecimalConstructor","enabled":true},{"patternId":"basic_CheckSkipResult","enabled":true},{"patternId":"javabeans_MissingSerialVersionUID","enabled":true},{"patternId":"migrating_ShortInstantiation","enabled":true},{"patternId":"design_AvoidInstanceofChecksInCatchClause","enabled":true},{"patternId":"naming_LongVariable","enabled":true},{"patternId":"migrating_ReplaceEnumerationWithIterator","enabled":true},{"patternId":"j2ee_DoNotCallSystemExit","enabled":true},{"patternId":"unusedcode_UnusedLocalVariable","enabled":true},{"patternId":"strings_InefficientStringBuffering","enabled":true},{"patternId":"basic_DontUseFloatTypeForLoopIndices","enabled":true},{"patternId":"basic_AvoidBranchingStatementAsLastInLoop","enabled":true},{"patternId":"migrating_JUnit4TestShouldUseTestAnnotation","enabled":true},{"patternId":"optimizations_AddEmptyString","enabled":true},{"patternId":"logging-jakarta-commons_ProperLogger","enabled":true},{"patternId":"optimizations_RedundantFieldInitializer","enabled":true},{"patternId":"logging-java_AvoidPrintStackTrace","enabled":true},{"patternId":"empty_EmptyFinallyBlock","enabled":true},{"patternId":"design_CompareObjectsWithEquals","enabled":true},{"patternId":"basic_ClassCastExceptionWithToArray","enabled":true},{"patternId":"strictexception_DoNotExtendJavaLangError","enabled":true},{"patternId":"junit_UnnecessaryBooleanAssertion","enabled":true},{"patternId":"design_SimplifyBooleanExpressions","enabled":true},{"patternId":"basic_ForLoopShouldBeWhileLoop","enabled":true},{"patternId":"basic_BigIntegerInstantiation","enabled":true},{"patternId":"optimizations_UseArrayListInsteadOfVector","enabled":true},{"patternId":"optimizations_UnnecessaryWrapperObjectCreation","enabled":true},{"patternId":"strings_StringBufferInstantiationWithChar","enabled":true},{"patternId":"basic_JumbledIncrementer","enabled":true},{"patternId":"design_SwitchStmtsShouldHaveDefault","enabled":true},{"patternId":"strictexception_AvoidThrowingRawExceptionTypes","enabled":true},{"patternId":"migrating_LongInstantiation","enabled":true},{"patternId":"design_SimplifyBooleanReturns","enabled":true},{"patternId":"empty_EmptyInitializer","enabled":true},{"patternId":"design_FieldDeclarationsShouldBeAtStartOfClass","enabled":true},{"patternId":"unnecessary_UnnecessaryConversionTemporary","enabled":true},{"patternId":"design_AvoidProtectedFieldInFinalClass","enabled":true},{"patternId":"junit_UseAssertTrueInsteadOfAssertEquals","enabled":true},{"patternId":"naming_PackageCase","enabled":true},{"patternId":"migrating_JUnitUseExpected","enabled":true},{"patternId":"controversial_UnnecessaryConstructor","enabled":true},{"patternId":"naming_MethodNamingConventions","enabled":true},{"patternId":"design_DefaultLabelNotLastInSwitchStmt","enabled":true},{"patternId":"basic_UnconditionalIfStatement","enabled":true},{"patternId":"design_SingularField","enabled":true},{"patternId":"design_AssignmentToNonFinalStatic","enabled":true},{"patternId":"braces_WhileLoopsMustUseBraces","enabled":true},{"patternId":"logging-java_SystemPrintln","enabled":true},{"patternId":"strings_UseStringBufferLength","enabled":true},{"patternId":"controversial_AvoidUsingNativeCode","enabled":true},{"patternId":"strictexception_AvoidLosingExceptionInformation","enabled":true},{"patternId":"imports_ImportFromSamePackage","enabled":true},{"patternId":"finalizers_AvoidCallingFinalize","enabled":true},{"patternId":"finalizers_FinalizeOverloaded","enabled":true},{"patternId":"naming_ClassNamingConventions","enabled":true},{"patternId":"logging-java_LoggerIsNotStaticFinal","enabled":true},{"patternId":"finalizers_FinalizeOnlyCallsSuperFinalize","enabled":true},{"patternId":"unnecessary_UselessOverridingMethod","enabled":true},{"patternId":"naming_SuspiciousConstantFieldName","enabled":true},{"patternId":"design_OptimizableToArrayCall","enabled":true},{"patternId":"imports_UnnecessaryFullyQualifiedName","enabled":true},{"patternId":"migrating_ReplaceHashtableWithMap","enabled":true},{"patternId":"unusedcode_UnusedPrivateField","enabled":true},{"patternId":"strings_UnnecessaryCaseChange","enabled":true},{"patternId":"migrating_IntegerInstantiation","enabled":true},{"patternId":"design_NonStaticInitializer","enabled":true},{"patternId":"design_MissingBreakInSwitch","enabled":true},{"patternId":"design_AvoidReassigningParameters","enabled":true},{"patternId":"basic_AvoidThreadGroup","enabled":true},{"patternId":"empty_EmptyCatchBlock","parameters":{"allowCommentedBlocks":"true"},"enabled":true},{"patternId":"codesize_ExcessiveParameterList","parameters":{"minimum":"8","violationSuppressRegex":"\"\"","violationSuppressXPath":"\"\""},"enabled":true},{"patternId":"naming_SuspiciousHashcodeMethodName","enabled":true},{"patternId":"migrating_JUnit4TestShouldUseBeforeAnnotation","enabled":true},{"patternId":"design_UncommentedEmptyMethodBody","enabled":true},{"patternId":"basic_BrokenNullCheck","enabled":true},{"patternId":"strings_ConsecutiveLiteralAppends","enabled":true},{"patternId":"strings_StringInstantiation","enabled":true},{"patternId":"design_EqualsNull","enabled":true},{"patternId":"basic_OverrideBothEqualsAndHashcode","enabled":true},{"patternId":"design_InstantiationToGetClass","enabled":true},{"patternId":"basic_BooleanInstantiation","enabled":true},{"patternId":"strings_AvoidStringBufferField","enabled":true},{"patternId":"basic_ReturnFromFinallyBlock","enabled":true},{"patternId":"empty_EmptyTryBlock","enabled":true},{"patternId":"naming_SuspiciousEqualsMethodName","enabled":true},{"patternId":"basic_ExtendsObject","enabled":true},{"patternId":"strings_UselessStringValueOf","enabled":true},{"patternId":"design_UnsynchronizedStaticDateFormatter","enabled":true},{"patternId":"design_UseCollectionIsEmpty","enabled":true},{"patternId":"controversial_AvoidFinalLocalVariable","enabled":true},{"patternId":"strictexception_AvoidThrowingNullPointerException","enabled":true},{"patternId":"design_AvoidProtectedMethodInFinalClassNotExtending","enabled":true},{"patternId":"optimizations_PrematureDeclaration","enabled":true},{"patternId":"empty_EmptySwitchStatements","enabled":true},{"patternId":"basic_MisplacedNullCheck","enabled":true},{"patternId":"optimizations_UseStringBufferForStringAppends","enabled":true},{"patternId":"strings_StringToString","enabled":true},{"patternId":"naming_MethodWithSameNameAsEnclosingClass","enabled":true},{"patternId":"migrating_ReplaceVectorWithList","enabled":true},{"patternId":"imports_UnusedImports","enabled":true},{"patternId":"unnecessary_UnnecessaryFinalModifier","enabled":true},{"patternId":"basic_AvoidMultipleUnaryOperators","enabled":true},{"patternId":"junit_SimplifyBooleanAssertion","enabled":true},{"patternId":"unnecessary_UselessParentheses","enabled":true},{"patternId":"design_IdempotentOperations","enabled":true},{"patternId":"braces_IfStmtsMustUseBraces","enabled":true},{"patternId":"strings_UseIndexOfChar","enabled":true},{"patternId":"naming_NoPackage","enabled":true},{"patternId":"finalizers_FinalizeDoesNotCallSuperFinalize","enabled":true},{"patternId":"design_UseVarargs","enabled":true},{"patternId":"unusedcode_UnusedFormalParameter","enabled":true},{"patternId":"design_ReturnEmptyArrayRatherThanNull","enabled":true},{"patternId":"junit_UseAssertNullInsteadOfAssertTrue","enabled":true},{"patternId":"design_UseUtilityClass","enabled":true},{"patternId":"design_AvoidDeeplyNestedIfStmts","enabled":true},{"patternId":"empty_EmptyStatementNotInLoop","enabled":true},{"patternId":"junit_UseAssertSameInsteadOfAssertTrue","enabled":true},{"patternId":"braces_ForLoopsMustUseBraces","enabled":true},{"patternId":"controversial_DoNotCallGarbageCollectionExplicitly","enabled":true},{"patternId":"naming_GenericsNaming","enabled":true},{"patternId":"strings_UseEqualsToCompareStrings","enabled":true},{"patternId":"optimizations_AvoidArrayLoops","enabled":true},{"patternId":"empty_EmptyStaticInitializer","enabled":true},{"patternId":"design_UncommentedEmptyConstructor","enabled":true},{"patternId":"empty_EmptyStatementBlock","enabled":true},{"patternId":"basic_CollapsibleIfStatements","enabled":true},{"patternId":"design_FinalFieldCouldBeStatic","enabled":true},{"patternId":"logging-java_MoreThanOneLogger","enabled":true},{"patternId":"codesize_ExcessiveClassLength","enabled":true},{"patternId":"design_ImmutableField","enabled":true},{"patternId":"controversial_OneDeclarationPerLine","enabled":true},{"patternId":"empty_EmptyWhileStmt","enabled":true},{"patternId":"unnecessary_UnnecessaryReturn","enabled":true},{"patternId":"strings_InefficientEmptyStringCheck","enabled":true},{"patternId":"design_UseNotifyAllInsteadOfNotify","enabled":true},{"patternId":"strictexception_DoNotThrowExceptionInFinally","enabled":true},{"patternId":"junit_UseAssertEqualsInsteadOfAssertTrue","enabled":true},{"patternId":"typeresolution_CloneMethodMustImplementCloneable","enabled":true},{"patternId":"codesize_NPathComplexity","enabled":true},{"patternId":"imports_DontImportJavaLang","enabled":true},{"patternId":"empty_EmptySynchronizedBlock","enabled":true},{"patternId":"migrating_JUnit4TestShouldUseAfterAnnotation","enabled":true},{"patternId":"design_AvoidConstantsInterface","enabled":true},{"patternId":"unnecessary_UselessOperationOnImmutable","enabled":true},{"patternId":"design_PositionLiteralsFirstInComparisons","enabled":true},{"patternId":"migrating_ByteInstantiation","enabled":true},{"patternId":"junit_JUnitSpelling","enabled":true},{"patternId":"junit_JUnitTestsShouldIncludeAssert","enabled":true},{"patternId":"finalizers_EmptyFinalizer","enabled":true},{"patternId":"design_NonCaseLabelInSwitchStatement","enabled":true},{"patternId":"android_DoNotHardCodeSDCard","enabled":true},{"patternId":"design_LogicInversion","enabled":true},{"patternId":"unusedcode_UnusedPrivateMethod","enabled":true},{"patternId":"naming_AvoidDollarSigns","enabled":true},{"patternId":"finalizers_FinalizeShouldBeProtected","enabled":true},{"patternId":"clone_ProperCloneImplementation","enabled":true},{"patternId":"basic_CheckResultSet","enabled":true},{"patternId":"controversial_AvoidPrefixingMethodParameters","enabled":true},{"patternId":"migrating_JUnit4SuitesShouldUseSuiteAnnotation","enabled":true},{"patternId":"empty_EmptyIfStmt","enabled":true},{"patternId":"basic_DontCallThreadRun","enabled":true},{"patternId":"junit_JUnitStaticSuite","enabled":true},{"patternId":"optimizations_UseArraysAsList","enabled":true},{"patternId":"design_MissingStaticMethodInNonInstantiatableClass","enabled":true},{"patternId":"unusedcode_UnusedModifier","enabled":true},{"patternId":"Style_MethodName","enabled":true},{"patternId":"Metrics_CyclomaticComplexity","enabled":true},{"patternId":"Lint_DuplicateMethods","enabled":true},{"patternId":"Style_Lambda","enabled":true},{"patternId":"Lint_UselessSetterCall","enabled":true},{"patternId":"Style_VariableName","enabled":true},{"patternId":"Lint_AmbiguousOperator","enabled":true},{"patternId":"Style_LeadingCommentSpace","enabled":true},{"patternId":"Style_CaseEquality","enabled":true},{"patternId":"Lint_StringConversionInInterpolation","enabled":true},{"patternId":"Performance_ReverseEach","enabled":true},{"patternId":"Lint_LiteralInCondition","enabled":true},{"patternId":"Performance_Sample","enabled":true},{"patternId":"Style_NonNilCheck","enabled":true},{"patternId":"Lint_RescueException","enabled":true},{"patternId":"Lint_UselessElseWithoutRescue","enabled":true},{"patternId":"Style_ConstantName","enabled":true},{"patternId":"Lint_LiteralInInterpolation","enabled":true},{"patternId":"Lint_NestedMethodDefinition","enabled":true},{"patternId":"Style_DoubleNegation","enabled":true},{"patternId":"Lint_SpaceBeforeFirstArg","enabled":true},{"patternId":"Lint_Debugger","enabled":true},{"patternId":"Style_ClassVars","enabled":true},{"patternId":"Lint_EmptyEnsure","enabled":true},{"patternId":"Style_MultilineBlockLayout","enabled":true},{"patternId":"Lint_UnusedBlockArgument","enabled":true},{"patternId":"Lint_UselessAccessModifier","enabled":true},{"patternId":"Performance_Size","enabled":true},{"patternId":"Lint_EachWithObjectArgument","enabled":true},{"patternId":"Style_Alias","enabled":true},{"patternId":"Lint_Loop","enabled":true},{"patternId":"Style_NegatedWhile","enabled":true},{"patternId":"Style_ColonMethodCall","enabled":true},{"patternId":"Lint_AmbiguousRegexpLiteral","enabled":true},{"patternId":"Lint_UnusedMethodArgument","enabled":true},{"patternId":"Style_MultilineIfThen","enabled":true},{"patternId":"Lint_EnsureReturn","enabled":true},{"patternId":"Style_NegatedIf","enabled":true},{"patternId":"Lint_Eval","enabled":true},{"patternId":"Style_NilComparison","enabled":true},{"patternId":"Style_ArrayJoin","enabled":true},{"patternId":"Lint_ConditionPosition","enabled":true},{"patternId":"Lint_UnreachableCode","enabled":true},{"patternId":"Performance_Count","enabled":true},{"patternId":"Lint_EmptyInterpolation","enabled":true},{"patternId":"Style_LambdaCall","enabled":true},{"patternId":"Lint_HandleExceptions","enabled":true},{"patternId":"Lint_ShadowingOuterLocalVariable","enabled":true},{"patternId":"Lint_EndAlignment","enabled":true},{"patternId":"Style_MultilineTernaryOperator","enabled":true},{"patternId":"Style_AutoResourceCleanup","enabled":true},{"patternId":"Lint_ElseLayout","enabled":true},{"patternId":"Style_NestedTernaryOperator","enabled":true},{"patternId":"Style_OneLineConditional","enabled":true},{"patternId":"Style_EmptyElse","enabled":true},{"patternId":"Lint_UselessComparison","enabled":true},{"patternId":"Metrics_PerceivedComplexity","enabled":true},{"patternId":"Style_InfiniteLoop","enabled":true},{"patternId":"Rails_Date","enabled":true},{"patternId":"Style_EvenOdd","enabled":true},{"patternId":"Style_IndentationConsistency","enabled":true},{"patternId":"Style_ModuleFunction","enabled":true},{"patternId":"Lint_UselessAssignment","enabled":true},{"patternId":"Style_EachWithObject","enabled":true},{"patternId":"Performance_Detect","enabled":true},{"patternId":"duplicate_key","enabled":true},{"patternId":"no_interpolation_in_single_quotes","enabled":true},{"patternId":"no_backticks","enabled":true},{"patternId":"no_unnecessary_fat_arrows","enabled":true},{"patternId":"indentation","enabled":true},{"patternId":"ensure_comprehensions","enabled":true},{"patternId":"no_stand_alone_at","enabled":true},{"patternId":"cyclomatic_complexity","enabled":true},{"patternId":"Deserialize","enabled":true},{"patternId":"SymbolDoS","enabled":true},{"patternId":"SkipBeforeFilter","enabled":true},{"patternId":"SanitizeMethods","enabled":true},{"patternId":"SelectTag","enabled":true},{"patternId":"XMLDoS","enabled":true},{"patternId":"SimpleFormat","enabled":true},{"patternId":"Evaluation","enabled":true},{"patternId":"BasicAuth","enabled":true},{"patternId":"JRubyXML","enabled":true},{"patternId":"RenderInline","enabled":true},{"patternId":"YAMLParsing","enabled":true},{"patternId":"Redirect","enabled":true},{"patternId":"UnsafeReflection","enabled":true},{"patternId":"SSLVerify","enabled":true},{"patternId":"HeaderDoS","enabled":true},{"patternId":"TranslateBug","enabled":true},{"patternId":"Execute","enabled":true},{"patternId":"JSONParsing","enabled":true},{"patternId":"LinkTo","enabled":true},{"patternId":"FileDisclosure","enabled":true},{"patternId":"SafeBufferManipulation","enabled":true},{"patternId":"ModelAttributes","enabled":true},{"patternId":"ResponseSplitting","enabled":true},{"patternId":"DigestDoS","enabled":true},{"patternId":"Send","enabled":true},{"patternId":"MailTo","enabled":true},{"patternId":"SymbolDoSCVE","enabled":true},{"patternId":"StripTags","enabled":true},{"patternId":"MassAssignment","enabled":true},{"patternId":"RegexDoS","enabled":true},{"patternId":"SelectVulnerability","enabled":true},{"patternId":"FileAccess","enabled":true},{"patternId":"ContentTag","enabled":true},{"patternId":"SessionSettings","enabled":true},{"patternId":"FilterSkipping","enabled":true},{"patternId":"CreateWith","enabled":true},{"patternId":"JSONEncoding","enabled":true},{"patternId":"SQLCVEs","enabled":true},{"patternId":"ForgerySetting","enabled":true},{"patternId":"QuoteTableName","enabled":true},{"patternId":"I18nXSS","enabled":true},{"patternId":"WithoutProtection","enabled":true},{"patternId":"CrossSiteScripting","enabled":true},{"patternId":"SingleQuotes","enabled":true},{"patternId":"NestedAttributes","enabled":true},{"patternId":"DetailedExceptions","enabled":true},{"patternId":"LinkToHref","enabled":true},{"patternId":"RenderDoS","enabled":true},{"patternId":"ModelSerialize","enabled":true},{"patternId":"SQL","enabled":true},{"patternId":"Render","enabled":true},{"patternId":"UnscopedFind","enabled":true},{"patternId":"ValidationRegex","enabled":true},{"patternId":"EscapeFunction","enabled":true},{"patternId":"Custom_Scala_FieldNamesChecker","enabled":true},{"patternId":"Custom_Scala_ObjDeserialization","enabled":true},{"patternId":"Custom_Scala_RSAPadding","enabled":true},{"patternId":"ESLint_no-extra-boolean-cast","enabled":true},{"patternId":"ESLint_no-iterator","enabled":true},{"patternId":"ESLint_no-invalid-regexp","enabled":true},{"patternId":"ESLint_no-obj-calls","enabled":true},{"patternId":"ESLint_no-sparse-arrays","enabled":true},{"patternId":"ESLint_no-unreachable","enabled":true},{"patternId":"ESLint_no-dupe-keys","enabled":true},{"patternId":"ESLint_no-multi-str","enabled":true},{"patternId":"ESLint_no-extend-native","enabled":true},{"patternId":"ESLint_guard-for-in","enabled":true},{"patternId":"ESLint_no-func-assign","enabled":true},{"patternId":"ESLint_no-extra-semi","enabled":true},{"patternId":"ESLint_camelcase","enabled":true},{"patternId":"ESLint_no-mixed-spaces-and-tabs","enabled":true},{"patternId":"ESLint_no-undef","enabled":true},{"patternId":"ESLint_semi","enabled":true},{"patternId":"ESLint_no-empty-character-class","enabled":true},{"patternId":"ESLint_complexity","enabled":true},{"patternId":"ESLint_no-dupe-class-members","enabled":true},{"patternId":"ESLint_no-debugger","enabled":true},{"patternId":"ESLint_block-scoped-var","enabled":true},{"patternId":"ESLint_no-loop-func","enabled":true},{"patternId":"ESLint_no-use-before-define","enabled":true},{"patternId":"ESLint_no-console","enabled":true},{"patternId":"ESLint_require-yield","enabled":true},{"patternId":"ESLint_no-redeclare","enabled":true},{"patternId":"ESLint_no-undefined","enabled":true},{"patternId":"ESLint_use-isnan","enabled":true},{"patternId":"ESLint_no-control-regex","enabled":true},{"patternId":"ESLint_no-const-assign","enabled":true},{"patternId":"ESLint_no-new","enabled":true},{"patternId":"ESLint_new-cap","enabled":true},{"patternId":"ESLint_no-irregular-whitespace","enabled":true},{"patternId":"ESLint_object-shorthand","enabled":true},{"patternId":"ESLint_no-ex-assign","enabled":true},{"patternId":"ESLint_wrap-iife","enabled":true},{"patternId":"ESLint_arrow-parens","enabled":true},{"patternId":"ESLint_no-constant-condition","enabled":true},{"patternId":"ESLint_no-octal","enabled":true},{"patternId":"ESLint_no-dupe-args","enabled":true},{"patternId":"ESLint_quotes","enabled":true},{"patternId":"ESLint_no-fallthrough","enabled":true},{"patternId":"ESLint_no-delete-var","enabled":true},{"patternId":"ESLint_no-caller","enabled":true},{"patternId":"ESLint_no-cond-assign","enabled":true},{"patternId":"ESLint_no-this-before-super","enabled":true},{"patternId":"ESLint_no-negated-in-lhs","enabled":true},{"patternId":"ESLint_no-inner-declarations","enabled":true},{"patternId":"ESLint_eqeqeq","enabled":true},{"patternId":"ESLint_curly","enabled":true},{"patternId":"ESLint_arrow-spacing","enabled":true},{"patternId":"ESLint_no-empty","enabled":true},{"patternId":"ESLint_no-unused-vars","enabled":true},{"patternId":"ESLint_generator-star-spacing","enabled":true},{"patternId":"ESLint_no-duplicate-case","enabled":true},{"patternId":"ESLint_valid-typeof","enabled":true},{"patternId":"ESLint_no-regex-spaces","enabled":true},{"patternId":"ESLint_no-class-assign","enabled":true},{"patternId":"PyLint_W0221","enabled":true},{"patternId":"PyLint_E0117","enabled":true},{"patternId":"PyLint_E0001","enabled":true},{"patternId":"PyLint_E0241","enabled":true},{"patternId":"PyLint_W0404","enabled":true},{"patternId":"PyLint_E0704","enabled":true},{"patternId":"PyLint_E0703","enabled":true},{"patternId":"PyLint_E0302","enabled":true},{"patternId":"PyLint_W1301","enabled":true},{"patternId":"PyLint_R0201","enabled":true},{"patternId":"PyLint_E0113","enabled":true},{"patternId":"PyLint_W0410","enabled":true},{"patternId":"PyLint_C0123","enabled":true},{"patternId":"PyLint_E0115","enabled":true},{"patternId":"PyLint_E0114","enabled":true},{"patternId":"PyLint_E1126","enabled":true},{"patternId":"PyLint_W0702","enabled":true},{"patternId":"PyLint_W1303","enabled":true},{"patternId":"PyLint_W0622","enabled":true},{"patternId":"PyLint_W0222","enabled":true},{"patternId":"PyLint_W0233","enabled":true},{"patternId":"PyLint_W1305","enabled":true},{"patternId":"PyLint_E1127","enabled":true},{"patternId":"PyLint_E0112","enabled":true},{"patternId":"PyLint_W0611","enabled":true},{"patternId":"PyLint_W0601","enabled":true},{"patternId":"PyLint_W1300","enabled":true},{"patternId":"PyLint_W0124","enabled":true},{"patternId":"PyLint_R0203","enabled":true},{"patternId":"PyLint_E0236","enabled":true},{"patternId":"PyLint_W0612","enabled":true},{"patternId":"PyLint_W0604","enabled":true},{"patternId":"PyLint_W0705","enabled":true},{"patternId":"PyLint_E0238","enabled":true},{"patternId":"PyLint_W0602","enabled":true},{"patternId":"PyLint_R0102","enabled":true},{"patternId":"PyLint_R0202","enabled":true},{"patternId":"PyLint_E0240","enabled":true},{"patternId":"PyLint_W0623","enabled":true},{"patternId":"PyLint_W0711","enabled":true},{"patternId":"PyLint_E0116","enabled":true},{"patternId":"PyLint_E0239","enabled":true},{"patternId":"PyLint_E1132","enabled":true},{"patternId":"PyLint_W1307","enabled":true},{"patternId":"PyLint_C0200","enabled":true},{"patternId":"PyLint_E0301","enabled":true},{"patternId":"PyLint_W1306","enabled":true},{"patternId":"PyLint_W1302","enabled":true},{"patternId":"PyLint_E0110","enabled":true},{"patternId":"PyLint_E1125","enabled":true}]} \ No newline at end of file diff --git a/gcloud-java-bigquery/README.md b/gcloud-java-bigquery/README.md index af68402fbf70..58633ba635f9 100644 --- a/gcloud-java-bigquery/README.md +++ b/gcloud-java-bigquery/README.md @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud BigQuery] (https://cloud.google.com/bigq [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-bigquery.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-bigquery.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/bigquery/package-summary.html) @@ -20,22 +22,22 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java-bigquery - 0.1.2 + 0.1.4 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java-bigquery:0.1.2' +compile 'com.google.gcloud:gcloud-java-bigquery:0.1.4' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java-bigquery" % "0.1.2" +libraryDependencies += "com.google.gcloud" % "gcloud-java-bigquery" % "0.1.4" ``` Example Application ------------------- -- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality. -Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html). +- [`BigQueryExample`](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality. +Read more about using this application on the [`BigQueryExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/bigquery/BigQueryExample.html). Authentication -------------- @@ -111,9 +113,10 @@ are created from a BigQuery SQL query. In this code snippet we show how to creat with only one string field. Add the following imports at the top of your file: ```java -import com.google.gcloud.bigquery.BaseTableInfo; import com.google.gcloud.bigquery.Field; import com.google.gcloud.bigquery.Schema; +import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; import com.google.gcloud.bigquery.TableId; import com.google.gcloud.bigquery.TableInfo; ``` @@ -126,7 +129,8 @@ Field stringField = Field.of("StringField", Field.Type.string()); // Table schema definition Schema schema = Schema.of(stringField); // Create a table -TableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, schema)); +StandardTableDefinition tableDefinition = StandardTableDefinition.of(schema); +Table createdTable = bigquery.create(TableInfo.of(tableId, tableDefinition)); ``` #### Loading data into a table @@ -198,89 +202,13 @@ while (rowIterator.hasNext()) { ``` #### Complete source code -Here we put together all the code shown above into one program. This program assumes that you are -running on Compute Engine or from your own desktop. To run this example on App Engine, simply move +In +[InsertDataAndQueryTable.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/InsertDataAndQueryTable.java) +we put together all the code shown above into one program. The program assumes that you are +running on Compute Engine or from your own desktop. To run the example on App Engine, simply move the code from the main method to your application's servlet class and change the print statements to display on your webpage. -```java -import com.google.gcloud.bigquery.BaseTableInfo; -import com.google.gcloud.bigquery.BigQuery; -import com.google.gcloud.bigquery.BigQueryOptions; -import com.google.gcloud.bigquery.DatasetInfo; -import com.google.gcloud.bigquery.Field; -import com.google.gcloud.bigquery.FieldValue; -import com.google.gcloud.bigquery.InsertAllRequest; -import com.google.gcloud.bigquery.InsertAllResponse; -import com.google.gcloud.bigquery.QueryRequest; -import com.google.gcloud.bigquery.QueryResponse; -import com.google.gcloud.bigquery.Schema; -import com.google.gcloud.bigquery.TableId; -import com.google.gcloud.bigquery.TableInfo; - -import java.util.HashMap; -import java.util.Iterator; -import java.util.List; -import java.util.Map; - -public class GcloudBigQueryExample { - - public static void main(String[] args) throws InterruptedException { - - // Create a service instance - BigQuery bigquery = BigQueryOptions.defaultInstance().service(); - - // Create a dataset - String datasetId = "my_dataset_id"; - bigquery.create(DatasetInfo.builder(datasetId).build()); - - TableId tableId = TableId.of(datasetId, "my_table_id"); - // Table field definition - Field stringField = Field.of("StringField", Field.Type.string()); - // Table schema definition - Schema schema = Schema.of(stringField); - // Create a table - TableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, schema)); - - // Define rows to insert - Map firstRow = new HashMap<>(); - Map secondRow = new HashMap<>(); - firstRow.put("StringField", "value1"); - secondRow.put("StringField", "value2"); - // Create an insert request - InsertAllRequest insertRequest = InsertAllRequest.builder(tableId) - .addRow(firstRow) - .addRow(secondRow) - .build(); - // Insert rows - InsertAllResponse insertResponse = bigquery.insertAll(insertRequest); - // Check if errors occurred - if (insertResponse.hasErrors()) { - System.out.println("Errors occurred while inserting rows"); - } - - // Create a query request - QueryRequest queryRequest = - QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id") - .maxWaitTime(60000L) - .maxResults(1000L) - .build(); - // Request query to be executed and wait for results - QueryResponse queryResponse = bigquery.query(queryRequest); - while (!queryResponse.jobComplete()) { - Thread.sleep(1000L); - queryResponse = bigquery.getQueryResults(queryResponse.jobId()); - } - // Read rows - Iterator> rowIterator = queryResponse.result().iterateAll(); - System.out.println("Table rows:"); - while (rowIterator.hasNext()) { - System.out.println(rowIterator.next()); - } - } -} -``` - Troubleshooting --------------- diff --git a/gcloud-java-bigquery/pom.xml b/gcloud-java-bigquery/pom.xml index aba002b6dce4..34ddd7679e97 100644 --- a/gcloud-java-bigquery/pom.xml +++ b/gcloud-java-bigquery/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java-bigquery jar GCloud Java bigquery @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.3-SNAPSHOT + 0.1.5-SNAPSHOT gcloud-java-bigquery @@ -31,13 +30,13 @@ com.google.apis google-api-services-bigquery - v2-rev254-1.21.0 + v2-rev270-1.21.0 compile - - com.google.guava - guava-jdk5 - + + com.google.guava + guava-jdk5 + @@ -49,7 +48,7 @@ org.easymock easymock - 3.3 + 3.4 test diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Acl.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Acl.java index b8e9926ce8c8..b8e1a817c836 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Acl.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Acl.java @@ -325,6 +325,8 @@ Access toPb() { */ public static final class View extends Entity { + private static final long serialVersionUID = -6851072781269419383L; + private final TableId id; /** diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BaseTableInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BaseTableInfo.java deleted file mode 100644 index 1b49a026ac14..000000000000 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BaseTableInfo.java +++ /dev/null @@ -1,437 +0,0 @@ -/* - * Copyright 2015 Google Inc. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.google.gcloud.bigquery; - -import static com.google.common.base.MoreObjects.firstNonNull; -import static com.google.common.base.Preconditions.checkNotNull; - -import com.google.api.client.util.Data; -import com.google.api.services.bigquery.model.Table; -import com.google.common.base.Function; -import com.google.common.base.MoreObjects; -import com.google.common.base.MoreObjects.ToStringHelper; - -import java.io.Serializable; -import java.math.BigInteger; -import java.util.Objects; - -/** - * Base class for Google BigQuery table information. Use {@link TableInfo} for a simple BigQuery - * Table. Use {@link ViewInfo} for a BigQuery View Table. Use {@link ExternalTableInfo} for a - * BigQuery Table backed by external data. - * - * @see Managing Tables - */ -public abstract class BaseTableInfo implements Serializable { - - static final Function FROM_PB_FUNCTION = - new Function() { - @Override - public BaseTableInfo apply(Table pb) { - return BaseTableInfo.fromPb(pb); - } - }; - static final Function TO_PB_FUNCTION = - new Function() { - @Override - public Table apply(BaseTableInfo tableInfo) { - return tableInfo.toPb(); - } - }; - - private static final long serialVersionUID = -7679032506430816205L; - - /** - * The table type. - */ - public enum Type { - /** - * A normal BigQuery table. - */ - TABLE, - - /** - * A virtual table defined by a SQL query. - * - * @see Views - */ - VIEW, - - /** - * A BigQuery table backed by external data. - * - * @see Federated Data - * Sources - */ - EXTERNAL - } - - private final String etag; - private final String id; - private final String selfLink; - private final TableId tableId; - private final Type type; - private final Schema schema; - private final String friendlyName; - private final String description; - private final Long numBytes; - private final Long numRows; - private final Long creationTime; - private final Long expirationTime; - private final Long lastModifiedTime; - - /** - * Base builder for tables. - * - * @param the table type - * @param the table builder - */ - public abstract static class Builder> { - - private String etag; - private String id; - private String selfLink; - private TableId tableId; - private Type type; - private Schema schema; - private String friendlyName; - private String description; - private Long numBytes; - private Long numRows; - private Long creationTime; - private Long expirationTime; - private Long lastModifiedTime; - - protected Builder() {} - - protected Builder(BaseTableInfo tableInfo) { - this.etag = tableInfo.etag; - this.id = tableInfo.id; - this.selfLink = tableInfo.selfLink; - this.tableId = tableInfo.tableId; - this.type = tableInfo.type; - this.schema = tableInfo.schema; - this.friendlyName = tableInfo.friendlyName; - this.description = tableInfo.description; - this.numBytes = tableInfo.numBytes; - this.numRows = tableInfo.numRows; - this.creationTime = tableInfo.creationTime; - this.expirationTime = tableInfo.expirationTime; - this.lastModifiedTime = tableInfo.lastModifiedTime; - } - - protected Builder(Table tablePb) { - this.type = Type.valueOf(tablePb.getType()); - this.tableId = TableId.fromPb(tablePb.getTableReference()); - if (tablePb.getSchema() != null) { - this.schema(Schema.fromPb(tablePb.getSchema())); - } - if (tablePb.getLastModifiedTime() != null) { - this.lastModifiedTime(tablePb.getLastModifiedTime().longValue()); - } - if (tablePb.getNumRows() != null) { - this.numRows(tablePb.getNumRows().longValue()); - } - this.description = tablePb.getDescription(); - this.expirationTime = tablePb.getExpirationTime(); - this.friendlyName = tablePb.getFriendlyName(); - this.creationTime = tablePb.getCreationTime(); - this.etag = tablePb.getEtag(); - this.id = tablePb.getId(); - this.numBytes = tablePb.getNumBytes(); - this.selfLink = tablePb.getSelfLink(); - } - - @SuppressWarnings("unchecked") - protected B self() { - return (B) this; - } - - B creationTime(Long creationTime) { - this.creationTime = creationTime; - return self(); - } - - /** - * Sets a user-friendly description for the table. - */ - public B description(String description) { - this.description = firstNonNull(description, Data.nullOf(String.class)); - return self(); - } - - B etag(String etag) { - this.etag = etag; - return self(); - } - - /** - * Sets the time when this table expires, in milliseconds since the epoch. If not present, the - * table will persist indefinitely. Expired tables will be deleted and their storage reclaimed. - */ - public B expirationTime(Long expirationTime) { - this.expirationTime = firstNonNull(expirationTime, Data.nullOf(Long.class)); - return self(); - } - - /** - * Sets a user-friendly name for the table. - */ - public B friendlyName(String friendlyName) { - this.friendlyName = firstNonNull(friendlyName, Data.nullOf(String.class)); - return self(); - } - - B id(String id) { - this.id = id; - return self(); - } - - B lastModifiedTime(Long lastModifiedTime) { - this.lastModifiedTime = lastModifiedTime; - return self(); - } - - B numBytes(Long numBytes) { - this.numBytes = numBytes; - return self(); - } - - B numRows(Long numRows) { - this.numRows = numRows; - return self(); - } - - B selfLink(String selfLink) { - this.selfLink = selfLink; - return self(); - } - - /** - * Sets the table identity. - */ - public B tableId(TableId tableId) { - this.tableId = checkNotNull(tableId); - return self(); - } - - B type(Type type) { - this.type = type; - return self(); - } - - /** - * Sets the table schema. - */ - public B schema(Schema schema) { - this.schema = checkNotNull(schema); - return self(); - } - - /** - * Creates an object. - */ - public abstract T build(); - } - - protected BaseTableInfo(Builder builder) { - this.tableId = checkNotNull(builder.tableId); - this.etag = builder.etag; - this.id = builder.id; - this.selfLink = builder.selfLink; - this.friendlyName = builder.friendlyName; - this.description = builder.description; - this.type = builder.type; - this.schema = builder.schema; - this.numBytes = builder.numBytes; - this.numRows = builder.numRows; - this.creationTime = builder.creationTime; - this.expirationTime = builder.expirationTime; - this.lastModifiedTime = builder.lastModifiedTime; - } - - /** - * Returns the hash of the table resource. - */ - public String etag() { - return etag; - } - - /** - * Returns an opaque id for the table. - */ - public String id() { - return id; - } - - /** - * Returns the table's type. If this table is simple table the method returns {@link Type#TABLE}. - * If this table is an external table this method returns {@link Type#EXTERNAL}. If this table is - * a view table this method returns {@link Type#VIEW}. - */ - public Type type() { - return type; - } - - /** - * Returns the table's schema. - */ - public Schema schema() { - return schema; - } - - /** - * Returns an URL that can be used to access the resource again. The returned URL can be used for - * get or update requests. - */ - public String selfLink() { - return selfLink; - } - - /** - * Returns the table identity. - */ - public TableId tableId() { - return tableId; - } - - /** - * Returns a user-friendly name for the table. - */ - public String friendlyName() { - return Data.isNull(friendlyName) ? null : friendlyName; - } - - /** - * Returns a user-friendly description for the table. - */ - public String description() { - return Data.isNull(description) ? null : description; - } - - /** - * Returns the size of this table in bytes, excluding any data in the streaming buffer. - */ - public Long numBytes() { - return numBytes; - } - - /** - * Returns the number of rows in this table, excluding any data in the streaming buffer. - */ - public Long numRows() { - return numRows; - } - - /** - * Returns the time when this table was created, in milliseconds since the epoch. - */ - public Long creationTime() { - return creationTime; - } - - /** - * Returns the time when this table expires, in milliseconds since the epoch. If not present, the - * table will persist indefinitely. Expired tables will be deleted and their storage reclaimed. - */ - public Long expirationTime() { - return Data.isNull(expirationTime) ? null : expirationTime; - } - - /** - * Returns the time when this table was last modified, in milliseconds since the epoch. - */ - public Long lastModifiedTime() { - return lastModifiedTime; - } - - /** - * Returns a builder for the object. - */ - public abstract Builder toBuilder(); - - ToStringHelper toStringHelper() { - return MoreObjects.toStringHelper(this) - .add("tableId", tableId) - .add("type", type) - .add("schema", schema) - .add("etag", etag) - .add("id", id) - .add("selfLink", selfLink) - .add("friendlyName", friendlyName) - .add("description", description) - .add("numBytes", numBytes) - .add("numRows", numRows) - .add("expirationTime", expirationTime) - .add("creationTime", creationTime) - .add("lastModifiedTime", lastModifiedTime); - } - - @Override - public String toString() { - return toStringHelper().toString(); - } - - @Override - public int hashCode() { - return Objects.hash(tableId); - } - - @Override - public boolean equals(Object obj) { - return obj instanceof BaseTableInfo && Objects.equals(toPb(), ((BaseTableInfo) obj).toPb()); - } - - Table toPb() { - Table tablePb = new Table(); - tablePb.setTableReference(tableId.toPb()); - if (lastModifiedTime != null) { - tablePb.setLastModifiedTime(BigInteger.valueOf(lastModifiedTime)); - } - if (numRows != null) { - tablePb.setNumRows(BigInteger.valueOf(numRows)); - } - if (schema != null) { - tablePb.setSchema(schema.toPb()); - } - tablePb.setType(type.name()); - tablePb.setCreationTime(creationTime); - tablePb.setDescription(description); - tablePb.setEtag(etag); - tablePb.setExpirationTime(expirationTime); - tablePb.setFriendlyName(friendlyName); - tablePb.setId(id); - tablePb.setNumBytes(numBytes); - tablePb.setSelfLink(selfLink); - return tablePb; - } - - @SuppressWarnings("unchecked") - static T fromPb(Table tablePb) { - switch (Type.valueOf(tablePb.getType())) { - case TABLE: - return (T) TableInfo.fromPb(tablePb); - case VIEW: - return (T) ViewInfo.fromPb(tablePb); - case EXTERNAL: - return (T) ExternalTableInfo.fromPb(tablePb); - default: - // never reached - throw new IllegalArgumentException("Format " + tablePb.getType() + " is not supported"); - } - } -} diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQuery.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQuery.java index d3c712229348..4b5d3ef0c81a 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQuery.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQuery.java @@ -206,7 +206,7 @@ private DatasetOption(BigQueryRpc.Option option, Object value) { /** * Returns an option to specify the dataset's fields to be returned by the RPC call. If this * option is not provided all dataset's fields are returned. {@code DatasetOption.fields} can - * be used to specify only the fields of interest. {@link DatasetInfo#datasetId()} is always + * be used to specify only the fields of interest. {@link Dataset#datasetId()} is always * returned, even if not specified. */ public static DatasetOption fields(DatasetField... fields) { @@ -275,8 +275,8 @@ private TableOption(BigQueryRpc.Option option, Object value) { /** * Returns an option to specify the table's fields to be returned by the RPC call. If this * option is not provided all table's fields are returned. {@code TableOption.fields} can be - * used to specify only the fields of interest. {@link BaseTableInfo#tableId()} and - * {@link BaseTableInfo#type()} are always returned, even if not specified. + * used to specify only the fields of interest. {@link Table#tableId()} and type (which is part + * of {@link Table#definition()}) are always returned, even if not specified. */ public static TableOption fields(TableField... fields) { return new TableOption(BigQueryRpc.Option.FIELDS, TableField.selector(fields)); @@ -369,10 +369,11 @@ public static JobListOption startPageToken(String pageToken) { /** * Returns an option to specify the job's fields to be returned by the RPC call. If this option * is not provided all job's fields are returned. {@code JobOption.fields()} can be used to - * specify only the fields of interest. {@link JobInfo#jobId()}, {@link JobStatus#state()}, + * specify only the fields of interest. {@link Job#jobId()}, {@link JobStatus#state()}, * {@link JobStatus#error()} as well as type-specific configuration (e.g. - * {@link QueryJobInfo#query()} for Query Jobs) are always returned, even if not specified. - * {@link JobField#SELF_LINK} and {@link JobField#ETAG} can not be selected when listing jobs. + * {@link QueryJobConfiguration#query()} for Query Jobs) are always returned, even if not + * specified. {@link JobField#SELF_LINK} and {@link JobField#ETAG} can not be selected when + * listing jobs. */ public static JobListOption fields(JobField... fields) { String selector = JobField.selector(fields); @@ -396,9 +397,9 @@ private JobOption(BigQueryRpc.Option option, Object value) { /** * Returns an option to specify the job's fields to be returned by the RPC call. If this option * is not provided all job's fields are returned. {@code JobOption.fields()} can be used to - * specify only the fields of interest. {@link JobInfo#jobId()} as well as type-specific - * configuration (e.g. {@link QueryJobInfo#query()} for Query Jobs) are always returned, even if - * not specified. + * specify only the fields of interest. {@link Job#jobId()} as well as type-specific + * configuration (e.g. {@link QueryJobConfiguration#query()} for Query Jobs) are always + * returned, even if not specified. */ public static JobOption fields(JobField... fields) { return new JobOption(BigQueryRpc.Option.FIELDS, JobField.selector(fields)); @@ -456,46 +457,45 @@ public static QueryResultsOption maxWaitTime(long maxWaitTime) { * * @throws BigQueryException upon failure */ - DatasetInfo create(DatasetInfo dataset, DatasetOption... options) throws BigQueryException; + Dataset create(DatasetInfo dataset, DatasetOption... options) throws BigQueryException; /** * Creates a new table. * * @throws BigQueryException upon failure */ - T create(T table, TableOption... options) throws BigQueryException; + Table create(TableInfo table, TableOption... options) throws BigQueryException; /** * Creates a new job. * * @throws BigQueryException upon failure */ - T create(T job, JobOption... options) throws BigQueryException; + Job create(JobInfo job, JobOption... options) throws BigQueryException; /** * Returns the requested dataset or {@code null} if not found. * * @throws BigQueryException upon failure */ - DatasetInfo getDataset(String datasetId, DatasetOption... options) throws BigQueryException; + Dataset getDataset(String datasetId, DatasetOption... options) throws BigQueryException; /** * Returns the requested dataset or {@code null} if not found. * * @throws BigQueryException upon failure */ - DatasetInfo getDataset(DatasetId datasetId, DatasetOption... options) throws BigQueryException; + Dataset getDataset(DatasetId datasetId, DatasetOption... options) throws BigQueryException; /** * Lists the project's datasets. This method returns partial information on each dataset - * ({@link DatasetInfo#datasetId()}, {@link DatasetInfo#friendlyName()} and - * {@link DatasetInfo#id()}). To get complete information use either - * {@link #getDataset(String, DatasetOption...)} or + * ({@link Dataset#datasetId()}, {@link Dataset#friendlyName()} and {@link Dataset#id()}). To get + * complete information use either {@link #getDataset(String, DatasetOption...)} or * {@link #getDataset(DatasetId, DatasetOption...)}. * * @throws BigQueryException upon failure */ - Page listDatasets(DatasetListOption... options) throws BigQueryException; + Page listDatasets(DatasetListOption... options) throws BigQueryException; /** * Deletes the requested dataset. @@ -534,54 +534,50 @@ public static QueryResultsOption maxWaitTime(long maxWaitTime) { * * @throws BigQueryException upon failure */ - DatasetInfo update(DatasetInfo dataset, DatasetOption... options) throws BigQueryException; + Dataset update(DatasetInfo dataset, DatasetOption... options) throws BigQueryException; /** * Updates table information. * * @throws BigQueryException upon failure */ - T update(T table, TableOption... options) throws BigQueryException; + Table update(TableInfo table, TableOption... options) throws BigQueryException; /** * Returns the requested table or {@code null} if not found. * * @throws BigQueryException upon failure */ - T getTable(String datasetId, String tableId, TableOption... options) - throws BigQueryException; + Table getTable(String datasetId, String tableId, TableOption... options) throws BigQueryException; /** * Returns the requested table or {@code null} if not found. * * @throws BigQueryException upon failure */ - T getTable(TableId tableId, TableOption... options) - throws BigQueryException; + Table getTable(TableId tableId, TableOption... options) throws BigQueryException; /** * Lists the tables in the dataset. This method returns partial information on each table - * ({@link BaseTableInfo#tableId()}, {@link BaseTableInfo#friendlyName()}, - * {@link BaseTableInfo#id()} and {@link BaseTableInfo#type()}). To get complete information use - * either {@link #getTable(TableId, TableOption...)} or + * ({@link Table#tableId()}, {@link Table#friendlyName()}, {@link Table#id()} and type, which + * is part of {@link Table#definition()}). To get complete information use either + * {@link #getTable(TableId, TableOption...)} or * {@link #getTable(String, String, TableOption...)}. * * @throws BigQueryException upon failure */ - Page listTables(String datasetId, TableListOption... options) - throws BigQueryException; + Page listTables(String datasetId, TableListOption... options) throws BigQueryException; /** * Lists the tables in the dataset. This method returns partial information on each table - * ({@link BaseTableInfo#tableId()}, {@link BaseTableInfo#friendlyName()}, - * {@link BaseTableInfo#id()} and {@link BaseTableInfo#type()}). To get complete information use - * either {@link #getTable(TableId, TableOption...)} or + * ({@link Table#tableId()}, {@link Table#friendlyName()}, {@link Table#id()} and type, which + * is part of {@link Table#definition()}). To get complete information use either + * {@link #getTable(TableId, TableOption...)} or * {@link #getTable(String, String, TableOption...)}. * * @throws BigQueryException upon failure */ - Page listTables(DatasetId datasetId, TableListOption... options) - throws BigQueryException; + Page
listTables(DatasetId datasetId, TableListOption... options) throws BigQueryException; /** * Sends an insert all request. @@ -611,21 +607,21 @@ Page> listTableData(TableId tableId, TableDataListOption... opt * * @throws BigQueryException upon failure */ - T getJob(String jobId, JobOption... options) throws BigQueryException; + Job getJob(String jobId, JobOption... options) throws BigQueryException; /** * Returns the requested job or {@code null} if not found. * * @throws BigQueryException upon failure */ - T getJob(JobId jobId, JobOption... options) throws BigQueryException; + Job getJob(JobId jobId, JobOption... options) throws BigQueryException; /** * Lists the jobs. * * @throws BigQueryException upon failure */ - Page listJobs(JobListOption... options) throws BigQueryException; + Page listJobs(JobListOption... options) throws BigQueryException; /** * Sends a job cancel request. This call will return immediately. The job status can then be @@ -665,9 +661,9 @@ Page> listTableData(TableId tableId, TableDataListOption... opt /** * Returns a channel to write data to be inserted into a BigQuery table. Data format and other - * options can be configured using the {@link LoadConfiguration} parameter. + * options can be configured using the {@link WriteChannelConfiguration} parameter. * * @throws BigQueryException upon failure */ - TableDataWriteChannel writer(LoadConfiguration loadConfiguration); + TableDataWriteChannel writer(WriteChannelConfiguration writeChannelConfiguration); } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryException.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryException.java index c42bbd96db66..a157afd25db2 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryException.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryException.java @@ -16,8 +16,6 @@ package com.google.gcloud.bigquery; -import com.google.api.client.googleapis.json.GoogleJsonError; -import com.google.api.client.googleapis.json.GoogleJsonResponseException; import com.google.common.collect.ImmutableSet; import com.google.gcloud.BaseServiceException; import com.google.gcloud.RetryHelper.RetryHelperException; diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryImpl.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryImpl.java index dde918d1f516..1158dd86c83d 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryImpl.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryImpl.java @@ -19,13 +19,9 @@ import static com.google.common.base.Preconditions.checkArgument; import static com.google.gcloud.RetryHelper.runWithRetries; -import com.google.api.services.bigquery.model.Dataset; import com.google.api.services.bigquery.model.GetQueryResultsResponse; -import com.google.api.services.bigquery.model.Job; -import com.google.api.services.bigquery.model.Table; import com.google.api.services.bigquery.model.TableDataInsertAllRequest; import com.google.api.services.bigquery.model.TableDataInsertAllRequest.Rows; -import com.google.api.services.bigquery.model.TableReference; import com.google.api.services.bigquery.model.TableRow; import com.google.common.base.Function; import com.google.common.collect.ImmutableList; @@ -47,7 +43,7 @@ final class BigQueryImpl extends BaseService implements BigQuery { - private static class DatasetPageFetcher implements NextPageFetcher { + private static class DatasetPageFetcher implements NextPageFetcher { private static final long serialVersionUID = -3057564042439021278L; private final Map requestOptions; @@ -61,12 +57,12 @@ private static class DatasetPageFetcher implements NextPageFetcher } @Override - public Page nextPage() { + public Page nextPage() { return listDatasets(serviceOptions, requestOptions); } } - private static class TablePageFetcher implements NextPageFetcher { + private static class TablePageFetcher implements NextPageFetcher
{ private static final long serialVersionUID = 8611248840504201187L; private final Map requestOptions; @@ -82,12 +78,12 @@ private static class TablePageFetcher implements NextPageFetcher } @Override - public Page nextPage() { + public Page
nextPage() { return listTables(dataset, serviceOptions, requestOptions); } } - private static class JobPageFetcher implements NextPageFetcher { + private static class JobPageFetcher implements NextPageFetcher { private static final long serialVersionUID = 8536533282558245472L; private final Map requestOptions; @@ -101,7 +97,7 @@ private static class JobPageFetcher implements NextPageFetcher { } @Override - public Page nextPage() { + public Page nextPage() { return listJobs(serviceOptions, requestOptions); } } @@ -157,96 +153,108 @@ public QueryResult nextPage() { } @Override - public DatasetInfo create(DatasetInfo dataset, DatasetOption... options) - throws BigQueryException { - final Dataset datasetPb = setProjectId(dataset).toPb(); + public Dataset create(DatasetInfo dataset, DatasetOption... options) throws BigQueryException { + final com.google.api.services.bigquery.model.Dataset datasetPb = + dataset.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return DatasetInfo.fromPb(runWithRetries(new Callable() { - @Override - public Dataset call() { - return bigQueryRpc.create(datasetPb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Dataset.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Dataset call() { + return bigQueryRpc.create(datasetPb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public T create(T table, TableOption... options) - throws BigQueryException { - final Table tablePb = setProjectId(table).toPb(); + public Table create(TableInfo table, TableOption... options) throws BigQueryException { + final com.google.api.services.bigquery.model.Table tablePb = + table.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return BaseTableInfo.fromPb(runWithRetries(new Callable
() { - @Override - public Table call() { - return bigQueryRpc.create(tablePb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Table.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Table call() { + return bigQueryRpc.create(tablePb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public T create(T job, JobOption... options) throws BigQueryException { - final Job jobPb = setProjectId(job).toPb(); + public Job create(JobInfo job, JobOption... options) throws BigQueryException { + final com.google.api.services.bigquery.model.Job jobPb = + job.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return JobInfo.fromPb(runWithRetries(new Callable() { - @Override - public Job call() { - return bigQueryRpc.create(jobPb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Job.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Job call() { + return bigQueryRpc.create(jobPb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public DatasetInfo getDataset(String datasetId, DatasetOption... options) - throws BigQueryException { + public Dataset getDataset(String datasetId, DatasetOption... options) throws BigQueryException { return getDataset(DatasetId.of(datasetId), options); } @Override - public DatasetInfo getDataset(final DatasetId datasetId, DatasetOption... options) + public Dataset getDataset(final DatasetId datasetId, DatasetOption... options) throws BigQueryException { final Map optionsMap = optionMap(options); try { - Dataset answer = runWithRetries(new Callable() { - @Override - public Dataset call() { - return bigQueryRpc.getDataset(datasetId.dataset(), optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER); - return answer == null ? null : DatasetInfo.fromPb(answer); + com.google.api.services.bigquery.model.Dataset answer = + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Dataset call() { + return bigQueryRpc.getDataset(datasetId.dataset(), optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER); + return answer == null ? null : Dataset.fromPb(this, answer); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public Page listDatasets(DatasetListOption... options) throws BigQueryException { + public Page listDatasets(DatasetListOption... options) throws BigQueryException { return listDatasets(options(), optionMap(options)); } - private static Page listDatasets(final BigQueryOptions serviceOptions, + private static Page listDatasets(final BigQueryOptions serviceOptions, final Map optionsMap) { try { - BigQueryRpc.Tuple> result = - runWithRetries(new Callable>>() { - @Override - public BigQueryRpc.Tuple> call() { - return serviceOptions.rpc().listDatasets(optionsMap); - } - }, serviceOptions.retryParams(), EXCEPTION_HANDLER); + BigQueryRpc.Tuple> result = + runWithRetries(new Callable>>() { + @Override + public BigQueryRpc.Tuple> call() { + return serviceOptions.rpc().listDatasets(optionsMap); + } + }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); return new PageImpl<>(new DatasetPageFetcher(serviceOptions, cursor, optionsMap), cursor, - Iterables.transform(result.y(), DatasetInfo.FROM_PB_FUNCTION)); + Iterables.transform(result.y(), + new Function() { + @Override + public Dataset apply(com.google.api.services.bigquery.model.Dataset dataset) { + return Dataset.fromPb(serviceOptions.service(), dataset); + } + })); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } @@ -293,87 +301,96 @@ public Boolean call() { } @Override - public DatasetInfo update(DatasetInfo dataset, DatasetOption... options) - throws BigQueryException { - final Dataset datasetPb = setProjectId(dataset).toPb(); + public Dataset update(DatasetInfo dataset, DatasetOption... options) throws BigQueryException { + final com.google.api.services.bigquery.model.Dataset datasetPb = + dataset.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return DatasetInfo.fromPb(runWithRetries(new Callable() { - @Override - public Dataset call() { - return bigQueryRpc.patch(datasetPb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Dataset.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Dataset call() { + return bigQueryRpc.patch(datasetPb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public T update(T table, TableOption... options) - throws BigQueryException { - final Table tablePb = setProjectId(table).toPb(); + public Table update(TableInfo table, TableOption... options) throws BigQueryException { + final com.google.api.services.bigquery.model.Table tablePb = + table.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return BaseTableInfo.fromPb(runWithRetries(new Callable
() { - @Override - public Table call() { - return bigQueryRpc.patch(tablePb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Table.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Table call() { + return bigQueryRpc.patch(tablePb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public T getTable(final String datasetId, final String tableId, - TableOption... options) throws BigQueryException { + public Table getTable(final String datasetId, final String tableId, TableOption... options) + throws BigQueryException { return getTable(TableId.of(datasetId, tableId), options); } @Override - public T getTable(final TableId tableId, TableOption... options) - throws BigQueryException { + public Table getTable(final TableId tableId, TableOption... options) throws BigQueryException { final Map optionsMap = optionMap(options); try { - Table answer = runWithRetries(new Callable
() { - @Override - public Table call() { - return bigQueryRpc.getTable(tableId.dataset(), tableId.table(), optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER); - return answer == null ? null : BaseTableInfo.fromPb(answer); + com.google.api.services.bigquery.model.Table answer = + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Table call() { + return bigQueryRpc.getTable(tableId.dataset(), tableId.table(), optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER); + return answer == null ? null : Table.fromPb(this, answer); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public Page listTables(String datasetId, TableListOption... options) + public Page
listTables(String datasetId, TableListOption... options) throws BigQueryException { return listTables(datasetId, options(), optionMap(options)); } @Override - public Page listTables(DatasetId datasetId, TableListOption... options) + public Page
listTables(DatasetId datasetId, TableListOption... options) throws BigQueryException { return listTables(datasetId.dataset(), options(), optionMap(options)); } - private static Page listTables(final String datasetId, final BigQueryOptions + private static Page
listTables(final String datasetId, final BigQueryOptions serviceOptions, final Map optionsMap) { try { - BigQueryRpc.Tuple> result = - runWithRetries(new Callable>>() { + BigQueryRpc.Tuple> result = + runWithRetries(new Callable>>() { @Override - public BigQueryRpc.Tuple> call() { - return serviceOptions.rpc().listTables(datasetId, optionsMap); - } + public BigQueryRpc.Tuple> + call() { + return serviceOptions.rpc().listTables(datasetId, optionsMap); + } }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); - Iterable tables = Iterables.transform(result.y(), - BaseTableInfo.FROM_PB_FUNCTION); + Iterable
tables = Iterables.transform(result.y(), + new Function() { + @Override + public Table apply(com.google.api.services.bigquery.model.Table table) { + return Table.fromPb(serviceOptions.service(), table); + } + }); return new PageImpl<>(new TablePageFetcher(datasetId, serviceOptions, cursor, optionsMap), cursor, tables); } catch (RetryHelper.RetryHelperException e) { @@ -442,43 +459,51 @@ public List apply(TableRow rowPb) { } @Override - public T getJob(String jobId, JobOption... options) throws BigQueryException { + public Job getJob(String jobId, JobOption... options) throws BigQueryException { return getJob(JobId.of(jobId), options); } @Override - public T getJob(final JobId jobId, JobOption... options) - throws BigQueryException { + public Job getJob(final JobId jobId, JobOption... options) throws BigQueryException { final Map optionsMap = optionMap(options); try { - Job answer = runWithRetries(new Callable() { - @Override - public Job call() { - return bigQueryRpc.getJob(jobId.job(), optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER); - return answer == null ? null : JobInfo.fromPb(answer); + com.google.api.services.bigquery.model.Job answer = + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Job call() { + return bigQueryRpc.getJob(jobId.job(), optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER); + return answer == null ? null : Job.fromPb(this, answer); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public Page listJobs(JobListOption... options) throws BigQueryException { + public Page listJobs(JobListOption... options) throws BigQueryException { return listJobs(options(), optionMap(options)); } - private static Page listJobs(final BigQueryOptions serviceOptions, + private static Page listJobs(final BigQueryOptions serviceOptions, final Map optionsMap) { - BigQueryRpc.Tuple> result = - runWithRetries(new Callable>>() { + BigQueryRpc.Tuple> result = + runWithRetries(new Callable>>() { @Override - public BigQueryRpc.Tuple> call() { + public BigQueryRpc.Tuple> + call() { return serviceOptions.rpc().listJobs(optionsMap); } }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); - Iterable jobs = Iterables.transform(result.y(), JobInfo.FROM_PB_FUNCTION); + Iterable jobs = Iterables.transform(result.y(), + new Function() { + @Override + public Job apply(com.google.api.services.bigquery.model.Job job) { + return Job.fromPb(serviceOptions.service(), job); + } + }); return new PageImpl<>(new JobPageFetcher(serviceOptions, cursor, optionsMap), cursor, jobs); } @@ -508,7 +533,7 @@ public QueryResponse query(final QueryRequest request) throws BigQueryException runWithRetries(new Callable() { @Override public com.google.api.services.bigquery.model.QueryResponse call() { - return bigQueryRpc.query(setProjectId(request).toPb()); + return bigQueryRpc.query(request.setProjectId(options().projectId()).toPb()); } }, options().retryParams(), EXCEPTION_HANDLER); QueryResponse.Builder builder = QueryResponse.builder(); @@ -596,8 +621,10 @@ private static QueryResult.Builder transformQueryResults(JobId jobId, List
optionMap(Option... options) { @@ -608,93 +635,4 @@ public TableDataWriteChannel writer(LoadConfiguration loadConfiguration) { } return optionMap; } - - private DatasetInfo setProjectId(DatasetInfo dataset) { - DatasetInfo.Builder datasetBuilder = dataset.toBuilder(); - datasetBuilder.datasetId(setProjectId(dataset.datasetId())); - if (dataset.acl() != null) { - List acls = Lists.newArrayListWithCapacity(dataset.acl().size()); - for (Acl acl : dataset.acl()) { - if (acl.entity().type() == Acl.Entity.Type.VIEW) { - Dataset.Access accessPb = acl.toPb(); - TableReference viewReferencePb = accessPb.getView(); - if (viewReferencePb.getProjectId() == null) { - viewReferencePb.setProjectId(options().projectId()); - } - acls.add(Acl.of(new Acl.View(TableId.fromPb(viewReferencePb)))); - } else { - acls.add(acl); - } - } - datasetBuilder.acl(acls); - } - return datasetBuilder.build(); - } - - private DatasetId setProjectId(DatasetId dataset) { - return dataset.project() != null ? dataset - : DatasetId.of(options().projectId(), dataset.dataset()); - } - - private BaseTableInfo setProjectId(BaseTableInfo table) { - return table.toBuilder().tableId(setProjectId(table.tableId())).build(); - } - - private TableId setProjectId(TableId table) { - return table.project() != null ? table - : TableId.of(options().projectId(), table.dataset(), table.table()); - } - - private JobInfo setProjectId(JobInfo job) { - if (job instanceof CopyJobInfo) { - CopyJobInfo copyJob = (CopyJobInfo) job; - CopyJobInfo.Builder copyBuilder = copyJob.toBuilder(); - copyBuilder.destinationTable(setProjectId(copyJob.destinationTable())); - copyBuilder.sourceTables( - Lists.transform(copyJob.sourceTables(), new Function() { - @Override - public TableId apply(TableId tableId) { - return setProjectId(tableId); - } - })); - return copyBuilder.build(); - } - if (job instanceof QueryJobInfo) { - QueryJobInfo queryJob = (QueryJobInfo) job; - QueryJobInfo.Builder queryBuilder = queryJob.toBuilder(); - if (queryJob.destinationTable() != null) { - queryBuilder.destinationTable(setProjectId(queryJob.destinationTable())); - } - if (queryJob.defaultDataset() != null) { - queryBuilder.defaultDataset(setProjectId(queryJob.defaultDataset())); - } - return queryBuilder.build(); - } - if (job instanceof ExtractJobInfo) { - ExtractJobInfo extractJob = (ExtractJobInfo) job; - ExtractJobInfo.Builder extractBuilder = extractJob.toBuilder(); - extractBuilder.sourceTable(setProjectId(extractJob.sourceTable())); - return extractBuilder.build(); - } - if (job instanceof LoadJobInfo) { - LoadJobInfo loadJob = (LoadJobInfo) job; - LoadJobInfo.Builder loadBuilder = loadJob.toBuilder(); - return loadBuilder.configuration(setProjectId(loadJob.configuration())).build(); - } - return job; - } - - private QueryRequest setProjectId(QueryRequest request) { - QueryRequest.Builder builder = request.toBuilder(); - if (request.defaultDataset() != null) { - builder.defaultDataset(setProjectId(request.defaultDataset())); - } - return builder.build(); - } - - private LoadConfiguration setProjectId(LoadConfiguration configuration) { - LoadConfiguration.Builder builder = configuration.toBuilder(); - builder.destinationTable(setProjectId(configuration.destinationTable())); - return builder.build(); - } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CopyJobInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CopyJobConfiguration.java similarity index 56% rename from gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CopyJobInfo.java rename to gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CopyJobConfiguration.java index bd346a8e1633..c12cbf5fe432 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CopyJobInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CopyJobConfiguration.java @@ -1,5 +1,5 @@ /* - * Copyright 2015 Google Inc. All Rights Reserved. + * Copyright 2016 Google Inc. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -18,9 +18,8 @@ import static com.google.common.base.Preconditions.checkNotNull; -import com.google.api.services.bigquery.model.Job; -import com.google.api.services.bigquery.model.JobConfiguration; import com.google.api.services.bigquery.model.JobConfigurationTableCopy; +import com.google.common.base.Function; import com.google.common.base.MoreObjects.ToStringHelper; import com.google.common.collect.ImmutableList; import com.google.common.collect.Lists; @@ -29,37 +28,41 @@ import java.util.Objects; /** - * Google BigQuery Copy Job. A Copy Job copies an existing table to another new or existing table. + * Google BigQuery copy job configuration. A copy job copies an existing table to another new or + * existing table. Copy job configurations have {@link JobConfiguration.Type#COPY} type. */ -public class CopyJobInfo extends JobInfo { +public final class CopyJobConfiguration extends JobConfiguration { - private static final long serialVersionUID = 7830335512951916299L; + private static final long serialVersionUID = 1140509641399762967L; private final List sourceTables; private final TableId destinationTable; - private final CreateDisposition createDisposition; - private final WriteDisposition writeDisposition; + private final JobInfo.CreateDisposition createDisposition; + private final JobInfo.WriteDisposition writeDisposition; - public static final class Builder extends JobInfo.Builder { + public static final class Builder + extends JobConfiguration.Builder { private List sourceTables; private TableId destinationTable; - private CreateDisposition createDisposition; - private WriteDisposition writeDisposition; + private JobInfo.CreateDisposition createDisposition; + private JobInfo.WriteDisposition writeDisposition; - private Builder() {} + private Builder() { + super(Type.COPY); + } - private Builder(CopyJobInfo jobInfo) { - super(jobInfo); - this.sourceTables = jobInfo.sourceTables; - this.destinationTable = jobInfo.destinationTable; - this.createDisposition = jobInfo.createDisposition; - this.writeDisposition = jobInfo.writeDisposition; + private Builder(CopyJobConfiguration jobConfiguration) { + this(); + this.sourceTables = jobConfiguration.sourceTables; + this.destinationTable = jobConfiguration.destinationTable; + this.createDisposition = jobConfiguration.createDisposition; + this.writeDisposition = jobConfiguration.writeDisposition; } - private Builder(Job jobPb) { - super(jobPb); - JobConfigurationTableCopy copyConfigurationPb = jobPb.getConfiguration().getCopy(); + private Builder(com.google.api.services.bigquery.model.JobConfiguration configurationPb) { + this(); + JobConfigurationTableCopy copyConfigurationPb = configurationPb.getCopy(); this.destinationTable = TableId.fromPb(copyConfigurationPb.getDestinationTable()); if (copyConfigurationPb.getSourceTables() != null) { this.sourceTables = @@ -69,10 +72,11 @@ private Builder(Job jobPb) { } if (copyConfigurationPb.getCreateDisposition() != null) { this.createDisposition = - CreateDisposition.valueOf(copyConfigurationPb.getCreateDisposition()); + JobInfo.CreateDisposition.valueOf(copyConfigurationPb.getCreateDisposition()); } if (copyConfigurationPb.getWriteDisposition() != null) { - this.writeDisposition = WriteDisposition.valueOf(copyConfigurationPb.getWriteDisposition()); + this.writeDisposition = JobInfo.WriteDisposition.valueOf( + copyConfigurationPb.getWriteDisposition()); } } @@ -81,7 +85,7 @@ private Builder(Job jobPb) { */ public Builder sourceTables(List sourceTables) { this.sourceTables = sourceTables != null ? ImmutableList.copyOf(sourceTables) : null; - return self(); + return this; } /** @@ -89,7 +93,7 @@ public Builder sourceTables(List sourceTables) { */ public Builder destinationTable(TableId destinationTable) { this.destinationTable = destinationTable; - return self(); + return this; } /** @@ -98,9 +102,9 @@ public Builder destinationTable(TableId destinationTable) { * @see * Create Disposition */ - public Builder createDisposition(CreateDisposition createDisposition) { + public Builder createDisposition(JobInfo.CreateDisposition createDisposition) { this.createDisposition = createDisposition; - return self(); + return this; } /** @@ -109,18 +113,17 @@ public Builder createDisposition(CreateDisposition createDisposition) { * @see * Write Disposition */ - public Builder writeDisposition(WriteDisposition writeDisposition) { + public Builder writeDisposition(JobInfo.WriteDisposition writeDisposition) { this.writeDisposition = writeDisposition; - return self(); + return this; } - @Override - public CopyJobInfo build() { - return new CopyJobInfo(this); + public CopyJobConfiguration build() { + return new CopyJobConfiguration(this); } } - private CopyJobInfo(Builder builder) { + private CopyJobConfiguration(Builder builder) { super(builder); this.sourceTables = checkNotNull(builder.sourceTables); this.destinationTable = checkNotNull(builder.destinationTable); @@ -148,7 +151,7 @@ public TableId destinationTable() { * @see * Create Disposition */ - public CreateDisposition createDisposition() { + public JobInfo.CreateDisposition createDisposition() { return this.createDisposition; } @@ -158,7 +161,7 @@ public CreateDisposition createDisposition() { * @see * Write Disposition */ - public WriteDisposition writeDisposition() { + public JobInfo.WriteDisposition writeDisposition() { return writeDisposition; } @@ -178,81 +181,78 @@ ToStringHelper toStringHelper() { @Override public boolean equals(Object obj) { - return obj instanceof CopyJobInfo && Objects.equals(toPb(), ((CopyJobInfo) obj).toPb()); + return obj instanceof CopyJobConfiguration && baseEquals((CopyJobConfiguration) obj); } @Override public int hashCode() { - return Objects.hash(super.hashCode(), sourceTables, destinationTable, createDisposition, + return Objects.hash(baseHashCode(), sourceTables, destinationTable, createDisposition, writeDisposition); } @Override - Job toPb() { - JobConfigurationTableCopy copyConfigurationPb = new JobConfigurationTableCopy(); - copyConfigurationPb.setDestinationTable(destinationTable.toPb()); + CopyJobConfiguration setProjectId(final String projectId) { + Builder builder = toBuilder(); + builder.sourceTables( + Lists.transform(sourceTables(), new Function() { + @Override + public TableId apply(TableId tableId) { + return tableId.setProjectId(projectId); + } + })); + builder.destinationTable(destinationTable().setProjectId(projectId)); + return builder.build(); + } + + @Override + com.google.api.services.bigquery.model.JobConfiguration toPb() { + JobConfigurationTableCopy configurationPb = new JobConfigurationTableCopy(); + configurationPb.setDestinationTable(destinationTable.toPb()); if (sourceTables.size() == 1) { - copyConfigurationPb.setSourceTable(sourceTables.get(0).toPb()); + configurationPb.setSourceTable(sourceTables.get(0).toPb()); } else { - copyConfigurationPb.setSourceTables(Lists.transform(sourceTables, TableId.TO_PB_FUNCTION)); + configurationPb.setSourceTables(Lists.transform(sourceTables, TableId.TO_PB_FUNCTION)); } if (createDisposition != null) { - copyConfigurationPb.setCreateDisposition(createDisposition.toString()); + configurationPb.setCreateDisposition(createDisposition.toString()); } if (writeDisposition != null) { - copyConfigurationPb.setWriteDisposition(writeDisposition.toString()); + configurationPb.setWriteDisposition(writeDisposition.toString()); } - return super.toPb().setConfiguration(new JobConfiguration().setCopy(copyConfigurationPb)); + return new com.google.api.services.bigquery.model.JobConfiguration().setCopy(configurationPb); } /** - * Creates a builder for a BigQuery Copy Job given destination and source table. + * Creates a builder for a BigQuery Copy Job configuration given destination and source table. */ public static Builder builder(TableId destinationTable, TableId sourceTable) { return builder(destinationTable, ImmutableList.of(checkNotNull(sourceTable))); } /** - * Creates a builder for a BigQuery Copy Job given destination and source tables. + * Creates a builder for a BigQuery Copy Job configuration given destination and source tables. */ public static Builder builder(TableId destinationTable, List sourceTables) { return new Builder().destinationTable(destinationTable).sourceTables(sourceTables); } /** - * Returns a BigQuery Copy Job for the given destination and source table. Job's id is chosen by - * the service. + * Returns a BigQuery Copy Job configuration for the given destination and source table. */ - public static CopyJobInfo of(TableId destinationTable, TableId sourceTable) { + public static CopyJobConfiguration of(TableId destinationTable, TableId sourceTable) { return builder(destinationTable, sourceTable).build(); } /** - * Returns a BigQuery Copy Job for the given destination and source tables. Job's id is chosen by - * the service. + * Returns a BigQuery Copy Job configuration for the given destination and source tables. */ - public static CopyJobInfo of(TableId destinationTable, List sourceTables) { + public static CopyJobConfiguration of(TableId destinationTable, List sourceTables) { return builder(destinationTable, sourceTables).build(); } - /** - * Returns a BigQuery Copy Job for the given destination and source table. Job's id is set to the - * provided value. - */ - public static CopyJobInfo of(JobId jobId, TableId destinationTable, TableId sourceTable) { - return builder(destinationTable, sourceTable).jobId(jobId).build(); - } - - /** - * Returns a BigQuery Copy Job for the given destination and source tables. Job's id is set to the - * provided value. - */ - public static CopyJobInfo of(JobId jobId, TableId destinationTable, List sourceTables) { - return builder(destinationTable, sourceTables).jobId(jobId).build(); - } - @SuppressWarnings("unchecked") - static CopyJobInfo fromPb(Job jobPb) { + static CopyJobConfiguration fromPb( + com.google.api.services.bigquery.model.JobConfiguration jobPb) { return new Builder(jobPb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CsvOptions.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CsvOptions.java index 274ef5678a8a..9576e7d75640 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CsvOptions.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/CsvOptions.java @@ -144,7 +144,7 @@ private CsvOptions(Builder builder) { * Returns whether BigQuery should accept rows that are missing trailing optional columns. If * {@code true}, BigQuery treats missing trailing columns as null values. If {@code false}, * records with missing trailing columns are treated as bad records, and if the number of bad - * records exceeds {@link ExternalDataConfiguration#maxBadRecords()}, an invalid error is returned + * records exceeds {@link ExternalTableDefinition#maxBadRecords()}, an invalid error is returned * in the job result. */ public Boolean allowJaggedRows() { diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Dataset.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Dataset.java index facf5e659f99..e17d3e82c4ef 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Dataset.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Dataset.java @@ -16,18 +16,12 @@ package com.google.gcloud.bigquery; -import static com.google.common.base.Preconditions.checkArgument; import static com.google.common.base.Preconditions.checkNotNull; -import com.google.common.base.Function; -import com.google.common.collect.Iterators; import com.google.gcloud.Page; -import com.google.gcloud.PageImpl; import java.io.IOException; import java.io.ObjectInputStream; -import java.io.Serializable; -import java.util.Iterator; import java.util.List; import java.util.Objects; @@ -36,111 +30,112 @@ * *

Objects of this class are immutable. Operations that modify the dataset like {@link #update} * return a new object. To get a {@code Dataset} object with the most recent information use - * {@link #reload}. + * {@link #reload}. {@code Dataset} adds a layer of service-related functionality over + * {@link DatasetInfo}. *

*/ -public final class Dataset { +public final class Dataset extends DatasetInfo { - private final BigQuery bigquery; - private final DatasetInfo info; + private static final long serialVersionUID = -4272921483363065593L; - private static class TablePageFetcher implements PageImpl.NextPageFetcher
{ + private final BigQueryOptions options; + private transient BigQuery bigquery; - private static final long serialVersionUID = 6906197848579250598L; + /** + * A builder for {@code Dataset} objects. + */ + public static final class Builder extends DatasetInfo.Builder { - private final BigQueryOptions options; - private final Page infoPage; + private final BigQuery bigquery; + private final DatasetInfo.BuilderImpl infoBuilder; - TablePageFetcher(BigQueryOptions options, Page infoPage) { - this.options = options; - this.infoPage = infoPage; + Builder(BigQuery bigquery, DatasetId datasetId) { + this.bigquery = bigquery; + this.infoBuilder = new DatasetInfo.BuilderImpl(); + this.infoBuilder.datasetId(datasetId); + } + + Builder(Dataset dataset) { + this.bigquery = dataset.bigquery; + this.infoBuilder = new DatasetInfo.BuilderImpl(dataset); } @Override - public Page
nextPage() { - Page nextInfoPage = infoPage.nextPage(); - return new PageImpl<>(new TablePageFetcher(options, nextInfoPage), - nextInfoPage.nextPageCursor(), new LazyTableIterable(options, nextInfoPage.values())); + public Builder datasetId(DatasetId datasetId) { + infoBuilder.datasetId(datasetId); + return this; } - } - private static class LazyTableIterable implements Iterable
, Serializable { + @Override + public Builder acl(List acl) { + infoBuilder.acl(acl); + return this; + } - private static final long serialVersionUID = 3312744215731674032L; + @Override + Builder creationTime(Long creationTime) { + infoBuilder.creationTime(creationTime); + return this; + } - private final BigQueryOptions options; - private final Iterable infoIterable; - private transient BigQuery bigquery; + @Override + public Builder defaultTableLifetime(Long defaultTableLifetime) { + infoBuilder.defaultTableLifetime(defaultTableLifetime); + return this; + } - public LazyTableIterable(BigQueryOptions options, Iterable infoIterable) { - this.options = options; - this.infoIterable = infoIterable; - this.bigquery = options.service(); + @Override + public Builder description(String description) { + infoBuilder.description(description); + return this; } - private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { - in.defaultReadObject(); - this.bigquery = options.service(); + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; } @Override - public Iterator
iterator() { - return Iterators.transform(infoIterable.iterator(), new Function() { - @Override - public Table apply(BaseTableInfo tableInfo) { - return new Table(bigquery, tableInfo); - } - }); + public Builder friendlyName(String friendlyName) { + infoBuilder.friendlyName(friendlyName); + return this; } @Override - public int hashCode() { - return Objects.hash(options, infoIterable); + Builder id(String id) { + infoBuilder.id(id); + return this; } @Override - public boolean equals(Object obj) { - if (!(obj instanceof LazyTableIterable)) { - return false; - } - LazyTableIterable other = (LazyTableIterable) obj; - return Objects.equals(options, other.options) - && Objects.equals(infoIterable, other.infoIterable); + Builder lastModified(Long lastModified) { + infoBuilder.lastModified(lastModified); + return this; } - } - /** - * Constructs a {@code Dataset} object for the provided {@code DatasetInfo}. The BigQuery service - * is used to issue requests. - * - * @param bigquery the BigQuery service used for issuing requests - * @param info dataset's info - */ - public Dataset(BigQuery bigquery, DatasetInfo info) { - this.bigquery = checkNotNull(bigquery); - this.info = checkNotNull(info); - } + @Override + public Builder location(String location) { + infoBuilder.location(location); + return this; + } - /** - * Creates a {@code Dataset} object for the provided dataset's user-defined id. Performs an RPC - * call to get the latest dataset information. - * - * @param bigquery the BigQuery service used for issuing requests - * @param dataset dataset's user-defined id - * @param options dataset options - * @return the {@code Dataset} object or {@code null} if not found - * @throws BigQueryException upon failure - */ - public static Dataset get(BigQuery bigquery, String dataset, BigQuery.DatasetOption... options) { - DatasetInfo info = bigquery.getDataset(dataset, options); - return info != null ? new Dataset(bigquery, info) : null; + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + public Dataset build() { + return new Dataset(bigquery, infoBuilder); + } } - /** - * Returns the dataset's information. - */ - public DatasetInfo info() { - return info; + Dataset(BigQuery bigquery, DatasetInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.bigquery = checkNotNull(bigquery); + this.options = bigquery.options(); } /** @@ -150,7 +145,7 @@ public DatasetInfo info() { * @throws BigQueryException upon failure */ public boolean exists() { - return bigquery.getDataset(info.datasetId(), BigQuery.DatasetOption.fields()) != null; + return bigquery.getDataset(datasetId(), BigQuery.DatasetOption.fields()) != null; } /** @@ -162,23 +157,19 @@ public boolean exists() { * @throws BigQueryException upon failure */ public Dataset reload(BigQuery.DatasetOption... options) { - return Dataset.get(bigquery, info.datasetId().dataset(), options); + return bigquery.getDataset(datasetId().dataset(), options); } /** - * Updates the dataset's information. Dataset's user-defined id cannot be changed. A new - * {@code Dataset} object is returned. + * Updates the dataset's information with this dataset's information. Dataset's user-defined id + * cannot be changed. A new {@code Dataset} object is returned. * - * @param datasetInfo new dataset's information. User-defined id must match the one of the current - * dataset * @param options dataset options * @return a {@code Dataset} object with updated information * @throws BigQueryException upon failure */ - public Dataset update(DatasetInfo datasetInfo, BigQuery.DatasetOption... options) { - checkArgument(Objects.equals(datasetInfo.datasetId().dataset(), - info.datasetId().dataset()), "Dataset's user-defined ids must match"); - return new Dataset(bigquery, bigquery.update(datasetInfo, options)); + public Dataset update(BigQuery.DatasetOption... options) { + return bigquery.update(this, options); } /** @@ -188,7 +179,7 @@ public Dataset update(DatasetInfo datasetInfo, BigQuery.DatasetOption... options * @throws BigQueryException upon failure */ public boolean delete() { - return bigquery.delete(info.datasetId()); + return bigquery.delete(datasetId()); } /** @@ -198,10 +189,7 @@ public boolean delete() { * @throws BigQueryException upon failure */ public Page
list(BigQuery.TableListOption... options) { - Page infoPage = bigquery.listTables(info.datasetId(), options); - BigQueryOptions bigqueryOptions = bigquery.options(); - return new PageImpl<>(new TablePageFetcher(bigqueryOptions, infoPage), - infoPage.nextPageCursor(), new LazyTableIterable(bigqueryOptions, infoPage.values())); + return bigquery.listTables(datasetId(), options); } /** @@ -212,76 +200,54 @@ public Page
list(BigQuery.TableListOption... options) { * @throws BigQueryException upon failure */ public Table get(String table, BigQuery.TableOption... options) { - BaseTableInfo tableInfo = - bigquery.getTable(TableId.of(info.datasetId().dataset(), table), options); - return tableInfo != null ? new Table(bigquery, tableInfo) : null; + return bigquery.getTable(TableId.of(datasetId().dataset(), table), options); } /** - * Creates a new simple table in this dataset. + * Creates a new table in this dataset. * * @param table the table's user-defined id - * @param schema the table's schema + * @param definition the table's definition * @param options options for table creation * @return a {@code Table} object for the created table * @throws BigQueryException upon failure */ - public Table create(String table, Schema schema, BigQuery.TableOption... options) { - BaseTableInfo tableInfo = TableInfo.of(TableId.of(info.datasetId().dataset(), table), schema); - return new Table(bigquery, bigquery.create(tableInfo, options)); + public Table create(String table, TableDefinition definition, BigQuery.TableOption... options) { + TableInfo tableInfo = TableInfo.of(TableId.of(datasetId().dataset(), table), definition); + return bigquery.create(tableInfo, options); } /** - * Creates a new view table in this dataset. - * - * @param table the table's user-defined id - * @param query the query used to generate the table - * @param functions user-defined functions that can be used by the query - * @param options options for table creation - * @return a {@code Table} object for the created table - * @throws BigQueryException upon failure + * Returns the dataset's {@code BigQuery} object used to issue requests. */ - public Table create(String table, String query, List functions, - BigQuery.TableOption... options) { - BaseTableInfo tableInfo = - ViewInfo.of(TableId.of(info.datasetId().dataset(), table), query, functions); - return new Table(bigquery, bigquery.create(tableInfo, options)); + public BigQuery bigquery() { + return bigquery; } - /** - * Creates a new view table in this dataset. - * - * @param table the table's user-defined id - * @param query the query used to generate the table - * @param options options for table creation - * @return a {@code Table} object for the created table - * @throws BigQueryException upon failure - */ - public Table create(String table, String query, BigQuery.TableOption... options) { - BaseTableInfo tableInfo = ViewInfo.of(TableId.of(info.datasetId().dataset(), table), query); - return new Table(bigquery, bigquery.create(tableInfo, options)); + @Override + public Builder toBuilder() { + return new Builder(this); } - /** - * Creates a new external table in this dataset. - * - * @param table the table's user-defined id - * @param configuration data format, location and other properties of an external table - * @param options options for table creation - * @return a {@code Table} object for the created table - * @throws BigQueryException upon failure - */ - public Table create(String table, ExternalDataConfiguration configuration, - BigQuery.TableOption... options) { - BaseTableInfo tableInfo = - ExternalTableInfo.of(TableId.of(info.datasetId().dataset(), table), configuration); - return new Table(bigquery, bigquery.create(tableInfo, options)); + @Override + public boolean equals(Object obj) { + return obj instanceof Dataset + && Objects.equals(toPb(), ((Dataset) obj).toPb()) + && Objects.equals(options, ((Dataset) obj).options); } - /** - * Returns the dataset's {@code BigQuery} object used to issue requests. - */ - public BigQuery bigquery() { - return bigquery; + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); + } + + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.bigquery = options.service(); + } + + static Dataset fromPb(BigQuery bigquery, + com.google.api.services.bigquery.model.Dataset datasetPb) { + return new Dataset(bigquery, new DatasetInfo.BuilderImpl(datasetPb)); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetId.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetId.java index 942322ea51d3..006c089f8d63 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetId.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetId.java @@ -81,6 +81,10 @@ public String toString() { return toPb().toString(); } + DatasetId setProjectId(String projectId) { + return project() != null ? this : DatasetId.of(projectId, dataset()); + } + DatasetReference toPb() { return new DatasetReference().setProjectId(project).setDatasetId(dataset); } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetInfo.java index 95897ba3a801..aa767b97631b 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetInfo.java @@ -21,6 +21,7 @@ import com.google.api.client.util.Data; import com.google.api.services.bigquery.model.Dataset; +import com.google.api.services.bigquery.model.TableReference; import com.google.common.base.Function; import com.google.common.base.MoreObjects; import com.google.common.collect.ImmutableList; @@ -38,7 +39,7 @@ * @see * Managing Jobs, Datasets, and Projects */ -public final class DatasetInfo implements Serializable { +public class DatasetInfo implements Serializable { static final Function FROM_PB_FUNCTION = new Function() { @@ -69,7 +70,72 @@ public Dataset apply(DatasetInfo datasetInfo) { private final String location; private final String selfLink; - public static final class Builder { + /** + * A builder for {@code DatasetInfo} objects. + */ + public abstract static class Builder { + + /** + * Sets the dataset identity. + */ + public abstract Builder datasetId(DatasetId datasetId); + + /** + * Sets the dataset's access control configuration. + * + * @see Access Control + */ + public abstract Builder acl(List acl); + + abstract Builder creationTime(Long creationTime); + + /** + * Sets the default lifetime of all tables in the dataset, in milliseconds. The minimum value is + * 3600000 milliseconds (one hour). Once this property is set, all newly-created tables in the + * dataset will have an expirationTime property set to the creation time plus the value in this + * property, and changing the value will only affect new tables, not existing ones. When the + * expirationTime for a given table is reached, that table will be deleted automatically. If a + * table's expirationTime is modified or removed before the table expires, or if you provide an + * explicit expirationTime when creating a table, that value takes precedence over the default + * expiration time indicated by this property. This property is experimental and might be + * subject to change or removed. + */ + public abstract Builder defaultTableLifetime(Long defaultTableLifetime); + + /** + * Sets a user-friendly description for the dataset. + */ + public abstract Builder description(String description); + + abstract Builder etag(String etag); + + /** + * Sets a user-friendly name for the dataset. + */ + public abstract Builder friendlyName(String friendlyName); + + abstract Builder id(String id); + + abstract Builder lastModified(Long lastModified); + + /** + * Sets the geographic location where the dataset should reside. This property is experimental + * and might be subject to change or removed. + * + * @see Dataset + * Location + */ + public abstract Builder location(String location); + + abstract Builder selfLink(String selfLink); + + /** + * Creates a {@code DatasetInfo} object. + */ + public abstract DatasetInfo build(); + } + + static final class BuilderImpl extends Builder { private DatasetId datasetId; private List acl; @@ -83,9 +149,9 @@ public static final class Builder { private String location; private String selfLink; - private Builder() {} + BuilderImpl() {} - private Builder(DatasetInfo datasetInfo) { + BuilderImpl(DatasetInfo datasetInfo) { this.datasetId = datasetInfo.datasetId; this.acl = datasetInfo.acl; this.creationTime = datasetInfo.creationTime; @@ -99,103 +165,103 @@ private Builder(DatasetInfo datasetInfo) { this.selfLink = datasetInfo.selfLink; } - /** - * Sets the dataset identity. - */ + BuilderImpl(com.google.api.services.bigquery.model.Dataset datasetPb) { + if (datasetPb.getDatasetReference() != null) { + this.datasetId = DatasetId.fromPb(datasetPb.getDatasetReference()); + } + if (datasetPb.getAccess() != null) { + this.acl = Lists.transform(datasetPb.getAccess(), new Function() { + @Override + public Acl apply(Dataset.Access accessPb) { + return Acl.fromPb(accessPb); + } + }); + } + this.creationTime = datasetPb.getCreationTime(); + this.defaultTableLifetime = datasetPb.getDefaultTableExpirationMs(); + this.description = datasetPb.getDescription(); + this.etag = datasetPb.getEtag(); + this.friendlyName = datasetPb.getFriendlyName(); + this.id = datasetPb.getId(); + this.lastModified = datasetPb.getLastModifiedTime(); + this.location = datasetPb.getLocation(); + this.selfLink = datasetPb.getSelfLink(); + } + + @Override public Builder datasetId(DatasetId datasetId) { this.datasetId = checkNotNull(datasetId); return this; } - /** - * Sets the dataset's access control configuration. - * - * @see Access Control - */ + @Override public Builder acl(List acl) { this.acl = acl != null ? ImmutableList.copyOf(acl) : null; return this; } + @Override Builder creationTime(Long creationTime) { this.creationTime = creationTime; return this; } - /** - * Sets the default lifetime of all tables in the dataset, in milliseconds. The minimum value is - * 3600000 milliseconds (one hour). Once this property is set, all newly-created tables in the - * dataset will have an expirationTime property set to the creation time plus the value in this - * property, and changing the value will only affect new tables, not existing ones. When the - * expirationTime for a given table is reached, that table will be deleted automatically. If a - * table's expirationTime is modified or removed before the table expires, or if you provide an - * explicit expirationTime when creating a table, that value takes precedence over the default - * expiration time indicated by this property. This property is experimental and might be - * subject to change or removed. - */ + @Override public Builder defaultTableLifetime(Long defaultTableLifetime) { this.defaultTableLifetime = firstNonNull(defaultTableLifetime, Data.nullOf(Long.class)); return this; } - /** - * Sets a user-friendly description for the dataset. - */ + @Override public Builder description(String description) { this.description = firstNonNull(description, Data.nullOf(String.class)); return this; } + @Override Builder etag(String etag) { this.etag = etag; return this; } - /** - * Sets a user-friendly name for the dataset. - */ + @Override public Builder friendlyName(String friendlyName) { this.friendlyName = firstNonNull(friendlyName, Data.nullOf(String.class)); return this; } + @Override Builder id(String id) { this.id = id; return this; } + @Override Builder lastModified(Long lastModified) { this.lastModified = lastModified; return this; } - /** - * Sets the geographic location where the dataset should reside. This property is experimental - * and might be subject to change or removed. - * - * @see Dataset - * Location - */ + @Override public Builder location(String location) { this.location = firstNonNull(location, Data.nullOf(String.class)); return this; } + @Override Builder selfLink(String selfLink) { this.selfLink = selfLink; return this; } - /** - * Creates a {@code DatasetInfo} object. - */ + @Override public DatasetInfo build() { return new DatasetInfo(this); } } - private DatasetInfo(Builder builder) { + DatasetInfo(BuilderImpl builder) { datasetId = checkNotNull(builder.datasetId); acl = builder.acl; creationTime = builder.creationTime; @@ -300,10 +366,10 @@ public String selfLink() { } /** - * Returns a builder for the {@code DatasetInfo} object. + * Returns a builder for the dataset object. */ public Builder toBuilder() { - return new Builder(this); + return new BuilderImpl(this); } @Override @@ -330,7 +396,31 @@ public int hashCode() { @Override public boolean equals(Object obj) { - return obj instanceof DatasetInfo && Objects.equals(toPb(), ((DatasetInfo) obj).toPb()); + return obj != null + && obj.getClass().equals(DatasetInfo.class) + && Objects.equals(toPb(), ((DatasetInfo) obj).toPb()); + } + + DatasetInfo setProjectId(String projectId) { + Builder builder = toBuilder(); + builder.datasetId(datasetId().setProjectId(projectId)); + if (acl() != null) { + List acls = Lists.newArrayListWithCapacity(acl().size()); + for (Acl acl : acl()) { + if (acl.entity().type() == Acl.Entity.Type.VIEW) { + Dataset.Access accessPb = acl.toPb(); + TableReference viewReferencePb = accessPb.getView(); + if (viewReferencePb.getProjectId() == null) { + viewReferencePb.setProjectId(projectId); + } + acls.add(Acl.of(new Acl.View(TableId.fromPb(viewReferencePb)))); + } else { + acls.add(acl); + } + } + builder.acl(acls); + } + return builder.build(); } Dataset toPb() { @@ -357,65 +447,27 @@ public Dataset.Access apply(Acl acl) { } /** - * Returns a builder for the DatasetInfo object given it's user-defined id. + * Returns a builder for a {@code DatasetInfo} object given it's identity. */ - public static Builder builder(String datasetId) { - return new Builder().datasetId(DatasetId.of(datasetId)); + public static Builder builder(DatasetId datasetId) { + return new BuilderImpl().datasetId(datasetId); } /** - * Returns a builder for the DatasetInfo object given it's project and user-defined id. + * Returns a builder for a {@code DatasetInfo} object given it's user-defined id. */ - public static Builder builder(String projectId, String datasetId) { - return new Builder().datasetId(DatasetId.of(projectId, datasetId)); + public static Builder builder(String datasetId) { + return builder(DatasetId.of(datasetId)); } /** - * Returns a builder for the DatasetInfo object given it's identity. + * Returns a builder for the DatasetInfo object given it's user-defined project and dataset ids. */ - public static Builder builder(DatasetId datasetId) { - return new Builder().datasetId(datasetId); + public static Builder builder(String projectId, String datasetId) { + return builder(DatasetId.of(projectId, datasetId)); } static DatasetInfo fromPb(Dataset datasetPb) { - Builder builder = builder(datasetPb.getDatasetReference().getProjectId(), - datasetPb.getDatasetReference().getDatasetId()); - if (datasetPb.getAccess() != null) { - builder.acl(Lists.transform(datasetPb.getAccess(), - new Function() { - @Override - public Acl apply(Dataset.Access accessPb) { - return Acl.fromPb(accessPb); - } - })); - } - if (datasetPb.getCreationTime() != null) { - builder.creationTime(datasetPb.getCreationTime()); - } - if (datasetPb.getDefaultTableExpirationMs() != null) { - builder.defaultTableLifetime(datasetPb.getDefaultTableExpirationMs()); - } - if (datasetPb.getDescription() != null) { - builder.description(datasetPb.getDescription()); - } - if (datasetPb.getEtag() != null) { - builder.etag(datasetPb.getEtag()); - } - if (datasetPb.getFriendlyName() != null) { - builder.friendlyName(datasetPb.getFriendlyName()); - } - if (datasetPb.getId() != null) { - builder.id(datasetPb.getId()); - } - if (datasetPb.getLastModifiedTime() != null) { - builder.lastModified(datasetPb.getLastModifiedTime()); - } - if (datasetPb.getLocation() != null) { - builder.location(datasetPb.getLocation()); - } - if (datasetPb.getSelfLink() != null) { - builder.selfLink(datasetPb.getSelfLink()); - } - return builder.build(); + return new BuilderImpl(datasetPb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalDataConfiguration.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableDefinition.java similarity index 71% rename from gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalDataConfiguration.java rename to gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableDefinition.java index 4344aeba186b..5f396d948f5a 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalDataConfiguration.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableDefinition.java @@ -1,5 +1,5 @@ /* - * Copyright 2015 Google Inc. All Rights Reserved. + * Copyright 2016 Google Inc. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -18,66 +18,91 @@ import static com.google.common.base.Preconditions.checkNotNull; +import com.google.api.services.bigquery.model.ExternalDataConfiguration; +import com.google.api.services.bigquery.model.Table; import com.google.common.base.Function; -import com.google.common.base.MoreObjects; +import com.google.common.base.MoreObjects.ToStringHelper; import com.google.common.collect.ImmutableList; -import java.io.Serializable; import java.util.List; import java.util.Objects; /** - * Google BigQuery configuration for tables backed by external data. Objects of this class describe - * the data format, location, and other properties of a table stored outside of BigQuery. - * By defining these properties, the data source can then be queried as if it were a standard - * BigQuery table. Support for external tables is experimental and might be subject to changes or - * removed. + * Google BigQuery external table definition. BigQuery's external tables are tables whose data + * reside outside of BigQuery but can be queried as normal BigQuery tables. External tables are + * experimental and might be subject to change or removed. * * @see Federated Data Sources * */ -public class ExternalDataConfiguration implements Serializable { +public class ExternalTableDefinition extends TableDefinition { - static final Function FROM_PB_FUNCTION = - new Function() { + static final Function + FROM_EXTERNAL_DATA_FUNCTION = + new Function() { @Override - public ExternalDataConfiguration apply( - com.google.api.services.bigquery.model.ExternalDataConfiguration configurationPb) { - return ExternalDataConfiguration.fromPb(configurationPb); + public ExternalTableDefinition apply(ExternalDataConfiguration pb) { + return ExternalTableDefinition.fromExternalDataConfiguration(pb); } }; - static final Function TO_PB_FUNCTION = - new Function() { + static final Function + TO_EXTERNAL_DATA_FUNCTION = + new Function() { @Override - public com.google.api.services.bigquery.model.ExternalDataConfiguration apply( - ExternalDataConfiguration configuration) { - return configuration.toPb(); + public ExternalDataConfiguration apply(ExternalTableDefinition tableInfo) { + return tableInfo.toExternalDataConfigurationPb(); } }; - private static final long serialVersionUID = -8004288831035566549L; + private static final long serialVersionUID = -5951580238459622025L; private final List sourceUris; - private final Schema schema; private final FormatOptions formatOptions; private final Integer maxBadRecords; private final Boolean ignoreUnknownValues; private final String compression; - public static final class Builder { + public static final class Builder + extends TableDefinition.Builder { private List sourceUris; - private Schema schema; private FormatOptions formatOptions; private Integer maxBadRecords; private Boolean ignoreUnknownValues; private String compression; - private Builder() {} + private Builder() { + super(Type.EXTERNAL); + } + + private Builder(ExternalTableDefinition tableDefinition) { + super(tableDefinition); + this.sourceUris = tableDefinition.sourceUris; + this.formatOptions = tableDefinition.formatOptions; + this.maxBadRecords = tableDefinition.maxBadRecords; + this.ignoreUnknownValues = tableDefinition.ignoreUnknownValues; + this.compression = tableDefinition.compression; + } + + private Builder(Table tablePb) { + super(tablePb); + com.google.api.services.bigquery.model.ExternalDataConfiguration externalDataConfiguration = + tablePb.getExternalDataConfiguration(); + if (externalDataConfiguration != null) { + if (externalDataConfiguration.getSourceUris() != null) { + this.sourceUris = ImmutableList.copyOf(externalDataConfiguration.getSourceUris()); + } + if (externalDataConfiguration.getSourceFormat() != null) { + this.formatOptions = FormatOptions.of(externalDataConfiguration.getSourceFormat()); + } + this.compression = externalDataConfiguration.getCompression(); + this.ignoreUnknownValues = externalDataConfiguration.getIgnoreUnknownValues(); + if (externalDataConfiguration.getCsvOptions() != null) { + this.formatOptions = CsvOptions.fromPb(externalDataConfiguration.getCsvOptions()); + } + this.maxBadRecords = externalDataConfiguration.getMaxBadRecords(); + } + } /** * Sets the fully-qualified URIs that point to your data in Google Cloud Storage (e.g. @@ -92,14 +117,6 @@ public Builder sourceUris(List sourceUris) { return this; } - /** - * Sets the schema for the external data. - */ - public Builder schema(Schema schema) { - this.schema = checkNotNull(schema); - return this; - } - /** * Sets the source format, and possibly some parsing options, of the external data. Supported * formats are {@code CSV} and {@code NEWLINE_DELIMITED_JSON}. @@ -149,18 +166,19 @@ public Builder compression(String compression) { } /** - * Creates an {@code ExternalDataConfiguration} object. + * Creates an {@code ExternalTableDefinition} object. */ - public ExternalDataConfiguration build() { - return new ExternalDataConfiguration(this); + @Override + public ExternalTableDefinition build() { + return new ExternalTableDefinition(this); } } - ExternalDataConfiguration(Builder builder) { + private ExternalTableDefinition(Builder builder) { + super(builder); this.compression = builder.compression; this.ignoreUnknownValues = builder.ignoreUnknownValues; this.maxBadRecords = builder.maxBadRecords; - this.schema = builder.schema; this.formatOptions = builder.formatOptions; this.sourceUris = builder.sourceUris; } @@ -197,13 +215,6 @@ public Integer maxBadRecords() { return maxBadRecords; } - /** - * Returns the schema for the external data. - */ - public Schema schema() { - return schema; - } - /** * Returns the fully-qualified URIs that point to your data in Google Cloud Storage. Each URI can * contain one '*' wildcard character that must come after the bucket's name. Size limits @@ -226,43 +237,42 @@ public F formatOptions() { } /** - * Returns a builder for the {@code ExternalDataConfiguration} object. + * Returns a builder for the {@code ExternalTableDefinition} object. */ + @Override public Builder toBuilder() { - return new Builder() - .compression(compression) - .ignoreUnknownValues(ignoreUnknownValues) - .maxBadRecords(maxBadRecords) - .schema(schema) - .formatOptions(formatOptions) - .sourceUris(sourceUris); + return new Builder(this); } @Override - public String toString() { - return MoreObjects.toStringHelper(this) + ToStringHelper toStringHelper() { + return super.toStringHelper() .add("sourceUris", sourceUris) .add("formatOptions", formatOptions) - .add("schema", schema) .add("compression", compression) .add("ignoreUnknownValues", ignoreUnknownValues) - .add("maxBadRecords", maxBadRecords) - .toString(); + .add("maxBadRecords", maxBadRecords); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof ExternalTableDefinition && baseEquals((ExternalTableDefinition) obj); } @Override public int hashCode() { - return Objects.hash(compression, ignoreUnknownValues, maxBadRecords, schema, formatOptions, - sourceUris); + return Objects.hash(baseHashCode(), compression, ignoreUnknownValues, maxBadRecords, + formatOptions, sourceUris); } @Override - public boolean equals(Object obj) { - return obj instanceof ExternalDataConfiguration - && Objects.equals(toPb(), ((ExternalDataConfiguration) obj).toPb()); + com.google.api.services.bigquery.model.Table toPb() { + Table tablePb = super.toPb(); + tablePb.setExternalDataConfiguration(toExternalDataConfigurationPb()); + return tablePb; } - com.google.api.services.bigquery.model.ExternalDataConfiguration toPb() { + com.google.api.services.bigquery.model.ExternalDataConfiguration toExternalDataConfigurationPb() { com.google.api.services.bigquery.model.ExternalDataConfiguration externalConfigurationPb = new com.google.api.services.bigquery.model.ExternalDataConfiguration(); if (compression != null) { @@ -274,8 +284,8 @@ com.google.api.services.bigquery.model.ExternalDataConfiguration toPb() { if (maxBadRecords != null) { externalConfigurationPb.setMaxBadRecords(maxBadRecords); } - if (schema != null) { - externalConfigurationPb.setSchema(schema.toPb()); + if (schema() != null) { + externalConfigurationPb.setSchema(schema().toPb()); } if (formatOptions != null) { externalConfigurationPb.setSourceFormat(formatOptions.type()); @@ -290,7 +300,7 @@ com.google.api.services.bigquery.model.ExternalDataConfiguration toPb() { } /** - * Creates a builder for an ExternalDataConfiguration object. + * Creates a builder for an ExternalTableDefinition object. * * @param sourceUris the fully-qualified URIs that point to your data in Google Cloud Storage. * Each URI can contain one '*' wildcard character that must come after the bucket's name. @@ -298,7 +308,7 @@ com.google.api.services.bigquery.model.ExternalDataConfiguration toPb() { * of 10 GB maximum size across all URIs. * @param schema the schema for the external data * @param format the source format of the external data - * @return a builder for an ExternalDataConfiguration object given source URIs, schema and format + * @return a builder for an ExternalTableDefinition object given source URIs, schema and format * * @see Quota * @see @@ -309,28 +319,25 @@ public static Builder builder(List sourceUris, Schema schema, FormatOpti } /** - * Creates a builder for an ExternalDataConfiguration object. + * Creates a builder for an ExternalTableDefinition object. * * @param sourceUri a fully-qualified URI that points to your data in Google Cloud Storage. The * URI can contain one '*' wildcard character that must come after the bucket's name. Size * limits related to load jobs apply to external data sources. * @param schema the schema for the external data * @param format the source format of the external data - * @return a builder for an ExternalDataConfiguration object given source URI, schema and format + * @return a builder for an ExternalTableDefinition object given source URI, schema and format * * @see Quota * @see * Source Format */ public static Builder builder(String sourceUri, Schema schema, FormatOptions format) { - return new Builder() - .sourceUris(ImmutableList.of(sourceUri)) - .schema(schema) - .formatOptions(format); + return builder(ImmutableList.of(sourceUri), schema, format); } /** - * Creates an ExternalDataConfiguration object. + * Creates an ExternalTableDefinition object. * * @param sourceUris the fully-qualified URIs that point to your data in Google Cloud Storage. * Each URI can contain one '*' wildcard character that must come after the bucket's name. @@ -338,38 +345,42 @@ public static Builder builder(String sourceUri, Schema schema, FormatOptions for * of 10 GB maximum size across all URIs. * @param schema the schema for the external data * @param format the source format of the external data - * @return an ExternalDataConfiguration object given source URIs, schema and format + * @return an ExternalTableDefinition object given source URIs, schema and format * * @see Quota * @see * Source Format */ - public static ExternalDataConfiguration of(List sourceUris, Schema schema, + public static ExternalTableDefinition of(List sourceUris, Schema schema, FormatOptions format) { return builder(sourceUris, schema, format).build(); } /** - * Creates an ExternalDataConfiguration object. + * Creates an ExternalTableDefinition object. * * @param sourceUri a fully-qualified URI that points to your data in Google Cloud Storage. The * URI can contain one '*' wildcard character that must come after the bucket's name. Size * limits related to load jobs apply to external data sources. * @param schema the schema for the external data * @param format the source format of the external data - * @return an ExternalDataConfiguration object given source URIs, schema and format + * @return an ExternalTableDefinition object given source URIs, schema and format * * @see Quota * @see * Source Format */ - public static ExternalDataConfiguration of(String sourceUri, Schema schema, - FormatOptions format) { + public static ExternalTableDefinition of(String sourceUri, Schema schema, FormatOptions format) { return builder(sourceUri, schema, format).build(); } - static ExternalDataConfiguration fromPb( - com.google.api.services.bigquery.model.ExternalDataConfiguration externalDataConfiguration) { + @SuppressWarnings("unchecked") + static ExternalTableDefinition fromPb(Table tablePb) { + return new Builder(tablePb).build(); + } + + static ExternalTableDefinition fromExternalDataConfiguration( + ExternalDataConfiguration externalDataConfiguration) { Builder builder = new Builder(); if (externalDataConfiguration.getSourceUris() != null) { builder.sourceUris(externalDataConfiguration.getSourceUris()); diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableInfo.java deleted file mode 100644 index 21ccb3fc1642..000000000000 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableInfo.java +++ /dev/null @@ -1,150 +0,0 @@ -/* - * Copyright 2015 Google Inc. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.google.gcloud.bigquery; - -import static com.google.common.base.Preconditions.checkNotNull; - -import com.google.api.services.bigquery.model.Table; -import com.google.common.base.MoreObjects.ToStringHelper; - -import java.util.Objects; - -/** - * Google BigQuery External Table information. BigQuery's external tables are tables whose data - * reside outside of BigQuery but can be queried as normal BigQuery tables. External tables are - * experimental and might be subject to change or removed. - * - * @see Federated Data Sources - * - */ -public class ExternalTableInfo extends BaseTableInfo { - - private static final long serialVersionUID = -5893406738246214865L; - - private final ExternalDataConfiguration configuration; - - public static final class Builder extends BaseTableInfo.Builder { - - private ExternalDataConfiguration configuration; - - private Builder() {} - - private Builder(ExternalTableInfo tableInfo) { - super(tableInfo); - this.configuration = tableInfo.configuration; - } - - protected Builder(Table tablePb) { - super(tablePb); - if (tablePb.getExternalDataConfiguration() != null) { - this.configuration = - ExternalDataConfiguration.fromPb(tablePb.getExternalDataConfiguration()); - } - } - - /** - * Sets the data format, location and other properties of a table stored outside of BigQuery. - * - * @see Federated Data - * Sources - */ - public Builder configuration(ExternalDataConfiguration configuration) { - this.configuration = checkNotNull(configuration); - return self(); - } - - /** - * Creates a {@code ExternalTableInfo} object. - */ - @Override - public ExternalTableInfo build() { - return new ExternalTableInfo(this); - } - } - - private ExternalTableInfo(Builder builder) { - super(builder); - this.configuration = builder.configuration; - } - - /** - * Returns the data format, location and other properties of a table stored outside of BigQuery. - * This property is experimental and might be subject to change or removed. - * - * @see Federated Data Sources - * - */ - public ExternalDataConfiguration configuration() { - return configuration; - } - - /** - * Returns a builder for the {@code ExternalTableInfo} object. - */ - @Override - public Builder toBuilder() { - return new Builder(this); - } - - @Override - ToStringHelper toStringHelper() { - return super.toStringHelper().add("configuration", configuration); - } - - @Override - public boolean equals(Object obj) { - return obj instanceof ExternalTableInfo - && Objects.equals(toPb(), ((ExternalTableInfo) obj).toPb()); - } - - @Override - public int hashCode() { - return Objects.hash(super.hashCode(), configuration); - } - - @Override - Table toPb() { - Table tablePb = super.toPb(); - tablePb.setExternalDataConfiguration(configuration.toPb()); - return tablePb; - } - - /** - * Returns a builder for a BigQuery External Table. - * - * @param tableId table id - * @param configuration data format, location and other properties of an External Table - */ - public static Builder builder(TableId tableId, ExternalDataConfiguration configuration) { - return new Builder().tableId(tableId).type(Type.EXTERNAL).configuration(configuration); - } - - /** - * Returns a BigQuery External Table. - * - * @param table table id - * @param configuration data format, location and other properties of an External Table - */ - public static ExternalTableInfo of(TableId table, ExternalDataConfiguration configuration) { - return builder(table, configuration).build(); - } - - @SuppressWarnings("unchecked") - static ExternalTableInfo fromPb(Table tablePb) { - return new Builder(tablePb).build(); - } -} diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobConfiguration.java similarity index 62% rename from gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobInfo.java rename to gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobConfiguration.java index effdab84f265..7c5a2698b159 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobConfiguration.java @@ -1,5 +1,5 @@ /* - * Copyright 2015 Google Inc. All Rights Reserved. + * Copyright 2016 Google Inc. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -18,23 +18,21 @@ import static com.google.common.base.Preconditions.checkNotNull; -import com.google.api.services.bigquery.model.Job; -import com.google.api.services.bigquery.model.JobConfiguration; import com.google.api.services.bigquery.model.JobConfigurationExtract; import com.google.common.base.MoreObjects.ToStringHelper; import com.google.common.collect.ImmutableList; -import com.google.gcloud.bigquery.JobStatistics.ExtractStatistics; import java.util.List; import java.util.Objects; /** - * Google BigQuery Extract Jobs. An Extract Job exports a BigQuery table to Google Cloud Storage. - * The extract destination provided as URIs that point to objects in Google Cloud Storage. + * Google BigQuery extract job configuration. An extract job exports a BigQuery table to Google + * Cloud Storage. The extract destination provided as URIs that point to objects in Google Cloud + * Storage. Extract job configurations have {@link JobConfiguration.Type#EXTRACT} type. */ -public class ExtractJobInfo extends JobInfo { +public final class ExtractJobConfiguration extends JobConfiguration { - private static final long serialVersionUID = -9126951217071361576L; + private static final long serialVersionUID = 4147749733166593761L; private final TableId sourceTable; private final List destinationUris; @@ -43,8 +41,8 @@ public class ExtractJobInfo extends JobInfo { private final String format; private final String compression; - public static final class Builder extends JobInfo.Builder { + public static final class Builder + extends JobConfiguration.Builder { private TableId sourceTable; private List destinationUris; @@ -53,10 +51,12 @@ public static final class Builder extends JobInfo.Builder destinationUris) { this.destinationUris = destinationUris != null ? ImmutableList.copyOf(destinationUris) : null; - return self(); + return this; } /** @@ -98,7 +98,7 @@ public Builder destinationUris(List destinationUris) { */ public Builder printHeader(Boolean printHeader) { this.printHeader = printHeader; - return self(); + return this; } /** @@ -106,7 +106,7 @@ public Builder printHeader(Boolean printHeader) { */ public Builder fieldDelimiter(String fieldDelimiter) { this.fieldDelimiter = fieldDelimiter; - return self(); + return this; } /** @@ -117,7 +117,7 @@ public Builder fieldDelimiter(String fieldDelimiter) { */ public Builder format(String format) { this.format = format; - return self(); + return this; } /** @@ -129,16 +129,15 @@ public Builder format(String format) { */ public Builder compression(String compression) { this.compression = compression; - return self(); + return this; } - @Override - public ExtractJobInfo build() { - return new ExtractJobInfo(this); + public ExtractJobConfiguration build() { + return new ExtractJobConfiguration(this); } } - private ExtractJobInfo(Builder builder) { + private ExtractJobConfiguration(Builder builder) { super(builder); this.sourceTable = checkNotNull(builder.sourceTable); this.destinationUris = checkNotNull(builder.destinationUris); @@ -212,17 +211,22 @@ ToStringHelper toStringHelper() { @Override public boolean equals(Object obj) { - return obj instanceof ExtractJobInfo && Objects.equals(toPb(), ((ExtractJobInfo) obj).toPb()); + return obj instanceof ExtractJobConfiguration && baseEquals((ExtractJobConfiguration) obj); } @Override public int hashCode() { - return Objects.hash(super.hashCode(), sourceTable, destinationUris, printHeader, fieldDelimiter, + return Objects.hash(baseHashCode(), sourceTable, destinationUris, printHeader, fieldDelimiter, format, compression); } @Override - Job toPb() { + ExtractJobConfiguration setProjectId(String projectId) { + return toBuilder().sourceTable(sourceTable().setProjectId(projectId)).build(); + } + + @Override + com.google.api.services.bigquery.model.JobConfiguration toPb() { JobConfigurationExtract extractConfigurationPb = new JobConfigurationExtract(); extractConfigurationPb.setDestinationUris(destinationUris); extractConfigurationPb.setSourceTable(sourceTable.toPb()); @@ -230,92 +234,61 @@ Job toPb() { extractConfigurationPb.setFieldDelimiter(fieldDelimiter); extractConfigurationPb.setDestinationFormat(format); extractConfigurationPb.setCompression(compression); - return super.toPb().setConfiguration(new JobConfiguration().setExtract(extractConfigurationPb)); + return new com.google.api.services.bigquery.model.JobConfiguration() + .setExtract(extractConfigurationPb); } /** - * Creates a builder for a BigQuery Extract Job given source table and destination URI. + * Creates a builder for a BigQuery Extract Job configuration given source table and destination + * URI. */ public static Builder builder(TableId sourceTable, String destinationUri) { return builder(sourceTable, ImmutableList.of(checkNotNull(destinationUri))); } /** - * Creates a builder for a BigQuery Extract Job given source table and destination URIs. + * Creates a builder for a BigQuery Extract Job configuration given source table and destination + * URIs. */ public static Builder builder(TableId sourceTable, List destinationUris) { return new Builder().sourceTable(sourceTable).destinationUris(destinationUris); } /** - * Returns a BigQuery Extract Job for the given source table and destination URI. Job's id is - * chosen by the service. + * Returns a BigQuery Extract Job configuration for the given source table and destination URI. */ - public static ExtractJobInfo of(TableId sourceTable, String destinationUri) { + public static ExtractJobConfiguration of(TableId sourceTable, String destinationUri) { return builder(sourceTable, destinationUri).build(); } /** - * Returns a BigQuery Extract Job for the given source table and destination URIs. Job's id is - * chosen by the service. + * Returns a BigQuery Extract Job configuration for the given source table and destination URIs. */ - public static ExtractJobInfo of(TableId sourceTable, List destinationUris) { + public static ExtractJobConfiguration of(TableId sourceTable, List destinationUris) { return builder(sourceTable, destinationUris).build(); } /** - * Returns a BigQuery Extract Job for the given source table, format and destination URI. Job's id - * is chosen by the service. + * Returns a BigQuery Extract Job configuration for the given source table, format and destination + * URI. */ - public static ExtractJobInfo of(TableId sourceTable, String format, String destinationUri) { + public static ExtractJobConfiguration of(TableId sourceTable, String destinationUri, + String format) { return builder(sourceTable, destinationUri).format(format).build(); } /** - * Returns a BigQuery Extract Job for the given source table, format and destination URIs. Job's - * id is chosen by the service. + * Returns a BigQuery Extract Job configuration for the given source table, format and destination + * URIs. */ - public static ExtractJobInfo of(TableId sourceTable, String format, - List destinationUris) { + public static ExtractJobConfiguration of(TableId sourceTable, List destinationUris, + String format) { return builder(sourceTable, destinationUris).format(format).build(); } - /** - * Returns a BigQuery Extract Job for the given source table and destination URI. Job's id is set - * to the provided value. - */ - public static ExtractJobInfo of(JobId jobId, TableId sourceTable, String destinationUri) { - return builder(sourceTable, destinationUri).jobId(jobId).build(); - } - - /** - * Returns a BigQuery Extract Job for the given source table and destination URIs. Job's id is set - * to the provided value. - */ - public static ExtractJobInfo of(JobId jobId, TableId sourceTable, List destinationUris) { - return builder(sourceTable, destinationUris).jobId(jobId).build(); - } - - /** - * Returns a BigQuery Extract Job for the given source table, format and destination URI. Job's id - * is set to the provided value. - */ - public static ExtractJobInfo of(JobId jobId, TableId sourceTable, String format, - String destinationUri) { - return builder(sourceTable, destinationUri).format(format).jobId(jobId).build(); - } - - /** - * Returns a BigQuery Extract Job for the given source table, format and destination URIs. Job's - * id is set to the provided value. - */ - public static ExtractJobInfo of(JobId jobId, TableId sourceTable, String format, - List destinationUris) { - return builder(sourceTable, destinationUris).format(format).jobId(jobId).build(); - } - @SuppressWarnings("unchecked") - static ExtractJobInfo fromPb(Job jobPb) { - return new Builder(jobPb).build(); + static ExtractJobConfiguration fromPb( + com.google.api.services.bigquery.model.JobConfiguration confPb) { + return new Builder(confPb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FieldValue.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FieldValue.java index 24c4b28b7613..8b27c70db782 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FieldValue.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FieldValue.java @@ -20,9 +20,9 @@ import static com.google.common.base.Preconditions.checkState; import com.google.api.client.util.Data; -import com.google.api.client.util.Lists; import com.google.common.base.Function; import com.google.common.base.MoreObjects; +import com.google.common.collect.Lists; import java.io.Serializable; import java.util.List; diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/InsertAllRequest.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/InsertAllRequest.java index bd86f208480f..f0d61583f83f 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/InsertAllRequest.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/InsertAllRequest.java @@ -52,15 +52,15 @@ public class InsertAllRequest implements Serializable { * id used by BigQuery to detect duplicate insertion requests on a best-effort basis. * *

Example usage of creating a row to insert: - *

    {@code
-   *   List repeatedFieldValue = Arrays.asList(1L, 2L);
-   *   Map recordContent = new HashMap();
-   *   recordContent.put("subfieldName1", "value");
-   *   recordContent.put("subfieldName2", repeatedFieldValue);
-   *   Map rowContent = new HashMap();
-   *   rowContent.put("fieldName1", true);
-   *   rowContent.put("fieldName2", recordContent);
-   *   RowToInsert row = new RowToInsert("rowId", rowContent);
+   * 
 {@code
+   * List repeatedFieldValue = Arrays.asList(1L, 2L);
+   * Map recordContent = new HashMap();
+   * recordContent.put("subfieldName1", "value");
+   * recordContent.put("subfieldName2", repeatedFieldValue);
+   * Map rowContent = new HashMap();
+   * rowContent.put("fieldName1", true);
+   * rowContent.put("fieldName2", recordContent);
+   * RowToInsert row = new RowToInsert("rowId", rowContent);
    * }
* * @see @@ -177,16 +177,16 @@ public Builder addRow(RowToInsert rowToInsert) { * Adds a row to be inserted with associated id. * *

Example usage of adding a row with associated id: - *

    {@code
-     *   InsertAllRequest.Builder builder = InsertAllRequest.builder(tableId);
-     *   List repeatedFieldValue = Arrays.asList(1L, 2L);
-     *   Map recordContent = new HashMap();
-     *   recordContent.put("subfieldName1", "value");
-     *   recordContent.put("subfieldName2", repeatedFieldValue);
-     *   Map rowContent = new HashMap();
-     *   rowContent.put("fieldName1", true);
-     *   rowContent.put("fieldName2", recordContent);
-     *   builder.addRow("rowId", rowContent);
+     * 
 {@code
+     * InsertAllRequest.Builder builder = InsertAllRequest.builder(tableId);
+     * List repeatedFieldValue = Arrays.asList(1L, 2L);
+     * Map recordContent = new HashMap();
+     * recordContent.put("subfieldName1", "value");
+     * recordContent.put("subfieldName2", repeatedFieldValue);
+     * Map rowContent = new HashMap();
+     * rowContent.put("fieldName1", true);
+     * rowContent.put("fieldName2", recordContent);
+     * builder.addRow("rowId", rowContent);
      * }
*/ public Builder addRow(String id, Map content) { @@ -198,16 +198,16 @@ public Builder addRow(String id, Map content) { * Adds a row to be inserted without an associated id. * *

Example usage of adding a row without an associated id: - *

    {@code
-     *   InsertAllRequest.Builder builder = InsertAllRequest.builder(tableId);
-     *   List repeatedFieldValue = Arrays.asList(1L, 2L);
-     *   Map recordContent = new HashMap();
-     *   recordContent.put("subfieldName1", "value");
-     *   recordContent.put("subfieldName2", repeatedFieldValue);
-     *   Map rowContent = new HashMap();
-     *   rowContent.put("fieldName1", true);
-     *   rowContent.put("fieldName2", recordContent);
-     *   builder.addRow(rowContent);
+     * 
 {@code
+     * InsertAllRequest.Builder builder = InsertAllRequest.builder(tableId);
+     * List repeatedFieldValue = Arrays.asList(1L, 2L);
+     * Map recordContent = new HashMap();
+     * recordContent.put("subfieldName1", "value");
+     * recordContent.put("subfieldName2", repeatedFieldValue);
+     * Map rowContent = new HashMap();
+     * rowContent.put("fieldName1", true);
+     * rowContent.put("fieldName2", recordContent);
+     * builder.addRow(rowContent);
      * }
*/ public Builder addRow(Map content) { @@ -241,7 +241,7 @@ public Builder ignoreUnknownValues(boolean ignoreUnknownValues) { * is called use: *
 {@code
      * String suffixTableId = ...;
-     * BaseTableInfo suffixTable = bigquery.getTable(DATASET, suffixTableId);
+     * TableInfo suffixTable = bigquery.getTable(DATASET, suffixTableId);
      * while (suffixTable == null) {
      *   Thread.sleep(1000L);
      *   suffixTable = bigquery.getTable(DATASET, suffixTableId);
@@ -307,7 +307,7 @@ public Boolean skipInvalidRows() {
    * called use:
    * 
 {@code
    * String suffixTableId = ...;
-   * BaseTableInfo suffixTable = bigquery.getTable(DATASET, suffixTableId);
+   * TableInfo suffixTable = bigquery.getTable(DATASET, suffixTableId);
    * while (suffixTable == null) {
    *   Thread.sleep(1000L);
    *   suffixTable = bigquery.getTable(DATASET, suffixTableId);
@@ -371,7 +371,7 @@ public static Builder builder(String datasetId, String tableId, RowToInsert... r
    * Returns a builder for an {@code InsertAllRequest} object given the destination table and the
    * rows to insert.
    */
-  public static Builder builder(BaseTableInfo tableInfo, Iterable rows) {
+  public static Builder builder(TableInfo tableInfo, Iterable rows) {
     return builder(tableInfo.tableId(), rows);
   }
 
@@ -379,7 +379,7 @@ public static Builder builder(BaseTableInfo tableInfo, Iterable row
    * Returns a builder for an {@code InsertAllRequest} object given the destination table and the
    * rows to insert.
    */
-  public static Builder builder(BaseTableInfo tableInfo, RowToInsert... rows) {
+  public static Builder builder(TableInfo tableInfo, RowToInsert... rows) {
     return builder(tableInfo.tableId(), rows);
   }
 
@@ -414,14 +414,14 @@ public static InsertAllRequest of(String datasetId, String tableId, RowToInsert.
   /**
    * Returns a {@code InsertAllRequest} object given the destination table and the rows to insert.
    */
-  public static InsertAllRequest of(BaseTableInfo tableInfo, Iterable rows) {
+  public static InsertAllRequest of(TableInfo tableInfo, Iterable rows) {
     return builder(tableInfo.tableId(), rows).build();
   }
 
   /**
    * Returns a {@code InsertAllRequest} object given the destination table and the rows to insert.
    */
-  public static InsertAllRequest of(BaseTableInfo tableInfo, RowToInsert... rows) {
+  public static InsertAllRequest of(TableInfo tableInfo, RowToInsert... rows) {
     return builder(tableInfo.tableId(), rows).build();
   }
 
diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Job.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Job.java
index c0d7ddc29c37..1e63344a600d 100644
--- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Job.java
+++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Job.java
@@ -18,50 +18,102 @@
 
 import static com.google.common.base.Preconditions.checkNotNull;
 
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.util.Objects;
+
 /**
  * A Google BigQuery Job.
  *
  * 

Objects of this class are immutable. To get a {@code Job} object with the most recent - * information use {@link #reload}. + * information use {@link #reload}. {@code Job} adds a layer of service-related functionality over + * {@link JobInfo}. *

*/ -public final class Job { +public final class Job extends JobInfo { - private final BigQuery bigquery; - private final JobInfo info; + private static final long serialVersionUID = -4324100991693024704L; - /** - * Constructs a {@code Job} object for the provided {@code JobInfo}. The BigQuery service - * is used to issue requests. - * - * @param bigquery the BigQuery service used for issuing requests - * @param info jobs's info - */ - public Job(BigQuery bigquery, JobInfo info) { - this.bigquery = checkNotNull(bigquery); - this.info = checkNotNull(info); - } + private final BigQueryOptions options; + private transient BigQuery bigquery; /** - * Creates a {@code Job} object for the provided job's user-defined id. Performs an RPC call to - * get the latest job information. - * - * @param bigquery the BigQuery service used for issuing requests - * @param job job's id, either user-defined or picked by the BigQuery service - * @param options job options - * @return the {@code Job} object or {@code null} if not found - * @throws BigQueryException upon failure + * A builder for {@code Job} objects. */ - public static Job get(BigQuery bigquery, String job, BigQuery.JobOption... options) { - JobInfo info = bigquery.getJob(job, options); - return info != null ? new Job(bigquery, info) : null; + public static final class Builder extends JobInfo.Builder { + + private final BigQuery bigquery; + private final JobInfo.BuilderImpl infoBuilder; + + Builder(BigQuery bigquery, JobConfiguration configuration) { + this.bigquery = bigquery; + this.infoBuilder = new JobInfo.BuilderImpl(); + this.infoBuilder.configuration(configuration); + } + + Builder(Job job) { + this.bigquery = job.bigquery; + this.infoBuilder = new JobInfo.BuilderImpl(job); + } + + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; + } + + @Override + Builder id(String id) { + infoBuilder.id(id); + return this; + } + + @Override + public Builder jobId(JobId jobId) { + infoBuilder.jobId(jobId); + return this; + } + + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + Builder status(JobStatus status) { + infoBuilder.status(status); + return this; + } + + @Override + Builder statistics(JobStatistics statistics) { + infoBuilder.statistics(statistics); + return this; + } + + @Override + Builder userEmail(String userEmail) { + infoBuilder.userEmail(userEmail); + return this; + } + + @Override + public Builder configuration(JobConfiguration configuration) { + infoBuilder.configuration(configuration); + return this; + } + + @Override + public Job build() { + return new Job(bigquery, infoBuilder); + } } - /** - * Returns the job's information. - */ - public JobInfo info() { - return info; + Job(BigQuery bigquery, JobInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.bigquery = checkNotNull(bigquery); + this.options = bigquery.options(); } /** @@ -71,7 +123,7 @@ public JobInfo info() { * @throws BigQueryException upon failure */ public boolean exists() { - return bigquery.getJob(info.jobId(), BigQuery.JobOption.fields()) != null; + return bigquery.getJob(jobId(), BigQuery.JobOption.fields()) != null; } /** @@ -90,8 +142,7 @@ public boolean exists() { * @throws BigQueryException upon failure */ public boolean isDone() { - JobInfo job = bigquery.getJob(info.jobId(), - BigQuery.JobOption.fields(BigQuery.JobField.STATUS)); + Job job = bigquery.getJob(jobId(), BigQuery.JobOption.fields(BigQuery.JobField.STATUS)); return job != null && job.status().state() == JobStatus.State.DONE; } @@ -103,7 +154,7 @@ public boolean isDone() { * @throws BigQueryException upon failure */ public Job reload(BigQuery.JobOption... options) { - return Job.get(bigquery, info.jobId().job(), options); + return bigquery.getJob(jobId().job(), options); } /** @@ -114,7 +165,7 @@ public Job reload(BigQuery.JobOption... options) { * @throws BigQueryException upon failure */ public boolean cancel() { - return bigquery.cancel(info.jobId()); + return bigquery.cancel(jobId()); } /** @@ -123,4 +174,30 @@ public boolean cancel() { public BigQuery bigquery() { return bigquery; } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof Job + && Objects.equals(toPb(), ((Job) obj).toPb()) + && Objects.equals(options, ((Job) obj).options); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); + } + + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.bigquery = options.service(); + } + + static Job fromPb(BigQuery bigquery, com.google.api.services.bigquery.model.Job jobPb) { + return new Job(bigquery, new JobInfo.BuilderImpl(jobPb)); + } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobConfiguration.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobConfiguration.java new file mode 100644 index 000000000000..2244969567ef --- /dev/null +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobConfiguration.java @@ -0,0 +1,145 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.common.base.MoreObjects; +import com.google.common.base.MoreObjects.ToStringHelper; + +import java.io.Serializable; +import java.util.Objects; + +/** + * Base class for a BigQuery job configuration. + */ +public abstract class JobConfiguration implements Serializable { + + private static final long serialVersionUID = -548132177415406526L; + + private final Type type; + + /** + * Type of a BigQuery Job. + */ + enum Type { + /** + * A Copy Job copies an existing table to another new or existing table. Instances of + * {@code JobConfiguration} for this type are implemented by {@link CopyJobConfiguration}. + */ + COPY, + /** + * An Extract Job exports a BigQuery table to Google Cloud Storage. Instances of + * {@code JobConfiguration} for this type are implemented by {@link ExtractJobConfiguration}. + */ + EXTRACT, + /** + * A Load Job loads data from one of several formats into a table. Instances of + * {@code JobConfiguration} for this type are implemented by {@link LoadJobConfiguration}. + */ + LOAD, + /** + * A Query Job runs a query against BigQuery data. Instances of + * {@code JobConfiguration} for this type are implemented by {@link QueryJobConfiguration}. + */ + QUERY + } + + /** + * Base builder for job configurations. + * + * @param the job configuration type + * @param the job configuration builder + */ + public abstract static class Builder> { + + private Type type; + + Builder(Type type) { + this.type = checkNotNull(type); + } + + @SuppressWarnings("unchecked") + B self() { + return (B) this; + } + + B type(Type type) { + this.type = checkNotNull(type); + return self(); + } + + /** + * Creates an object. + */ + public abstract T build(); + } + + JobConfiguration(Builder builder) { + this.type = builder.type; + } + + /** + * Returns the type of the job configuration. + */ + public Type type() { + return type; + } + + /** + * Returns a builder for the object. + */ + public abstract Builder toBuilder(); + + ToStringHelper toStringHelper() { + return MoreObjects.toStringHelper(this).add("type", type); + } + + @Override + public String toString() { + return toStringHelper().toString(); + } + + final int baseHashCode() { + return Objects.hash(type); + } + + final boolean baseEquals(JobConfiguration jobConfiguration) { + return Objects.equals(toPb(), jobConfiguration.toPb()); + } + + abstract JobConfiguration setProjectId(String projectId); + + abstract com.google.api.services.bigquery.model.JobConfiguration toPb(); + + @SuppressWarnings("unchecked") + static T fromPb( + com.google.api.services.bigquery.model.JobConfiguration configurationPb) { + if (configurationPb.getCopy() != null) { + return (T) CopyJobConfiguration.fromPb(configurationPb); + } else if (configurationPb.getExtract() != null) { + return (T) ExtractJobConfiguration.fromPb(configurationPb); + } else if (configurationPb.getLoad() != null) { + return (T) LoadJobConfiguration.fromPb(configurationPb); + } else if (configurationPb.getQuery() != null) { + return (T) QueryJobConfiguration.fromPb(configurationPb); + } else { + // never reached + throw new IllegalArgumentException("Job configuration is not supported"); + } + } +} diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobInfo.java index de33c483393d..1adf7fabafc1 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobInfo.java @@ -19,23 +19,20 @@ import com.google.api.services.bigquery.model.Job; import com.google.common.base.Function; import com.google.common.base.MoreObjects; -import com.google.common.base.MoreObjects.ToStringHelper; import java.io.Serializable; import java.util.Objects; /** - * Base class for Google BigQuery Job information. Jobs are objects that manage asynchronous tasks - * such as running queries, loading data, and exporting data. Use {@link CopyJobInfo} for a job that - * copies an existing table. Use {@link ExtractJobInfo} for a job that exports a table to Google - * Cloud Storage. Use {@link LoadJobInfo} for a job that loads data from Google Cloud Storage into - * a table. Use {@link QueryJobInfo} for a job that runs a query. + * Google BigQuery Job information. Jobs are objects that manage asynchronous tasks such as running + * queries, loading data, and exporting data. Use {@link CopyJobConfiguration} for a job that + * copies an existing table. Use {@link ExtractJobConfiguration} for a job that exports a table to + * Google Cloud Storage. Use {@link LoadJobConfiguration} for a job that loads data from Google + * Cloud Storage into a table. Use {@link QueryJobConfiguration} for a job that runs a query. * * @see
Jobs - * - * @param the statistics type */ -public abstract class JobInfo implements Serializable { +public class JobInfo implements Serializable { static final Function FROM_PB_FUNCTION = new Function() { @@ -44,7 +41,17 @@ public JobInfo apply(Job pb) { return JobInfo.fromPb(pb); } }; - private static final long serialVersionUID = -7086529810736715842L; + + private static final long serialVersionUID = -3272941007234620265L; + + private final String etag; + private final String id; + private final JobId jobId; + private final String selfLink; + private final JobStatus status; + private final JobStatistics statistics; + private final String userEmail; + private final JobConfiguration configuration; /** * Specifies whether the job is allowed to create new tables. @@ -81,35 +88,57 @@ public enum WriteDisposition { WRITE_EMPTY } - private final String etag; - private final String id; - private final JobId jobId; - private final String selfLink; - private final JobStatus status; - private final S statistics; - private final String userEmail; - /** - * Base builder for jobs. - * - * @param the job type - * @param the job statistics type - * @param the job builder + * A builder for {@code JobInfo} objects. */ - public abstract static class Builder> { + public abstract static class Builder { + + abstract Builder etag(String etag); + + abstract Builder id(String id); + + /** + * Sets the job identity. + */ + public abstract Builder jobId(JobId jobId); + + abstract Builder selfLink(String selfLink); + + abstract Builder status(JobStatus status); + + abstract Builder statistics(JobStatistics statistics); + + abstract Builder userEmail(String userEmail); + + /** + * Sets a configuration for the {@code JobInfo} object. Use {@link CopyJobConfiguration} for a + * job that copies an existing table. Use {@link ExtractJobConfiguration} for a job that exports + * a table to Google Cloud Storage. Use {@link LoadJobConfiguration} for a job that loads data + * from Google Cloud Storage into a table. Use {@link QueryJobConfiguration} for a job that runs + * a query. + */ + public abstract Builder configuration(JobConfiguration configuration); + + /** + * Creates a {@code JobInfo} object. + */ + public abstract JobInfo build(); + } + + static final class BuilderImpl extends Builder { private String etag; private String id; private JobId jobId; private String selfLink; private JobStatus status; - private S statistics; + private JobStatistics statistics; private String userEmail; + private JobConfiguration configuration; - protected Builder() {} + BuilderImpl() {} - protected Builder(JobInfo jobInfo) { + BuilderImpl(JobInfo jobInfo) { this.etag = jobInfo.etag; this.id = jobInfo.id; this.jobId = jobInfo.jobId; @@ -117,10 +146,10 @@ protected Builder(JobInfo jobInfo) { this.status = jobInfo.status; this.statistics = jobInfo.statistics; this.userEmail = jobInfo.userEmail; - + this.configuration = jobInfo.configuration; } - protected Builder(Job jobPb) { + BuilderImpl(Job jobPb) { this.etag = jobPb.getEtag(); this.id = jobPb.getId(); if (jobPb.getJobReference() != null) { @@ -134,55 +163,64 @@ protected Builder(Job jobPb) { this.statistics = JobStatistics.fromPb(jobPb.getStatistics()); } this.userEmail = jobPb.getUserEmail(); + this.configuration = JobConfiguration.fromPb(jobPb.getConfiguration()); } - @SuppressWarnings("unchecked") - protected B self() { - return (B) this; - } - - B etag(String etag) { + @Override + Builder etag(String etag) { this.etag = etag; - return self(); + return this; } - B id(String id) { + @Override + Builder id(String id) { this.id = id; - return self(); + return this; } - /** - * Sets the job identity. - */ - public B jobId(JobId jobId) { + @Override + public Builder jobId(JobId jobId) { this.jobId = jobId; - return self(); + return this; } - B selfLink(String selfLink) { + @Override + Builder selfLink(String selfLink) { this.selfLink = selfLink; - return self(); + return this; } - B status(JobStatus status) { + @Override + Builder status(JobStatus status) { this.status = status; - return self(); + return this; } - B statistics(S statistics) { + @Override + Builder statistics(JobStatistics statistics) { this.statistics = statistics; - return self(); + return this; } - B userEmail(String userEmail) { + @Override + Builder userEmail(String userEmail) { this.userEmail = userEmail; - return self(); + return this; } - public abstract T build(); + @Override + public Builder configuration(JobConfiguration configuration) { + this.configuration = configuration; + return this; + } + + @Override + public JobInfo build() { + return new JobInfo(this); + } } - protected JobInfo(Builder builder) { + JobInfo(BuilderImpl builder) { this.jobId = builder.jobId; this.etag = builder.etag; this.id = builder.id; @@ -190,6 +228,7 @@ protected JobInfo(Builder builder) { this.status = builder.status; this.statistics = builder.statistics; this.userEmail = builder.userEmail; + this.configuration = builder.configuration; } /** @@ -232,8 +271,9 @@ public JobStatus status() { /** * Returns information about the job, including starting time and ending time of the job. */ - public S statistics() { - return statistics; + @SuppressWarnings("unchecked") + public S statistics() { + return (S) statistics; } /** @@ -244,11 +284,22 @@ public String userEmail() { } /** - * Returns a builder for the job. + * Returns the job's configuration. + */ + @SuppressWarnings("unchecked") + public C configuration() { + return (C) configuration; + } + + /** + * Returns a builder for the job object. */ - public abstract Builder toBuilder(); + public Builder toBuilder() { + return new BuilderImpl(this); + } - ToStringHelper toStringHelper() { + @Override + public String toString() { return MoreObjects.toStringHelper(this) .add("job", jobId) .add("status", status) @@ -256,12 +307,9 @@ ToStringHelper toStringHelper() { .add("userEmail", userEmail) .add("etag", etag) .add("id", id) - .add("selfLink", selfLink); - } - - @Override - public String toString() { - return toStringHelper().toString(); + .add("selfLink", selfLink) + .add("configuration", configuration) + .toString(); } @Override @@ -271,7 +319,13 @@ public int hashCode() { @Override public boolean equals(Object obj) { - return obj instanceof JobInfo && Objects.equals(toPb(), ((JobInfo) obj).toPb()); + return obj != null + && obj.getClass().equals(JobInfo.class) + && Objects.equals(toPb(), ((JobInfo) obj).toPb()); + } + + JobInfo setProjectId(String projectId) { + return toBuilder().configuration(configuration.setProjectId(projectId)).build(); } Job toPb() { @@ -289,22 +343,44 @@ Job toPb() { if (statistics != null) { jobPb.setStatistics(statistics.toPb()); } + jobPb.setConfiguration(configuration.toPb()); return jobPb; } - @SuppressWarnings("unchecked") - static T fromPb(Job jobPb) { - if (jobPb.getConfiguration().getLoad() != null) { - return (T) LoadJobInfo.fromPb(jobPb); - } else if (jobPb.getConfiguration().getCopy() != null) { - return (T) CopyJobInfo.fromPb(jobPb); - } else if (jobPb.getConfiguration().getExtract() != null) { - return (T) ExtractJobInfo.fromPb(jobPb); - } else if (jobPb.getConfiguration().getQuery() != null) { - return (T) QueryJobInfo.fromPb(jobPb); - } else { - // never reached - throw new IllegalArgumentException("Job configuration is not supported"); - } + /** + * Returns a builder for a {@code JobInfo} object given the job configuration. Use + * {@link CopyJobConfiguration} for a job that copies an existing table. Use + * {@link ExtractJobConfiguration} for a job that exports a table to Google Cloud Storage. Use + * {@link LoadJobConfiguration} for a job that loads data from Google Cloud Storage into a table. + * Use {@link QueryJobConfiguration} for a job that runs a query. + */ + public static Builder builder(JobConfiguration configuration) { + return new BuilderImpl().configuration(configuration); + } + + /** + * Returns a {@code JobInfo} object given the job configuration. Use {@link CopyJobConfiguration} + * for a job that copies an existing table. Use {@link ExtractJobConfiguration} for a job that + * exports a table to Google Cloud Storage. Use {@link LoadJobConfiguration} for a job that loads + * data from Google Cloud Storage into a table. Use {@link QueryJobConfiguration} for a job that + * runs a query. + */ + public static JobInfo of(JobConfiguration configuration) { + return builder(configuration).build(); + } + + /** + * Returns a builder for a {@code JobInfo} object given the job identity and configuration. Use + * {@link CopyJobConfiguration} for a job that copies an existing table. Use + * {@link ExtractJobConfiguration} for a job that exports a table to Google Cloud Storage. Use + * {@link LoadJobConfiguration} for a job that loads data from Google Cloud Storage into a table. + * Use {@link QueryJobConfiguration} for a job that runs a query. + */ + public static JobInfo of(JobId jobId, JobConfiguration configuration) { + return builder(configuration).jobId(jobId).build(); + } + + static JobInfo fromPb(Job jobPb) { + return new BuilderImpl(jobPb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobStatistics.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobStatistics.java index b2d50882aabb..34e4917921ba 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobStatistics.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobStatistics.java @@ -61,7 +61,7 @@ private ExtractStatistics(Builder builder) { /** * Returns the number of files per destination URI or URI pattern specified in the extract job. * These values will be in the same order as the URIs specified by - * {@link ExtractJobInfo#destinationUris()}. + * {@link ExtractJobConfiguration#destinationUris()}. */ public List destinationUriFileCounts() { return destinationUriFileCounts; diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadConfiguration.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadConfiguration.java index 18cb8ae6bedb..223a25a478e0 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadConfiguration.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadConfiguration.java @@ -16,103 +16,26 @@ package com.google.gcloud.bigquery; -import static com.google.common.base.Preconditions.checkNotNull; - -import com.google.api.services.bigquery.model.JobConfigurationLoad; -import com.google.common.base.MoreObjects; -import com.google.common.collect.ImmutableList; import com.google.gcloud.bigquery.JobInfo.CreateDisposition; import com.google.gcloud.bigquery.JobInfo.WriteDisposition; -import java.io.Serializable; -import java.nio.channels.SeekableByteChannel; import java.util.List; -import java.util.Objects; /** - * Google BigQuery Configuration for a load operation. A load configuration can be used to build a - * {@link LoadJobInfo} or to load data into a table with a {@link com.google.gcloud.WriteChannel} - * ({@link BigQuery#writer(LoadConfiguration)}). + * Common interface for a load configuration. A load configuration + * ({@link WriteChannelConfiguration}) can be used to load data into a table with a + * {@link com.google.gcloud.WriteChannel} ({@link BigQuery#writer(WriteChannelConfiguration)}). + * A load configuration ({@link LoadJobConfiguration}) can also be used to create a load job + * ({@link JobInfo#of(JobConfiguration)}). */ -public class LoadConfiguration implements Serializable { - - private static final long serialVersionUID = 470267591917413578L; - - private final TableId destinationTable; - private final CreateDisposition createDisposition; - private final WriteDisposition writeDisposition; - private final FormatOptions formatOptions; - private final Integer maxBadRecords; - private final Schema schema; - private final Boolean ignoreUnknownValues; - private final List projectionFields; - - public static final class Builder { - - private TableId destinationTable; - private CreateDisposition createDisposition; - private WriteDisposition writeDisposition; - private FormatOptions formatOptions; - private Integer maxBadRecords; - private Schema schema; - private Boolean ignoreUnknownValues; - private List projectionFields; - - private Builder() {} +public interface LoadConfiguration { - private Builder(LoadConfiguration loadConfiguration) { - this.destinationTable = loadConfiguration.destinationTable; - this.createDisposition = loadConfiguration.createDisposition; - this.writeDisposition = loadConfiguration.writeDisposition; - this.formatOptions = loadConfiguration.formatOptions; - this.maxBadRecords = loadConfiguration.maxBadRecords; - this.schema = loadConfiguration.schema; - this.ignoreUnknownValues = loadConfiguration.ignoreUnknownValues; - this.projectionFields = loadConfiguration.projectionFields; - } - - private Builder(JobConfigurationLoad loadConfigurationPb) { - this.destinationTable = TableId.fromPb(loadConfigurationPb.getDestinationTable()); - if (loadConfigurationPb.getCreateDisposition() != null) { - this.createDisposition = - CreateDisposition.valueOf(loadConfigurationPb.getCreateDisposition()); - } - if (loadConfigurationPb.getWriteDisposition() != null) { - this.writeDisposition = WriteDisposition.valueOf(loadConfigurationPb.getWriteDisposition()); - } - if (loadConfigurationPb.getSourceFormat() != null) { - this.formatOptions = FormatOptions.of(loadConfigurationPb.getSourceFormat()); - } - if (loadConfigurationPb.getAllowJaggedRows() != null - || loadConfigurationPb.getAllowQuotedNewlines() != null - || loadConfigurationPb.getEncoding() != null - || loadConfigurationPb.getFieldDelimiter() != null - || loadConfigurationPb.getQuote() != null - || loadConfigurationPb.getSkipLeadingRows() != null) { - CsvOptions.Builder builder = CsvOptions.builder() - .allowJaggedRows(loadConfigurationPb.getAllowJaggedRows()) - .allowQuotedNewLines(loadConfigurationPb.getAllowQuotedNewlines()) - .encoding(loadConfigurationPb.getEncoding()) - .fieldDelimiter(loadConfigurationPb.getFieldDelimiter()) - .quote(loadConfigurationPb.getQuote()) - .skipLeadingRows(loadConfigurationPb.getSkipLeadingRows()); - this.formatOptions = builder.build(); - } - this.maxBadRecords = loadConfigurationPb.getMaxBadRecords(); - if (loadConfigurationPb.getSchema() != null) { - this.schema = Schema.fromPb(loadConfigurationPb.getSchema()); - } - this.ignoreUnknownValues = loadConfigurationPb.getIgnoreUnknownValues(); - this.projectionFields = loadConfigurationPb.getProjectionFields(); - } + interface Builder { /** * Sets the destination table to load the data into. */ - public Builder destinationTable(TableId destinationTable) { - this.destinationTable = destinationTable; - return this; - } + Builder destinationTable(TableId destinationTable); /** * Sets whether the job is allowed to create new tables. @@ -120,10 +43,7 @@ public Builder destinationTable(TableId destinationTable) { * @see * Create Disposition */ - public Builder createDisposition(CreateDisposition createDisposition) { - this.createDisposition = createDisposition; - return this; - } + Builder createDisposition(CreateDisposition createDisposition); /** * Sets the action that should occur if the destination table already exists. @@ -131,10 +51,7 @@ public Builder createDisposition(CreateDisposition createDisposition) { * @see * Write Disposition */ - public Builder writeDisposition(WriteDisposition writeDisposition) { - this.writeDisposition = writeDisposition; - return this; - } + Builder writeDisposition(WriteDisposition writeDisposition); /** * Sets the source format, and possibly some parsing options, of the external data. Supported @@ -144,30 +61,21 @@ public Builder writeDisposition(WriteDisposition writeDisposition) { * * Source Format */ - public Builder formatOptions(FormatOptions formatOptions) { - this.formatOptions = formatOptions; - return this; - } + Builder formatOptions(FormatOptions formatOptions); /** * Sets the maximum number of bad records that BigQuery can ignore when running the job. If the * number of bad records exceeds this value, an invalid error is returned in the job result. * By default no bad record is ignored. */ - public Builder maxBadRecords(Integer maxBadRecords) { - this.maxBadRecords = maxBadRecords; - return this; - } + Builder maxBadRecords(Integer maxBadRecords); /** * Sets the schema for the destination table. The schema can be omitted if the destination table * already exists, or if you're loading data from a Google Cloud Datastore backup (i.e. * {@code DATASTORE_BACKUP} format option). */ - public Builder schema(Schema schema) { - this.schema = schema; - return this; - } + Builder schema(Schema schema); /** * Sets whether BigQuery should allow extra values that are not represented in the table schema. @@ -175,10 +83,7 @@ public Builder schema(Schema schema) { * are treated as bad records, and if there are too many bad records, an invalid error is * returned in the job result. By default unknown values are not allowed. */ - public Builder ignoreUnknownValues(Boolean ignoreUnknownValues) { - this.ignoreUnknownValues = ignoreUnknownValues; - return this; - } + Builder ignoreUnknownValues(Boolean ignoreUnknownValues); /** * Sets which entity properties to load into BigQuery from a Cloud Datastore backup. This field @@ -187,34 +92,15 @@ public Builder ignoreUnknownValues(Boolean ignoreUnknownValues) { * all properties. If any named property isn't found in the Cloud Datastore backup, an invalid * error is returned in the job result. */ - public Builder projectionFields(List projectionFields) { - this.projectionFields = - projectionFields != null ? ImmutableList.copyOf(projectionFields) : null; - return this; - } - - public LoadConfiguration build() { - return new LoadConfiguration(this); - } - } + Builder projectionFields(List projectionFields); - private LoadConfiguration(Builder builder) { - this.destinationTable = checkNotNull(builder.destinationTable); - this.createDisposition = builder.createDisposition; - this.writeDisposition = builder.writeDisposition; - this.formatOptions = builder.formatOptions; - this.maxBadRecords = builder.maxBadRecords; - this.schema = builder.schema; - this.ignoreUnknownValues = builder.ignoreUnknownValues; - this.projectionFields = builder.projectionFields; + LoadConfiguration build(); } /** * Returns the destination table to load the data into. */ - public TableId destinationTable() { - return destinationTable; - } + TableId destinationTable(); /** * Returns whether the job is allowed to create new tables. @@ -222,9 +108,7 @@ public TableId destinationTable() { * @see * Create Disposition */ - public CreateDisposition createDisposition() { - return this.createDisposition; - } + CreateDisposition createDisposition(); /** * Returns the action that should occur if the destination table already exists. @@ -232,40 +116,30 @@ public CreateDisposition createDisposition() { * @see * Write Disposition */ - public WriteDisposition writeDisposition() { - return writeDisposition; - } + WriteDisposition writeDisposition(); /** * Returns additional properties used to parse CSV data (used when {@link #format()} is set * to CSV). Returns {@code null} if not set. */ - public CsvOptions csvOptions() { - return formatOptions instanceof CsvOptions ? (CsvOptions) formatOptions : null; - } + CsvOptions csvOptions(); /** * Returns the maximum number of bad records that BigQuery can ignore when running the job. If the * number of bad records exceeds this value, an invalid error is returned in the job result. * By default no bad record is ignored. */ - public Integer maxBadRecords() { - return maxBadRecords; - } + Integer maxBadRecords(); /** * Returns the schema for the destination table, if set. Returns {@code null} otherwise. */ - public Schema schema() { - return schema; - } + Schema schema(); /** * Returns the format of the data files. */ - public String format() { - return formatOptions != null ? formatOptions.type() : null; - } + String format(); /** * Returns whether BigQuery should allow extra values that are not represented in the table @@ -273,9 +147,7 @@ public String format() { * columns are treated as bad records, and if there are too many bad records, an invalid error is * returned in the job result. By default unknown values are not allowed. */ - public Boolean ignoreUnknownValues() { - return ignoreUnknownValues; - } + Boolean ignoreUnknownValues(); /** * Returns which entity properties to load into BigQuery from a Cloud Datastore backup. This field @@ -284,99 +156,10 @@ public Boolean ignoreUnknownValues() { * all properties. If any named property isn't found in the Cloud Datastore backup, an invalid * error is returned in the job result. */ - public List projectionFields() { - return projectionFields; - } - - public Builder toBuilder() { - return new Builder(this); - } - - @Override - public String toString() { - return MoreObjects.toStringHelper(this) - .add("destinationTable", destinationTable) - .add("createDisposition", createDisposition) - .add("writeDisposition", writeDisposition) - .add("formatOptions", formatOptions) - .add("maxBadRecords", maxBadRecords) - .add("schema", schema) - .add("ignoreUnknownValue", ignoreUnknownValues) - .add("projectionFields", projectionFields) - .toString(); - } - - @Override - public boolean equals(Object obj) { - return obj instanceof LoadConfiguration - && Objects.equals(toPb(), ((LoadConfiguration) obj).toPb()); - } - - @Override - public int hashCode() { - return Objects.hash(destinationTable, createDisposition, writeDisposition, formatOptions, - maxBadRecords, schema, ignoreUnknownValues, projectionFields); - } - - JobConfigurationLoad toPb() { - JobConfigurationLoad loadConfigurationPb = new JobConfigurationLoad(); - loadConfigurationPb.setDestinationTable(destinationTable.toPb()); - if (createDisposition != null) { - loadConfigurationPb.setCreateDisposition(createDisposition.toString()); - } - if (writeDisposition != null) { - loadConfigurationPb.setWriteDisposition(writeDisposition.toString()); - } - if (csvOptions() != null) { - CsvOptions csvOptions = csvOptions(); - loadConfigurationPb.setFieldDelimiter(csvOptions.fieldDelimiter()) - .setAllowJaggedRows(csvOptions.allowJaggedRows()) - .setAllowQuotedNewlines(csvOptions.allowQuotedNewLines()) - .setEncoding(csvOptions.encoding()) - .setQuote(csvOptions.quote()) - .setSkipLeadingRows(csvOptions.skipLeadingRows()); - } - if (schema != null) { - loadConfigurationPb.setSchema(schema.toPb()); - } - if (formatOptions != null) { - loadConfigurationPb.setSourceFormat(formatOptions.type()); - } - loadConfigurationPb.setMaxBadRecords(maxBadRecords); - loadConfigurationPb.setIgnoreUnknownValues(ignoreUnknownValues); - loadConfigurationPb.setProjectionFields(projectionFields); - return loadConfigurationPb; - } - - static LoadConfiguration fromPb(JobConfigurationLoad configurationPb) { - return new Builder(configurationPb).build(); - } + List projectionFields(); /** - * Creates a builder for a BigQuery Load Configuration given the destination table. + * Returns a builder for the load configuration object. */ - public static Builder builder(TableId destinationTable) { - return new Builder().destinationTable(destinationTable); - } - - /** - * Creates a builder for a BigQuery Load Configuration given the destination table and format. - */ - public static Builder builder(TableId destinationTable, FormatOptions format) { - return new Builder().destinationTable(destinationTable).formatOptions(format); - } - - /** - * Returns a BigQuery Load Configuration for the given destination table. - */ - public static LoadConfiguration of(TableId destinationTable) { - return builder(destinationTable).build(); - } - - /** - * Returns a BigQuery Load Configuration for the given destination table and format. - */ - public static LoadConfiguration of(TableId destinationTable, FormatOptions format) { - return builder(destinationTable).formatOptions(format).build(); - } + Builder toBuilder(); } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobConfiguration.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobConfiguration.java new file mode 100644 index 000000000000..9c9fa7a769b6 --- /dev/null +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobConfiguration.java @@ -0,0 +1,390 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.api.services.bigquery.model.JobConfigurationLoad; +import com.google.common.base.MoreObjects.ToStringHelper; +import com.google.common.collect.ImmutableList; + +import java.util.List; +import java.util.Objects; + +/** + * Google BigQuery load job configuration. A load job loads data from one of several formats into a + * table. Data is provided as URIs that point to objects in Google Cloud Storage. Load job + * configurations have {@link JobConfiguration.Type#LOAD} type. + */ +public final class LoadJobConfiguration extends JobConfiguration implements LoadConfiguration { + + private static final long serialVersionUID = -2673554846792429829L; + + private final List sourceUris; + private final TableId destinationTable; + private final JobInfo.CreateDisposition createDisposition; + private final JobInfo.WriteDisposition writeDisposition; + private final FormatOptions formatOptions; + private final Integer maxBadRecords; + private final Schema schema; + private final Boolean ignoreUnknownValues; + private final List projectionFields; + + public static final class Builder + extends JobConfiguration.Builder + implements LoadConfiguration.Builder { + + private List sourceUris; + private TableId destinationTable; + private JobInfo.CreateDisposition createDisposition; + private JobInfo.WriteDisposition writeDisposition; + private FormatOptions formatOptions; + private Integer maxBadRecords; + private Schema schema; + private Boolean ignoreUnknownValues; + private List projectionFields; + + private Builder() { + super(Type.LOAD); + } + + private Builder(LoadJobConfiguration loadConfiguration) { + this(); + this.destinationTable = loadConfiguration.destinationTable; + this.createDisposition = loadConfiguration.createDisposition; + this.writeDisposition = loadConfiguration.writeDisposition; + this.formatOptions = loadConfiguration.formatOptions; + this.maxBadRecords = loadConfiguration.maxBadRecords; + this.schema = loadConfiguration.schema; + this.ignoreUnknownValues = loadConfiguration.ignoreUnknownValues; + this.projectionFields = loadConfiguration.projectionFields; + this.sourceUris = loadConfiguration.sourceUris; + } + + private Builder(com.google.api.services.bigquery.model.JobConfiguration configurationPb) { + this(); + JobConfigurationLoad loadConfigurationPb = configurationPb.getLoad(); + this.destinationTable = TableId.fromPb(loadConfigurationPb.getDestinationTable()); + if (loadConfigurationPb.getCreateDisposition() != null) { + this.createDisposition = + JobInfo.CreateDisposition.valueOf(loadConfigurationPb.getCreateDisposition()); + } + if (loadConfigurationPb.getWriteDisposition() != null) { + this.writeDisposition = + JobInfo.WriteDisposition.valueOf(loadConfigurationPb.getWriteDisposition()); + } + if (loadConfigurationPb.getSourceFormat() != null) { + this.formatOptions = FormatOptions.of(loadConfigurationPb.getSourceFormat()); + } + if (loadConfigurationPb.getAllowJaggedRows() != null + || loadConfigurationPb.getAllowQuotedNewlines() != null + || loadConfigurationPb.getEncoding() != null + || loadConfigurationPb.getFieldDelimiter() != null + || loadConfigurationPb.getQuote() != null + || loadConfigurationPb.getSkipLeadingRows() != null) { + CsvOptions.Builder builder = CsvOptions.builder() + .allowJaggedRows(loadConfigurationPb.getAllowJaggedRows()) + .allowQuotedNewLines(loadConfigurationPb.getAllowQuotedNewlines()) + .encoding(loadConfigurationPb.getEncoding()) + .fieldDelimiter(loadConfigurationPb.getFieldDelimiter()) + .quote(loadConfigurationPb.getQuote()) + .skipLeadingRows(loadConfigurationPb.getSkipLeadingRows()); + this.formatOptions = builder.build(); + } + this.maxBadRecords = loadConfigurationPb.getMaxBadRecords(); + if (loadConfigurationPb.getSchema() != null) { + this.schema = Schema.fromPb(loadConfigurationPb.getSchema()); + } + this.ignoreUnknownValues = loadConfigurationPb.getIgnoreUnknownValues(); + this.projectionFields = loadConfigurationPb.getProjectionFields(); + if (loadConfigurationPb.getSourceUris() != null) { + this.sourceUris = ImmutableList.copyOf(configurationPb.getLoad().getSourceUris()); + } + } + + @Override + public Builder destinationTable(TableId destinationTable) { + this.destinationTable = destinationTable; + return this; + } + + @Override + public Builder createDisposition(JobInfo.CreateDisposition createDisposition) { + this.createDisposition = createDisposition; + return this; + } + + @Override + public Builder writeDisposition(JobInfo.WriteDisposition writeDisposition) { + this.writeDisposition = writeDisposition; + return this; + } + + @Override + public Builder formatOptions(FormatOptions formatOptions) { + this.formatOptions = formatOptions; + return this; + } + + @Override + public Builder maxBadRecords(Integer maxBadRecords) { + this.maxBadRecords = maxBadRecords; + return this; + } + + @Override + public Builder schema(Schema schema) { + this.schema = schema; + return this; + } + + @Override + public Builder ignoreUnknownValues(Boolean ignoreUnknownValues) { + this.ignoreUnknownValues = ignoreUnknownValues; + return this; + } + + @Override + public Builder projectionFields(List projectionFields) { + this.projectionFields = + projectionFields != null ? ImmutableList.copyOf(projectionFields) : null; + return this; + } + + /** + * Sets the fully-qualified URIs that point to source data in Google Cloud Storage (e.g. + * gs://bucket/path). Each URI can contain one '*' wildcard character and it must come after the + * 'bucket' name. + */ + public Builder sourceUris(List sourceUris) { + this.sourceUris = ImmutableList.copyOf(checkNotNull(sourceUris)); + return this; + } + + @Override + public LoadJobConfiguration build() { + return new LoadJobConfiguration(this); + } + } + + private LoadJobConfiguration(Builder builder) { + super(builder); + this.sourceUris = builder.sourceUris; + this.destinationTable = builder.destinationTable; + this.createDisposition = builder.createDisposition; + this.writeDisposition = builder.writeDisposition; + this.formatOptions = builder.formatOptions; + this.maxBadRecords = builder.maxBadRecords; + this.schema = builder.schema; + this.ignoreUnknownValues = builder.ignoreUnknownValues; + this.projectionFields = builder.projectionFields; + } + + @Override + public TableId destinationTable() { + return destinationTable; + } + + @Override + public JobInfo.CreateDisposition createDisposition() { + return this.createDisposition; + } + + @Override + public JobInfo.WriteDisposition writeDisposition() { + return writeDisposition; + } + + @Override + public CsvOptions csvOptions() { + return formatOptions instanceof CsvOptions ? (CsvOptions) formatOptions : null; + } + + @Override + public Integer maxBadRecords() { + return maxBadRecords; + } + + @Override + public Schema schema() { + return schema; + } + + @Override + public String format() { + return formatOptions != null ? formatOptions.type() : null; + } + + @Override + public Boolean ignoreUnknownValues() { + return ignoreUnknownValues; + } + + @Override + public List projectionFields() { + return projectionFields; + } + + /** + * Returns the fully-qualified URIs that point to source data in Google Cloud Storage (e.g. + * gs://bucket/path). Each URI can contain one '*' wildcard character and it must come after the + * 'bucket' name. + */ + public List sourceUris() { + return sourceUris; + } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + ToStringHelper toStringHelper() { + return super.toStringHelper() + .add("destinationTable", destinationTable) + .add("createDisposition", createDisposition) + .add("writeDisposition", writeDisposition) + .add("formatOptions", formatOptions) + .add("maxBadRecords", maxBadRecords) + .add("schema", schema) + .add("ignoreUnknownValue", ignoreUnknownValues) + .add("projectionFields", projectionFields) + .add("sourceUris", sourceUris); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof LoadJobConfiguration && baseEquals((LoadJobConfiguration) obj); + } + + @Override + public int hashCode() { + return Objects.hash(baseHashCode(), sourceUris); + } + + @Override + LoadJobConfiguration setProjectId(String projectId) { + return toBuilder().destinationTable(destinationTable().setProjectId(projectId)).build(); + } + + @Override + com.google.api.services.bigquery.model.JobConfiguration toPb() { + JobConfigurationLoad loadConfigurationPb = new JobConfigurationLoad(); + loadConfigurationPb.setDestinationTable(destinationTable.toPb()); + if (createDisposition != null) { + loadConfigurationPb.setCreateDisposition(createDisposition.toString()); + } + if (writeDisposition != null) { + loadConfigurationPb.setWriteDisposition(writeDisposition.toString()); + } + if (csvOptions() != null) { + CsvOptions csvOptions = csvOptions(); + loadConfigurationPb.setFieldDelimiter(csvOptions.fieldDelimiter()) + .setAllowJaggedRows(csvOptions.allowJaggedRows()) + .setAllowQuotedNewlines(csvOptions.allowQuotedNewLines()) + .setEncoding(csvOptions.encoding()) + .setQuote(csvOptions.quote()) + .setSkipLeadingRows(csvOptions.skipLeadingRows()); + } + if (schema != null) { + loadConfigurationPb.setSchema(schema.toPb()); + } + if (formatOptions != null) { + loadConfigurationPb.setSourceFormat(formatOptions.type()); + } + loadConfigurationPb.setMaxBadRecords(maxBadRecords); + loadConfigurationPb.setIgnoreUnknownValues(ignoreUnknownValues); + loadConfigurationPb.setProjectionFields(projectionFields); + if (sourceUris != null) { + loadConfigurationPb.setSourceUris(ImmutableList.copyOf(sourceUris)); + } + return new com.google.api.services.bigquery.model.JobConfiguration() + .setLoad(loadConfigurationPb); + } + + /** + * Creates a builder for a BigQuery Load Job configuration given the destination table and source + * URIs. + */ + public static Builder builder(TableId destinationTable, List sourceUris) { + return new Builder().destinationTable(destinationTable).sourceUris(sourceUris); + } + + /** + * Creates a builder for a BigQuery Load Job configuration given the destination table and source + * URI. + */ + public static Builder builder(TableId destinationTable, String sourceUri) { + return builder(destinationTable, ImmutableList.of(sourceUri)); + } + + /** + * Creates a builder for a BigQuery Load Job configuration given the destination table, format and + * source URIs. + */ + public static Builder builder(TableId destinationTable, List sourceUris, + FormatOptions format) { + return builder(destinationTable, sourceUris).formatOptions(format); + } + + /** + * Creates a builder for a BigQuery Load Job configuration given the destination table, format and + * source URI. + */ + public static Builder builder(TableId destinationTable, String sourceUri, FormatOptions format) { + return builder(destinationTable, ImmutableList.of(sourceUri), format); + } + + /** + * Returns a BigQuery Load Job Configuration for the given destination table and source URIs. + */ + public static LoadJobConfiguration of(TableId destinationTable, List sourceUris) { + return builder(destinationTable, sourceUris).build(); + } + + /** + * Returns a BigQuery Load Job Configuration for the given destination table and source URI. + */ + public static LoadJobConfiguration of(TableId destinationTable, String sourceUri) { + return of(destinationTable, ImmutableList.of(sourceUri)); + } + + /** + * Returns a BigQuery Load Job Configuration for the given destination table, format and source + * URI. + */ + public static LoadJobConfiguration of(TableId destinationTable, List sourceUris, + FormatOptions format) { + return builder(destinationTable, sourceUris, format).build(); + } + + /** + * Returns a BigQuery Load Job Configuration for the given destination table, format and source + * URI. + */ + public static LoadJobConfiguration of(TableId destinationTable, String sourceUri, + FormatOptions format) { + return of(destinationTable, ImmutableList.of(sourceUri), format); + } + + @SuppressWarnings("unchecked") + static LoadJobConfiguration fromPb( + com.google.api.services.bigquery.model.JobConfiguration confPb) { + return new Builder(confPb).build(); + } +} diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobInfo.java deleted file mode 100644 index 4f8d03cbc6a9..000000000000 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobInfo.java +++ /dev/null @@ -1,186 +0,0 @@ -/* - * Copyright 2015 Google Inc. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.google.gcloud.bigquery; - -import static com.google.common.base.Preconditions.checkNotNull; - -import com.google.api.services.bigquery.model.Job; -import com.google.api.services.bigquery.model.JobConfiguration; -import com.google.api.services.bigquery.model.JobConfigurationLoad; -import com.google.common.base.MoreObjects.ToStringHelper; -import com.google.common.collect.ImmutableList; -import com.google.gcloud.bigquery.JobStatistics.LoadStatistics; - -import java.util.List; -import java.util.Objects; - -/** - * Google BigQuery Load Job. A Load Job loads data from one of several formats into a table. Data is - * provided as URIs that point to objects in Google Cloud Storage. - */ -public class LoadJobInfo extends JobInfo { - - private static final long serialVersionUID = 6349304826867750535L; - - private final List sourceUris; - private final LoadConfiguration configuration; - - public static final class Builder extends JobInfo.Builder { - - private List sourceUris; - private LoadConfiguration configuration; - - private Builder() {} - - private Builder(LoadJobInfo jobInfo) { - super(jobInfo); - this.sourceUris = jobInfo.sourceUris; - this.configuration = jobInfo.configuration; - } - - private Builder(Job jobPb) { - super(jobPb); - JobConfigurationLoad loadConfigurationPb = jobPb.getConfiguration().getLoad(); - this.configuration = LoadConfiguration.fromPb(loadConfigurationPb); - this.sourceUris = loadConfigurationPb.getSourceUris(); - } - - /** - * Sets the fully-qualified URIs that point to source data in Google Cloud Storage (e.g. - * gs://bucket/path). Each URI can contain one '*' wildcard character and it must come after the - * 'bucket' name. - */ - public Builder sourceUris(List sourceUris) { - this.sourceUris = sourceUris != null ? ImmutableList.copyOf(sourceUris) : null; - return this; - } - - /** - * Sets the configuration for the BigQuery Load Job. - */ - public Builder configuration(LoadConfiguration configuration) { - this.configuration = configuration; - return this; - } - - @Override - public LoadJobInfo build() { - return new LoadJobInfo(this); - } - } - - private LoadJobInfo(Builder builder) { - super(builder); - this.sourceUris = builder.sourceUris; - this.configuration = builder.configuration; - } - - /** - * Returns the fully-qualified URIs that point to source data in Google Cloud Storage (e.g. - * gs://bucket/path). Each URI can contain one '*' wildcard character and it must come after the - * 'bucket' name. - */ - public List sourceUris() { - return sourceUris; - } - - /** - * Returns the configuration for the BigQuery Load Job. - */ - public LoadConfiguration configuration() { - return configuration; - } - - @Override - public Builder toBuilder() { - return new Builder(this); - } - - @Override - ToStringHelper toStringHelper() { - return super.toStringHelper().add("sourceUris", sourceUris).add("configuration", configuration); - } - - @Override - public boolean equals(Object obj) { - return obj instanceof LoadJobInfo && Objects.equals(toPb(), ((LoadJobInfo) obj).toPb()); - } - - @Override - public int hashCode() { - return Objects.hash(super.hashCode(), sourceUris, configuration); - } - - @Override - Job toPb() { - JobConfigurationLoad loadConfigurationPb = configuration.toPb(); - loadConfigurationPb.setSourceUris(sourceUris); - return super.toPb().setConfiguration(new JobConfiguration().setLoad(loadConfigurationPb)); - } - - /** - * Creates a builder for a BigQuery Load Job given the load configuration and source URI. - */ - public static Builder builder(LoadConfiguration configuration, String sourceUri) { - return builder(configuration, ImmutableList.of(checkNotNull(sourceUri))); - } - - /** - * Creates a builder for a BigQuery Load Job given the load configuration and source URIs. - */ - public static Builder builder(LoadConfiguration configuration, List sourceUris) { - return new Builder().configuration(configuration).sourceUris(sourceUris); - } - - /** - * Returns a BigQuery Load Job for the given load configuration and source URI. Job's id is chosen - * by the service. - */ - public static LoadJobInfo of(LoadConfiguration configuration, String sourceUri) { - return builder(configuration, sourceUri).build(); - } - - /** - * Returns a BigQuery Load Job for the given load configuration and source URIs. Job's id is - * chosen by the service. - */ - public static LoadJobInfo of(LoadConfiguration configuration, List sourceUris) { - return builder(configuration, sourceUris).build(); - } - - /** - * Returns a BigQuery Load Job for the given load configuration and source URI. Job's id is set to - * the provided value. - */ - public static LoadJobInfo of(JobId jobId, LoadConfiguration configuration, String sourceUri) { - return builder(configuration, sourceUri).jobId(jobId).build(); - } - - /** - * Returns a BigQuery Load Job for the given load configuration and source URIs. Job's id is set - * to the provided value. - */ - public static LoadJobInfo of(JobId jobId, LoadConfiguration configuration, - List sourceUris) { - return builder(configuration, sourceUris).jobId(jobId).build(); - } - - @SuppressWarnings("unchecked") - static LoadJobInfo fromPb(Job jobPb) { - return new Builder(jobPb).build(); - } -} diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobConfiguration.java similarity index 80% rename from gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobInfo.java rename to gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobConfiguration.java index ad76d229bf2d..688611d07526 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobConfiguration.java @@ -1,5 +1,5 @@ /* - * Copyright 2015 Google Inc. All Rights Reserved. + * Copyright 2016 Google Inc. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -18,26 +18,26 @@ import static com.google.common.base.Preconditions.checkNotNull; -import com.google.api.services.bigquery.model.Job; -import com.google.api.services.bigquery.model.JobConfiguration; import com.google.api.services.bigquery.model.JobConfigurationQuery; import com.google.common.base.MoreObjects.ToStringHelper; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Lists; import com.google.common.collect.Maps; -import com.google.gcloud.bigquery.JobStatistics.QueryStatistics; +import com.google.gcloud.bigquery.JobInfo.CreateDisposition; +import com.google.gcloud.bigquery.JobInfo.WriteDisposition; import java.util.List; import java.util.Map; import java.util.Objects; /** - * Google BigQuery Query Job. A Query Job runs a query against BigQuery data. + * Google BigQuery Query Job configuration. A Query Job runs a query against BigQuery data. Query + * job configurations have {@link JobConfiguration.Type#QUERY} type. */ -public class QueryJobInfo extends JobInfo { +public final class QueryJobConfiguration extends JobConfiguration { - private static final long serialVersionUID = -8708709356039780158L; + private static final long serialVersionUID = -1108948249081804890L; /** * Priority levels for a query. If not specified the priority is assumed to be @@ -61,7 +61,7 @@ public enum Priority { private final String query; private final TableId destinationTable; - private final Map tableDefinitions; + private final Map tableDefinitions; private final List userDefinedFunctions; private final CreateDisposition createDisposition; private final WriteDisposition writeDisposition; @@ -72,12 +72,12 @@ public enum Priority { private final Boolean flattenResults; private final Boolean dryRun; - public static final class Builder extends JobInfo.Builder { + public static final class Builder + extends JobConfiguration.Builder { private String query; private TableId destinationTable; - private Map tableDefinitions; + private Map tableDefinitions; private List userDefinedFunctions; private CreateDisposition createDisposition; private WriteDisposition writeDisposition; @@ -88,32 +88,34 @@ public static final class Builder extends JobInfo.Builder tableDefinitions) { + public Builder tableDefinitions(Map tableDefinitions) { this.tableDefinitions = tableDefinitions != null ? Maps.newHashMap(tableDefinitions) : null; - return self(); + return this; } /** @@ -175,12 +179,12 @@ public Builder tableDefinitions(Map tableDefi * @param tableName name of the table * @param tableDefinition external data configuration for the table used by this query */ - public Builder addTableDefinition(String tableName, ExternalDataConfiguration tableDefinition) { + public Builder addTableDefinition(String tableName, ExternalTableDefinition tableDefinition) { if (this.tableDefinitions == null) { this.tableDefinitions = Maps.newHashMap(); } this.tableDefinitions.put(checkNotNull(tableName), checkNotNull(tableDefinition)); - return self(); + return this; } /** @@ -191,7 +195,7 @@ public Builder addTableDefinition(String tableName, ExternalDataConfiguration ta public Builder userDefinedFunctions(List userDefinedFunctions) { this.userDefinedFunctions = userDefinedFunctions != null ? ImmutableList.copyOf(userDefinedFunctions) : null; - return self(); + return this; } /** @@ -202,7 +206,7 @@ public Builder userDefinedFunctions(List userDefinedFunctio */ public Builder createDisposition(CreateDisposition createDisposition) { this.createDisposition = createDisposition; - return self(); + return this; } /** @@ -213,7 +217,7 @@ public Builder createDisposition(CreateDisposition createDisposition) { */ public Builder writeDisposition(WriteDisposition writeDisposition) { this.writeDisposition = writeDisposition; - return self(); + return this; } /** @@ -222,7 +226,7 @@ public Builder writeDisposition(WriteDisposition writeDisposition) { */ public Builder defaultDataset(DatasetId defaultDataset) { this.defaultDataset = defaultDataset; - return self(); + return this; } /** @@ -239,7 +243,7 @@ public Builder defaultDataset(String defaultDataset) { */ public Builder priority(Priority priority) { this.priority = priority; - return self(); + return this; } /** @@ -252,7 +256,7 @@ public Builder priority(Priority priority) { */ public Builder allowLargeResults(Boolean allowLargeResults) { this.allowLargeResults = allowLargeResults; - return self(); + return this; } /** @@ -264,7 +268,7 @@ public Builder allowLargeResults(Boolean allowLargeResults) { */ public Builder useQueryCache(Boolean useQueryCache) { this.useQueryCache = useQueryCache; - return self(); + return this; } /** @@ -276,7 +280,7 @@ public Builder useQueryCache(Boolean useQueryCache) { */ public Builder flattenResults(Boolean flattenResults) { this.flattenResults = flattenResults; - return self(); + return this; } /** @@ -286,16 +290,15 @@ public Builder flattenResults(Boolean flattenResults) { */ public Builder dryRun(Boolean dryRun) { this.dryRun = dryRun; - return self(); + return this; } - @Override - public QueryJobInfo build() { - return new QueryJobInfo(this); + public QueryJobConfiguration build() { + return new QueryJobConfiguration(this); } } - private QueryJobInfo(Builder builder) { + private QueryJobConfiguration(Builder builder) { super(builder); this.query = checkNotNull(builder.query); this.allowLargeResults = builder.allowLargeResults; @@ -380,7 +383,7 @@ public String query() { * sources. By defining these properties, the data sources can be queried as if they were * standard BigQuery tables. */ - public Map tableDefinitions() { + public Map tableDefinitions() { return tableDefinitions; } @@ -447,19 +450,32 @@ ToStringHelper toStringHelper() { @Override public boolean equals(Object obj) { - return obj instanceof QueryJobInfo && Objects.equals(toPb(), ((QueryJobInfo) obj).toPb()); + return obj instanceof QueryJobConfiguration && baseEquals((QueryJobConfiguration) obj); } @Override public int hashCode() { - return Objects.hash(super.hashCode(), allowLargeResults, createDisposition, destinationTable, + return Objects.hash(baseHashCode(), allowLargeResults, createDisposition, destinationTable, defaultDataset, flattenResults, priority, query, tableDefinitions, useQueryCache, userDefinedFunctions, writeDisposition, dryRun); } @Override - Job toPb() { - JobConfiguration configurationPb = new JobConfiguration(); + QueryJobConfiguration setProjectId(String projectId) { + Builder builder = toBuilder(); + if (destinationTable() != null) { + builder.destinationTable(destinationTable().setProjectId(projectId)); + } + if (defaultDataset() != null) { + builder.defaultDataset(defaultDataset().setProjectId(projectId)); + } + return builder.build(); + } + + @Override + com.google.api.services.bigquery.model.JobConfiguration toPb() { + com.google.api.services.bigquery.model.JobConfiguration configurationPb = + new com.google.api.services.bigquery.model.JobConfiguration(); JobConfigurationQuery queryConfigurationPb = new JobConfigurationQuery(); queryConfigurationPb.setQuery(query); configurationPb.setDryRun(dryRun()); @@ -482,8 +498,8 @@ Job toPb() { queryConfigurationPb.setPriority(priority.toString()); } if (tableDefinitions != null) { - queryConfigurationPb.setTableDefinitions( - Maps.transformValues(tableDefinitions, ExternalDataConfiguration.TO_PB_FUNCTION)); + queryConfigurationPb.setTableDefinitions(Maps.transformValues(tableDefinitions, + ExternalTableDefinition.TO_EXTERNAL_DATA_FUNCTION)); } if (useQueryCache != null) { queryConfigurationPb.setUseQueryCache(useQueryCache); @@ -495,7 +511,7 @@ Job toPb() { if (writeDisposition != null) { queryConfigurationPb.setWriteDisposition(writeDisposition.toString()); } - return super.toPb().setConfiguration(configurationPb.setQuery(queryConfigurationPb)); + return configurationPb.setQuery(queryConfigurationPb); } /** @@ -509,20 +525,13 @@ public static Builder builder(String query) { * Returns a BigQuery Copy Job for the given the query to be run. Job's id is chosen by the * service. */ - public static QueryJobInfo of(String query) { + public static QueryJobConfiguration of(String query) { return builder(query).build(); } - /** - * Returns a BigQuery Copy Job for the given the query to be run. Job's id is set to the provided - * value. - */ - public static QueryJobInfo of(JobId jobId, String query) { - return builder(query).jobId(jobId).build(); - } - @SuppressWarnings("unchecked") - static QueryJobInfo fromPb(Job jobPb) { + static QueryJobConfiguration fromPb( + com.google.api.services.bigquery.model.JobConfiguration jobPb) { return new Builder(jobPb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryRequest.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryRequest.java index 64fbb3e931fc..5f99f3c5b4ee 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryRequest.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryRequest.java @@ -35,26 +35,26 @@ * {@link QueryResponse#jobCompleted()} returns {@code true}. * *

Example usage of a query request: - *

    {@code
- *    // Substitute "field", "table" and "dataset" with real field, table and dataset identifiers
- *    QueryRequest request = QueryRequest.builder("SELECT field FROM table")
- *      .defaultDataset(DatasetId.of("dataset"))
- *      .maxWaitTime(60000L)
- *      .maxResults(1000L)
- *      .build();
- *    QueryResponse response = bigquery.query(request);
- *    while (!response.jobCompleted()) {
- *      Thread.sleep(1000);
- *      response = bigquery.getQueryResults(response.jobId());
- *    }
- *    List executionErrors = response.executionErrors();
- *    // look for errors in executionErrors
- *    QueryResult result = response.result();
- *    Iterator> rowIterator = result.iterateAll();
- *    while(rowIterator.hasNext()) {
- *      List row = rowIterator.next();
- *      // do something with row
- *    }
+ * 
 {@code
+ * // Substitute "field", "table" and "dataset" with real field, table and dataset identifiers
+ * QueryRequest request = QueryRequest.builder("SELECT field FROM table")
+ *     .defaultDataset(DatasetId.of("dataset"))
+ *     .maxWaitTime(60000L)
+ *     .maxResults(1000L)
+ *     .build();
+ * QueryResponse response = bigquery.query(request);
+ * while (!response.jobCompleted()) {
+ *   Thread.sleep(1000);
+ *   response = bigquery.getQueryResults(response.jobId());
+ * }
+ * List executionErrors = response.executionErrors();
+ * // look for errors in executionErrors
+ * QueryResult result = response.result();
+ * Iterator> rowIterator = result.iterateAll();
+ * while(rowIterator.hasNext()) {
+ *   List row = rowIterator.next();
+ *   // do something with row
+ * }
  * }
* * @see Query @@ -120,8 +120,8 @@ public Builder defaultDataset(String defaultDataset) { * Sets how long to wait for the query to complete, in milliseconds, before the request times * out and returns. Note that this is only a timeout for the request, not the query. If the * query takes longer to run than the timeout value, the call returns without any results and - * with the {@link QueryResponse#jobCompleted()} set to {@code false}. If not set, a wait time of - * 10000 milliseconds (10 seconds) is used. + * with the {@link QueryResponse#jobCompleted()} set to {@code false}. If not set, a wait time + * of 10000 milliseconds (10 seconds) is used. */ public Builder maxWaitTime(Long maxWaitTime) { this.maxWaitTime = maxWaitTime; @@ -253,6 +253,14 @@ public boolean equals(Object obj) { return obj instanceof QueryRequest && Objects.equals(toPb(), ((QueryRequest) obj).toPb()); } + QueryRequest setProjectId(String projectId) { + Builder builder = toBuilder(); + if (defaultDataset() != null) { + builder.defaultDataset(defaultDataset().setProjectId(projectId)); + } + return builder.build(); + } + com.google.api.services.bigquery.model.QueryRequest toPb() { com.google.api.services.bigquery.model.QueryRequest queryRequestPb = new com.google.api.services.bigquery.model.QueryRequest().setQuery(query); diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryResponse.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryResponse.java index 77386747754f..12000cc1cbd2 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryResponse.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryResponse.java @@ -29,20 +29,20 @@ * Query Request ({@link BigQuery#query(QueryRequest)}). * *

Example usage of a query response: - *

    {@code
- *    QueryResponse response = bigquery.query(request);
- *    while (!response.jobCompleted()) {
- *      Thread.sleep(1000);
- *      response = bigquery.getQueryResults(response.jobId());
- *    }
- *    List executionErrors = response.executionErrors();
- *    // look for errors in executionErrors
- *    QueryResult result = response.result();
- *    Iterator> rowIterator = result.iterateAll();
- *    while(rowIterator.hasNext()) {
- *      List row = rowIterator.next();
- *      // do something with row
- *    }
+ * 
 {@code
+ * QueryResponse response = bigquery.query(request);
+ * while (!response.jobCompleted()) {
+ *   Thread.sleep(1000);
+ *   response = bigquery.getQueryResults(response.jobId());
+ * }
+ * List executionErrors = response.executionErrors();
+ * // look for errors in executionErrors
+ * QueryResult result = response.result();
+ * Iterator> rowIterator = result.iterateAll();
+ * while(rowIterator.hasNext()) {
+ *   List row = rowIterator.next();
+ *   // do something with row
+ * }
  * }
* * @see Get Query diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/StandardTableDefinition.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/StandardTableDefinition.java new file mode 100644 index 000000000000..d0e49157a99c --- /dev/null +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/StandardTableDefinition.java @@ -0,0 +1,282 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import com.google.api.services.bigquery.model.Streamingbuffer; +import com.google.api.services.bigquery.model.Table; +import com.google.common.base.MoreObjects; +import com.google.common.base.MoreObjects.ToStringHelper; + +import java.io.Serializable; +import java.math.BigInteger; +import java.util.Objects; + +/** + * A Google BigQuery default table definition. This definition is used for standard, two-dimensional + * tables with individual records organized in rows, and a data type assigned to each column (also + * called a field). Individual fields within a record may contain nested and repeated children + * fields. Every table is described by a schema that describes field names, types, and other + * information. + * + * @see Managing Tables + */ +public class StandardTableDefinition extends TableDefinition { + + private static final long serialVersionUID = 2113445776046717900L; + + private final Long numBytes; + private final Long numRows; + private final String location; + private final StreamingBuffer streamingBuffer; + + /** + * Google BigQuery Table's Streaming Buffer information. This class contains information on a + * table's streaming buffer as the estimated size in number of rows/bytes. + */ + public static class StreamingBuffer implements Serializable { + + private static final long serialVersionUID = 822027055549277843L; + private final long estimatedRows; + private final long estimatedBytes; + private final long oldestEntryTime; + + StreamingBuffer(long estimatedRows, long estimatedBytes, long oldestEntryTime) { + this.estimatedRows = estimatedRows; + this.estimatedBytes = estimatedBytes; + this.oldestEntryTime = oldestEntryTime; + } + + /** + * Returns a lower-bound estimate of the number of rows currently in the streaming buffer. + */ + public long estimatedRows() { + return estimatedRows; + } + + /** + * Returns a lower-bound estimate of the number of bytes currently in the streaming buffer. + */ + public long estimatedBytes() { + return estimatedBytes; + } + + /** + * Returns the timestamp of the oldest entry in the streaming buffer, in milliseconds since + * epoch. + */ + public long oldestEntryTime() { + return oldestEntryTime; + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this) + .add("estimatedRows", estimatedRows) + .add("estimatedBytes", estimatedBytes) + .add("oldestEntryTime", oldestEntryTime) + .toString(); + } + + @Override + public int hashCode() { + return Objects.hash(estimatedRows, estimatedBytes, oldestEntryTime); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof StreamingBuffer + && Objects.equals(toPb(), ((StreamingBuffer) obj).toPb()); + } + + Streamingbuffer toPb() { + return new Streamingbuffer() + .setEstimatedBytes(BigInteger.valueOf(estimatedBytes)) + .setEstimatedRows(BigInteger.valueOf(estimatedRows)) + .setOldestEntryTime(BigInteger.valueOf(oldestEntryTime)); + } + + static StreamingBuffer fromPb(Streamingbuffer streamingBufferPb) { + return new StreamingBuffer(streamingBufferPb.getEstimatedRows().longValue(), + streamingBufferPb.getEstimatedBytes().longValue(), + streamingBufferPb.getOldestEntryTime().longValue()); + } + } + + public static final class Builder + extends TableDefinition.Builder { + + private Long numBytes; + private Long numRows; + private String location; + private StreamingBuffer streamingBuffer; + + private Builder() { + super(Type.TABLE); + } + + private Builder(StandardTableDefinition tableDefinition) { + super(tableDefinition); + this.numBytes = tableDefinition.numBytes; + this.numRows = tableDefinition.numRows; + this.location = tableDefinition.location; + this.streamingBuffer = tableDefinition.streamingBuffer; + } + + private Builder(Table tablePb) { + super(tablePb); + if (tablePb.getNumRows() != null) { + this.numRows(tablePb.getNumRows().longValue()); + } + this.numBytes = tablePb.getNumBytes(); + this.location = tablePb.getLocation(); + if (tablePb.getStreamingBuffer() != null) { + this.streamingBuffer = StreamingBuffer.fromPb(tablePb.getStreamingBuffer()); + } + } + + Builder numBytes(Long numBytes) { + this.numBytes = numBytes; + return self(); + } + + Builder numRows(Long numRows) { + this.numRows = numRows; + return self(); + } + + Builder location(String location) { + this.location = location; + return self(); + } + + Builder streamingBuffer(StreamingBuffer streamingBuffer) { + this.streamingBuffer = streamingBuffer; + return self(); + } + + /** + * Creates a {@code StandardTableDefinition} object. + */ + @Override + public StandardTableDefinition build() { + return new StandardTableDefinition(this); + } + } + + private StandardTableDefinition(Builder builder) { + super(builder); + this.numBytes = builder.numBytes; + this.numRows = builder.numRows; + this.location = builder.location; + this.streamingBuffer = builder.streamingBuffer; + } + + /** + * Returns the size of this table in bytes, excluding any data in the streaming buffer. + */ + public Long numBytes() { + return numBytes; + } + + /** + * Returns the number of rows in this table, excluding any data in the streaming buffer. + */ + public Long numRows() { + return numRows; + } + + /** + * Returns the geographic location where the table should reside. This value is inherited from the + * dataset. + * + * @see + * Dataset Location + */ + public String location() { + return location; + } + + /** + * Returns information on the table's streaming buffer if any exists. Returns {@code null} if no + * streaming buffer exists. + */ + public StreamingBuffer streamingBuffer() { + return streamingBuffer; + } + + /** + * Returns a builder for a BigQuery standard table definition. + */ + public static Builder builder() { + return new Builder(); + } + + /** + * Creates a BigQuery standard table definition given its schema. + * + * @param schema the schema of the table + */ + public static StandardTableDefinition of(Schema schema) { + return builder().schema(schema).build(); + } + + /** + * Returns a builder for the {@code StandardTableDefinition} object. + */ + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + ToStringHelper toStringHelper() { + return super.toStringHelper() + .add("numBytes", numBytes) + .add("numRows", numRows) + .add("location", location) + .add("streamingBuffer", streamingBuffer); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof StandardTableDefinition && baseEquals((StandardTableDefinition) obj); + } + + @Override + public int hashCode() { + return Objects.hash(baseHashCode(), numBytes, numRows, location, streamingBuffer); + } + + @Override + Table toPb() { + Table tablePb = super.toPb(); + if (numRows != null) { + tablePb.setNumRows(BigInteger.valueOf(numRows)); + } + tablePb.setNumBytes(numBytes); + tablePb.setLocation(location); + if (streamingBuffer != null) { + tablePb.setStreamingBuffer(streamingBuffer.toPb()); + } + return tablePb; + } + + @SuppressWarnings("unchecked") + static StandardTableDefinition fromPb(Table tablePb) { + return new Builder(tablePb).build(); + } +} diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Table.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Table.java index b4cc1df1d997..3f902d2ff242 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Table.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Table.java @@ -16,12 +16,13 @@ package com.google.gcloud.bigquery; -import static com.google.common.base.Preconditions.checkArgument; import static com.google.common.base.Preconditions.checkNotNull; import com.google.common.collect.ImmutableList; import com.google.gcloud.Page; +import java.io.IOException; +import java.io.ObjectInputStream; import java.util.List; import java.util.Objects; @@ -30,62 +31,106 @@ * *

Objects of this class are immutable. Operations that modify the table like {@link #update} * return a new object. To get a {@code Table} object with the most recent information use - * {@link #reload}. + * {@link #reload}. {@code Table} adds a layer of service-related functionality over + * {@link TableInfo}. *

*/ -public final class Table { +public final class Table extends TableInfo { - private final BigQuery bigquery; - private final BaseTableInfo info; + private static final long serialVersionUID = 5744556727066570096L; - /** - * Constructs a {@code Table} object for the provided {@code TableInfo}. The BigQuery service - * is used to issue requests. - * - * @param bigquery the BigQuery service used for issuing requests - * @param info table's info - */ - public Table(BigQuery bigquery, BaseTableInfo info) { - this.bigquery = checkNotNull(bigquery); - this.info = checkNotNull(info); - } + private final BigQueryOptions options; + private transient BigQuery bigquery; /** - * Creates a {@code Table} object for the provided dataset and table's user-defined ids. Performs - * an RPC call to get the latest table information. - * - * @param bigquery the BigQuery service used for issuing requests - * @param dataset the dataset's user-defined id - * @param table the table's user-defined id - * @param options table options - * @return the {@code Table} object or {@code null} if not found - * @throws BigQueryException upon failure + * A builder for {@code Table} objects. */ - public static Table get(BigQuery bigquery, String dataset, String table, - BigQuery.TableOption... options) { - return get(bigquery, TableId.of(dataset, table), options); - } + public static class Builder extends TableInfo.Builder { - /** - * Creates a {@code Table} object for the provided table identity. Performs an RPC call to get the - * latest table information. - * - * @param bigquery the BigQuery service used for issuing requests - * @param table the table's identity - * @param options table options - * @return the {@code Table} object or {@code null} if not found - * @throws BigQueryException upon failure - */ - public static Table get(BigQuery bigquery, TableId table, BigQuery.TableOption... options) { - BaseTableInfo info = bigquery.getTable(table, options); - return info != null ? new Table(bigquery, info) : null; + private final BigQuery bigquery; + private final TableInfo.BuilderImpl infoBuilder; + + Builder(BigQuery bigquery, TableId tableId, TableDefinition defintion) { + this.bigquery = bigquery; + this.infoBuilder = new TableInfo.BuilderImpl(); + this.infoBuilder.tableId(tableId).definition(defintion); + } + + Builder(Table table) { + this.bigquery = table.bigquery; + this.infoBuilder = new TableInfo.BuilderImpl(table); + } + + @Override + Builder creationTime(Long creationTime) { + infoBuilder.creationTime(creationTime); + return this; + } + + @Override + public Builder description(String description) { + infoBuilder.description(description); + return this; + } + + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; + } + + @Override + public Builder expirationTime(Long expirationTime) { + infoBuilder.expirationTime(expirationTime); + return this; + } + + @Override + public Builder friendlyName(String friendlyName) { + infoBuilder.friendlyName(friendlyName); + return this; + } + + @Override + Builder id(String id) { + infoBuilder.id(id); + return this; + } + + @Override + Builder lastModifiedTime(Long lastModifiedTime) { + infoBuilder.lastModifiedTime(lastModifiedTime); + return this; + } + + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + public Builder tableId(TableId tableId) { + infoBuilder.tableId(tableId); + return this; + } + + @Override + public Builder definition(TableDefinition definition) { + infoBuilder.definition(definition); + return this; + } + + @Override + public Table build() { + return new Table(bigquery, infoBuilder); + } } - /** - * Returns the table's information. - */ - public BaseTableInfo info() { - return info; + Table(BigQuery bigquery, TableInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.bigquery = checkNotNull(bigquery); + this.options = bigquery.options(); } /** @@ -95,7 +140,7 @@ public BaseTableInfo info() { * @throws BigQueryException upon failure */ public boolean exists() { - return bigquery.getTable(info.tableId(), BigQuery.TableOption.fields()) != null; + return bigquery.getTable(tableId(), BigQuery.TableOption.fields()) != null; } /** @@ -106,25 +151,19 @@ public boolean exists() { * @throws BigQueryException upon failure */ public Table reload(BigQuery.TableOption... options) { - return Table.get(bigquery, info.tableId(), options); + return bigquery.getTable(tableId(), options); } /** - * Updates the table's information. Dataset's and table's user-defined ids cannot be changed. A - * new {@code Table} object is returned. + * Updates the table's information with this table's information. Dataset's and table's + * user-defined ids cannot be changed. A new {@code Table} object is returned. * - * @param tableInfo new table's information. Dataset's and table's user-defined ids must match the - * ones of the current table * @param options dataset options * @return a {@code Table} object with updated information * @throws BigQueryException upon failure */ - public Table update(BaseTableInfo tableInfo, BigQuery.TableOption... options) { - checkArgument(Objects.equals(tableInfo.tableId().dataset(), - info.tableId().dataset()), "Dataset's user-defined ids must match"); - checkArgument(Objects.equals(tableInfo.tableId().table(), - info.tableId().table()), "Table's user-defined ids must match"); - return new Table(bigquery, bigquery.update(tableInfo, options)); + public Table update(BigQuery.TableOption... options) { + return bigquery.update(this, options); } /** @@ -134,7 +173,7 @@ public Table update(BaseTableInfo tableInfo, BigQuery.TableOption... options) { * @throws BigQueryException upon failure */ public boolean delete() { - return bigquery.delete(info.tableId()); + return bigquery.delete(tableId()); } /** @@ -143,8 +182,9 @@ public boolean delete() { * @param rows rows to be inserted * @throws BigQueryException upon failure */ - InsertAllResponse insert(Iterable rows) throws BigQueryException { - return bigquery.insertAll(InsertAllRequest.of(info.tableId(), rows)); + public InsertAllResponse insert(Iterable rows) + throws BigQueryException { + return bigquery.insertAll(InsertAllRequest.of(tableId(), rows)); } /** @@ -158,9 +198,9 @@ InsertAllResponse insert(Iterable rows) throws Big * to be invalid * @throws BigQueryException upon failure */ - InsertAllResponse insert(Iterable rows, boolean skipInvalidRows, - boolean ignoreUnknownValues) throws BigQueryException { - InsertAllRequest request = InsertAllRequest.builder(info.tableId(), rows) + public InsertAllResponse insert(Iterable rows, + boolean skipInvalidRows, boolean ignoreUnknownValues) throws BigQueryException { + InsertAllRequest request = InsertAllRequest.builder(tableId(), rows) .skipInvalidRows(skipInvalidRows) .ignoreUnknownValues(ignoreUnknownValues) .build(); @@ -173,8 +213,9 @@ InsertAllResponse insert(Iterable rows, boolean sk * @param options table data list options * @throws BigQueryException upon failure */ - Page> list(BigQuery.TableDataListOption... options) throws BigQueryException { - return bigquery.listTableData(info.tableId(), options); + public Page> list(BigQuery.TableDataListOption... options) + throws BigQueryException { + return bigquery.listTableData(tableId(), options); } /** @@ -186,7 +227,7 @@ Page> list(BigQuery.TableDataListOption... options) throws BigQ * @param options job options * @throws BigQueryException upon failure */ - Job copy(String destinationDataset, String destinationTable, BigQuery.JobOption... options) + public Job copy(String destinationDataset, String destinationTable, BigQuery.JobOption... options) throws BigQueryException { return copy(TableId.of(destinationDataset, destinationTable), options); } @@ -199,9 +240,10 @@ Job copy(String destinationDataset, String destinationTable, BigQuery.JobOption. * @param options job options * @throws BigQueryException upon failure */ - Job copy(TableId destinationTable, BigQuery.JobOption... options) throws BigQueryException { - JobInfo job = bigquery.create(CopyJobInfo.of(destinationTable, info.tableId()), options); - return new Job(bigquery, job); + public Job copy(TableId destinationTable, BigQuery.JobOption... options) + throws BigQueryException { + CopyJobConfiguration configuration = CopyJobConfiguration.of(destinationTable, tableId()); + return bigquery.create(JobInfo.of(configuration), options); } /** @@ -214,7 +256,7 @@ Job copy(TableId destinationTable, BigQuery.JobOption... options) throws BigQuer * @param options job options * @throws BigQueryException upon failure */ - Job extract(String format, String destinationUri, BigQuery.JobOption... options) + public Job extract(String format, String destinationUri, BigQuery.JobOption... options) throws BigQueryException { return extract(format, ImmutableList.of(destinationUri), options); } @@ -229,10 +271,11 @@ Job extract(String format, String destinationUri, BigQuery.JobOption... options) * @param options job options * @throws BigQueryException upon failure */ - Job extract(String format, List destinationUris, BigQuery.JobOption... options) + public Job extract(String format, List destinationUris, BigQuery.JobOption... options) throws BigQueryException { - return new Job(bigquery, - bigquery.create(ExtractJobInfo.of(info.tableId(), format, destinationUris), options)); + ExtractJobConfiguration extractConfiguration = + ExtractJobConfiguration.of(tableId(), destinationUris, format); + return bigquery.create(JobInfo.of(extractConfiguration), options); } /** @@ -245,7 +288,7 @@ Job extract(String format, List destinationUris, BigQuery.JobOption... o * @param options job options * @throws BigQueryException upon failure */ - Job load(FormatOptions format, String sourceUri, BigQuery.JobOption... options) + public Job load(FormatOptions format, String sourceUri, BigQuery.JobOption... options) throws BigQueryException { return load(format, ImmutableList.of(sourceUri), options); } @@ -260,10 +303,10 @@ Job load(FormatOptions format, String sourceUri, BigQuery.JobOption... options) * @param options job options * @throws BigQueryException upon failure */ - Job load(FormatOptions format, List sourceUris, BigQuery.JobOption... options) + public Job load(FormatOptions format, List sourceUris, BigQuery.JobOption... options) throws BigQueryException { - LoadConfiguration configuration = LoadConfiguration.of(info.tableId(), format); - return new Job(bigquery, bigquery.create(LoadJobInfo.of(configuration, sourceUris), options)); + LoadJobConfiguration loadConfig = LoadJobConfiguration.of(tableId(), sourceUris, format); + return bigquery.create(JobInfo.of(loadConfig), options); } /** @@ -272,4 +315,30 @@ Job load(FormatOptions format, List sourceUris, BigQuery.JobOption... op public BigQuery bigquery() { return bigquery; } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof Table + && Objects.equals(toPb(), ((Table) obj).toPb()) + && Objects.equals(options, ((Table) obj).options); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); + } + + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.bigquery = options.service(); + } + + static Table fromPb(BigQuery bigquery, com.google.api.services.bigquery.model.Table tablePb) { + return new Table(bigquery, new TableInfo.BuilderImpl(tablePb)); + } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java index c4cee5a9a303..9c6a950ca27f 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java @@ -24,18 +24,18 @@ import com.google.gcloud.RetryHelper; import com.google.gcloud.WriteChannel; -import java.util.Arrays; - /** * WriteChannel implementation to stream data into a BigQuery table. */ -class TableDataWriteChannel extends BaseWriteChannel { +class TableDataWriteChannel extends BaseWriteChannel { - TableDataWriteChannel(BigQueryOptions options, LoadConfiguration loadConfiguration) { - this(options, loadConfiguration, options.rpc().open(loadConfiguration.toPb())); + TableDataWriteChannel(BigQueryOptions options, + WriteChannelConfiguration writeChannelConfiguration) { + this(options, writeChannelConfiguration, options.rpc().open(writeChannelConfiguration.toPb())); } - TableDataWriteChannel(BigQueryOptions options, LoadConfiguration config, String uploadId) { + TableDataWriteChannel(BigQueryOptions options, WriteChannelConfiguration config, + String uploadId) { super(options, config, uploadId); } @@ -53,11 +53,13 @@ public void run() { } } + @Override protected StateImpl.Builder stateBuilder() { return StateImpl.builder(options(), entity(), uploadId()); } - static class StateImpl extends BaseWriteChannel.BaseState { + static class StateImpl + extends BaseWriteChannel.BaseState { private static final long serialVersionUID = -787362105981823738L; @@ -66,9 +68,10 @@ static class StateImpl extends BaseWriteChannel.BaseState { + extends BaseWriteChannel.BaseState.Builder { - private Builder(BigQueryOptions options, LoadConfiguration configuration, String uploadId) { + private Builder(BigQueryOptions options, WriteChannelConfiguration configuration, + String uploadId) { super(options, configuration, uploadId); } @@ -77,7 +80,8 @@ public RestorableState build() { } } - static Builder builder(BigQueryOptions options, LoadConfiguration config, String uploadId) { + static Builder builder(BigQueryOptions options, WriteChannelConfiguration config, + String uploadId) { return new Builder(options, config, uploadId); } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDefinition.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDefinition.java new file mode 100644 index 000000000000..26e7bcc76f55 --- /dev/null +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDefinition.java @@ -0,0 +1,182 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.api.services.bigquery.model.Table; +import com.google.common.base.MoreObjects; + +import java.io.Serializable; +import java.util.Objects; + +/** + * Base class for a Google BigQuery table definition. + */ +public abstract class TableDefinition implements Serializable { + + private static final long serialVersionUID = -374760330662959529L; + + private final Type type; + private final Schema schema; + + /** + * The table type. + */ + public enum Type { + /** + * A normal BigQuery table. Instances of {@code TableDefinition} for this type are implemented + * by {@link StandardTableDefinition}. + */ + TABLE, + + /** + * A virtual table defined by a SQL query. Instances of {@code TableDefinition} for this type + * are implemented by {@link ViewDefinition}. + * + * @see Views + */ + VIEW, + + /** + * A BigQuery table backed by external data. Instances of {@code TableDefinition} for this type + * are implemented by {@link ExternalTableDefinition}. + * + * @see Federated Data + * Sources + */ + EXTERNAL + } + + /** + * Base builder for table definitions. + * + * @param the table definition class + * @param the table definition builder + */ + public abstract static class Builder> { + + private Type type; + private Schema schema; + + Builder(Type type) { + this.type = type; + } + + Builder(TableDefinition tableDefinition) { + this.type = tableDefinition.type; + this.schema = tableDefinition.schema; + } + + Builder(Table tablePb) { + this.type = Type.valueOf(tablePb.getType()); + if (tablePb.getSchema() != null) { + this.schema(Schema.fromPb(tablePb.getSchema())); + } + } + + @SuppressWarnings("unchecked") + B self() { + return (B) this; + } + + B type(Type type) { + this.type = type; + return self(); + } + + /** + * Sets the table schema. + */ + public B schema(Schema schema) { + this.schema = checkNotNull(schema); + return self(); + } + + /** + * Creates an object. + */ + public abstract T build(); + } + + TableDefinition(Builder builder) { + this.type = builder.type; + this.schema = builder.schema; + } + + /** + * Returns the table's type. If this table is simple table the method returns {@link Type#TABLE}. + * If this table is an external table this method returns {@link Type#EXTERNAL}. If this table is + * a view table this method returns {@link Type#VIEW}. + */ + public Type type() { + return type; + } + + /** + * Returns the table's schema. + */ + public Schema schema() { + return schema; + } + + /** + * Returns a builder for the object. + */ + public abstract Builder toBuilder(); + + MoreObjects.ToStringHelper toStringHelper() { + return MoreObjects.toStringHelper(this).add("type", type).add("schema", schema); + } + + @Override + public String toString() { + return toStringHelper().toString(); + } + + final int baseHashCode() { + return Objects.hash(type); + } + + final boolean baseEquals(TableDefinition tableDefinition) { + return Objects.equals(toPb(), tableDefinition.toPb()); + } + + Table toPb() { + Table tablePb = new Table(); + if (schema != null) { + tablePb.setSchema(schema.toPb()); + } + tablePb.setType(type.name()); + return tablePb; + } + + @SuppressWarnings("unchecked") + static T fromPb(Table tablePb) { + switch (Type.valueOf(tablePb.getType())) { + case TABLE: + return (T) StandardTableDefinition.fromPb(tablePb); + case VIEW: + return (T) ViewDefinition.fromPb(tablePb); + case EXTERNAL: + return (T) ExternalTableDefinition.fromPb(tablePb); + default: + // never reached + throw new IllegalArgumentException("Format " + tablePb.getType() + " is not supported"); + } + } +} diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableId.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableId.java index 7a4e0bbb38b4..20ed53cc1a5d 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableId.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableId.java @@ -105,6 +105,10 @@ public String toString() { return toPb().toString(); } + TableId setProjectId(String projectId) { + return project() != null ? this : TableId.of(projectId, dataset(), table()); + } + TableReference toPb() { return new TableReference().setProjectId(project).setDatasetId(dataset).setTableId(table); } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableInfo.java index 54258abc6ddd..de331350e978 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableInfo.java @@ -16,225 +16,375 @@ package com.google.gcloud.bigquery; -import com.google.api.services.bigquery.model.Streamingbuffer; +import static com.google.common.base.MoreObjects.firstNonNull; +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.api.client.util.Data; import com.google.api.services.bigquery.model.Table; +import com.google.common.base.Function; import com.google.common.base.MoreObjects; -import com.google.common.base.MoreObjects.ToStringHelper; import java.io.Serializable; import java.math.BigInteger; import java.util.Objects; /** - * A Google BigQuery Table information. A BigQuery table is a standard, two-dimensional table with - * individual records organized in rows, and a data type assigned to each column (also called a - * field). Individual fields within a record may contain nested and repeated children fields. Every - * table is described by a schema that describes field names, types, and other information. + * Google BigQuery table information. Use {@link StandardTableDefinition} to create simple BigQuery + * table. Use {@link ViewDefinition} to create a BigQuery view. Use {@link ExternalTableDefinition} + * to create a BigQuery a table backed by external data. * * @see Managing Tables */ -public class TableInfo extends BaseTableInfo { - - private static final long serialVersionUID = -5910575573063546949L; - - private final String location; - private final StreamingBuffer streamingBuffer; +public class TableInfo implements Serializable { + + static final Function FROM_PB_FUNCTION = + new Function() { + @Override + public TableInfo apply(Table pb) { + return TableInfo.fromPb(pb); + } + }; + static final Function TO_PB_FUNCTION = + new Function() { + @Override + public Table apply(TableInfo tableInfo) { + return tableInfo.toPb(); + } + }; + + private static final long serialVersionUID = -7679032506430816205L; + + private final String etag; + private final String id; + private final String selfLink; + private final TableId tableId; + private final String friendlyName; + private final String description; + private final Long creationTime; + private final Long expirationTime; + private final Long lastModifiedTime; + private final TableDefinition definition; /** - * Google BigQuery Table's Streaming Buffer information. This class contains information on a - * table's streaming buffer as the estimated size in number of rows/bytes. + * A builder for {@code TableInfo} objects. */ - public static class StreamingBuffer implements Serializable { + public abstract static class Builder { - private static final long serialVersionUID = -6713971364725267597L; - private final long estimatedRows; - private final long estimatedBytes; - private final long oldestEntryTime; + abstract Builder creationTime(Long creationTime); - StreamingBuffer(long estimatedRows, long estimatedBytes, long oldestEntryTime) { - this.estimatedRows = estimatedRows; - this.estimatedBytes = estimatedBytes; - this.oldestEntryTime = oldestEntryTime; - } + /** + * Sets a user-friendly description for the table. + */ + public abstract Builder description(String description); + + abstract Builder etag(String etag); /** - * Returns a lower-bound estimate of the number of rows currently in the streaming buffer. + * Sets the time when this table expires, in milliseconds since the epoch. If not present, the + * table will persist indefinitely. Expired tables will be deleted and their storage reclaimed. */ - public long estimatedRows() { - return estimatedRows; - } + public abstract Builder expirationTime(Long expirationTime); /** - * Returns a lower-bound estimate of the number of bytes currently in the streaming buffer. + * Sets a user-friendly name for the table. */ - public long estimatedBytes() { - return estimatedBytes; - } + public abstract Builder friendlyName(String friendlyName); + + abstract Builder id(String id); + + abstract Builder lastModifiedTime(Long lastModifiedTime); + + abstract Builder selfLink(String selfLink); + + /** + * Sets the table identity. + */ + public abstract Builder tableId(TableId tableId); + + /** + * Sets the table definition. Use {@link StandardTableDefinition} to create simple BigQuery + * table. Use {@link ViewDefinition} to create a BigQuery view. Use + * {@link ExternalTableDefinition} to create a BigQuery a table backed by external data. + */ + public abstract Builder definition(TableDefinition definition); /** - * Returns the timestamp of the oldest entry in the streaming buffer, in milliseconds since - * epoch. + * Creates a {@code TableInfo} object. */ - public long oldestEntryTime() { - return oldestEntryTime; + public abstract TableInfo build(); + } + + static class BuilderImpl extends Builder { + + private String etag; + private String id; + private String selfLink; + private TableId tableId; + private String friendlyName; + private String description; + private Long creationTime; + private Long expirationTime; + private Long lastModifiedTime; + private TableDefinition definition; + + BuilderImpl() {} + + BuilderImpl(TableInfo tableInfo) { + this.etag = tableInfo.etag; + this.id = tableInfo.id; + this.selfLink = tableInfo.selfLink; + this.tableId = tableInfo.tableId; + this.friendlyName = tableInfo.friendlyName; + this.description = tableInfo.description; + this.creationTime = tableInfo.creationTime; + this.expirationTime = tableInfo.expirationTime; + this.lastModifiedTime = tableInfo.lastModifiedTime; + this.definition = tableInfo.definition; } - @Override - public String toString() { - return MoreObjects.toStringHelper(this) - .add("estimatedRows", estimatedRows) - .add("estimatedBytes", estimatedBytes) - .add("oldestEntryTime", oldestEntryTime) - .toString(); + BuilderImpl(Table tablePb) { + this.tableId = TableId.fromPb(tablePb.getTableReference()); + if (tablePb.getLastModifiedTime() != null) { + this.lastModifiedTime(tablePb.getLastModifiedTime().longValue()); + } + this.description = tablePb.getDescription(); + this.expirationTime = tablePb.getExpirationTime(); + this.friendlyName = tablePb.getFriendlyName(); + this.creationTime = tablePb.getCreationTime(); + this.etag = tablePb.getEtag(); + this.id = tablePb.getId(); + this.selfLink = tablePb.getSelfLink(); + this.definition = TableDefinition.fromPb(tablePb); } @Override - public int hashCode() { - return Objects.hash(estimatedRows, estimatedBytes, oldestEntryTime); + Builder creationTime(Long creationTime) { + this.creationTime = creationTime; + return this; } @Override - public boolean equals(Object obj) { - return obj instanceof StreamingBuffer - && Objects.equals(toPb(), ((StreamingBuffer) obj).toPb()); + public Builder description(String description) { + this.description = firstNonNull(description, Data.nullOf(String.class)); + return this; } - Streamingbuffer toPb() { - return new Streamingbuffer() - .setEstimatedBytes(BigInteger.valueOf(estimatedBytes)) - .setEstimatedRows(BigInteger.valueOf(estimatedRows)) - .setOldestEntryTime(BigInteger.valueOf(oldestEntryTime)); + @Override + Builder etag(String etag) { + this.etag = etag; + return this; } - static StreamingBuffer fromPb(Streamingbuffer streamingBufferPb) { - return new StreamingBuffer(streamingBufferPb.getEstimatedRows().longValue(), - streamingBufferPb.getEstimatedBytes().longValue(), - streamingBufferPb.getOldestEntryTime().longValue()); + @Override + public Builder expirationTime(Long expirationTime) { + this.expirationTime = firstNonNull(expirationTime, Data.nullOf(Long.class)); + return this; } - } - public static final class Builder extends BaseTableInfo.Builder { - - private String location; - private StreamingBuffer streamingBuffer; + @Override + public Builder friendlyName(String friendlyName) { + this.friendlyName = firstNonNull(friendlyName, Data.nullOf(String.class)); + return this; + } - private Builder() {} + @Override + Builder id(String id) { + this.id = id; + return this; + } - private Builder(TableInfo tableInfo) { - super(tableInfo); - this.location = tableInfo.location; - this.streamingBuffer = tableInfo.streamingBuffer; + @Override + Builder lastModifiedTime(Long lastModifiedTime) { + this.lastModifiedTime = lastModifiedTime; + return this; } - protected Builder(Table tablePb) { - super(tablePb); - this.location = tablePb.getLocation(); - if (tablePb.getStreamingBuffer() != null) { - this.streamingBuffer = StreamingBuffer.fromPb(tablePb.getStreamingBuffer()); - } + @Override + Builder selfLink(String selfLink) { + this.selfLink = selfLink; + return this; } - Builder location(String location) { - this.location = location; - return self(); + @Override + public Builder tableId(TableId tableId) { + this.tableId = checkNotNull(tableId); + return this; } - Builder streamingBuffer(StreamingBuffer streamingBuffer) { - this.streamingBuffer = streamingBuffer; - return self(); + @Override + public Builder definition(TableDefinition definition) { + this.definition = checkNotNull(definition); + return this; } - /** - * Creates a {@code TableInfo} object. - */ @Override public TableInfo build() { return new TableInfo(this); } } - private TableInfo(Builder builder) { - super(builder); - this.location = builder.location; - this.streamingBuffer = builder.streamingBuffer; + TableInfo(BuilderImpl builder) { + this.tableId = checkNotNull(builder.tableId); + this.etag = builder.etag; + this.id = builder.id; + this.selfLink = builder.selfLink; + this.friendlyName = builder.friendlyName; + this.description = builder.description; + this.creationTime = builder.creationTime; + this.expirationTime = builder.expirationTime; + this.lastModifiedTime = builder.lastModifiedTime; + this.definition = builder.definition; } /** - * Returns the geographic location where the table should reside. This value is inherited from the - * dataset. - * - * @see - * Dataset Location + * Returns the hash of the table resource. */ - public String location() { - return location; + public String etag() { + return etag; } /** - * Returns information on the table's streaming buffer if any exists. Returns {@code null} if no - * streaming buffer exists. + * Returns an opaque id for the table. */ - public StreamingBuffer streamingBuffer() { - return streamingBuffer; + public String id() { + return id; } /** - * Returns a builder for a BigQuery Table. - * - * @param tableId table id - * @param schema the schema of the table + * Returns an URL that can be used to access the resource again. The returned URL can be used for + * get or update requests. */ - public static Builder builder(TableId tableId, Schema schema) { - return new Builder().tableId(tableId).type(Type.TABLE).schema(schema); + public String selfLink() { + return selfLink; } /** - * Creates BigQuery table given its type. - * - * @param tableId table id - * @param schema the schema of the table + * Returns the table identity. */ - public static TableInfo of(TableId tableId, Schema schema) { - return builder(tableId, schema).build(); + public TableId tableId() { + return tableId; } /** - * Returns a builder for the {@code TableInfo} object. + * Returns a user-friendly name for the table. */ - @Override - public Builder toBuilder() { - return new Builder(this); + public String friendlyName() { + return Data.isNull(friendlyName) ? null : friendlyName; } - @Override - ToStringHelper toStringHelper() { - return super.toStringHelper() - .add("location", location) - .add("streamingBuffer", streamingBuffer); + /** + * Returns a user-friendly description for the table. + */ + public String description() { + return Data.isNull(description) ? null : description; + } + + /** + * Returns the time when this table was created, in milliseconds since the epoch. + */ + public Long creationTime() { + return creationTime; + } + + /** + * Returns the time when this table expires, in milliseconds since the epoch. If not present, the + * table will persist indefinitely. Expired tables will be deleted and their storage reclaimed. + */ + public Long expirationTime() { + return Data.isNull(expirationTime) ? null : expirationTime; + } + + /** + * Returns the time when this table was last modified, in milliseconds since the epoch. + */ + public Long lastModifiedTime() { + return lastModifiedTime; + } + + /** + * Returns the table definition. + */ + @SuppressWarnings("unchecked") + public T definition() { + return (T) definition; + } + + /** + * Returns a builder for the table object. + */ + public Builder toBuilder() { + return new BuilderImpl(this); } @Override - public boolean equals(Object obj) { - return obj instanceof TableInfo && Objects.equals(toPb(), ((TableInfo) obj).toPb()); + public String toString() { + return MoreObjects.toStringHelper(this) + .add("tableId", tableId) + .add("etag", etag) + .add("id", id) + .add("selfLink", selfLink) + .add("friendlyName", friendlyName) + .add("description", description) + .add("expirationTime", expirationTime) + .add("creationTime", creationTime) + .add("lastModifiedTime", lastModifiedTime) + .add("definition", definition) + .toString(); } @Override public int hashCode() { - return Objects.hash(super.hashCode(), location, streamingBuffer); + return Objects.hash(tableId); } @Override + public boolean equals(Object obj) { + return obj != null + && obj.getClass().equals(TableInfo.class) + && Objects.equals(toPb(), ((TableInfo) obj).toPb()); + } + + /** + * Returns a builder for a {@code TableInfo} object given table identity and definition. Use + * {@link StandardTableDefinition} to create simple BigQuery table. Use {@link ViewDefinition} to + * create a BigQuery view. Use {@link ExternalTableDefinition} to create a BigQuery a table backed + * by external data. + */ + public static Builder builder(TableId tableId, TableDefinition definition) { + return new BuilderImpl().tableId(tableId).definition(definition); + } + + /** + * Returns a {@code TableInfo} object given table identity and definition. Use + * {@link StandardTableDefinition} to create simple BigQuery table. Use {@link ViewDefinition} to + * create a BigQuery view. Use {@link ExternalTableDefinition} to create a BigQuery a table backed + * by external data. + */ + public static TableInfo of(TableId tableId, TableDefinition definition) { + return builder(tableId, definition).build(); + } + + TableInfo setProjectId(String projectId) { + return toBuilder().tableId(tableId().setProjectId(projectId)).build(); + } + Table toPb() { - Table tablePb = super.toPb(); - tablePb.setLocation(location); - if (streamingBuffer != null) { - tablePb.setStreamingBuffer(streamingBuffer.toPb()); + Table tablePb = definition.toPb(); + tablePb.setTableReference(tableId.toPb()); + if (lastModifiedTime != null) { + tablePb.setLastModifiedTime(BigInteger.valueOf(lastModifiedTime)); } + tablePb.setCreationTime(creationTime); + tablePb.setDescription(description); + tablePb.setEtag(etag); + tablePb.setExpirationTime(expirationTime); + tablePb.setFriendlyName(friendlyName); + tablePb.setId(id); + tablePb.setSelfLink(selfLink); return tablePb; } - @SuppressWarnings("unchecked") static TableInfo fromPb(Table tablePb) { - return new Builder(tablePb).build(); + return new BuilderImpl(tablePb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewDefinition.java similarity index 63% rename from gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewInfo.java rename to gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewDefinition.java index 9af9e9d7a08e..796dd411b4a1 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewDefinition.java @@ -1,5 +1,5 @@ /* - * Copyright 2015 Google Inc. All Rights Reserved. + * Copyright 2016 Google Inc. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -19,7 +19,6 @@ import static com.google.common.base.Preconditions.checkNotNull; import com.google.api.services.bigquery.model.Table; -import com.google.api.services.bigquery.model.ViewDefinition; import com.google.common.base.MoreObjects.ToStringHelper; import com.google.common.collect.ImmutableList; import com.google.common.collect.Lists; @@ -28,35 +27,37 @@ import java.util.Objects; /** - * Google BigQuery View Table information. BigQuery's views are logical views, not materialized + * Google BigQuery view table definition. BigQuery's views are logical views, not materialized * views, which means that the query that defines the view is re-executed every time the view is * queried. * * @see Views */ -public class ViewInfo extends BaseTableInfo { +public final class ViewDefinition extends TableDefinition { - private static final long serialVersionUID = 7567772157817454901L; + private static final long serialVersionUID = -8789311196910794545L; private final String query; private final List userDefinedFunctions; - public static final class Builder extends BaseTableInfo.Builder { + public static final class Builder extends TableDefinition.Builder { private String query; private List userDefinedFunctions; - private Builder() {} + private Builder() { + super(Type.VIEW); + } - private Builder(ViewInfo viewInfo) { - super(viewInfo); - this.query = viewInfo.query; - this.userDefinedFunctions = viewInfo.userDefinedFunctions; + private Builder(ViewDefinition viewDefinition) { + super(viewDefinition); + this.query = viewDefinition.query; + this.userDefinedFunctions = viewDefinition.userDefinedFunctions; } - protected Builder(Table tablePb) { + private Builder(Table tablePb) { super(tablePb); - ViewDefinition viewPb = tablePb.getView(); + com.google.api.services.bigquery.model.ViewDefinition viewPb = tablePb.getView(); if (viewPb != null) { this.query = viewPb.getQuery(); if (viewPb.getUserDefinedFunctionResources() != null) { @@ -97,15 +98,15 @@ public Builder userDefinedFunctions(UserDefinedFunction... userDefinedFunctions) } /** - * Creates a {@code ViewInfo} object. + * Creates a {@code ViewDefinition} object. */ @Override - public ViewInfo build() { - return new ViewInfo(this); + public ViewDefinition build() { + return new ViewDefinition(this); } } - private ViewInfo(Builder builder) { + private ViewDefinition(Builder builder) { super(builder); this.query = builder.query; this.userDefinedFunctions = builder.userDefinedFunctions; @@ -146,18 +147,19 @@ ToStringHelper toStringHelper() { @Override public boolean equals(Object obj) { - return obj instanceof ViewInfo && Objects.equals(toPb(), ((ViewInfo) obj).toPb()); + return obj instanceof ViewDefinition && baseEquals((ViewDefinition) obj); } @Override public int hashCode() { - return Objects.hash(super.hashCode(), query, userDefinedFunctions); + return Objects.hash(baseHashCode(), query, userDefinedFunctions); } @Override Table toPb() { Table tablePb = super.toPb(); - ViewDefinition viewDefinition = new ViewDefinition().setQuery(query); + com.google.api.services.bigquery.model.ViewDefinition viewDefinition = + new com.google.api.services.bigquery.model.ViewDefinition().setQuery(query); if (userDefinedFunctions != null) { viewDefinition.setUserDefinedFunctionResources(Lists.transform(userDefinedFunctions, UserDefinedFunction.TO_PB_FUNCTION)); @@ -167,79 +169,65 @@ Table toPb() { } /** - * Returns a builder for a BigQuery View Table. + * Returns a builder for a BigQuery view definition. * - * @param tableId table id - * @param query the query used to generate the table + * @param query the query used to generate the view */ - public static Builder builder(TableId tableId, String query) { - return new Builder().tableId(tableId).type(Type.VIEW).query(query); + public static Builder builder(String query) { + return new Builder().query(query); } /** - * Returns a builder for a BigQuery View Table. + * Returns a builder for a BigQuery view definition. * - * @param table table id * @param query the query used to generate the table * @param functions user-defined functions that can be used by the query */ - public static Builder builder(TableId table, String query, List functions) { - return new Builder() - .tableId(table) - .type(Type.VIEW) - .userDefinedFunctions(functions) - .query(query); + public static Builder builder(String query, List functions) { + return new Builder().type(Type.VIEW).userDefinedFunctions(functions).query(query); } /** - * Returns a builder for a BigQuery View Table. + * Returns a builder for a BigQuery view definition. * - * @param table table id * @param query the query used to generate the table * @param functions user-defined functions that can be used by the query */ - public static Builder builder(TableId table, String query, UserDefinedFunction... functions) { - return new Builder() - .tableId(table) - .type(Type.VIEW) - .userDefinedFunctions(functions) - .query(query); + public static Builder builder(String query, UserDefinedFunction... functions) { + return new Builder().type(Type.VIEW).userDefinedFunctions(functions).query(query); } /** - * Creates a BigQuery View given table identity and query. + * Creates a BigQuery view definition given the query used to generate the table. * - * @param tableId table id * @param query the query used to generate the table */ - public static ViewInfo of(TableId tableId, String query) { - return builder(tableId, query).build(); + public static ViewDefinition of(String query) { + return builder(query).build(); } /** - * Creates a BigQuery View given table identity, a query and some user-defined functions. + * Creates a BigQuery view definition given a query and some user-defined functions. * - * @param table table id * @param query the query used to generate the table * @param functions user-defined functions that can be used by the query */ - public static ViewInfo of(TableId table, String query, List functions) { - return builder(table, query, functions).build(); + public static ViewDefinition of(String query, List functions) { + return builder(query, functions).build(); } /** - * Creates a BigQuery View given table identity, a query and some user-defined functions. + * Creates a BigQuery view definition given a query and some user-defined functions. * - * @param table table id * @param query the query used to generate the table * @param functions user-defined functions that can be used by the query */ - public static ViewInfo of(TableId table, String query, UserDefinedFunction... functions) { - return builder(table, query, functions).build(); + public static ViewDefinition of(String query, UserDefinedFunction... functions) { + return builder(query, functions).build(); } @SuppressWarnings("unchecked") - static ViewInfo fromPb(Table tablePb) { + static ViewDefinition fromPb(Table tablePb) { return new Builder(tablePb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/WriteChannelConfiguration.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/WriteChannelConfiguration.java new file mode 100644 index 000000000000..6cc44ce7d5d6 --- /dev/null +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/WriteChannelConfiguration.java @@ -0,0 +1,321 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.api.services.bigquery.model.JobConfigurationLoad; +import com.google.common.base.MoreObjects; +import com.google.common.collect.ImmutableList; +import com.google.gcloud.bigquery.JobInfo.CreateDisposition; +import com.google.gcloud.bigquery.JobInfo.WriteDisposition; + +import java.io.Serializable; +import java.util.List; +import java.util.Objects; + +/** + * Google BigQuery Configuration for a load operation. A load configuration can be used to load data + * into a table with a {@link com.google.gcloud.WriteChannel} + * ({@link BigQuery#writer(WriteChannelConfiguration)}). + */ +public class WriteChannelConfiguration implements LoadConfiguration, Serializable { + + private static final long serialVersionUID = 470267591917413578L; + + private final TableId destinationTable; + private final CreateDisposition createDisposition; + private final WriteDisposition writeDisposition; + private final FormatOptions formatOptions; + private final Integer maxBadRecords; + private final Schema schema; + private final Boolean ignoreUnknownValues; + private final List projectionFields; + + public static final class Builder implements LoadConfiguration.Builder { + + private TableId destinationTable; + private CreateDisposition createDisposition; + private WriteDisposition writeDisposition; + private FormatOptions formatOptions; + private Integer maxBadRecords; + private Schema schema; + private Boolean ignoreUnknownValues; + private List projectionFields; + + private Builder() {} + + private Builder(WriteChannelConfiguration writeChannelConfiguration) { + this.destinationTable = writeChannelConfiguration.destinationTable; + this.createDisposition = writeChannelConfiguration.createDisposition; + this.writeDisposition = writeChannelConfiguration.writeDisposition; + this.formatOptions = writeChannelConfiguration.formatOptions; + this.maxBadRecords = writeChannelConfiguration.maxBadRecords; + this.schema = writeChannelConfiguration.schema; + this.ignoreUnknownValues = writeChannelConfiguration.ignoreUnknownValues; + this.projectionFields = writeChannelConfiguration.projectionFields; + } + + private Builder(com.google.api.services.bigquery.model.JobConfiguration configurationPb) { + JobConfigurationLoad loadConfigurationPb = configurationPb.getLoad(); + this.destinationTable = TableId.fromPb(loadConfigurationPb.getDestinationTable()); + if (loadConfigurationPb.getCreateDisposition() != null) { + this.createDisposition = + CreateDisposition.valueOf(loadConfigurationPb.getCreateDisposition()); + } + if (loadConfigurationPb.getWriteDisposition() != null) { + this.writeDisposition = WriteDisposition.valueOf(loadConfigurationPb.getWriteDisposition()); + } + if (loadConfigurationPb.getSourceFormat() != null) { + this.formatOptions = FormatOptions.of(loadConfigurationPb.getSourceFormat()); + } + if (loadConfigurationPb.getAllowJaggedRows() != null + || loadConfigurationPb.getAllowQuotedNewlines() != null + || loadConfigurationPb.getEncoding() != null + || loadConfigurationPb.getFieldDelimiter() != null + || loadConfigurationPb.getQuote() != null + || loadConfigurationPb.getSkipLeadingRows() != null) { + CsvOptions.Builder builder = CsvOptions.builder() + .allowJaggedRows(loadConfigurationPb.getAllowJaggedRows()) + .allowQuotedNewLines(loadConfigurationPb.getAllowQuotedNewlines()) + .encoding(loadConfigurationPb.getEncoding()) + .fieldDelimiter(loadConfigurationPb.getFieldDelimiter()) + .quote(loadConfigurationPb.getQuote()) + .skipLeadingRows(loadConfigurationPb.getSkipLeadingRows()); + this.formatOptions = builder.build(); + } + this.maxBadRecords = loadConfigurationPb.getMaxBadRecords(); + if (loadConfigurationPb.getSchema() != null) { + this.schema = Schema.fromPb(loadConfigurationPb.getSchema()); + } + this.ignoreUnknownValues = loadConfigurationPb.getIgnoreUnknownValues(); + this.projectionFields = loadConfigurationPb.getProjectionFields(); + } + + @Override + public Builder destinationTable(TableId destinationTable) { + this.destinationTable = destinationTable; + return this; + } + + @Override + public Builder createDisposition(CreateDisposition createDisposition) { + this.createDisposition = createDisposition; + return this; + } + + @Override + public Builder writeDisposition(WriteDisposition writeDisposition) { + this.writeDisposition = writeDisposition; + return this; + } + + @Override + public Builder formatOptions(FormatOptions formatOptions) { + this.formatOptions = formatOptions; + return this; + } + + @Override + public Builder maxBadRecords(Integer maxBadRecords) { + this.maxBadRecords = maxBadRecords; + return this; + } + + @Override + public Builder schema(Schema schema) { + this.schema = schema; + return this; + } + + @Override + public Builder ignoreUnknownValues(Boolean ignoreUnknownValues) { + this.ignoreUnknownValues = ignoreUnknownValues; + return this; + } + + @Override + public Builder projectionFields(List projectionFields) { + this.projectionFields = + projectionFields != null ? ImmutableList.copyOf(projectionFields) : null; + return this; + } + + @Override + public WriteChannelConfiguration build() { + return new WriteChannelConfiguration(this); + } + } + + protected WriteChannelConfiguration(Builder builder) { + this.destinationTable = checkNotNull(builder.destinationTable); + this.createDisposition = builder.createDisposition; + this.writeDisposition = builder.writeDisposition; + this.formatOptions = builder.formatOptions; + this.maxBadRecords = builder.maxBadRecords; + this.schema = builder.schema; + this.ignoreUnknownValues = builder.ignoreUnknownValues; + this.projectionFields = builder.projectionFields; + } + + @Override + public TableId destinationTable() { + return destinationTable; + } + + @Override + public CreateDisposition createDisposition() { + return this.createDisposition; + } + + @Override + public WriteDisposition writeDisposition() { + return writeDisposition; + } + + @Override + public CsvOptions csvOptions() { + return formatOptions instanceof CsvOptions ? (CsvOptions) formatOptions : null; + } + + @Override + public Integer maxBadRecords() { + return maxBadRecords; + } + + @Override + public Schema schema() { + return schema; + } + + @Override + public String format() { + return formatOptions != null ? formatOptions.type() : null; + } + + @Override + public Boolean ignoreUnknownValues() { + return ignoreUnknownValues; + } + + @Override + public List projectionFields() { + return projectionFields; + } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + MoreObjects.ToStringHelper toStringHelper() { + return MoreObjects.toStringHelper(this) + .add("destinationTable", destinationTable) + .add("createDisposition", createDisposition) + .add("writeDisposition", writeDisposition) + .add("formatOptions", formatOptions) + .add("maxBadRecords", maxBadRecords) + .add("schema", schema) + .add("ignoreUnknownValue", ignoreUnknownValues) + .add("projectionFields", projectionFields); + } + + @Override + public String toString() { + return toStringHelper().toString(); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof WriteChannelConfiguration + && Objects.equals(toPb(), ((WriteChannelConfiguration) obj).toPb()); + } + + @Override + public int hashCode() { + return Objects.hash(destinationTable, createDisposition, writeDisposition, formatOptions, + maxBadRecords, schema, ignoreUnknownValues, projectionFields); + } + + WriteChannelConfiguration setProjectId(String projectId) { + return toBuilder().destinationTable(destinationTable().setProjectId(projectId)).build(); + } + + com.google.api.services.bigquery.model.JobConfiguration toPb() { + JobConfigurationLoad loadConfigurationPb = new JobConfigurationLoad(); + loadConfigurationPb.setDestinationTable(destinationTable.toPb()); + if (createDisposition != null) { + loadConfigurationPb.setCreateDisposition(createDisposition.toString()); + } + if (writeDisposition != null) { + loadConfigurationPb.setWriteDisposition(writeDisposition.toString()); + } + if (csvOptions() != null) { + CsvOptions csvOptions = csvOptions(); + loadConfigurationPb.setFieldDelimiter(csvOptions.fieldDelimiter()) + .setAllowJaggedRows(csvOptions.allowJaggedRows()) + .setAllowQuotedNewlines(csvOptions.allowQuotedNewLines()) + .setEncoding(csvOptions.encoding()) + .setQuote(csvOptions.quote()) + .setSkipLeadingRows(csvOptions.skipLeadingRows()); + } + if (schema != null) { + loadConfigurationPb.setSchema(schema.toPb()); + } + if (formatOptions != null) { + loadConfigurationPb.setSourceFormat(formatOptions.type()); + } + loadConfigurationPb.setMaxBadRecords(maxBadRecords); + loadConfigurationPb.setIgnoreUnknownValues(ignoreUnknownValues); + loadConfigurationPb.setProjectionFields(projectionFields); + return new com.google.api.services.bigquery.model.JobConfiguration() + .setLoad(loadConfigurationPb); + } + + static WriteChannelConfiguration fromPb( + com.google.api.services.bigquery.model.JobConfiguration configurationPb) { + return new Builder(configurationPb).build(); + } + + /** + * Creates a builder for a BigQuery Load Configuration given the destination table. + */ + public static Builder builder(TableId destinationTable) { + return new Builder().destinationTable(destinationTable); + } + + /** + * Creates a builder for a BigQuery Load Configuration given the destination table and format. + */ + public static Builder builder(TableId destinationTable, FormatOptions format) { + return builder(destinationTable).formatOptions(format); + } + + /** + * Returns a BigQuery Load Configuration for the given destination table. + */ + public static WriteChannelConfiguration of(TableId destinationTable) { + return builder(destinationTable).build(); + } + + /** + * Returns a BigQuery Load Configuration for the given destination table and format. + */ + public static WriteChannelConfiguration of(TableId destinationTable, FormatOptions format) { + return builder(destinationTable).formatOptions(format).build(); + } +} diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/package-info.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/package-info.java index 8553de221fdc..db5e956e0a12 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/package-info.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/package-info.java @@ -17,28 +17,29 @@ /** * A client to Google Cloud BigQuery. * - *

A simple usage example: + *

A simple usage example showing how to create a table if it does not exist and load data into + * it. For the complete source code see + * + * CreateTableAndLoadData.java. *

 {@code
  * BigQuery bigquery = BigQueryOptions.defaultInstance().service();
  * TableId tableId = TableId.of("dataset", "table");
- * BaseTableInfo info = bigquery.getTable(tableId);
- * if (info == null) {
+ * Table table = bigquery.getTable(tableId);
+ * if (table == null) {
  *   System.out.println("Creating table " + tableId);
  *   Field integerField = Field.of("fieldName", Field.Type.integer());
- *   bigquery.create(TableInfo.of(tableId, Schema.of(integerField)));
+ *   Schema schema = Schema.of(integerField);
+ *   table = bigquery.create(TableInfo.of(tableId, StandardTableDefinition.of(schema)));
+ * }
+ * System.out.println("Loading data into table " + tableId);
+ * Job loadJob = table.load(FormatOptions.csv(), "gs://bucket/path");
+ * while (!loadJob.isDone()) {
+ *   Thread.sleep(1000L);
+ * }
+ * if (loadJob.status().error() != null) {
+ *   System.out.println("Job completed with errors");
  * } else {
- *   System.out.println("Loading data into table " + tableId);
- *   LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path");
- *   loadJob = bigquery.create(loadJob);
- *   while (loadJob.status().state() != JobStatus.State.DONE) {
- *     Thread.sleep(1000L);
- *     loadJob = bigquery.getJob(loadJob.jobId());
- *   }
- *   if (loadJob.status().error() != null) {
- *     System.out.println("Job completed with errors");
- *   } else {
- *     System.out.println("Job succeeded");
- *   }
+ *   System.out.println("Job succeeded");
  * }}
* * @see Google Cloud BigQuery diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/BigQueryRpc.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/BigQueryRpc.java index 5f17f60f2bb5..6062e19950e0 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/BigQueryRpc.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/BigQueryRpc.java @@ -19,7 +19,7 @@ import com.google.api.services.bigquery.model.Dataset; import com.google.api.services.bigquery.model.GetQueryResultsResponse; import com.google.api.services.bigquery.model.Job; -import com.google.api.services.bigquery.model.JobConfigurationLoad; +import com.google.api.services.bigquery.model.JobConfiguration; import com.google.api.services.bigquery.model.QueryRequest; import com.google.api.services.bigquery.model.QueryResponse; import com.google.api.services.bigquery.model.Table; @@ -193,7 +193,7 @@ GetQueryResultsResponse getQueryResults(String jobId, Map options) * @param configuration load configuration * @throws BigQueryException upon failure */ - String open(JobConfigurationLoad configuration) throws BigQueryException; + String open(JobConfiguration configuration) throws BigQueryException; /** * Uploads the provided data to the resumable upload session at the specified position. diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/DefaultBigQueryRpc.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/DefaultBigQueryRpc.java index 0a1dc046bf74..b57f1dc8a128 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/DefaultBigQueryRpc.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/DefaultBigQueryRpc.java @@ -24,7 +24,6 @@ import static java.net.HttpURLConnection.HTTP_NOT_FOUND; import static java.net.HttpURLConnection.HTTP_OK; -import com.google.api.client.googleapis.json.GoogleJsonError; import com.google.api.client.http.ByteArrayContent; import com.google.api.client.http.GenericUrl; import com.google.api.client.http.HttpRequest; @@ -43,7 +42,6 @@ import com.google.api.services.bigquery.model.GetQueryResultsResponse; import com.google.api.services.bigquery.model.Job; import com.google.api.services.bigquery.model.JobConfiguration; -import com.google.api.services.bigquery.model.JobConfigurationLoad; import com.google.api.services.bigquery.model.JobList; import com.google.api.services.bigquery.model.JobStatus; import com.google.api.services.bigquery.model.QueryRequest; @@ -408,9 +406,9 @@ public QueryResponse query(QueryRequest request) throws BigQueryException { } @Override - public String open(JobConfigurationLoad configuration) throws BigQueryException { + public String open(JobConfiguration configuration) throws BigQueryException { try { - Job loadJob = new Job().setConfiguration(new JobConfiguration().setLoad(configuration)); + Job loadJob = new Job().setConfiguration(configuration); StringBuilder builder = new StringBuilder() .append(BASE_RESUMABLE_URI) .append(options.projectId()) diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/BigQueryImplTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/BigQueryImplTest.java index 3fe1824fd3c2..385ee6dcc8bd 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/BigQueryImplTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/BigQueryImplTest.java @@ -26,10 +26,7 @@ import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; -import com.google.api.services.bigquery.model.Dataset; import com.google.api.services.bigquery.model.ErrorProto; -import com.google.api.services.bigquery.model.Job; -import com.google.api.services.bigquery.model.Table; import com.google.api.services.bigquery.model.TableCell; import com.google.api.services.bigquery.model.TableDataInsertAllRequest; import com.google.api.services.bigquery.model.TableDataInsertAllResponse; @@ -104,46 +101,48 @@ public class BigQueryImplTest { .description("FieldDescription3") .build(); private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA1, FIELD_SCHEMA2, FIELD_SCHEMA3); - private static final TableInfo TABLE_INFO = TableInfo.of(TABLE_ID, TABLE_SCHEMA); - private static final TableInfo OTHER_TABLE_INFO = TableInfo.of(OTHER_TABLE_ID, TABLE_SCHEMA); + private static final StandardTableDefinition TABLE_DEFINITION = + StandardTableDefinition.of(TABLE_SCHEMA); + private static final TableInfo TABLE_INFO = TableInfo.of(TABLE_ID, TABLE_DEFINITION); + private static final TableInfo OTHER_TABLE_INFO = TableInfo.of(OTHER_TABLE_ID, TABLE_DEFINITION); private static final TableInfo TABLE_INFO_WITH_PROJECT = - TableInfo.of(TABLE_ID_WITH_PROJECT, TABLE_SCHEMA); - private static final LoadJobInfo LOAD_JOB = LoadJobInfo.of(LoadConfiguration.of(TABLE_ID), "URI"); - private static final LoadJobInfo LOAD_JOB_WITH_PROJECT = - LoadJobInfo.of(LoadConfiguration.of(TABLE_ID_WITH_PROJECT), "URI"); - private static final LoadJobInfo COMPLETE_LOAD_JOB = - LoadJobInfo.builder(LoadConfiguration.of(TABLE_ID_WITH_PROJECT), "URI") - .jobId(JobId.of(PROJECT, JOB)) - .build(); - private static final CopyJobInfo COPY_JOB = - CopyJobInfo.of(TABLE_ID, ImmutableList.of(TABLE_ID, TABLE_ID)); - private static final CopyJobInfo COPY_JOB_WITH_PROJECT = - CopyJobInfo.of(TABLE_ID_WITH_PROJECT, ImmutableList.of(TABLE_ID_WITH_PROJECT, + TableInfo.of(TABLE_ID_WITH_PROJECT, TABLE_DEFINITION); + private static final LoadJobConfiguration LOAD_JOB_CONFIGURATION = + LoadJobConfiguration.of(TABLE_ID, "URI"); + private static final LoadJobConfiguration LOAD_JOB_CONFIGURATION_WITH_PROJECT = + LoadJobConfiguration.of(TABLE_ID_WITH_PROJECT, "URI"); + private static final JobInfo LOAD_JOB = + JobInfo.of(LOAD_JOB_CONFIGURATION); + private static final JobInfo COMPLETE_LOAD_JOB = + JobInfo.of(JobId.of(PROJECT, JOB), LOAD_JOB_CONFIGURATION_WITH_PROJECT); + private static final CopyJobConfiguration COPY_JOB_CONFIGURATION = + CopyJobConfiguration.of(TABLE_ID, ImmutableList.of(TABLE_ID, TABLE_ID)); + private static final CopyJobConfiguration COPY_JOB_CONFIGURATION_WITH_PROJECT = + CopyJobConfiguration.of(TABLE_ID_WITH_PROJECT, ImmutableList.of(TABLE_ID_WITH_PROJECT, TABLE_ID_WITH_PROJECT)); - private static final CopyJobInfo COMPLETE_COPY_JOB = - CopyJobInfo.builder(TABLE_ID_WITH_PROJECT, ImmutableList.of(TABLE_ID_WITH_PROJECT, - TABLE_ID_WITH_PROJECT)) - .jobId(JobId.of(PROJECT, JOB)) + private static final JobInfo COPY_JOB = JobInfo.of(COPY_JOB_CONFIGURATION); + private static final JobInfo COMPLETE_COPY_JOB = + JobInfo.of(JobId.of(PROJECT, JOB), COPY_JOB_CONFIGURATION_WITH_PROJECT); + private static final QueryJobConfiguration QUERY_JOB_CONFIGURATION = + QueryJobConfiguration.builder("SQL") + .defaultDataset(DatasetId.of(DATASET)) + .destinationTable(TABLE_ID) .build(); - private static final QueryJobInfo QUERY_JOB = QueryJobInfo.builder("SQL") - .defaultDataset(DatasetId.of(DATASET)) - .destinationTable(TABLE_ID) - .build(); - private static final QueryJobInfo QUERY_JOB_WITH_PROJECT = QueryJobInfo.builder("SQL") - .defaultDataset(DatasetId.of(PROJECT, DATASET)) - .destinationTable(TABLE_ID_WITH_PROJECT) - .build(); - private static final QueryJobInfo COMPLETE_QUERY_JOB = QueryJobInfo.builder("SQL") - .defaultDataset(DatasetId.of(PROJECT, DATASET)).destinationTable(TABLE_ID_WITH_PROJECT) - .jobId(JobId.of(PROJECT, JOB)) - .build(); - private static final ExtractJobInfo EXTRACT_JOB = ExtractJobInfo.of(TABLE_ID, "URI"); - private static final ExtractJobInfo EXTRACT_JOB_WITH_PROJECT = - ExtractJobInfo.of(TABLE_ID_WITH_PROJECT, "URI"); - private static final ExtractJobInfo COMPLETE_EXTRACT_JOB = - ExtractJobInfo.builder(TABLE_ID_WITH_PROJECT, "URI") - .jobId(JobId.of(PROJECT, JOB)) + private static final QueryJobConfiguration QUERY_JOB_CONFIGURATION_WITH_PROJECT = + QueryJobConfiguration.builder("SQL") + .defaultDataset(DatasetId.of(PROJECT, DATASET)) + .destinationTable(TABLE_ID_WITH_PROJECT) .build(); + private static final JobInfo QUERY_JOB = JobInfo.of(QUERY_JOB_CONFIGURATION); + private static final JobInfo COMPLETE_QUERY_JOB = + JobInfo.of(JobId.of(PROJECT, JOB), QUERY_JOB_CONFIGURATION_WITH_PROJECT); + private static final ExtractJobConfiguration EXTRACT_JOB_CONFIGURATION = + ExtractJobConfiguration.of(TABLE_ID, "URI"); + private static final ExtractJobConfiguration EXTRACT_JOB_CONFIGURATION_WITH_PROJECT = + ExtractJobConfiguration.of(TABLE_ID_WITH_PROJECT, "URI"); + private static final JobInfo EXTRACT_JOB = JobInfo.of(EXTRACT_JOB_CONFIGURATION); + private static final JobInfo COMPLETE_EXTRACT_JOB = + JobInfo.of(JobId.of(PROJECT, JOB), EXTRACT_JOB_CONFIGURATION_WITH_PROJECT); private static final TableCell BOOLEAN_FIELD = new TableCell().setV("false"); private static final TableCell INTEGER_FIELD = new TableCell().setV("1"); private static final TableRow TABLE_ROW = @@ -285,8 +284,9 @@ public void testCreateDataset() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.create(DATASET_INFO); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + Dataset dataset = bigquery.create(DATASET_INFO); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test @@ -297,13 +297,14 @@ public void testCreateDatasetWithSelectedFields() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.create(DATASET_INFO, DATASET_OPTION_FIELDS); + Dataset dataset = bigquery.create(DATASET_INFO, DATASET_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(DATASET_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("datasetReference")); assertTrue(selector.contains("access")); assertTrue(selector.contains("etag")); assertEquals(28, selector.length()); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test @@ -312,8 +313,9 @@ public void testGetDataset() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.getDataset(DATASET); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + Dataset dataset = bigquery.getDataset(DATASET); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test @@ -322,8 +324,9 @@ public void testGetDatasetFromDatasetId() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.getDataset(DatasetId.of(PROJECT, DATASET)); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + Dataset dataset = bigquery.getDataset(DatasetId.of(PROJECT, DATASET)); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test @@ -333,54 +336,58 @@ public void testGetDatasetWithSelectedFields() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.getDataset(DATASET, DATASET_OPTION_FIELDS); + Dataset dataset = bigquery.getDataset(DATASET, DATASET_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(DATASET_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("datasetReference")); assertTrue(selector.contains("access")); assertTrue(selector.contains("etag")); assertEquals(28, selector.length()); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test public void testListDatasets() { String cursor = "cursor"; - ImmutableList datasetList = ImmutableList.of(DATASET_INFO_WITH_PROJECT, - OTHER_DATASET_INFO); - Tuple> result = + bigquery = options.service(); + ImmutableList datasetList = ImmutableList.of( + new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + new Dataset(bigquery, new DatasetInfo.BuilderImpl(OTHER_DATASET_INFO))); + Tuple> result = Tuple.of(cursor, Iterables.transform(datasetList, DatasetInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listDatasets(EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listDatasets(); + Page page = bigquery.listDatasets(); assertEquals(cursor, page.nextPageCursor()); assertArrayEquals(datasetList.toArray(), Iterables.toArray(page.values(), DatasetInfo.class)); } @Test public void testListEmptyDatasets() { - ImmutableList datasets = ImmutableList.of(); - Tuple> result = Tuple.>of(null, datasets); + ImmutableList datasets = ImmutableList.of(); + Tuple> result = + Tuple.>of(null, datasets); EasyMock.expect(bigqueryRpcMock.listDatasets(EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - Page page = bigquery.listDatasets(); + Page page = bigquery.listDatasets(); assertNull(page.nextPageCursor()); assertArrayEquals(ImmutableList.of().toArray(), - Iterables.toArray(page.values(), DatasetInfo.class)); + Iterables.toArray(page.values(), Dataset.class)); } @Test public void testListDatasetsWithOptions() { String cursor = "cursor"; - ImmutableList datasetList = ImmutableList.of(DATASET_INFO_WITH_PROJECT, - OTHER_DATASET_INFO); - Tuple> result = + bigquery = options.service(); + ImmutableList datasetList = ImmutableList.of( + new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + new Dataset(bigquery, new DatasetInfo.BuilderImpl(OTHER_DATASET_INFO))); + Tuple> result = Tuple.of(cursor, Iterables.transform(datasetList, DatasetInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listDatasets(DATASET_LIST_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listDatasets(DATASET_LIST_ALL, DATASET_LIST_PAGE_TOKEN, + Page page = bigquery.listDatasets(DATASET_LIST_ALL, DATASET_LIST_PAGE_TOKEN, DATASET_LIST_MAX_RESULTS); assertEquals(cursor, page.nextPageCursor()); assertArrayEquals(datasetList.toArray(), Iterables.toArray(page.values(), DatasetInfo.class)); @@ -420,8 +427,9 @@ public void testUpdateDataset() { .andReturn(updatedDatasetInfoWithProject.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.update(updatedDatasetInfo); - assertEquals(updatedDatasetInfoWithProject, dataset); + Dataset dataset = bigquery.update(updatedDatasetInfo); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(updatedDatasetInfoWithProject)), + dataset); } @Test @@ -436,13 +444,14 @@ public void testUpdateDatasetWithSelectedFields() { .andReturn(updatedDatasetInfoWithProject.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.update(updatedDatasetInfo, DATASET_OPTION_FIELDS); + Dataset dataset = bigquery.update(updatedDatasetInfo, DATASET_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(DATASET_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("datasetReference")); assertTrue(selector.contains("access")); assertTrue(selector.contains("etag")); assertEquals(28, selector.length()); - assertEquals(updatedDatasetInfoWithProject, dataset); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(updatedDatasetInfoWithProject)), + dataset); } @Test @@ -451,8 +460,8 @@ public void testCreateTable() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.create(TABLE_INFO); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + Table table = bigquery.create(TABLE_INFO); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test @@ -463,13 +472,13 @@ public void testCreateTableWithSelectedFields() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.create(TABLE_INFO, TABLE_OPTION_FIELDS); + Table table = bigquery.create(TABLE_INFO, TABLE_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(TABLE_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("tableReference")); assertTrue(selector.contains("schema")); assertTrue(selector.contains("etag")); assertEquals(31, selector.length()); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test @@ -478,8 +487,8 @@ public void testGetTable() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.getTable(DATASET, TABLE); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + Table table = bigquery.getTable(DATASET, TABLE); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test @@ -488,8 +497,8 @@ public void testGetTableFromTableId() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.getTable(TABLE_ID); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + Table table = bigquery.getTable(TABLE_ID); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test @@ -499,59 +508,61 @@ public void testGetTableWithSelectedFields() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.getTable(TABLE_ID, TABLE_OPTION_FIELDS); + Table table = bigquery.getTable(TABLE_ID, TABLE_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(TABLE_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("tableReference")); assertTrue(selector.contains("schema")); assertTrue(selector.contains("etag")); assertEquals(31, selector.length()); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test public void testListTables() { String cursor = "cursor"; - ImmutableList tableList = - ImmutableList.of(TABLE_INFO_WITH_PROJECT, OTHER_TABLE_INFO); - Tuple> result = - Tuple.of(cursor, Iterables.transform(tableList, BaseTableInfo.TO_PB_FUNCTION)); + bigquery = options.service(); + ImmutableList
tableList = ImmutableList.of( + new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), + new Table(bigquery, new TableInfo.BuilderImpl(OTHER_TABLE_INFO))); + Tuple> result = + Tuple.of(cursor, Iterables.transform(tableList, TableInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listTables(DATASET, EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listTables(DATASET); + Page
page = bigquery.listTables(DATASET); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), BaseTableInfo.class)); + assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), Table.class)); } @Test public void testListTablesFromDatasetId() { String cursor = "cursor"; - ImmutableList tableList = - ImmutableList.of(TABLE_INFO_WITH_PROJECT, OTHER_TABLE_INFO); - Tuple> result = - Tuple.of(cursor, Iterables.transform(tableList, BaseTableInfo.TO_PB_FUNCTION)); + bigquery = options.service(); + ImmutableList
tableList = ImmutableList.of( + new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), + new Table(bigquery, new TableInfo.BuilderImpl(OTHER_TABLE_INFO))); + Tuple> result = + Tuple.of(cursor, Iterables.transform(tableList, TableInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listTables(DATASET, EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listTables(DatasetId.of(PROJECT, DATASET)); + Page
page = bigquery.listTables(DatasetId.of(PROJECT, DATASET)); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), BaseTableInfo.class)); + assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), Table.class)); } @Test public void testListTablesWithOptions() { String cursor = "cursor"; - ImmutableList tableList = - ImmutableList.of(TABLE_INFO_WITH_PROJECT, OTHER_TABLE_INFO); - Tuple> result = - Tuple.of(cursor, Iterables.transform(tableList, BaseTableInfo.TO_PB_FUNCTION)); + bigquery = options.service(); + ImmutableList
tableList = ImmutableList.of( + new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), + new Table(bigquery, new TableInfo.BuilderImpl(OTHER_TABLE_INFO))); + Tuple> result = + Tuple.of(cursor, Iterables.transform(tableList, TableInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listTables(DATASET, TABLE_LIST_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listTables(DATASET, TABLE_LIST_MAX_RESULTS, - TABLE_LIST_PAGE_TOKEN); + Page
page = bigquery.listTables(DATASET, TABLE_LIST_MAX_RESULTS, TABLE_LIST_PAGE_TOKEN); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), BaseTableInfo.class)); + assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), Table.class)); } @Test @@ -580,8 +591,9 @@ public void testUpdateTable() { .andReturn(updatedTableInfoWithProject.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.update(updatedTableInfo); - assertEquals(updatedTableInfoWithProject, table); + Table table = bigquery.update(updatedTableInfo); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(updatedTableInfoWithProject)), + table); } @Test @@ -595,13 +607,14 @@ public void testUpdateTableWithSelectedFields() { capture(capturedOptions))).andReturn(updatedTableInfoWithProject.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.update(updatedTableInfo, TABLE_OPTION_FIELDS); + Table table = bigquery.update(updatedTableInfo, TABLE_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(TABLE_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("tableReference")); assertTrue(selector.contains("schema")); assertTrue(selector.contains("etag")); assertEquals(31, selector.length()); - assertEquals(updatedTableInfoWithProject, table); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(updatedTableInfoWithProject)), + table); } @Test @@ -625,8 +638,7 @@ public TableDataInsertAllRequest.Rows apply(RowToInsert rowToInsert) { return new TableDataInsertAllRequest.Rows().setInsertId(rowToInsert.id()) .setJson(rowToInsert.content()); } - }) - ).setSkipInvalidRows(false).setIgnoreUnknownValues(true).setTemplateSuffix("suffix"); + })).setSkipInvalidRows(false).setIgnoreUnknownValues(true).setTemplateSuffix("suffix"); TableDataInsertAllResponse responsePb = new TableDataInsertAllResponse().setInsertErrors( ImmutableList.of(new TableDataInsertAllResponse.InsertErrors().setIndex(0L).setErrors( ImmutableList.of(new ErrorProto().setMessage("ErrorMessage"))))); @@ -728,54 +740,58 @@ public void testListTableDataWithOptions() { @Test public void testCreateQueryJob() { - EasyMock.expect(bigqueryRpcMock.create(QUERY_JOB_WITH_PROJECT.toPb(), EMPTY_RPC_OPTIONS)) - .andReturn(COMPLETE_QUERY_JOB.toPb()); + EasyMock.expect(bigqueryRpcMock.create( + JobInfo.of(QUERY_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) + .andReturn(COMPLETE_QUERY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - QueryJobInfo job = bigquery.create(QUERY_JOB); - assertEquals(COMPLETE_QUERY_JOB, job); + Job job = bigquery.create(QUERY_JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), job); } @Test public void testCreateLoadJob() { - EasyMock.expect(bigqueryRpcMock.create(LOAD_JOB_WITH_PROJECT.toPb(), EMPTY_RPC_OPTIONS)) - .andReturn(COMPLETE_LOAD_JOB.toPb()); + EasyMock.expect(bigqueryRpcMock.create( + JobInfo.of(LOAD_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) + .andReturn(COMPLETE_LOAD_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - LoadJobInfo job = bigquery.create(LOAD_JOB); - assertEquals(COMPLETE_LOAD_JOB, job); + Job job = bigquery.create(LOAD_JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_LOAD_JOB)), job); } @Test public void testCreateCopyJob() { - EasyMock.expect(bigqueryRpcMock.create(COPY_JOB_WITH_PROJECT.toPb(), EMPTY_RPC_OPTIONS)) - .andReturn(COMPLETE_COPY_JOB.toPb()); + EasyMock.expect(bigqueryRpcMock.create( + JobInfo.of(COPY_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) + .andReturn(COMPLETE_COPY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - CopyJobInfo job = bigquery.create(COPY_JOB); - assertEquals(COMPLETE_COPY_JOB, job); + Job job = bigquery.create(COPY_JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_COPY_JOB)), job); } @Test public void testCreateExtractJob() { - EasyMock.expect(bigqueryRpcMock.create(EXTRACT_JOB_WITH_PROJECT.toPb(), EMPTY_RPC_OPTIONS)) - .andReturn(COMPLETE_EXTRACT_JOB.toPb()); + EasyMock.expect(bigqueryRpcMock.create( + JobInfo.of(EXTRACT_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) + .andReturn(COMPLETE_EXTRACT_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - ExtractJobInfo job = bigquery.create(EXTRACT_JOB); - assertEquals(COMPLETE_EXTRACT_JOB, job); + Job job = bigquery.create(EXTRACT_JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_EXTRACT_JOB)), job); } @Test public void testCreateJobWithSelectedFields() { Capture> capturedOptions = Capture.newInstance(); - EasyMock.expect( - bigqueryRpcMock.create(eq(QUERY_JOB_WITH_PROJECT.toPb()), capture(capturedOptions))) - .andReturn(COMPLETE_QUERY_JOB.toPb()); + EasyMock.expect(bigqueryRpcMock.create( + eq(JobInfo.of(QUERY_JOB_CONFIGURATION_WITH_PROJECT).toPb()), capture(capturedOptions))) + .andReturn(COMPLETE_QUERY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - QueryJobInfo job = bigquery.create(QUERY_JOB, JOB_OPTION_FIELDS); - assertEquals(COMPLETE_QUERY_JOB, job); + Job job = bigquery.create(QUERY_JOB, JOB_OPTION_FIELDS); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), job); String selector = (String) capturedOptions.getValue().get(JOB_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("jobReference")); assertTrue(selector.contains("configuration")); @@ -789,8 +805,8 @@ public void testGetJob() { .andReturn(COMPLETE_COPY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - CopyJobInfo job = bigquery.getJob(JOB); - assertEquals(COMPLETE_COPY_JOB, job); + Job job = bigquery.getJob(JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_COPY_JOB)), job); } @Test @@ -799,70 +815,76 @@ public void testGetJobFromJobId() { .andReturn(COMPLETE_COPY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - CopyJobInfo job = bigquery.getJob(JobId.of(PROJECT, JOB)); - assertEquals(COMPLETE_COPY_JOB, job); + Job job = bigquery.getJob(JobId.of(PROJECT, JOB)); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_COPY_JOB)), job); } @Test public void testListJobs() { String cursor = "cursor"; - ImmutableList jobList = - ImmutableList.of(QUERY_JOB_WITH_PROJECT, LOAD_JOB_WITH_PROJECT); - Tuple> result = - Tuple.of(cursor, Iterables.transform(jobList, new Function() { - @Override - public Job apply(JobInfo jobInfo) { - return jobInfo.toPb(); - } - })); + bigquery = options.service(); + ImmutableList jobList = ImmutableList.of( + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_LOAD_JOB))); + Tuple> result = + Tuple.of(cursor, Iterables.transform(jobList, + new Function() { + @Override + public com.google.api.services.bigquery.model.Job apply(Job job) { + return job.toPb(); + } + })); EasyMock.expect(bigqueryRpcMock.listJobs(EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listJobs(); + Page page = bigquery.listJobs(); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), JobInfo.class)); + assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), Job.class)); } @Test public void testListJobsWithOptions() { String cursor = "cursor"; - ImmutableList jobList = - ImmutableList.of(QUERY_JOB_WITH_PROJECT, LOAD_JOB_WITH_PROJECT); - Tuple> result = - Tuple.of(cursor, Iterables.transform(jobList, new Function() { - @Override - public Job apply(JobInfo jobInfo) { - return jobInfo.toPb(); - } - })); + bigquery = options.service(); + ImmutableList jobList = ImmutableList.of( + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_LOAD_JOB))); + Tuple> result = + Tuple.of(cursor, Iterables.transform(jobList, + new Function() { + @Override + public com.google.api.services.bigquery.model.Job apply(Job job) { + return job.toPb(); + } + })); EasyMock.expect(bigqueryRpcMock.listJobs(JOB_LIST_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listJobs(JOB_LIST_ALL_USERS, JOB_LIST_STATE_FILTER, + Page page = bigquery.listJobs(JOB_LIST_ALL_USERS, JOB_LIST_STATE_FILTER, JOB_LIST_PAGE_TOKEN, JOB_LIST_MAX_RESULTS); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), JobInfo.class)); + assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), Job.class)); } @Test public void testListJobsWithSelectedFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList jobList = - ImmutableList.of(QUERY_JOB_WITH_PROJECT, LOAD_JOB_WITH_PROJECT); - Tuple> result = - Tuple.of(cursor, Iterables.transform(jobList, new Function() { - @Override - public Job apply(JobInfo jobInfo) { - return jobInfo.toPb(); - } - })); + bigquery = options.service(); + ImmutableList jobList = ImmutableList.of( + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_LOAD_JOB))); + Tuple> result = + Tuple.of(cursor, Iterables.transform(jobList, + new Function() { + @Override + public com.google.api.services.bigquery.model.Job apply(Job job) { + return job.toPb(); + } + })); EasyMock.expect(bigqueryRpcMock.listJobs(capture(capturedOptions))).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listJobs(JOB_LIST_OPTION_FIELD); + Page page = bigquery.listJobs(JOB_LIST_OPTION_FIELD); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), JobInfo.class)); + assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), Job.class)); String selector = (String) capturedOptions.getValue().get(JOB_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("etag,jobs(")); assertTrue(selector.contains("configuration")); @@ -1009,12 +1031,13 @@ public void testGetQueryResultsWithOptions() { @Test public void testWriter() { - LoadConfiguration loadConfiguration = LoadConfiguration.of(TABLE_ID); - EasyMock.expect(bigqueryRpcMock.open(LoadConfiguration.of(TABLE_ID_WITH_PROJECT).toPb())) - .andReturn("upload-id"); + WriteChannelConfiguration writeChannelConfiguration = WriteChannelConfiguration.of(TABLE_ID); + EasyMock.expect( + bigqueryRpcMock.open(WriteChannelConfiguration.of(TABLE_ID_WITH_PROJECT).toPb())) + .andReturn("upload-id"); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - WriteChannel channel = bigquery.writer(loadConfiguration); + WriteChannel channel = bigquery.writer(writeChannelConfiguration); assertNotNull(channel); assertTrue(channel.isOpen()); } @@ -1026,8 +1049,9 @@ public void testRetryableException() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.toBuilder().retryParams(RetryParams.defaultInstance()).build().service(); - DatasetInfo dataset = bigquery.getDataset(DATASET); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + Dataset dataset = bigquery.getDataset(DATASET); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/CopyJobConfigurationTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/CopyJobConfigurationTest.java new file mode 100644 index 000000000000..3f3f6f0fd15c --- /dev/null +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/CopyJobConfigurationTest.java @@ -0,0 +1,130 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; + +import com.google.common.collect.ImmutableList; +import com.google.gcloud.bigquery.JobInfo.CreateDisposition; +import com.google.gcloud.bigquery.JobInfo.WriteDisposition; + +import org.junit.Test; + +import java.util.List; + +public class CopyJobConfigurationTest { + + private static final TableId SOURCE_TABLE = TableId.of("dataset", "sourceTable"); + private static final List SOURCE_TABLES = ImmutableList.of( + TableId.of("dataset", "sourceTable1"), + TableId.of("dataset", "sourceTable2")); + private static final TableId DESTINATION_TABLE = TableId.of("dataset", "destinationTable"); + private static final CreateDisposition CREATE_DISPOSITION = CreateDisposition.CREATE_IF_NEEDED; + private static final WriteDisposition WRITE_DISPOSITION = WriteDisposition.WRITE_APPEND; + private static final CopyJobConfiguration COPY_JOB_CONFIGURATION = + CopyJobConfiguration.builder(DESTINATION_TABLE, SOURCE_TABLE) + .createDisposition(CREATE_DISPOSITION) + .writeDisposition(WRITE_DISPOSITION) + .build(); + private static final CopyJobConfiguration COPY_JOB_CONFIGURATION_MULTIPLE_TABLES = + CopyJobConfiguration.builder(DESTINATION_TABLE, SOURCE_TABLES) + .createDisposition(CREATE_DISPOSITION) + .writeDisposition(WRITE_DISPOSITION) + .build(); + + @Test + public void testToBuilder() { + compareCopyJobConfiguration(COPY_JOB_CONFIGURATION, COPY_JOB_CONFIGURATION.toBuilder().build()); + compareCopyJobConfiguration(COPY_JOB_CONFIGURATION_MULTIPLE_TABLES, + COPY_JOB_CONFIGURATION_MULTIPLE_TABLES.toBuilder().build()); + CopyJobConfiguration jobConfiguration = COPY_JOB_CONFIGURATION.toBuilder() + .destinationTable(TableId.of("dataset", "newTable")) + .build(); + assertEquals("newTable", jobConfiguration.destinationTable().table()); + jobConfiguration = jobConfiguration.toBuilder().destinationTable(DESTINATION_TABLE).build(); + compareCopyJobConfiguration(COPY_JOB_CONFIGURATION, jobConfiguration); + } + + @Test + public void testOf() { + CopyJobConfiguration job = CopyJobConfiguration.of(DESTINATION_TABLE, SOURCE_TABLES); + assertEquals(DESTINATION_TABLE, job.destinationTable()); + assertEquals(SOURCE_TABLES, job.sourceTables()); + job = CopyJobConfiguration.of(DESTINATION_TABLE, SOURCE_TABLE); + assertEquals(DESTINATION_TABLE, job.destinationTable()); + assertEquals(ImmutableList.of(SOURCE_TABLE), job.sourceTables()); + } + + @Test + public void testToBuilderIncomplete() { + CopyJobConfiguration jobConfiguration = + CopyJobConfiguration.of(DESTINATION_TABLE, SOURCE_TABLES); + compareCopyJobConfiguration(jobConfiguration, jobConfiguration.toBuilder().build()); + } + + @Test + public void testBuilder() { + assertEquals(DESTINATION_TABLE, COPY_JOB_CONFIGURATION_MULTIPLE_TABLES.destinationTable()); + assertEquals(SOURCE_TABLES, COPY_JOB_CONFIGURATION_MULTIPLE_TABLES.sourceTables()); + assertEquals(CREATE_DISPOSITION, COPY_JOB_CONFIGURATION_MULTIPLE_TABLES.createDisposition()); + assertEquals(WRITE_DISPOSITION, COPY_JOB_CONFIGURATION_MULTIPLE_TABLES.writeDisposition()); + assertEquals(DESTINATION_TABLE, COPY_JOB_CONFIGURATION.destinationTable()); + assertEquals(ImmutableList.of(SOURCE_TABLE), COPY_JOB_CONFIGURATION.sourceTables()); + assertEquals(CREATE_DISPOSITION, COPY_JOB_CONFIGURATION.createDisposition()); + assertEquals(WRITE_DISPOSITION, COPY_JOB_CONFIGURATION.writeDisposition()); + } + + @Test + public void testToPbAndFromPb() { + assertNotNull(COPY_JOB_CONFIGURATION.toPb().getCopy()); + assertNull(COPY_JOB_CONFIGURATION.toPb().getExtract()); + assertNull(COPY_JOB_CONFIGURATION.toPb().getLoad()); + assertNull(COPY_JOB_CONFIGURATION.toPb().getQuery()); + assertNull(COPY_JOB_CONFIGURATION.toPb().getCopy().getSourceTables()); + assertNull(COPY_JOB_CONFIGURATION_MULTIPLE_TABLES.toPb().getCopy().getSourceTable()); + compareCopyJobConfiguration(COPY_JOB_CONFIGURATION, + CopyJobConfiguration.fromPb(COPY_JOB_CONFIGURATION.toPb())); + compareCopyJobConfiguration(COPY_JOB_CONFIGURATION_MULTIPLE_TABLES, + CopyJobConfiguration.fromPb(COPY_JOB_CONFIGURATION_MULTIPLE_TABLES.toPb())); + CopyJobConfiguration jobConfiguration = + CopyJobConfiguration.of(DESTINATION_TABLE, SOURCE_TABLES); + compareCopyJobConfiguration( + jobConfiguration, CopyJobConfiguration.fromPb(jobConfiguration.toPb())); + } + + @Test + public void testSetProjectId() { + CopyJobConfiguration configuration = COPY_JOB_CONFIGURATION_MULTIPLE_TABLES.setProjectId("p"); + assertEquals("p", configuration.destinationTable().project()); + for (TableId sourceTable : configuration.sourceTables()) { + assertEquals("p", sourceTable.project()); + } + } + + private void compareCopyJobConfiguration(CopyJobConfiguration expected, + CopyJobConfiguration value) { + assertEquals(expected, value); + assertEquals(expected.hashCode(), value.hashCode()); + assertEquals(expected.toString(), value.toString()); + assertEquals(expected.destinationTable(), value.destinationTable()); + assertEquals(expected.sourceTables(), value.sourceTables()); + assertEquals(expected.createDisposition(), value.createDisposition()); + assertEquals(expected.writeDisposition(), value.writeDisposition()); + } +} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/CopyJobInfoTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/CopyJobInfoTest.java deleted file mode 100644 index 81da59644cf0..000000000000 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/CopyJobInfoTest.java +++ /dev/null @@ -1,174 +0,0 @@ -/* - * Copyright 2015 Google Inc. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.google.gcloud.bigquery; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNotNull; -import static org.junit.Assert.assertNull; - -import com.google.common.collect.ImmutableList; -import com.google.gcloud.bigquery.JobInfo.CreateDisposition; -import com.google.gcloud.bigquery.JobInfo.WriteDisposition; - -import org.junit.Test; - -import java.util.List; - -public class CopyJobInfoTest { - - private static final String ETAG = "etag"; - private static final String ID = "id"; - private static final String SELF_LINK = "selfLink"; - private static final String EMAIL = "email"; - private static final TableId SOURCE_TABLE = TableId.of("dataset", "sourceTable"); - private static final List SOURCE_TABLES = ImmutableList.of( - TableId.of("dataset", "sourceTable1"), - TableId.of("dataset", "sourceTable2") - ); - private static final TableId DESTINATION_TABLE = TableId.of("dataset", "destinationTable"); - private static final CreateDisposition CREATE_DISPOSITION = CreateDisposition.CREATE_IF_NEEDED; - private static final WriteDisposition WRITE_DISPOSITION = WriteDisposition.WRITE_APPEND; - private static final JobId JOB_ID = JobId.of("job"); - private static final JobStatus JOB_STATUS = new JobStatus(JobStatus.State.DONE); - private static final JobStatistics JOB_STATISTICS = JobStatistics.builder() - .creationTime(1L) - .endTime(3L) - .startTime(2L) - .build(); - private static final CopyJobInfo COPY_JOB = - CopyJobInfo.builder(DESTINATION_TABLE, SOURCE_TABLE) - .etag(ETAG) - .id(ID) - .selfLink(SELF_LINK) - .userEmail(EMAIL) - .jobId(JOB_ID) - .status(JOB_STATUS) - .createDisposition(CREATE_DISPOSITION) - .writeDisposition(WRITE_DISPOSITION) - .statistics(JOB_STATISTICS) - .build(); - private static final CopyJobInfo COPY_JOB_INFO_MULTIPLE_TABLES = - CopyJobInfo.builder(DESTINATION_TABLE, SOURCE_TABLES) - .etag(ETAG) - .id(ID) - .selfLink(SELF_LINK) - .userEmail(EMAIL) - .jobId(JOB_ID) - .status(JOB_STATUS) - .createDisposition(CREATE_DISPOSITION) - .writeDisposition(WRITE_DISPOSITION) - .build(); - - @Test - public void testToBuilder() { - compareCopyJobInfo(COPY_JOB, COPY_JOB.toBuilder().build()); - compareCopyJobInfo(COPY_JOB_INFO_MULTIPLE_TABLES, - COPY_JOB_INFO_MULTIPLE_TABLES.toBuilder().build()); - CopyJobInfo job = COPY_JOB.toBuilder() - .destinationTable(TableId.of("dataset", "newTable")) - .build(); - assertEquals("newTable", job.destinationTable().table()); - job = job.toBuilder().destinationTable(DESTINATION_TABLE).build(); - compareCopyJobInfo(COPY_JOB, job); - } - - @Test - public void testOf() { - CopyJobInfo job = CopyJobInfo.of(DESTINATION_TABLE, SOURCE_TABLES); - assertEquals(DESTINATION_TABLE, job.destinationTable()); - assertEquals(SOURCE_TABLES, job.sourceTables()); - job = CopyJobInfo.of(DESTINATION_TABLE, SOURCE_TABLE); - assertEquals(DESTINATION_TABLE, job.destinationTable()); - assertEquals(ImmutableList.of(SOURCE_TABLE), job.sourceTables()); - job = CopyJobInfo.of(JOB_ID, DESTINATION_TABLE, SOURCE_TABLES); - assertEquals(JOB_ID, job.jobId()); - assertEquals(DESTINATION_TABLE, job.destinationTable()); - assertEquals(SOURCE_TABLES, job.sourceTables()); - job = CopyJobInfo.of(JOB_ID, DESTINATION_TABLE, SOURCE_TABLE); - assertEquals(JOB_ID, job.jobId()); - assertEquals(DESTINATION_TABLE, job.destinationTable()); - assertEquals(ImmutableList.of(SOURCE_TABLE), job.sourceTables()); - } - - @Test - public void testToBuilderIncomplete() { - CopyJobInfo job = CopyJobInfo.of(DESTINATION_TABLE, SOURCE_TABLES); - compareCopyJobInfo(job, job.toBuilder().build()); - } - - @Test - public void testBuilder() { - assertEquals(ETAG, COPY_JOB_INFO_MULTIPLE_TABLES.etag()); - assertEquals(ID, COPY_JOB_INFO_MULTIPLE_TABLES.id()); - assertEquals(SELF_LINK, COPY_JOB_INFO_MULTIPLE_TABLES.selfLink()); - assertEquals(EMAIL, COPY_JOB_INFO_MULTIPLE_TABLES.userEmail()); - assertEquals(JOB_ID, COPY_JOB_INFO_MULTIPLE_TABLES.jobId()); - assertEquals(JOB_STATUS, COPY_JOB_INFO_MULTIPLE_TABLES.status()); - assertEquals(DESTINATION_TABLE, COPY_JOB_INFO_MULTIPLE_TABLES.destinationTable()); - assertEquals(SOURCE_TABLES, COPY_JOB_INFO_MULTIPLE_TABLES.sourceTables()); - assertEquals(CREATE_DISPOSITION, COPY_JOB_INFO_MULTIPLE_TABLES.createDisposition()); - assertEquals(WRITE_DISPOSITION, COPY_JOB_INFO_MULTIPLE_TABLES.writeDisposition()); - assertEquals(ETAG, COPY_JOB.etag()); - assertEquals(ID, COPY_JOB.id()); - assertEquals(SELF_LINK, COPY_JOB.selfLink()); - assertEquals(EMAIL, COPY_JOB.userEmail()); - assertEquals(JOB_ID, COPY_JOB.jobId()); - assertEquals(JOB_STATUS, COPY_JOB.status()); - assertEquals(DESTINATION_TABLE, COPY_JOB.destinationTable()); - assertEquals(ImmutableList.of(SOURCE_TABLE), COPY_JOB.sourceTables()); - assertEquals(CREATE_DISPOSITION, COPY_JOB.createDisposition()); - assertEquals(WRITE_DISPOSITION, COPY_JOB.writeDisposition()); - assertEquals(JOB_STATISTICS, COPY_JOB.statistics()); - } - - @Test - public void testToPbAndFromPb() { - assertNotNull(COPY_JOB.toPb().getConfiguration().getCopy()); - assertNull(COPY_JOB.toPb().getConfiguration().getExtract()); - assertNull(COPY_JOB.toPb().getConfiguration().getLoad()); - assertNull(COPY_JOB.toPb().getConfiguration().getQuery()); - assertNull(COPY_JOB.toPb().getConfiguration().getCopy().getSourceTables()); - assertEquals(JOB_STATISTICS, JobStatistics.fromPb(COPY_JOB.statistics().toPb())); - assertNull(COPY_JOB_INFO_MULTIPLE_TABLES.toPb().getConfiguration().getCopy().getSourceTable()); - compareCopyJobInfo(COPY_JOB, CopyJobInfo.fromPb(COPY_JOB.toPb())); - compareCopyJobInfo(COPY_JOB, (CopyJobInfo) JobInfo.fromPb(COPY_JOB.toPb())); - compareCopyJobInfo(COPY_JOB_INFO_MULTIPLE_TABLES, - CopyJobInfo.fromPb(COPY_JOB_INFO_MULTIPLE_TABLES.toPb())); - compareCopyJobInfo(COPY_JOB_INFO_MULTIPLE_TABLES, - (CopyJobInfo) JobInfo.fromPb(COPY_JOB_INFO_MULTIPLE_TABLES.toPb())); - CopyJobInfo job = CopyJobInfo.of(DESTINATION_TABLE, SOURCE_TABLES); - compareCopyJobInfo(job, CopyJobInfo.fromPb(job.toPb())); - compareCopyJobInfo(job, (CopyJobInfo) JobInfo.fromPb(job.toPb())); - } - - private void compareCopyJobInfo(CopyJobInfo expected, CopyJobInfo value) { - assertEquals(expected, value); - assertEquals(expected.hashCode(), value.hashCode()); - assertEquals(expected.toString(), value.toString()); - assertEquals(expected.etag(), value.etag()); - assertEquals(expected.id(), value.id()); - assertEquals(expected.jobId(), value.jobId()); - assertEquals(expected.selfLink(), value.selfLink()); - assertEquals(expected.status(), value.status()); - assertEquals(expected.statistics(), value.statistics()); - assertEquals(expected.userEmail(), value.userEmail()); - assertEquals(expected.destinationTable(), value.destinationTable()); - assertEquals(expected.sourceTables(), value.sourceTables()); - assertEquals(expected.createDisposition(), value.createDisposition()); - assertEquals(expected.writeDisposition(), value.writeDisposition()); - } -} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetIdTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetIdTest.java index 0af665895d71..ec645d71c96f 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetIdTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetIdTest.java @@ -45,6 +45,11 @@ public void testToPbAndFromPb() { compareDatasetIds(DATASET_COMPLETE, DatasetId.fromPb(DATASET_COMPLETE.toPb())); } + @Test + public void testSetProjectId() { + assertEquals(DATASET_COMPLETE, DATASET.setProjectId("project")); + } + private void compareDatasetIds(DatasetId expected, DatasetId value) { assertEquals(expected, value); assertEquals(expected.project(), value.project()); diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetInfoTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetInfoTest.java index 733253a2d790..20875c0fc853 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetInfoTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetInfoTest.java @@ -29,7 +29,10 @@ public class DatasetInfoTest { private static final List ACCESS_RULES = ImmutableList.of( Acl.of(Acl.Group.ofAllAuthenticatedUsers(), Acl.Role.READER), - Acl.of(new Acl.View(TableId.of("project", "dataset", "table")), Acl.Role.WRITER)); + Acl.of(new Acl.View(TableId.of("dataset", "table")))); + private static final List ACCESS_RULES_COMPLETE = ImmutableList.of( + Acl.of(Acl.Group.ofAllAuthenticatedUsers(), Acl.Role.READER), + Acl.of(new Acl.View(TableId.of("project", "dataset", "table")))); private static final Long CREATION_TIME = System.currentTimeMillis(); private static final Long DEFAULT_TABLE_EXPIRATION = CREATION_TIME + 100; private static final String DESCRIPTION = "description"; @@ -55,6 +58,7 @@ public class DatasetInfoTest { .build(); private static final DatasetInfo DATASET_INFO_COMPLETE = DATASET_INFO.toBuilder() .datasetId(DATASET_ID_COMPLETE) + .acl(ACCESS_RULES_COMPLETE) .build(); @Test @@ -91,7 +95,7 @@ public void testBuilder() { assertEquals(LOCATION, DATASET_INFO.location()); assertEquals(SELF_LINK, DATASET_INFO.selfLink()); assertEquals(DATASET_ID_COMPLETE, DATASET_INFO_COMPLETE.datasetId()); - assertEquals(ACCESS_RULES, DATASET_INFO_COMPLETE.acl()); + assertEquals(ACCESS_RULES_COMPLETE, DATASET_INFO_COMPLETE.acl()); assertEquals(CREATION_TIME, DATASET_INFO_COMPLETE.creationTime()); assertEquals(DEFAULT_TABLE_EXPIRATION, DATASET_INFO_COMPLETE.defaultTableLifetime()); assertEquals(DESCRIPTION, DATASET_INFO_COMPLETE.description()); @@ -110,6 +114,11 @@ public void testToPbAndFromPb() { compareDatasets(datasetInfo, DatasetInfo.fromPb(datasetInfo.toPb())); } + @Test + public void testSetProjectId() { + assertEquals(DATASET_INFO_COMPLETE, DATASET_INFO.setProjectId("project")); + } + private void compareDatasets(DatasetInfo expected, DatasetInfo value) { assertEquals(expected, value); assertEquals(expected.datasetId(), value.datasetId()); diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetTest.java index 544fc2378b23..373291021b23 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetTest.java @@ -16,10 +16,13 @@ package com.google.gcloud.bigquery; +import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; +import static org.easymock.EasyMock.eq; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; +import static org.junit.Assert.assertArrayEquals; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; @@ -28,305 +31,343 @@ import static org.junit.Assert.assertTrue; import com.google.common.collect.ImmutableList; +import com.google.common.collect.Iterables; import com.google.gcloud.Page; import com.google.gcloud.PageImpl; import org.junit.After; -import org.junit.Before; -import org.junit.Rule; import org.junit.Test; -import org.junit.rules.ExpectedException; -import java.util.Iterator; import java.util.List; public class DatasetTest { private static final DatasetId DATASET_ID = DatasetId.of("dataset"); + private static final List ACCESS_RULES = ImmutableList.of( + Acl.of(Acl.Group.ofAllAuthenticatedUsers(), Acl.Role.READER), + Acl.of(new Acl.View(TableId.of("dataset", "table")))); + private static final Long CREATION_TIME = System.currentTimeMillis(); + private static final Long DEFAULT_TABLE_EXPIRATION = CREATION_TIME + 100; + private static final String DESCRIPTION = "description"; + private static final String ETAG = "0xFF00"; + private static final String FRIENDLY_NAME = "friendlyDataset"; + private static final String ID = "P/D:1"; + private static final Long LAST_MODIFIED = CREATION_TIME + 50; + private static final String LOCATION = ""; + private static final String SELF_LINK = "http://bigquery/p/d"; private static final DatasetInfo DATASET_INFO = DatasetInfo.builder(DATASET_ID).build(); private static final Field FIELD = Field.of("FieldName", Field.Type.integer()); - private static final Iterable TABLE_INFO_RESULTS = ImmutableList.of( - TableInfo.builder(TableId.of("dataset", "table1"), Schema.of(FIELD)).build(), - ViewInfo.builder(TableId.of("dataset", "table2"), "QUERY").build(), - ExternalTableInfo.builder(TableId.of("dataset", "table2"), - ExternalDataConfiguration.of(ImmutableList.of("URI"), Schema.of(), FormatOptions.csv())) - .build()); - private static final UserDefinedFunction FUNCTION1 = UserDefinedFunction.inline("inline"); - private static final UserDefinedFunction FUNCTION2 = UserDefinedFunction.inline("gs://b/f"); - private static final List FUNCTIONS = ImmutableList.of(FUNCTION1, FUNCTION2); - - @Rule - public ExpectedException thrown = ExpectedException.none(); + private static final StandardTableDefinition TABLE_DEFINITION = + StandardTableDefinition.of(Schema.of(FIELD)); + private static final ViewDefinition VIEW_DEFINITION = ViewDefinition.of("QUERY"); + private static final ExternalTableDefinition EXTERNAL_TABLE_DEFINITION = + ExternalTableDefinition.of(ImmutableList.of("URI"), Schema.of(), FormatOptions.csv()); + private static final TableInfo TABLE_INFO1 = + TableInfo.builder(TableId.of("dataset", "table1"), TABLE_DEFINITION).build(); + private static final TableInfo TABLE_INFO2 = + TableInfo.builder(TableId.of("dataset", "table2"), VIEW_DEFINITION).build(); + private static final TableInfo TABLE_INFO3 = + TableInfo.builder(TableId.of("dataset", "table3"), EXTERNAL_TABLE_DEFINITION).build(); + + private BigQuery serviceMockReturnsOptions = createStrictMock(BigQuery.class); + private BigQueryOptions mockOptions = createMock(BigQueryOptions.class); private BigQuery bigquery; + private Dataset expectedDataset; private Dataset dataset; - @Before - public void setUp() throws Exception { + private void initializeExpectedDataset(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); bigquery = createStrictMock(BigQuery.class); - dataset = new Dataset(bigquery, DATASET_INFO); + expectedDataset = new Dataset(serviceMockReturnsOptions, new Dataset.BuilderImpl(DATASET_INFO)); + } + + private void initializeDataset() { + dataset = new Dataset(bigquery, new Dataset.BuilderImpl(DATASET_INFO)); } @After public void tearDown() throws Exception { - verify(bigquery); + verify(bigquery, serviceMockReturnsOptions); } @Test - public void testInfo() throws Exception { - assertEquals(DATASET_INFO, dataset.info()); + public void testBuilder() { + initializeExpectedDataset(2); replay(bigquery); + Dataset builtDataset = new Dataset.Builder(serviceMockReturnsOptions, DATASET_ID) + .acl(ACCESS_RULES) + .creationTime(CREATION_TIME) + .defaultTableLifetime(DEFAULT_TABLE_EXPIRATION) + .description(DESCRIPTION) + .etag(ETAG) + .friendlyName(FRIENDLY_NAME) + .id(ID) + .lastModified(LAST_MODIFIED) + .location(LOCATION) + .selfLink(SELF_LINK) + .build(); + assertEquals(DATASET_ID, builtDataset.datasetId()); + assertEquals(ACCESS_RULES, builtDataset.acl()); + assertEquals(CREATION_TIME, builtDataset.creationTime()); + assertEquals(DEFAULT_TABLE_EXPIRATION, builtDataset.defaultTableLifetime()); + assertEquals(DESCRIPTION, builtDataset.description()); + assertEquals(ETAG, builtDataset.etag()); + assertEquals(FRIENDLY_NAME, builtDataset.friendlyName()); + assertEquals(ID, builtDataset.id()); + assertEquals(LAST_MODIFIED, builtDataset.lastModified()); + assertEquals(LOCATION, builtDataset.location()); + assertEquals(SELF_LINK, builtDataset.selfLink()); } @Test - public void testBigQuery() throws Exception { - assertSame(bigquery, dataset.bigquery()); + public void testToBuilder() { + initializeExpectedDataset(4); replay(bigquery); + compareDataset(expectedDataset, expectedDataset.toBuilder().build()); } @Test public void testExists_True() throws Exception { + initializeExpectedDataset(1); BigQuery.DatasetOption[] expectedOptions = {BigQuery.DatasetOption.fields()}; - expect(bigquery.getDataset(DATASET_ID, expectedOptions)).andReturn(DATASET_INFO); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId(), expectedOptions)) + .andReturn(expectedDataset); replay(bigquery); + initializeDataset(); assertTrue(dataset.exists()); } @Test public void testExists_False() throws Exception { + initializeExpectedDataset(1); BigQuery.DatasetOption[] expectedOptions = {BigQuery.DatasetOption.fields()}; - expect(bigquery.getDataset(DATASET_ID, expectedOptions)).andReturn(null); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId(), expectedOptions)).andReturn(null); replay(bigquery); + initializeDataset(); assertFalse(dataset.exists()); } @Test public void testReload() throws Exception { + initializeExpectedDataset(4); DatasetInfo updatedInfo = DATASET_INFO.toBuilder().description("Description").build(); - expect(bigquery.getDataset(DATASET_ID.dataset())).andReturn(updatedInfo); + Dataset expectedDataset = + new Dataset(serviceMockReturnsOptions, new DatasetInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset())).andReturn(expectedDataset); replay(bigquery); + initializeDataset(); Dataset updatedDataset = dataset.reload(); - assertSame(bigquery, updatedDataset.bigquery()); - assertEquals(updatedInfo, updatedDataset.info()); + compareDataset(expectedDataset, updatedDataset); } @Test public void testReloadNull() throws Exception { - expect(bigquery.getDataset(DATASET_ID.dataset())).andReturn(null); + initializeExpectedDataset(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset())).andReturn(null); replay(bigquery); + initializeDataset(); assertNull(dataset.reload()); } @Test public void testReloadWithOptions() throws Exception { + initializeExpectedDataset(4); DatasetInfo updatedInfo = DATASET_INFO.toBuilder().description("Description").build(); - expect(bigquery.getDataset(DATASET_ID.dataset(), BigQuery.DatasetOption.fields())) - .andReturn(updatedInfo); + Dataset expectedDataset = + new Dataset(serviceMockReturnsOptions, new DatasetInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset(), BigQuery.DatasetOption.fields())) + .andReturn(expectedDataset); replay(bigquery); + initializeDataset(); Dataset updatedDataset = dataset.reload(BigQuery.DatasetOption.fields()); - assertSame(bigquery, updatedDataset.bigquery()); - assertEquals(updatedInfo, updatedDataset.info()); + compareDataset(expectedDataset, updatedDataset); } @Test - public void testUpdate() throws Exception { - DatasetInfo updatedInfo = DATASET_INFO.toBuilder().description("Description").build(); - expect(bigquery.update(updatedInfo)).andReturn(updatedInfo); + public void testUpdate() { + initializeExpectedDataset(4); + Dataset expectedUpdatedDataset = expectedDataset.toBuilder().description("Description").build(); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.update(eq(expectedDataset))).andReturn(expectedUpdatedDataset); replay(bigquery); - Dataset updatedDataset = dataset.update(updatedInfo); - assertSame(bigquery, updatedDataset.bigquery()); - assertEquals(updatedInfo, updatedDataset.info()); + initializeDataset(); + Dataset actualUpdatedDataset = dataset.update(); + compareDataset(expectedUpdatedDataset, actualUpdatedDataset); } @Test - public void testUpdateWithDifferentId() throws Exception { - DatasetInfo updatedInfo = DATASET_INFO.toBuilder() - .datasetId(DatasetId.of("dataset2")) - .description("Description") - .build(); + public void testUpdateWithOptions() { + initializeExpectedDataset(4); + Dataset expectedUpdatedDataset = expectedDataset.toBuilder().description("Description").build(); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.update(eq(expectedDataset), eq(BigQuery.DatasetOption.fields()))) + .andReturn(expectedUpdatedDataset); replay(bigquery); - thrown.expect(IllegalArgumentException.class); - dataset.update(updatedInfo); + initializeDataset(); + Dataset actualUpdatedDataset = dataset.update(BigQuery.DatasetOption.fields()); + compareDataset(expectedUpdatedDataset, actualUpdatedDataset); } @Test - public void testUpdateWithOptions() throws Exception { - DatasetInfo updatedInfo = DATASET_INFO.toBuilder().description("Description").build(); - expect(bigquery.update(updatedInfo, BigQuery.DatasetOption.fields())).andReturn(updatedInfo); + public void testDeleteTrue() { + initializeExpectedDataset(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.delete(DATASET_INFO.datasetId())).andReturn(true); replay(bigquery); - Dataset updatedDataset = dataset.update(updatedInfo, BigQuery.DatasetOption.fields()); - assertSame(bigquery, updatedDataset.bigquery()); - assertEquals(updatedInfo, updatedDataset.info()); + initializeDataset(); + assertTrue(dataset.delete()); } @Test - public void testDelete() throws Exception { - expect(bigquery.delete(DATASET_INFO.datasetId())).andReturn(true); + public void testDeleteFalse() { + initializeExpectedDataset(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.delete(DATASET_INFO.datasetId())).andReturn(false); replay(bigquery); - assertTrue(dataset.delete()); + initializeDataset(); + assertFalse(dataset.delete()); } @Test public void testList() throws Exception { - BigQueryOptions bigqueryOptions = createStrictMock(BigQueryOptions.class); - PageImpl tableInfoPage = new PageImpl<>(null, "c", TABLE_INFO_RESULTS); - expect(bigquery.listTables(DATASET_INFO.datasetId())).andReturn(tableInfoPage); - expect(bigquery.options()).andReturn(bigqueryOptions); - expect(bigqueryOptions.service()).andReturn(bigquery); - replay(bigquery, bigqueryOptions); + initializeExpectedDataset(4); + List
tableResults = ImmutableList.of( + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO1)), + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO2)), + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO3))); + PageImpl
expectedPage = new PageImpl<>(null, "c", tableResults); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.listTables(DATASET_INFO.datasetId())).andReturn(expectedPage); + replay(bigquery); + initializeDataset(); Page
tablePage = dataset.list(); - Iterator tableInfoIterator = tableInfoPage.values().iterator(); - Iterator
tableIterator = tablePage.values().iterator(); - while (tableInfoIterator.hasNext() && tableIterator.hasNext()) { - assertEquals(tableInfoIterator.next(), tableIterator.next().info()); - } - assertFalse(tableInfoIterator.hasNext()); - assertFalse(tableIterator.hasNext()); - assertEquals(tableInfoPage.nextPageCursor(), tablePage.nextPageCursor()); - verify(bigqueryOptions); + assertArrayEquals(tableResults.toArray(), Iterables.toArray(tablePage.values(), Table.class)); + assertEquals(expectedPage.nextPageCursor(), tablePage.nextPageCursor()); } @Test public void testListWithOptions() throws Exception { - BigQueryOptions bigqueryOptions = createStrictMock(BigQueryOptions.class); - PageImpl tableInfoPage = new PageImpl<>(null, "c", TABLE_INFO_RESULTS); + initializeExpectedDataset(4); + List
tableResults = ImmutableList.of( + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO1)), + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO2)), + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO3))); + PageImpl
expectedPage = new PageImpl<>(null, "c", tableResults); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.listTables(DATASET_INFO.datasetId(), BigQuery.TableListOption.maxResults(10L))) - .andReturn(tableInfoPage); - expect(bigquery.options()).andReturn(bigqueryOptions); - expect(bigqueryOptions.service()).andReturn(bigquery); - replay(bigquery, bigqueryOptions); + .andReturn(expectedPage); + replay(bigquery); + initializeDataset(); Page
tablePage = dataset.list(BigQuery.TableListOption.maxResults(10L)); - Iterator tableInfoIterator = tableInfoPage.values().iterator(); - Iterator
tableIterator = tablePage.values().iterator(); - while (tableInfoIterator.hasNext() && tableIterator.hasNext()) { - assertEquals(tableInfoIterator.next(), tableIterator.next().info()); - } - assertFalse(tableInfoIterator.hasNext()); - assertFalse(tableIterator.hasNext()); - assertEquals(tableInfoPage.nextPageCursor(), tablePage.nextPageCursor()); - verify(bigqueryOptions); + assertArrayEquals(tableResults.toArray(), Iterables.toArray(tablePage.values(), Table.class)); + assertEquals(expectedPage.nextPageCursor(), tablePage.nextPageCursor()); } @Test public void testGet() throws Exception { - BaseTableInfo info = TableInfo.builder(TableId.of("dataset", "table1"), Schema.of()).build(); - expect(bigquery.getTable(TableId.of("dataset", "table1"))).andReturn(info); + initializeExpectedDataset(2); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO1)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO1.tableId())).andReturn(expectedTable); replay(bigquery); - Table table = dataset.get("table1"); + initializeDataset(); + Table table = dataset.get(TABLE_INFO1.tableId().table()); assertNotNull(table); - assertEquals(info, table.info()); + assertEquals(expectedTable, table); } @Test public void testGetNull() throws Exception { - expect(bigquery.getTable(TableId.of("dataset", "table1"))).andReturn(null); + initializeExpectedDataset(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO1.tableId())).andReturn(null); replay(bigquery); - assertNull(dataset.get("table1")); + initializeDataset(); + assertNull(dataset.get(TABLE_INFO1.tableId().table())); } @Test public void testGetWithOptions() throws Exception { - BaseTableInfo info = TableInfo.builder(TableId.of("dataset", "table1"), Schema.of()).build(); - expect(bigquery.getTable(TableId.of("dataset", "table1"), BigQuery.TableOption.fields())) - .andReturn(info); + initializeExpectedDataset(2); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO1)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO1.tableId(), BigQuery.TableOption.fields())) + .andReturn(expectedTable); replay(bigquery); - Table table = dataset.get("table1", BigQuery.TableOption.fields()); + initializeDataset(); + Table table = dataset.get(TABLE_INFO1.tableId().table(), BigQuery.TableOption.fields()); assertNotNull(table); - assertEquals(info, table.info()); + assertEquals(expectedTable, table); } @Test public void testCreateTable() throws Exception { - TableInfo info = TableInfo.builder(TableId.of("dataset", "table1"), Schema.of(FIELD)).build(); - expect(bigquery.create(info)).andReturn(info); + initializeExpectedDataset(2); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO1)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.create(TABLE_INFO1)).andReturn(expectedTable); replay(bigquery); - Table table = dataset.create("table1", Schema.of(FIELD)); - assertEquals(info, table.info()); + initializeDataset(); + Table table = dataset.create(TABLE_INFO1.tableId().table(), TABLE_DEFINITION); + assertEquals(expectedTable, table); } @Test public void testCreateTableWithOptions() throws Exception { - TableInfo info = TableInfo.builder(TableId.of("dataset", "table1"), Schema.of(FIELD)).build(); - expect(bigquery.create(info, BigQuery.TableOption.fields())).andReturn(info); - replay(bigquery); - Table table = dataset.create("table1", Schema.of(FIELD), BigQuery.TableOption.fields()); - assertEquals(info, table.info()); - } - - @Test - public void testCreateView() throws Exception { - ViewInfo info = ViewInfo.builder(TableId.of("dataset", "table2"), "QUERY").build(); - expect(bigquery.create(info)).andReturn(info); - replay(bigquery); - Table table = dataset.create("table2", "QUERY"); - assertEquals(info, table.info()); - } - - @Test - public void testCreateViewWithUserDefinedFunctions() throws Exception { - ViewInfo info = ViewInfo.builder(TableId.of("dataset", "table2"), "QUERY", FUNCTIONS).build(); - expect(bigquery.create(info)).andReturn(info); + initializeExpectedDataset(2); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO1)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.create(TABLE_INFO1, BigQuery.TableOption.fields())).andReturn(expectedTable); replay(bigquery); - Table table = dataset.create("table2", "QUERY", FUNCTIONS); - assertEquals(info, table.info()); + initializeDataset(); + Table table = dataset.create(TABLE_INFO1.tableId().table(), TABLE_DEFINITION, + BigQuery.TableOption.fields()); + assertEquals(expectedTable, table); } @Test - public void testCreateViewWithOptions() throws Exception { - ViewInfo info = ViewInfo.builder(TableId.of("dataset", "table2"), "QUERY").build(); - expect(bigquery.create(info, BigQuery.TableOption.fields())).andReturn(info); + public void testBigquery() { + initializeExpectedDataset(1); replay(bigquery); - Table table = dataset.create("table2", "QUERY", BigQuery.TableOption.fields()); - assertEquals(info, table.info()); + assertSame(serviceMockReturnsOptions, expectedDataset.bigquery()); } @Test - public void testCreateExternalTable() throws Exception { - ExternalTableInfo info = ExternalTableInfo.builder(TableId.of("dataset", "table3"), - ExternalDataConfiguration.of(ImmutableList.of("URI"), Schema.of(), FormatOptions.csv())) - .build(); - expect(bigquery.create(info)).andReturn(info); + public void testToAndFromPb() { + initializeExpectedDataset(4); replay(bigquery); - Table table = dataset.create("table3", ExternalDataConfiguration.of( - ImmutableList.of("URI"), Schema.of(), FormatOptions.csv())); - assertEquals(info, table.info()); + compareDataset(expectedDataset, + Dataset.fromPb(serviceMockReturnsOptions, expectedDataset.toPb())); } - @Test - public void testCreateExternalTableWithOptions() throws Exception { - ExternalTableInfo info = ExternalTableInfo.builder(TableId.of("dataset", "table3"), - ExternalDataConfiguration.of(ImmutableList.of("URI"), Schema.of(), FormatOptions.csv())) - .build(); - expect(bigquery.create(info, BigQuery.TableOption.fields())).andReturn(info); - replay(bigquery); - Table table = dataset.create("table3", ExternalDataConfiguration.of( - ImmutableList.of("URI"), Schema.of(), FormatOptions.csv()), BigQuery.TableOption.fields()); - assertEquals(info, table.info()); + private void compareDataset(Dataset expected, Dataset value) { + assertEquals(expected, value); + compareDatasetInfo(expected, value); + assertEquals(expected.bigquery().options(), value.bigquery().options()); } - @Test - public void testStaticGet() throws Exception { - expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset())).andReturn(DATASET_INFO); - replay(bigquery); - Dataset loadedDataset = Dataset.get(bigquery, DATASET_INFO.datasetId().dataset()); - assertNotNull(loadedDataset); - assertEquals(DATASET_INFO, loadedDataset.info()); - } - - @Test - public void testStaticGetNull() throws Exception { - expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset())).andReturn(null); - replay(bigquery); - assertNull(Dataset.get(bigquery, DATASET_INFO.datasetId().dataset())); - } - - @Test - public void testStaticGetWithOptions() throws Exception { - expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset(), BigQuery.DatasetOption.fields())) - .andReturn(DATASET_INFO); - replay(bigquery); - Dataset loadedDataset = Dataset.get(bigquery, DATASET_INFO.datasetId().dataset(), - BigQuery.DatasetOption.fields()); - assertNotNull(loadedDataset); - assertEquals(DATASET_INFO, loadedDataset.info()); + private void compareDatasetInfo(DatasetInfo expected, DatasetInfo value) { + assertEquals(expected, value); + assertEquals(expected.datasetId(), value.datasetId()); + assertEquals(expected.description(), value.description()); + assertEquals(expected.etag(), value.etag()); + assertEquals(expected.friendlyName(), value.friendlyName()); + assertEquals(expected.id(), value.id()); + assertEquals(expected.location(), value.location()); + assertEquals(expected.selfLink(), value.selfLink()); + assertEquals(expected.acl(), value.acl()); + assertEquals(expected.creationTime(), value.creationTime()); + assertEquals(expected.defaultTableLifetime(), value.defaultTableLifetime()); + assertEquals(expected.lastModified(), value.lastModified()); } } diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExternalDataConfigurationTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExternalTableDefinitionTest.java similarity index 52% rename from gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExternalDataConfigurationTest.java rename to gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExternalTableDefinitionTest.java index f9b7c31e1071..247032dff890 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExternalDataConfigurationTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExternalTableDefinitionTest.java @@ -1,5 +1,5 @@ /* - * Copyright 2015 Google Inc. All Rights Reserved. + * Copyright 2016 Google Inc. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -24,7 +24,7 @@ import java.util.List; -public class ExternalDataConfigurationTest { +public class ExternalTableDefinitionTest { private static final List SOURCE_URIS = ImmutableList.of("uri1", "uri2"); private static final Field FIELD_SCHEMA1 = @@ -47,51 +47,56 @@ public class ExternalDataConfigurationTest { private static final Boolean IGNORE_UNKNOWN_VALUES = true; private static final String COMPRESSION = "GZIP"; private static final CsvOptions CSV_OPTIONS = CsvOptions.builder().build(); - private static final ExternalDataConfiguration CONFIGURATION = ExternalDataConfiguration - .builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS) - .compression(COMPRESSION) - .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) - .maxBadRecords(MAX_BAD_RECORDS) - .build(); + private static final ExternalTableDefinition EXTERNAL_TABLE_DEFINITION = + ExternalTableDefinition.builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS) + .compression(COMPRESSION) + .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) + .maxBadRecords(MAX_BAD_RECORDS) + .build(); @Test public void testToBuilder() { - compareConfiguration(CONFIGURATION, CONFIGURATION.toBuilder().build()); - ExternalDataConfiguration configuration = CONFIGURATION.toBuilder().compression("NONE").build(); - assertEquals("NONE", configuration.compression()); - configuration = configuration.toBuilder() + compareExternalTableDefinition(EXTERNAL_TABLE_DEFINITION, + EXTERNAL_TABLE_DEFINITION.toBuilder().build()); + ExternalTableDefinition externalTableDefinition = + EXTERNAL_TABLE_DEFINITION.toBuilder().compression("NONE").build(); + assertEquals("NONE", externalTableDefinition.compression()); + externalTableDefinition = externalTableDefinition.toBuilder() .compression(COMPRESSION) .build(); - compareConfiguration(CONFIGURATION, configuration); + compareExternalTableDefinition(EXTERNAL_TABLE_DEFINITION, externalTableDefinition); } @Test public void testToBuilderIncomplete() { - ExternalDataConfiguration configuration = - ExternalDataConfiguration.of(SOURCE_URIS, TABLE_SCHEMA, FormatOptions.json()); - assertEquals(configuration, configuration.toBuilder().build()); + ExternalTableDefinition externalTableDefinition = + ExternalTableDefinition.of(SOURCE_URIS, TABLE_SCHEMA, FormatOptions.json()); + assertEquals(externalTableDefinition, externalTableDefinition.toBuilder().build()); } @Test public void testBuilder() { - assertEquals(COMPRESSION, CONFIGURATION.compression()); - assertEquals(CSV_OPTIONS, CONFIGURATION.formatOptions()); - assertEquals(IGNORE_UNKNOWN_VALUES, CONFIGURATION.ignoreUnknownValues()); - assertEquals(MAX_BAD_RECORDS, CONFIGURATION.maxBadRecords()); - assertEquals(TABLE_SCHEMA, CONFIGURATION.schema()); - assertEquals(SOURCE_URIS, CONFIGURATION.sourceUris()); + assertEquals(TableDefinition.Type.EXTERNAL, EXTERNAL_TABLE_DEFINITION.type()); + assertEquals(COMPRESSION, EXTERNAL_TABLE_DEFINITION.compression()); + assertEquals(CSV_OPTIONS, EXTERNAL_TABLE_DEFINITION.formatOptions()); + assertEquals(IGNORE_UNKNOWN_VALUES, EXTERNAL_TABLE_DEFINITION.ignoreUnknownValues()); + assertEquals(MAX_BAD_RECORDS, EXTERNAL_TABLE_DEFINITION.maxBadRecords()); + assertEquals(TABLE_SCHEMA, EXTERNAL_TABLE_DEFINITION.schema()); + assertEquals(SOURCE_URIS, EXTERNAL_TABLE_DEFINITION.sourceUris()); } @Test public void testToAndFromPb() { - compareConfiguration(CONFIGURATION, ExternalDataConfiguration.fromPb(CONFIGURATION.toPb())); - ExternalDataConfiguration configuration = - ExternalDataConfiguration.builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS).build(); - compareConfiguration(configuration, ExternalDataConfiguration.fromPb(configuration.toPb())); + compareExternalTableDefinition(EXTERNAL_TABLE_DEFINITION, + ExternalTableDefinition.fromPb(EXTERNAL_TABLE_DEFINITION.toPb())); + ExternalTableDefinition externalTableDefinition = + ExternalTableDefinition.builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS).build(); + compareExternalTableDefinition(externalTableDefinition, + ExternalTableDefinition.fromPb(externalTableDefinition.toPb())); } - private void compareConfiguration(ExternalDataConfiguration expected, - ExternalDataConfiguration value) { + private void compareExternalTableDefinition(ExternalTableDefinition expected, + ExternalTableDefinition value) { assertEquals(expected, value); assertEquals(expected.compression(), value.compression()); assertEquals(expected.formatOptions(), value.formatOptions()); @@ -99,5 +104,6 @@ private void compareConfiguration(ExternalDataConfiguration expected, assertEquals(expected.maxBadRecords(), value.maxBadRecords()); assertEquals(expected.schema(), value.schema()); assertEquals(expected.sourceUris(), value.sourceUris()); + assertEquals(expected.hashCode(), value.hashCode()); } } diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExtractJobConfigurationTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExtractJobConfigurationTest.java new file mode 100644 index 000000000000..7ac67f41b1f8 --- /dev/null +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExtractJobConfigurationTest.java @@ -0,0 +1,139 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; + +import com.google.common.collect.ImmutableList; + +import org.junit.Test; + +import java.util.List; + +public class ExtractJobConfigurationTest { + + private static final List DESTINATION_URIS = ImmutableList.of("uri1", "uri2"); + private static final String DESTINATION_URI = "uri1"; + private static final TableId TABLE_ID = TableId.of("dataset", "table"); + private static final String FIELD_DELIMITER = ","; + private static final String FORMAT = "CSV"; + private static final String JSON_FORMAT = "NEWLINE_DELIMITED_JSON"; + private static final Boolean PRINT_HEADER = true; + private static final String COMPRESSION = "GZIP"; + private static final ExtractJobConfiguration EXTRACT_CONFIGURATION = + ExtractJobConfiguration.builder(TABLE_ID, DESTINATION_URIS) + .printHeader(PRINT_HEADER) + .fieldDelimiter(FIELD_DELIMITER) + .compression(COMPRESSION) + .format(FORMAT) + .build(); + private static final ExtractJobConfiguration EXTRACT_CONFIGURATION_ONE_URI = + ExtractJobConfiguration.builder(TABLE_ID, DESTINATION_URI) + .printHeader(PRINT_HEADER) + .fieldDelimiter(FIELD_DELIMITER) + .compression(COMPRESSION) + .format(FORMAT) + .build(); + + @Test + public void testToBuilder() { + compareExtractJobConfiguration( + EXTRACT_CONFIGURATION, EXTRACT_CONFIGURATION.toBuilder().build()); + ExtractJobConfiguration job = EXTRACT_CONFIGURATION.toBuilder() + .sourceTable(TableId.of("dataset", "newTable")) + .build(); + assertEquals("newTable", job.sourceTable().table()); + job = job.toBuilder().sourceTable(TABLE_ID).build(); + compareExtractJobConfiguration(EXTRACT_CONFIGURATION, job); + } + + @Test + public void testOf() { + ExtractJobConfiguration job = ExtractJobConfiguration.of(TABLE_ID, DESTINATION_URIS); + assertEquals(TABLE_ID, job.sourceTable()); + assertEquals(DESTINATION_URIS, job.destinationUris()); + job = ExtractJobConfiguration.of(TABLE_ID, DESTINATION_URI); + assertEquals(TABLE_ID, job.sourceTable()); + assertEquals(ImmutableList.of(DESTINATION_URI), job.destinationUris()); + job = ExtractJobConfiguration.of(TABLE_ID, DESTINATION_URIS, JSON_FORMAT); + assertEquals(TABLE_ID, job.sourceTable()); + assertEquals(DESTINATION_URIS, job.destinationUris()); + assertEquals(JSON_FORMAT, job.format()); + job = ExtractJobConfiguration.of(TABLE_ID, DESTINATION_URI, JSON_FORMAT); + assertEquals(TABLE_ID, job.sourceTable()); + assertEquals(ImmutableList.of(DESTINATION_URI), job.destinationUris()); + assertEquals(JSON_FORMAT, job.format()); + } + + @Test + public void testToBuilderIncomplete() { + ExtractJobConfiguration job = ExtractJobConfiguration.of(TABLE_ID, DESTINATION_URIS); + compareExtractJobConfiguration(job, job.toBuilder().build()); + } + + @Test + public void testBuilder() { + assertEquals(TABLE_ID, EXTRACT_CONFIGURATION.sourceTable()); + assertEquals(DESTINATION_URIS, EXTRACT_CONFIGURATION.destinationUris()); + assertEquals(FIELD_DELIMITER, EXTRACT_CONFIGURATION.fieldDelimiter()); + assertEquals(COMPRESSION, EXTRACT_CONFIGURATION.compression()); + assertEquals(PRINT_HEADER, EXTRACT_CONFIGURATION.printHeader()); + assertEquals(FORMAT, EXTRACT_CONFIGURATION.format()); + assertEquals(TABLE_ID, EXTRACT_CONFIGURATION_ONE_URI.sourceTable()); + assertEquals(ImmutableList.of(DESTINATION_URI), + EXTRACT_CONFIGURATION_ONE_URI.destinationUris()); + assertEquals(FIELD_DELIMITER, EXTRACT_CONFIGURATION_ONE_URI.fieldDelimiter()); + assertEquals(COMPRESSION, EXTRACT_CONFIGURATION_ONE_URI.compression()); + assertEquals(PRINT_HEADER, EXTRACT_CONFIGURATION_ONE_URI.printHeader()); + assertEquals(FORMAT, EXTRACT_CONFIGURATION_ONE_URI.format()); + } + + @Test + public void testToPbAndFromPb() { + assertNotNull(EXTRACT_CONFIGURATION.toPb().getExtract()); + assertNull(EXTRACT_CONFIGURATION.toPb().getCopy()); + assertNull(EXTRACT_CONFIGURATION.toPb().getLoad()); + assertNull(EXTRACT_CONFIGURATION.toPb().getQuery()); + compareExtractJobConfiguration(EXTRACT_CONFIGURATION, + ExtractJobConfiguration.fromPb(EXTRACT_CONFIGURATION.toPb())); + compareExtractJobConfiguration(EXTRACT_CONFIGURATION_ONE_URI, + ExtractJobConfiguration.fromPb(EXTRACT_CONFIGURATION_ONE_URI.toPb())); + ExtractJobConfiguration job = ExtractJobConfiguration.of(TABLE_ID, DESTINATION_URIS); + compareExtractJobConfiguration(job, ExtractJobConfiguration.fromPb(job.toPb())); + } + + @Test + public void testSetProjectId() { + ExtractJobConfiguration configuration = EXTRACT_CONFIGURATION.setProjectId("p"); + assertEquals("p", configuration.sourceTable().project()); + } + + private void compareExtractJobConfiguration(ExtractJobConfiguration expected, + ExtractJobConfiguration value) { + assertEquals(expected, value); + assertEquals(expected.hashCode(), value.hashCode()); + assertEquals(expected.toString(), value.toString()); + assertEquals(expected.sourceTable(), value.sourceTable()); + assertEquals(expected.destinationUris(), value.destinationUris()); + assertEquals(expected.compression(), value.compression()); + assertEquals(expected.printHeader(), value.printHeader()); + assertEquals(expected.fieldDelimiter(), value.fieldDelimiter()); + assertEquals(expected.format(), value.format()); + } +} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExtractJobInfoTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExtractJobInfoTest.java deleted file mode 100644 index bb47112b6410..000000000000 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ExtractJobInfoTest.java +++ /dev/null @@ -1,201 +0,0 @@ -/* - * Copyright 2015 Google Inc. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.google.gcloud.bigquery; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNotNull; -import static org.junit.Assert.assertNull; - -import com.google.common.collect.ImmutableList; -import com.google.gcloud.bigquery.JobStatistics.ExtractStatistics; - -import org.junit.Test; - -import java.util.List; - -public class ExtractJobInfoTest { - - private static final String ETAG = "etag"; - private static final String ID = "id"; - private static final String SELF_LINK = "selfLink"; - private static final String EMAIL = "email"; - private static final List DESTINATION_URIS = ImmutableList.of("uri1", "uri2"); - private static final String DESTINATION_URI = "uri1"; - private static final TableId TABLE_ID = TableId.of("dataset", "table"); - private static final String FIELD_DELIMITER = ","; - private static final String FORMAT = "CSV"; - private static final String JSON_FORMAT = "NEWLINE_DELIMITED_JSON"; - private static final Boolean PRINT_HEADER = true; - private static final String COMPRESSION = "GZIP"; - private static final JobId JOB_ID = JobId.of("job"); - private static final JobStatus JOB_STATUS = new JobStatus(JobStatus.State.DONE); - private static final ExtractStatistics JOB_STATISTICS = ExtractStatistics.builder() - .creationTime(1L) - .endTime(3L) - .startTime(2L) - .destinationUriFileCounts(ImmutableList.of(42L)) - .build(); - private static final ExtractJobInfo EXTRACT_JOB = - ExtractJobInfo.builder(TABLE_ID, DESTINATION_URIS) - .etag(ETAG) - .id(ID) - .selfLink(SELF_LINK) - .userEmail(EMAIL) - .jobId(JOB_ID) - .status(JOB_STATUS) - .printHeader(PRINT_HEADER) - .fieldDelimiter(FIELD_DELIMITER) - .compression(COMPRESSION) - .format(FORMAT) - .statistics(JOB_STATISTICS) - .build(); - private static final ExtractJobInfo EXTRACT_JOB_ONE_URI = - ExtractJobInfo.builder(TABLE_ID, DESTINATION_URI) - .etag(ETAG) - .id(ID) - .selfLink(SELF_LINK) - .userEmail(EMAIL) - .jobId(JOB_ID) - .status(JOB_STATUS) - .printHeader(PRINT_HEADER) - .fieldDelimiter(FIELD_DELIMITER) - .compression(COMPRESSION) - .format(FORMAT) - .build(); - - @Test - public void testToBuilder() { - compareExtractJobInfo(EXTRACT_JOB, EXTRACT_JOB.toBuilder().build()); - ExtractJobInfo job = EXTRACT_JOB.toBuilder() - .sourceTable(TableId.of("dataset", "newTable")) - .build(); - assertEquals("newTable", job.sourceTable().table()); - job = job.toBuilder().sourceTable(TABLE_ID).build(); - compareExtractJobInfo(EXTRACT_JOB, job); - } - - @Test - public void testOf() { - ExtractJobInfo job = ExtractJobInfo.of(TABLE_ID, DESTINATION_URIS); - assertEquals(TABLE_ID, job.sourceTable()); - assertEquals(DESTINATION_URIS, job.destinationUris()); - job = ExtractJobInfo.of(TABLE_ID, DESTINATION_URI); - assertEquals(TABLE_ID, job.sourceTable()); - assertEquals(ImmutableList.of(DESTINATION_URI), job.destinationUris()); - job = ExtractJobInfo.of(TABLE_ID, JSON_FORMAT, DESTINATION_URIS); - assertEquals(TABLE_ID, job.sourceTable()); - assertEquals(DESTINATION_URIS, job.destinationUris()); - assertEquals(JSON_FORMAT, job.format()); - job = ExtractJobInfo.of(TABLE_ID, JSON_FORMAT, DESTINATION_URI); - assertEquals(TABLE_ID, job.sourceTable()); - assertEquals(ImmutableList.of(DESTINATION_URI), job.destinationUris()); - assertEquals(JSON_FORMAT, job.format()); - job = ExtractJobInfo.of(JOB_ID, TABLE_ID, DESTINATION_URIS); - assertEquals(JOB_ID, job.jobId()); - assertEquals(TABLE_ID, job.sourceTable()); - assertEquals(DESTINATION_URIS, job.destinationUris()); - job = ExtractJobInfo.of(JOB_ID, TABLE_ID, DESTINATION_URI); - assertEquals(JOB_ID, job.jobId()); - assertEquals(TABLE_ID, job.sourceTable()); - assertEquals(ImmutableList.of(DESTINATION_URI), job.destinationUris()); - job = ExtractJobInfo.of(JOB_ID, TABLE_ID, JSON_FORMAT, DESTINATION_URIS); - assertEquals(JOB_ID, job.jobId()); - assertEquals(TABLE_ID, job.sourceTable()); - assertEquals(DESTINATION_URIS, job.destinationUris()); - assertEquals(JSON_FORMAT, job.format()); - job = ExtractJobInfo.of(JOB_ID, TABLE_ID, JSON_FORMAT, DESTINATION_URI); - assertEquals(JOB_ID, job.jobId()); - assertEquals(TABLE_ID, job.sourceTable()); - assertEquals(ImmutableList.of(DESTINATION_URI), job.destinationUris()); - assertEquals(JSON_FORMAT, job.format()); - } - - @Test - public void testToBuilderIncomplete() { - ExtractJobInfo job = ExtractJobInfo.of(TABLE_ID, DESTINATION_URIS); - compareExtractJobInfo(job, job.toBuilder().build()); - } - - @Test - public void testBuilder() { - assertEquals(ETAG, EXTRACT_JOB.etag()); - assertEquals(ID, EXTRACT_JOB.id()); - assertEquals(SELF_LINK, EXTRACT_JOB.selfLink()); - assertEquals(EMAIL, EXTRACT_JOB.userEmail()); - assertEquals(JOB_ID, EXTRACT_JOB.jobId()); - assertEquals(JOB_STATUS, EXTRACT_JOB.status()); - assertEquals(TABLE_ID, EXTRACT_JOB.sourceTable()); - assertEquals(DESTINATION_URIS, EXTRACT_JOB.destinationUris()); - assertEquals(FIELD_DELIMITER, EXTRACT_JOB.fieldDelimiter()); - assertEquals(COMPRESSION, EXTRACT_JOB.compression()); - assertEquals(PRINT_HEADER, EXTRACT_JOB.printHeader()); - assertEquals(FORMAT, EXTRACT_JOB.format()); - assertEquals(JOB_STATISTICS, EXTRACT_JOB.statistics()); - assertEquals(ETAG, EXTRACT_JOB_ONE_URI.etag()); - assertEquals(ID, EXTRACT_JOB_ONE_URI.id()); - assertEquals(SELF_LINK, EXTRACT_JOB_ONE_URI.selfLink()); - assertEquals(EMAIL, EXTRACT_JOB_ONE_URI.userEmail()); - assertEquals(JOB_ID, EXTRACT_JOB_ONE_URI.jobId()); - assertEquals(JOB_STATUS, EXTRACT_JOB_ONE_URI.status()); - assertEquals(TABLE_ID, EXTRACT_JOB_ONE_URI.sourceTable()); - assertEquals(ImmutableList.of(DESTINATION_URI), - EXTRACT_JOB_ONE_URI.destinationUris()); - assertEquals(FIELD_DELIMITER, EXTRACT_JOB_ONE_URI.fieldDelimiter()); - assertEquals(COMPRESSION, EXTRACT_JOB_ONE_URI.compression()); - assertEquals(PRINT_HEADER, EXTRACT_JOB_ONE_URI.printHeader()); - assertEquals(FORMAT, EXTRACT_JOB_ONE_URI.format()); - } - - @Test - public void testToPbAndFromPb() { - assertNotNull(EXTRACT_JOB.toPb().getConfiguration().getExtract()); - assertNull(EXTRACT_JOB.toPb().getConfiguration().getCopy()); - assertNull(EXTRACT_JOB.toPb().getConfiguration().getLoad()); - assertNull(EXTRACT_JOB.toPb().getConfiguration().getQuery()); - assertEquals(JOB_STATISTICS, JobStatistics.fromPb(EXTRACT_JOB.toPb().getStatistics())); - compareExtractJobInfo(EXTRACT_JOB, - ExtractJobInfo.fromPb(EXTRACT_JOB.toPb())); - compareExtractJobInfo(EXTRACT_JOB, - (ExtractJobInfo) JobInfo.fromPb(EXTRACT_JOB.toPb())); - compareExtractJobInfo(EXTRACT_JOB_ONE_URI, - ExtractJobInfo.fromPb(EXTRACT_JOB_ONE_URI.toPb())); - compareExtractJobInfo(EXTRACT_JOB_ONE_URI, - (ExtractJobInfo) JobInfo.fromPb(EXTRACT_JOB_ONE_URI.toPb())); - ExtractJobInfo job = ExtractJobInfo.of(TABLE_ID, DESTINATION_URIS); - compareExtractJobInfo(job, ExtractJobInfo.fromPb(job.toPb())); - compareExtractJobInfo(job, (ExtractJobInfo) JobInfo.fromPb(job.toPb())); - } - - private void compareExtractJobInfo(ExtractJobInfo expected, ExtractJobInfo value) { - assertEquals(expected, value); - assertEquals(expected.hashCode(), value.hashCode()); - assertEquals(expected.toString(), value.toString()); - assertEquals(expected.etag(), value.etag()); - assertEquals(expected.id(), value.id()); - assertEquals(expected.jobId(), value.jobId()); - assertEquals(expected.selfLink(), value.selfLink()); - assertEquals(expected.status(), value.status()); - assertEquals(expected.statistics(), value.statistics()); - assertEquals(expected.userEmail(), value.userEmail()); - assertEquals(expected.sourceTable(), value.sourceTable()); - assertEquals(expected.destinationUris(), value.destinationUris()); - assertEquals(expected.compression(), value.compression()); - assertEquals(expected.printHeader(), value.printHeader()); - assertEquals(expected.fieldDelimiter(), value.fieldDelimiter()); - assertEquals(expected.format(), value.format()); - } -} \ No newline at end of file diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/InsertAllRequestTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/InsertAllRequestTest.java index d2e1de14a571..0866f0b9349e 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/InsertAllRequestTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/InsertAllRequestTest.java @@ -43,7 +43,8 @@ public class InsertAllRequestTest { InsertAllRequest.RowToInsert.of("id2", CONTENT2)); private static final TableId TABLE_ID = TableId.of("dataset", "table"); private static final Schema TABLE_SCHEMA = Schema.of(); - private static final BaseTableInfo TABLE_INFO = TableInfo.of(TABLE_ID, TABLE_SCHEMA); + private static final TableDefinition TABLE_DEFINITION = StandardTableDefinition.of(TABLE_SCHEMA); + private static final TableInfo TABLE_INFO = TableInfo.of(TABLE_ID, TABLE_DEFINITION); private static final boolean SKIP_INVALID_ROWS = true; private static final boolean IGNORE_UNKNOWN_VALUES = false; private static final String TEMPLATE_SUFFIX = "templateSuffix"; diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobInfoTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobInfoTest.java new file mode 100644 index 000000000000..260088470aff --- /dev/null +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobInfoTest.java @@ -0,0 +1,370 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.gcloud.bigquery.JobInfo.CreateDisposition; +import com.google.gcloud.bigquery.JobInfo.WriteDisposition; +import com.google.gcloud.bigquery.JobStatistics.ExtractStatistics; +import com.google.gcloud.bigquery.JobStatistics.LoadStatistics; +import com.google.gcloud.bigquery.JobStatistics.QueryStatistics; + +import org.junit.Test; + +import java.util.List; +import java.util.Map; + +public class JobInfoTest { + + private static final String ETAG = "etag"; + private static final String ID = "id"; + private static final String SELF_LINK = "selfLink"; + private static final String EMAIL = "email"; + private static final JobId JOB_ID = JobId.of("job"); + private static final JobStatus JOB_STATUS = new JobStatus(JobStatus.State.DONE); + private static final JobStatistics COPY_JOB_STATISTICS = JobStatistics.builder() + .creationTime(1L) + .endTime(3L) + .startTime(2L) + .build(); + private static final ExtractStatistics EXTRACT_JOB_STATISTICS = + ExtractStatistics.builder() + .creationTime(1L) + .endTime(3L) + .startTime(2L) + .destinationUriFileCounts(ImmutableList.of(42L)) + .build(); + private static final LoadStatistics LOAD_JOB_STATISTICS = + LoadStatistics.builder() + .creationTime(1L) + .endTime(3L) + .startTime(2L) + .inputFiles(42L) + .outputBytes(1024L) + .inputBytes(2048L) + .outputRows(24L) + .build(); + private static final QueryStatistics QUERY_JOB_STATISTICS = + QueryStatistics.builder() + .creationTime(1L) + .endTime(3L) + .startTime(2L) + .totalBytesProcessed(2048L) + .totalBytesBilled(1024L) + .cacheHit(false) + .billingTier(42) + .build(); + private static final TableId SOURCE_TABLE = TableId.of("dataset", "sourceTable"); + private static final TableId DESTINATION_TABLE = TableId.of("dataset", "destinationTable"); + private static final CreateDisposition CREATE_DISPOSITION = CreateDisposition.CREATE_IF_NEEDED; + private static final WriteDisposition WRITE_DISPOSITION = WriteDisposition.WRITE_APPEND; + private static final CopyJobConfiguration COPY_CONFIGURATION = + CopyJobConfiguration.builder(DESTINATION_TABLE, SOURCE_TABLE) + .createDisposition(CREATE_DISPOSITION) + .writeDisposition(WRITE_DISPOSITION) + .build(); + private static final List DESTINATION_URIS = ImmutableList.of("uri1", "uri2"); + private static final TableId TABLE_ID = TableId.of("dataset", "table"); + private static final DatasetId DATASET_ID = DatasetId.of("dataset"); + private static final List SOURCE_URIS = ImmutableList.of("uri1", "uri2"); + private static final Field FIELD_SCHEMA1 = + Field.builder("StringField", Field.Type.string()) + .mode(Field.Mode.NULLABLE) + .description("FieldDescription1") + .build(); + private static final Field FIELD_SCHEMA2 = + Field.builder("IntegerField", Field.Type.integer()) + .mode(Field.Mode.REPEATED) + .description("FieldDescription2") + .build(); + private static final Field FIELD_SCHEMA3 = + Field.builder("RecordField", Field.Type.record(FIELD_SCHEMA1, FIELD_SCHEMA2)) + .mode(Field.Mode.REQUIRED) + .description("FieldDescription3") + .build(); + private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA1, FIELD_SCHEMA2, FIELD_SCHEMA3); + private static final String FIELD_DELIMITER = ","; + private static final String FORMAT = "CSV"; + private static final Boolean PRINT_HEADER = true; + private static final String COMPRESSION = "GZIP"; + private static final ExtractJobConfiguration EXTRACT_CONFIGURATION = + ExtractJobConfiguration.builder(TABLE_ID, DESTINATION_URIS) + .printHeader(PRINT_HEADER) + .fieldDelimiter(FIELD_DELIMITER) + .compression(COMPRESSION) + .format(FORMAT) + .build(); + private static final List PROJECTION_FIELDS = ImmutableList.of("field1", "field2"); + private static final Integer MAX_BAD_RECORDS = 42; + private static final Boolean IGNORE_UNKNOWN_VALUES = true; + private static final CsvOptions CSV_OPTIONS = CsvOptions.builder().build(); + private static final ExternalTableDefinition TABLE_CONFIGURATION = + ExternalTableDefinition.builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS) + .compression(COMPRESSION) + .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) + .maxBadRecords(MAX_BAD_RECORDS) + .build(); + private static final LoadJobConfiguration LOAD_CONFIGURATION = + LoadJobConfiguration.builder(TABLE_ID, SOURCE_URIS) + .createDisposition(CREATE_DISPOSITION) + .writeDisposition(WRITE_DISPOSITION) + .formatOptions(CSV_OPTIONS) + .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) + .maxBadRecords(MAX_BAD_RECORDS) + .projectionFields(PROJECTION_FIELDS) + .schema(TABLE_SCHEMA) + .build(); + private static final String QUERY = "BigQuery SQL"; + private static final Map TABLE_DEFINITIONS = + ImmutableMap.of("tableName", TABLE_CONFIGURATION); + private static final QueryJobConfiguration.Priority PRIORITY = + QueryJobConfiguration.Priority.BATCH; + private static final boolean ALLOW_LARGE_RESULTS = true; + private static final boolean USE_QUERY_CACHE = false; + private static final boolean FLATTEN_RESULTS = true; + private static final List USER_DEFINED_FUNCTIONS = ImmutableList.of( + UserDefinedFunction.inline("Function"), UserDefinedFunction.fromUri("URI")); + private static final QueryJobConfiguration QUERY_CONFIGURATION = + QueryJobConfiguration.builder(QUERY) + .useQueryCache(USE_QUERY_CACHE) + .tableDefinitions(TABLE_DEFINITIONS) + .allowLargeResults(ALLOW_LARGE_RESULTS) + .createDisposition(CREATE_DISPOSITION) + .defaultDataset(DATASET_ID) + .destinationTable(TABLE_ID) + .writeDisposition(WRITE_DISPOSITION) + .priority(PRIORITY) + .flattenResults(FLATTEN_RESULTS) + .userDefinedFunctions(USER_DEFINED_FUNCTIONS) + .dryRun(true) + .build(); + private static final JobInfo COPY_JOB = JobInfo.builder(COPY_CONFIGURATION) + .jobId(JOB_ID) + .statistics(COPY_JOB_STATISTICS) + .jobId(JOB_ID) + .etag(ETAG) + .id(ID) + .selfLink(SELF_LINK) + .userEmail(EMAIL) + .status(JOB_STATUS) + .build(); + private static final JobInfo EXTRACT_JOB = JobInfo.builder(EXTRACT_CONFIGURATION) + .jobId(JOB_ID) + .statistics(EXTRACT_JOB_STATISTICS) + .jobId(JOB_ID) + .etag(ETAG) + .id(ID) + .selfLink(SELF_LINK) + .userEmail(EMAIL) + .status(JOB_STATUS) + .build(); + private static final JobInfo LOAD_JOB = JobInfo.builder(LOAD_CONFIGURATION) + .jobId(JOB_ID) + .statistics(LOAD_JOB_STATISTICS) + .jobId(JOB_ID) + .etag(ETAG) + .id(ID) + .selfLink(SELF_LINK) + .userEmail(EMAIL) + .status(JOB_STATUS) + .build(); + private static final JobInfo QUERY_JOB = JobInfo.builder(QUERY_CONFIGURATION) + .jobId(JOB_ID) + .statistics(QUERY_JOB_STATISTICS) + .jobId(JOB_ID) + .etag(ETAG) + .id(ID) + .selfLink(SELF_LINK) + .userEmail(EMAIL) + .status(JOB_STATUS) + .build(); + + + @Test + public void testToBuilder() { + compareJobInfo(COPY_JOB, COPY_JOB.toBuilder().build()); + compareJobInfo(EXTRACT_JOB, EXTRACT_JOB.toBuilder().build()); + compareJobInfo(LOAD_JOB, LOAD_JOB.toBuilder().build()); + compareJobInfo(QUERY_JOB, QUERY_JOB.toBuilder().build()); + JobInfo job = COPY_JOB.toBuilder() + .userEmail("newEmail") + .build(); + assertEquals("newEmail", job.userEmail()); + job = job.toBuilder().userEmail(EMAIL).build(); + compareJobInfo(COPY_JOB, job); + job = EXTRACT_JOB.toBuilder() + .userEmail("newEmail") + .build(); + assertEquals("newEmail", job.userEmail()); + job = job.toBuilder().userEmail(EMAIL).build(); + compareJobInfo(EXTRACT_JOB, job); + job = LOAD_JOB.toBuilder() + .userEmail("newEmail") + .build(); + assertEquals("newEmail", job.userEmail()); + job = job.toBuilder().userEmail(EMAIL).build(); + compareJobInfo(LOAD_JOB, job); + job = QUERY_JOB.toBuilder() + .userEmail("newEmail") + .build(); + assertEquals("newEmail", job.userEmail()); + job = job.toBuilder().userEmail(EMAIL).build(); + compareJobInfo(QUERY_JOB, job); + } + + @Test + public void testOf() { + JobInfo job = JobInfo.of(COPY_CONFIGURATION); + assertEquals(COPY_CONFIGURATION, job.configuration()); + job = JobInfo.of(EXTRACT_CONFIGURATION); + assertEquals(EXTRACT_CONFIGURATION, job.configuration()); + job = JobInfo.of(LOAD_CONFIGURATION); + assertEquals(LOAD_CONFIGURATION, job.configuration()); + job = JobInfo.of(QUERY_CONFIGURATION); + assertEquals(QUERY_CONFIGURATION, job.configuration()); + job = JobInfo.of(JOB_ID, COPY_CONFIGURATION); + assertEquals(JOB_ID, job.jobId()); + assertEquals(COPY_CONFIGURATION, job.configuration()); + job = JobInfo.of(JOB_ID, EXTRACT_CONFIGURATION); + assertEquals(JOB_ID, job.jobId()); + assertEquals(EXTRACT_CONFIGURATION, job.configuration()); + job = JobInfo.of(JOB_ID, LOAD_CONFIGURATION); + assertEquals(JOB_ID, job.jobId()); + assertEquals(LOAD_CONFIGURATION, job.configuration()); + job = JobInfo.of(JOB_ID, QUERY_CONFIGURATION); + assertEquals(JOB_ID, job.jobId()); + assertEquals(QUERY_CONFIGURATION, job.configuration()); + + } + + @Test + public void testToBuilderIncomplete() { + JobInfo job = JobInfo.of(COPY_CONFIGURATION); + compareJobInfo(job, job.toBuilder().build()); + } + + @Test + public void testBuilder() { + assertEquals(ETAG, COPY_JOB.etag()); + assertEquals(ID, COPY_JOB.id()); + assertEquals(SELF_LINK, COPY_JOB.selfLink()); + assertEquals(EMAIL, COPY_JOB.userEmail()); + assertEquals(JOB_ID, COPY_JOB.jobId()); + assertEquals(JOB_STATUS, COPY_JOB.status()); + assertEquals(COPY_CONFIGURATION, COPY_JOB.configuration()); + assertEquals(COPY_JOB_STATISTICS, COPY_JOB.statistics()); + + assertEquals(ETAG, EXTRACT_JOB.etag()); + assertEquals(ID, EXTRACT_JOB.id()); + assertEquals(SELF_LINK, EXTRACT_JOB.selfLink()); + assertEquals(EMAIL, EXTRACT_JOB.userEmail()); + assertEquals(JOB_ID, EXTRACT_JOB.jobId()); + assertEquals(JOB_STATUS, EXTRACT_JOB.status()); + assertEquals(EXTRACT_CONFIGURATION, EXTRACT_JOB.configuration()); + assertEquals(EXTRACT_JOB_STATISTICS, EXTRACT_JOB.statistics()); + + assertEquals(ETAG, LOAD_JOB.etag()); + assertEquals(ID, LOAD_JOB.id()); + assertEquals(SELF_LINK, LOAD_JOB.selfLink()); + assertEquals(EMAIL, LOAD_JOB.userEmail()); + assertEquals(JOB_ID, LOAD_JOB.jobId()); + assertEquals(JOB_STATUS, LOAD_JOB.status()); + assertEquals(LOAD_CONFIGURATION, LOAD_JOB.configuration()); + assertEquals(LOAD_JOB_STATISTICS, LOAD_JOB.statistics()); + + assertEquals(ETAG, QUERY_JOB.etag()); + assertEquals(ID, QUERY_JOB.id()); + assertEquals(SELF_LINK, QUERY_JOB.selfLink()); + assertEquals(EMAIL, QUERY_JOB.userEmail()); + assertEquals(JOB_ID, QUERY_JOB.jobId()); + assertEquals(JOB_STATUS, QUERY_JOB.status()); + assertEquals(QUERY_CONFIGURATION, QUERY_JOB.configuration()); + assertEquals(QUERY_JOB_STATISTICS, QUERY_JOB.statistics()); + } + + @Test + public void testToPbAndFromPb() { + assertNotNull(COPY_JOB.toPb().getConfiguration().getCopy()); + assertNull(COPY_JOB.toPb().getConfiguration().getExtract()); + assertNull(COPY_JOB.toPb().getConfiguration().getLoad()); + assertNull(COPY_JOB.toPb().getConfiguration().getQuery()); + assertEquals(COPY_JOB_STATISTICS, JobStatistics.fromPb(COPY_JOB.statistics().toPb())); + compareJobInfo(COPY_JOB, JobInfo.fromPb(COPY_JOB.toPb())); + assertTrue(JobInfo.fromPb(COPY_JOB.toPb()).configuration() instanceof CopyJobConfiguration); + assertNull(EXTRACT_JOB.toPb().getConfiguration().getCopy()); + assertNotNull(EXTRACT_JOB.toPb().getConfiguration().getExtract()); + assertNull(EXTRACT_JOB.toPb().getConfiguration().getLoad()); + assertNull(EXTRACT_JOB.toPb().getConfiguration().getQuery()); + assertEquals(EXTRACT_JOB_STATISTICS, JobStatistics.fromPb(EXTRACT_JOB.statistics().toPb())); + compareJobInfo(EXTRACT_JOB, JobInfo.fromPb(EXTRACT_JOB.toPb())); + assertTrue( + JobInfo.fromPb(EXTRACT_JOB.toPb()).configuration() instanceof ExtractJobConfiguration); + assertTrue(JobInfo.fromPb(EXTRACT_JOB.toPb()).statistics() instanceof ExtractStatistics); + assertNull(LOAD_JOB.toPb().getConfiguration().getCopy()); + assertNull(LOAD_JOB.toPb().getConfiguration().getExtract()); + assertNotNull(LOAD_JOB.toPb().getConfiguration().getLoad()); + assertNull(LOAD_JOB.toPb().getConfiguration().getQuery()); + assertEquals(LOAD_JOB_STATISTICS, JobStatistics.fromPb(LOAD_JOB.statistics().toPb())); + compareJobInfo(LOAD_JOB, JobInfo.fromPb(LOAD_JOB.toPb())); + assertTrue(JobInfo.fromPb(LOAD_JOB.toPb()).configuration() instanceof LoadJobConfiguration); + assertTrue(JobInfo.fromPb(LOAD_JOB.toPb()).statistics() instanceof LoadStatistics); + assertNull(QUERY_JOB.toPb().getConfiguration().getCopy()); + assertNull(QUERY_JOB.toPb().getConfiguration().getExtract()); + assertNull(QUERY_JOB.toPb().getConfiguration().getLoad()); + assertNotNull(QUERY_JOB.toPb().getConfiguration().getQuery()); + assertEquals(QUERY_JOB_STATISTICS, JobStatistics.fromPb(QUERY_JOB.statistics().toPb())); + compareJobInfo(QUERY_JOB, JobInfo.fromPb(QUERY_JOB.toPb())); + assertTrue(JobInfo.fromPb(QUERY_JOB.toPb()).configuration() instanceof QueryJobConfiguration); + assertTrue(JobInfo.fromPb(QUERY_JOB.toPb()).statistics() instanceof QueryStatistics); + } + + @Test + public void testSetProjectId() { + CopyJobConfiguration copyConfiguration = COPY_JOB.setProjectId("p").configuration(); + assertEquals("p", copyConfiguration.destinationTable().project()); + for (TableId sourceTable : copyConfiguration.sourceTables()) { + assertEquals("p", sourceTable.project()); + } + ExtractJobConfiguration extractConfiguration = EXTRACT_JOB.setProjectId("p").configuration(); + assertEquals("p", extractConfiguration.sourceTable().project()); + LoadJobConfiguration loadConfiguration = LOAD_JOB.setProjectId("p").configuration(); + assertEquals("p", loadConfiguration.destinationTable().project()); + QueryJobConfiguration queryConfiguration = QUERY_JOB.setProjectId("p").configuration(); + assertEquals("p", queryConfiguration.defaultDataset().project()); + assertEquals("p", queryConfiguration.destinationTable().project()); + } + + private void compareJobInfo(JobInfo expected, JobInfo value) { + assertEquals(expected, value); + assertEquals(expected.hashCode(), value.hashCode()); + assertEquals(expected.toString(), value.toString()); + assertEquals(expected.etag(), value.etag()); + assertEquals(expected.id(), value.id()); + assertEquals(expected.jobId(), value.jobId()); + assertEquals(expected.selfLink(), value.selfLink()); + assertEquals(expected.status(), value.status()); + assertEquals(expected.statistics(), value.statistics()); + assertEquals(expected.userEmail(), value.userEmail()); + assertEquals(expected.configuration(), value.configuration()); + } +} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobTest.java index 6a5cef6508cc..db51706fff5a 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobTest.java @@ -16,160 +16,243 @@ package com.google.gcloud.bigquery; +import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; import org.junit.After; -import org.junit.Before; import org.junit.Test; public class JobTest { - private static final JobId JOB_ID = JobId.of("dataset", "job"); + private static final JobId JOB_ID = JobId.of("project", "job"); private static final TableId TABLE_ID1 = TableId.of("dataset", "table1"); private static final TableId TABLE_ID2 = TableId.of("dataset", "table2"); - private static final JobInfo JOB_INFO = CopyJobInfo.of(JOB_ID, TABLE_ID1, TABLE_ID2); + private static final String ETAG = "etag"; + private static final String ID = "id"; + private static final String SELF_LINK = "selfLink"; + private static final String EMAIL = "email"; + private static final JobStatus JOB_STATUS = new JobStatus(JobStatus.State.DONE); + private static final JobStatistics COPY_JOB_STATISTICS = JobStatistics.builder() + .creationTime(1L) + .endTime(3L) + .startTime(2L) + .build(); + private static final CopyJobConfiguration COPY_CONFIGURATION = + CopyJobConfiguration.of(TABLE_ID1, TABLE_ID2); + private static final JobInfo JOB_INFO = JobInfo.builder(COPY_CONFIGURATION) + .jobId(JOB_ID) + .statistics(COPY_JOB_STATISTICS) + .jobId(JOB_ID) + .etag(ETAG) + .id(ID) + .selfLink(SELF_LINK) + .userEmail(EMAIL) + .status(JOB_STATUS) + .build(); + private BigQuery serviceMockReturnsOptions = createStrictMock(BigQuery.class); + private BigQueryOptions mockOptions = createMock(BigQueryOptions.class); private BigQuery bigquery; + private Job expectedJob; private Job job; - @Before - public void setUp() throws Exception { + private void initializeExpectedJob(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); bigquery = createStrictMock(BigQuery.class); - job = new Job(bigquery, JOB_INFO); + expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(JOB_INFO)); + } + + private void initializeJob() { + job = new Job(bigquery, new JobInfo.BuilderImpl(JOB_INFO)); } @After public void tearDown() throws Exception { - verify(bigquery); + verify(bigquery, serviceMockReturnsOptions); } @Test - public void testInfo() throws Exception { - assertEquals(JOB_INFO, job.info()); + public void testBuilder() { + initializeExpectedJob(2); replay(bigquery); + Job builtJob = new Job.Builder(serviceMockReturnsOptions, COPY_CONFIGURATION) + .jobId(JOB_ID) + .statistics(COPY_JOB_STATISTICS) + .jobId(JOB_ID) + .etag(ETAG) + .id(ID) + .selfLink(SELF_LINK) + .userEmail(EMAIL) + .status(JOB_STATUS) + .build(); + assertEquals(ETAG, builtJob.etag()); + assertEquals(ID, builtJob.id()); + assertEquals(SELF_LINK, builtJob.selfLink()); + assertEquals(EMAIL, builtJob.userEmail()); + assertEquals(JOB_ID, builtJob.jobId()); + assertEquals(JOB_STATUS, builtJob.status()); + assertEquals(COPY_CONFIGURATION, builtJob.configuration()); + assertEquals(COPY_JOB_STATISTICS, builtJob.statistics()); + assertSame(serviceMockReturnsOptions, builtJob.bigquery()); } @Test - public void testBigQuery() throws Exception { - assertSame(bigquery, job.bigquery()); + public void testToBuilder() { + initializeExpectedJob(4); replay(bigquery); + compareJob(expectedJob, expectedJob.toBuilder().build()); } @Test public void testExists_True() throws Exception { + initializeExpectedJob(1); BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields()}; - expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)).andReturn(JOB_INFO); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)).andReturn(expectedJob); replay(bigquery); + initializeJob(); assertTrue(job.exists()); } @Test public void testExists_False() throws Exception { + initializeExpectedJob(1); BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields()}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)).andReturn(null); replay(bigquery); + initializeJob(); assertFalse(job.exists()); } @Test public void testIsDone_True() throws Exception { + initializeExpectedJob(2); + BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; JobStatus status = createStrictMock(JobStatus.class); expect(status.state()).andReturn(JobStatus.State.DONE); - BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)) - .andReturn(JOB_INFO.toBuilder().status(status).build()); + .andReturn(expectedJob.toBuilder().status(status).build()); replay(status, bigquery); + initializeJob(); assertTrue(job.isDone()); verify(status); } @Test public void testIsDone_False() throws Exception { + initializeExpectedJob(2); + BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; JobStatus status = createStrictMock(JobStatus.class); expect(status.state()).andReturn(JobStatus.State.RUNNING); - BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)) - .andReturn(JOB_INFO.toBuilder().status(status).build()); + .andReturn(expectedJob.toBuilder().status(status).build()); replay(status, bigquery); + initializeJob(); assertFalse(job.isDone()); verify(status); } @Test public void testIsDone_NotExists() throws Exception { + initializeExpectedJob(1); BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)).andReturn(null); replay(bigquery); + initializeJob(); assertFalse(job.isDone()); } @Test public void testReload() throws Exception { + initializeExpectedJob(4); JobInfo updatedInfo = JOB_INFO.toBuilder().etag("etag").build(); - expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(updatedInfo); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(expectedJob); replay(bigquery); + initializeJob(); Job updatedJob = job.reload(); - assertSame(bigquery, updatedJob.bigquery()); - assertEquals(updatedInfo, updatedJob.info()); + compareJob(expectedJob, updatedJob); } @Test public void testReloadNull() throws Exception { + initializeExpectedJob(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(null); replay(bigquery); + initializeJob(); assertNull(job.reload()); } @Test public void testReloadWithOptions() throws Exception { + initializeExpectedJob(4); JobInfo updatedInfo = JOB_INFO.toBuilder().etag("etag").build(); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId().job(), BigQuery.JobOption.fields())) - .andReturn(updatedInfo); + .andReturn(expectedJob); replay(bigquery); + initializeJob(); Job updatedJob = job.reload(BigQuery.JobOption.fields()); - assertSame(bigquery, updatedJob.bigquery()); - assertEquals(updatedInfo, updatedJob.info()); + compareJob(expectedJob, updatedJob); } @Test public void testCancel() throws Exception { + initializeExpectedJob(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.cancel(JOB_INFO.jobId())).andReturn(true); replay(bigquery); + initializeJob(); assertTrue(job.cancel()); } @Test - public void testGet() throws Exception { - expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(JOB_INFO); + public void testBigquery() { + initializeExpectedJob(1); replay(bigquery); - Job loadedJob = Job.get(bigquery, JOB_INFO.jobId().job()); - assertNotNull(loadedJob); - assertEquals(JOB_INFO, loadedJob.info()); + assertSame(serviceMockReturnsOptions, expectedJob.bigquery()); } @Test - public void testGetNull() throws Exception { - expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(null); + public void testToAndFromPb() { + initializeExpectedJob(4); replay(bigquery); - assertNull(Job.get(bigquery, JOB_INFO.jobId().job())); + compareJob(expectedJob, Job.fromPb(serviceMockReturnsOptions, expectedJob.toPb())); } - @Test - public void testGetWithOptions() throws Exception { - expect(bigquery.getJob(JOB_INFO.jobId().job(), BigQuery.JobOption.fields())) - .andReturn(JOB_INFO); - replay(bigquery); - Job loadedJob = Job.get(bigquery, JOB_INFO.jobId().job(), BigQuery.JobOption.fields()); - assertNotNull(loadedJob); - assertEquals(JOB_INFO, loadedJob.info()); + private void compareJob(Job expected, Job value) { + assertEquals(expected, value); + compareJobInfo(expected, value); + assertEquals(expected.bigquery().options(), value.bigquery().options()); + } + + private void compareJobInfo(JobInfo expected, JobInfo value) { + assertEquals(expected, value); + assertEquals(expected.hashCode(), value.hashCode()); + assertEquals(expected.toString(), value.toString()); + assertEquals(expected.etag(), value.etag()); + assertEquals(expected.id(), value.id()); + assertEquals(expected.jobId(), value.jobId()); + assertEquals(expected.selfLink(), value.selfLink()); + assertEquals(expected.status(), value.status()); + assertEquals(expected.statistics(), value.statistics()); + assertEquals(expected.userEmail(), value.userEmail()); + assertEquals(expected.configuration(), value.configuration()); } } diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/LoadJobConfigurationTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/LoadJobConfigurationTest.java new file mode 100644 index 000000000000..88ae6a4fc1b8 --- /dev/null +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/LoadJobConfigurationTest.java @@ -0,0 +1,140 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static org.junit.Assert.assertEquals; + +import com.google.common.collect.ImmutableList; +import com.google.gcloud.bigquery.JobInfo.CreateDisposition; +import com.google.gcloud.bigquery.JobInfo.WriteDisposition; + +import org.junit.Test; + +import java.nio.charset.StandardCharsets; +import java.util.List; + +public class LoadJobConfigurationTest { + + private static final CsvOptions CSV_OPTIONS = CsvOptions.builder() + .allowJaggedRows(true) + .allowQuotedNewLines(false) + .encoding(StandardCharsets.UTF_8) + .build(); + private static final TableId TABLE_ID = TableId.of("dataset", "table"); + private static final CreateDisposition CREATE_DISPOSITION = CreateDisposition.CREATE_IF_NEEDED; + private static final WriteDisposition WRITE_DISPOSITION = WriteDisposition.WRITE_APPEND; + private static final Integer MAX_BAD_RECORDS = 42; + private static final String FORMAT = "CSV"; + private static final Boolean IGNORE_UNKNOWN_VALUES = true; + private static final List PROJECTION_FIELDS = ImmutableList.of("field1", "field2"); + private static final Field FIELD_SCHEMA = Field.builder("IntegerField", Field.Type.integer()) + .mode(Field.Mode.REQUIRED) + .description("FieldDescription") + .build(); + private static final List SOURCE_URIS = ImmutableList.of("uri1", "uri2"); + private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA); + private static final LoadJobConfiguration LOAD_CONFIGURATION = + LoadJobConfiguration.builder(TABLE_ID, SOURCE_URIS) + .createDisposition(CREATE_DISPOSITION) + .writeDisposition(WRITE_DISPOSITION) + .formatOptions(CSV_OPTIONS) + .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) + .maxBadRecords(MAX_BAD_RECORDS) + .projectionFields(PROJECTION_FIELDS) + .schema(TABLE_SCHEMA) + .build(); + + @Test + public void testToBuilder() { + compareLoadJobConfiguration(LOAD_CONFIGURATION, LOAD_CONFIGURATION.toBuilder().build()); + LoadJobConfiguration configuration = LOAD_CONFIGURATION.toBuilder() + .destinationTable(TableId.of("dataset", "newTable")) + .build(); + assertEquals("newTable", configuration.destinationTable().table()); + configuration = configuration.toBuilder().destinationTable(TABLE_ID).build(); + compareLoadJobConfiguration(LOAD_CONFIGURATION, configuration); + } + + @Test + public void testOf() { + LoadJobConfiguration configuration = LoadJobConfiguration.of(TABLE_ID, SOURCE_URIS); + assertEquals(TABLE_ID, configuration.destinationTable()); + assertEquals(SOURCE_URIS, configuration.sourceUris()); + configuration = LoadJobConfiguration.of(TABLE_ID, SOURCE_URIS, CSV_OPTIONS); + assertEquals(TABLE_ID, configuration.destinationTable()); + assertEquals(FORMAT, configuration.format()); + assertEquals(CSV_OPTIONS, configuration.csvOptions()); + assertEquals(SOURCE_URIS, configuration.sourceUris()); + configuration = LoadJobConfiguration.of(TABLE_ID, "uri1"); + assertEquals(TABLE_ID, configuration.destinationTable()); + assertEquals(ImmutableList.of("uri1"), configuration.sourceUris()); + configuration = LoadJobConfiguration.of(TABLE_ID, "uri1", CSV_OPTIONS); + assertEquals(TABLE_ID, configuration.destinationTable()); + assertEquals(FORMAT, configuration.format()); + assertEquals(CSV_OPTIONS, configuration.csvOptions()); + assertEquals(ImmutableList.of("uri1"), configuration.sourceUris()); + } + + @Test + public void testToBuilderIncomplete() { + LoadJobConfiguration configuration = LoadJobConfiguration.of(TABLE_ID, SOURCE_URIS); + compareLoadJobConfiguration(configuration, configuration.toBuilder().build()); + } + + @Test + public void testBuilder() { + assertEquals(TABLE_ID, LOAD_CONFIGURATION.destinationTable()); + assertEquals(CREATE_DISPOSITION, LOAD_CONFIGURATION.createDisposition()); + assertEquals(WRITE_DISPOSITION, LOAD_CONFIGURATION.writeDisposition()); + assertEquals(CSV_OPTIONS, LOAD_CONFIGURATION.csvOptions()); + assertEquals(FORMAT, LOAD_CONFIGURATION.format()); + assertEquals(IGNORE_UNKNOWN_VALUES, LOAD_CONFIGURATION.ignoreUnknownValues()); + assertEquals(MAX_BAD_RECORDS, LOAD_CONFIGURATION.maxBadRecords()); + assertEquals(PROJECTION_FIELDS, LOAD_CONFIGURATION.projectionFields()); + assertEquals(TABLE_SCHEMA, LOAD_CONFIGURATION.schema()); + } + + @Test + public void testToPbAndFromPb() { + compareLoadJobConfiguration(LOAD_CONFIGURATION, + LoadJobConfiguration.fromPb(LOAD_CONFIGURATION.toPb())); + LoadJobConfiguration configuration = LoadJobConfiguration.of(TABLE_ID, SOURCE_URIS); + compareLoadJobConfiguration(configuration, LoadJobConfiguration.fromPb(configuration.toPb())); + } + + @Test + public void testSetProjectId() { + LoadConfiguration configuration = LOAD_CONFIGURATION.setProjectId("p"); + assertEquals("p", configuration.destinationTable().project()); + } + + private void compareLoadJobConfiguration(LoadJobConfiguration expected, + LoadJobConfiguration value) { + assertEquals(expected, value); + assertEquals(expected.hashCode(), value.hashCode()); + assertEquals(expected.toString(), value.toString()); + assertEquals(expected.destinationTable(), value.destinationTable()); + assertEquals(expected.createDisposition(), value.createDisposition()); + assertEquals(expected.writeDisposition(), value.writeDisposition()); + assertEquals(expected.csvOptions(), value.csvOptions()); + assertEquals(expected.format(), value.format()); + assertEquals(expected.ignoreUnknownValues(), value.ignoreUnknownValues()); + assertEquals(expected.maxBadRecords(), value.maxBadRecords()); + assertEquals(expected.projectionFields(), value.projectionFields()); + assertEquals(expected.schema(), value.schema()); + } +} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/LoadJobInfoTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/LoadJobInfoTest.java deleted file mode 100644 index 499d0d939698..000000000000 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/LoadJobInfoTest.java +++ /dev/null @@ -1,161 +0,0 @@ -/* - * Copyright 2015 Google Inc. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.google.gcloud.bigquery; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNotNull; -import static org.junit.Assert.assertNull; - -import com.google.common.collect.ImmutableList; -import com.google.gcloud.bigquery.JobInfo.CreateDisposition; -import com.google.gcloud.bigquery.JobInfo.WriteDisposition; -import com.google.gcloud.bigquery.JobStatistics.LoadStatistics; - -import org.junit.Test; - -import java.nio.charset.StandardCharsets; -import java.util.List; - -public class LoadJobInfoTest { - - private static final String ETAG = "etag"; - private static final String ID = "id"; - private static final String SELF_LINK = "selfLink"; - private static final String EMAIL = "email"; - private static final CsvOptions CSV_OPTIONS = CsvOptions.builder() - .allowJaggedRows(true) - .allowQuotedNewLines(false) - .encoding(StandardCharsets.UTF_8) - .build(); - private static final String SOURCE_URI = "uri"; - private static final List SOURCE_URIS = ImmutableList.of("uri1", "uri2"); - private static final TableId TABLE_ID = TableId.of("dataset", "table"); - private static final CreateDisposition CREATE_DISPOSITION = CreateDisposition.CREATE_IF_NEEDED; - private static final WriteDisposition WRITE_DISPOSITION = WriteDisposition.WRITE_APPEND; - private static final Integer MAX_BAD_RECORDS = 42; - private static final Boolean IGNORE_UNKNOWN_VALUES = true; - private static final List PROJECTION_FIELDS = ImmutableList.of("field1", "field2"); - private static final JobId JOB_ID = JobId.of("job"); - private static final JobStatus JOB_STATUS = new JobStatus(JobStatus.State.DONE); - private static final Field FIELD_SCHEMA = Field.builder("IntegerField", Field.Type.integer()) - .mode(Field.Mode.REQUIRED) - .description("FieldDescription") - .build(); - private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA); - private static final LoadStatistics JOB_STATISTICS = LoadStatistics.builder() - .creationTime(1L) - .endTime(3L) - .startTime(2L) - .inputFiles(42L) - .outputBytes(1024L) - .inputBytes(2048L) - .outputRows(24L) - .build(); - private static final LoadConfiguration LOAD_CONFIGURATION = LoadConfiguration.builder(TABLE_ID) - .createDisposition(CREATE_DISPOSITION) - .writeDisposition(WRITE_DISPOSITION) - .formatOptions(CSV_OPTIONS) - .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) - .maxBadRecords(MAX_BAD_RECORDS) - .projectionFields(PROJECTION_FIELDS) - .schema(TABLE_SCHEMA) - .build(); - private static final LoadJobInfo LOAD_JOB = LoadJobInfo.builder(LOAD_CONFIGURATION, SOURCE_URIS) - .etag(ETAG) - .id(ID) - .selfLink(SELF_LINK) - .userEmail(EMAIL) - .jobId(JOB_ID) - .status(JOB_STATUS) - .statistics(JOB_STATISTICS) - .build(); - - @Test - public void testToBuilder() { - compareLoadJobInfo(LOAD_JOB, LOAD_JOB.toBuilder().build()); - LoadJobInfo job = LOAD_JOB.toBuilder().etag("newEtag").build(); - assertEquals("newEtag", job.etag()); - job = job.toBuilder().etag(ETAG).build(); - compareLoadJobInfo(LOAD_JOB, job); - } - - @Test - public void testOf() { - LoadJobInfo job = LoadJobInfo.of(LOAD_CONFIGURATION, SOURCE_URIS); - assertEquals(LOAD_CONFIGURATION, job.configuration()); - assertEquals(SOURCE_URIS, job.sourceUris()); - job = LoadJobInfo.of(LOAD_CONFIGURATION, SOURCE_URI); - assertEquals(LOAD_CONFIGURATION, job.configuration()); - assertEquals(ImmutableList.of(SOURCE_URI), job.sourceUris()); - job = LoadJobInfo.of(JOB_ID, LOAD_CONFIGURATION, SOURCE_URIS); - assertEquals(JOB_ID, job.jobId()); - assertEquals(LOAD_CONFIGURATION, job.configuration()); - assertEquals(SOURCE_URIS, job.sourceUris()); - job = LoadJobInfo.of(JOB_ID, LOAD_CONFIGURATION, SOURCE_URI); - assertEquals(JOB_ID, job.jobId()); - assertEquals(LOAD_CONFIGURATION, job.configuration()); - assertEquals(ImmutableList.of(SOURCE_URI), job.sourceUris()); - } - - @Test - public void testToBuilderIncomplete() { - LoadJobInfo job = LoadJobInfo.of(LOAD_CONFIGURATION, SOURCE_URIS); - compareLoadJobInfo(job, job.toBuilder().build()); - } - - @Test - public void testBuilder() { - assertEquals(ETAG, LOAD_JOB.etag()); - assertEquals(ID, LOAD_JOB.id()); - assertEquals(SELF_LINK, LOAD_JOB.selfLink()); - assertEquals(EMAIL, LOAD_JOB.userEmail()); - assertEquals(JOB_ID, LOAD_JOB.jobId()); - assertEquals(JOB_STATUS, LOAD_JOB.status()); - assertEquals(LOAD_CONFIGURATION, LOAD_JOB.configuration()); - assertEquals(SOURCE_URIS, LOAD_JOB.sourceUris()); - assertEquals(JOB_STATISTICS, LOAD_JOB.statistics()); - } - - @Test - public void testToPbAndFromPb() { - assertNotNull(LOAD_JOB.toPb().getConfiguration().getLoad()); - assertNull(LOAD_JOB.toPb().getConfiguration().getExtract()); - assertNull(LOAD_JOB.toPb().getConfiguration().getCopy()); - assertNull(LOAD_JOB.toPb().getConfiguration().getQuery()); - assertEquals(JOB_STATISTICS, JobStatistics.fromPb(LOAD_JOB.toPb().getStatistics())); - compareLoadJobInfo(LOAD_JOB, LoadJobInfo.fromPb(LOAD_JOB.toPb())); - compareLoadJobInfo(LOAD_JOB, (LoadJobInfo) JobInfo.fromPb(LOAD_JOB.toPb())); - LoadJobInfo job = LoadJobInfo.of(LOAD_CONFIGURATION, SOURCE_URIS); - compareLoadJobInfo(job, LoadJobInfo.fromPb(job.toPb())); - compareLoadJobInfo(job, (LoadJobInfo) JobInfo.fromPb(job.toPb())); - } - - private void compareLoadJobInfo(LoadJobInfo expected, LoadJobInfo value) { - assertEquals(expected, value); - assertEquals(expected.hashCode(), value.hashCode()); - assertEquals(expected.toString(), value.toString()); - assertEquals(expected.etag(), value.etag()); - assertEquals(expected.id(), value.id()); - assertEquals(expected.jobId(), value.jobId()); - assertEquals(expected.selfLink(), value.selfLink()); - assertEquals(expected.status(), value.status()); - assertEquals(expected.statistics(), value.statistics()); - assertEquals(expected.userEmail(), value.userEmail()); - assertEquals(expected.configuration(), value.configuration()); - assertEquals(expected.sourceUris(), value.sourceUris()); - } -} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryJobConfigurationTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryJobConfigurationTest.java new file mode 100644 index 000000000000..1ef270ee69cf --- /dev/null +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryJobConfigurationTest.java @@ -0,0 +1,169 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.gcloud.bigquery.JobInfo.CreateDisposition; +import com.google.gcloud.bigquery.JobInfo.WriteDisposition; +import com.google.gcloud.bigquery.QueryJobConfiguration.Priority; + +import org.junit.Test; + +import java.util.List; +import java.util.Map; + +public class QueryJobConfigurationTest { + + private static final String QUERY = "BigQuery SQL"; + private static final DatasetId DATASET_ID = DatasetId.of("dataset"); + private static final TableId TABLE_ID = TableId.of("dataset", "table"); + private static final List SOURCE_URIS = ImmutableList.of("uri1", "uri2"); + private static final Field FIELD_SCHEMA1 = + Field.builder("StringField", Field.Type.string()) + .mode(Field.Mode.NULLABLE) + .description("FieldDescription1") + .build(); + private static final Field FIELD_SCHEMA2 = + Field.builder("IntegerField", Field.Type.integer()) + .mode(Field.Mode.REPEATED) + .description("FieldDescription2") + .build(); + private static final Field FIELD_SCHEMA3 = + Field.builder("RecordField", Field.Type.record(FIELD_SCHEMA1, FIELD_SCHEMA2)) + .mode(Field.Mode.REQUIRED) + .description("FieldDescription3") + .build(); + private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA1, FIELD_SCHEMA2, FIELD_SCHEMA3); + private static final Integer MAX_BAD_RECORDS = 42; + private static final Boolean IGNORE_UNKNOWN_VALUES = true; + private static final String COMPRESSION = "GZIP"; + private static final CsvOptions CSV_OPTIONS = CsvOptions.builder().build(); + private static final ExternalTableDefinition TABLE_CONFIGURATION = + ExternalTableDefinition.builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS) + .compression(COMPRESSION) + .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) + .maxBadRecords(MAX_BAD_RECORDS) + .build(); + private static final Map TABLE_DEFINITIONS = + ImmutableMap.of("tableName", TABLE_CONFIGURATION); + private static final CreateDisposition CREATE_DISPOSITION = CreateDisposition.CREATE_IF_NEEDED; + private static final WriteDisposition WRITE_DISPOSITION = WriteDisposition.WRITE_APPEND; + private static final Priority PRIORITY = Priority.BATCH; + private static final boolean ALLOW_LARGE_RESULTS = true; + private static final boolean USE_QUERY_CACHE = false; + private static final boolean FLATTEN_RESULTS = true; + private static final List USER_DEFINED_FUNCTIONS = ImmutableList.of( + UserDefinedFunction.inline("Function"), UserDefinedFunction.fromUri("URI")); + private static final QueryJobConfiguration QUERY_JOB_CONFIGURATION = + QueryJobConfiguration.builder(QUERY) + .useQueryCache(USE_QUERY_CACHE) + .tableDefinitions(TABLE_DEFINITIONS) + .allowLargeResults(ALLOW_LARGE_RESULTS) + .createDisposition(CREATE_DISPOSITION) + .defaultDataset(DATASET_ID) + .destinationTable(TABLE_ID) + .writeDisposition(WRITE_DISPOSITION) + .priority(PRIORITY) + .flattenResults(FLATTEN_RESULTS) + .userDefinedFunctions(USER_DEFINED_FUNCTIONS) + .dryRun(true) + .build(); + + @Test + public void testToBuilder() { + compareQueryJobConfiguration(QUERY_JOB_CONFIGURATION, + QUERY_JOB_CONFIGURATION.toBuilder().build()); + QueryJobConfiguration job = QUERY_JOB_CONFIGURATION.toBuilder() + .query("New BigQuery SQL") + .build(); + assertEquals("New BigQuery SQL", job.query()); + job = job.toBuilder().query(QUERY).build(); + compareQueryJobConfiguration(QUERY_JOB_CONFIGURATION, job); + } + + @Test + public void testOf() { + QueryJobConfiguration job = QueryJobConfiguration.of(QUERY); + assertEquals(QUERY, job.query()); + } + + @Test + public void testToBuilderIncomplete() { + QueryJobConfiguration job = QueryJobConfiguration.of(QUERY); + compareQueryJobConfiguration(job, job.toBuilder().build()); + } + + @Test + public void testBuilder() { + assertEquals(ALLOW_LARGE_RESULTS, QUERY_JOB_CONFIGURATION.allowLargeResults()); + assertEquals(CREATE_DISPOSITION, QUERY_JOB_CONFIGURATION.createDisposition()); + assertEquals(DATASET_ID, QUERY_JOB_CONFIGURATION.defaultDataset()); + assertEquals(TABLE_ID, QUERY_JOB_CONFIGURATION.destinationTable()); + assertEquals(FLATTEN_RESULTS, QUERY_JOB_CONFIGURATION.flattenResults()); + assertEquals(PRIORITY, QUERY_JOB_CONFIGURATION.priority()); + assertEquals(QUERY, QUERY_JOB_CONFIGURATION.query()); + assertEquals(TABLE_DEFINITIONS, QUERY_JOB_CONFIGURATION.tableDefinitions()); + assertEquals(USE_QUERY_CACHE, QUERY_JOB_CONFIGURATION.useQueryCache()); + assertEquals(USER_DEFINED_FUNCTIONS, QUERY_JOB_CONFIGURATION.userDefinedFunctions()); + assertEquals(WRITE_DISPOSITION, QUERY_JOB_CONFIGURATION.writeDisposition()); + assertTrue(QUERY_JOB_CONFIGURATION.dryRun()); + } + + @Test + public void testToPbAndFromPb() { + assertNotNull(QUERY_JOB_CONFIGURATION.toPb().getQuery()); + assertNull(QUERY_JOB_CONFIGURATION.toPb().getExtract()); + assertNull(QUERY_JOB_CONFIGURATION.toPb().getCopy()); + assertNull(QUERY_JOB_CONFIGURATION.toPb().getLoad()); + compareQueryJobConfiguration(QUERY_JOB_CONFIGURATION, + QueryJobConfiguration.fromPb(QUERY_JOB_CONFIGURATION.toPb())); + QueryJobConfiguration job = QueryJobConfiguration.of(QUERY); + compareQueryJobConfiguration(job, QueryJobConfiguration.fromPb(job.toPb())); + } + + @Test + public void testSetProjectId() { + QueryJobConfiguration configuration = QUERY_JOB_CONFIGURATION.setProjectId("p"); + assertEquals("p", configuration.defaultDataset().project()); + assertEquals("p", configuration.destinationTable().project()); + } + + private void compareQueryJobConfiguration(QueryJobConfiguration expected, + QueryJobConfiguration value) { + assertEquals(expected, value); + assertEquals(expected.hashCode(), value.hashCode()); + assertEquals(expected.toString(), value.toString()); + assertEquals(expected.dryRun(), value.dryRun()); + assertEquals(expected.allowLargeResults(), value.allowLargeResults()); + assertEquals(expected.createDisposition(), value.createDisposition()); + assertEquals(expected.defaultDataset(), value.defaultDataset()); + assertEquals(expected.destinationTable(), value.destinationTable()); + assertEquals(expected.flattenResults(), value.flattenResults()); + assertEquals(expected.priority(), value.priority()); + assertEquals(expected.query(), value.query()); + assertEquals(expected.tableDefinitions(), value.tableDefinitions()); + assertEquals(expected.useQueryCache(), value.useQueryCache()); + assertEquals(expected.userDefinedFunctions(), value.userDefinedFunctions()); + assertEquals(expected.writeDisposition(), value.writeDisposition()); + } +} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryJobInfoTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryJobInfoTest.java deleted file mode 100644 index f99bec19efd9..000000000000 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryJobInfoTest.java +++ /dev/null @@ -1,202 +0,0 @@ -/* - * Copyright 2015 Google Inc. All Rights Reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.google.gcloud.bigquery; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNotNull; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertTrue; - -import com.google.common.collect.ImmutableList; -import com.google.common.collect.ImmutableMap; -import com.google.gcloud.bigquery.JobInfo.CreateDisposition; -import com.google.gcloud.bigquery.JobInfo.WriteDisposition; -import com.google.gcloud.bigquery.JobStatistics.QueryStatistics; -import com.google.gcloud.bigquery.QueryJobInfo.Priority; - -import org.junit.Test; - -import java.util.List; -import java.util.Map; - -public class QueryJobInfoTest { - - private static final String ETAG = "etag"; - private static final String ID = "id"; - private static final String SELF_LINK = "selfLink"; - private static final String EMAIL = "email"; - private static final String QUERY = "BigQuery SQL"; - private static final DatasetId DATASET_ID = DatasetId.of("project", "dataset"); - private static final TableId TABLE_ID = TableId.of("project", "dataset", "table"); - private static final List SOURCE_URIS = ImmutableList.of("uri1", "uri2"); - private static final Field FIELD_SCHEMA1 = - Field.builder("StringField", Field.Type.string()) - .mode(Field.Mode.NULLABLE) - .description("FieldDescription1") - .build(); - private static final Field FIELD_SCHEMA2 = - Field.builder("IntegerField", Field.Type.integer()) - .mode(Field.Mode.REPEATED) - .description("FieldDescription2") - .build(); - private static final Field FIELD_SCHEMA3 = - Field.builder("RecordField", Field.Type.record(FIELD_SCHEMA1, FIELD_SCHEMA2)) - .mode(Field.Mode.REQUIRED) - .description("FieldDescription3") - .build(); - private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA1, FIELD_SCHEMA2, FIELD_SCHEMA3); - private static final Integer MAX_BAD_RECORDS = 42; - private static final Boolean IGNORE_UNKNOWN_VALUES = true; - private static final String COMPRESSION = "GZIP"; - private static final CsvOptions CSV_OPTIONS = CsvOptions.builder().build(); - private static final ExternalDataConfiguration TABLE_CONFIGURATION = ExternalDataConfiguration - .builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS) - .compression(COMPRESSION) - .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) - .maxBadRecords(MAX_BAD_RECORDS) - .build(); - private static final Map TABLE_DEFINITIONS = - ImmutableMap.of("tableName", TABLE_CONFIGURATION); - private static final CreateDisposition CREATE_DISPOSITION = CreateDisposition.CREATE_IF_NEEDED; - private static final WriteDisposition WRITE_DISPOSITION = WriteDisposition.WRITE_APPEND; - private static final Priority PRIORITY = Priority.BATCH; - private static final boolean ALLOW_LARGE_RESULTS = true; - private static final boolean USE_QUERY_CACHE = false; - private static final boolean FLATTEN_RESULTS = true; - private static final List USER_DEFINED_FUNCTIONS = ImmutableList.of( - UserDefinedFunction.inline("Function"), UserDefinedFunction.fromUri("URI")); - private static final JobId JOB_ID = JobId.of("job"); - private static final JobStatus JOB_STATUS = new JobStatus(JobStatus.State.DONE); - private static final QueryStatistics JOB_STATISTICS = QueryStatistics.builder() - .creationTime(1L) - .endTime(3L) - .startTime(2L) - .totalBytesProcessed(2048L) - .totalBytesBilled(1024L) - .cacheHit(false) - .billingTier(42) - .build(); - private static final QueryJobInfo QUERY_JOB = QueryJobInfo.builder(QUERY) - .etag(ETAG) - .id(ID) - .selfLink(SELF_LINK) - .userEmail(EMAIL) - .jobId(JOB_ID) - .status(JOB_STATUS) - .useQueryCache(USE_QUERY_CACHE) - .tableDefinitions(TABLE_DEFINITIONS) - .allowLargeResults(ALLOW_LARGE_RESULTS) - .createDisposition(CREATE_DISPOSITION) - .defaultDataset(DATASET_ID) - .destinationTable(TABLE_ID) - .writeDisposition(WRITE_DISPOSITION) - .priority(PRIORITY) - .flattenResults(FLATTEN_RESULTS) - .userDefinedFunctions(USER_DEFINED_FUNCTIONS) - .dryRun(true) - .statistics(JOB_STATISTICS) - .build(); - - @Test - public void testToBuilder() { - compareQueryJobInfo(QUERY_JOB, QUERY_JOB.toBuilder().build()); - QueryJobInfo job = QUERY_JOB.toBuilder() - .query("New BigQuery SQL") - .build(); - assertEquals("New BigQuery SQL", job.query()); - job = job.toBuilder().query(QUERY).build(); - compareQueryJobInfo(QUERY_JOB, job); - } - - @Test - public void testOf() { - QueryJobInfo job = QueryJobInfo.of(QUERY); - assertEquals(QUERY, job.query()); - job = QueryJobInfo.of(JOB_ID, QUERY); - assertEquals(JOB_ID, job.jobId()); - assertEquals(QUERY, job.query()); - } - - @Test - public void testToBuilderIncomplete() { - QueryJobInfo job = QueryJobInfo.of(QUERY); - compareQueryJobInfo(job, job.toBuilder().build()); - } - - @Test - public void testBuilder() { - assertEquals(ETAG, QUERY_JOB.etag()); - assertEquals(ID, QUERY_JOB.id()); - assertEquals(SELF_LINK, QUERY_JOB.selfLink()); - assertEquals(EMAIL, QUERY_JOB.userEmail()); - assertEquals(JOB_ID, QUERY_JOB.jobId()); - assertEquals(JOB_STATUS, QUERY_JOB.status()); - assertEquals(ALLOW_LARGE_RESULTS, QUERY_JOB.allowLargeResults()); - assertEquals(CREATE_DISPOSITION, QUERY_JOB.createDisposition()); - assertEquals(DATASET_ID, QUERY_JOB.defaultDataset()); - assertEquals(TABLE_ID, QUERY_JOB.destinationTable()); - assertEquals(FLATTEN_RESULTS, QUERY_JOB.flattenResults()); - assertEquals(PRIORITY, QUERY_JOB.priority()); - assertEquals(QUERY, QUERY_JOB.query()); - assertEquals(TABLE_DEFINITIONS, QUERY_JOB.tableDefinitions()); - assertEquals(USE_QUERY_CACHE, QUERY_JOB.useQueryCache()); - assertEquals(USER_DEFINED_FUNCTIONS, QUERY_JOB.userDefinedFunctions()); - assertEquals(WRITE_DISPOSITION, QUERY_JOB.writeDisposition()); - assertTrue(QUERY_JOB.dryRun()); - assertEquals(JOB_STATISTICS, QUERY_JOB.statistics()); - } - - @Test - public void testToPbAndFromPb() { - assertNotNull(QUERY_JOB.toPb().getConfiguration().getQuery()); - assertNull(QUERY_JOB.toPb().getConfiguration().getExtract()); - assertNull(QUERY_JOB.toPb().getConfiguration().getCopy()); - assertNull(QUERY_JOB.toPb().getConfiguration().getLoad()); - assertEquals(JOB_STATISTICS, JobStatistics.fromPb(QUERY_JOB.statistics().toPb())); - compareQueryJobInfo(QUERY_JOB, QueryJobInfo.fromPb(QUERY_JOB.toPb())); - compareQueryJobInfo(QUERY_JOB, - (QueryJobInfo) JobInfo.fromPb(QUERY_JOB.toPb())); - QueryJobInfo job = QueryJobInfo.of(QUERY); - compareQueryJobInfo(job, QueryJobInfo.fromPb(job.toPb())); - compareQueryJobInfo(job, (QueryJobInfo) JobInfo.fromPb(job.toPb())); - } - - private void compareQueryJobInfo(QueryJobInfo expected, QueryJobInfo value) { - assertEquals(expected, value); - assertEquals(expected.hashCode(), value.hashCode()); - assertEquals(expected.toString(), value.toString()); - assertEquals(expected.etag(), value.etag()); - assertEquals(expected.id(), value.id()); - assertEquals(expected.jobId(), value.jobId()); - assertEquals(expected.selfLink(), value.selfLink()); - assertEquals(expected.status(), value.status()); - assertEquals(expected.statistics(), value.statistics()); - assertEquals(expected.dryRun(), value.dryRun()); - assertEquals(expected.userEmail(), value.userEmail()); - assertEquals(expected.allowLargeResults(), value.allowLargeResults()); - assertEquals(expected.createDisposition(), value.createDisposition()); - assertEquals(expected.defaultDataset(), value.defaultDataset()); - assertEquals(expected.destinationTable(), value.destinationTable()); - assertEquals(expected.flattenResults(), value.flattenResults()); - assertEquals(expected.priority(), value.priority()); - assertEquals(expected.query(), value.query()); - assertEquals(expected.tableDefinitions(), value.tableDefinitions()); - assertEquals(expected.useQueryCache(), value.useQueryCache()); - assertEquals(expected.userDefinedFunctions(), value.userDefinedFunctions()); - assertEquals(expected.writeDisposition(), value.writeDisposition()); - } -} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryRequestTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryRequestTest.java index 276e4f6792b3..370b4d614cbf 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryRequestTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/QueryRequestTest.java @@ -26,7 +26,7 @@ public class QueryRequestTest { private static final String QUERY = "BigQuery SQL"; - private static final DatasetId DATASET_ID = DatasetId.of("project", "dataset"); + private static final DatasetId DATASET_ID = DatasetId.of("dataset"); private static final Boolean USE_QUERY_CACHE = true; private static final Boolean DRY_RUN = false; private static final Long MAX_RESULTS = 42L; @@ -91,6 +91,11 @@ public void testToPbAndFromPb() { compareQueryRequest(queryRequest, QueryRequest.fromPb(queryRequest.toPb())); } + @Test + public void testSetProjectId() { + assertEquals("p", QUERY_REQUEST.setProjectId("p").defaultDataset().project()); + } + private void compareQueryRequest(QueryRequest expected, QueryRequest value) { assertEquals(expected, value); assertEquals(expected.query(), value.query()); diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/RemoteBigQueryHelperTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/RemoteBigQueryHelperTest.java index 62a88c1860cd..267ae161b7aa 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/RemoteBigQueryHelperTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/RemoteBigQueryHelperTest.java @@ -65,6 +65,7 @@ public class RemoteBigQueryHelperTest { @Rule public ExpectedException thrown = ExpectedException.none(); + @Test public void testForceDelete() throws InterruptedException, ExecutionException { BigQuery bigqueryMock = EasyMock.createMock(BigQuery.class); diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/SerializationTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/SerializationTest.java index 1a20682aa447..d877bff2138c 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/SerializationTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/SerializationTest.java @@ -25,7 +25,7 @@ import com.google.gcloud.RestorableState; import com.google.gcloud.RetryParams; import com.google.gcloud.WriteChannel; -import com.google.gcloud.bigquery.TableInfo.StreamingBuffer; +import com.google.gcloud.bigquery.StandardTableDefinition.StreamingBuffer; import org.junit.Test; @@ -99,8 +99,8 @@ public class SerializationTest { private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA1, FIELD_SCHEMA2, FIELD_SCHEMA3); private static final StreamingBuffer STREAMING_BUFFER = new StreamingBuffer(1L, 2L, 3L); private static final List SOURCE_URIS = ImmutableList.of("uri1", "uri2"); - private static final ExternalDataConfiguration EXTERNAL_DATA_CONFIGURATION = - ExternalDataConfiguration.builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS) + private static final ExternalTableDefinition EXTERNAL_TABLE_DEFINITION = + ExternalTableDefinition.builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS) .ignoreUnknownValues(true) .maxBadRecords(42) .build(); @@ -108,24 +108,26 @@ public class SerializationTest { new UserDefinedFunction.InlineFunction("inline"); private static final UserDefinedFunction URI_FUNCTION = new UserDefinedFunction.UriFunction("URI"); - private static final BaseTableInfo TABLE_INFO = - TableInfo.builder(TABLE_ID, TABLE_SCHEMA) - .creationTime(CREATION_TIME) - .description(DESCRIPTION) - .etag(ETAG) - .id(ID) - .location(LOCATION) - .streamingBuffer(STREAMING_BUFFER) - .build(); - private static final ViewInfo VIEW_INFO = - ViewInfo.builder(TABLE_ID, "QUERY") - .creationTime(CREATION_TIME) - .description(DESCRIPTION) - .etag(ETAG) - .id(ID) - .build(); - private static final ExternalTableInfo EXTERNAL_TABLE_INFO = - ExternalTableInfo.builder(TABLE_ID, EXTERNAL_DATA_CONFIGURATION) + private static final TableDefinition TABLE_DEFINITION = StandardTableDefinition.builder() + .schema(TABLE_SCHEMA) + .location(LOCATION) + .streamingBuffer(STREAMING_BUFFER) + .build(); + private static final TableInfo TABLE_INFO = TableInfo.builder(TABLE_ID, TABLE_DEFINITION) + .creationTime(CREATION_TIME) + .description(DESCRIPTION) + .etag(ETAG) + .id(ID) + .build(); + private static final TableDefinition VIEW_DEFINITION = ViewDefinition.of("QUERY"); + private static final TableInfo VIEW_INFO = TableInfo.builder(TABLE_ID, VIEW_DEFINITION) + .creationTime(CREATION_TIME) + .description(DESCRIPTION) + .etag(ETAG) + .id(ID) + .build(); + private static final TableInfo EXTERNAL_TABLE_INFO = + TableInfo.builder(TABLE_ID, EXTERNAL_TABLE_DEFINITION) .creationTime(CREATION_TIME) .description(DESCRIPTION) .etag(ETAG) @@ -168,18 +170,24 @@ public class SerializationTest { private static final JobStatus JOB_STATUS = new JobStatus(JobStatus.State.DONE, BIGQUERY_ERROR, ImmutableList.of(BIGQUERY_ERROR)); private static final JobId JOB_ID = JobId.of("project", "job"); - private static final CopyJobInfo COPY_JOB = CopyJobInfo.of(TABLE_ID, TABLE_ID); - private static final ExtractJobInfo EXTRACT_JOB = ExtractJobInfo.of(TABLE_ID, SOURCE_URIS); - private static final LoadConfiguration LOAD_CONFIGURATION = LoadConfiguration.builder(TABLE_ID) - .createDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED) - .writeDisposition(JobInfo.WriteDisposition.WRITE_APPEND) - .formatOptions(CSV_OPTIONS) - .ignoreUnknownValues(true) - .maxBadRecords(10) - .schema(TABLE_SCHEMA) - .build(); - private static final LoadJobInfo LOAD_JOB = LoadJobInfo.of(LOAD_CONFIGURATION, SOURCE_URIS); - private static final QueryJobInfo QUERY_JOB = QueryJobInfo.of("query"); + private static final CopyJobConfiguration COPY_JOB_CONFIGURATION = + CopyJobConfiguration.of(TABLE_ID, TABLE_ID); + private static final ExtractJobConfiguration EXTRACT_JOB_CONFIGURATION = + ExtractJobConfiguration.of(TABLE_ID, SOURCE_URIS); + private static final WriteChannelConfiguration LOAD_CONFIGURATION = + WriteChannelConfiguration.builder(TABLE_ID) + .createDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED) + .writeDisposition(JobInfo.WriteDisposition.WRITE_APPEND) + .formatOptions(CSV_OPTIONS) + .ignoreUnknownValues(true) + .maxBadRecords(10) + .schema(TABLE_SCHEMA) + .build(); + private static final LoadJobConfiguration LOAD_JOB_CONFIGURATION = + LoadJobConfiguration.of(TABLE_ID, SOURCE_URIS); + private static final QueryJobConfiguration QUERY_JOB_CONFIGURATION = + QueryJobConfiguration.of("query"); + private static final JobInfo JOB_INFO = JobInfo.of(COPY_JOB_CONFIGURATION); private static final Map CONTENT1 = ImmutableMap.of("key", "val1"); private static final Map CONTENT2 = @@ -216,6 +224,12 @@ public class SerializationTest { .jobCompleted(true) .result(QUERY_RESULT) .build(); + private static final BigQuery BIGQUERY = + BigQueryOptions.builder().projectId("p1").build().service(); + private static final Dataset DATASET = + new Dataset(BIGQUERY, new DatasetInfo.BuilderImpl(DATASET_INFO)); + private static final Table TABLE = new Table(BIGQUERY, new TableInfo.BuilderImpl(TABLE_INFO)); + private static final Job JOB = new Job(BIGQUERY, new JobInfo.BuilderImpl(JOB_INFO)); @Test public void testServiceOptions() throws Exception { @@ -238,15 +252,17 @@ public void testServiceOptions() throws Exception { @Test public void testModelAndRequests() throws Exception { Serializable[] objects = {DOMAIN_ACCESS, GROUP_ACCESS, USER_ACCESS, VIEW_ACCESS, DATASET_ID, - DATASET_INFO, TABLE_ID, CSV_OPTIONS, STREAMING_BUFFER, EXTERNAL_DATA_CONFIGURATION, - TABLE_SCHEMA, TABLE_INFO, VIEW_INFO, EXTERNAL_TABLE_INFO, INLINE_FUNCTION, URI_FUNCTION, - JOB_STATISTICS, EXTRACT_STATISTICS, LOAD_STATISTICS, QUERY_STATISTICS, BIGQUERY_ERROR, - JOB_STATUS, JOB_ID, COPY_JOB, EXTRACT_JOB, LOAD_CONFIGURATION, LOAD_JOB, QUERY_JOB, - INSERT_ALL_REQUEST, INSERT_ALL_RESPONSE, FIELD_VALUE, QUERY_REQUEST, QUERY_RESPONSE, + DATASET_INFO, TABLE_ID, CSV_OPTIONS, STREAMING_BUFFER, TABLE_DEFINITION, + EXTERNAL_TABLE_DEFINITION, VIEW_DEFINITION, TABLE_SCHEMA, TABLE_INFO, VIEW_INFO, + EXTERNAL_TABLE_INFO, INLINE_FUNCTION, URI_FUNCTION, JOB_STATISTICS, EXTRACT_STATISTICS, + LOAD_STATISTICS, QUERY_STATISTICS, BIGQUERY_ERROR, JOB_STATUS, JOB_ID, + COPY_JOB_CONFIGURATION, EXTRACT_JOB_CONFIGURATION, LOAD_CONFIGURATION, + LOAD_JOB_CONFIGURATION, QUERY_JOB_CONFIGURATION, JOB_INFO, INSERT_ALL_REQUEST, + INSERT_ALL_RESPONSE, FIELD_VALUE, QUERY_REQUEST, QUERY_RESPONSE, BigQuery.DatasetOption.fields(), BigQuery.DatasetDeleteOption.deleteContents(), BigQuery.DatasetListOption.all(), BigQuery.TableOption.fields(), BigQuery.TableListOption.maxResults(42L), BigQuery.JobOption.fields(), - BigQuery.JobListOption.allUsers()}; + BigQuery.JobListOption.allUsers(), DATASET, TABLE, JOB}; for (Serializable obj : objects) { Object copy = serializeAndDeserialize(obj); assertEquals(obj, obj); diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableDataWriteChannelTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableDataWriteChannelTest.java index 67933407e377..6b7edcd76db1 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableDataWriteChannelTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableDataWriteChannelTest.java @@ -50,13 +50,14 @@ public class TableDataWriteChannelTest { private static final String UPLOAD_ID = "uploadid"; private static final TableId TABLE_ID = TableId.of("dataset", "table"); - private static final LoadConfiguration LOAD_CONFIGURATION = LoadConfiguration.builder(TABLE_ID) - .createDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED) - .writeDisposition(JobInfo.WriteDisposition.WRITE_APPEND) - .formatOptions(FormatOptions.json()) - .ignoreUnknownValues(true) - .maxBadRecords(10) - .build(); + private static final WriteChannelConfiguration LOAD_CONFIGURATION = + WriteChannelConfiguration.builder(TABLE_ID) + .createDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED) + .writeDisposition(JobInfo.WriteDisposition.WRITE_APPEND) + .formatOptions(FormatOptions.json()) + .ignoreUnknownValues(true) + .maxBadRecords(10) + .build(); private static final int MIN_CHUNK_SIZE = 256 * 1024; private static final int DEFAULT_CHUNK_SIZE = 8 * MIN_CHUNK_SIZE; private static final int CUSTOM_CHUNK_SIZE = 4 * MIN_CHUNK_SIZE; diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableDefinitionTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableDefinitionTest.java new file mode 100644 index 000000000000..d1e3635d00cb --- /dev/null +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableDefinitionTest.java @@ -0,0 +1,103 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import com.google.gcloud.bigquery.StandardTableDefinition.StreamingBuffer; + +import org.junit.Test; + +public class TableDefinitionTest { + + private static final Field FIELD_SCHEMA1 = + Field.builder("StringField", Field.Type.string()) + .mode(Field.Mode.NULLABLE) + .description("FieldDescription1") + .build(); + private static final Field FIELD_SCHEMA2 = + Field.builder("IntegerField", Field.Type.integer()) + .mode(Field.Mode.REPEATED) + .description("FieldDescription2") + .build(); + private static final Field FIELD_SCHEMA3 = + Field.builder("RecordField", Field.Type.record(FIELD_SCHEMA1, FIELD_SCHEMA2)) + .mode(Field.Mode.REQUIRED) + .description("FieldDescription3") + .build(); + private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA1, FIELD_SCHEMA2, FIELD_SCHEMA3); + private static final Long NUM_BYTES = 42L; + private static final Long NUM_ROWS = 43L; + private static final String LOCATION = "US"; + private static final StreamingBuffer STREAMING_BUFFER = new StreamingBuffer(1L, 2L, 3L); + private static final StandardTableDefinition TABLE_DEFINITION = + StandardTableDefinition.builder() + .location(LOCATION) + .numBytes(NUM_BYTES) + .numRows(NUM_ROWS) + .streamingBuffer(STREAMING_BUFFER) + .schema(TABLE_SCHEMA) + .build(); + + @Test + public void testToBuilder() { + compareTableDefinition(TABLE_DEFINITION, TABLE_DEFINITION.toBuilder().build()); + StandardTableDefinition tableDefinition = TABLE_DEFINITION.toBuilder().location("EU").build(); + assertEquals("EU", tableDefinition.location()); + tableDefinition = tableDefinition.toBuilder() + .location(LOCATION) + .build(); + compareTableDefinition(TABLE_DEFINITION, tableDefinition); + } + + @Test + public void testToBuilderIncomplete() { + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + assertEquals(tableDefinition, tableDefinition.toBuilder().build()); + } + + @Test + public void testBuilder() { + assertEquals(TableDefinition.Type.TABLE, TABLE_DEFINITION.type()); + assertEquals(TABLE_SCHEMA, TABLE_DEFINITION.schema()); + assertEquals(LOCATION, TABLE_DEFINITION.location()); + assertEquals(NUM_BYTES, TABLE_DEFINITION.numBytes()); + assertEquals(NUM_ROWS, TABLE_DEFINITION.numRows()); + assertEquals(STREAMING_BUFFER, TABLE_DEFINITION.streamingBuffer()); + } + + @Test + public void testToAndFromPb() { + assertTrue(TableDefinition.fromPb(TABLE_DEFINITION.toPb()) instanceof StandardTableDefinition); + compareTableDefinition(TABLE_DEFINITION, + TableDefinition.fromPb(TABLE_DEFINITION.toPb())); + } + + private void compareTableDefinition(StandardTableDefinition expected, + StandardTableDefinition value) { + assertEquals(expected, value); + assertEquals(expected.schema(), value.schema()); + assertEquals(expected.type(), value.type()); + assertEquals(expected.numBytes(), value.numBytes()); + assertEquals(expected.numRows(), value.numRows()); + assertEquals(expected.location(), value.location()); + assertEquals(expected.streamingBuffer(), value.streamingBuffer()); + assertEquals(expected.type(), value.type()); + assertEquals(expected.hashCode(), value.hashCode()); + } +} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableIdTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableIdTest.java index 9da050bf5951..bc013bfa5c31 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableIdTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableIdTest.java @@ -47,6 +47,11 @@ public void testToPbAndFromPb() { compareTableIds(TABLE_COMPLETE, TableId.fromPb(TABLE_COMPLETE.toPb())); } + @Test + public void testSetProjectId() { + assertEquals(TABLE_COMPLETE, TABLE.setProjectId("project")); + } + private void compareTableIds(TableId expected, TableId value) { assertEquals(expected, value); assertEquals(expected.project(), value.project()); diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableInfoTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableInfoTest.java index c636a31ad1ff..18b8be10d71e 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableInfoTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableInfoTest.java @@ -17,10 +17,8 @@ package com.google.gcloud.bigquery; import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; import com.google.common.collect.ImmutableList; -import com.google.gcloud.bigquery.TableInfo.StreamingBuffer; import org.junit.Test; @@ -28,6 +26,16 @@ public class TableInfoTest { + private static final String ETAG = "etag"; + private static final String ID = "project:dataset:table"; + private static final String SELF_LINK = "selfLink"; + private static final TableId TABLE_ID = TableId.of("dataset", "table"); + private static final String FRIENDLY_NAME = "friendlyName"; + private static final String DESCRIPTION = "description"; + private static final Long CREATION_TIME = 10L; + private static final Long EXPIRATION_TIME = 100L; + private static final Long LAST_MODIFIED_TIME = 20L; + private static final Field FIELD_SCHEMA1 = Field.builder("StringField", Field.Type.string()) .mode(Field.Mode.NULLABLE) @@ -44,62 +52,59 @@ public class TableInfoTest { .description("FieldDescription3") .build(); private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA1, FIELD_SCHEMA2, FIELD_SCHEMA3); - private static final String VIEW_QUERY = "VIEW QUERY"; + private static final Long NUM_BYTES = 42L; + private static final Long NUM_ROWS = 43L; + private static final String LOCATION = "US"; + private static final StandardTableDefinition.StreamingBuffer STREAMING_BUFFER = + new StandardTableDefinition.StreamingBuffer(1L, 2L, 3L); + private static final StandardTableDefinition TABLE_DEFINITION = StandardTableDefinition.builder() + .location(LOCATION) + .numBytes(NUM_BYTES) + .numRows(NUM_ROWS) + .streamingBuffer(STREAMING_BUFFER) + .schema(TABLE_SCHEMA) + .build(); + private static final List SOURCE_URIS = ImmutableList.of("uri1", "uri2"); private static final Integer MAX_BAD_RECORDS = 42; private static final Boolean IGNORE_UNKNOWN_VALUES = true; private static final String COMPRESSION = "GZIP"; - private static final ExternalDataConfiguration CONFIGURATION = ExternalDataConfiguration - .builder(SOURCE_URIS, TABLE_SCHEMA, FormatOptions.datastoreBackup()) - .compression(COMPRESSION) - .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) - .maxBadRecords(MAX_BAD_RECORDS) - .build(); - private static final String ETAG = "etag"; - private static final String ID = "project:dataset:table"; - private static final String SELF_LINK = "selfLink"; - private static final TableId TABLE_ID = TableId.of("dataset", "table"); - private static final String FRIENDLY_NAME = "friendlyName"; - private static final String DESCRIPTION = "description"; - private static final Long NUM_BYTES = 42L; - private static final Long NUM_ROWS = 43L; - private static final Long CREATION_TIME = 10L; - private static final Long EXPIRATION_TIME = 100L; - private static final Long LAST_MODIFIED_TIME = 20L; - private static final String LOCATION = "US"; - private static final StreamingBuffer STREAMING_BUFFER = new StreamingBuffer(1L, 2L, 3L); - private static final TableInfo TABLE_INFO = - TableInfo.builder(TABLE_ID, TABLE_SCHEMA) - .creationTime(CREATION_TIME) - .description(DESCRIPTION) - .etag(ETAG) - .expirationTime(EXPIRATION_TIME) - .friendlyName(FRIENDLY_NAME) - .id(ID) - .lastModifiedTime(LAST_MODIFIED_TIME) - .location(LOCATION) - .numBytes(NUM_BYTES) - .numRows(NUM_ROWS) - .selfLink(SELF_LINK) - .streamingBuffer(STREAMING_BUFFER) - .build(); - private static final ExternalTableInfo EXTERNAL_TABLE_INFO = - ExternalTableInfo.builder(TABLE_ID, CONFIGURATION) - .creationTime(CREATION_TIME) - .description(DESCRIPTION) - .etag(ETAG) - .expirationTime(EXPIRATION_TIME) - .friendlyName(FRIENDLY_NAME) - .id(ID) - .lastModifiedTime(LAST_MODIFIED_TIME) - .numBytes(NUM_BYTES) - .numRows(NUM_ROWS) - .selfLink(SELF_LINK) + private static final CsvOptions CSV_OPTIONS = CsvOptions.builder().build(); + private static final ExternalTableDefinition EXTERNAL_TABLE_DEFINITION = + ExternalTableDefinition.builder(SOURCE_URIS, TABLE_SCHEMA, CSV_OPTIONS) + .compression(COMPRESSION) + .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) + .maxBadRecords(MAX_BAD_RECORDS) .build(); + + private static final String VIEW_QUERY = "VIEW QUERY"; private static final List USER_DEFINED_FUNCTIONS = ImmutableList.of(UserDefinedFunction.inline("Function"), UserDefinedFunction.fromUri("URI")); - private static final ViewInfo VIEW_INFO = - ViewInfo.builder(TABLE_ID, VIEW_QUERY, USER_DEFINED_FUNCTIONS) + private static final ViewDefinition VIEW_TYPE = + ViewDefinition.builder(VIEW_QUERY, USER_DEFINED_FUNCTIONS).build(); + + private static final TableInfo TABLE_INFO = TableInfo.builder(TABLE_ID, TABLE_DEFINITION) + .creationTime(CREATION_TIME) + .description(DESCRIPTION) + .etag(ETAG) + .expirationTime(EXPIRATION_TIME) + .friendlyName(FRIENDLY_NAME) + .id(ID) + .lastModifiedTime(LAST_MODIFIED_TIME) + .selfLink(SELF_LINK) + .build(); + private static final TableInfo VIEW_INFO = TableInfo.builder(TABLE_ID, VIEW_TYPE) + .creationTime(CREATION_TIME) + .description(DESCRIPTION) + .etag(ETAG) + .expirationTime(EXPIRATION_TIME) + .friendlyName(FRIENDLY_NAME) + .id(ID) + .lastModifiedTime(LAST_MODIFIED_TIME) + .selfLink(SELF_LINK) + .build(); + private static final TableInfo EXTERNAL_TABLE_INFO = + TableInfo.builder(TABLE_ID, EXTERNAL_TABLE_DEFINITION) .creationTime(CREATION_TIME) .description(DESCRIPTION) .etag(ETAG) @@ -107,40 +112,37 @@ public class TableInfoTest { .friendlyName(FRIENDLY_NAME) .id(ID) .lastModifiedTime(LAST_MODIFIED_TIME) - .numBytes(NUM_BYTES) - .numRows(NUM_ROWS) .selfLink(SELF_LINK) .build(); @Test public void testToBuilder() { compareTableInfo(TABLE_INFO, TABLE_INFO.toBuilder().build()); - compareViewInfo(VIEW_INFO, VIEW_INFO.toBuilder().build()); - compareExternalTableInfo(EXTERNAL_TABLE_INFO, EXTERNAL_TABLE_INFO.toBuilder().build()); - BaseTableInfo tableInfo = TABLE_INFO.toBuilder() + compareTableInfo(VIEW_INFO, VIEW_INFO.toBuilder().build()); + compareTableInfo(EXTERNAL_TABLE_INFO, EXTERNAL_TABLE_INFO.toBuilder().build()); + TableInfo tableInfo = TABLE_INFO.toBuilder() .description("newDescription") .build(); assertEquals("newDescription", tableInfo.description()); tableInfo = tableInfo.toBuilder() .description("description") .build(); - compareBaseTableInfo(TABLE_INFO, tableInfo); + compareTableInfo(TABLE_INFO, tableInfo); } @Test public void testToBuilderIncomplete() { - BaseTableInfo tableInfo = TableInfo.of(TABLE_ID, TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(TABLE_ID, TABLE_DEFINITION); assertEquals(tableInfo, tableInfo.toBuilder().build()); - tableInfo = ViewInfo.of(TABLE_ID, VIEW_QUERY); + tableInfo = TableInfo.of(TABLE_ID, VIEW_TYPE); assertEquals(tableInfo, tableInfo.toBuilder().build()); - tableInfo = ExternalTableInfo.of(TABLE_ID, CONFIGURATION); + tableInfo = TableInfo.of(TABLE_ID, EXTERNAL_TABLE_DEFINITION); assertEquals(tableInfo, tableInfo.toBuilder().build()); } @Test public void testBuilder() { assertEquals(TABLE_ID, TABLE_INFO.tableId()); - assertEquals(TABLE_SCHEMA, TABLE_INFO.schema()); assertEquals(CREATION_TIME, TABLE_INFO.creationTime()); assertEquals(DESCRIPTION, TABLE_INFO.description()); assertEquals(ETAG, TABLE_INFO.etag()); @@ -148,16 +150,10 @@ public void testBuilder() { assertEquals(FRIENDLY_NAME, TABLE_INFO.friendlyName()); assertEquals(ID, TABLE_INFO.id()); assertEquals(LAST_MODIFIED_TIME, TABLE_INFO.lastModifiedTime()); - assertEquals(LOCATION, TABLE_INFO.location()); - assertEquals(NUM_BYTES, TABLE_INFO.numBytes()); - assertEquals(NUM_ROWS, TABLE_INFO.numRows()); + assertEquals(TABLE_DEFINITION, TABLE_INFO.definition()); assertEquals(SELF_LINK, TABLE_INFO.selfLink()); - assertEquals(STREAMING_BUFFER, TABLE_INFO.streamingBuffer()); - assertEquals(BaseTableInfo.Type.TABLE, TABLE_INFO.type()); assertEquals(TABLE_ID, VIEW_INFO.tableId()); - assertEquals(null, VIEW_INFO.schema()); - assertEquals(VIEW_QUERY, VIEW_INFO.query()); - assertEquals(BaseTableInfo.Type.VIEW, VIEW_INFO.type()); + assertEquals(VIEW_TYPE, VIEW_INFO.definition()); assertEquals(CREATION_TIME, VIEW_INFO.creationTime()); assertEquals(DESCRIPTION, VIEW_INFO.description()); assertEquals(ETAG, VIEW_INFO.etag()); @@ -165,13 +161,9 @@ public void testBuilder() { assertEquals(FRIENDLY_NAME, VIEW_INFO.friendlyName()); assertEquals(ID, VIEW_INFO.id()); assertEquals(LAST_MODIFIED_TIME, VIEW_INFO.lastModifiedTime()); - assertEquals(NUM_BYTES, VIEW_INFO.numBytes()); - assertEquals(NUM_ROWS, VIEW_INFO.numRows()); + assertEquals(VIEW_TYPE, VIEW_INFO.definition()); assertEquals(SELF_LINK, VIEW_INFO.selfLink()); - assertEquals(BaseTableInfo.Type.VIEW, VIEW_INFO.type()); assertEquals(TABLE_ID, EXTERNAL_TABLE_INFO.tableId()); - assertEquals(null, EXTERNAL_TABLE_INFO.schema()); - assertEquals(CONFIGURATION, EXTERNAL_TABLE_INFO.configuration()); assertEquals(CREATION_TIME, EXTERNAL_TABLE_INFO.creationTime()); assertEquals(DESCRIPTION, EXTERNAL_TABLE_INFO.description()); assertEquals(ETAG, EXTERNAL_TABLE_INFO.etag()); @@ -179,28 +171,28 @@ public void testBuilder() { assertEquals(FRIENDLY_NAME, EXTERNAL_TABLE_INFO.friendlyName()); assertEquals(ID, EXTERNAL_TABLE_INFO.id()); assertEquals(LAST_MODIFIED_TIME, EXTERNAL_TABLE_INFO.lastModifiedTime()); - assertEquals(NUM_BYTES, EXTERNAL_TABLE_INFO.numBytes()); - assertEquals(NUM_ROWS, EXTERNAL_TABLE_INFO.numRows()); + assertEquals(EXTERNAL_TABLE_DEFINITION, EXTERNAL_TABLE_INFO.definition()); assertEquals(SELF_LINK, EXTERNAL_TABLE_INFO.selfLink()); - assertEquals(BaseTableInfo.Type.EXTERNAL, EXTERNAL_TABLE_INFO.type()); } @Test public void testToAndFromPb() { - assertTrue(BaseTableInfo.fromPb(TABLE_INFO.toPb()) instanceof TableInfo); - compareTableInfo(TABLE_INFO, BaseTableInfo.fromPb(TABLE_INFO.toPb())); - assertTrue(BaseTableInfo.fromPb(VIEW_INFO.toPb()) instanceof ViewInfo); - compareViewInfo(VIEW_INFO, BaseTableInfo.fromPb(VIEW_INFO.toPb())); - assertTrue(BaseTableInfo.fromPb(EXTERNAL_TABLE_INFO.toPb()) instanceof ExternalTableInfo); - compareExternalTableInfo(EXTERNAL_TABLE_INFO, - BaseTableInfo.fromPb(EXTERNAL_TABLE_INFO.toPb())); + compareTableInfo(TABLE_INFO, TableInfo.fromPb(TABLE_INFO.toPb())); + compareTableInfo(VIEW_INFO, TableInfo.fromPb(VIEW_INFO.toPb())); + compareTableInfo(EXTERNAL_TABLE_INFO, TableInfo.fromPb(EXTERNAL_TABLE_INFO.toPb())); + } + + @Test + public void testSetProjectId() { + assertEquals("project", TABLE_INFO.setProjectId("project").tableId().project()); + assertEquals("project", EXTERNAL_TABLE_INFO.setProjectId("project").tableId().project()); + assertEquals("project", VIEW_INFO.setProjectId("project").tableId().project()); } - private void compareBaseTableInfo(BaseTableInfo expected, BaseTableInfo value) { + private void compareTableInfo(TableInfo expected, TableInfo value) { assertEquals(expected, value); assertEquals(expected.tableId(), value.tableId()); - assertEquals(expected.schema(), value.schema()); - assertEquals(expected.type(), value.type()); + assertEquals(expected.definition(), value.definition()); assertEquals(expected.creationTime(), value.creationTime()); assertEquals(expected.description(), value.description()); assertEquals(expected.etag(), value.etag()); @@ -208,29 +200,8 @@ private void compareBaseTableInfo(BaseTableInfo expected, BaseTableInfo value) { assertEquals(expected.friendlyName(), value.friendlyName()); assertEquals(expected.id(), value.id()); assertEquals(expected.lastModifiedTime(), value.lastModifiedTime()); - assertEquals(expected.numBytes(), value.numBytes()); - assertEquals(expected.numRows(), value.numRows()); assertEquals(expected.selfLink(), value.selfLink()); - assertEquals(expected.type(), value.type()); - } - - private void compareTableInfo(TableInfo expected, TableInfo value) { - compareBaseTableInfo(expected, value); - assertEquals(expected, value); - assertEquals(expected.location(), value.location()); - assertEquals(expected.streamingBuffer(), value.streamingBuffer()); - } - - private void compareViewInfo(ViewInfo expected, ViewInfo value) { - compareBaseTableInfo(expected, value); - assertEquals(expected, value); - assertEquals(expected.query(), value.query()); - assertEquals(expected.userDefinedFunctions(), value.userDefinedFunctions()); - } - - private void compareExternalTableInfo(ExternalTableInfo expected, ExternalTableInfo value) { - compareBaseTableInfo(expected, value); - assertEquals(expected, value); - assertEquals(expected.configuration(), value.configuration()); + assertEquals(expected.definition(), value.definition()); + assertEquals(expected.hashCode(), value.hashCode()); } } diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableTest.java index c931d768def1..4866ee9ab8ec 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableTest.java @@ -16,13 +16,14 @@ package com.google.gcloud.bigquery; +import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; +import static org.easymock.EasyMock.eq; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; @@ -35,29 +36,34 @@ import com.google.gcloud.bigquery.InsertAllRequest.RowToInsert; import org.junit.After; -import org.junit.Before; -import org.junit.Rule; import org.junit.Test; -import org.junit.rules.ExpectedException; import java.util.Iterator; import java.util.List; public class TableTest { + private static final String ETAG = "etag"; + private static final String ID = "project:dataset:table1"; + private static final String SELF_LINK = "selfLink"; + private static final String FRIENDLY_NAME = "friendlyName"; + private static final String DESCRIPTION = "description"; + private static final Long CREATION_TIME = 10L; + private static final Long EXPIRATION_TIME = 100L; + private static final Long LAST_MODIFIED_TIME = 20L; private static final TableId TABLE_ID1 = TableId.of("dataset", "table1"); private static final TableId TABLE_ID2 = TableId.of("dataset", "table2"); - private static final JobInfo COPY_JOB_INFO = CopyJobInfo.of(TABLE_ID2, TABLE_ID1); - private static final JobInfo LOAD_JOB_INFO = LoadJobInfo.builder( - LoadConfiguration.builder(TABLE_ID1).formatOptions(FormatOptions.json()).build(), - ImmutableList.of("URI")) - .build(); + private static final CopyJobConfiguration COPY_JOB_CONFIGURATION = + CopyJobConfiguration.of(TABLE_ID2, TABLE_ID1); + private static final JobInfo COPY_JOB_INFO = JobInfo.of(COPY_JOB_CONFIGURATION); + private static final JobInfo LOAD_JOB_INFO = + JobInfo.of(LoadJobConfiguration.of(TABLE_ID1, ImmutableList.of("URI"), FormatOptions.json())); private static final JobInfo EXTRACT_JOB_INFO = - ExtractJobInfo.builder(TABLE_ID1, ImmutableList.of("URI")) - .format("CSV") - .build(); + JobInfo.of(ExtractJobConfiguration.of(TABLE_ID1, ImmutableList.of("URI"), "CSV")); private static final Field FIELD = Field.of("FieldName", Field.Type.integer()); - private static final TableInfo TABLE_INFO = TableInfo.of(TABLE_ID1, Schema.of(FIELD)); + private static final TableDefinition TABLE_DEFINITION = + StandardTableDefinition.of(Schema.of(FIELD)); + private static final TableInfo TABLE_INFO = TableInfo.of(TABLE_ID1, TABLE_DEFINITION); private static final List ROWS_TO_INSERT = ImmutableList.of( RowToInsert.of("id1", ImmutableMap.of("key", "val1")), RowToInsert.of("id2", ImmutableMap.of("key", "val2"))); @@ -77,148 +83,198 @@ public class TableTest { private static final Iterable> ROWS = ImmutableList.of( (List) ImmutableList.of(FIELD_VALUE1), ImmutableList.of(FIELD_VALUE2)); - @Rule - public ExpectedException thrown = ExpectedException.none(); + private BigQuery serviceMockReturnsOptions = createStrictMock(BigQuery.class); + private BigQueryOptions mockOptions = createMock(BigQueryOptions.class); private BigQuery bigquery; + private Table expectedTable; private Table table; - @Before - public void setUp() throws Exception { + private void initializeExpectedTable(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); bigquery = createStrictMock(BigQuery.class); - table = new Table(bigquery, TABLE_INFO); + expectedTable = new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO)); + } + + private void initializeTable() { + table = new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO)); } @After public void tearDown() throws Exception { - verify(bigquery); + verify(bigquery, serviceMockReturnsOptions); } @Test - public void testInfo() throws Exception { - assertEquals(TABLE_INFO, table.info()); + public void testBuilder() { + initializeExpectedTable(2); replay(bigquery); + Table builtTable = new Table.Builder(serviceMockReturnsOptions, TABLE_ID1, TABLE_DEFINITION) + .creationTime(CREATION_TIME) + .description(DESCRIPTION) + .etag(ETAG) + .expirationTime(EXPIRATION_TIME) + .friendlyName(FRIENDLY_NAME) + .id(ID) + .lastModifiedTime(LAST_MODIFIED_TIME) + .selfLink(SELF_LINK) + .build(); + assertEquals(TABLE_ID1, builtTable.tableId()); + assertEquals(CREATION_TIME, builtTable.creationTime()); + assertEquals(DESCRIPTION, builtTable.description()); + assertEquals(ETAG, builtTable.etag()); + assertEquals(EXPIRATION_TIME, builtTable.expirationTime()); + assertEquals(FRIENDLY_NAME, builtTable.friendlyName()); + assertEquals(ID, builtTable.id()); + assertEquals(LAST_MODIFIED_TIME, builtTable.lastModifiedTime()); + assertEquals(TABLE_DEFINITION, builtTable.definition()); + assertEquals(SELF_LINK, builtTable.selfLink()); + assertSame(serviceMockReturnsOptions, builtTable.bigquery()); } @Test - public void testBigQuery() throws Exception { - assertSame(bigquery, table.bigquery()); + public void testToBuilder() { + initializeExpectedTable(4); replay(bigquery); + compareTable(expectedTable, expectedTable.toBuilder().build()); } @Test public void testExists_True() throws Exception { + initializeExpectedTable(1); BigQuery.TableOption[] expectedOptions = {BigQuery.TableOption.fields()}; - expect(bigquery.getTable(TABLE_INFO.tableId(), expectedOptions)).andReturn(TABLE_INFO); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO.tableId(), expectedOptions)).andReturn(expectedTable); replay(bigquery); + initializeTable(); assertTrue(table.exists()); } @Test public void testExists_False() throws Exception { + initializeExpectedTable(1); BigQuery.TableOption[] expectedOptions = {BigQuery.TableOption.fields()}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getTable(TABLE_INFO.tableId(), expectedOptions)).andReturn(null); replay(bigquery); + initializeTable(); assertFalse(table.exists()); } @Test public void testReload() throws Exception { + initializeExpectedTable(4); TableInfo updatedInfo = TABLE_INFO.toBuilder().description("Description").build(); - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(updatedInfo); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(expectedTable); replay(bigquery); + initializeTable(); Table updatedTable = table.reload(); - assertSame(bigquery, updatedTable.bigquery()); - assertEquals(updatedInfo, updatedTable.info()); + compareTable(expectedTable, updatedTable); } @Test public void testReloadNull() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(null); replay(bigquery); + initializeTable(); assertNull(table.reload()); } @Test public void testReloadWithOptions() throws Exception { + initializeExpectedTable(4); TableInfo updatedInfo = TABLE_INFO.toBuilder().description("Description").build(); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getTable(TABLE_INFO.tableId(), BigQuery.TableOption.fields())) - .andReturn(updatedInfo); + .andReturn(expectedTable); replay(bigquery); + initializeTable(); Table updatedTable = table.reload(BigQuery.TableOption.fields()); - assertSame(bigquery, updatedTable.bigquery()); - assertEquals(updatedInfo, updatedTable.info()); - } - - @Test - public void testUpdate() throws Exception { - BaseTableInfo updatedInfo = TABLE_INFO.toBuilder().description("Description").build(); - expect(bigquery.update(updatedInfo)).andReturn(updatedInfo); - replay(bigquery); - Table updatedTable = table.update(updatedInfo); - assertSame(bigquery, updatedTable.bigquery()); - assertEquals(updatedInfo, updatedTable.info()); + compareTable(expectedTable, updatedTable); } @Test - public void testUpdateWithDifferentId() throws Exception { - TableInfo updatedInfo = TABLE_INFO.toBuilder() - .tableId(TableId.of("dataset", "table3")) - .description("Description") - .build(); + public void testUpdate() { + initializeExpectedTable(4); + Table expectedUpdatedTable = expectedTable.toBuilder().description("Description").build(); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.update(eq(expectedTable))).andReturn(expectedUpdatedTable); replay(bigquery); - thrown.expect(IllegalArgumentException.class); - table.update(updatedInfo); + initializeTable(); + Table actualUpdatedTable = table.update(); + compareTable(expectedUpdatedTable, actualUpdatedTable); } @Test - public void testUpdateWithDifferentDatasetId() throws Exception { - TableInfo updatedInfo = TABLE_INFO.toBuilder() - .tableId(TableId.of("dataset1", "table1")) - .description("Description") - .build(); + public void testUpdateWithOptions() { + initializeExpectedTable(4); + Table expectedUpdatedTable = expectedTable.toBuilder().description("Description").build(); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.update(eq(expectedTable), eq(BigQuery.TableOption.fields()))) + .andReturn(expectedUpdatedTable); replay(bigquery); - thrown.expect(IllegalArgumentException.class); - table.update(updatedInfo); + initializeTable(); + Table actualUpdatedTable = table.update(BigQuery.TableOption.fields()); + compareTable(expectedUpdatedTable, actualUpdatedTable); } @Test - public void testUpdateWithOptions() throws Exception { - BaseTableInfo updatedInfo = TABLE_INFO.toBuilder().description("Description").build(); - expect(bigquery.update(updatedInfo, BigQuery.TableOption.fields())).andReturn(updatedInfo); + public void testDeleteTrue() { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.delete(TABLE_INFO.tableId())).andReturn(true); replay(bigquery); - Table updatedTable = table.update(updatedInfo, BigQuery.TableOption.fields()); - assertSame(bigquery, updatedTable.bigquery()); - assertEquals(updatedInfo, updatedTable.info()); + initializeTable(); + assertTrue(table.delete()); } @Test - public void testDelete() throws Exception { - expect(bigquery.delete(TABLE_INFO.tableId())).andReturn(true); + public void testDeleteFalse() { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.delete(TABLE_INFO.tableId())).andReturn(false); replay(bigquery); - assertTrue(table.delete()); + initializeTable(); + assertFalse(table.delete()); } @Test public void testInsert() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.insertAll(INSERT_ALL_REQUEST)).andReturn(EMPTY_INSERT_ALL_RESPONSE); replay(bigquery); + initializeTable(); InsertAllResponse response = table.insert(ROWS_TO_INSERT); assertSame(EMPTY_INSERT_ALL_RESPONSE, response); } @Test public void testInsertComplete() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.insertAll(INSERT_ALL_REQUEST_COMPLETE)).andReturn(EMPTY_INSERT_ALL_RESPONSE); replay(bigquery); + initializeTable(); InsertAllResponse response = table.insert(ROWS_TO_INSERT, true, true); assertSame(EMPTY_INSERT_ALL_RESPONSE, response); } @Test public void testList() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); PageImpl> tableDataPage = new PageImpl<>(null, "c", ROWS); expect(bigquery.listTableData(TABLE_ID1)).andReturn(tableDataPage); replay(bigquery); + initializeTable(); Page> dataPage = table.list(); Iterator> tableDataIterator = tableDataPage.values().iterator(); Iterator> dataIterator = dataPage.values().iterator(); @@ -227,10 +283,13 @@ public void testList() throws Exception { @Test public void testListWithOptions() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); PageImpl> tableDataPage = new PageImpl<>(null, "c", ROWS); expect(bigquery.listTableData(TABLE_ID1, BigQuery.TableDataListOption.maxResults(10L))) .andReturn(tableDataPage); replay(bigquery); + initializeTable(); Page> dataPage = table.list(BigQuery.TableDataListOption.maxResults(10L)); Iterator> tableDataIterator = tableDataPage.values().iterator(); Iterator> dataIterator = dataPage.values().iterator(); @@ -239,108 +298,110 @@ public void testListWithOptions() throws Exception { @Test public void testCopyFromString() throws Exception { - expect(bigquery.create(COPY_JOB_INFO)).andReturn(COPY_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(COPY_JOB_INFO)); + expect(bigquery.create(COPY_JOB_INFO)) + .andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.copy(TABLE_ID2.dataset(), TABLE_ID2.table()); - assertSame(bigquery, job.bigquery()); - assertEquals(COPY_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test public void testCopyFromId() throws Exception { - expect(bigquery.create(COPY_JOB_INFO)).andReturn(COPY_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(COPY_JOB_INFO)); + expect(bigquery.create(COPY_JOB_INFO)).andReturn(expectedJob); replay(bigquery); - Job job = table.copy(TABLE_ID2); - assertSame(bigquery, job.bigquery()); - assertEquals(COPY_JOB_INFO, job.info()); + initializeTable(); + Job job = table.copy(TABLE_ID2.dataset(), TABLE_ID2.table()); + assertSame(expectedJob, job); } @Test public void testLoadDataUri() throws Exception { - expect(bigquery.create(LOAD_JOB_INFO)).andReturn(LOAD_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(LOAD_JOB_INFO)); + expect(bigquery.create(LOAD_JOB_INFO)).andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.load(FormatOptions.json(), "URI"); - assertSame(bigquery, job.bigquery()); - assertEquals(LOAD_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test public void testLoadDataUris() throws Exception { - expect(bigquery.create(LOAD_JOB_INFO)).andReturn(LOAD_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(LOAD_JOB_INFO)); + expect(bigquery.create(LOAD_JOB_INFO)).andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.load(FormatOptions.json(), ImmutableList.of("URI")); - assertSame(bigquery, job.bigquery()); - assertEquals(LOAD_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test public void testExtractDataUri() throws Exception { - expect(bigquery.create(EXTRACT_JOB_INFO)).andReturn(EXTRACT_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(EXTRACT_JOB_INFO)); + expect(bigquery.create(EXTRACT_JOB_INFO)).andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.extract("CSV", "URI"); - assertSame(bigquery, job.bigquery()); - assertEquals(EXTRACT_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test public void testExtractDataUris() throws Exception { - expect(bigquery.create(EXTRACT_JOB_INFO)).andReturn(EXTRACT_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(EXTRACT_JOB_INFO)); + expect(bigquery.create(EXTRACT_JOB_INFO)).andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.extract("CSV", ImmutableList.of("URI")); - assertSame(bigquery, job.bigquery()); - assertEquals(EXTRACT_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test - public void testGetFromId() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(TABLE_INFO); + public void testBigquery() { + initializeExpectedTable(1); replay(bigquery); - Table loadedTable = Table.get(bigquery, TABLE_INFO.tableId()); - assertNotNull(loadedTable); - assertEquals(TABLE_INFO, loadedTable.info()); + assertSame(serviceMockReturnsOptions, expectedTable.bigquery()); } @Test - public void testGetFromStrings() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(TABLE_INFO); + public void testToAndFromPb() { + initializeExpectedTable(4); replay(bigquery); - Table loadedTable = Table.get(bigquery, TABLE_ID1.dataset(), TABLE_ID1.table()); - assertNotNull(loadedTable); - assertEquals(TABLE_INFO, loadedTable.info()); + compareTable(expectedTable, Table.fromPb(serviceMockReturnsOptions, expectedTable.toPb())); } - @Test - public void testGetFromIdNull() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(null); - replay(bigquery); - assertNull(Table.get(bigquery, TABLE_INFO.tableId())); - } - - @Test - public void testGetFromStringsNull() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(null); - replay(bigquery); - assertNull(Table.get(bigquery, TABLE_ID1.dataset(), TABLE_ID1.table())); + private void compareTable(Table expected, Table value) { + assertEquals(expected, value); + compareTableInfo(expected, value); + assertEquals(expected.bigquery().options(), value.bigquery().options()); } - @Test - public void testGetFromIdWithOptions() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId(), BigQuery.TableOption.fields())) - .andReturn(TABLE_INFO); - replay(bigquery); - Table loadedTable = Table.get(bigquery, TABLE_INFO.tableId(), BigQuery.TableOption.fields()); - assertNotNull(loadedTable); - assertEquals(TABLE_INFO, loadedTable.info()); - } - - @Test - public void testGetFromStringsWithOptions() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId(), BigQuery.TableOption.fields())) - .andReturn(TABLE_INFO); - replay(bigquery); - Table loadedTable = - Table.get(bigquery, TABLE_ID1.dataset(), TABLE_ID1.table(), BigQuery.TableOption.fields()); - assertNotNull(loadedTable); - assertEquals(TABLE_INFO, loadedTable.info()); + private void compareTableInfo(TableInfo expected, TableInfo value) { + assertEquals(expected, value); + assertEquals(expected.tableId(), value.tableId()); + assertEquals(expected.definition(), value.definition()); + assertEquals(expected.creationTime(), value.creationTime()); + assertEquals(expected.description(), value.description()); + assertEquals(expected.etag(), value.etag()); + assertEquals(expected.expirationTime(), value.expirationTime()); + assertEquals(expected.friendlyName(), value.friendlyName()); + assertEquals(expected.id(), value.id()); + assertEquals(expected.lastModifiedTime(), value.lastModifiedTime()); + assertEquals(expected.selfLink(), value.selfLink()); + assertEquals(expected.definition(), value.definition()); + assertEquals(expected.hashCode(), value.hashCode()); } } diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ViewDefinitionTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ViewDefinitionTest.java new file mode 100644 index 000000000000..ebab7a6e87ca --- /dev/null +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ViewDefinitionTest.java @@ -0,0 +1,75 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.bigquery; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import com.google.common.collect.ImmutableList; + +import org.junit.Test; + +import java.util.List; + +public class ViewDefinitionTest { + + private static final String VIEW_QUERY = "VIEW QUERY"; + private static final List USER_DEFINED_FUNCTIONS = + ImmutableList.of(UserDefinedFunction.inline("Function"), UserDefinedFunction.fromUri("URI")); + private static final ViewDefinition VIEW_DEFINITION = + ViewDefinition.builder(VIEW_QUERY, USER_DEFINED_FUNCTIONS).build(); + + @Test + public void testToBuilder() { + compareViewDefinition(VIEW_DEFINITION, VIEW_DEFINITION.toBuilder().build()); + ViewDefinition viewDefinition = VIEW_DEFINITION.toBuilder() + .query("NEW QUERY") + .build(); + assertEquals("NEW QUERY", viewDefinition.query()); + viewDefinition = viewDefinition.toBuilder() + .query(VIEW_QUERY) + .build(); + compareViewDefinition(VIEW_DEFINITION, viewDefinition); + } + + @Test + public void testToBuilderIncomplete() { + TableDefinition viewDefinition = ViewDefinition.of(VIEW_QUERY); + assertEquals(viewDefinition, viewDefinition.toBuilder().build()); + } + + @Test + public void testBuilder() { + assertEquals(VIEW_QUERY, VIEW_DEFINITION.query()); + assertEquals(TableDefinition.Type.VIEW, VIEW_DEFINITION.type()); + assertEquals(USER_DEFINED_FUNCTIONS, VIEW_DEFINITION.userDefinedFunctions()); + } + + @Test + public void testToAndFromPb() { + assertTrue(TableDefinition.fromPb(VIEW_DEFINITION.toPb()) instanceof ViewDefinition); + compareViewDefinition(VIEW_DEFINITION, + TableDefinition.fromPb(VIEW_DEFINITION.toPb())); + } + + private void compareViewDefinition(ViewDefinition expected, ViewDefinition value) { + assertEquals(expected, value); + assertEquals(expected.query(), value.query()); + assertEquals(expected.userDefinedFunctions(), value.userDefinedFunctions()); + assertEquals(expected.hashCode(), value.hashCode()); + } +} diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/LoadConfigurationTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/WriteChannelConfigurationTest.java similarity index 77% rename from gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/LoadConfigurationTest.java rename to gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/WriteChannelConfigurationTest.java index e72101829cdf..dfde4795dacd 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/LoadConfigurationTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/WriteChannelConfigurationTest.java @@ -28,7 +28,7 @@ import java.nio.charset.StandardCharsets; import java.util.List; -public class LoadConfigurationTest { +public class WriteChannelConfigurationTest { private static final CsvOptions CSV_OPTIONS = CsvOptions.builder() .allowJaggedRows(true) @@ -47,20 +47,21 @@ public class LoadConfigurationTest { .description("FieldDescription") .build(); private static final Schema TABLE_SCHEMA = Schema.of(FIELD_SCHEMA); - private static final LoadConfiguration LOAD_CONFIGURATION = LoadConfiguration.builder(TABLE_ID) - .createDisposition(CREATE_DISPOSITION) - .writeDisposition(WRITE_DISPOSITION) - .formatOptions(CSV_OPTIONS) - .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) - .maxBadRecords(MAX_BAD_RECORDS) - .projectionFields(PROJECTION_FIELDS) - .schema(TABLE_SCHEMA) - .build(); + private static final WriteChannelConfiguration LOAD_CONFIGURATION = + WriteChannelConfiguration.builder(TABLE_ID) + .createDisposition(CREATE_DISPOSITION) + .writeDisposition(WRITE_DISPOSITION) + .formatOptions(CSV_OPTIONS) + .ignoreUnknownValues(IGNORE_UNKNOWN_VALUES) + .maxBadRecords(MAX_BAD_RECORDS) + .projectionFields(PROJECTION_FIELDS) + .schema(TABLE_SCHEMA) + .build(); @Test public void testToBuilder() { compareLoadConfiguration(LOAD_CONFIGURATION, LOAD_CONFIGURATION.toBuilder().build()); - LoadConfiguration configuration = LOAD_CONFIGURATION.toBuilder() + WriteChannelConfiguration configuration = LOAD_CONFIGURATION.toBuilder() .destinationTable(TableId.of("dataset", "newTable")) .build(); assertEquals("newTable", configuration.destinationTable().table()); @@ -70,9 +71,9 @@ public void testToBuilder() { @Test public void testOf() { - LoadConfiguration configuration = LoadConfiguration.of(TABLE_ID); + WriteChannelConfiguration configuration = WriteChannelConfiguration.of(TABLE_ID); assertEquals(TABLE_ID, configuration.destinationTable()); - configuration = LoadConfiguration.of(TABLE_ID, CSV_OPTIONS); + configuration = WriteChannelConfiguration.of(TABLE_ID, CSV_OPTIONS); assertEquals(TABLE_ID, configuration.destinationTable()); assertEquals(FORMAT, configuration.format()); assertEquals(CSV_OPTIONS, configuration.csvOptions()); @@ -80,7 +81,7 @@ public void testOf() { @Test public void testToBuilderIncomplete() { - LoadConfiguration configuration = LoadConfiguration.of(TABLE_ID); + WriteChannelConfiguration configuration = WriteChannelConfiguration.of(TABLE_ID); compareLoadConfiguration(configuration, configuration.toBuilder().build()); } @@ -99,14 +100,15 @@ public void testBuilder() { @Test public void testToPbAndFromPb() { - assertNull(LOAD_CONFIGURATION.toPb().getSourceUris()); + assertNull(LOAD_CONFIGURATION.toPb().getLoad().getSourceUris()); compareLoadConfiguration(LOAD_CONFIGURATION, - LoadConfiguration.fromPb(LOAD_CONFIGURATION.toPb())); - LoadConfiguration configuration = LoadConfiguration.of(TABLE_ID); - compareLoadConfiguration(configuration, LoadConfiguration.fromPb(configuration.toPb())); + WriteChannelConfiguration.fromPb(LOAD_CONFIGURATION.toPb())); + WriteChannelConfiguration configuration = WriteChannelConfiguration.of(TABLE_ID); + compareLoadConfiguration(configuration, WriteChannelConfiguration.fromPb(configuration.toPb())); } - private void compareLoadConfiguration(LoadConfiguration expected, LoadConfiguration value) { + private void compareLoadConfiguration(WriteChannelConfiguration expected, + WriteChannelConfiguration value) { assertEquals(expected, value); assertEquals(expected.hashCode(), value.hashCode()); assertEquals(expected.toString(), value.toString()); diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ITBigQueryTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/it/ITBigQueryTest.java similarity index 66% rename from gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ITBigQueryTest.java rename to gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/it/ITBigQueryTest.java index f672815bcb7a..63a0551ece33 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ITBigQueryTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/it/ITBigQueryTest.java @@ -14,14 +14,8 @@ * limitations under the License. */ -package com.google.gcloud.bigquery; - -import static com.google.gcloud.bigquery.BigQuery.DatasetField; -import static com.google.gcloud.bigquery.BigQuery.JobField; -import static com.google.gcloud.bigquery.BigQuery.JobListOption; -import static com.google.gcloud.bigquery.BigQuery.JobOption; -import static com.google.gcloud.bigquery.BigQuery.TableField; -import static com.google.gcloud.bigquery.BigQuery.TableOption; +package com.google.gcloud.bigquery.it; + import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; @@ -32,7 +26,43 @@ import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.gcloud.Page; +import com.google.gcloud.WriteChannel; +import com.google.gcloud.bigquery.BigQuery; +import com.google.gcloud.bigquery.BigQuery.DatasetField; import com.google.gcloud.bigquery.BigQuery.DatasetOption; +import com.google.gcloud.bigquery.BigQuery.JobField; +import com.google.gcloud.bigquery.BigQuery.JobListOption; +import com.google.gcloud.bigquery.BigQuery.JobOption; +import com.google.gcloud.bigquery.BigQuery.TableField; +import com.google.gcloud.bigquery.BigQuery.TableOption; +import com.google.gcloud.bigquery.BigQueryError; +import com.google.gcloud.bigquery.BigQueryException; +import com.google.gcloud.bigquery.CopyJobConfiguration; +import com.google.gcloud.bigquery.Dataset; +import com.google.gcloud.bigquery.DatasetId; +import com.google.gcloud.bigquery.DatasetInfo; +import com.google.gcloud.bigquery.ExternalTableDefinition; +import com.google.gcloud.bigquery.ExtractJobConfiguration; +import com.google.gcloud.bigquery.Field; +import com.google.gcloud.bigquery.FieldValue; +import com.google.gcloud.bigquery.FormatOptions; +import com.google.gcloud.bigquery.InsertAllRequest; +import com.google.gcloud.bigquery.InsertAllResponse; +import com.google.gcloud.bigquery.Job; +import com.google.gcloud.bigquery.JobInfo; +import com.google.gcloud.bigquery.JobStatistics; +import com.google.gcloud.bigquery.LoadJobConfiguration; +import com.google.gcloud.bigquery.QueryJobConfiguration; +import com.google.gcloud.bigquery.QueryRequest; +import com.google.gcloud.bigquery.QueryResponse; +import com.google.gcloud.bigquery.Schema; +import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; +import com.google.gcloud.bigquery.TableDefinition; +import com.google.gcloud.bigquery.TableId; +import com.google.gcloud.bigquery.TableInfo; +import com.google.gcloud.bigquery.ViewDefinition; +import com.google.gcloud.bigquery.WriteChannelConfiguration; import com.google.gcloud.bigquery.testing.RemoteBigQueryHelper; import com.google.gcloud.storage.BlobInfo; import com.google.gcloud.storage.BucketInfo; @@ -45,7 +75,6 @@ import org.junit.Test; import org.junit.rules.Timeout; -import java.io.FileNotFoundException; import java.io.IOException; import java.nio.ByteBuffer; import java.nio.charset.StandardCharsets; @@ -58,7 +87,7 @@ public class ITBigQueryTest { - private static final Logger log = Logger.getLogger(ITBigQueryTest.class.getName()); + private static final Logger LOG = Logger.getLogger(ITBigQueryTest.class.getName()); private static final String DATASET = RemoteBigQueryHelper.generateDatasetName(); private static final String DESCRIPTION = "Test dataset"; private static final String OTHER_DATASET = RemoteBigQueryHelper.generateDatasetName(); @@ -140,7 +169,7 @@ public class ITBigQueryTest { public Timeout globalTimeout = Timeout.seconds(300); @BeforeClass - public static void beforeClass() throws IOException, InterruptedException { + public static void beforeClass() throws InterruptedException { RemoteBigQueryHelper bigqueryHelper = RemoteBigQueryHelper.create(); RemoteGcsHelper gcsHelper = RemoteGcsHelper.create(); bigquery = bigqueryHelper.options().service(); @@ -152,15 +181,14 @@ public static void beforeClass() throws IOException, InterruptedException { JSON_CONTENT.getBytes(StandardCharsets.UTF_8)); DatasetInfo info = DatasetInfo.builder(DATASET).description(DESCRIPTION).build(); bigquery.create(info); - LoadConfiguration configuration = LoadConfiguration.builder(TABLE_ID, FormatOptions.json()) + LoadJobConfiguration configuration = LoadJobConfiguration.builder( + TABLE_ID, "gs://" + BUCKET + "/" + JSON_LOAD_FILE, FormatOptions.json()) .createDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED) .schema(TABLE_SCHEMA) .build(); - LoadJobInfo job = LoadJobInfo.of(configuration, "gs://" + BUCKET + "/" + JSON_LOAD_FILE); - job = bigquery.create(job); - while (job.status().state() != JobStatus.State.DONE) { + Job job = bigquery.create(JobInfo.of(configuration)); + while (!job.isDone()) { Thread.sleep(1000); - job = bigquery.getJob(job.jobId()); } assertNull(job.status().error()); } @@ -170,16 +198,17 @@ public static void afterClass() throws ExecutionException, InterruptedException if (bigquery != null) { RemoteBigQueryHelper.forceDelete(bigquery, DATASET); } - if (storage != null && !RemoteGcsHelper.forceDelete(storage, BUCKET, 10, TimeUnit.SECONDS)) { - if (log.isLoggable(Level.WARNING)) { - log.log(Level.WARNING, "Deletion of bucket {0} timed out, bucket is not empty", BUCKET); + if (storage != null) { + boolean wasDeleted = RemoteGcsHelper.forceDelete(storage, BUCKET, 10, TimeUnit.SECONDS); + if (!wasDeleted && LOG.isLoggable(Level.WARNING)) { + LOG.log(Level.WARNING, "Deletion of bucket {0} timed out, bucket is not empty", BUCKET); } } } @Test public void testGetDataset() { - DatasetInfo dataset = bigquery.getDataset(DATASET); + Dataset dataset = bigquery.getDataset(DATASET); assertEquals(bigquery.options().projectId(), dataset.datasetId().project()); assertEquals(DATASET, dataset.datasetId().dataset()); assertEquals(DESCRIPTION, dataset.description()); @@ -192,7 +221,7 @@ public void testGetDataset() { @Test public void testGetDatasetWithSelectedFields() { - DatasetInfo dataset = bigquery.getDataset(DATASET, + Dataset dataset = bigquery.getDataset(DATASET, DatasetOption.fields(DatasetField.CREATION_TIME)); assertEquals(bigquery.options().projectId(), dataset.datasetId().project()); assertEquals(DATASET, dataset.datasetId().dataset()); @@ -210,29 +239,29 @@ public void testGetDatasetWithSelectedFields() { @Test public void testUpdateDataset() { - DatasetInfo dataset = bigquery.create(DatasetInfo.builder(OTHER_DATASET) + Dataset dataset = bigquery.create(DatasetInfo.builder(OTHER_DATASET) .description("Some Description") .build()); assertNotNull(dataset); assertEquals(bigquery.options().projectId(), dataset.datasetId().project()); assertEquals(OTHER_DATASET, dataset.datasetId().dataset()); assertEquals("Some Description", dataset.description()); - DatasetInfo updatedDataset = + Dataset updatedDataset = bigquery.update(dataset.toBuilder().description("Updated Description").build()); assertEquals("Updated Description", updatedDataset.description()); - assertTrue(bigquery.delete(OTHER_DATASET)); + assertTrue(dataset.delete()); } @Test public void testUpdateDatasetWithSelectedFields() { - DatasetInfo dataset = bigquery.create(DatasetInfo.builder(OTHER_DATASET) + Dataset dataset = bigquery.create(DatasetInfo.builder(OTHER_DATASET) .description("Some Description") .build()); assertNotNull(dataset); assertEquals(bigquery.options().projectId(), dataset.datasetId().project()); assertEquals(OTHER_DATASET, dataset.datasetId().dataset()); assertEquals("Some Description", dataset.description()); - DatasetInfo updatedDataset = + Dataset updatedDataset = bigquery.update(dataset.toBuilder().description("Updated Description").build(), DatasetOption.fields(DatasetField.DESCRIPTION)); assertEquals("Updated Description", updatedDataset.description()); @@ -245,7 +274,7 @@ public void testUpdateDatasetWithSelectedFields() { assertNull(updatedDataset.lastModified()); assertNull(updatedDataset.location()); assertNull(updatedDataset.selfLink()); - assertTrue(bigquery.delete(OTHER_DATASET)); + assertTrue(dataset.delete()); } @Test @@ -257,61 +286,63 @@ public void testGetNonExistingTable() { public void testCreateAndGetTable() { String tableName = "test_create_and_get_table"; TableId tableId = TableId.of(DATASET, tableName); - BaseTableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, TABLE_SCHEMA)); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(tableName, createdTableInfo.tableId().table()); - BaseTableInfo remoteTableInfo = bigquery.getTable(DATASET, tableName); - assertNotNull(remoteTableInfo); - assertTrue(remoteTableInfo instanceof TableInfo); - assertEquals(createdTableInfo.tableId(), remoteTableInfo.tableId()); - assertEquals(BaseTableInfo.Type.TABLE, remoteTableInfo.type()); - assertEquals(TABLE_SCHEMA, remoteTableInfo.schema()); - assertNotNull(remoteTableInfo.creationTime()); - assertNotNull(remoteTableInfo.lastModifiedTime()); - assertNotNull(remoteTableInfo.numBytes()); - assertNotNull(remoteTableInfo.numRows()); - assertTrue(bigquery.delete(DATASET, tableName)); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + Table createdTable = bigquery.create(TableInfo.of(tableId, tableDefinition)); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(tableName, createdTable.tableId().table()); + Table remoteTable = bigquery.getTable(DATASET, tableName); + assertNotNull(remoteTable); + assertTrue(remoteTable.definition() instanceof StandardTableDefinition); + assertEquals(createdTable.tableId(), remoteTable.tableId()); + assertEquals(TableDefinition.Type.TABLE, remoteTable.definition().type()); + assertEquals(TABLE_SCHEMA, remoteTable.definition().schema()); + assertNotNull(remoteTable.creationTime()); + assertNotNull(remoteTable.lastModifiedTime()); + assertNotNull(remoteTable.definition().numBytes()); + assertNotNull(remoteTable.definition().numRows()); + assertTrue(remoteTable.delete()); } @Test public void testCreateAndGetTableWithSelectedField() { String tableName = "test_create_and_get_selected_fields_table"; TableId tableId = TableId.of(DATASET, tableName); - BaseTableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, TABLE_SCHEMA)); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(tableName, createdTableInfo.tableId().table()); - BaseTableInfo remoteTableInfo = bigquery.getTable(DATASET, tableName, + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + Table createdTable = bigquery.create(TableInfo.of(tableId, tableDefinition)); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(tableName, createdTable.tableId().table()); + Table remoteTable = bigquery.getTable(DATASET, tableName, TableOption.fields(TableField.CREATION_TIME)); - assertNotNull(remoteTableInfo); - assertTrue(remoteTableInfo instanceof TableInfo); - assertEquals(createdTableInfo.tableId(), remoteTableInfo.tableId()); - assertEquals(BaseTableInfo.Type.TABLE, remoteTableInfo.type()); - assertNotNull(remoteTableInfo.creationTime()); - assertNull(remoteTableInfo.schema()); - assertNull(remoteTableInfo.lastModifiedTime()); - assertNull(remoteTableInfo.numBytes()); - assertNull(remoteTableInfo.numRows()); - assertTrue(bigquery.delete(DATASET, tableName)); + assertNotNull(remoteTable); + assertTrue(remoteTable.definition() instanceof StandardTableDefinition); + assertEquals(createdTable.tableId(), remoteTable.tableId()); + assertEquals(TableDefinition.Type.TABLE, remoteTable.definition().type()); + assertNotNull(remoteTable.creationTime()); + assertNull(remoteTable.definition().schema()); + assertNull(remoteTable.lastModifiedTime()); + assertNull(remoteTable.definition().numBytes()); + assertNull(remoteTable.definition().numRows()); + assertTrue(remoteTable.delete()); } @Test public void testCreateExternalTable() throws InterruptedException { String tableName = "test_create_external_table"; TableId tableId = TableId.of(DATASET, tableName); - ExternalDataConfiguration externalDataConfiguration = ExternalDataConfiguration.of( + ExternalTableDefinition externalTableDefinition = ExternalTableDefinition.of( "gs://" + BUCKET + "/" + JSON_LOAD_FILE, TABLE_SCHEMA, FormatOptions.json()); - BaseTableInfo tableInfo = ExternalTableInfo.of(tableId, externalDataConfiguration); - BaseTableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(tableName, createdTableInfo.tableId().table()); - BaseTableInfo remoteTableInfo = bigquery.getTable(DATASET, tableName); - assertNotNull(remoteTableInfo); - assertTrue(remoteTableInfo instanceof ExternalTableInfo); - assertEquals(createdTableInfo.tableId(), remoteTableInfo.tableId()); - assertEquals(TABLE_SCHEMA, remoteTableInfo.schema()); + TableInfo tableInfo = TableInfo.of(tableId, externalTableDefinition); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(tableName, createdTable.tableId().table()); + Table remoteTable = bigquery.getTable(DATASET, tableName); + assertNotNull(remoteTable); + assertTrue(remoteTable.definition() instanceof ExternalTableDefinition); + assertEquals(createdTable.tableId(), remoteTable.tableId()); + assertEquals(TABLE_SCHEMA, remoteTable.definition().schema()); QueryRequest request = QueryRequest.builder( "SELECT TimestampField, StringField, IntegerField, BooleanField FROM " + DATASET + "." + tableName) @@ -343,24 +374,25 @@ public void testCreateExternalTable() throws InterruptedException { rowCount++; } assertEquals(4, rowCount); - assertTrue(bigquery.delete(DATASET, tableName)); + assertTrue(remoteTable.delete()); } @Test public void testCreateViewTable() throws InterruptedException { String tableName = "test_create_view_table"; TableId tableId = TableId.of(DATASET, tableName); - BaseTableInfo tableInfo = ViewInfo.of(tableId, - "SELECT TimestampField, StringField, BooleanField FROM " + DATASET + "." + ViewDefinition viewDefinition = + ViewDefinition.of("SELECT TimestampField, StringField, BooleanField FROM " + DATASET + "." + TABLE_ID.table()); - BaseTableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(tableName, createdTableInfo.tableId().table()); - BaseTableInfo remoteTableInfo = bigquery.getTable(DATASET, tableName); - assertNotNull(remoteTableInfo); - assertEquals(createdTableInfo.tableId(), remoteTableInfo.tableId()); - assertTrue(remoteTableInfo instanceof ViewInfo); + TableInfo tableInfo = TableInfo.of(tableId, viewDefinition); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(tableName, createdTable.tableId().table()); + Table remoteTable = bigquery.getTable(DATASET, tableName); + assertNotNull(remoteTable); + assertEquals(createdTable.tableId(), remoteTable.tableId()); + assertTrue(remoteTable.definition() instanceof ViewDefinition); Schema expectedSchema = Schema.builder() .addField( Field.builder("TimestampField", Field.Type.timestamp()) @@ -375,7 +407,7 @@ public void testCreateViewTable() throws InterruptedException { .mode(Field.Mode.NULLABLE) .build()) .build(); - assertEquals(expectedSchema, remoteTableInfo.schema()); + assertEquals(expectedSchema, remoteTable.definition().schema()); QueryRequest request = QueryRequest.builder("SELECT * FROM " + tableName) .defaultDataset(DatasetId.of(DATASET)) .maxWaitTime(60000L) @@ -400,65 +432,68 @@ public void testCreateViewTable() throws InterruptedException { rowCount++; } assertEquals(2, rowCount); - assertTrue(bigquery.delete(DATASET, tableName)); + assertTrue(remoteTable.delete()); } @Test public void testListTables() { String tableName = "test_list_tables"; - BaseTableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), TABLE_SCHEMA); - BaseTableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - Page tables = bigquery.listTables(DATASET); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), tableDefinition); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + Page
tables = bigquery.listTables(DATASET); boolean found = false; - Iterator tableIterator = tables.values().iterator(); + Iterator
tableIterator = tables.values().iterator(); while (tableIterator.hasNext() && !found) { - if (tableIterator.next().tableId().equals(createdTableInfo.tableId())) { + if (tableIterator.next().tableId().equals(createdTable.tableId())) { found = true; } } assertTrue(found); - assertTrue(bigquery.delete(DATASET, tableName)); + assertTrue(createdTable.delete()); } @Test public void testUpdateTable() { String tableName = "test_update_table"; - BaseTableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), TABLE_SCHEMA); - BaseTableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - BaseTableInfo updatedTableInfo = bigquery.update(tableInfo.toBuilder() - .description("newDescription").build()); - assertEquals(DATASET, updatedTableInfo.tableId().dataset()); - assertEquals(tableName, updatedTableInfo.tableId().table()); - assertEquals(TABLE_SCHEMA, updatedTableInfo.schema()); - assertEquals("newDescription", updatedTableInfo.description()); - assertTrue(bigquery.delete(DATASET, tableName)); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), tableDefinition); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + Table updatedTable = + bigquery.update(tableInfo.toBuilder().description("newDescription").build()); + assertEquals(DATASET, updatedTable.tableId().dataset()); + assertEquals(tableName, updatedTable.tableId().table()); + assertEquals(TABLE_SCHEMA, updatedTable.definition().schema()); + assertEquals("newDescription", updatedTable.description()); + assertTrue(updatedTable.delete()); } @Test public void testUpdateTableWithSelectedFields() { String tableName = "test_update_with_selected_fields_table"; - BaseTableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), TABLE_SCHEMA); - BaseTableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - BaseTableInfo updatedTableInfo = bigquery.update(tableInfo.toBuilder().description("newDescr") - .build(), TableOption.fields(TableField.DESCRIPTION)); - assertTrue(updatedTableInfo instanceof TableInfo); - assertEquals(DATASET, updatedTableInfo.tableId().dataset()); - assertEquals(tableName, updatedTableInfo.tableId().table()); - assertEquals("newDescr", updatedTableInfo.description()); - assertNull(updatedTableInfo.schema()); - assertNull(updatedTableInfo.lastModifiedTime()); - assertNull(updatedTableInfo.numBytes()); - assertNull(updatedTableInfo.numRows()); - assertTrue(bigquery.delete(DATASET, tableName)); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), tableDefinition); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + Table updatedTable = bigquery.update(tableInfo.toBuilder().description("newDescr").build(), + TableOption.fields(TableField.DESCRIPTION)); + assertTrue(updatedTable.definition() instanceof StandardTableDefinition); + assertEquals(DATASET, updatedTable.tableId().dataset()); + assertEquals(tableName, updatedTable.tableId().table()); + assertEquals("newDescr", updatedTable.description()); + assertNull(updatedTable.definition().schema()); + assertNull(updatedTable.lastModifiedTime()); + assertNull(updatedTable.definition().numBytes()); + assertNull(updatedTable.definition().numRows()); + assertTrue(createdTable.delete()); } @Test public void testUpdateNonExistingTable() { - TableInfo tableInfo = - TableInfo.of(TableId.of(DATASET, "test_update_non_existing_table"), SIMPLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, "test_update_non_existing_table"), + StandardTableDefinition.of(SIMPLE_SCHEMA)); try { bigquery.update(tableInfo); fail("BigQueryException was expected"); @@ -478,7 +513,8 @@ public void testDeleteNonExistingTable() { @Test public void testInsertAll() { String tableName = "test_insert_all_table"; - BaseTableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), TABLE_SCHEMA); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), tableDefinition); assertNotNull(bigquery.create(tableInfo)); InsertAllRequest request = InsertAllRequest.builder(tableInfo.tableId()) .addRow(ImmutableMap.of( @@ -509,7 +545,8 @@ public void testInsertAll() { @Test public void testInsertAllWithSuffix() throws InterruptedException { String tableName = "test_insert_all_with_suffix_table"; - BaseTableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), TABLE_SCHEMA); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), tableDefinition); assertNotNull(bigquery.create(tableInfo)); InsertAllRequest request = InsertAllRequest.builder(tableInfo.tableId()) .addRow(ImmutableMap.of( @@ -536,20 +573,21 @@ public void testInsertAllWithSuffix() throws InterruptedException { assertFalse(response.hasErrors()); assertEquals(0, response.insertErrors().size()); String newTableName = tableName + "_suffix"; - BaseTableInfo suffixTable = bigquery.getTable(DATASET, newTableName, TableOption.fields()); + Table suffixTable = bigquery.getTable(DATASET, newTableName, TableOption.fields()); // wait until the new table is created. If the table is never created the test will time-out while (suffixTable == null) { Thread.sleep(1000L); suffixTable = bigquery.getTable(DATASET, newTableName, TableOption.fields()); } assertTrue(bigquery.delete(TableId.of(DATASET, tableName))); - assertTrue(bigquery.delete(TableId.of(DATASET, newTableName))); + assertTrue(suffixTable.delete()); } @Test public void testInsertAllWithErrors() { String tableName = "test_insert_all_with_errors_table"; - BaseTableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), TABLE_SCHEMA); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), tableDefinition); assertNotNull(bigquery.create(tableInfo)); InsertAllRequest request = InsertAllRequest.builder(tableInfo.tableId()) .addRow(ImmutableMap.of( @@ -646,14 +684,15 @@ public void testQuery() throws InterruptedException { rowCount++; } assertEquals(2, rowCount); - QueryJobInfo queryJob = bigquery.getJob(response.jobId()); - assertNotNull(queryJob.statistics().queryPlan()); + Job queryJob = bigquery.getJob(response.jobId()); + JobStatistics.QueryStatistics statistics = queryJob.statistics(); + assertNotNull(statistics.queryPlan()); } @Test public void testListJobs() { - Page jobs = bigquery.listJobs(); - for (JobInfo job : jobs.values()) { + Page jobs = bigquery.listJobs(); + for (Job job : jobs.values()) { assertNotNull(job.jobId()); assertNotNull(job.statistics()); assertNotNull(job.status()); @@ -664,8 +703,8 @@ public void testListJobs() { @Test public void testListJobsWithSelectedFields() { - Page jobs = bigquery.listJobs(JobListOption.fields(JobField.USER_EMAIL)); - for (JobInfo job : jobs.values()) { + Page jobs = bigquery.listJobs(JobListOption.fields(JobField.USER_EMAIL)); + for (Job job : jobs.values()) { assertNotNull(job.jobId()); assertNotNull(job.status()); assertNotNull(job.userEmail()); @@ -675,67 +714,73 @@ public void testListJobsWithSelectedFields() { } @Test - public void testCreateAndGetJob() throws InterruptedException { + public void testCreateAndGetJob() { String sourceTableName = "test_create_and_get_job_source_table"; String destinationTableName = "test_create_and_get_job_destination_table"; TableId sourceTable = TableId.of(DATASET, sourceTableName); - BaseTableInfo tableInfo = TableInfo.of(sourceTable, SIMPLE_SCHEMA); - BaseTableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(sourceTableName, createdTableInfo.tableId().table()); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(sourceTable, tableDefinition); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(sourceTableName, createdTable.tableId().table()); TableId destinationTable = TableId.of(DATASET, destinationTableName); - CopyJobInfo job = CopyJobInfo.of(destinationTable, sourceTable); - CopyJobInfo createdJob = bigquery.create(job); - CopyJobInfo remoteJob = bigquery.getJob(createdJob.jobId()); + CopyJobConfiguration copyJobConfiguration = + CopyJobConfiguration.of(destinationTable, sourceTable); + Job createdJob = bigquery.create(JobInfo.of(copyJobConfiguration)); + Job remoteJob = bigquery.getJob(createdJob.jobId()); assertEquals(createdJob.jobId(), remoteJob.jobId()); - assertEquals(createdJob.sourceTables(), remoteJob.sourceTables()); - assertEquals(createdJob.destinationTable(), remoteJob.destinationTable()); - assertEquals(createdJob.createDisposition(), remoteJob.createDisposition()); - assertEquals(createdJob.writeDisposition(), remoteJob.writeDisposition()); + CopyJobConfiguration createdConfiguration = createdJob.configuration(); + CopyJobConfiguration remoteConfiguration = remoteJob.configuration(); + assertEquals(createdConfiguration.sourceTables(), remoteConfiguration.sourceTables()); + assertEquals(createdConfiguration.destinationTable(), remoteConfiguration.destinationTable()); + assertEquals(createdConfiguration.createDisposition(), remoteConfiguration.createDisposition()); + assertEquals(createdConfiguration.writeDisposition(), remoteConfiguration.writeDisposition()); assertNotNull(remoteJob.etag()); assertNotNull(remoteJob.statistics()); assertNotNull(remoteJob.status()); assertEquals(createdJob.selfLink(), remoteJob.selfLink()); assertEquals(createdJob.userEmail(), remoteJob.userEmail()); - assertTrue(bigquery.delete(DATASET, sourceTableName)); + assertTrue(createdTable.delete()); assertTrue(bigquery.delete(DATASET, destinationTableName)); } @Test - public void testCreateAndGetJobWithSelectedFields() throws InterruptedException { + public void testCreateAndGetJobWithSelectedFields() { String sourceTableName = "test_create_and_get_job_with_selected_fields_source_table"; String destinationTableName = "test_create_and_get_job_with_selected_fields_destination_table"; TableId sourceTable = TableId.of(DATASET, sourceTableName); - BaseTableInfo tableInfo = TableInfo.of(sourceTable, SIMPLE_SCHEMA); - BaseTableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(sourceTableName, createdTableInfo.tableId().table()); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(sourceTable, tableDefinition); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(sourceTableName, createdTable.tableId().table()); TableId destinationTable = TableId.of(DATASET, destinationTableName); - CopyJobInfo job = CopyJobInfo.of(destinationTable, sourceTable); - CopyJobInfo createdJob = bigquery.create(job, JobOption.fields(JobField.ETAG)); + CopyJobConfiguration configuration = CopyJobConfiguration.of(destinationTable, sourceTable); + Job createdJob = bigquery.create(JobInfo.of(configuration), JobOption.fields(JobField.ETAG)); + CopyJobConfiguration createdConfiguration = createdJob.configuration(); assertNotNull(createdJob.jobId()); - assertNotNull(createdJob.sourceTables()); - assertNotNull(createdJob.destinationTable()); + assertNotNull(createdConfiguration.sourceTables()); + assertNotNull(createdConfiguration.destinationTable()); assertNotNull(createdJob.etag()); assertNull(createdJob.statistics()); assertNull(createdJob.status()); assertNull(createdJob.selfLink()); assertNull(createdJob.userEmail()); - CopyJobInfo remoteJob = bigquery.getJob(createdJob.jobId(), - JobOption.fields(JobField.ETAG)); + Job remoteJob = bigquery.getJob(createdJob.jobId(), JobOption.fields(JobField.ETAG)); + CopyJobConfiguration remoteConfiguration = remoteJob.configuration(); assertEquals(createdJob.jobId(), remoteJob.jobId()); - assertEquals(createdJob.sourceTables(), remoteJob.sourceTables()); - assertEquals(createdJob.destinationTable(), remoteJob.destinationTable()); - assertEquals(createdJob.createDisposition(), remoteJob.createDisposition()); - assertEquals(createdJob.writeDisposition(), remoteJob.writeDisposition()); + assertEquals(createdConfiguration.sourceTables(), remoteConfiguration.sourceTables()); + assertEquals(createdConfiguration.destinationTable(), remoteConfiguration.destinationTable()); + assertEquals(createdConfiguration.createDisposition(), remoteConfiguration.createDisposition()); + assertEquals(createdConfiguration.writeDisposition(), remoteConfiguration.writeDisposition()); assertNotNull(remoteJob.etag()); assertNull(remoteJob.statistics()); assertNull(remoteJob.status()); assertNull(remoteJob.selfLink()); assertNull(remoteJob.userEmail()); - assertTrue(bigquery.delete(DATASET, sourceTableName)); + assertTrue(createdTable.delete()); assertTrue(bigquery.delete(DATASET, destinationTableName)); } @@ -744,26 +789,26 @@ public void testCopyJob() throws InterruptedException { String sourceTableName = "test_copy_job_source_table"; String destinationTableName = "test_copy_job_destination_table"; TableId sourceTable = TableId.of(DATASET, sourceTableName); - BaseTableInfo tableInfo = TableInfo.of(sourceTable, SIMPLE_SCHEMA); - BaseTableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(sourceTableName, createdTableInfo.tableId().table()); + StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); + TableInfo tableInfo = TableInfo.of(sourceTable, tableDefinition); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(sourceTableName, createdTable.tableId().table()); TableId destinationTable = TableId.of(DATASET, destinationTableName); - CopyJobInfo job = CopyJobInfo.of(destinationTable, sourceTable); - CopyJobInfo remoteJob = bigquery.create(job); - while (remoteJob.status().state() != JobStatus.State.DONE) { + CopyJobConfiguration configuration = CopyJobConfiguration.of(destinationTable, sourceTable); + Job remoteJob = bigquery.create(JobInfo.of(configuration)); + while (!remoteJob.isDone()) { Thread.sleep(1000); - remoteJob = bigquery.getJob(remoteJob.jobId()); } assertNull(remoteJob.status().error()); - BaseTableInfo remoteTableInfo = bigquery.getTable(DATASET, destinationTableName); - assertNotNull(remoteTableInfo); - assertEquals(destinationTable.dataset(), remoteTableInfo.tableId().dataset()); - assertEquals(destinationTableName, remoteTableInfo.tableId().table()); - assertEquals(SIMPLE_SCHEMA, remoteTableInfo.schema()); - assertTrue(bigquery.delete(DATASET, sourceTableName)); - assertTrue(bigquery.delete(DATASET, destinationTableName)); + Table remoteTable = bigquery.getTable(DATASET, destinationTableName); + assertNotNull(remoteTable); + assertEquals(destinationTable.dataset(), remoteTable.tableId().dataset()); + assertEquals(destinationTableName, remoteTable.tableId().table()); + assertEquals(TABLE_SCHEMA, remoteTable.definition().schema()); + assertTrue(createdTable.delete()); + assertTrue(remoteTable.delete()); } @Test @@ -774,14 +819,13 @@ public void testQueryJob() throws InterruptedException { .append(TABLE_ID.table()) .toString(); TableId destinationTable = TableId.of(DATASET, tableName); - QueryJobInfo job = QueryJobInfo.builder(query) + QueryJobConfiguration configuration = QueryJobConfiguration.builder(query) .defaultDataset(DatasetId.of(DATASET)) .destinationTable(destinationTable) .build(); - QueryJobInfo remoteJob = bigquery.create(job); - while (remoteJob.status().state() != JobStatus.State.DONE) { + Job remoteJob = bigquery.create(JobInfo.of(configuration)); + while (!remoteJob.isDone()) { Thread.sleep(1000); - remoteJob = bigquery.getJob(remoteJob.jobId()); } assertNull(remoteJob.status().error()); @@ -807,33 +851,32 @@ public void testQueryJob() throws InterruptedException { } assertEquals(2, rowCount); assertTrue(bigquery.delete(DATASET, tableName)); - QueryJobInfo queryJob = bigquery.getJob(remoteJob.jobId()); - assertNotNull(queryJob.statistics().queryPlan()); + Job queryJob = bigquery.getJob(remoteJob.jobId()); + JobStatistics.QueryStatistics statistics = queryJob.statistics(); + assertNotNull(statistics.queryPlan()); } @Test public void testExtractJob() throws InterruptedException { String tableName = "test_export_job_table"; TableId destinationTable = TableId.of(DATASET, tableName); - LoadConfiguration configuration = LoadConfiguration.builder(destinationTable) - .schema(SIMPLE_SCHEMA) - .build(); - LoadJobInfo remoteLoadJob = - bigquery.create(LoadJobInfo.of(configuration, "gs://" + BUCKET + "/" + LOAD_FILE)); - while (remoteLoadJob.status().state() != JobStatus.State.DONE) { + LoadJobConfiguration configuration = + LoadJobConfiguration.builder(destinationTable, "gs://" + BUCKET + "/" + LOAD_FILE) + .schema(SIMPLE_SCHEMA) + .build(); + Job remoteLoadJob = bigquery.create(JobInfo.of(configuration)); + while (!remoteLoadJob.isDone()) { Thread.sleep(1000); - remoteLoadJob = bigquery.getJob(remoteLoadJob.jobId()); } assertNull(remoteLoadJob.status().error()); - ExtractJobInfo extractJob = - ExtractJobInfo.builder(destinationTable, "gs://" + BUCKET + "/" + EXTRACT_FILE) - .printHeader(false) - .build(); - ExtractJobInfo remoteExtractJob = bigquery.create(extractJob); - while (remoteExtractJob.status().state() != JobStatus.State.DONE) { + ExtractJobConfiguration extractConfiguration = + ExtractJobConfiguration.builder(destinationTable, "gs://" + BUCKET + "/" + EXTRACT_FILE) + .printHeader(false) + .build(); + Job remoteExtractJob = bigquery.create(JobInfo.of(extractConfiguration)); + while (!remoteExtractJob.isDone()) { Thread.sleep(1000); - remoteExtractJob = bigquery.getJob(remoteExtractJob.jobId()); } assertNull(remoteExtractJob.status().error()); assertEquals(CSV_CONTENT, @@ -846,34 +889,33 @@ public void testCancelJob() throws InterruptedException { String destinationTableName = "test_cancel_query_job_table"; String query = "SELECT TimestampField, StringField, BooleanField FROM " + TABLE_ID.table(); TableId destinationTable = TableId.of(DATASET, destinationTableName); - QueryJobInfo job = QueryJobInfo.builder(query) + QueryJobConfiguration configuration = QueryJobConfiguration.builder(query) .defaultDataset(DatasetId.of(DATASET)) .destinationTable(destinationTable) .build(); - JobInfo remoteJob = bigquery.create(job); - assertTrue(bigquery.cancel(remoteJob.jobId())); - while (remoteJob.status().state() != JobStatus.State.DONE) { + Job remoteJob = bigquery.create(JobInfo.of(configuration)); + assertTrue(remoteJob.cancel()); + while (!remoteJob.isDone()) { Thread.sleep(1000); - remoteJob = bigquery.getJob(remoteJob.jobId()); } assertNull(remoteJob.status().error()); } @Test - public void testCancelNonExistingJob() throws InterruptedException { + public void testCancelNonExistingJob() { assertFalse(bigquery.cancel("test_cancel_non_existing_job")); } @Test - public void testInsertFromFile() throws InterruptedException, FileNotFoundException { + public void testInsertFromFile() throws InterruptedException { String destinationTableName = "test_insert_from_file_table"; TableId tableId = TableId.of(DATASET, destinationTableName); - LoadConfiguration configuration = LoadConfiguration.builder(tableId) + WriteChannelConfiguration configuration = WriteChannelConfiguration.builder(tableId) .formatOptions(FormatOptions.json()) .createDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED) .schema(TABLE_SCHEMA) .build(); - try (TableDataWriteChannel channel = bigquery.writer(configuration)) { + try (WriteChannel channel = bigquery.writer(configuration)) { channel.write(ByteBuffer.wrap(JSON_CONTENT.getBytes(StandardCharsets.UTF_8))); } catch (IOException e) { fail("IOException was not expected"); diff --git a/gcloud-java-contrib/README.md b/gcloud-java-contrib/README.md new file mode 100644 index 000000000000..7a935192891d --- /dev/null +++ b/gcloud-java-contrib/README.md @@ -0,0 +1,64 @@ +Google Cloud Java Contributions +=============================== + +Packages that provide higher-level abstraction/functionality for common gcloud-java use cases. + +[![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) +[![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) +[![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-bigquery.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-bigquery.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) + +Quickstart +---------- +If you are using Maven, add this to your pom.xml file +```xml + + com.google.gcloud + gcloud-java-contrib + 0.1.4 + +``` +If you are using Gradle, add this to your dependencies +```Groovy +compile 'com.google.gcloud:gcloud-java-contrib:0.1.4' +``` +If you are using SBT, add this to your dependencies +```Scala +libraryDependencies += "com.google.gcloud" % "gcloud-java-contrib" % "0.1.4" +``` + +Java Versions +------------- + +Java 7 or above is required for using this client. + +Versioning +---------- + +This library follows [Semantic Versioning] (http://semver.org/). + +It is currently in major version zero (``0.y.z``), which means that anything +may change at any time and the public API should not be considered +stable. + +Contributing +------------ + +Contributions to this library are always welcome and highly encouraged. + +See `gcloud-java`'s [CONTRIBUTING] documentation and the `gcloud-*` [shared documentation](https://github.com/GoogleCloudPlatform/gcloud-common/blob/master/contributing/readme.md#how-to-contribute-to-gcloud) for more information on how to get started. + +Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See [Code of Conduct][code-of-conduct] for more information. + +License +------- + +Apache 2.0 - See [LICENSE] for more information. + + +[CONTRIBUTING]:https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/CONTRIBUTING.md +[code-of-conduct]:https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/CODE_OF_CONDUCT.md#contributor-code-of-conduct +[LICENSE]: https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/LICENSE +[cloud-platform]: https://cloud.google.com/ +[developers-console]:https://console.developers.google.com/ diff --git a/gcloud-java-contrib/pom.xml b/gcloud-java-contrib/pom.xml new file mode 100644 index 000000000000..dd976991e2af --- /dev/null +++ b/gcloud-java-contrib/pom.xml @@ -0,0 +1,36 @@ + + + 4.0.0 + gcloud-java-contrib + jar + GCloud Java contributions + + Contains packages that provide higher-level abstraction/functionality for common gcloud-java use cases. + + + com.google.gcloud + gcloud-java-pom + 0.1.5-SNAPSHOT + + + gcloud-java-contrib + + + + ${project.groupId} + gcloud-java + ${project.version} + + + + + + org.codehaus.mojo + exec-maven-plugin + + false + + + + + diff --git a/gcloud-java-core/README.md b/gcloud-java-core/README.md index f47d99779a05..bc9463b9cc2b 100644 --- a/gcloud-java-core/README.md +++ b/gcloud-java-core/README.md @@ -6,6 +6,8 @@ This module provides common functionality required by service-specific modules o [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-core.svg)](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-core.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/package-summary.html) @@ -17,16 +19,16 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java-core - 0.1.2 + 0.1.4 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java-core:0.1.2' +compile 'com.google.gcloud:gcloud-java-core:0.1.4' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java-core" % "0.1.2" +libraryDependencies += "com.google.gcloud" % "gcloud-java-core" % "0.1.4" ``` Troubleshooting diff --git a/gcloud-java-core/pom.xml b/gcloud-java-core/pom.xml index d87e0d7eae85..d07a567b7e5a 100644 --- a/gcloud-java-core/pom.xml +++ b/gcloud-java-core/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java-core jar GCloud Java core @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.3-SNAPSHOT + 0.1.5-SNAPSHOT gcloud-java-core @@ -36,24 +35,24 @@ com.google.http-client google-http-client - 1.20.0 + 1.21.0 compile com.google.oauth-client google-oauth-client - 1.20.0 + 1.21.0 compile com.google.guava guava - 18.0 + 19.0 com.google.api-client google-api-client-appengine - 1.20.0 + 1.21.0 compile @@ -65,7 +64,7 @@ com.google.http-client google-http-client-jackson - 1.20.0 + 1.21.0 compile @@ -83,19 +82,19 @@ joda-time joda-time - 2.8.2 + 2.9.2 compile org.json json - 20090211 + 20151123 compile org.easymock easymock - 3.3 + 3.4 test diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java b/gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java index e05383a65826..1d18a5a27e81 100644 --- a/gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java +++ b/gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java @@ -21,6 +21,7 @@ import java.io.IOException; import java.io.Serializable; import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; import java.util.Arrays; import java.util.Objects; @@ -114,9 +115,9 @@ private void flush() { } } - private void validateOpen() throws IOException { + private void validateOpen() throws ClosedChannelException { if (!isOpen) { - throw new IOException("stream is closed"); + throw new ClosedChannelException(); } } diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/IamPolicy.java b/gcloud-java-core/src/main/java/com/google/gcloud/IamPolicy.java new file mode 100644 index 000000000000..748eaba2ab4c --- /dev/null +++ b/gcloud-java-core/src/main/java/com/google/gcloud/IamPolicy.java @@ -0,0 +1,256 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud; + +import static com.google.common.base.Preconditions.checkArgument; + +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; + +import java.io.Serializable; +import java.util.Arrays; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; + +/** + * Base class for Identity and Access Management (IAM) policies. IAM policies are used to specify + * access settings for Cloud Platform resources. A policy is a map of bindings. A binding assigns + * a set of identities to a role, where the identities can be user accounts, Google groups, Google + * domains, and service accounts. A role is a named list of permissions defined by IAM. + * + * @param the data type of roles (should be serializable) + * @see Policy + */ +public abstract class IamPolicy implements Serializable { + + private static final long serialVersionUID = 1114489978726897720L; + + private final Map> bindings; + private final String etag; + private final Integer version; + + /** + * Builder for an IAM Policy. + * + * @param the data type of roles + * @param the subclass extending this abstract builder + */ + public abstract static class Builder> { + + private final Map> bindings = new HashMap<>(); + private String etag; + private Integer version; + + /** + * Constructor for IAM Policy builder. + */ + protected Builder() {} + + /** + * Replaces the builder's map of bindings with the given map of bindings. + * + * @throws IllegalArgumentException if the provided map is null or contain any null values + */ + public final B bindings(Map> bindings) { + checkArgument(bindings != null, "The provided map of bindings cannot be null."); + for (Map.Entry> binding : bindings.entrySet()) { + verifyBinding(binding.getKey(), binding.getValue()); + } + this.bindings.clear(); + for (Map.Entry> binding : bindings.entrySet()) { + this.bindings.put(binding.getKey(), new HashSet(binding.getValue())); + } + return self(); + } + + /** + * Adds a binding to the policy. + * + * @throws IllegalArgumentException if the policy already contains a binding with the same role + * or if the role or any identities are null + */ + public final B addBinding(R role, Set identities) { + verifyBinding(role, identities); + checkArgument(!bindings.containsKey(role), + "The policy already contains a binding with the role " + role.toString() + "."); + bindings.put(role, new HashSet(identities)); + return self(); + } + + /** + * Adds a binding to the policy. + * + * @throws IllegalArgumentException if the policy already contains a binding with the same role + * or if the role or any identities are null + */ + public final B addBinding(R role, Identity first, Identity... others) { + HashSet identities = new HashSet<>(); + identities.add(first); + identities.addAll(Arrays.asList(others)); + return addBinding(role, identities); + } + + private void verifyBinding(R role, Collection identities) { + checkArgument(role != null, "The role cannot be null."); + verifyIdentities(identities); + } + + private void verifyIdentities(Collection identities) { + checkArgument(identities != null, "A role cannot be assigned to a null set of identities."); + checkArgument(!identities.contains(null), "Null identities are not permitted."); + } + + /** + * Removes the binding associated with the specified role. + */ + public final B removeBinding(R role) { + bindings.remove(role); + return self(); + } + + /** + * Adds one or more identities to an existing binding. + * + * @throws IllegalArgumentException if the policy doesn't contain a binding with the specified + * role or any identities are null + */ + public final B addIdentity(R role, Identity first, Identity... others) { + checkArgument(bindings.containsKey(role), + "The policy doesn't contain the role " + role.toString() + "."); + List toAdd = new LinkedList<>(); + toAdd.add(first); + toAdd.addAll(Arrays.asList(others)); + verifyIdentities(toAdd); + bindings.get(role).addAll(toAdd); + return self(); + } + + /** + * Removes one or more identities from an existing binding. + * + * @throws IllegalArgumentException if the policy doesn't contain a binding with the specified + * role + */ + public final B removeIdentity(R role, Identity first, Identity... others) { + checkArgument(bindings.containsKey(role), + "The policy doesn't contain the role " + role.toString() + "."); + bindings.get(role).remove(first); + bindings.get(role).removeAll(Arrays.asList(others)); + return self(); + } + + /** + * Sets the policy's etag. + * + *

Etags are used for optimistic concurrency control as a way to help prevent simultaneous + * updates of a policy from overwriting each other. It is strongly suggested that systems make + * use of the etag in the read-modify-write cycle to perform policy updates in order to avoid + * race conditions. An etag is returned in the response to getIamPolicy, and systems are + * expected to put that etag in the request to setIamPolicy to ensure that their change will be + * applied to the same version of the policy. If no etag is provided in the call to + * setIamPolicy, then the existing policy is overwritten blindly. + */ + protected final B etag(String etag) { + this.etag = etag; + return self(); + } + + /** + * Sets the version of the policy. The default version is 0, meaning only the "owner", "editor", + * and "viewer" roles are permitted. If the version is 1, you may also use other roles. + */ + protected final B version(Integer version) { + this.version = version; + return self(); + } + + @SuppressWarnings("unchecked") + private B self() { + return (B) this; + } + + public abstract IamPolicy build(); + } + + protected IamPolicy(Builder> builder) { + ImmutableMap.Builder> bindingsBuilder = ImmutableMap.builder(); + for (Map.Entry> binding : builder.bindings.entrySet()) { + bindingsBuilder.put(binding.getKey(), ImmutableSet.copyOf(binding.getValue())); + } + this.bindings = bindingsBuilder.build(); + this.etag = builder.etag; + this.version = builder.version; + } + + /** + * Returns a builder containing the properties of this IAM Policy. + */ + public abstract Builder> toBuilder(); + + /** + * The map of bindings that comprises the policy. + */ + public Map> bindings() { + return bindings; + } + + /** + * The policy's etag. + * + *

Etags are used for optimistic concurrency control as a way to help prevent simultaneous + * updates of a policy from overwriting each other. It is strongly suggested that systems make + * use of the etag in the read-modify-write cycle to perform policy updates in order to avoid + * race conditions. An etag is returned in the response to getIamPolicy, and systems are + * expected to put that etag in the request to setIamPolicy to ensure that their change will be + * applied to the same version of the policy. If no etag is provided in the call to + * setIamPolicy, then the existing policy is overwritten blindly. + */ + public String etag() { + return etag; + } + + /** + * Sets the version of the policy. The default version is 0, meaning only the "owner", "editor", + * and "viewer" roles are permitted. If the version is 1, you may also use other roles. + */ + public Integer version() { + return version; + } + + @Override + public final int hashCode() { + return Objects.hash(getClass(), bindings, etag, version); + } + + @Override + public final boolean equals(Object obj) { + if (obj == null || !getClass().equals(obj.getClass())) { + return false; + } + @SuppressWarnings("rawtypes") + IamPolicy other = (IamPolicy) obj; + return Objects.equals(bindings, other.bindings()) + && Objects.equals(etag, other.etag()) + && Objects.equals(version, other.version()); + } +} diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/Identity.java b/gcloud-java-core/src/main/java/com/google/gcloud/Identity.java new file mode 100644 index 000000000000..d1644198f759 --- /dev/null +++ b/gcloud-java-core/src/main/java/com/google/gcloud/Identity.java @@ -0,0 +1,225 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud; + +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.common.base.CaseFormat; + +import java.io.Serializable; +import java.util.Objects; + +/** + * An identity in an {@link IamPolicy}. The following types of identities are permitted in IAM + * policies: + *

    + *
  • Google account + *
  • Service account + *
  • Google group + *
  • Google Apps domain + *
+ * + *

There are also two special identities that represent all users and all Google-authenticated + * accounts. + * + * @see Concepts + * related to identity + */ +public final class Identity implements Serializable { + + private static final long serialVersionUID = -8181841964597657446L; + + private final Type type; + private final String id; + + /** + * The types of IAM identities. + */ + public enum Type { + + /** + * Represents anyone who is on the internet; with or without a Google account. + */ + ALL_USERS, + + /** + * Represents anyone who is authenticated with a Google account or a service account. + */ + ALL_AUTHENTICATED_USERS, + + /** + * Represents a specific Google account. + */ + USER, + + /** + * Represents a service account. + */ + SERVICE_ACCOUNT, + + /** + * Represents a Google group. + */ + GROUP, + + /** + * Represents all the users of a Google Apps domain name. + */ + DOMAIN + } + + private Identity(Type type, String id) { + this.type = type; + this.id = id; + } + + public Type type() { + return type; + } + + /** + * Returns the string identifier for this identity. The id corresponds to: + *

    + *
  • email address (for identities of type {@code USER}, {@code SERVICE_ACCOUNT}, and + * {@code GROUP}) + *
  • domain (for identities of type {@code DOMAIN}) + *
  • {@code null} (for identities of type {@code ALL_USERS} and + * {@code ALL_AUTHENTICATED_USERS}) + *
+ */ + public String id() { + return id; + } + + /** + * Returns a new identity representing anyone who is on the internet; with or without a Google + * account. + */ + public static Identity allUsers() { + return new Identity(Type.ALL_USERS, null); + } + + /** + * Returns a new identity representing anyone who is authenticated with a Google account or a + * service account. + */ + public static Identity allAuthenticatedUsers() { + return new Identity(Type.ALL_AUTHENTICATED_USERS, null); + } + + /** + * Returns a new user identity. + * + * @param email An email address that represents a specific Google account. For example, + * alice@gmail.com or joe@example.com. + */ + public static Identity user(String email) { + return new Identity(Type.USER, checkNotNull(email)); + } + + /** + * Returns a new service account identity. + * + * @param email An email address that represents a service account. For example, + * my-other-app@appspot.gserviceaccount.com. + */ + public static Identity serviceAccount(String email) { + return new Identity(Type.SERVICE_ACCOUNT, checkNotNull(email)); + } + + /** + * Returns a new group identity. + * + * @param email An email address that represents a Google group. For example, + * admins@example.com. + */ + public static Identity group(String email) { + return new Identity(Type.GROUP, checkNotNull(email)); + } + + /** + * Returns a new domain identity. + * + * @param domain A Google Apps domain name that represents all the users of that domain. For + * example, google.com or example.com. + */ + public static Identity domain(String domain) { + return new Identity(Type.DOMAIN, checkNotNull(domain)); + } + + @Override + public int hashCode() { + return Objects.hash(id, type); + } + + @Override + public boolean equals(Object obj) { + if (!(obj instanceof Identity)) { + return false; + } + Identity other = (Identity) obj; + return Objects.equals(id, other.id()) && Objects.equals(type, other.type()); + } + + /** + * Returns the string value associated with the identity. Used primarily for converting from + * {@code Identity} objects to strings for protobuf-generated policies. + */ + public String strValue() { + switch (type) { + case ALL_USERS: + return "allUsers"; + case ALL_AUTHENTICATED_USERS: + return "allAuthenticatedUsers"; + case USER: + return "user:" + id; + case SERVICE_ACCOUNT: + return "serviceAccount:" + id; + case GROUP: + return "group:" + id; + case DOMAIN: + return "domain:" + id; + default: + throw new IllegalStateException("Unexpected identity type: " + type); + } + } + + /** + * Converts a string to an {@code Identity}. Used primarily for converting protobuf-generated + * policy identities to {@code Identity} objects. + */ + public static Identity valueOf(String identityStr) { + String[] info = identityStr.split(":"); + Type type = Type.valueOf(CaseFormat.LOWER_CAMEL.to(CaseFormat.UPPER_UNDERSCORE, info[0])); + switch (type) { + case ALL_USERS: + return Identity.allUsers(); + case ALL_AUTHENTICATED_USERS: + return Identity.allAuthenticatedUsers(); + case USER: + return Identity.user(info[1]); + case SERVICE_ACCOUNT: + return Identity.serviceAccount(info[1]); + case GROUP: + return Identity.group(info[1]); + case DOMAIN: + return Identity.domain(info[1]); + default: + throw new IllegalStateException("Unexpected identity type " + type); + } + } +} diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/Restorable.java b/gcloud-java-core/src/main/java/com/google/gcloud/Restorable.java index 90633c70046f..0b573522e370 100644 --- a/gcloud-java-core/src/main/java/com/google/gcloud/Restorable.java +++ b/gcloud-java-core/src/main/java/com/google/gcloud/Restorable.java @@ -21,14 +21,14 @@ * *

* A typical capture usage: - *

  {@code
+ * 
 {@code
  * X restorableObj; // X instanceof Restorable
  * RestorableState state = restorableObj.capture();
  * .. persist state
  * }
* * A typical restore usage: - *
  {@code
+ * 
 {@code
  * RestorableState state = ... // read from persistence
  * X restorableObj = state.restore();
  * ...
diff --git a/gcloud-java-core/src/test/java/com/google/gcloud/BaseWriteChannelTest.java b/gcloud-java-core/src/test/java/com/google/gcloud/BaseWriteChannelTest.java
index e49a17b019e0..6d5306a3bc7f 100644
--- a/gcloud-java-core/src/test/java/com/google/gcloud/BaseWriteChannelTest.java
+++ b/gcloud-java-core/src/test/java/com/google/gcloud/BaseWriteChannelTest.java
@@ -32,6 +32,7 @@
 import java.io.IOException;
 import java.io.Serializable;
 import java.nio.ByteBuffer;
+import java.nio.channels.ClosedChannelException;
 import java.util.Arrays;
 import java.util.Random;
 
@@ -102,8 +103,7 @@ public void testClose() throws IOException {
   @Test
   public void testValidateOpen() throws IOException {
     channel.close();
-    thrown.expect(IOException.class);
-    thrown.expectMessage("stream is closed");
+    thrown.expect(ClosedChannelException.class);
     channel.write(ByteBuffer.allocate(42));
   }
 
diff --git a/gcloud-java-core/src/test/java/com/google/gcloud/IamPolicyTest.java b/gcloud-java-core/src/test/java/com/google/gcloud/IamPolicyTest.java
new file mode 100644
index 000000000000..db0935c4766d
--- /dev/null
+++ b/gcloud-java-core/src/test/java/com/google/gcloud/IamPolicyTest.java
@@ -0,0 +1,180 @@
+/*
+ * Copyright 2016 Google Inc. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.google.gcloud;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+
+import org.junit.Test;
+
+import java.util.Map;
+import java.util.Set;
+
+public class IamPolicyTest {
+
+  private static final Identity ALL_USERS = Identity.allUsers();
+  private static final Identity ALL_AUTH_USERS = Identity.allAuthenticatedUsers();
+  private static final Identity USER = Identity.user("abc@gmail.com");
+  private static final Identity SERVICE_ACCOUNT =
+      Identity.serviceAccount("service-account@gmail.com");
+  private static final Identity GROUP = Identity.group("group@gmail.com");
+  private static final Identity DOMAIN = Identity.domain("google.com");
+  private static final Map> BINDINGS = ImmutableMap.of(
+      "viewer",
+      ImmutableSet.of(USER, SERVICE_ACCOUNT, ALL_USERS),
+      "editor",
+      ImmutableSet.of(ALL_AUTH_USERS, GROUP, DOMAIN));
+  private static final PolicyImpl SIMPLE_POLICY = PolicyImpl.builder()
+      .addBinding("viewer", ImmutableSet.of(USER, SERVICE_ACCOUNT, ALL_USERS))
+      .addBinding("editor", ImmutableSet.of(ALL_AUTH_USERS, GROUP, DOMAIN))
+      .build();
+  private static final PolicyImpl FULL_POLICY =
+      new PolicyImpl.Builder(SIMPLE_POLICY.bindings(), "etag", 1).build();
+
+  static class PolicyImpl extends IamPolicy {
+
+    static class Builder extends IamPolicy.Builder {
+
+      private Builder() {}
+
+      private Builder(Map> bindings, String etag, Integer version) {
+        bindings(bindings).etag(etag).version(version);
+      }
+
+      @Override
+      public PolicyImpl build() {
+        return new PolicyImpl(this);
+      }
+    }
+
+    PolicyImpl(Builder builder) {
+      super(builder);
+    }
+
+    @Override
+    public Builder toBuilder() {
+      return new Builder(bindings(), etag(), version());
+    }
+
+    static Builder builder() {
+      return new Builder();
+    }
+  }
+
+  @Test
+  public void testBuilder() {
+    assertEquals(BINDINGS, FULL_POLICY.bindings());
+    assertEquals("etag", FULL_POLICY.etag());
+    assertEquals(1, FULL_POLICY.version().intValue());
+    Map> editorBinding =
+        ImmutableMap.>builder().put("editor", BINDINGS.get("editor")).build();
+    PolicyImpl policy = FULL_POLICY.toBuilder().bindings(editorBinding).build();
+    assertEquals(editorBinding, policy.bindings());
+    assertEquals("etag", policy.etag());
+    assertEquals(1, policy.version().intValue());
+    policy = SIMPLE_POLICY.toBuilder().removeBinding("editor").build();
+    assertEquals(ImmutableMap.of("viewer", BINDINGS.get("viewer")), policy.bindings());
+    assertNull(policy.etag());
+    assertNull(policy.version());
+    policy = policy.toBuilder()
+        .removeIdentity("viewer", USER, ALL_USERS)
+        .addIdentity("viewer", DOMAIN, GROUP)
+        .build();
+    assertEquals(ImmutableMap.of("viewer", ImmutableSet.of(SERVICE_ACCOUNT, DOMAIN, GROUP)),
+        policy.bindings());
+    assertNull(policy.etag());
+    assertNull(policy.version());
+    policy = PolicyImpl.builder().addBinding("owner", USER, SERVICE_ACCOUNT).build();
+    assertEquals(
+        ImmutableMap.of("owner", ImmutableSet.of(USER, SERVICE_ACCOUNT)), policy.bindings());
+    assertNull(policy.etag());
+    assertNull(policy.version());
+    try {
+      SIMPLE_POLICY.toBuilder().addBinding("viewer", USER);
+      fail("Should have failed due to duplicate role.");
+    } catch (IllegalArgumentException e) {
+      assertEquals("The policy already contains a binding with the role viewer.", e.getMessage());
+    }
+    try {
+      SIMPLE_POLICY.toBuilder().addBinding("editor", ImmutableSet.of(USER));
+      fail("Should have failed due to duplicate role.");
+    } catch (IllegalArgumentException e) {
+      assertEquals("The policy already contains a binding with the role editor.", e.getMessage());
+    }
+  }
+
+  @Test
+  public void testEqualsHashCode() {
+    assertNotNull(FULL_POLICY);
+    PolicyImpl emptyPolicy = PolicyImpl.builder().build();
+    AnotherPolicyImpl anotherPolicy = new AnotherPolicyImpl.Builder().build();
+    assertNotEquals(emptyPolicy, anotherPolicy);
+    assertNotEquals(emptyPolicy.hashCode(), anotherPolicy.hashCode());
+    assertNotEquals(FULL_POLICY, SIMPLE_POLICY);
+    assertNotEquals(FULL_POLICY.hashCode(), SIMPLE_POLICY.hashCode());
+    PolicyImpl copy = SIMPLE_POLICY.toBuilder().build();
+    assertEquals(SIMPLE_POLICY, copy);
+    assertEquals(SIMPLE_POLICY.hashCode(), copy.hashCode());
+  }
+
+  @Test
+  public void testBindings() {
+    assertTrue(PolicyImpl.builder().build().bindings().isEmpty());
+    assertEquals(BINDINGS, SIMPLE_POLICY.bindings());
+  }
+
+  @Test
+  public void testEtag() {
+    assertNull(SIMPLE_POLICY.etag());
+    assertEquals("etag", FULL_POLICY.etag());
+  }
+
+  @Test
+  public void testVersion() {
+    assertNull(SIMPLE_POLICY.version());
+    assertEquals(1, FULL_POLICY.version().intValue());
+  }
+
+  static class AnotherPolicyImpl extends IamPolicy {
+
+    static class Builder extends IamPolicy.Builder {
+
+      private Builder() {}
+
+      @Override
+      public AnotherPolicyImpl build() {
+        return new AnotherPolicyImpl(this);
+      }
+    }
+
+    AnotherPolicyImpl(Builder builder) {
+      super(builder);
+    }
+
+    @Override
+    public Builder toBuilder() {
+      return new Builder();
+    }
+  }
+}
diff --git a/gcloud-java-core/src/test/java/com/google/gcloud/IdentityTest.java b/gcloud-java-core/src/test/java/com/google/gcloud/IdentityTest.java
new file mode 100644
index 000000000000..828f1c839431
--- /dev/null
+++ b/gcloud-java-core/src/test/java/com/google/gcloud/IdentityTest.java
@@ -0,0 +1,105 @@
+/*
+ * Copyright 2016 Google Inc. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.google.gcloud;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+import org.junit.Test;
+
+public class IdentityTest {
+
+  private static final Identity ALL_USERS = Identity.allUsers();
+  private static final Identity ALL_AUTH_USERS = Identity.allAuthenticatedUsers();
+  private static final Identity USER = Identity.user("abc@gmail.com");
+  private static final Identity SERVICE_ACCOUNT =
+      Identity.serviceAccount("service-account@gmail.com");
+  private static final Identity GROUP = Identity.group("group@gmail.com");
+  private static final Identity DOMAIN = Identity.domain("google.com");
+
+  @Test
+  public void testAllUsers() {
+    assertEquals(Identity.Type.ALL_USERS, ALL_USERS.type());
+    assertNull(ALL_USERS.id());
+  }
+
+  @Test
+  public void testAllAuthenticatedUsers() {
+    assertEquals(Identity.Type.ALL_AUTHENTICATED_USERS, ALL_AUTH_USERS.type());
+    assertNull(ALL_AUTH_USERS.id());
+  }
+
+  @Test
+  public void testUser() {
+    assertEquals(Identity.Type.USER, USER.type());
+    assertEquals("abc@gmail.com", USER.id());
+  }
+
+  @Test(expected = NullPointerException.class)
+  public void testUserNullEmail() {
+    Identity.user(null);
+  }
+
+  @Test
+  public void testServiceAccount() {
+    assertEquals(Identity.Type.SERVICE_ACCOUNT, SERVICE_ACCOUNT.type());
+    assertEquals("service-account@gmail.com", SERVICE_ACCOUNT.id());
+  }
+
+  @Test(expected = NullPointerException.class)
+  public void testServiceAccountNullEmail() {
+    Identity.serviceAccount(null);
+  }
+
+  @Test
+  public void testGroup() {
+    assertEquals(Identity.Type.GROUP, GROUP.type());
+    assertEquals("group@gmail.com", GROUP.id());
+  }
+
+  @Test(expected = NullPointerException.class)
+  public void testGroupNullEmail() {
+    Identity.group(null);
+  }
+
+  @Test
+  public void testDomain() {
+    assertEquals(Identity.Type.DOMAIN, DOMAIN.type());
+    assertEquals("google.com", DOMAIN.id());
+  }
+
+  @Test(expected = NullPointerException.class)
+  public void testDomainNullId() {
+    Identity.domain(null);
+  }
+
+  @Test
+  public void testIdentityToAndFromPb() {
+    compareIdentities(ALL_USERS, Identity.valueOf(ALL_USERS.strValue()));
+    compareIdentities(ALL_AUTH_USERS, Identity.valueOf(ALL_AUTH_USERS.strValue()));
+    compareIdentities(USER, Identity.valueOf(USER.strValue()));
+    compareIdentities(SERVICE_ACCOUNT, Identity.valueOf(SERVICE_ACCOUNT.strValue()));
+    compareIdentities(GROUP, Identity.valueOf(GROUP.strValue()));
+    compareIdentities(DOMAIN, Identity.valueOf(DOMAIN.strValue()));
+  }
+
+  private void compareIdentities(Identity expected, Identity actual) {
+    assertEquals(expected, actual);
+    assertEquals(expected.type(), actual.type());
+    assertEquals(expected.id(), actual.id());
+  }
+}
diff --git a/gcloud-java-datastore/README.md b/gcloud-java-datastore/README.md
index 15c2b3961dc2..dd341ba244c3 100644
--- a/gcloud-java-datastore/README.md
+++ b/gcloud-java-datastore/README.md
@@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Datastore] (https://cloud.google.com/dat
 [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java)
 [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master)
 [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-datastore.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-datastore.svg)
+[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java)
+[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969)
 
 -  [Homepage] (https://googlecloudplatform.github.io/gcloud-java/)
 -  [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/datastore/package-summary.html)
@@ -20,21 +22,21 @@ If you are using Maven, add this to your pom.xml file
 
   com.google.gcloud
   gcloud-java-datastore
-  0.1.2
+  0.1.4
 
 ```
 If you are using Gradle, add this to your dependencies
 ```Groovy
-compile 'com.google.gcloud:gcloud-java-datastore:0.1.2'
+compile 'com.google.gcloud:gcloud-java-datastore:0.1.4'
 ```
 If you are using SBT, add this to your dependencies
 ```Scala
-libraryDependencies += "com.google.gcloud" % "gcloud-java-datastore" % "0.1.2"
+libraryDependencies += "com.google.gcloud" % "gcloud-java-datastore" % "0.1.4"
 ```
 
 Example Application
 --------------------
-[`DatastoreExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java) is a simple command line interface for the Cloud Datastore.  Read more about using the application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/DatastoreExample.html).
+[`DatastoreExample`](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java) is a simple command line interface for the Cloud Datastore.  Read more about using the application on the [`DatastoreExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/datastore/DatastoreExample.html).
 
 Authentication
 --------------
@@ -134,67 +136,12 @@ Cloud Datastore relies on indexing to run queries. Indexing is turned on by defa
 
 #### Complete source code
 
-Here we put together all the code shown above into one program.  This program assumes that you are running on Compute Engine or from your own desktop. To run this example on App Engine, move this code to your application's servlet class and print the query output to the webpage instead of `System.out`.
-
-```java
-import com.google.gcloud.datastore.Datastore;
-import com.google.gcloud.datastore.DatastoreOptions;
-import com.google.gcloud.datastore.Entity;
-import com.google.gcloud.datastore.Key;
-import com.google.gcloud.datastore.KeyFactory;
-import com.google.gcloud.datastore.Query;
-import com.google.gcloud.datastore.QueryResults;
-import com.google.gcloud.datastore.StructuredQuery;
-import com.google.gcloud.datastore.StructuredQuery.PropertyFilter;
-
-public class GcloudDatastoreExample {
-
-  public static void main(String[] args) {
-    // Create datastore service object.
-    // By default, credentials are inferred from the runtime environment.
-    Datastore datastore = DatastoreOptions.defaultInstance().service();
-
-    // Add an entity to Datastore
-    KeyFactory keyFactory = datastore.newKeyFactory().kind("Person");
-    Key key = keyFactory.newKey("john.doe@gmail.com");
-    Entity entity = Entity.builder(key)
-        .set("name", "John Doe")
-        .set("age", 51)
-        .set("favorite_food", "pizza")
-        .build();
-    datastore.put(entity);
-
-    // Get an entity from Datastore
-    Entity johnEntity = datastore.get(key);
-
-    // Add a couple more entities to make the query results more interesting
-    Key janeKey = keyFactory.newKey("jane.doe@gmail.com");
-    Entity janeEntity = Entity.builder(janeKey)
-        .set("name", "Jane Doe")
-        .set("age", 44)
-        .set("favorite_food", "pizza")
-        .build();
-    Key joeKey = keyFactory.newKey("joe.shmoe@gmail.com");
-    Entity joeEntity = Entity.builder(joeKey)
-        .set("name", "Joe Shmoe")
-        .set("age", 27)
-        .set("favorite_food", "sushi")
-        .build();
-    datastore.put(janeEntity, joeEntity);
-
-    // Run a query
-    Query query = Query.entityQueryBuilder()
-        .kind("Person")
-        .filter(PropertyFilter.eq("favorite_food", "pizza"))
-        .build();
-    QueryResults results = datastore.run(query);
-    while (results.hasNext()) {
-      Entity currentEntity = results.next();
-      System.out.println(currentEntity.getString("name") + ", you're invited to a pizza party!");
-    }
-  }
-}
-```
+In
+[AddEntitiesAndRunQuery.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/AddEntitiesAndRunQuery.java)
+we put together all the code shown above into one program. The program assumes that you are
+running on Compute Engine or from your own desktop. To run the example on App Engine, simply move
+the code from the main method to your application's servlet class and change the print statements to
+display on your webpage.
 
 Troubleshooting
 ---------------
diff --git a/gcloud-java-datastore/pom.xml b/gcloud-java-datastore/pom.xml
index 071dae4ad3f7..452986ba5ea3 100644
--- a/gcloud-java-datastore/pom.xml
+++ b/gcloud-java-datastore/pom.xml
@@ -1,7 +1,6 @@
 
 
   4.0.0
-  com.google.gcloud
   gcloud-java-datastore
   jar
   GCloud Java datastore
@@ -11,7 +10,7 @@
   
     com.google.gcloud
     gcloud-java-pom
-    0.1.3-SNAPSHOT
+    0.1.5-SNAPSHOT
   
   
     gcloud-java-datastore
@@ -25,7 +24,7 @@
     
       com.google.apis
       google-api-services-datastore-protobuf
-      v1beta2-rev1-2.1.2
+      v1beta2-rev1-4.0.0
       compile
       
         
@@ -43,7 +42,7 @@
     
       org.easymock
       easymock
-      3.3
+      3.4
       test
     
   
diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseEntity.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseEntity.java
index d674a5e242ad..20c0b13e5001 100644
--- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseEntity.java
+++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseEntity.java
@@ -33,6 +33,7 @@
 import com.google.protobuf.InvalidProtocolBufferException;
 
 import java.util.HashMap;
+import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
@@ -128,61 +129,286 @@ public B remove(String name) {
       return self();
     }
 
+    /**
+     * Sets a property.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, Value value) {
       properties.put(name, value);
       return self();
     }
 
+    /**
+     * Sets a property of type {@link StringValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, String value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link StringValue}.
+     *
+     * @param name name of the property
+     * @param first the first string in the list
+     * @param second the second string in the list
+     * @param others other strings in the list
+     */
+    public B set(String name, String first, String second, String... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (String other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link LongValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, long value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link LongValue}.
+     *
+     * @param name name of the property
+     * @param first the first long in the list
+     * @param second the second long in the list
+     * @param others other longs in the list
+     */
+    public B set(String name, long first, long second, long... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (long other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link DoubleValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, double value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link DoubleValue}.
+     *
+     * @param name name of the property
+     * @param first the first double in the list
+     * @param second the second double in the list
+     * @param others other doubles in the list
+     */
+    public B set(String name, double first, double second, double... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (double other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link BooleanValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, boolean value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link BooleanValue}.
+     *
+     * @param name name of the property
+     * @param first the first boolean in the list
+     * @param second the second boolean in the list
+     * @param others other booleans in the list
+     */
+    public B set(String name, boolean first, boolean second, boolean... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (boolean other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link DateTimeValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, DateTime value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link DateTimeValue}.
+     *
+     * @param name name of the property
+     * @param first the first {@link DateTime} in the list
+     * @param second the second {@link DateTime} in the list
+     * @param others other {@link DateTime}s in the list
+     */
+    public B set(String name, DateTime first, DateTime second, DateTime... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (DateTime other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link KeyValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, Key value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link KeyValue}.
+     *
+     * @param name name of the property
+     * @param first the first {@link Key} in the list
+     * @param second the second {@link Key} in the list
+     * @param others other {@link Key}s in the list
+     */
+    public B set(String name, Key first, Key second, Key... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (Key other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link EntityValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, FullEntity value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link EntityValue}.
+     *
+     * @param name name of the property
+     * @param first the first {@link FullEntity} in the list
+     * @param second the second {@link FullEntity} in the list
+     * @param others other entities in the list
+     */
+    public B set(String name, FullEntity first, FullEntity second, FullEntity... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (FullEntity other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link ListValue}.
+     *
+     * @param name name of the property
+     * @param values list of values associated with the property
+     */
     public B set(String name, List> values) {
       properties.put(name, of(values));
       return self();
     }
 
-    public B set(String name, Value value, Value... other) {
-      properties.put(name, of(value, other));
+    /**
+     * Sets a property of type {@link ListValue}.
+     *
+     * @param name name of the property
+     * @param first the first value in the list
+     * @param second the second value in the list
+     * @param others other values in the list
+     */
+    public B set(String name, Value first, Value second, Value... others) {
+      properties.put(name, ListValue.builder().addValue(first).addValue(second, others).build());
       return self();
     }
 
+    /**
+     * Sets a property of type {@link BlobValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, Blob value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link BlobValue}.
+     *
+     * @param name name of the property
+     * @param first the first {@link Blob} in the list
+     * @param second the second {@link Blob} in the list
+     * @param others other {@link Blob}s in the list
+     */
+    public B set(String name, Blob first, Blob second, Blob... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (Blob other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@code NullValue}.
+     *
+     * @param name name of the property
+     */
     public B setNull(String name) {
       properties.put(name, of());
       return self();
@@ -348,8 +574,8 @@ public  FullEntity getEntity(String name) {
    * @throws ClassCastException if value is not a list of values
    */
   @SuppressWarnings("unchecked")
-  public List> getList(String name) {
-    return ((Value>>) getValue(name)).get();
+  public > List getList(String name) {
+    return (List) getValue(name).get();
   }
 
   /**
diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseKey.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseKey.java
index a8ad7d4e7734..4ab6f51b6767 100644
--- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseKey.java
+++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseKey.java
@@ -157,6 +157,8 @@ public String kind() {
     return leaf().kind();
   }
 
+  abstract BaseKey parent();
+
   @Override
   public int hashCode() {
     return Objects.hash(projectId(), namespace(), path());
diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Batch.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Batch.java
index 75a5d1381403..5306a685195a 100644
--- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Batch.java
+++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Batch.java
@@ -24,14 +24,14 @@
  * to the Datastore upon {@link #submit}.
  * A usage example:
  * 
 {@code
- *   Entity entity1 = datastore.get(key1);
- *   Batch batch = datastore.newBatch();
- *   Entity entity2 = Entity.builder(key2).set("name", "John").build();
- *   entity1 = Entity.builder(entity1).clear().setNull("bla").build();
- *   Entity entity3 = Entity.builder(key3).set("title", "title").build();
- *   batch.update(entity1);
- *   batch.add(entity2, entity3);
- *   batch.submit();
+ * Entity entity1 = datastore.get(key1);
+ * Batch batch = datastore.newBatch();
+ * Entity entity2 = Entity.builder(key2).set("name", "John").build();
+ * entity1 = Entity.builder(entity1).clear().setNull("bla").build();
+ * Entity entity3 = Entity.builder(key3).set("title", "title").build();
+ * batch.update(entity1);
+ * batch.add(entity2, entity3);
+ * batch.submit();
  * } 
*/ public interface Batch extends DatastoreBatchWriter { diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Cursor.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Cursor.java index 667f3cc5e427..5e577f7feb6c 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Cursor.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Cursor.java @@ -17,21 +17,14 @@ package com.google.gcloud.datastore; import static com.google.common.base.Preconditions.checkNotNull; -import static java.nio.charset.StandardCharsets.UTF_8; import com.google.api.services.datastore.DatastoreV1; import com.google.api.services.datastore.DatastoreV1.Value; import com.google.common.base.MoreObjects; import com.google.common.base.MoreObjects.ToStringHelper; -import com.google.common.base.Preconditions; +import com.google.common.io.BaseEncoding; import com.google.protobuf.ByteString; import com.google.protobuf.InvalidProtocolBufferException; -import com.google.protobuf.TextFormat; -import com.google.protobuf.TextFormat.ParseException; - -import java.io.UnsupportedEncodingException; -import java.net.URLDecoder; -import java.net.URLEncoder; /** * A Google Cloud Datastore cursor. @@ -44,7 +37,6 @@ public final class Cursor extends Serializable { private final transient ByteString byteString; Cursor(ByteString byteString) { - Preconditions.checkArgument(byteString.isValidUtf8(), "content is not a valid UTF-8"); this.byteString = byteString; } @@ -76,11 +68,7 @@ ByteString byteString() { * Returns the cursor in an encoded form that can be used as part of a URL. */ public String toUrlSafe() { - try { - return URLEncoder.encode(TextFormat.printToString(toPb()), UTF_8.name()); - } catch (UnsupportedEncodingException e) { - throw new IllegalStateException("Unexpected encoding exception", e); - } + return BaseEncoding.base64Url().encode(byteString.toByteArray()); } /** @@ -88,11 +76,8 @@ public String toUrlSafe() { */ public static Cursor fromUrlSafe(String urlSafe) { try { - String utf8Str = URLDecoder.decode(urlSafe, UTF_8.name()); - DatastoreV1.Value.Builder builder = DatastoreV1.Value.newBuilder(); - TextFormat.merge(utf8Str, builder); - return fromPb(builder.build()); - } catch (UnsupportedEncodingException | ParseException e) { + return Cursor.copyFrom(BaseEncoding.base64Url().decode(urlSafe)); + } catch (IllegalArgumentException e) { throw new IllegalStateException("Unexpected decoding exception", e); } } diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/EntityQuery.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/EntityQuery.java new file mode 100644 index 000000000000..902168f20f48 --- /dev/null +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/EntityQuery.java @@ -0,0 +1,67 @@ +/* + * Copyright 2015 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.datastore; + +import com.google.api.services.datastore.DatastoreV1; + +/** + * An implementation of a Google Cloud Datastore entity query that can be constructed by providing + * all the specific query elements. + * + * @see Datastore + * queries + */ +public final class EntityQuery extends StructuredQuery { + + private static final long serialVersionUID = 2990565454831019471L; + + /** + * A {@code EntityQuery} builder for queries that return {@link Entity} results. + */ + public static final class Builder extends StructuredQuery.BuilderImpl { + + Builder(EntityQuery query) { + super(query); + } + + Builder() { + super(ResultType.ENTITY); + } + + @Override + Builder mergeFrom(DatastoreV1.Query queryPb) { + super.mergeFrom(queryPb); + clearProjection(); + clearGroupBy(); + return this; + } + + @Override + public EntityQuery build() { + return new EntityQuery(this); + } + } + + EntityQuery(Builder builder) { + super(builder); + } + + @Override + public Builder toBuilder() { + return new Builder(this); + } +} diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/GqlQuery.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/GqlQuery.java index e6ae166dbf07..7c03b69d9f39 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/GqlQuery.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/GqlQuery.java @@ -43,27 +43,27 @@ *

A usage example:

* *

When the type of the results is known the preferred usage would be: - *

{@code
- *   Query query =
- *       Query.gqlQueryBuilder(Query.ResultType.ENTITY, "select * from kind").build();
- *   QueryResults results = datastore.run(query);
- *   while (results.hasNext()) {
- *     Entity entity = results.next();
- *     ...
- *   }
+ * 
 {@code
+ * Query query =
+ *     Query.gqlQueryBuilder(Query.ResultType.ENTITY, "select * from kind").build();
+ * QueryResults results = datastore.run(query);
+ * while (results.hasNext()) {
+ *   Entity entity = results.next();
+ *   ...
+ * }
  * } 
* *

When the type of the results is unknown you can use this approach: - *

{@code
- *   Query query = Query.gqlQueryBuilder("select __key__ from kind").build();
- *   QueryResults results = datastore.run(query);
- *   if (Key.class.isAssignableFrom(results.resultClass())) {
- *     QueryResults keys = (QueryResults) results;
- *     while (keys.hasNext()) {
- *       Key key = keys.next();
- *       ...
- *     }
+ * 
 {@code
+ * Query query = Query.gqlQueryBuilder("select __key__ from kind").build();
+ * QueryResults results = datastore.run(query);
+ * if (Key.class.isAssignableFrom(results.resultClass())) {
+ *   QueryResults keys = (QueryResults) results;
+ *   while (keys.hasNext()) {
+ *     Key key = keys.next();
+ *     ...
  *   }
+ * }
  * } 
* * @param the type of the result values this query will produce diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/IncompleteKey.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/IncompleteKey.java index 2ccd59e725a8..2192384ef70b 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/IncompleteKey.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/IncompleteKey.java @@ -84,6 +84,29 @@ static IncompleteKey fromPb(DatastoreV1.Key keyPb) { return new IncompleteKey(projectId, namespace, path); } + /** + * Returns the key's parent. + */ + @Override + public Key parent() { + List ancestors = ancestors(); + if (ancestors.isEmpty()) { + return null; + } + PathElement parent = ancestors.get(ancestors.size() - 1); + Key.Builder keyBuilder; + if (parent.hasName()) { + keyBuilder = Key.builder(projectId(), parent.kind(), parent.name()); + } else { + keyBuilder = Key.builder(projectId(), parent.kind(), parent.id()); + } + String namespace = namespace(); + if (namespace != null) { + keyBuilder.namespace(namespace); + } + return keyBuilder.ancestors(ancestors.subList(0, ancestors.size() - 1)).build(); + } + public static Builder builder(String projectId, String kind) { return new Builder(projectId, kind); } diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/KeyQuery.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/KeyQuery.java new file mode 100644 index 000000000000..7afa0f5099d6 --- /dev/null +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/KeyQuery.java @@ -0,0 +1,68 @@ +/* + * Copyright 2015 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.datastore; + +import com.google.api.services.datastore.DatastoreV1; + +/** + * An implementation of a Google Cloud Datastore key-only query that can be constructed by providing + * all the specific query elements. + * + * @see Datastore + * queries + */ +public final class KeyQuery extends StructuredQuery { + + private static final long serialVersionUID = -746768461459070045L; + + /** + * A {@code KeyQuery} builder for queries that return {@link Key} results. + */ + public static final class Builder extends StructuredQuery.BuilderImpl { + + Builder(KeyQuery query) { + super(query); + } + + Builder() { + super(ResultType.KEY); + projection(Projection.property(KEY_PROPERTY_NAME)); + } + + @Override + Builder mergeFrom(DatastoreV1.Query queryPb) { + super.mergeFrom(queryPb); + projection(Projection.property(KEY_PROPERTY_NAME)); + clearGroupBy(); + return this; + } + + @Override + public KeyQuery build() { + return new KeyQuery(this); + } + } + + KeyQuery(Builder builder) { + super(builder); + } + + @Override + public Builder toBuilder() { + return new Builder(this); + } +} diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ListValue.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ListValue.java index 41c7e82788b5..06282a2c79d1 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ListValue.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ListValue.java @@ -70,17 +70,16 @@ private Builder() { super(ValueType.LIST); } - public Builder addValue(Value value) { + private void addValueHelper(Value value) { // see datastore_v1.proto definition for list_value Preconditions.checkArgument(value.type() != ValueType.LIST, "Cannot contain another list"); listBuilder.add(value); - return this; } public Builder addValue(Value first, Value... other) { - addValue(first); + addValueHelper(first); for (Value value : other) { - addValue(value); + addValueHelper(value); } return this; } diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ProjectionEntityQuery.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ProjectionEntityQuery.java new file mode 100644 index 000000000000..bad9fc5af2d0 --- /dev/null +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ProjectionEntityQuery.java @@ -0,0 +1,112 @@ +/* + * Copyright 2015 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.datastore; + +/** + * An implementation of a Google Cloud Datastore projection entity query that can be constructed by + * providing all the specific query elements. + * + * @see Datastore + * queries + */ +public final class ProjectionEntityQuery extends StructuredQuery { + + private static final long serialVersionUID = 5488451194542425391L; + + /** + * A {@code ProjectionEntityQuery} builder for queries that return {@link ProjectionEntity} + * results. + */ + public static final class Builder extends StructuredQuery.BuilderImpl { + + Builder(ProjectionEntityQuery query) { + super(query); + } + + Builder() { + super(ResultType.PROJECTION_ENTITY); + } + + /** + * Clears the projection clause. + */ + @Override + public Builder clearProjection() { + super.clearProjection(); + return this; + } + + /** + * Sets the query's projection clause (clearing any previously specified Projection settings). + */ + @Override + public Builder projection(Projection projection, Projection... others) { + super.projection(projection, others); + return this; + } + + /** + * Adds one or more projections to the existing projection clause. + */ + @Override + public Builder addProjection(Projection projection, Projection... others) { + super.addProjection(projection, others); + return this; + } + + /** + * Clears the group by clause. + */ + @Override + public Builder clearGroupBy() { + super.clearGroupBy(); + return this; + } + + /** + * Sets the query's group by clause (clearing any previously specified GroupBy settings). + */ + @Override + public Builder groupBy(String property, String... others) { + super.groupBy(property, others); + return this; + } + + /** + * Adds one or more properties to the existing group by clause. + */ + @Override + public Builder addGroupBy(String property, String... others) { + super.addGroupBy(property, others); + return this; + } + + @Override + public ProjectionEntityQuery build() { + return new ProjectionEntityQuery(this); + } + } + + ProjectionEntityQuery(Builder builder) { + super(builder); + } + + @Override + public Builder toBuilder() { + return new Builder(this); + } +} diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Query.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Query.java index 0dbd1633928e..50591a87a6a4 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Query.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Query.java @@ -22,9 +22,6 @@ import com.google.common.base.MoreObjects; import com.google.common.base.MoreObjects.ToStringHelper; import com.google.common.collect.Maps; -import com.google.gcloud.datastore.StructuredQuery.EntityQueryBuilder; -import com.google.gcloud.datastore.StructuredQuery.KeyQueryBuilder; -import com.google.gcloud.datastore.StructuredQuery.ProjectionEntityQueryBuilder; import com.google.protobuf.GeneratedMessage; import com.google.protobuf.InvalidProtocolBufferException; @@ -217,21 +214,21 @@ public static GqlQuery.Builder gqlQueryBuilder(ResultType resultType, /** * Returns a new {@link StructuredQuery} builder for full (complete entities) queries. */ - public static EntityQueryBuilder entityQueryBuilder() { - return new EntityQueryBuilder(); + public static EntityQuery.Builder entityQueryBuilder() { + return new EntityQuery.Builder(); } /** * Returns a new {@link StructuredQuery} builder for key only queries. */ - public static KeyQueryBuilder keyQueryBuilder() { - return new KeyQueryBuilder(); + public static KeyQuery.Builder keyQueryBuilder() { + return new KeyQuery.Builder(); } /** * Returns a new {@link StructuredQuery} builder for projection queries. */ - public static ProjectionEntityQueryBuilder projectionEntityQueryBuilder() { - return new ProjectionEntityQueryBuilder(); + public static ProjectionEntityQuery.Builder projectionEntityQueryBuilder() { + return new ProjectionEntityQuery.Builder(); } } diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/StructuredQuery.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/StructuredQuery.java index bbb73df4a79c..5892268f859c 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/StructuredQuery.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/StructuredQuery.java @@ -46,45 +46,45 @@ *

A usage example:

* *

A simple query that returns all entities for a specific kind - *

{@code
- *   Query query = Query.entityQueryBuilder().kind(kind).build();
- *   QueryResults results = datastore.run(query);
- *   while (results.hasNext()) {
- *     Entity entity = results.next();
- *     ...
- *   }
+ * 
 {@code
+ * Query query = Query.entityQueryBuilder().kind(kind).build();
+ * QueryResults results = datastore.run(query);
+ * while (results.hasNext()) {
+ *   Entity entity = results.next();
+ *   ...
+ * }
  * } 
* *

A simple key-only query of all entities for a specific kind - *

{@code
- *   Query keyOnlyQuery =  Query.keyQueryBuilder().kind(KIND1).build();
- *   QueryResults results = datastore.run(keyOnlyQuery);
- *   ...
+ * 
 {@code
+ * Query keyOnlyQuery =  Query.keyQueryBuilder().kind(KIND1).build();
+ * QueryResults results = datastore.run(keyOnlyQuery);
+ * ...
  * } 
* *

A less trivial example of a projection query that returns the first 10 results * of "age" and "name" properties (sorted and grouped by "age") with an age greater than 18 - *

{@code
- *   Query query = Query.projectionEntityQueryBuilder()
- *       .kind(kind)
- *       .projection(Projection.property("age"), Projection.first("name"))
- *       .filter(PropertyFilter.gt("age", 18))
- *       .groupBy("age")
- *       .orderBy(OrderBy.asc("age"))
- *       .limit(10)
- *       .build();
- *   QueryResults results = datastore.run(query);
- *   ...
+ * 
 {@code
+ * Query query = Query.projectionEntityQueryBuilder()
+ *     .kind(kind)
+ *     .projection(Projection.property("age"), Projection.first("name"))
+ *     .filter(PropertyFilter.gt("age", 18))
+ *     .groupBy("age")
+ *     .orderBy(OrderBy.asc("age"))
+ *     .limit(10)
+ *     .build();
+ * QueryResults results = datastore.run(query);
+ * ...
  * } 
* * @param the type of the result values this query will produce * @see Datastore * queries */ -public class StructuredQuery extends Query { +public abstract class StructuredQuery extends Query { private static final long serialVersionUID = 546838955624019594L; - private static final String KEY_PROPERTY_NAME = "__key__"; + static final String KEY_PROPERTY_NAME = "__key__"; private final transient String kind; private final ImmutableList projection; @@ -609,13 +609,48 @@ public static Projection first(String property) { } } + /** + * Interface for StructuredQuery builders. + * + * @param the type of result the query returns. + */ + public interface Builder { + Builder namespace(String namespace); + + Builder kind(String kind); + + Builder startCursor(Cursor startCursor); + + Builder endCursor(Cursor endCursor); + + Builder offset(int offset); + + Builder limit(Integer limit); + + Builder filter(Filter filter); + + Builder clearOrderBy(); + + /** + * Sets the query's order by clause (clearing any previously specified OrderBy settings). + */ + Builder orderBy(OrderBy orderBy, OrderBy... others); + + /** + * Adds settings to the existing order by clause. + */ + Builder addOrderBy(OrderBy orderBy, OrderBy... others); + + StructuredQuery build(); + } + /** * Base class for StructuredQuery builders. * * @param the type of result the query returns. * @param the query builder. */ - protected static class BaseBuilder> { + abstract static class BuilderImpl> implements Builder { private final ResultType resultType; private String namespace; @@ -629,12 +664,12 @@ protected static class BaseBuilder> { private int offset; private Integer limit; - BaseBuilder(ResultType resultType) { + BuilderImpl(ResultType resultType) { this.resultType = resultType; } - BaseBuilder(StructuredQuery query) { - resultType = query.type(); + BuilderImpl(StructuredQuery query) { + this(query.type()); namespace = query.namespace(); kind = query.kind; projection.addAll(query.projection); @@ -652,60 +687,64 @@ B self() { return (B) this; } + @Override public B namespace(String namespace) { this.namespace = namespace; return self(); } + @Override public B kind(String kind) { this.kind = kind; return self(); } + @Override public B startCursor(Cursor startCursor) { this.startCursor = startCursor; return self(); } + @Override public B endCursor(Cursor endCursor) { this.endCursor = endCursor; return self(); } + @Override public B offset(int offset) { Preconditions.checkArgument(offset >= 0, "offset must not be negative"); this.offset = offset; return self(); } + @Override public B limit(Integer limit) { Preconditions.checkArgument(limit == null || limit > 0, "limit must be positive"); this.limit = limit; return self(); } + @Override public B filter(Filter filter) { this.filter = filter; return self(); } + @Override public B clearOrderBy() { orderBy.clear(); return self(); } - /** - * Sets the query's order by clause (clearing any previously specified OrderBy settings). - */ + @Override public B orderBy(OrderBy orderBy, OrderBy... others) { clearOrderBy(); addOrderBy(orderBy, others); return self(); } - /** - * Adds settings to the existing order by clause. - */ + @Override public B addOrderBy(OrderBy orderBy, OrderBy... others) { this.orderBy.add(orderBy); Collections.addAll(this.orderBy, others); @@ -776,121 +815,9 @@ B mergeFrom(DatastoreV1.Query queryPb) { } return self(); } - - public StructuredQuery build() { - return new StructuredQuery<>(this); - } } - static final class Builder extends BaseBuilder> { - - Builder(ResultType resultType) { - super(resultType); - } - - Builder(StructuredQuery query) { - super(query); - } - } - - /** - * A StructuredQuery builder for queries that return Entity results. - */ - public static final class EntityQueryBuilder extends BaseBuilder { - - EntityQueryBuilder() { - super(ResultType.ENTITY); - } - - @Override - public StructuredQuery build() { - return new StructuredQuery<>(this); - } - } - - /** - * A StructuredQuery builder for queries that return Key results. - */ - public static final class KeyQueryBuilder extends BaseBuilder { - - KeyQueryBuilder() { - super(ResultType.KEY); - projection(Projection.property(KEY_PROPERTY_NAME)); - } - - @Override - KeyQueryBuilder mergeFrom(DatastoreV1.Query queryPb) { - super.mergeFrom(queryPb); - projection(Projection.property(KEY_PROPERTY_NAME)); - clearGroupBy(); - return this; - } - - @Override - public StructuredQuery build() { - return new StructuredQuery<>(this); - } - } - - /** - * A StructuredQuery builder for projection queries. - */ - public static final class ProjectionEntityQueryBuilder - extends BaseBuilder { - - ProjectionEntityQueryBuilder() { - super(ResultType.PROJECTION_ENTITY); - } - - @Override - public StructuredQuery build() { - return new StructuredQuery<>(this); - } - - @Override - public ProjectionEntityQueryBuilder clearProjection() { - return super.clearProjection(); - } - - /** - * Sets the query's projection clause (clearing any previously specified Projection settings). - */ - @Override - public ProjectionEntityQueryBuilder projection(Projection projection, Projection... others) { - return super.projection(projection, others); - } - - /** - * Adds one or more projections to the existing projection clause. - */ - @Override - public ProjectionEntityQueryBuilder addProjection(Projection projection, Projection... others) { - return super.addProjection(projection, others); - } - - @Override - public ProjectionEntityQueryBuilder clearGroupBy() { - return super.clearGroupBy(); - } - - /** - * Sets the query's group by clause (clearing any previously specified GroupBy settings). - */ - @Override - public ProjectionEntityQueryBuilder groupBy(String property, String... others) { - return super.groupBy(property, others); - } - - /** - * Adds one or more properties to the existing group by clause. - */ - @Override - public ProjectionEntityQueryBuilder addGroupBy(String property, String... others) { - return super.addGroupBy(property, others); - } - } - - StructuredQuery(BaseBuilder builder) { + StructuredQuery(BuilderImpl builder) { super(builder.resultType, builder.namespace); kind = builder.kind; projection = ImmutableList.copyOf(builder.projection); @@ -971,9 +898,7 @@ public Integer limit() { return limit; } - public Builder toBuilder() { - return new Builder<>(this); - } + public abstract Builder toBuilder(); @Override void populatePb(DatastoreV1.RunQueryRequest.Builder requestPb) { @@ -982,8 +907,7 @@ void populatePb(DatastoreV1.RunQueryRequest.Builder requestPb) { @Override StructuredQuery nextQuery(DatastoreV1.QueryResultBatch responsePb) { - Builder builder = new Builder<>(type()); - builder.mergeFrom(toPb()); + Builder builder = toBuilder(); builder.startCursor(new Cursor(responsePb.getEndCursor())); if (offset > 0 && responsePb.getSkippedResults() < offset) { builder.offset(offset - responsePb.getSkippedResults()); @@ -1035,15 +959,15 @@ Object fromPb(ResultType resultType, String namespace, byte[] bytesPb) return fromPb(resultType, namespace, DatastoreV1.Query.parseFrom(bytesPb)); } - private static StructuredQuery fromPb(ResultType resultType, String namespace, + static StructuredQuery fromPb(ResultType resultType, String namespace, DatastoreV1.Query queryPb) { - BaseBuilder builder; + BuilderImpl builder; if (resultType.equals(ResultType.ENTITY)) { - builder = new EntityQueryBuilder(); + builder = new EntityQuery.Builder(); } else if (resultType.equals(ResultType.KEY)) { - builder = new KeyQueryBuilder(); + builder = new KeyQuery.Builder(); } else { - builder = new ProjectionEntityQueryBuilder(); + builder = new ProjectionEntityQuery.Builder(); } return builder.namespace(namespace).mergeFrom(queryPb).build(); } diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Transaction.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Transaction.java index 8089c0130f5d..78ee217f31e7 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Transaction.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Transaction.java @@ -30,21 +30,21 @@ * the Datastore upon {@code commit}. * A usage example: *
 {@code
- *   Transaction transaction = datastore.newTransaction();
- *   try {
- *     Entity entity = transaction.get(key);
- *     if (!entity.contains("last_name") || entity.isNull("last_name")) {
- *       String[] name = entity.getString("name").split(" ");
- *       entity = Entity.builder(entity).remove("name").set("first_name", name[0])
- *           .set("last_name", name[1]).build();
- *       transaction.update(entity);
- *       transaction.commit();
- *     }
- *   } finally {
- *     if (transaction.active()) {
- *       transaction.rollback();
- *     }
+ * Transaction transaction = datastore.newTransaction();
+ * try {
+ *   Entity entity = transaction.get(key);
+ *   if (!entity.contains("last_name") || entity.isNull("last_name")) {
+ *     String[] name = entity.getString("name").split(" ");
+ *     entity = Entity.builder(entity).remove("name").set("first_name", name[0])
+ *         .set("last_name", name[1]).build();
+ *     transaction.update(entity);
+ *     transaction.commit();
  *   }
+ * } finally {
+ *   if (transaction.active()) {
+ *     transaction.rollback();
+ *   }
+ * }
  * } 
* * @see diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/package-info.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/package-info.java index 1404b2817802..fbf468d6458d 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/package-info.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/package-info.java @@ -17,35 +17,38 @@ /** * A client to the Google Cloud Datastore. * - *

Here's a simple usage example for using gcloud-java from App/Compute Engine: + *

Here's a simple usage example for using gcloud-java from App/Compute Engine. This example + * shows how to create a Datastore entity. For the complete source code see + * + * CreateEntity.java. *

 {@code
  * Datastore datastore = DatastoreOptions.defaultInstance().service();
- * KeyFactory keyFactory = datastore.newKeyFactory().kind(kind);
- * Key key = keyFactory.newKey(keyName);
+ * KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind");
+ * Key key = keyFactory.newKey("keyName");
+ * Entity entity = Entity.builder(key)
+ *     .set("name", "John Doe")
+ *     .set("age", 30)
+ *     .set("access_time", DateTime.now())
+ *     .build();
+ * datastore.put(entity);
+ * } 
+ *

+ * This second example shows how to get and update a Datastore entity if it exists. For the complete + * source code see + * + * UpdateEntity.java. + *

 {@code
+ * Datastore datastore = DatastoreOptions.defaultInstance().service();
+ * KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind");
+ * Key key = keyFactory.newKey("keyName");
  * Entity entity = datastore.get(key);
- * if (entity == null) {
- *   entity = Entity.builder(key)
- *       .set("name", "John Do")
- *       .set("age", LongValue.builder(100).indexed(false).build())
- *       .set("updated", false)
+ * if (entity != null) {
+ *   System.out.println("Updating access_time for " + entity.getString("name"));
+ *   entity = Entity.builder(entity)
+ *       .set("access_time", DateTime.now())
  *       .build();
- *   datastore.put(entity);
- * } else {
- *   boolean updated = entity.getBoolean("updated");
- *   if (!updated) {
- *     String[] name = entity.getString("name").split(" ");
- *     entity = Entity.builder(entity)
- *         .set("name", name[0])
- *         .set("last_name", StringValue.builder(name[1]).indexed(false).build())
- *         .set("updated", true)
- *         .remove("old_property")
- *         .set("new_property", 1.1)
- *         .build();
- *     datastore.update(entity);
- *   }
- * }
- * } 
- * + * datastore.update(entity); + * }}
*

When using gcloud-java from outside of App/Compute Engine, you have to specify a * project ID and diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/testing/LocalGcdHelper.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/testing/LocalGcdHelper.java index 7c60da50b0b0..fdb6774f810f 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/testing/LocalGcdHelper.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/testing/LocalGcdHelper.java @@ -17,6 +17,7 @@ package com.google.gcloud.datastore.testing; import static com.google.common.base.MoreObjects.firstNonNull; +import static com.google.common.base.Preconditions.checkArgument; import static java.nio.charset.StandardCharsets.UTF_8; import com.google.common.base.Strings; @@ -85,6 +86,7 @@ public class LocalGcdHelper { private static final String GCLOUD = "gcloud"; private static final Path INSTALLED_GCD_PATH; private static final String GCD_VERSION_PREFIX = "gcd-emulator "; + private static final double DEFAULT_CONSISTENCY = 0.9; static { INSTALLED_GCD_PATH = installedGcdPath(); @@ -398,9 +400,15 @@ public LocalGcdHelper(String projectId, int port) { * All content is written to a temporary directory that will be deleted when * {@link #stop()} is called or when the program terminates) to make sure that no left-over * data from prior runs is used. + * + * @param consistency the fraction of job application attempts that will succeed, with 0.0 + * resulting in no attempts succeeding, and 1.0 resulting in all attempts succeeding. Defaults + * to 0.9. Note that setting this to 1.0 may mask incorrect assumptions about the consistency + * of non-ancestor queries; non-ancestor queries are eventually consistent. */ - public void start() throws IOException, InterruptedException { + public void start(double consistency) throws IOException, InterruptedException { // send a quick request in case we have a hanging process from a previous run + checkArgument(consistency >= 0.0 && consistency <= 1.0, "Consistency must be between 0 and 1"); sendQuitRequest(port); // Each run is associated with its own folder that is deleted once test completes. gcdPath = Files.createTempDirectory("gcd"); @@ -415,7 +423,7 @@ public void start() throws IOException, InterruptedException { } else { gcdExecutablePath = INSTALLED_GCD_PATH; } - startGcd(gcdExecutablePath); + startGcd(gcdExecutablePath, consistency); } private void downloadGcd() throws IOException { @@ -453,7 +461,8 @@ private void downloadGcd() throws IOException { } } - private void startGcd(Path executablePath) throws IOException, InterruptedException { + private void startGcd(Path executablePath, double consistency) + throws IOException, InterruptedException { // cleanup any possible data for the same project File datasetFolder = new File(gcdPath.toFile(), projectId); deleteRecurse(datasetFolder.toPath()); @@ -486,7 +495,8 @@ private void startGcd(Path executablePath) throws IOException, InterruptedExcept startProcess = CommandWrapper.create() .command(gcdAbsolutePath.toString(), "start", "--testing", "--allow_remote_shutdown", - "--port=" + Integer.toString(port), projectId) + "--port=" + Integer.toString(port), "--consistency=" + Double.toString(consistency), + projectId) .directory(gcdPath) .start(); processReader = ProcessStreamReader.start(startProcess.getInputStream()); @@ -526,6 +536,7 @@ private static void extractFile(ZipInputStream zipIn, File filePath) throws IOEx public static boolean sendQuitRequest(int port) { StringBuilder result = new StringBuilder(); + String shutdownMsg = "Shutting down local server"; try { URL url = new URL("http", "localhost", port, "/_ah/admin/quit"); HttpURLConnection con = (HttpURLConnection) url.openConnection(); @@ -537,13 +548,13 @@ public static boolean sendQuitRequest(int port) { out.flush(); InputStream in = con.getInputStream(); int currByte = 0; - while ((currByte = in.read()) != -1) { + while ((currByte = in.read()) != -1 && result.length() < shutdownMsg.length()) { result.append(((char) currByte)); } } catch (IOException ignore) { // ignore } - return result.toString().startsWith("Shutting down local server"); + return result.toString().startsWith(shutdownMsg); } public void stop() throws IOException, InterruptedException { @@ -578,10 +589,10 @@ public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IO }); } - public static LocalGcdHelper start(String projectId, int port) + public static LocalGcdHelper start(String projectId, int port, double consistency) throws IOException, InterruptedException { LocalGcdHelper helper = new LocalGcdHelper(projectId, port); - helper.start(); + helper.start(consistency); return helper; } @@ -593,7 +604,9 @@ public static void main(String... args) throws IOException, InterruptedException switch (action) { case "START": if (!isActive(DEFAULT_PROJECT_ID, port)) { - LocalGcdHelper helper = start(DEFAULT_PROJECT_ID, port); + double consistency = parsedArgs.get("consistency") == null + ? DEFAULT_CONSISTENCY : Double.parseDouble(parsedArgs.get("consistency")); + LocalGcdHelper helper = start(DEFAULT_PROJECT_ID, port, consistency); try (FileWriter writer = new FileWriter(".local_gcd_helper")) { writer.write(helper.gcdPath.toAbsolutePath().toString() + System.lineSeparator()); writer.write(Integer.toString(port)); diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseEntityTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseEntityTest.java index 5ece01508d3a..a69ea5e20e3b 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseEntityTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseEntityTest.java @@ -67,6 +67,16 @@ public void setUp() { builder.set("list1", NullValue.of(), StringValue.of("foo")); builder.set("list2", ImmutableList.of(LongValue.of(10), DoubleValue.of(2))); builder.set("list3", Collections.singletonList(BooleanValue.of(true))); + builder.set( + "blobList", BLOB, Blob.copyFrom(new byte[] {3, 4}), Blob.copyFrom(new byte[] {5, 6})); + builder.set("booleanList", true, false, true); + builder.set("dateTimeList", DateTime.now(), DateTime.now(), DateTime.now()); + builder.set("doubleList", 12.3, 4.56, .789); + builder.set("keyList", KEY, Key.builder("ds2", "k2", "n2").build(), + Key.builder("ds3", "k3", "n3").build()); + builder.set("entityList", ENTITY, PARTIAL_ENTITY); + builder.set("stringList", "s1", "s2", "s3"); + builder.set("longList", 1, 23, 456); } @Test @@ -183,6 +193,17 @@ public void testGetList() throws Exception { assertEquals(Boolean.TRUE, list.get(0).get()); entity = builder.set("list1", ListValue.of(list)).build(); assertEquals(list, entity.getList("list1")); + List> stringList = entity.getList("stringList"); + assertEquals( + ImmutableList.of(StringValue.of("s1"), StringValue.of("s2"), StringValue.of("s3")), + stringList); + List> doubleList = entity.getList("doubleList"); + assertEquals( + ImmutableList.of(DoubleValue.of(12.3), DoubleValue.of(4.56), DoubleValue.of(.789)), + doubleList); + List entityList = entity.getList("entityList"); + assertEquals( + ImmutableList.of(EntityValue.of(ENTITY), EntityValue.of(PARTIAL_ENTITY)), entityList); } @Test @@ -198,7 +219,9 @@ public void testGetBlob() throws Exception { public void testNames() throws Exception { Set names = ImmutableSet.builder() .add("string", "stringValue", "boolean", "double", "long", "list1", "list2", "list3") - .add("entity", "partialEntity", "null", "dateTime", "blob", "key") + .add("entity", "partialEntity", "null", "dateTime", "blob", "key", "blobList") + .add("booleanList", "dateTimeList", "doubleList", "keyList", "entityList", "stringList") + .add("longList") .build(); BaseEntity entity = builder.build(); assertEquals(names, entity.names()); diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseKeyTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseKeyTest.java index 8615ee025bd1..43db4695b191 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseKeyTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseKeyTest.java @@ -50,6 +50,11 @@ protected BaseKey build() { protected Object fromPb(byte[] bytesPb) throws InvalidProtocolBufferException { return null; } + + @Override + protected BaseKey parent() { + return null; + } }; } } diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreExceptionTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreExceptionTest.java index 4d62224172f9..f7bdcb89bcec 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreExceptionTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreExceptionTest.java @@ -78,7 +78,8 @@ public void testDatastoreException() throws Exception { @Test public void testTranslateAndThrow() throws Exception { DatastoreException cause = new DatastoreException(503, "message", "UNAVAILABLE"); - RetryHelper.RetryHelperException exceptionMock = createMock(RetryHelper.RetryHelperException.class); + RetryHelper.RetryHelperException exceptionMock = + createMock(RetryHelper.RetryHelperException.class); expect(exceptionMock.getCause()).andReturn(cause).times(2); replay(exceptionMock); try { diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreTest.java index 8cb88f9e7795..a289610fe841 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreTest.java @@ -31,6 +31,7 @@ import com.google.api.services.datastore.DatastoreV1.RunQueryRequest; import com.google.api.services.datastore.DatastoreV1.RunQueryResponse; import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; import com.google.gcloud.RetryParams; import com.google.gcloud.datastore.Query.ResultType; import com.google.gcloud.datastore.StructuredQuery.OrderBy; @@ -39,6 +40,7 @@ import com.google.gcloud.datastore.testing.LocalGcdHelper; import com.google.gcloud.spi.DatastoreRpc; import com.google.gcloud.spi.DatastoreRpcFactory; +import com.google.protobuf.ByteString; import org.easymock.EasyMock; import org.junit.AfterClass; @@ -88,8 +90,8 @@ public class DatastoreTest { FullEntity.builder(INCOMPLETE_KEY2).set("str", STR_VALUE).set("bool", BOOL_VALUE) .set("list", LIST_VALUE1).build(); private static final FullEntity PARTIAL_ENTITY2 = - FullEntity.builder(PARTIAL_ENTITY1).remove("str").set("bool", true). - set("list", LIST_VALUE1.get()).build(); + FullEntity.builder(PARTIAL_ENTITY1).remove("str").set("bool", true) + .set("list", LIST_VALUE1.get()).build(); private static final FullEntity PARTIAL_ENTITY3 = FullEntity.builder(PARTIAL_ENTITY1).key(IncompleteKey.builder(PROJECT_ID, KIND3).build()) .build(); @@ -117,7 +119,7 @@ public class DatastoreTest { @BeforeClass public static void beforeClass() throws IOException, InterruptedException { if (!LocalGcdHelper.isActive(PROJECT_ID, PORT)) { - gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT); + gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT, 1.0); } } @@ -470,9 +472,13 @@ public void testQueryPaginationWithLimit() throws DatastoreException { EasyMock.expect(rpcFactoryMock.create(EasyMock.anyObject(DatastoreOptions.class))) .andReturn(rpcMock); List responses = buildResponsesForQueryPaginationWithLimit(); - for (int i = 0; i < responses.size(); i++) { + List endCursors = Lists.newArrayListWithCapacity(responses.size()); + for (RunQueryResponse response : responses) { EasyMock.expect(rpcMock.runQuery(EasyMock.anyObject(RunQueryRequest.class))) - .andReturn(responses.get(i)); + .andReturn(response); + if (response.getBatch().getMoreResults() != QueryResultBatch.MoreResultsType.NOT_FINISHED) { + endCursors.add(response.getBatch().getEndCursor()); + } } EasyMock.replay(rpcFactoryMock, rpcMock); Datastore mockDatastore = options.toBuilder() @@ -482,6 +488,7 @@ public void testQueryPaginationWithLimit() throws DatastoreException { .service(); int limit = 2; int totalCount = 0; + Iterator cursorIter = endCursors.iterator(); StructuredQuery query = Query.entityQueryBuilder().limit(limit).build(); while (true) { QueryResults results = mockDatastore.run(query); @@ -491,12 +498,15 @@ public void testQueryPaginationWithLimit() throws DatastoreException { resultCount++; totalCount++; } + assertTrue(cursorIter.hasNext()); + Cursor expectedEndCursor = Cursor.copyFrom(cursorIter.next().toByteArray()); + assertEquals(expectedEndCursor, results.cursorAfter()); if (resultCount < limit) { break; } query = query.toBuilder().startCursor(results.cursorAfter()).build(); } - assertEquals(totalCount, 5); + assertEquals(5, totalCount); EasyMock.verify(rpcFactoryMock, rpcMock); } @@ -504,19 +514,20 @@ private List buildResponsesForQueryPaginationWithLimit() { Entity entity4 = Entity.builder(KEY4).set("value", StringValue.of("value")).build(); Entity entity5 = Entity.builder(KEY5).set("value", "value").build(); datastore.add(ENTITY3, entity4, entity5); + DatastoreRpc datastoreRpc = datastore.options().rpc(); List responses = new ArrayList<>(); Query query = Query.entityQueryBuilder().build(); RunQueryRequest.Builder requestPb = RunQueryRequest.newBuilder(); query.populatePb(requestPb); QueryResultBatch queryResultBatchPb = RunQueryResponse.newBuilder() - .mergeFrom(((DatastoreImpl) datastore).runQuery(requestPb.build())) + .mergeFrom(datastoreRpc.runQuery(requestPb.build())) .getBatch(); QueryResultBatch queryResultBatchPb1 = QueryResultBatch.newBuilder() .mergeFrom(queryResultBatchPb) .setMoreResults(QueryResultBatch.MoreResultsType.NOT_FINISHED) .clearEntityResult() .addAllEntityResult(queryResultBatchPb.getEntityResultList().subList(0, 1)) - .setEndCursor(queryResultBatchPb.getEntityResultList().get(0).getCursor()) + .setEndCursor(ByteString.copyFromUtf8("a")) .build(); responses.add(RunQueryResponse.newBuilder().setBatch(queryResultBatchPb1).build()); QueryResultBatch queryResultBatchPb2 = QueryResultBatch.newBuilder() @@ -524,7 +535,8 @@ private List buildResponsesForQueryPaginationWithLimit() { .setMoreResults(QueryResultBatch.MoreResultsType.MORE_RESULTS_AFTER_LIMIT) .clearEntityResult() .addAllEntityResult(queryResultBatchPb.getEntityResultList().subList(1, 2)) - .setEndCursor(queryResultBatchPb.getEntityResultList().get(1).getCursor()) + .setEndCursor( + ByteString.copyFrom(new byte[] {(byte) 0x80})) // test invalid UTF-8 string .build(); responses.add(RunQueryResponse.newBuilder().setBatch(queryResultBatchPb2).build()); QueryResultBatch queryResultBatchPb3 = QueryResultBatch.newBuilder() @@ -532,7 +544,7 @@ private List buildResponsesForQueryPaginationWithLimit() { .setMoreResults(QueryResultBatch.MoreResultsType.MORE_RESULTS_AFTER_LIMIT) .clearEntityResult() .addAllEntityResult(queryResultBatchPb.getEntityResultList().subList(2, 4)) - .setEndCursor(queryResultBatchPb.getEntityResultList().get(3).getCursor()) + .setEndCursor(ByteString.copyFromUtf8("b")) .build(); responses.add(RunQueryResponse.newBuilder().setBatch(queryResultBatchPb3).build()); QueryResultBatch queryResultBatchPb4 = QueryResultBatch.newBuilder() @@ -540,12 +552,23 @@ private List buildResponsesForQueryPaginationWithLimit() { .setMoreResults(QueryResultBatch.MoreResultsType.NO_MORE_RESULTS) .clearEntityResult() .addAllEntityResult(queryResultBatchPb.getEntityResultList().subList(4, 5)) - .setEndCursor(queryResultBatchPb.getEntityResultList().get(4).getCursor()) + .setEndCursor(ByteString.copyFromUtf8("c")) .build(); responses.add(RunQueryResponse.newBuilder().setBatch(queryResultBatchPb4).build()); return responses; } + @Test + public void testToUrlSafe() { + byte[][] invalidUtf8 = + new byte[][] {{(byte) 0xfe}, {(byte) 0xc1, (byte) 0xbf}, {(byte) 0xc0}, {(byte) 0x80}}; + for (byte[] bytes : invalidUtf8) { + assertFalse(ByteString.copyFrom(bytes).isValidUtf8()); + Cursor cursor = new Cursor(ByteString.copyFrom(bytes)); + assertEquals(cursor, Cursor.fromUrlSafe(cursor.toUrlSafe())); + } + } + @Test public void testAllocateId() { KeyFactory keyFactory = datastore.newKeyFactory().kind(KIND1); diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/IncompleteKeyTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/IncompleteKeyTest.java index 7edbf133d330..acd1dfd3c9e3 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/IncompleteKeyTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/IncompleteKeyTest.java @@ -17,29 +17,47 @@ package com.google.gcloud.datastore; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; +import org.junit.Before; import org.junit.Test; public class IncompleteKeyTest { + private static IncompleteKey pk1, pk2; + private static Key parent1; + + @Before + public void setUp() { + pk1 = IncompleteKey.builder("ds", "kind1").build(); + parent1 = Key.builder("ds", "kind2", 10).namespace("ns").build(); + pk2 = IncompleteKey.builder(parent1, "kind3").build(); + } + @Test public void testBuilders() throws Exception { - IncompleteKey pk1 = IncompleteKey.builder("ds", "kind1").build(); assertEquals("ds", pk1.projectId()); assertEquals("kind1", pk1.kind()); assertTrue(pk1.ancestors().isEmpty()); - Key parent = Key.builder("ds", "kind2", 10).build(); - IncompleteKey pk2 = IncompleteKey.builder(parent, "kind3").build(); assertEquals("ds", pk2.projectId()); assertEquals("kind3", pk2.kind()); - assertEquals(parent.path(), pk2.ancestors()); + assertEquals(parent1.path(), pk2.ancestors()); assertEquals(pk2, IncompleteKey.builder(pk2).build()); IncompleteKey pk3 = IncompleteKey.builder(pk2).kind("kind4").build(); assertEquals("ds", pk3.projectId()); assertEquals("kind4", pk3.kind()); - assertEquals(parent.path(), pk3.ancestors()); + assertEquals(parent1.path(), pk3.ancestors()); + } + + @Test + public void testParent() { + assertNull(pk1.parent()); + assertEquals(parent1, pk2.parent()); + Key parent2 = Key.builder("ds", "kind3", "name").namespace("ns").build(); + IncompleteKey pk3 = IncompleteKey.builder(parent2, "kind3").build(); + assertEquals(parent2, pk3.parent()); } } diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/LocalGcdHelperTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/LocalGcdHelperTest.java index 40ea62c5a7e0..5d761a713506 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/LocalGcdHelperTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/LocalGcdHelperTest.java @@ -49,7 +49,7 @@ public void testFindAvailablePort() { @Test public void testSendQuitRequest() throws IOException, InterruptedException { - LocalGcdHelper gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT); + LocalGcdHelper gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT, 0.75); assertTrue(LocalGcdHelper.sendQuitRequest(PORT)); long timeoutMillis = 30000; long startTime = System.currentTimeMillis(); @@ -64,7 +64,7 @@ public void testSendQuitRequest() throws IOException, InterruptedException { @Test public void testStartStop() throws IOException, InterruptedException { - LocalGcdHelper gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT); + LocalGcdHelper gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT, 0.75); assertFalse(LocalGcdHelper.isActive("wrong-project-id", PORT)); assertTrue(LocalGcdHelper.isActive(PROJECT_ID, PORT)); gcdHelper.stop(); diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/StructuredQueryTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/StructuredQueryTest.java new file mode 100644 index 000000000000..4b6589efd723 --- /dev/null +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/StructuredQueryTest.java @@ -0,0 +1,172 @@ +/* + * Copyright 2015 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.datastore; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import com.google.common.collect.ImmutableList; +import com.google.gcloud.datastore.Query.ResultType; +import com.google.gcloud.datastore.StructuredQuery.CompositeFilter; +import com.google.gcloud.datastore.StructuredQuery.Filter; +import com.google.gcloud.datastore.StructuredQuery.OrderBy; +import com.google.gcloud.datastore.StructuredQuery.Projection; +import com.google.gcloud.datastore.StructuredQuery.PropertyFilter; + +import org.junit.Test; + +import java.util.List; + +public class StructuredQueryTest { + + private static final String NAMESPACE = "ns"; + private static final String KIND = "k"; + private static final Cursor START_CURSOR = Cursor.copyFrom(new byte[] {1, 2}); + private static final Cursor END_CURSOR = Cursor.copyFrom(new byte[] {10}); + private static final int OFFSET = 42; + private static final Integer LIMIT = 43; + private static final Filter FILTER = + CompositeFilter.and(PropertyFilter.gt("p1", 10), PropertyFilter.eq("a", "v")); + private static final OrderBy ORDER_BY_1 = OrderBy.asc("p2"); + private static final OrderBy ORDER_BY_2 = OrderBy.desc("p3"); + private static final List ORDER_BY = ImmutableList.of(ORDER_BY_1, ORDER_BY_2); + private static final Projection PROJECTION1 = Projection.property("p4"); + private static final Projection PROJECTION2 = Projection.property("p5"); + private static final List PROJECTION = ImmutableList.of(PROJECTION1, PROJECTION2); + private static final String GROUP_BY1 = "p6"; + private static final String GROUP_BY2 = "p7"; + private static final List GROUP_BY = ImmutableList.of(GROUP_BY1, GROUP_BY2); + private static final EntityQuery ENTITY_QUERY = Query.entityQueryBuilder() + .namespace(NAMESPACE) + .kind(KIND) + .startCursor(START_CURSOR) + .endCursor(END_CURSOR) + .offset(OFFSET) + .limit(LIMIT) + .filter(FILTER) + .orderBy(ORDER_BY_1, ORDER_BY_2) + .build(); + private static final KeyQuery KEY_QUERY = Query.keyQueryBuilder() + .namespace(NAMESPACE) + .kind(KIND) + .startCursor(START_CURSOR) + .endCursor(END_CURSOR) + .offset(OFFSET) + .limit(LIMIT) + .filter(FILTER) + .orderBy(ORDER_BY_1, ORDER_BY_2) + .build(); + private static final ProjectionEntityQuery PROJECTION_QUERY = + Query.projectionEntityQueryBuilder() + .namespace(NAMESPACE) + .kind(KIND) + .startCursor(START_CURSOR) + .endCursor(END_CURSOR) + .offset(OFFSET) + .limit(LIMIT) + .filter(FILTER) + .orderBy(ORDER_BY_1, ORDER_BY_2) + .projection(PROJECTION1, PROJECTION2) + .groupBy(GROUP_BY1, GROUP_BY2) + .build(); + + @Test + public void testEntityQueryBuilder() { + compareBaseBuilderFields(ENTITY_QUERY); + assertTrue(ENTITY_QUERY.projection().isEmpty()); + assertTrue(ENTITY_QUERY.groupBy().isEmpty()); + } + + @Test + public void testKeyQueryBuilder() { + compareBaseBuilderFields(KEY_QUERY); + assertEquals( + ImmutableList.of(Projection.property(StructuredQuery.KEY_PROPERTY_NAME)), + KEY_QUERY.projection()); + assertTrue(KEY_QUERY.groupBy().isEmpty()); + } + + @Test + public void testProjectionEntityQueryBuilder() { + compareBaseBuilderFields(PROJECTION_QUERY); + assertEquals(PROJECTION, PROJECTION_QUERY.projection()); + assertEquals(GROUP_BY, PROJECTION_QUERY.groupBy()); + } + + private void compareBaseBuilderFields(StructuredQuery query) { + assertEquals(NAMESPACE, query.namespace()); + assertEquals(KIND, query.kind()); + assertEquals(START_CURSOR, query.startCursor()); + assertEquals(END_CURSOR, query.endCursor()); + assertEquals(OFFSET, query.offset()); + assertEquals(LIMIT, query.limit()); + assertEquals(FILTER, query.filter()); + assertEquals(ORDER_BY, query.orderBy()); + } + + @Test + public void mergeFrom() { + compareMergedQuery( + ENTITY_QUERY, new EntityQuery.Builder().mergeFrom(ENTITY_QUERY.toPb()).build()); + compareMergedQuery(KEY_QUERY, new KeyQuery.Builder().mergeFrom(KEY_QUERY.toPb()).build()); + compareMergedQuery( + PROJECTION_QUERY, + new ProjectionEntityQuery.Builder().mergeFrom(PROJECTION_QUERY.toPb()).build()); + } + + private void compareMergedQuery(StructuredQuery expected, StructuredQuery actual) { + assertEquals(expected.kind(), actual.kind()); + assertEquals(expected.startCursor(), actual.startCursor()); + assertEquals(expected.endCursor(), actual.endCursor()); + assertEquals(expected.offset(), actual.offset()); + assertEquals(expected.limit(), actual.limit()); + assertEquals(expected.filter(), actual.filter()); + assertEquals(expected.orderBy(), actual.orderBy()); + assertEquals(expected.projection(), actual.projection()); + assertEquals(expected.groupBy(), actual.groupBy()); + } + + @Test + public void testToAndFromPb() { + assertEquals( + ENTITY_QUERY, + StructuredQuery.fromPb(ResultType.ENTITY, ENTITY_QUERY.namespace(), ENTITY_QUERY.toPb())); + assertEquals( + KEY_QUERY, StructuredQuery.fromPb(ResultType.KEY, KEY_QUERY.namespace(), KEY_QUERY.toPb())); + assertEquals( + PROJECTION_QUERY, + StructuredQuery.fromPb( + ResultType.PROJECTION_ENTITY, PROJECTION_QUERY.namespace(), PROJECTION_QUERY.toPb())); + } + + @Test + public void testToBuilder() { + List> queries = + ImmutableList.>of(ENTITY_QUERY, KEY_QUERY, PROJECTION_QUERY); + for (StructuredQuery query : queries) { + assertEquals(query, query.toBuilder().build()); + } + } + + @Test + public void testKeyOnly() { + assertTrue(KEY_QUERY.keyOnly()); + assertFalse(ENTITY_QUERY.keyOnly()); + assertFalse(PROJECTION_QUERY.keyOnly()); + } +} diff --git a/gcloud-java-dns/pom.xml b/gcloud-java-dns/pom.xml index 5f04f261d500..1a559473cc82 100644 --- a/gcloud-java-dns/pom.xml +++ b/gcloud-java-dns/pom.xml @@ -13,7 +13,7 @@ com.google.gcloud gcloud-java-pom - 0.1.3-SNAPSHOT + 0.1.5-SNAPSHOT gcloud-java-dns diff --git a/gcloud-java-examples/README.md b/gcloud-java-examples/README.md index 8d64c8ea202f..59fbca11e219 100644 --- a/gcloud-java-examples/README.md +++ b/gcloud-java-examples/README.md @@ -6,6 +6,8 @@ Examples for gcloud-java (Java idiomatic client for [Google Cloud Platform][clou [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-examples.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-examples.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [Examples] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/examples/package-summary.html) @@ -17,16 +19,16 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java-examples - 0.1.2 + 0.1.4 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java-examples:0.1.2' +compile 'com.google.gcloud:gcloud-java-examples:0.1.4' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java-examples" % "0.1.2" +libraryDependencies += "com.google.gcloud" % "gcloud-java-examples" % "0.1.4" ``` To run examples from your command line: @@ -39,54 +41,54 @@ To run examples from your command line: 4. Run an example using Maven from command line. - Here's an example run of `BigQueryExample`. - - Before running the example, go to the [Google Developers Console][developers-console] to ensure - that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket - (replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS - [web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to - load data into a BigQuery table and should look something like: - ```csv - value1 - value2 - value3 - ``` - Then you are ready to run the following example: - ``` - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id" - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string" - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id" - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file" - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'" - ``` - - Here's an example run of `DatastoreExample`. + * Here's an example run of `BigQueryExample`. + + Before running the example, go to the [Google Developers Console][developers-console] to ensure + that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket + (replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS + [web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to + load data into a BigQuery table and should look something like: + ```csv + value1 + value2 + value3 + ``` + Then you are ready to run the following example: + ``` + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="create dataset new_dataset_id" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="list tables new_dataset_id" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'" + ``` + + * Here's an example run of `DatastoreExample`. - Note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands. - ``` - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name add my\ comment" - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name display" - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name delete" - ``` - - Here's an example run of `ResourceManagerExample`. - - Be sure to change the placeholder project ID "my-project-id" with your own globally unique project ID. - ``` - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="create my-project-id" - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="list" - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get my-project-id" - ``` - - Here's an example run of `StorageExample`. - - Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine. - ``` - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt " - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list " - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download test.txt" - $mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete test.txt" - ``` + Be sure to change the placeholder project ID "your-project-id" with your own project ID. Also note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands. + ``` + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name add my\ comment" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name display" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name delete" + ``` + + * Here's an example run of `ResourceManagerExample`. + + Be sure to change the placeholder project ID "your-project-id" with your own globally unique project ID. + ``` + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="create your-project-id" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="list" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="get your-project-id" + ``` + + * Here's an example run of `StorageExample`. + + Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine. + ``` + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="upload /path/to/test.txt " + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="list " + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="download test.txt" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="delete test.txt" + ``` Troubleshooting --------------- diff --git a/gcloud-java-examples/pom.xml b/gcloud-java-examples/pom.xml index a5f50b4fe89a..862d48c89eaa 100644 --- a/gcloud-java-examples/pom.xml +++ b/gcloud-java-examples/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java-examples jar GCloud Java examples @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.3-SNAPSHOT + 0.1.5-SNAPSHOT gcloud-java-examples diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java similarity index 89% rename from gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java rename to gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java index 895502e3d6d5..c8fbe7289f9c 100644 --- a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java @@ -14,34 +14,35 @@ * limitations under the License. */ -package com.google.gcloud.examples; +package com.google.gcloud.examples.bigquery; import com.google.common.collect.ImmutableMap; import com.google.gcloud.WriteChannel; -import com.google.gcloud.bigquery.BaseTableInfo; import com.google.gcloud.bigquery.BigQuery; import com.google.gcloud.bigquery.BigQueryError; import com.google.gcloud.bigquery.BigQueryOptions; -import com.google.gcloud.bigquery.CopyJobInfo; +import com.google.gcloud.bigquery.CopyJobConfiguration; +import com.google.gcloud.bigquery.Dataset; import com.google.gcloud.bigquery.DatasetId; import com.google.gcloud.bigquery.DatasetInfo; -import com.google.gcloud.bigquery.ExternalDataConfiguration; -import com.google.gcloud.bigquery.ExternalTableInfo; -import com.google.gcloud.bigquery.ExtractJobInfo; +import com.google.gcloud.bigquery.ExternalTableDefinition; +import com.google.gcloud.bigquery.ExtractJobConfiguration; import com.google.gcloud.bigquery.Field; import com.google.gcloud.bigquery.FieldValue; import com.google.gcloud.bigquery.FormatOptions; +import com.google.gcloud.bigquery.Job; import com.google.gcloud.bigquery.JobId; import com.google.gcloud.bigquery.JobInfo; -import com.google.gcloud.bigquery.JobStatus; -import com.google.gcloud.bigquery.LoadConfiguration; -import com.google.gcloud.bigquery.LoadJobInfo; +import com.google.gcloud.bigquery.LoadJobConfiguration; import com.google.gcloud.bigquery.QueryRequest; import com.google.gcloud.bigquery.QueryResponse; import com.google.gcloud.bigquery.Schema; +import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; import com.google.gcloud.bigquery.TableId; import com.google.gcloud.bigquery.TableInfo; -import com.google.gcloud.bigquery.ViewInfo; +import com.google.gcloud.bigquery.ViewDefinition; +import com.google.gcloud.bigquery.WriteChannelConfiguration; import com.google.gcloud.spi.BigQueryRpc.Tuple; import java.nio.channels.FileChannel; @@ -62,7 +63,7 @@ *

  • login using gcloud SDK - {@code gcloud auth login}.
  • *
  • compile using maven - {@code mvn compile}
  • *
  • run using maven - - *
    {@code mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample"
    + * 
    {@code mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample"
      *  -Dexec.args="[]
      *  list datasets |
      *  list tables  |
    @@ -177,7 +178,7 @@ Void parse(String... args) throws Exception {
       private static class ListDatasetsAction extends NoArgsAction {
         @Override
         public void run(BigQuery bigquery, Void arg) {
    -      Iterator datasetInfoIterator = bigquery.listDatasets().iterateAll();
    +      Iterator datasetInfoIterator = bigquery.listDatasets().iterateAll();
           while (datasetInfoIterator.hasNext()) {
             System.out.println(datasetInfoIterator.next());
           }
    @@ -212,7 +213,7 @@ public String params() {
       private static class ListTablesAction extends DatasetAction {
         @Override
         public void run(BigQuery bigquery, DatasetId datasetId) {
    -      Iterator tableInfoIterator = bigquery.listTables(datasetId).iterateAll();
    +      Iterator
  • tableInfoIterator = bigquery.listTables(datasetId).iterateAll(); while (tableInfoIterator.hasNext()) { System.out.println(tableInfoIterator.next()); } @@ -356,7 +357,7 @@ public String params() { private static class ListJobsAction extends NoArgsAction { @Override public void run(BigQuery bigquery, Void arg) { - Iterator datasetInfoIterator = bigquery.listJobs().iterateAll(); + Iterator datasetInfoIterator = bigquery.listJobs().iterateAll(); while (datasetInfoIterator.hasNext()) { System.out.println(datasetInfoIterator.next()); } @@ -391,10 +392,10 @@ public void run(BigQuery bigquery, JobId jobId) { } } - private abstract static class CreateTableAction extends BigQueryAction { + private abstract static class CreateTableAction extends BigQueryAction { @Override - void run(BigQuery bigquery, BaseTableInfo table) throws Exception { - BaseTableInfo createTable = bigquery.create(table); + void run(BigQuery bigquery, TableInfo table) throws Exception { + Table createTable = bigquery.create(table); System.out.println("Created table:"); System.out.println(createTable.toString()); } @@ -435,20 +436,20 @@ static Schema parseSchema(String[] args, int start, int end) { } /** - * This class demonstrates how to create a simple BigQuery Table (i.e. a table of type - * {@link BaseTableInfo.Type#TABLE}). + * This class demonstrates how to create a simple BigQuery Table (i.e. a table created from a + * {@link StandardTableDefinition}). * * @see Tables: insert * */ private static class CreateSimpleTableAction extends CreateTableAction { @Override - BaseTableInfo parse(String... args) throws Exception { + TableInfo parse(String... args) throws Exception { if (args.length >= 3) { String dataset = args[0]; String table = args[1]; TableId tableId = TableId.of(dataset, table); - return TableInfo.of(tableId, parseSchema(args, 2, args.length)); + return TableInfo.of(tableId, StandardTableDefinition.of(parseSchema(args, 2, args.length))); } throw new IllegalArgumentException("Missing required arguments."); } @@ -460,23 +461,23 @@ protected String params() { } /** - * This class demonstrates how to create a BigQuery External Table (i.e. a table of type - * {@link BaseTableInfo.Type#EXTERNAL}). + * This class demonstrates how to create a BigQuery External Table (i.e. a table created from a + * {@link ExternalTableDefinition}). * * @see Tables: insert * */ private static class CreateExternalTableAction extends CreateTableAction { @Override - BaseTableInfo parse(String... args) throws Exception { + TableInfo parse(String... args) throws Exception { if (args.length >= 5) { String dataset = args[0]; String table = args[1]; TableId tableId = TableId.of(dataset, table); - ExternalDataConfiguration configuration = - ExternalDataConfiguration.of(args[args.length - 1], + ExternalTableDefinition externalTableDefinition = + ExternalTableDefinition.of(args[args.length - 1], parseSchema(args, 3, args.length - 1), FormatOptions.of(args[2])); - return ExternalTableInfo.of(tableId, configuration); + return TableInfo.of(tableId, externalTableDefinition); } throw new IllegalArgumentException("Missing required arguments."); } @@ -488,22 +489,22 @@ protected String params() { } /** - * This class demonstrates how to create a BigQuery View Table (i.e. a table of type - * {@link BaseTableInfo.Type#VIEW}). + * This class demonstrates how to create a BigQuery View Table (i.e. a table created from a + * {@link ViewDefinition}). * * @see Tables: insert * */ private static class CreateViewAction extends CreateTableAction { @Override - BaseTableInfo parse(String... args) throws Exception { + TableInfo parse(String... args) throws Exception { String message; if (args.length == 3) { String dataset = args[0]; String table = args[1]; String query = args[2]; TableId tableId = TableId.of(dataset, table); - return ViewInfo.of(tableId, query); + return TableInfo.of(tableId, ViewDefinition.of(query)); } else if (args.length < 3) { message = "Missing required dataset id, table id or query."; } else { @@ -522,11 +523,10 @@ private abstract static class JobRunAction extends BigQueryAction { @Override void run(BigQuery bigquery, JobInfo job) throws Exception { System.out.println("Creating job"); - JobInfo startedJob = bigquery.create(job); - while (startedJob.status().state() != JobStatus.State.DONE) { + Job startedJob = bigquery.create(job); + while (!startedJob.isDone()) { System.out.println("Waiting for job " + startedJob.jobId().job() + " to complete"); Thread.sleep(1000L); - startedJob = bigquery.getJob(startedJob.jobId()); } if (startedJob.status().error() == null) { System.out.println("Job " + startedJob.jobId().job() + " succeeded"); @@ -544,15 +544,15 @@ void run(BigQuery bigquery, JobInfo job) throws Exception { */ private static class LoadAction extends JobRunAction { @Override - LoadJobInfo parse(String... args) throws Exception { + JobInfo parse(String... args) throws Exception { if (args.length >= 4) { String dataset = args[0]; String table = args[1]; String format = args[2]; TableId tableId = TableId.of(dataset, table); - LoadConfiguration configuration = LoadConfiguration.of(tableId, FormatOptions.of(format)); - return LoadJobInfo.builder(configuration, Arrays.asList(args).subList(3, args.length)) - .build(); + LoadJobConfiguration configuration = LoadJobConfiguration.of( + tableId, Arrays.asList(args).subList(3, args.length), FormatOptions.of(format)); + return JobInfo.of(configuration); } throw new IllegalArgumentException("Missing required arguments."); } @@ -570,15 +570,15 @@ protected String params() { */ private static class ExtractAction extends JobRunAction { @Override - ExtractJobInfo parse(String... args) throws Exception { + JobInfo parse(String... args) throws Exception { if (args.length >= 4) { String dataset = args[0]; String table = args[1]; String format = args[2]; TableId tableId = TableId.of(dataset, table); - return ExtractJobInfo.builder(tableId, Arrays.asList(args).subList(3, args.length)) - .format(format) - .build(); + ExtractJobConfiguration configuration = ExtractJobConfiguration.of( + tableId, Arrays.asList(args).subList(3, args.length), format); + return JobInfo.of(configuration); } throw new IllegalArgumentException("Missing required arguments."); } @@ -596,12 +596,12 @@ protected String params() { */ private static class CopyAction extends JobRunAction { @Override - CopyJobInfo parse(String... args) throws Exception { + JobInfo parse(String... args) throws Exception { String message; if (args.length == 4) { TableId sourceTableId = TableId.of(args[0], args[1]); TableId destinationTableId = TableId.of(args[2], args[3]); - return CopyJobInfo.of(destinationTableId, sourceTableId); + return JobInfo.of(CopyJobConfiguration.of(destinationTableId, sourceTableId)); } else if (args.length < 3) { message = "Missing required source or destination table."; } else { @@ -671,9 +671,11 @@ protected String params() { * @see Resumable * Upload */ - private static class LoadFileAction extends BigQueryAction> { + private static class LoadFileAction + extends BigQueryAction> { @Override - void run(BigQuery bigquery, Tuple configuration) throws Exception { + void run(BigQuery bigquery, Tuple configuration) + throws Exception { System.out.println("Running insert"); try (FileChannel fileChannel = FileChannel.open(Paths.get(configuration.y()))) { WriteChannel writeChannel = bigquery.writer(configuration.x()); @@ -688,13 +690,14 @@ void run(BigQuery bigquery, Tuple configuration) thro } @Override - Tuple parse(String... args) throws Exception { + Tuple parse(String... args) throws Exception { if (args.length == 4) { String dataset = args[0]; String table = args[1]; String format = args[2]; TableId tableId = TableId.of(dataset, table); - LoadConfiguration configuration = LoadConfiguration.of(tableId, FormatOptions.of(format)); + WriteChannelConfiguration configuration = + WriteChannelConfiguration.of(tableId, FormatOptions.of(format)); return Tuple.of(configuration, args[3]); } throw new IllegalArgumentException("Missing required arguments."); diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/CreateTableAndLoadData.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/CreateTableAndLoadData.java new file mode 100644 index 000000000000..857f6b43d013 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/CreateTableAndLoadData.java @@ -0,0 +1,64 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.bigquery.snippets; + +import com.google.gcloud.bigquery.BigQuery; +import com.google.gcloud.bigquery.BigQueryOptions; +import com.google.gcloud.bigquery.Field; +import com.google.gcloud.bigquery.FormatOptions; +import com.google.gcloud.bigquery.Job; +import com.google.gcloud.bigquery.Schema; +import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; +import com.google.gcloud.bigquery.TableId; +import com.google.gcloud.bigquery.TableInfo; + +/** + * A snippet for Google Cloud BigQuery showing how to get a BigQuery table or create it if it does + * not exist. The snippet also starts a BigQuery job to load data into the table from a Cloud + * Storage blob and wait until the job completes. + */ +public class CreateTableAndLoadData { + + public static void main(String... args) throws InterruptedException { + BigQuery bigquery = BigQueryOptions.defaultInstance().service(); + TableId tableId = TableId.of("dataset", "table"); + Table table = bigquery.getTable(tableId); + if (table == null) { + System.out.println("Creating table " + tableId); + Field integerField = Field.of("fieldName", Field.Type.integer()); + Schema schema = Schema.of(integerField); + table = bigquery.create(TableInfo.of(tableId, StandardTableDefinition.of(schema))); + } + System.out.println("Loading data into table " + tableId); + Job loadJob = table.load(FormatOptions.csv(), "gs://bucket/path"); + while (!loadJob.isDone()) { + Thread.sleep(1000L); + } + if (loadJob.status().error() != null) { + System.out.println("Job completed with errors"); + } else { + System.out.println("Job succeeded"); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/InsertDataAndQueryTable.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/InsertDataAndQueryTable.java new file mode 100644 index 000000000000..f421bc832441 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/InsertDataAndQueryTable.java @@ -0,0 +1,102 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.bigquery.snippets; + +import com.google.gcloud.bigquery.BigQuery; +import com.google.gcloud.bigquery.BigQueryOptions; +import com.google.gcloud.bigquery.DatasetInfo; +import com.google.gcloud.bigquery.Field; +import com.google.gcloud.bigquery.FieldValue; +import com.google.gcloud.bigquery.InsertAllRequest; +import com.google.gcloud.bigquery.InsertAllResponse; +import com.google.gcloud.bigquery.QueryRequest; +import com.google.gcloud.bigquery.QueryResponse; +import com.google.gcloud.bigquery.Schema; +import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.TableId; +import com.google.gcloud.bigquery.TableInfo; + +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; + +/** + * A snippet for Google Cloud BigQuery showing how to create a BigQuery dataset and table. Once + * created, the snippet streams data into the table and then queries it. + */ +public class InsertDataAndQueryTable { + + public static void main(String... args) throws InterruptedException { + // Create a service instance + BigQuery bigquery = BigQueryOptions.defaultInstance().service(); + + // Create a dataset + String datasetId = "my_dataset_id"; + bigquery.create(DatasetInfo.builder(datasetId).build()); + + TableId tableId = TableId.of(datasetId, "my_table_id"); + // Table field definition + Field stringField = Field.of("StringField", Field.Type.string()); + // Table schema definition + Schema schema = Schema.of(stringField); + // Create a table + StandardTableDefinition tableDefinition = StandardTableDefinition.of(schema); + bigquery.create(TableInfo.of(tableId, tableDefinition)); + + // Define rows to insert + Map firstRow = new HashMap<>(); + Map secondRow = new HashMap<>(); + firstRow.put("StringField", "value1"); + secondRow.put("StringField", "value2"); + // Create an insert request + InsertAllRequest insertRequest = InsertAllRequest.builder(tableId) + .addRow(firstRow) + .addRow(secondRow) + .build(); + // Insert rows + InsertAllResponse insertResponse = bigquery.insertAll(insertRequest); + // Check if errors occurred + if (insertResponse.hasErrors()) { + System.out.println("Errors occurred while inserting rows"); + } + + // Create a query request + QueryRequest queryRequest = QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id") + .maxWaitTime(60000L) + .maxResults(1000L) + .build(); + // Request query to be executed and wait for results + QueryResponse queryResponse = bigquery.query(queryRequest); + while (!queryResponse.jobCompleted()) { + Thread.sleep(1000L); + queryResponse = bigquery.getQueryResults(queryResponse.jobId()); + } + // Read rows + Iterator> rowIterator = queryResponse.result().iterateAll(); + System.out.println("Table rows:"); + while (rowIterator.hasNext()) { + System.out.println(rowIterator.next()); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java similarity index 87% rename from gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java rename to gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java index e408bab1338e..cc4331734200 100644 --- a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java @@ -14,7 +14,7 @@ * limitations under the License. */ -package com.google.gcloud.examples; +package com.google.gcloud.examples.datastore; import com.google.gcloud.datastore.Datastore; import com.google.gcloud.datastore.DatastoreOptions; @@ -25,8 +25,8 @@ import com.google.gcloud.datastore.Key; import com.google.gcloud.datastore.KeyFactory; import com.google.gcloud.datastore.Query; -import com.google.gcloud.datastore.Query.ResultType; import com.google.gcloud.datastore.QueryResults; +import com.google.gcloud.datastore.StructuredQuery; import com.google.gcloud.datastore.StructuredQuery.PropertyFilter; import com.google.gcloud.datastore.Transaction; @@ -44,7 +44,7 @@ *
  • login using gcloud SDK - {@code gcloud auth login}.
  • *
  • compile using maven - {@code mvn compile}
  • *
  • run using maven - {@code mvn exec:java - * -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" + * -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" * -Dexec.args="[projectId] [user] [delete|display|add comment]"}
  • * */ @@ -67,7 +67,7 @@ private static class DeleteAction implements DatastoreAction { public void run(Transaction tx, Key userKey, String... args) { Entity user = tx.get(userKey); if (user == null) { - System.out.println("Nothing to delete, user does not exists."); + System.out.println("Nothing to delete, user does not exist."); return; } Query query = Query.keyQueryBuilder() @@ -100,21 +100,31 @@ public void run(Transaction tx, Key userKey, String... args) { return; } System.out.printf("User '%s' has %d comment[s].%n", userKey.name(), user.getLong("count")); - // ORDER BY timestamp"; - String gql = "SELECT * FROM " + COMMENT_KIND + " WHERE __key__ HAS ANCESTOR @1"; - Query query = Query.gqlQueryBuilder(ResultType.ENTITY, gql) - .namespace(NAMESPACE) - .addBinding(userKey) - .build(); - QueryResults results = tx.run(query); - // We could have added "ORDER BY timestamp" to the query to avoid the sorting bellow - // but that would require adding an ancestor index for timestamp - // see: https://cloud.google.com/datastore/docs/tools/indexconfig + int limit = 200; Map sortedComments = new TreeMap<>(); - while (results.hasNext()) { - Entity result = results.next(); - sortedComments.put(result.getDateTime("timestamp"), result.getString("content")); + StructuredQuery query = + Query.entityQueryBuilder() + .namespace(NAMESPACE) + .kind(COMMENT_KIND) + .filter(PropertyFilter.hasAncestor(userKey)) + .limit(limit) + .build(); + while (true) { + QueryResults results = tx.run(query); + int resultCount = 0; + while (results.hasNext()) { + Entity result = results.next(); + sortedComments.put(result.getDateTime("timestamp"), result.getString("content")); + resultCount++; + } + if (resultCount < limit) { + break; + } + query = query.toBuilder().startCursor(results.cursorAfter()).build(); } + // We could have added "ORDER BY timestamp" to the query to avoid sorting, but that would + // require adding an ancestor index for timestamp. + // See: https://cloud.google.com/datastore/docs/tools/indexconfig for (Map.Entry entry : sortedComments.entrySet()) { System.out.printf("\t%s: %s%n", entry.getKey(), entry.getValue()); } diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/AddEntitiesAndRunQuery.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/AddEntitiesAndRunQuery.java new file mode 100644 index 000000000000..f1e844c79b24 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/AddEntitiesAndRunQuery.java @@ -0,0 +1,84 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.datastore.snippets; + +import com.google.gcloud.datastore.Datastore; +import com.google.gcloud.datastore.DatastoreOptions; +import com.google.gcloud.datastore.Entity; +import com.google.gcloud.datastore.Key; +import com.google.gcloud.datastore.KeyFactory; +import com.google.gcloud.datastore.Query; +import com.google.gcloud.datastore.QueryResults; +import com.google.gcloud.datastore.StructuredQuery.PropertyFilter; + +/** + * A snippet for Google Cloud Datastore showing how to create and get entities. The snippet also + * shows how to run a query against Datastore. + */ +public class AddEntitiesAndRunQuery { + + public static void main(String... args) { + // Create datastore service object. + // By default, credentials are inferred from the runtime environment. + Datastore datastore = DatastoreOptions.defaultInstance().service(); + + // Add an entity to Datastore + KeyFactory keyFactory = datastore.newKeyFactory().kind("Person"); + Key key = keyFactory.newKey("john.doe@gmail.com"); + Entity entity = Entity.builder(key) + .set("name", "John Doe") + .set("age", 51) + .set("favorite_food", "pizza") + .build(); + datastore.put(entity); + + // Get an entity from Datastore + Entity johnEntity = datastore.get(key); + + // Add a couple more entities to make the query results more interesting + Key janeKey = keyFactory.newKey("jane.doe@gmail.com"); + Entity janeEntity = Entity.builder(janeKey) + .set("name", "Jane Doe") + .set("age", 44) + .set("favorite_food", "pizza") + .build(); + Key joeKey = keyFactory.newKey("joe.shmoe@gmail.com"); + Entity joeEntity = Entity.builder(joeKey) + .set("name", "Joe Shmoe") + .set("age", 27) + .set("favorite_food", "sushi") + .build(); + datastore.put(janeEntity, joeEntity); + + // Run a query + Query query = Query.entityQueryBuilder() + .kind("Person") + .filter(PropertyFilter.eq("favorite_food", "pizza")) + .build(); + QueryResults results = datastore.run(query); + while (results.hasNext()) { + Entity currentEntity = results.next(); + System.out.println(currentEntity.getString("name") + ", you're invited to a pizza party!"); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/CreateEntity.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/CreateEntity.java new file mode 100644 index 000000000000..3981162a2943 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/CreateEntity.java @@ -0,0 +1,48 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.datastore.snippets; + +import com.google.gcloud.datastore.Datastore; +import com.google.gcloud.datastore.DatastoreOptions; +import com.google.gcloud.datastore.DateTime; +import com.google.gcloud.datastore.Entity; +import com.google.gcloud.datastore.Key; +import com.google.gcloud.datastore.KeyFactory; + +/** + * A snippet for Google Cloud Datastore showing how to create an entity. + */ +public class CreateEntity { + + public static void main(String... args) { + Datastore datastore = DatastoreOptions.defaultInstance().service(); + KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind"); + Key key = keyFactory.newKey("keyName"); + Entity entity = Entity.builder(key) + .set("name", "John Doe") + .set("age", 30) + .set("access_time", DateTime.now()) + .build(); + datastore.put(entity); + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/UpdateEntity.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/UpdateEntity.java new file mode 100644 index 000000000000..cbc97f0784dd --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/UpdateEntity.java @@ -0,0 +1,50 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.datastore.snippets; + +import com.google.gcloud.datastore.Datastore; +import com.google.gcloud.datastore.DatastoreOptions; +import com.google.gcloud.datastore.DateTime; +import com.google.gcloud.datastore.Entity; +import com.google.gcloud.datastore.Key; +import com.google.gcloud.datastore.KeyFactory; + +/** + * A snippet for Google Cloud Datastore showing how to get an entity and update it if it exists. + */ +public class UpdateEntity { + + public static void main(String... args) { + Datastore datastore = DatastoreOptions.defaultInstance().service(); + KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind"); + Key key = keyFactory.newKey("keyName"); + Entity entity = datastore.get(key); + if (entity != null) { + System.out.println("Updating access_time for " + entity.getString("name")); + entity = Entity.builder(entity) + .set("access_time", DateTime.now()) + .build(); + datastore.update(entity); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java similarity index 97% rename from gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java rename to gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java index c1ba4e06cf7d..349c0eebe73d 100644 --- a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java @@ -14,9 +14,10 @@ * limitations under the License. */ -package com.google.gcloud.examples; +package com.google.gcloud.examples.resourcemanager; import com.google.common.base.Joiner; +import com.google.gcloud.resourcemanager.Project; import com.google.gcloud.resourcemanager.ProjectInfo; import com.google.gcloud.resourcemanager.ResourceManager; import com.google.gcloud.resourcemanager.ResourceManagerOptions; @@ -35,7 +36,7 @@ *
  • login using gcloud SDK - {@code gcloud auth login}.
  • *
  • compile using maven - {@code mvn compile}
  • *
  • run using maven - {@code mvn exec:java - * -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" + * -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" * -Dexec.args="[list | [create | delete | get] projectId]"}
  • * */ @@ -64,7 +65,7 @@ public void run(ResourceManager resourceManager, String... args) { labels.put(args[i], ""); } } - ProjectInfo project = + Project project = resourceManager.create(ProjectInfo.builder(projectId).labels(labels).build()); System.out.printf( "Successfully created project '%s': %s.%n", projectId, projectDetails(project)); diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/GetOrCreateProject.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/GetOrCreateProject.java new file mode 100644 index 000000000000..5a298107cc60 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/GetOrCreateProject.java @@ -0,0 +1,49 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.resourcemanager.snippets; + +import com.google.gcloud.resourcemanager.Project; +import com.google.gcloud.resourcemanager.ProjectInfo; +import com.google.gcloud.resourcemanager.ResourceManager; +import com.google.gcloud.resourcemanager.ResourceManagerOptions; + +/** + * A snippet for Google Cloud Resource Manager showing how to create a project if it does not exist. + */ +public class GetOrCreateProject { + + public static void main(String... args) { + // Create Resource Manager service object. + // By default, credentials are inferred from the runtime environment. + ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service(); + + String projectId = "my-globally-unique-project-id"; // Change to a unique project ID. + // Get a project from the server. + Project project = resourceManager.get(projectId); + if (project == null) { + // Create a project. + project = resourceManager.create(ProjectInfo.builder(projectId).build()); + } + System.out.println("Got project " + project.projectId() + " from the server."); + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java new file mode 100644 index 000000000000..b194de0815d5 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java @@ -0,0 +1,62 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.resourcemanager.snippets; + +import com.google.gcloud.resourcemanager.Project; +import com.google.gcloud.resourcemanager.ResourceManager; +import com.google.gcloud.resourcemanager.ResourceManagerOptions; + +import java.util.Iterator; + +/** + * A snippet for Google Cloud Resource Manager showing how to update a project and list all projects + * the user has permission to view. + */ +public class UpdateAndListProjects { + + public static void main(String... args) { + // Create Resource Manager service object + // By default, credentials are inferred from the runtime environment. + ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service(); + + // Get a project from the server + Project project = resourceManager.get("some-project-id"); // Use an existing project's ID + + // Update a project + if (project != null) { + Project newProject = project.toBuilder() + .addLabel("launch-status", "in-development") + .build() + .replace(); + System.out.println("Updated the labels of project " + newProject.projectId() + + " to be " + newProject.labels()); + } + + // List all the projects you have permission to view. + Iterator projectIterator = resourceManager.list().iterateAll(); + System.out.println("Projects I can view:"); + while (projectIterator.hasNext()) { + System.out.println(projectIterator.next().projectId()); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java similarity index 93% rename from gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java rename to gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java index e3bee626f49c..e73cfc427129 100644 --- a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java @@ -14,7 +14,7 @@ * limitations under the License. */ -package com.google.gcloud.examples; +package com.google.gcloud.examples.storage; import com.google.gcloud.AuthCredentials; import com.google.gcloud.AuthCredentials.ServiceAccountAuthCredentials; @@ -25,7 +25,6 @@ import com.google.gcloud.storage.BlobId; import com.google.gcloud.storage.BlobInfo; import com.google.gcloud.storage.Bucket; -import com.google.gcloud.storage.BucketInfo; import com.google.gcloud.storage.CopyWriter; import com.google.gcloud.storage.Storage; import com.google.gcloud.storage.Storage.ComposeRequest; @@ -66,7 +65,7 @@ *
  • login using gcloud SDK - {@code gcloud auth login}.
  • *
  • compile using maven - {@code mvn compile}
  • *
  • run using maven - - *
    {@code mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample"
    + * 
    {@code mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample"
      *  -Dexec.args="[]
      *  list [] |
      *  info [ []] |
    @@ -133,27 +132,27 @@ public void run(Storage storage, BlobId... blobIds) {
           if (blobIds.length == 1) {
             if (blobIds[0].name().isEmpty()) {
               // get Bucket
    -          Bucket bucket = Bucket.get(storage, blobIds[0].bucket());
    +          Bucket bucket = storage.get(blobIds[0].bucket());
               if (bucket == null) {
                 System.out.println("No such bucket");
                 return;
               }
    -          System.out.println("Bucket info: " + bucket.info());
    +          System.out.println("Bucket info: " + bucket);
             } else {
               // get Blob
    -          Blob blob = Blob.get(storage, blobIds[0]);
    +          Blob blob = storage.get(blobIds[0]);
               if (blob == null) {
                 System.out.println("No such object");
                 return;
               }
    -          System.out.println("Blob info: " + blob.info());
    +          System.out.println("Blob info: " + blob);
             }
           } else {
             // use batch to get multiple blobs.
    -        List blobs = Blob.get(storage, Arrays.asList(blobIds));
    +        List blobs = storage.get(blobIds);
             for (Blob blob : blobs) {
               if (blob != null) {
    -            System.out.println(blob.info());
    +            System.out.println(blob);
               }
             }
           }
    @@ -184,7 +183,7 @@ private static class DeleteAction extends BlobsAction {
         @Override
         public void run(Storage storage, BlobId... blobIds) {
           // use batch operation
    -      List deleteResults = Blob.delete(storage, blobIds);
    +      List deleteResults = storage.delete(blobIds);
           int index = 0;
           for (Boolean deleted : deleteResults) {
             if (deleted) {
    @@ -218,20 +217,20 @@ String parse(String... args) {
         public void run(Storage storage, String bucketName) {
           if (bucketName == null) {
             // list buckets
    -        Iterator bucketInfoIterator = storage.list().iterateAll();
    -        while (bucketInfoIterator.hasNext()) {
    -          System.out.println(bucketInfoIterator.next());
    +        Iterator bucketIterator = storage.list().iterateAll();
    +        while (bucketIterator.hasNext()) {
    +          System.out.println(bucketIterator.next());
             }
           } else {
             // list a bucket's blobs
    -        Bucket bucket = Bucket.get(storage, bucketName);
    +        Bucket bucket = storage.get(bucketName);
             if (bucket == null) {
               System.out.println("No such bucket");
               return;
             }
             Iterator blobIterator = bucket.list().iterateAll();
             while (blobIterator.hasNext()) {
    -          System.out.println(blobIterator.next().info());
    +          System.out.println(blobIterator.next());
             }
           }
         }
    @@ -257,8 +256,7 @@ private void run(Storage storage, Path uploadFrom, BlobInfo blobInfo) throws IOE
           if (Files.size(uploadFrom) > 1_000_000) {
             // When content is not available or large (1MB or more) it is recommended
             // to write it in chunks via the blob's channel writer.
    -        Blob blob = new Blob(storage, blobInfo);
    -        try (WriteChannel writer = blob.writer()) {
    +        try (WriteChannel writer = storage.writer(blobInfo)) {
               byte[] buffer = new byte[1024];
               try (InputStream input = Files.newInputStream(uploadFrom)) {
                 int limit;
    @@ -311,7 +309,7 @@ public void run(Storage storage, Tuple tuple) throws IOException {
         }
     
         private void run(Storage storage, BlobId blobId, Path downloadTo) throws IOException {
    -      Blob blob = Blob.get(storage, blobId);
    +      Blob blob = storage.get(blobId);
           if (blob == null) {
             System.out.println("No such object");
             return;
    @@ -320,7 +318,7 @@ private void run(Storage storage, BlobId blobId, Path downloadTo) throws IOExcep
           if (downloadTo != null) {
             writeTo = new PrintStream(new FileOutputStream(downloadTo.toFile()));
           }
    -      if (blob.info().size() < 1_000_000) {
    +      if (blob.size() < 1_000_000) {
             // Blob is small read all its content in one request
             byte[] content = blob.content();
             writeTo.write(content);
    @@ -438,13 +436,13 @@ public void run(Storage storage, Tuple> tuple)
         }
     
         private void run(Storage storage, BlobId blobId, Map metadata) {
    -      Blob blob = Blob.get(storage, blobId);
    +      Blob blob = storage.get(blobId);
           if (blob == null) {
             System.out.println("No such object");
             return;
           }
    -      Blob updateBlob = blob.update(blob.info().toBuilder().metadata(metadata).build());
    -      System.out.println("Updated " + updateBlob.info());
    +      Blob updateBlob = blob.toBuilder().metadata(metadata).build().update();
    +      System.out.println("Updated " + updateBlob);
         }
     
         @Override
    @@ -488,9 +486,8 @@ public void run(Storage storage, Tuple
           run(storage, tuple.x(), tuple.y());
         }
     
    -    private void run(Storage storage, ServiceAccountAuthCredentials cred, BlobInfo blobInfo)
    -        throws IOException {
    -      Blob blob = new Blob(storage, blobInfo);
    +    private void run(Storage storage, ServiceAccountAuthCredentials cred, BlobInfo blobInfo) {
    +      Blob blob = storage.get(blobInfo.blobId());
           System.out.println("Signed URL: "
               + blob.signUrl(1, TimeUnit.DAYS, SignUrlOption.serviceAccount(cred)));
         }
    diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateAndListBucketsAndBlobs.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateAndListBucketsAndBlobs.java
    new file mode 100644
    index 000000000000..435cc90b03d8
    --- /dev/null
    +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateAndListBucketsAndBlobs.java
    @@ -0,0 +1,70 @@
    +/*
    + * Copyright 2016 Google Inc. All Rights Reserved.
    + *
    + * Licensed under the Apache License, Version 2.0 (the "License");
    + * you may not use this file except in compliance with the License.
    + * You may obtain a copy of the License at
    + *
    + *       http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +/*
    + * EDITING INSTRUCTIONS
    + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in
    + * the project's READMEs and package-info.java.
    + */
    +
    +package com.google.gcloud.examples.storage.snippets;
    +
    +import static java.nio.charset.StandardCharsets.UTF_8;
    +
    +import com.google.gcloud.storage.Blob;
    +import com.google.gcloud.storage.Bucket;
    +import com.google.gcloud.storage.BucketInfo;
    +import com.google.gcloud.storage.Storage;
    +import com.google.gcloud.storage.StorageOptions;
    +
    +import java.util.Iterator;
    +
    +/**
    + * A snippet for Google Cloud Storage showing how to create a bucket and a blob in it. The snippet
    + * also shows how to get a blob's content, list buckets and list blobs.
    + */
    +public class CreateAndListBucketsAndBlobs {
    +
    +  public static void main(String... args) {
    +    // Create a service object
    +    // Credentials are inferred from the environment.
    +    Storage storage = StorageOptions.defaultInstance().service();
    +
    +    // Create a bucket
    +    String bucketName = "my_unique_bucket"; // Change this to something unique
    +    Bucket bucket = storage.create(BucketInfo.of(bucketName));
    +
    +    // Upload a blob to the newly created bucket
    +    Blob blob = bucket.create("my_blob_name", "a simple blob".getBytes(UTF_8), "text/plain");
    +
    +    // Read the blob content from the server
    +    String blobContent = new String(blob.content(), UTF_8);
    +
    +    // List all your buckets
    +    Iterator bucketIterator = storage.list().iterateAll();
    +    System.out.println("My buckets:");
    +    while (bucketIterator.hasNext()) {
    +      System.out.println(bucketIterator.next());
    +    }
    +
    +    // List the blobs in a particular bucket
    +    Iterator blobIterator = bucket.list().iterateAll();
    +    System.out.println("My blobs:");
    +    while (blobIterator.hasNext()) {
    +      System.out.println(blobIterator.next());
    +    }
    +  }
    +}
    diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateBlob.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateBlob.java
    new file mode 100644
    index 000000000000..2c1304a478ab
    --- /dev/null
    +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateBlob.java
    @@ -0,0 +1,44 @@
    +/*
    + * Copyright 2016 Google Inc. All Rights Reserved.
    + *
    + * Licensed under the Apache License, Version 2.0 (the "License");
    + * you may not use this file except in compliance with the License.
    + * You may obtain a copy of the License at
    + *
    + *       http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +/*
    + * EDITING INSTRUCTIONS
    + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in
    + * the project's READMEs and package-info.java.
    + */
    +
    +package com.google.gcloud.examples.storage.snippets;
    +
    +import static java.nio.charset.StandardCharsets.UTF_8;
    +
    +import com.google.gcloud.storage.Blob;
    +import com.google.gcloud.storage.BlobId;
    +import com.google.gcloud.storage.BlobInfo;
    +import com.google.gcloud.storage.Storage;
    +import com.google.gcloud.storage.StorageOptions;
    +
    +/**
    + * A snippet for Google Cloud Storage showing how to create a blob.
    + */
    +public class CreateBlob {
    +
    +  public static void main(String... args) {
    +    Storage storage = StorageOptions.defaultInstance().service();
    +    BlobId blobId = BlobId.of("bucket", "blob_name");
    +    BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
    +    Blob blob = storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
    +  }
    +}
    diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/UpdateBlob.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/UpdateBlob.java
    new file mode 100644
    index 000000000000..13290b201787
    --- /dev/null
    +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/UpdateBlob.java
    @@ -0,0 +1,53 @@
    +/*
    + * Copyright 2016 Google Inc. All Rights Reserved.
    + *
    + * Licensed under the Apache License, Version 2.0 (the "License");
    + * you may not use this file except in compliance with the License.
    + * You may obtain a copy of the License at
    + *
    + *       http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +/*
    + * EDITING INSTRUCTIONS
    + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in
    + * the project's READMEs and package-info.java.
    + */
    +
    +package com.google.gcloud.examples.storage.snippets;
    +
    +import static java.nio.charset.StandardCharsets.UTF_8;
    +
    +import com.google.gcloud.storage.Blob;
    +import com.google.gcloud.storage.BlobId;
    +import com.google.gcloud.storage.Storage;
    +import com.google.gcloud.storage.StorageOptions;
    +
    +import java.io.IOException;
    +import java.nio.ByteBuffer;
    +import java.nio.channels.WritableByteChannel;
    +
    +/**
    + * A snippet for Google Cloud Storage showing how to update the blob's content if the blob exists.
    + */
    +public class UpdateBlob {
    +
    +  public static void main(String... args) throws IOException {
    +    Storage storage = StorageOptions.defaultInstance().service();
    +    BlobId blobId = BlobId.of("bucket", "blob_name");
    +    Blob blob = storage.get(blobId);
    +    if (blob != null) {
    +      byte[] prevContent = blob.content();
    +      System.out.println(new String(prevContent, UTF_8));
    +      WritableByteChannel channel = blob.writer();
    +      channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
    +      channel.close();
    +    }
    +  }
    +}
    diff --git a/gcloud-java-resourcemanager/README.md b/gcloud-java-resourcemanager/README.md
    index d0e58af85e4e..cd48d6699311 100644
    --- a/gcloud-java-resourcemanager/README.md
    +++ b/gcloud-java-resourcemanager/README.md
    @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Resource Manager] (https://cloud.google.
     [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java)
     [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master)
     [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-resourcemanager.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-resourcemanager.svg)
    +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java)
    +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969)
     
     -  [Homepage] (https://googlecloudplatform.github.io/gcloud-java/)
     -  [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/resourcemanager/package-summary.html)
    @@ -20,21 +22,21 @@ If you are using Maven, add this to your pom.xml file
     
       com.google.gcloud
       gcloud-java-resourcemanager
    -  0.1.2
    +  0.1.4
     
     ```
     If you are using Gradle, add this to your dependencies
     ```Groovy
    -compile 'com.google.gcloud:gcloud-java-resourcemanager:0.1.2'
    +compile 'com.google.gcloud:gcloud-java-resourcemanager:0.1.4'
     ```
     If you are using SBT, add this to your dependencies
     ```Scala
    -libraryDependencies += "com.google.gcloud" % "gcloud-java-resourcemanager" % "0.1.2"
    +libraryDependencies += "com.google.gcloud" % "gcloud-java-resourcemanager" % "0.1.4"
     ```
     
     Example Application
     --------------------
    -[`ResourceManagerExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java) is a simple command line interface for the Cloud Resource Manager.  Read more about using the application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/ResourceManagerExample.html).
    +[`ResourceManagerExample`](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java) is a simple command line interface for the Cloud Resource Manager.  Read more about using the application on the [`ResourceManagerExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/resourcemanager/ResourceManagerExample.html).
     
     Authentication
     --------------
    @@ -70,7 +72,12 @@ You will need to set up the local development environment by [installing the Goo
     You'll need to obtain the `gcloud-java-resourcemanager` library.  See the [Quickstart](#quickstart) section to add `gcloud-java-resourcemanager` as a dependency in your code.
     
     #### Creating an authorized service object
    -To make authenticated requests to Google Cloud Resource Manager, you must create a service object with Google Cloud SDK credentials. You can then make API calls by calling methods on the Resource Manager service object. The simplest way to authenticate is to use [Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials). These credentials are automatically inferred from your environment, so you only need the following code to create your service object:
    +To make authenticated requests to Google Cloud Resource Manager, you must create a service object
    +with Google Cloud SDK credentials. You can then make API calls by calling methods on the Resource
    +Manager service object. The simplest way to authenticate is to use
    +[Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials).
    +These credentials are automatically inferred from your environment, so you only need the following
    +code to create your service object:
     
     ```java
     import com.google.gcloud.resourcemanager.ResourceManager;
    @@ -79,43 +86,68 @@ import com.google.gcloud.resourcemanager.ResourceManagerOptions;
     ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service();
     ```
     
    -#### Creating a project
    -All you need to create a project is a globally unique project ID.  You can also optionally attach a non-unique name and labels to your project. Read more about naming guidelines for project IDs, names, and labels [here](https://cloud.google.com/resource-manager/reference/rest/v1beta1/projects). To create a project, add the following import at the top of your file:
    +#### Getting a specific project
    +You can load a project if you know it's project ID and have read permissions to the project.
    +To get a project, add the following import at the top of your file:
     
     ```java
    -import com.google.gcloud.resourcemanager.ProjectInfo;
    +import com.google.gcloud.resourcemanager.Project;
     ```
     
    -Then add the following code to create a project (be sure to change `myProjectId` to your own unique project ID).
    +Then use the following code to get the project:
     
     ```java
    -String myProjectId = "my-globally-unique-project-id"; // Change to a unique project ID.
    -ProjectInfo myProject = resourceManager.create(ProjectInfo.builder(myProjectId).build());
    +String projectId = "my-globally-unique-project-id"; // Change to a unique project ID
    +Project project = resourceManager.get(projectId);
     ```
     
    -Note that the return value from `create` is a `ProjectInfo` that includes additional read-only information, like creation time, project number, and lifecycle state. Read more about these fields on the [Projects page](https://cloud.google.com/resource-manager/reference/rest/v1beta1/projects).
    +#### Creating a project
    +All you need to create a project is a globally unique project ID. You can also optionally attach a
    +non-unique name and labels to your project. Read more about naming guidelines for project IDs,
    +names, and labels [here](https://cloud.google.com/resource-manager/reference/rest/v1beta1/projects).
    +To create a project, add the following imports at the top of your file:
     
    -#### Getting a specific project
    -You can load a project if you know it's project ID and have read permissions to the project. For example, to get the project we just created we can do the following:
    +```java
    +import com.google.gcloud.resourcemanager.Project;
    +import com.google.gcloud.resourcemanager.ProjectInfo;
    +```
    +
    +Then add the following code to create a project (be sure to change `projectId` to your own unique
    +project ID).
     
     ```java
    -ProjectInfo projectFromServer = resourceManager.get(myProjectId);
    +String projectId = "my-globally-unique-project-id"; // Change to a unique project ID
    +Project project = resourceManager.create(ProjectInfo.builder(projectId).build());
     ```
     
    -#### Editing a project
    -To edit a project, create a new `ProjectInfo` object and pass it in to the `ResourceManager.replace` method.
    +Note that the return value from `create` is a `Project` that includes additional read-only
    +information, like creation time, project number, and lifecycle state. Read more about these fields
    +on the [Projects page](https://cloud.google.com/resource-manager/reference/rest/v1beta1/projects).
    +`Project`, a subclass of `ProjectInfo`, adds a layer of service-related functionality over
    +`ProjectInfo`.
     
    -For example, to add a label for the newly created project to denote that it's launch status is "in development", add the following code:
    +#### Editing a project
    +To edit a project, create a new `ProjectInfo` object and pass it in to the `Project.replace` method.
    +For example, to add a label to a project to denote that it's launch status is "in development", add
    +the following code:
     
     ```java
    -ProjectInfo newProjectInfo = resourceManager.replace(projectFromServer.toBuilder()
    -    .addLabel("launch-status", "in-development").build());
    +Project newProject = project.toBuilder()
    +    .addLabel("launch-status", "in-development")
    +    .build()
    +    .replace();
     ```
     
    -Note that the values of the project you pass in to `replace` overwrite the server's values for non-read-only fields, namely `projectName` and `labels`. For example, if you create a project with `projectName` "some-project-name" and subsequently call replace using a `ProjectInfo` object that didn't set the `projectName`, then the server will unset the project's name. The server ignores any attempted changes to the read-only fields `projectNumber`, `lifecycleState`, and `createTime`. The `projectId` cannot change.
    +Note that the values of the project you pass in to `replace` overwrite the server's values for
    +non-read-only fields, namely `projectName` and `labels`. For example, if you create a project with
    +`projectName` "some-project-name" and subsequently call replace using a `ProjectInfo` object that
    +didn't set the `projectName`, then the server will unset the project's name. The server ignores any
    +attempted changes to the read-only fields `projectNumber`, `lifecycleState`, and `createTime`.
    +The `projectId` cannot change.
     
     #### Listing all projects
    -Suppose that we want a list of all projects for which we have read permissions. Add the following import:
    +Suppose that we want a list of all projects for which we have read permissions. Add the following
    +import:
     
     ```java
     import java.util.Iterator;
    @@ -124,7 +156,7 @@ import java.util.Iterator;
     Then add the following code to print a list of projects you can view:
     
     ```java
    -Iterator projectIterator = resourceManager.list().iterateAll();
    +Iterator projectIterator = resourceManager.list().iterateAll();
     System.out.println("Projects I can view:");
     while (projectIterator.hasNext()) {
       System.out.println(projectIterator.next().projectId());
    @@ -133,45 +165,15 @@ while (projectIterator.hasNext()) {
     
     #### Complete source code
     
    -Here we put together all the code shown above into one program.  This program assumes that you are running from your own desktop and used the Google Cloud SDK to authenticate yourself.
    +We put together all the code shown above into two programs. Both programs assume that you are
    +running from your own desktop and used the Google Cloud SDK to authenticate yourself.
     
    -```java
    -import com.google.gcloud.resourcemanager.ProjectInfo;
    -import com.google.gcloud.resourcemanager.ResourceManager;
    -import com.google.gcloud.resourcemanager.ResourceManagerOptions;
    +The first program creates a project if it does not exist. Complete source code can be found at
    +[GetOrCreateProject.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/GetOrCreateProject.java).
     
    -import java.util.Iterator;
    -
    -public class GcloudJavaResourceManagerExample {
    -
    -  public static void main(String[] args) {
    -    // Create Resource Manager service object.
    -    // By default, credentials are inferred from the runtime environment.
    -    ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service();
    -
    -    // Create a project.
    -    String myProjectId = "my-globally-unique-project-id"; // Change to a unique project ID.
    -    ProjectInfo myProject = resourceManager.create(ProjectInfo.builder(myProjectId).build());
    -
    -    // Get a project from the server.
    -    ProjectInfo projectFromServer = resourceManager.get(myProjectId);
    -    System.out.println("Got project " + projectFromServer.projectId() + " from the server.");
    -
    -    // Update a project
    -    ProjectInfo newProjectInfo = resourceManager.replace(myProject.toBuilder()
    -        .addLabel("launch-status", "in-development").build());
    -    System.out.println("Updated the labels of project " + newProjectInfo.projectId()
    -        + " to be " + newProjectInfo.labels());
    -
    -    // List all the projects you have permission to view.
    -    Iterator projectIterator = resourceManager.list().iterateAll();
    -    System.out.println("Projects I can view:");
    -    while (projectIterator.hasNext()) {
    -      System.out.println(projectIterator.next().projectId());
    -    }
    -  }
    -}
    -```
    +The second program updates a project if it exists and lists all projects the user has permission to
    +view. Complete source code can be found at
    +[UpdateAndListProjects.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java).
     
     Java Versions
     -------------
    diff --git a/gcloud-java-resourcemanager/pom.xml b/gcloud-java-resourcemanager/pom.xml
    index 5101a963cf42..40a865f4db68 100644
    --- a/gcloud-java-resourcemanager/pom.xml
    +++ b/gcloud-java-resourcemanager/pom.xml
    @@ -1,7 +1,6 @@
     
     
       4.0.0
    -  com.google.gcloud
       gcloud-java-resourcemanager
       jar
       GCloud Java resource manager
    @@ -11,7 +10,7 @@
       
         com.google.gcloud
         gcloud-java-pom
    -    0.1.3-SNAPSHOT
    +    0.1.5-SNAPSHOT
       
       
         gcloud-java-resourcemanager
    @@ -25,7 +24,7 @@
         
           com.google.apis
           google-api-services-cloudresourcemanager
    -      v1beta1-rev6-1.19.0
    +      v1beta1-rev10-1.21.0
           compile
           
             
    @@ -43,7 +42,7 @@
         
           org.easymock
           easymock
    -      3.3
    +      3.4
           test
         
       
    diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Policy.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Policy.java
    new file mode 100644
    index 000000000000..0d7118dcbbd7
    --- /dev/null
    +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Policy.java
    @@ -0,0 +1,164 @@
    +/*
    + * Copyright 2016 Google Inc. All Rights Reserved.
    + *
    + * Licensed under the Apache License, Version 2.0 (the "License");
    + * you may not use this file except in compliance with the License.
    + * You may obtain a copy of the License at
    + *
    + *       http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package com.google.gcloud.resourcemanager;
    +
    +import com.google.common.annotations.VisibleForTesting;
    +import com.google.common.base.CaseFormat;
    +import com.google.common.base.Function;
    +import com.google.common.collect.ImmutableSet;
    +import com.google.common.collect.Lists;
    +import com.google.gcloud.IamPolicy;
    +import com.google.gcloud.Identity;
    +
    +import java.util.ArrayList;
    +import java.util.HashMap;
    +import java.util.LinkedList;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.Set;
    +
    +/**
    + * An Identity and Access Management (IAM) policy for a project. IAM policies are used to specify
    + * access settings for Cloud Platform resources. A policy is a map of bindings. A binding assigns
    + * a set of identities to a role, where the identities can be user accounts, Google groups, Google
    + * domains, and service accounts. A role is a named list of permissions defined by IAM. Policies set
    + * at the project level control access both to the project and to resources associated with the
    + * project.
    + *
    + * @see Policy
    + */
    +public class Policy extends IamPolicy {
    +
    +  private static final long serialVersionUID = -5573557282693961850L;
    +
    +  /**
    +   * Represents legacy roles in an IAM Policy.
    +   */
    +  public enum Role {
    +
    +    /**
    +     * Permissions for read-only actions that preserve state.
    +     */
    +    VIEWER("roles/viewer"),
    +
    +    /**
    +     * All viewer permissions and permissions for actions that modify state.
    +     */
    +    EDITOR("roles/editor"),
    +
    +    /**
    +     * All editor permissions and permissions for the following actions:
    +     * 
      + *
    • Manage access control for a resource. + *
    • Set up billing (for a project). + *
    + */ + OWNER("roles/owner"); + + private String strValue; + + private Role(String strValue) { + this.strValue = strValue; + } + + String strValue() { + return strValue; + } + + static Role fromStr(String roleStr) { + return Role.valueOf(CaseFormat.LOWER_CAMEL.to( + CaseFormat.UPPER_UNDERSCORE, roleStr.substring("roles/".length()))); + } + } + + /** + * Builder for an IAM Policy. + */ + public static class Builder extends IamPolicy.Builder { + + private Builder() {} + + @VisibleForTesting + Builder(Map> bindings, String etag, Integer version) { + bindings(bindings).etag(etag).version(version); + } + + @Override + public Policy build() { + return new Policy(this); + } + } + + private Policy(Builder builder) { + super(builder); + } + + public static Builder builder() { + return new Builder(); + } + + @Override + public Builder toBuilder() { + return new Builder(bindings(), etag(), version()); + } + + com.google.api.services.cloudresourcemanager.model.Policy toPb() { + com.google.api.services.cloudresourcemanager.model.Policy policyPb = + new com.google.api.services.cloudresourcemanager.model.Policy(); + List bindingPbList = + new LinkedList<>(); + for (Map.Entry> binding : bindings().entrySet()) { + com.google.api.services.cloudresourcemanager.model.Binding bindingPb = + new com.google.api.services.cloudresourcemanager.model.Binding(); + bindingPb.setRole(binding.getKey().strValue()); + bindingPb.setMembers( + Lists.transform( + new ArrayList<>(binding.getValue()), + new Function() { + @Override + public String apply(Identity identity) { + return identity.strValue(); + } + })); + bindingPbList.add(bindingPb); + } + policyPb.setBindings(bindingPbList); + policyPb.setEtag(etag()); + policyPb.setVersion(version()); + return policyPb; + } + + static Policy fromPb( + com.google.api.services.cloudresourcemanager.model.Policy policyPb) { + Map> bindings = new HashMap<>(); + for (com.google.api.services.cloudresourcemanager.model.Binding bindingPb : + policyPb.getBindings()) { + bindings.put( + Role.fromStr(bindingPb.getRole()), + ImmutableSet.copyOf( + Lists.transform( + bindingPb.getMembers(), + new Function() { + @Override + public Identity apply(String identityPb) { + return Identity.valueOf(identityPb); + } + }))); + } + return new Policy.Builder(bindings, policyPb.getEtag(), policyPb.getVersion()).build(); + } +} diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Project.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Project.java index e329e1aa3714..4d12a31274c0 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Project.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Project.java @@ -18,43 +18,108 @@ import static com.google.common.base.Preconditions.checkNotNull; +import java.io.IOException; +import java.io.ObjectInputStream; +import java.util.Map; +import java.util.Objects; + /** * A Google Cloud Resource Manager project object. * *

    A Project is a high-level Google Cloud Platform entity. It is a container for ACLs, APIs, * AppEngine Apps, VMs, and other Google Cloud Platform resources. This class' member variables are - * immutable. Methods that change or update the underlying Project information return a new Project - * instance. + * immutable. Methods that change or update the underlying Project information return a new Project + * instance. {@code Project} adds a layer of service-related functionality over {@link ProjectInfo}. */ -public class Project { +public class Project extends ProjectInfo { - private final ResourceManager resourceManager; - private final ProjectInfo info; + private static final long serialVersionUID = 6767630161335155133L; - /** - * Constructs a Project object that contains the ProjectInfo given. - */ - public Project(ResourceManager resourceManager, ProjectInfo projectInfo) { - this.resourceManager = checkNotNull(resourceManager); - this.info = checkNotNull(projectInfo); - } + private final ResourceManagerOptions options; + private transient ResourceManager resourceManager; /** - * Constructs a Project object that contains project information got from the server. - * - * @return Project object containing the project's metadata or {@code null} if not found - * @throws ResourceManagerException upon failure + * Builder for {@code Project}. */ - public static Project get(ResourceManager resourceManager, String projectId) { - ProjectInfo projectInfo = resourceManager.get(projectId); - return projectInfo != null ? new Project(resourceManager, projectInfo) : null; + public static class Builder extends ProjectInfo.Builder { + private final ResourceManager resourceManager; + private final ProjectInfo.BuilderImpl infoBuilder; + + Builder(Project project) { + this.resourceManager = project.resourceManager; + this.infoBuilder = new ProjectInfo.BuilderImpl(project); + } + + @Override + public Builder name(String name) { + infoBuilder.name(name); + return this; + } + + @Override + public Builder projectId(String projectId) { + infoBuilder.projectId(projectId); + return this; + } + + @Override + public Builder addLabel(String key, String value) { + infoBuilder.addLabel(key, value); + return this; + } + + @Override + public Builder removeLabel(String key) { + infoBuilder.removeLabel(key); + return this; + } + + @Override + public Builder clearLabels() { + infoBuilder.clearLabels(); + return this; + } + + @Override + public Builder labels(Map labels) { + infoBuilder.labels(labels); + return this; + } + + @Override + Builder projectNumber(Long projectNumber) { + infoBuilder.projectNumber(projectNumber); + return this; + } + + @Override + Builder state(State state) { + infoBuilder.state(state); + return this; + } + + @Override + Builder createTimeMillis(Long createTimeMillis) { + infoBuilder.createTimeMillis(createTimeMillis); + return this; + } + + @Override + Builder parent(ResourceId parent) { + infoBuilder.parent(parent); + return this; + } + + @Override + public Project build() { + return new Project(resourceManager, infoBuilder); + } } - /** - * Returns the {@link ProjectInfo} object associated with this Project. - */ - public ProjectInfo info() { - return info; + Project(ResourceManager resourceManager, ProjectInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.resourceManager = checkNotNull(resourceManager); + this.options = resourceManager.options(); } /** @@ -65,14 +130,14 @@ public ResourceManager resourceManager() { } /** - * Fetches the current project's latest information. Returns {@code null} if the job does not + * Fetches the project's latest information. Returns {@code null} if the project does not * exist. * * @return Project containing the project's updated metadata or {@code null} if not found * @throws ResourceManagerException upon failure */ public Project reload() { - return Project.get(resourceManager, info.projectId()); + return resourceManager.get(projectId()); } /** @@ -98,7 +163,7 @@ public Project reload() { * @throws ResourceManagerException upon failure */ public void delete() { - resourceManager.delete(info.projectId()); + resourceManager.delete(projectId()); } /** @@ -115,21 +180,48 @@ public void delete() { * @throws ResourceManagerException upon failure (including when the project can't be restored) */ public void undelete() { - resourceManager.undelete(info.projectId()); + resourceManager.undelete(projectId()); } /** - * Replaces the attributes of the project. + * Replaces the attributes of the project with the attributes of this project. * *

    The caller must have modify permissions for this project. * * @see Cloud * Resource Manager update - * @return the ProjectInfo representing the new project metadata + * @return the Project representing the new project metadata * @throws ResourceManagerException upon failure */ - public Project replace(ProjectInfo projectInfo) { - return new Project(resourceManager, resourceManager.replace(checkNotNull(projectInfo))); + public Project replace() { + return resourceManager.replace(this); + } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof Project && Objects.equals(toPb(), ((Project) obj).toPb()) + && Objects.equals(options, ((Project) obj).options); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); + } + + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.resourceManager = options.service(); + } + + static Project fromPb(ResourceManager resourceManager, + com.google.api.services.cloudresourcemanager.model.Project answer) { + ProjectInfo info = ProjectInfo.fromPb(answer); + return new Project(resourceManager, new ProjectInfo.BuilderImpl(info)); } } diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ProjectInfo.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ProjectInfo.java index 2cb8a2d93ad2..260e8a8e2f26 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ProjectInfo.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ProjectInfo.java @@ -115,7 +115,69 @@ static ResourceId fromPb( } } - public static class Builder { + /** + * Builder for {@code ProjectInfo}. + */ + public abstract static class Builder { + + /** + * Set the user-assigned name of the project. + * + *

    This field is optional and can remain unset. Allowed characters are: lowercase and + * uppercase letters, numbers, hyphen, single-quote, double-quote, space, and exclamation point. + * This field can be changed after project creation. + */ + public abstract Builder name(String name); + + /** + * Set the unique, user-assigned ID of the project. + * + *

    The ID must be 6 to 30 lowercase letters, digits, or hyphens. It must start with a letter. + * Trailing hyphens are prohibited. This field cannot be changed after the server creates the + * project. + */ + public abstract Builder projectId(String projectId); + + /** + * Add a label associated with this project. + * + *

    See {@link #labels} for label restrictions. + */ + public abstract Builder addLabel(String key, String value); + + /** + * Remove a label associated with this project. + */ + public abstract Builder removeLabel(String key); + + /** + * Clear the labels associated with this project. + */ + public abstract Builder clearLabels(); + + /** + * Set the labels associated with this project. + * + *

    Label keys must be between 1 and 63 characters long and must conform to the following + * regular expression: [a-z]([-a-z0-9]*[a-z0-9])?. Label values must be between 0 and 63 + * characters long and must conform to the regular expression ([a-z]([-a-z0-9]*[a-z0-9])?)?. No + * more than 256 labels can be associated with a given resource. This field can be changed after + * project creation. + */ + public abstract Builder labels(Map labels); + + abstract Builder projectNumber(Long projectNumber); + + abstract Builder state(State state); + + abstract Builder createTimeMillis(Long createTimeMillis); + + abstract Builder parent(ResourceId parent); + + public abstract ProjectInfo build(); + } + + static class BuilderImpl extends Builder { private String name; private String projectId; @@ -125,10 +187,11 @@ public static class Builder { private Long createTimeMillis; private ResourceId parent; - private Builder() { + BuilderImpl(String projectId) { + this.projectId = projectId; } - Builder(ProjectInfo info) { + BuilderImpl(ProjectInfo info) { this.name = info.name; this.projectId = info.projectId; this.labels.putAll(info.labels); @@ -138,96 +201,73 @@ private Builder() { this.parent = info.parent; } - /** - * Set the user-assigned name of the project. - * - *

    This field is optional and can remain unset. Allowed characters are: lowercase and - * uppercase letters, numbers, hyphen, single-quote, double-quote, space, and exclamation point. - * This field can be changed after project creation. - */ + @Override public Builder name(String name) { this.name = firstNonNull(name, Data.nullOf(String.class)); return this; } - /** - * Set the unique, user-assigned ID of the project. - * - *

    The ID must be 6 to 30 lowercase letters, digits, or hyphens. It must start with a letter. - * Trailing hyphens are prohibited. This field cannot be changed after the server creates the - * project. - */ + @Override public Builder projectId(String projectId) { this.projectId = checkNotNull(projectId); return this; } - /** - * Add a label associated with this project. - * - *

    See {@link #labels} for label restrictions. - */ + @Override public Builder addLabel(String key, String value) { this.labels.put(key, value); return this; } - /** - * Remove a label associated with this project. - */ + @Override public Builder removeLabel(String key) { this.labels.remove(key); return this; } - /** - * Clear the labels associated with this project. - */ + @Override public Builder clearLabels() { this.labels.clear(); return this; } - /** - * Set the labels associated with this project. - * - *

    Label keys must be between 1 and 63 characters long and must conform to the following - * regular expression: [a-z]([-a-z0-9]*[a-z0-9])?. Label values must be between 0 and 63 - * characters long and must conform to the regular expression ([a-z]([-a-z0-9]*[a-z0-9])?)?. No - * more than 256 labels can be associated with a given resource. This field can be changed after - * project creation. - */ + @Override public Builder labels(Map labels) { this.labels = Maps.newHashMap(checkNotNull(labels)); return this; } + @Override Builder projectNumber(Long projectNumber) { this.projectNumber = projectNumber; return this; } + @Override Builder state(State state) { this.state = state; return this; } + @Override Builder createTimeMillis(Long createTimeMillis) { this.createTimeMillis = createTimeMillis; return this; } + @Override Builder parent(ResourceId parent) { this.parent = parent; return this; } + @Override public ProjectInfo build() { return new ProjectInfo(this); } } - ProjectInfo(Builder builder) { + ProjectInfo(BuilderImpl builder) { this.name = builder.name; this.projectId = builder.projectId; this.labels = ImmutableMap.copyOf(builder.labels); @@ -296,7 +336,8 @@ public Long createTimeMillis() { @Override public boolean equals(Object obj) { - return obj instanceof ProjectInfo && Objects.equals(toPb(), ((ProjectInfo) obj).toPb()); + return obj != null && obj.getClass().equals(ProjectInfo.class) + && Objects.equals(toPb(), ((ProjectInfo) obj).toPb()); } @Override @@ -305,11 +346,11 @@ public int hashCode() { } public static Builder builder(String id) { - return new Builder().projectId(id); + return new BuilderImpl(id); } public Builder toBuilder() { - return new Builder(this); + return new BuilderImpl(this); } com.google.api.services.cloudresourcemanager.model.Project toPb() { diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManager.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManager.java index 3d658d18d28a..3d27f2a33ac8 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManager.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManager.java @@ -147,8 +147,6 @@ public static ProjectListOption pageToken(String pageToken) { * *

    The server can return fewer projects than requested. When there are more results than the * page size, the server will return a page token that can be used to fetch other results. - * Note: pagination is not yet supported; the server currently ignores this field and returns - * all results. */ public static ProjectListOption pageSize(int pageSize) { return new ProjectListOption(ResourceManagerRpc.Option.PAGE_SIZE, pageSize); @@ -164,7 +162,7 @@ public static ProjectListOption pageSize(int pageSize) { */ public static ProjectListOption fields(ProjectField... fields) { StringBuilder builder = new StringBuilder(); - builder.append("projects(").append(ProjectField.selector(fields)).append(")"); + builder.append("projects(").append(ProjectField.selector(fields)).append("),nextPageToken"); return new ProjectListOption(ResourceManagerRpc.Option.FIELDS, builder.toString()); } } @@ -179,12 +177,12 @@ public static ProjectListOption fields(ProjectField... fields) { * @see Cloud * Resource Manager create - * @return ProjectInfo object representing the new project's metadata. The returned object will + * @return Project object representing the new project's metadata. The returned object will * include the following read-only fields supplied by the server: project number, lifecycle * state, and creation time. * @throws ResourceManagerException upon failure */ - ProjectInfo create(ProjectInfo project); + Project create(ProjectInfo project); /** * Marks the project identified by the specified project ID for deletion. @@ -221,23 +219,22 @@ public static ProjectListOption fields(ProjectField... fields) { * Resource Manager get * @throws ResourceManagerException upon failure */ - ProjectInfo get(String projectId, ProjectGetOption... options); + Project get(String projectId, ProjectGetOption... options); /** * Lists the projects visible to the current user. * *

    This method returns projects in an unspecified order. New projects do not necessarily appear * at the end of the list. Use {@link ProjectListOption} to filter this list, set page size, and - * set page tokens. Note that pagination is currently not implemented by the Cloud Resource - * Manager API. + * set page tokens. * * @see Cloud * Resource Manager list - * @return {@code Page}, a page of projects + * @return {@code Page}, a page of projects * @throws ResourceManagerException upon failure */ - Page list(ProjectListOption... options); + Page list(ProjectListOption... options); /** * Replaces the attributes of the project. @@ -247,10 +244,10 @@ public static ProjectListOption fields(ProjectField... fields) { * @see Cloud * Resource Manager update - * @return the ProjectInfo representing the new project metadata + * @return the Project representing the new project metadata * @throws ResourceManagerException upon failure */ - ProjectInfo replace(ProjectInfo newProject); + Project replace(ProjectInfo newProject); /** * Restores the project identified by the specified project ID. diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManagerImpl.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManagerImpl.java index 22f2b350d2f3..4176b4e610ba 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManagerImpl.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManagerImpl.java @@ -46,9 +46,9 @@ final class ResourceManagerImpl } @Override - public ProjectInfo create(final ProjectInfo project) { + public Project create(final ProjectInfo project) { try { - return ProjectInfo.fromPb(runWithRetries( + return Project.fromPb(this, runWithRetries( new Callable() { @Override public com.google.api.services.cloudresourcemanager.model.Project call() { @@ -76,7 +76,7 @@ public Void call() { } @Override - public ProjectInfo get(final String projectId, ProjectGetOption... options) { + public Project get(final String projectId, ProjectGetOption... options) { final Map optionsMap = optionMap(options); try { com.google.api.services.cloudresourcemanager.model.Project answer = runWithRetries( @@ -86,13 +86,13 @@ public com.google.api.services.cloudresourcemanager.model.Project call() { return resourceManagerRpc.get(projectId, optionsMap); } }, options().retryParams(), EXCEPTION_HANDLER); - return answer == null ? null : ProjectInfo.fromPb(answer); + return answer == null ? null : Project.fromPb(this, answer); } catch (RetryHelperException e) { throw ResourceManagerException.translateAndThrow(e); } } - private static class ProjectPageFetcher implements NextPageFetcher { + private static class ProjectPageFetcher implements NextPageFetcher { private static final long serialVersionUID = 2158209410430566961L; private final Map requestOptions; @@ -106,17 +106,17 @@ private static class ProjectPageFetcher implements NextPageFetcher } @Override - public Page nextPage() { + public Page nextPage() { return listProjects(serviceOptions, requestOptions); } } @Override - public Page list(ProjectListOption... options) { + public Page list(ProjectListOption... options) { return listProjects(options(), optionMap(options)); } - private static Page listProjects(final ResourceManagerOptions serviceOptions, + private static Page listProjects(final ResourceManagerOptions serviceOptions, final Map optionsMap) { try { Tuple> result = @@ -130,16 +130,18 @@ Iterable> call() { }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); - Iterable projects = + Iterable projects = result.y() == null - ? ImmutableList.of() : Iterables.transform( + ? ImmutableList.of() : Iterables.transform( result.y(), new Function() { + Project>() { @Override - public ProjectInfo apply( + public Project apply( com.google.api.services.cloudresourcemanager.model.Project projectPb) { - return ProjectInfo.fromPb(projectPb); + return new Project( + serviceOptions.service(), + new ProjectInfo.BuilderImpl(ProjectInfo.fromPb(projectPb))); } }); return new PageImpl<>( @@ -150,9 +152,9 @@ public ProjectInfo apply( } @Override - public ProjectInfo replace(final ProjectInfo newProject) { + public Project replace(final ProjectInfo newProject) { try { - return ProjectInfo.fromPb(runWithRetries( + return Project.fromPb(this, runWithRetries( new Callable() { @Override public com.google.api.services.cloudresourcemanager.model.Project call() { diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/package-info.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/package-info.java index b8687fbf1314..d1794447e9fb 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/package-info.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/package-info.java @@ -17,19 +17,40 @@ /** * A client to Google Cloud Resource Manager. * - *

    Here's a simple usage example for using gcloud-java-resourcemanager: + *

    Here's a simple usage example for using gcloud-java from App/Compute Engine. This example + * creates a project if it does not exist. For the complete source code see + * + * GetOrCreateProject.java. *

     {@code
      * ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service();
    - * String myProjectId = "my-globally-unique-project-id"; // Change to a unique project ID.
    - * ProjectInfo myProject = resourceManager.create(ProjectInfo.builder(myProjectId).build());
    - * ProjectInfo newProjectInfo = resourceManager.replace(myProject.toBuilder()
    - *     .addLabel("launch-status", "in-development").build());
    - * Iterator projectIterator = resourceManager.list().iterateAll();
    + * String projectId = "my-globally-unique-project-id"; // Change to a unique project ID.
    + * Project project = resourceManager.get(projectId);
    + * if (project == null) {
    + *   project = resourceManager.create(ProjectInfo.builder(projectId).build());
    + * }
    + * System.out.println("Got project " + project.projectId() + " from the server.");
    + * }
    + *

    + * This second example shows how to update a project if it exists and list all projects the user has + * permission to view. For the complete source code see + * + * UpdateAndListProjects.java. + *

     {@code
    + * ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service();
    + * Project project = resourceManager.get("some-project-id"); // Use an existing project's ID
    + * if (project != null) {
    + *   Project newProject = project.toBuilder()
    + *       .addLabel("launch-status", "in-development")
    + *       .build()
    + *       .replace();
    + *   System.out.println("Updated the labels of project " + newProject.projectId()
    + *       + " to be " + newProject.labels());
    + * }
    + * Iterator projectIterator = resourceManager.list().iterateAll();
      * System.out.println("Projects I can view:");
      * while (projectIterator.hasNext()) {
      *   System.out.println(projectIterator.next().projectId());
      * }}
    - * *

    Remember that you must authenticate using the Google Cloud SDK. See more about * providing * credentials here. diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/testing/LocalResourceManagerHelper.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/testing/LocalResourceManagerHelper.java index 25c763276b3b..a077eb6144a5 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/testing/LocalResourceManagerHelper.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/testing/LocalResourceManagerHelper.java @@ -34,9 +34,11 @@ import java.util.Map; import java.util.Random; import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentSkipListMap; import java.util.logging.Level; import java.util.logging.Logger; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import java.util.zip.GZIPInputStream; /** @@ -56,6 +58,9 @@ public class LocalResourceManagerHelper { private static final URI BASE_CONTEXT; private static final Set SUPPORTED_COMPRESSION_ENCODINGS = ImmutableSet.of("gzip", "x-gzip"); + private static final Pattern LIST_FIELDS_PATTERN = + Pattern.compile("(.*?)projects\\((.*?)\\)(.*?)"); + private static final String[] NO_FIELDS = {}; static { try { @@ -71,7 +76,7 @@ public class LocalResourceManagerHelper { ImmutableSet.of('-', '\'', '"', ' ', '!'); private final HttpServer server; - private final ConcurrentHashMap projects = new ConcurrentHashMap<>(); + private final ConcurrentSkipListMap projects = new ConcurrentSkipListMap<>(); private final int port; private static class Response { @@ -228,7 +233,7 @@ private static String[] parseFields(String query) { return null; } - private static Map parseListOptions(String query) { + private static Map parseListOptions(String query) throws IOException { Map options = new HashMap<>(); if (query != null) { String[] args = query.split("&"); @@ -236,19 +241,28 @@ private static Map parseListOptions(String query) { String[] argEntry = arg.split("="); switch (argEntry[0]) { case "fields": - // List fields are in the form "projects(field1, field2, ...)" - options.put( - "fields", - argEntry[1].substring("projects(".length(), argEntry[1].length() - 1).split(",")); + // List fields are in the form "projects(field1, field2, ...),nextPageToken" + Matcher matcher = LIST_FIELDS_PATTERN.matcher(argEntry[1]); + if (matcher.matches()) { + options.put("projectFields", matcher.group(2).split(",")); + options.put("listFields", (matcher.group(1) + matcher.group(3)).split(",")); + } else { + options.put("projectFields", NO_FIELDS); + options.put("listFields", argEntry[1].split(",")); + } break; case "filter": options.put("filter", argEntry[1].split(" ")); break; case "pageToken": - // support pageToken when Cloud Resource Manager supports this (#421) + options.put("pageToken", argEntry[1]); break; case "pageSize": - // support pageSize when Cloud Resource Manager supports this (#421) + int pageSize = Integer.valueOf(argEntry[1]); + if (pageSize < 1) { + throw new IOException("Page size must be greater than 0."); + } + options.put("pageSize", pageSize); break; } } @@ -353,28 +367,55 @@ Response get(String projectId, String[] fields) { } Response list(Map options) { - // Use pageSize and pageToken options when Cloud Resource Manager does so (#421) List projectsSerialized = new ArrayList<>(); String[] filters = (String[]) options.get("filter"); if (filters != null && !isValidFilter(filters)) { return Error.INVALID_ARGUMENT.response("Could not parse the filter."); } - String[] fields = (String[]) options.get("fields"); - for (Project p : projects.values()) { + String[] projectFields = (String[]) options.get("projectFields"); + int count = 0; + String pageToken = (String) options.get("pageToken"); + Integer pageSize = (Integer) options.get("pageSize"); + String nextPageToken = null; + Map projectsToScan = projects; + if (pageToken != null) { + projectsToScan = projects.tailMap(pageToken); + } + for (Project p : projectsToScan.values()) { + if (pageSize != null && count >= pageSize) { + nextPageToken = p.getProjectId(); + break; + } boolean includeProject = includeProject(p, filters); if (includeProject) { + count++; try { - projectsSerialized.add(jsonFactory.toString(extractFields(p, fields))); + projectsSerialized.add(jsonFactory.toString(extractFields(p, projectFields))); } catch (IOException e) { return Error.INTERNAL_ERROR.response( "Error when serializing project " + p.getProjectId()); } } } + String[] listFields = (String[]) options.get("listFields"); StringBuilder responseBody = new StringBuilder(); - responseBody.append("{\"projects\": ["); - Joiner.on(",").appendTo(responseBody, projectsSerialized); - responseBody.append("]}"); + responseBody.append('{'); + // If fields parameter is set but no project field is selected we must return no projects. + if (!(projectFields != null && projectFields.length == 0)) { + responseBody.append("\"projects\": ["); + Joiner.on(",").appendTo(responseBody, projectsSerialized); + responseBody.append(']'); + } + if (nextPageToken != null && (listFields == null + || ImmutableSet.copyOf(listFields).contains("nextPageToken"))) { + if (responseBody.length() > 1) { + responseBody.append(','); + } + responseBody.append("\"nextPageToken\": \""); + responseBody.append(nextPageToken); + responseBody.append('"'); + } + responseBody.append('}'); return new Response(HTTP_OK, responseBody.toString()); } diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/LocalResourceManagerHelperTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/LocalResourceManagerHelperTest.java index 7eb0156d4e56..a3418fff98ab 100644 --- a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/LocalResourceManagerHelperTest.java +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/LocalResourceManagerHelperTest.java @@ -19,6 +19,7 @@ import org.junit.Test; import java.util.HashMap; +import java.util.Iterator; import java.util.Map; public class LocalResourceManagerHelperTest { @@ -278,7 +279,7 @@ public void testGetWithOptions() { public void testList() { Tuple> projects = rpc.list(EMPTY_RPC_OPTIONS); - assertNull(projects.x()); // change this when #421 is resolved + assertNull(projects.x()); assertFalse(projects.y().iterator().hasNext()); rpc.create(COMPLETE_PROJECT); RESOURCE_MANAGER_HELPER.changeLifecycleState( @@ -297,11 +298,43 @@ public void testList() { } @Test - public void testListFieldOptions() { + public void testInvalidListPaging() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, -1); + try { + rpc.list(rpcOptions); + } catch (ResourceManagerException e) { + assertEquals("Page size must be greater than 0.", e.getMessage()); + } + } + + @Test + public void testListPaging() { Map rpcOptions = new HashMap<>(); - rpcOptions.put(ResourceManagerRpc.Option.FIELDS, "projects(projectId,name,labels)"); - rpcOptions.put(ResourceManagerRpc.Option.PAGE_TOKEN, "somePageToken"); rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, 1); + rpc.create(PARTIAL_PROJECT); + rpc.create(COMPLETE_PROJECT); + Tuple> projects = + rpc.list(rpcOptions); + assertNotNull(projects.x()); + Iterator iterator = + projects.y().iterator(); + compareReadWriteFields(COMPLETE_PROJECT, iterator.next()); + assertFalse(iterator.hasNext()); + rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_TOKEN, projects.x()); + projects = rpc.list(rpcOptions); + iterator = projects.y().iterator(); + compareReadWriteFields(PARTIAL_PROJECT, iterator.next()); + assertFalse(iterator.hasNext()); + assertNull(projects.x()); + } + + @Test + public void testListFieldOptions() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.FIELDS, + "projects(projectId,name,labels),nextPageToken"); rpc.create(PROJECT_WITH_PARENT); Tuple> projects = rpc.list(rpcOptions); @@ -317,6 +350,81 @@ public void testListFieldOptions() { assertNull(returnedProject.getCreateTime()); } + @Test + public void testListPageTokenFieldOptions() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, 1); + rpcOptions.put(ResourceManagerRpc.Option.FIELDS, "nextPageToken,projects(projectId,name)"); + rpc.create(PARTIAL_PROJECT); + rpc.create(COMPLETE_PROJECT); + Tuple> projects = + rpc.list(rpcOptions); + assertNotNull(projects.x()); + Iterator iterator = + projects.y().iterator(); + com.google.api.services.cloudresourcemanager.model.Project returnedProject = iterator.next(); + assertEquals(COMPLETE_PROJECT.getProjectId(), returnedProject.getProjectId()); + assertEquals(COMPLETE_PROJECT.getName(), returnedProject.getName()); + assertNull(returnedProject.getLabels()); + assertNull(returnedProject.getParent()); + assertNull(returnedProject.getProjectNumber()); + assertNull(returnedProject.getLifecycleState()); + assertNull(returnedProject.getCreateTime()); + assertFalse(iterator.hasNext()); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_TOKEN, projects.x()); + projects = rpc.list(rpcOptions); + iterator = projects.y().iterator(); + returnedProject = iterator.next(); + assertEquals(PARTIAL_PROJECT.getProjectId(), returnedProject.getProjectId()); + assertEquals(PARTIAL_PROJECT.getName(), returnedProject.getName()); + assertNull(returnedProject.getLabels()); + assertNull(returnedProject.getParent()); + assertNull(returnedProject.getProjectNumber()); + assertNull(returnedProject.getLifecycleState()); + assertNull(returnedProject.getCreateTime()); + assertNull(projects.x()); + } + + @Test + public void testListNoPageTokenFieldOptions() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, 1); + rpcOptions.put(ResourceManagerRpc.Option.FIELDS, "projects(projectId,name)"); + rpc.create(PARTIAL_PROJECT); + rpc.create(COMPLETE_PROJECT); + Tuple> projects = + rpc.list(rpcOptions); + assertNull(projects.x()); + Iterator iterator = + projects.y().iterator(); + com.google.api.services.cloudresourcemanager.model.Project returnedProject = iterator.next(); + assertEquals(COMPLETE_PROJECT.getProjectId(), returnedProject.getProjectId()); + assertEquals(COMPLETE_PROJECT.getName(), returnedProject.getName()); + assertNull(returnedProject.getLabels()); + assertNull(returnedProject.getParent()); + assertNull(returnedProject.getProjectNumber()); + assertNull(returnedProject.getLifecycleState()); + assertNull(returnedProject.getCreateTime()); + assertFalse(iterator.hasNext()); + } + + @Test + public void testListPageTokenNoFieldsOptions() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, 1); + rpcOptions.put(ResourceManagerRpc.Option.FIELDS, "nextPageToken"); + rpc.create(PARTIAL_PROJECT); + rpc.create(COMPLETE_PROJECT); + Tuple> projects = + rpc.list(rpcOptions); + assertNotNull(projects.x()); + assertNull(projects.y()); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_TOKEN, projects.x()); + projects = rpc.list(rpcOptions); + assertNull(projects.x()); + assertNull(projects.y()); + } + @Test public void testListFilterOptions() { Map rpcFilterOptions = new HashMap<>(); diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/PolicyTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/PolicyTest.java new file mode 100644 index 000000000000..05d1b85bdbed --- /dev/null +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/PolicyTest.java @@ -0,0 +1,53 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.resourcemanager; + +import static org.junit.Assert.assertEquals; + +import com.google.common.collect.ImmutableSet; +import com.google.gcloud.Identity; + +import org.junit.Test; + +public class PolicyTest { + + private static final Identity ALL_USERS = Identity.allUsers(); + private static final Identity ALL_AUTH_USERS = Identity.allAuthenticatedUsers(); + private static final Identity USER = Identity.user("abc@gmail.com"); + private static final Identity SERVICE_ACCOUNT = + Identity.serviceAccount("service-account@gmail.com"); + private static final Identity GROUP = Identity.group("group@gmail.com"); + private static final Identity DOMAIN = Identity.domain("google.com"); + private static final Policy SIMPLE_POLICY = Policy.builder() + .addBinding(Policy.Role.VIEWER, ImmutableSet.of(USER, SERVICE_ACCOUNT, ALL_USERS)) + .addBinding(Policy.Role.EDITOR, ImmutableSet.of(ALL_AUTH_USERS, GROUP, DOMAIN)) + .build(); + private static final Policy FULL_POLICY = + new Policy.Builder(SIMPLE_POLICY.bindings(), "etag", 1).build(); + + @Test + public void testIamPolicyToBuilder() { + assertEquals(FULL_POLICY, FULL_POLICY.toBuilder().build()); + assertEquals(SIMPLE_POLICY, SIMPLE_POLICY.toBuilder().build()); + } + + @Test + public void testPolicyToAndFromPb() { + assertEquals(FULL_POLICY, Policy.fromPb(FULL_POLICY.toPb())); + assertEquals(SIMPLE_POLICY, Policy.fromPb(SIMPLE_POLICY.toPb())); + } +} diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ProjectTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ProjectTest.java index 077818bf2bb9..4e239acc45ef 100644 --- a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ProjectTest.java +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ProjectTest.java @@ -16,14 +16,14 @@ package com.google.gcloud.resourcemanager; +import static org.easymock.EasyMock.anyObject; +import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertSame; import com.google.common.collect.ImmutableMap; @@ -48,13 +48,15 @@ public class ProjectTest { .state(STATE) .build(); + private ResourceManager serviceMockReturnsOptions = createStrictMock(ResourceManager.class); + private ResourceManagerOptions mockOptions = createMock(ResourceManagerOptions.class); private ResourceManager resourceManager; + private Project expectedProject; private Project project; @Before - public void setUp() throws Exception { + public void setUp() { resourceManager = createStrictMock(ResourceManager.class); - project = new Project(resourceManager, PROJECT_INFO); } @After @@ -62,75 +64,142 @@ public void tearDown() throws Exception { verify(resourceManager); } + private void initializeExpectedProject(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); + expectedProject = + new Project(serviceMockReturnsOptions, new ProjectInfo.BuilderImpl(PROJECT_INFO)); + } + + private void initializeProject() { + project = new Project(resourceManager, new ProjectInfo.BuilderImpl(PROJECT_INFO)); + } + @Test - public void testLoad() { - expect(resourceManager.get(PROJECT_INFO.projectId())).andReturn(PROJECT_INFO); + public void testToBuilder() { + initializeExpectedProject(4); replay(resourceManager); - Project loadedProject = Project.get(resourceManager, PROJECT_INFO.projectId()); - assertEquals(PROJECT_INFO, loadedProject.info()); + compareProjects(expectedProject, expectedProject.toBuilder().build()); + } + + @Test + public void testBuilder() { + initializeExpectedProject(4); + expect(resourceManager.options()).andReturn(mockOptions).times(4); + replay(resourceManager); + Project.Builder builder = + new Project.Builder(new Project(resourceManager, new ProjectInfo.BuilderImpl(PROJECT_ID))); + Project project = builder.name(NAME) + .labels(LABELS) + .projectNumber(PROJECT_NUMBER) + .createTimeMillis(CREATE_TIME_MILLIS) + .state(STATE) + .build(); + assertEquals(PROJECT_ID, project.projectId()); + assertEquals(NAME, project.name()); + assertEquals(LABELS, project.labels()); + assertEquals(PROJECT_NUMBER, project.projectNumber()); + assertEquals(CREATE_TIME_MILLIS, project.createTimeMillis()); + assertEquals(STATE, project.state()); + assertEquals(resourceManager.options(), project.resourceManager().options()); + } + + @Test + public void testGet() { + initializeExpectedProject(1); + expect(resourceManager.get(PROJECT_INFO.projectId())).andReturn(expectedProject); + replay(resourceManager); + Project loadedProject = resourceManager.get(PROJECT_INFO.projectId()); + assertEquals(expectedProject, loadedProject); } @Test public void testReload() { + initializeExpectedProject(2); ProjectInfo newInfo = PROJECT_INFO.toBuilder().addLabel("k3", "v3").build(); - expect(resourceManager.get(PROJECT_INFO.projectId())).andReturn(newInfo); + Project expectedProject = + new Project(serviceMockReturnsOptions, new ProjectInfo.BuilderImpl(newInfo)); + expect(resourceManager.options()).andReturn(mockOptions); + expect(resourceManager.get(PROJECT_INFO.projectId())).andReturn(expectedProject); replay(resourceManager); + initializeProject(); Project newProject = project.reload(); - assertSame(resourceManager, newProject.resourceManager()); - assertEquals(newInfo, newProject.info()); + assertEquals(expectedProject, newProject); } @Test public void testLoadNull() { + initializeExpectedProject(1); expect(resourceManager.get(PROJECT_INFO.projectId())).andReturn(null); replay(resourceManager); - assertNull(Project.get(resourceManager, PROJECT_INFO.projectId())); + assertNull(resourceManager.get(PROJECT_INFO.projectId())); } @Test - public void testReloadDeletedProject() { - expect(resourceManager.get(PROJECT_INFO.projectId())).andReturn(PROJECT_INFO); + public void testReloadNull() { + initializeExpectedProject(1); + expect(resourceManager.options()).andReturn(mockOptions); expect(resourceManager.get(PROJECT_INFO.projectId())).andReturn(null); replay(resourceManager); - Project loadedProject = Project.get(resourceManager, PROJECT_INFO.projectId()); - assertNotNull(loadedProject); - Project reloadedProject = loadedProject.reload(); + Project reloadedProject = + new Project(resourceManager, new ProjectInfo.BuilderImpl(PROJECT_INFO)).reload(); assertNull(reloadedProject); } - @Test - public void testInfo() { - replay(resourceManager); - assertEquals(PROJECT_INFO, project.info()); - } - @Test public void testResourceManager() { + initializeExpectedProject(1); replay(resourceManager); - assertEquals(resourceManager, project.resourceManager()); + assertEquals(serviceMockReturnsOptions, expectedProject.resourceManager()); } @Test public void testDelete() { + initializeExpectedProject(1); + expect(resourceManager.options()).andReturn(mockOptions); resourceManager.delete(PROJECT_INFO.projectId()); replay(resourceManager); + initializeProject(); project.delete(); } @Test public void testUndelete() { + initializeExpectedProject(1); + expect(resourceManager.options()).andReturn(mockOptions); resourceManager.undelete(PROJECT_INFO.projectId()); replay(resourceManager); + initializeProject(); project.undelete(); } @Test public void testReplace() { - ProjectInfo newInfo = PROJECT_INFO.toBuilder().addLabel("k3", "v3").build(); - expect(resourceManager.replace(newInfo)).andReturn(newInfo); + initializeExpectedProject(2); + Project expectedReplacedProject = expectedProject.toBuilder().addLabel("k3", "v3").build(); + expect(resourceManager.options()).andReturn(mockOptions).times(2); + expect(resourceManager.replace(anyObject(Project.class))).andReturn(expectedReplacedProject); replay(resourceManager); - Project newProject = project.replace(newInfo); - assertSame(resourceManager, newProject.resourceManager()); - assertEquals(newInfo, newProject.info()); + initializeProject(); + Project newProject = + new Project(resourceManager, new ProjectInfo.BuilderImpl(expectedReplacedProject)); + Project actualReplacedProject = newProject.replace(); + compareProjectInfos(expectedReplacedProject, actualReplacedProject); + } + + private void compareProjects(Project expected, Project value) { + assertEquals(expected, value); + compareProjectInfos(expected, value); + assertEquals(expected.resourceManager().options(), value.resourceManager().options()); + } + + private void compareProjectInfos(ProjectInfo expected, ProjectInfo value) { + assertEquals(expected.projectId(), value.projectId()); + assertEquals(expected.name(), value.name()); + assertEquals(expected.labels(), value.labels()); + assertEquals(expected.projectNumber(), value.projectNumber()); + assertEquals(expected.createTimeMillis(), value.createTimeMillis()); + assertEquals(expected.state(), value.state()); + assertEquals(expected.parent(), value.parent()); } } diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ResourceManagerImplTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ResourceManagerImplTest.java index 7d1e00496463..1bc233311a4d 100644 --- a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ResourceManagerImplTest.java +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ResourceManagerImplTest.java @@ -20,6 +20,7 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -41,6 +42,7 @@ import org.junit.Test; import org.junit.rules.ExpectedException; +import java.util.Iterator; import java.util.Map; public class ResourceManagerImplTest { @@ -78,7 +80,7 @@ public void setUp() { } private void clearProjects() { - for (ProjectInfo project : RESOURCE_MANAGER.list().values()) { + for (Project project : RESOURCE_MANAGER.list().values()) { RESOURCE_MANAGER_HELPER.removeProject(project.projectId()); } } @@ -97,13 +99,14 @@ private void compareReadWriteFields(ProjectInfo expected, ProjectInfo actual) { @Test public void testCreate() { - ProjectInfo returnedProject = RESOURCE_MANAGER.create(PARTIAL_PROJECT); + Project returnedProject = RESOURCE_MANAGER.create(PARTIAL_PROJECT); compareReadWriteFields(PARTIAL_PROJECT, returnedProject); assertEquals(ProjectInfo.State.ACTIVE, returnedProject.state()); assertNull(returnedProject.name()); assertNull(returnedProject.parent()); assertNotNull(returnedProject.projectNumber()); assertNotNull(returnedProject.createTimeMillis()); + assertSame(RESOURCE_MANAGER, returnedProject.resourceManager()); try { RESOURCE_MANAGER.create(PARTIAL_PROJECT); fail("Should fail, project already exists."); @@ -117,6 +120,7 @@ public void testCreate() { assertEquals(ProjectInfo.State.ACTIVE, returnedProject.state()); assertNotNull(returnedProject.projectNumber()); assertNotNull(returnedProject.createTimeMillis()); + assertSame(RESOURCE_MANAGER, returnedProject.resourceManager()); } @Test @@ -137,16 +141,17 @@ public void testDelete() { @Test public void testGet() { RESOURCE_MANAGER.create(COMPLETE_PROJECT); - ProjectInfo returnedProject = RESOURCE_MANAGER.get(COMPLETE_PROJECT.projectId()); + Project returnedProject = RESOURCE_MANAGER.get(COMPLETE_PROJECT.projectId()); compareReadWriteFields(COMPLETE_PROJECT, returnedProject); + assertEquals(RESOURCE_MANAGER, returnedProject.resourceManager()); RESOURCE_MANAGER_HELPER.removeProject(COMPLETE_PROJECT.projectId()); assertNull(RESOURCE_MANAGER.get(COMPLETE_PROJECT.projectId())); } @Test public void testGetWithOptions() { - ProjectInfo originalProject = RESOURCE_MANAGER.create(COMPLETE_PROJECT); - ProjectInfo returnedProject = RESOURCE_MANAGER.get(COMPLETE_PROJECT.projectId(), GET_FIELDS); + Project originalProject = RESOURCE_MANAGER.create(COMPLETE_PROJECT); + Project returnedProject = RESOURCE_MANAGER.get(COMPLETE_PROJECT.projectId(), GET_FIELDS); assertFalse(COMPLETE_PROJECT.equals(returnedProject)); assertEquals(COMPLETE_PROJECT.projectId(), returnedProject.projectId()); assertEquals(COMPLETE_PROJECT.name(), returnedProject.name()); @@ -155,15 +160,17 @@ public void testGetWithOptions() { assertNull(returnedProject.projectNumber()); assertNull(returnedProject.state()); assertTrue(returnedProject.labels().isEmpty()); + assertEquals(RESOURCE_MANAGER, originalProject.resourceManager()); + assertEquals(RESOURCE_MANAGER, returnedProject.resourceManager()); } @Test public void testList() { - Page projects = RESOURCE_MANAGER.list(); - assertFalse(projects.values().iterator().hasNext()); // TODO: change this when #421 is resolved + Page projects = RESOURCE_MANAGER.list(); + assertFalse(projects.values().iterator().hasNext()); RESOURCE_MANAGER.create(PARTIAL_PROJECT); RESOURCE_MANAGER.create(COMPLETE_PROJECT); - for (ProjectInfo p : RESOURCE_MANAGER.list().values()) { + for (Project p : RESOURCE_MANAGER.list().values()) { if (p.projectId().equals(PARTIAL_PROJECT.projectId())) { compareReadWriteFields(PARTIAL_PROJECT, p); } else if (p.projectId().equals(COMPLETE_PROJECT.projectId())) { @@ -171,14 +178,49 @@ public void testList() { } else { fail("Some unexpected project returned by list."); } + assertSame(RESOURCE_MANAGER, p.resourceManager()); } } + @Test + public void testListPaging() { + RESOURCE_MANAGER.create(PARTIAL_PROJECT); + RESOURCE_MANAGER.create(COMPLETE_PROJECT); + Page page = RESOURCE_MANAGER.list(ProjectListOption.pageSize(1)); + assertNotNull(page.nextPageCursor()); + Iterator iterator = page.values().iterator(); + compareReadWriteFields(COMPLETE_PROJECT, iterator.next()); + assertFalse(iterator.hasNext()); + page = page.nextPage(); + iterator = page.values().iterator(); + compareReadWriteFields(PARTIAL_PROJECT, iterator.next()); + assertFalse(iterator.hasNext()); + assertNull(page.nextPageCursor()); + } + @Test public void testListFieldOptions() { RESOURCE_MANAGER.create(COMPLETE_PROJECT); - Page projects = RESOURCE_MANAGER.list(LIST_FIELDS); - ProjectInfo returnedProject = projects.iterateAll().next(); + Page projects = RESOURCE_MANAGER.list(LIST_FIELDS); + Project returnedProject = projects.iterateAll().next(); + assertEquals(COMPLETE_PROJECT.projectId(), returnedProject.projectId()); + assertEquals(COMPLETE_PROJECT.name(), returnedProject.name()); + assertEquals(COMPLETE_PROJECT.labels(), returnedProject.labels()); + assertNull(returnedProject.parent()); + assertNull(returnedProject.projectNumber()); + assertNull(returnedProject.state()); + assertNull(returnedProject.createTimeMillis()); + assertSame(RESOURCE_MANAGER, returnedProject.resourceManager()); + } + + @Test + public void testListPagingWithFieldOptions() { + RESOURCE_MANAGER.create(PARTIAL_PROJECT); + RESOURCE_MANAGER.create(COMPLETE_PROJECT); + Page projects = RESOURCE_MANAGER.list(LIST_FIELDS, ProjectListOption.pageSize(1)); + assertNotNull(projects.nextPageCursor()); + Iterator iterator = projects.values().iterator(); + Project returnedProject = iterator.next(); assertEquals(COMPLETE_PROJECT.projectId(), returnedProject.projectId()); assertEquals(COMPLETE_PROJECT.name(), returnedProject.name()); assertEquals(COMPLETE_PROJECT.labels(), returnedProject.labels()); @@ -186,6 +228,21 @@ public void testListFieldOptions() { assertNull(returnedProject.projectNumber()); assertNull(returnedProject.state()); assertNull(returnedProject.createTimeMillis()); + assertSame(RESOURCE_MANAGER, returnedProject.resourceManager()); + assertFalse(iterator.hasNext()); + projects = projects.nextPage(); + iterator = projects.values().iterator(); + returnedProject = iterator.next(); + assertEquals(PARTIAL_PROJECT.projectId(), returnedProject.projectId()); + assertEquals(PARTIAL_PROJECT.name(), returnedProject.name()); + assertEquals(PARTIAL_PROJECT.labels(), returnedProject.labels()); + assertNull(returnedProject.parent()); + assertNull(returnedProject.projectNumber()); + assertNull(returnedProject.state()); + assertNull(returnedProject.createTimeMillis()); + assertSame(RESOURCE_MANAGER, returnedProject.resourceManager()); + assertFalse(iterator.hasNext()); + assertNull(projects.nextPageCursor()); } @Test @@ -207,10 +264,11 @@ public void testListFilterOptions() { RESOURCE_MANAGER.create(nonMatchingProject1); RESOURCE_MANAGER.create(nonMatchingProject2); RESOURCE_MANAGER.create(nonMatchingProject3); - for (ProjectInfo p : RESOURCE_MANAGER.list(LIST_FILTER).values()) { + for (Project p : RESOURCE_MANAGER.list(LIST_FILTER).values()) { assertFalse(p.equals(nonMatchingProject1)); assertFalse(p.equals(nonMatchingProject2)); compareReadWriteFields(matchingProject, p); + assertSame(RESOURCE_MANAGER, p.resourceManager()); } } @@ -225,11 +283,12 @@ public void testReplace() { .state(ProjectInfo.State.DELETE_REQUESTED) .parent(createdProject.parent()) .build(); - ProjectInfo returnedProject = RESOURCE_MANAGER.replace(anotherCompleteProject); + Project returnedProject = RESOURCE_MANAGER.replace(anotherCompleteProject); compareReadWriteFields(anotherCompleteProject, returnedProject); assertEquals(createdProject.projectNumber(), returnedProject.projectNumber()); assertEquals(createdProject.createTimeMillis(), returnedProject.createTimeMillis()); assertEquals(createdProject.state(), returnedProject.state()); + assertEquals(RESOURCE_MANAGER, returnedProject.resourceManager()); ProjectInfo nonexistantProject = ProjectInfo.builder("some-project-id-that-does-not-exist").build(); try { @@ -276,8 +335,10 @@ public void testRetryableException() { .andThrow(new ResourceManagerException(500, "Internal Error")) .andReturn(PARTIAL_PROJECT.toPb()); EasyMock.replay(resourceManagerRpcMock); - ProjectInfo returnedProject = resourceManagerMock.get(PARTIAL_PROJECT.projectId()); - assertEquals(PARTIAL_PROJECT, returnedProject); + Project returnedProject = resourceManagerMock.get(PARTIAL_PROJECT.projectId()); + assertEquals( + new Project(resourceManagerMock, new ProjectInfo.BuilderImpl(PARTIAL_PROJECT)), + returnedProject); } @Test diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/SerializationTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/SerializationTest.java index 64e09449149b..35b72ae1713f 100644 --- a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/SerializationTest.java +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/SerializationTest.java @@ -20,6 +20,8 @@ import static org.junit.Assert.assertNotSame; import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import com.google.gcloud.Identity; import com.google.gcloud.PageImpl; import com.google.gcloud.RetryParams; @@ -35,6 +37,8 @@ public class SerializationTest { +private static final ResourceManager RESOURCE_MANAGER = + ResourceManagerOptions.defaultInstance().service(); private static final ProjectInfo PARTIAL_PROJECT_INFO = ProjectInfo.builder("id1").build(); private static final ProjectInfo FULL_PROJECT_INFO = ProjectInfo.builder("id") .name("name") @@ -43,12 +47,17 @@ public class SerializationTest { .state(ProjectInfo.State.ACTIVE) .createTimeMillis(1234L) .build(); - private static final PageImpl PAGE_RESULT = - new PageImpl<>(null, "c", Collections.singletonList(PARTIAL_PROJECT_INFO)); + private static final Project PROJECT = + new Project(RESOURCE_MANAGER, new ProjectInfo.BuilderImpl(FULL_PROJECT_INFO)); + private static final PageImpl PAGE_RESULT = + new PageImpl<>(null, "c", Collections.singletonList(PROJECT)); private static final ResourceManager.ProjectGetOption PROJECT_GET_OPTION = ResourceManager.ProjectGetOption.fields(ResourceManager.ProjectField.NAME); private static final ResourceManager.ProjectListOption PROJECT_LIST_OPTION = ResourceManager.ProjectListOption.filter("name:*"); + private static final Policy POLICY = Policy.builder() + .addBinding(Policy.Role.VIEWER, ImmutableSet.of(Identity.user("abc@gmail.com"))) + .build(); @Test public void testServiceOptions() throws Exception { @@ -65,8 +74,8 @@ public void testServiceOptions() throws Exception { @Test public void testModelAndRequests() throws Exception { - Serializable[] objects = {PARTIAL_PROJECT_INFO, FULL_PROJECT_INFO, PAGE_RESULT, - PROJECT_GET_OPTION, PROJECT_LIST_OPTION}; + Serializable[] objects = {PARTIAL_PROJECT_INFO, FULL_PROJECT_INFO, PROJECT, PAGE_RESULT, + PROJECT_GET_OPTION, PROJECT_LIST_OPTION, POLICY}; for (Serializable obj : objects) { Object copy = serializeAndDeserialize(obj); assertEquals(obj, obj); diff --git a/gcloud-java-storage/README.md b/gcloud-java-storage/README.md index 7e08264433c8..f7973544bba2 100644 --- a/gcloud-java-storage/README.md +++ b/gcloud-java-storage/README.md @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Storage] (https://cloud.google.com/stora [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-storage.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-storage.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/storage/package-summary.html) @@ -20,22 +22,22 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java-storage - 0.1.2 + 0.1.4 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java-storage:0.1.2' +compile 'com.google.gcloud:gcloud-java-storage:0.1.4' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java-storage" % "0.1.2" +libraryDependencies += "com.google.gcloud" % "gcloud-java-storage" % "0.1.4" ``` Example Application ------------------- -[`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) is a simple command line interface that provides some of Cloud Storage's functionality. Read more about using the application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html). +[`StorageExample`](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java) is a simple command line interface that provides some of Cloud Storage's functionality. Read more about using the application on the [`StorageExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/storage/StorageExample.html). Authentication -------------- @@ -77,15 +79,15 @@ Storage storage = StorageOptions.defaultInstance().service(); For other authentication options, see the [Authentication](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) page. #### Storing data -Stored objects are called "blobs" in `gcloud-java` and are organized into containers called "buckets". In this code snippet, we will create a new bucket and upload a blob to that bucket. +Stored objects are called "blobs" in `gcloud-java` and are organized into containers called "buckets". `Blob`, a subclass of `BlobInfo`, adds a layer of service-related functionality over `BlobInfo`. Similarly, `Bucket` adds a layer of service-related functionality over `BucketInfo`. In this code snippet, we will create a new bucket and upload a blob to that bucket. Add the following imports at the top of your file: ```java import static java.nio.charset.StandardCharsets.UTF_8; -import com.google.gcloud.storage.BlobId; -import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.Blob; +import com.google.gcloud.storage.Bucket; import com.google.gcloud.storage.BucketInfo; ``` @@ -96,13 +98,12 @@ Then add the following code to create a bucket and upload a simple blob. ```java // Create a bucket String bucketName = "my_unique_bucket"; // Change this to something unique -BucketInfo bucketInfo = storage.create(BucketInfo.of(bucketName)); +Bucket bucket = storage.create(BucketInfo.of(bucketName)); // Upload a blob to the newly created bucket BlobId blobId = BlobId.of(bucketName, "my_blob_name"); -BlobInfo blobInfo = storage.create( - BlobInfo.builder(blobId).contentType("text/plain").build(), - "a simple blob".getBytes(UTF_8)); +Blob blob = storage.create( + "my_blob_name", "a simple blob".getBytes(UTF_8), "text/plain"); ``` At this point, you will be able to see your newly created bucket and blob on the Google Developers Console. @@ -111,7 +112,7 @@ At this point, you will be able to see your newly created bucket and blob on the Now that we have content uploaded to the server, we can see how to read data from the server. Add the following line to your program to get back the blob we uploaded. ```java -String blobContent = new String(storage.readAllBytes(blobId), UTF_8); +String blobContent = new String(blob.content(), UTF_8); ``` #### Listing buckets and contents of buckets @@ -125,14 +126,14 @@ Then add the following code to list all your buckets and all the blobs inside yo ```java // List all your buckets -Iterator bucketInfoIterator = storage.list().iterateAll(); +Iterator bucketIterator = storage.list().iterateAll(); System.out.println("My buckets:"); -while (bucketInfoIterator.hasNext()) { - System.out.println(bucketInfoIterator.next()); +while (bucketIterator.hasNext()) { + System.out.println(bucketIterator.next()); } // List the blobs in a particular bucket -Iterator blobIterator = storage.list(bucketName).iterateAll(); +Iterator blobIterator = bucket.list().iterateAll(); System.out.println("My blobs:"); while (blobIterator.hasNext()) { System.out.println(blobIterator.next()); @@ -141,55 +142,12 @@ while (blobIterator.hasNext()) { #### Complete source code -Here we put together all the code shown above into one program. This program assumes that you are running on Compute Engine or from your own desktop. To run this example on App Engine, simply move the code from the main method to your application's servlet class and change the print statements to display on your webpage. - -```java -import static java.nio.charset.StandardCharsets.UTF_8; - -import com.google.gcloud.storage.BlobId; -import com.google.gcloud.storage.BlobInfo; -import com.google.gcloud.storage.BucketInfo; -import com.google.gcloud.storage.Storage; -import com.google.gcloud.storage.StorageOptions; - -import java.util.Iterator; - -public class GcloudStorageExample { - - public static void main(String[] args) { - // Create a service object - // Credentials are inferred from the environment. - Storage storage = StorageOptions.defaultInstance().service(); - - // Create a bucket - String bucketName = "my_unique_bucket"; // Change this to something unique - BucketInfo bucketInfo = storage.create(BucketInfo.of(bucketName)); - - // Upload a blob to the newly created bucket - BlobId blobId = BlobId.of(bucketName, "my_blob_name"); - BlobInfo blobInfo = storage.create( - BlobInfo.builder(blobId).contentType("text/plain").build(), - "a simple blob".getBytes(UTF_8)); - - // Retrieve a blob from the server - String blobContent = new String(storage.readAllBytes(blobId), UTF_8); - - // List all your buckets - Iterator bucketInfoIterator = storage.list().iterateAll(); - System.out.println("My buckets:"); - while (bucketInfoIterator.hasNext()) { - System.out.println(bucketInfoIterator.next()); - } - - // List the blobs in a particular bucket - Iterator blobIterator = storage.list(bucketName).iterateAll(); - System.out.println("My blobs:"); - while (blobIterator.hasNext()) { - System.out.println(blobIterator.next()); - } - } -} -``` +In +[CreateAndListBucketsAndBlobs.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateAndListBucketsAndBlobs.java) +we put together all the code shown above into one program. The program assumes that you are +running on Compute Engine or from your own desktop. To run the example on App Engine, simply move +the code from the main method to your application's servlet class and change the print statements to +display on your webpage. Troubleshooting --------------- diff --git a/gcloud-java-storage/pom.xml b/gcloud-java-storage/pom.xml index 0dc525db4e63..f18283b70bc8 100644 --- a/gcloud-java-storage/pom.xml +++ b/gcloud-java-storage/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java-storage jar GCloud Java storage @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.3-SNAPSHOT + 0.1.5-SNAPSHOT gcloud-java-storage @@ -25,13 +24,13 @@ com.google.apis google-api-services-storage - v1-rev33-1.20.0 + v1-rev62-1.21.0 compile - - com.google.guava - guava-jdk5 - + + com.google.guava + guava-jdk5 + com.google.api-client google-api-client @@ -47,7 +46,7 @@ org.easymock easymock - 3.3 + 3.4 test diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BatchResponse.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BatchResponse.java index 98e7ce09cef0..fe5f6f5743c8 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BatchResponse.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BatchResponse.java @@ -31,8 +31,8 @@ public final class BatchResponse implements Serializable { private static final long serialVersionUID = 1057416839397037706L; private final List> deleteResult; - private final List> updateResult; - private final List> getResult; + private final List> updateResult; + private final List> getResult; public static class Result implements Serializable { @@ -113,8 +113,8 @@ static Result empty() { } } - BatchResponse(List> deleteResult, List> updateResult, - List> getResult) { + BatchResponse(List> deleteResult, List> updateResult, + List> getResult) { this.deleteResult = ImmutableList.copyOf(deleteResult); this.updateResult = ImmutableList.copyOf(updateResult); this.getResult = ImmutableList.copyOf(getResult); @@ -146,14 +146,14 @@ public List> deletes() { /** * Returns the results for the update operations using the request order. */ - public List> updates() { + public List> updates() { return updateResult; } /** * Returns the results for the get operations using the request order. */ - public List> gets() { + public List> gets() { return getResult; } } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Blob.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Blob.java index fe65f6ee010b..aea424ca4063 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Blob.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Blob.java @@ -16,26 +16,27 @@ package com.google.gcloud.storage; -import static com.google.common.base.Preconditions.checkArgument; import static com.google.common.base.Preconditions.checkNotNull; import static com.google.gcloud.storage.Blob.BlobSourceOption.toGetOptions; import static com.google.gcloud.storage.Blob.BlobSourceOption.toSourceOptions; +import com.google.api.services.storage.model.StorageObject; import com.google.common.base.Function; -import com.google.common.collect.ImmutableList; -import com.google.common.collect.Lists; import com.google.gcloud.ReadChannel; import com.google.gcloud.WriteChannel; import com.google.gcloud.spi.StorageRpc; +import com.google.gcloud.spi.StorageRpc.Tuple; import com.google.gcloud.storage.Storage.BlobTargetOption; import com.google.gcloud.storage.Storage.BlobWriteOption; import com.google.gcloud.storage.Storage.CopyRequest; import com.google.gcloud.storage.Storage.SignUrlOption; +import java.io.IOException; +import java.io.ObjectInputStream; import java.net.URL; import java.util.Arrays; -import java.util.Collections; import java.util.List; +import java.util.Map; import java.util.Objects; import java.util.concurrent.TimeUnit; @@ -44,13 +45,24 @@ * *

    Objects of this class are immutable. Operations that modify the blob like {@link #update} and * {@link #copyTo} return a new object. To get a {@code Blob} object with the most recent - * information use {@link #reload}. + * information use {@link #reload}. {@code Blob} adds a layer of service-related functionality over + * {@link BlobInfo}. *

    */ -public final class Blob { +public final class Blob extends BlobInfo { - private final Storage storage; - private final BlobInfo info; + private static final long serialVersionUID = -6806832496717441434L; + + private final StorageOptions options; + private transient Storage storage; + + static final Function, Blob> BLOB_FROM_PB_FUNCTION = + new Function, Blob>() { + @Override + public Blob apply(Tuple pb) { + return Blob.fromPb(pb.x(), pb.y()); + } + }; /** * Class for specifying blob source options when {@code Blob} methods are used. @@ -146,60 +158,148 @@ static Storage.BlobGetOption[] toGetOptions(BlobInfo blobInfo, BlobSourceOption. } /** - * Constructs a {@code Blob} object for the provided {@code BlobInfo}. The storage service is used - * to issue requests. - * - * @param storage the storage service used for issuing requests - * @param info blob's info + * Builder for {@code Blob}. */ - public Blob(Storage storage, BlobInfo info) { - this.storage = checkNotNull(storage); - this.info = checkNotNull(info); - } + public static class Builder extends BlobInfo.Builder { - /** - * Creates a {@code Blob} object for the provided bucket and blob names. Performs an RPC call to - * get the latest blob information. Returns {@code null} if the blob does not exist. - * - * @param storage the storage service used for issuing requests - * @param bucket bucket's name - * @param options blob get options - * @param blob blob's name - * @return the {@code Blob} object or {@code null} if not found - * @throws StorageException upon failure - */ - public static Blob get(Storage storage, String bucket, String blob, - Storage.BlobGetOption... options) { - return get(storage, BlobId.of(bucket, blob), options); - } + private final Storage storage; + private final BlobInfo.BuilderImpl infoBuilder; - /** - * Creates a {@code Blob} object for the provided {@code blobId}. Performs an RPC call to get the - * latest blob information. Returns {@code null} if the blob does not exist. - * - * @param storage the storage service used for issuing requests - * @param blobId blob's identifier - * @param options blob get options - * @return the {@code Blob} object or {@code null} if not found - * @throws StorageException upon failure - */ - public static Blob get(Storage storage, BlobId blobId, Storage.BlobGetOption... options) { - BlobInfo info = storage.get(blobId, options); - return info != null ? new Blob(storage, info) : null; - } + Builder(Blob blob) { + this.storage = blob.storage(); + this.infoBuilder = new BlobInfo.BuilderImpl(blob); + } - /** - * Returns the blob's information. - */ - public BlobInfo info() { - return info; + @Override + public Builder blobId(BlobId blobId) { + infoBuilder.blobId(blobId); + return this; + } + + @Override + Builder id(String id) { + infoBuilder.id(id); + return this; + } + + @Override + public Builder contentType(String contentType) { + infoBuilder.contentType(contentType); + return this; + } + + @Override + public Builder contentDisposition(String contentDisposition) { + infoBuilder.contentDisposition(contentDisposition); + return this; + } + + @Override + public Builder contentLanguage(String contentLanguage) { + infoBuilder.contentLanguage(contentLanguage); + return this; + } + + @Override + public Builder contentEncoding(String contentEncoding) { + infoBuilder.contentEncoding(contentEncoding); + return this; + } + + @Override + Builder componentCount(Integer componentCount) { + infoBuilder.componentCount(componentCount); + return this; + } + + @Override + public Builder cacheControl(String cacheControl) { + infoBuilder.cacheControl(cacheControl); + return this; + } + + @Override + public Builder acl(List acl) { + infoBuilder.acl(acl); + return this; + } + + @Override + Builder owner(Acl.Entity owner) { + infoBuilder.owner(owner); + return this; + } + + @Override + Builder size(Long size) { + infoBuilder.size(size); + return this; + } + + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; + } + + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + public Builder md5(String md5) { + infoBuilder.md5(md5); + return this; + } + + @Override + public Builder crc32c(String crc32c) { + infoBuilder.crc32c(crc32c); + return this; + } + + @Override + Builder mediaLink(String mediaLink) { + infoBuilder.mediaLink(mediaLink); + return this; + } + + @Override + public Builder metadata(Map metadata) { + infoBuilder.metadata(metadata); + return this; + } + + @Override + Builder metageneration(Long metageneration) { + infoBuilder.metageneration(metageneration); + return this; + } + + @Override + Builder deleteTime(Long deleteTime) { + infoBuilder.deleteTime(deleteTime); + return this; + } + + @Override + Builder updateTime(Long updateTime) { + infoBuilder.updateTime(updateTime); + return this; + } + + @Override + public Blob build() { + return new Blob(storage, infoBuilder); + } } - /** - * Returns the blob's id. - */ - public BlobId id() { - return info.blobId(); + Blob(Storage storage, BlobInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.storage = checkNotNull(storage); + this.options = storage.options(); } /** @@ -211,9 +311,9 @@ public BlobId id() { */ public boolean exists(BlobSourceOption... options) { int length = options.length; - Storage.BlobGetOption[] getOptions = Arrays.copyOf(toGetOptions(info, options), length + 1); + Storage.BlobGetOption[] getOptions = Arrays.copyOf(toGetOptions(this, options), length + 1); getOptions[length] = Storage.BlobGetOption.fields(); - return storage.get(info.blobId(), getOptions) != null; + return storage.get(blobId(), getOptions) != null; } /** @@ -223,7 +323,7 @@ public boolean exists(BlobSourceOption... options) { * @throws StorageException upon failure */ public byte[] content(Storage.BlobSourceOption... options) { - return storage.readAllBytes(info.blobId(), options); + return storage.readAllBytes(blobId(), options); } /** @@ -234,7 +334,7 @@ public byte[] content(Storage.BlobSourceOption... options) { * @throws StorageException upon failure */ public Blob reload(BlobSourceOption... options) { - return Blob.get(storage, info.blobId(), toGetOptions(info, options)); + return storage.get(blobId(), toGetOptions(this, options)); } /** @@ -243,27 +343,26 @@ public Blob reload(BlobSourceOption... options) { * {@link #delete} operations. A new {@code Blob} object is returned. By default no checks are * made on the metadata generation of the current blob. If you want to update the information only * if the current blob metadata are at their latest version use the {@code metagenerationMatch} - * option: {@code blob.update(newInfo, BlobTargetOption.metagenerationMatch())}. + * option: {@code newBlob.update(BlobTargetOption.metagenerationMatch())}. * - *

    Original metadata are merged with metadata in the provided {@code blobInfo}. To replace - * metadata instead you first have to unset them. Unsetting metadata can be done by setting the - * provided {@code blobInfo}'s metadata to {@code null}. + *

    Original metadata are merged with metadata in the provided in this {@code blob}. To replace + * metadata instead you first have to unset them. Unsetting metadata can be done by setting this + * {@code blob}'s metadata to {@code null}. *

    * *

    Example usage of replacing blob's metadata: - *

        {@code blob.update(blob.info().toBuilder().metadata(null).build());}
    -   *    {@code blob.update(blob.info().toBuilder().metadata(newMetadata).build());}
    +   * 
     {@code
    +   * blob.toBuilder().metadata(null).build().update();
    +   * blob.toBuilder().metadata(newMetadata).build().update();
    +   * }
        * 
    * - * @param blobInfo new blob's information. Bucket and blob names must match the current ones * @param options update options * @return a {@code Blob} object with updated information * @throws StorageException upon failure */ - public Blob update(BlobInfo blobInfo, BlobTargetOption... options) { - checkArgument(Objects.equals(blobInfo.bucket(), info.bucket()), "Bucket name must match"); - checkArgument(Objects.equals(blobInfo.name(), info.name()), "Blob name must match"); - return new Blob(storage, storage.update(blobInfo, options)); + public Blob update(BlobTargetOption... options) { + return storage.update(this, options); } /** @@ -274,7 +373,7 @@ public Blob update(BlobInfo blobInfo, BlobTargetOption... options) { * @throws StorageException upon failure */ public boolean delete(BlobSourceOption... options) { - return storage.delete(info.blobId(), toSourceOptions(info, options)); + return storage.delete(blobId(), toSourceOptions(this, options)); } /** @@ -288,8 +387,11 @@ public boolean delete(BlobSourceOption... options) { * @throws StorageException upon failure */ public CopyWriter copyTo(BlobId targetBlob, BlobSourceOption... options) { - CopyRequest copyRequest = CopyRequest.builder().source(info.bucket(), info.name()) - .sourceOptions(toSourceOptions(info, options)).target(targetBlob).build(); + CopyRequest copyRequest = CopyRequest.builder() + .source(bucket(), name()) + .sourceOptions(toSourceOptions(this, options)) + .target(targetBlob) + .build(); return storage.copy(copyRequest); } @@ -304,7 +406,7 @@ public CopyWriter copyTo(BlobId targetBlob, BlobSourceOption... options) { * @throws StorageException upon failure */ public CopyWriter copyTo(String targetBucket, BlobSourceOption... options) { - return copyTo(targetBucket, info.name(), options); + return copyTo(targetBucket, name(), options); } /** @@ -329,7 +431,7 @@ public CopyWriter copyTo(String targetBucket, String targetBlob, BlobSourceOptio * @throws StorageException upon failure */ public ReadChannel reader(BlobSourceOption... options) { - return storage.reader(info.blobId(), toSourceOptions(info, options)); + return storage.reader(blobId(), toSourceOptions(this, options)); } /** @@ -341,7 +443,7 @@ public ReadChannel reader(BlobSourceOption... options) { * @throws StorageException upon failure */ public WriteChannel writer(BlobWriteOption... options) { - return storage.writer(info, options); + return storage.writer(this, options); } /** @@ -358,7 +460,7 @@ public WriteChannel writer(BlobWriteOption... options) { * @see Signed-URLs */ public URL signUrl(long duration, TimeUnit unit, SignUrlOption... options) { - return storage.signUrl(info, duration, unit, options); + return storage.signUrl(this, duration, unit, options); } /** @@ -368,97 +470,29 @@ public Storage storage() { return storage; } - /** - * Gets the requested blobs. A batch request is used to fetch blobs. - * - * @param storage the storage service used to issue the request - * @param first the first blob to get - * @param second the second blob to get - * @param other other blobs to get - * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it has - * been denied the corresponding item in the list is {@code null} - * @throws StorageException upon failure - */ - public static List get(Storage storage, BlobId first, BlobId second, BlobId... other) { - checkNotNull(storage); - checkNotNull(first); - checkNotNull(second); - checkNotNull(other); - ImmutableList blobs = ImmutableList.builder() - .add(first) - .add(second) - .addAll(Arrays.asList(other)) - .build(); - return get(storage, blobs); + @Override + public Builder toBuilder() { + return new Builder(this); } - /** - * Gets the requested blobs. A batch request is used to fetch blobs. - * - * @param storage the storage service used to issue the request - * @param blobs list of blobs to get - * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it has - * been denied the corresponding item in the list is {@code null} - * @throws StorageException upon failure - */ - public static List get(final Storage storage, List blobs) { - checkNotNull(storage); - checkNotNull(blobs); - BlobId[] blobArray = blobs.toArray(new BlobId[blobs.size()]); - return Collections.unmodifiableList(Lists.transform(storage.get(blobArray), - new Function() { - @Override - public Blob apply(BlobInfo blobInfo) { - return blobInfo != null ? new Blob(storage, blobInfo) : null; - } - })); + @Override + public boolean equals(Object obj) { + return obj instanceof Blob && Objects.equals(toPb(), ((Blob) obj).toPb()) + && Objects.equals(options, ((Blob) obj).options); } - /** - * Updates the requested blobs. A batch request is used to update blobs. Original metadata are - * merged with metadata in the provided {@code BlobInfo} objects. To replace metadata instead - * you first have to unset them. Unsetting metadata can be done by setting the provided - * {@code BlobInfo} objects metadata to {@code null}. See - * {@link #update(com.google.gcloud.storage.BlobInfo, - * com.google.gcloud.storage.Storage.BlobTargetOption...) } for a code example. - * - * @param storage the storage service used to issue the request - * @param infos the blobs to update - * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it has - * been denied the corresponding item in the list is {@code null} - * @throws StorageException upon failure - */ - public static List update(final Storage storage, BlobInfo... infos) { - checkNotNull(storage); - checkNotNull(infos); - if (infos.length == 0) { - return Collections.emptyList(); - } - return Collections.unmodifiableList(Lists.transform(storage.update(infos), - new Function() { - @Override - public Blob apply(BlobInfo blobInfo) { - return blobInfo != null ? new Blob(storage, blobInfo) : null; - } - })); + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); } - /** - * Deletes the requested blobs. A batch request is used to delete blobs. - * - * @param storage the storage service used to issue the request - * @param blobs the blobs to delete - * @return an immutable list of booleans. If a blob has been deleted the corresponding item in the - * list is {@code true}. If a blob was not found, deletion failed or access to the resource - * was denied the corresponding item is {@code false} - * @throws StorageException upon failure - */ - public static List delete(Storage storage, BlobId... blobs) { - checkNotNull(storage); - checkNotNull(blobs); - if (blobs.length == 0) { - return Collections.emptyList(); - } - return storage.delete(blobs); + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.storage = options.service(); + } + + static Blob fromPb(Storage storage, StorageObject storageObject) { + BlobInfo info = BlobInfo.fromPb(storageObject); + return new Blob(storage, new BlobInfo.BuilderImpl(info)); } } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobInfo.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobInfo.java index b27d00d68a16..54fabe87d766 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobInfo.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobInfo.java @@ -47,22 +47,16 @@ * @see Concepts and * Terminology */ -public final class BlobInfo implements Serializable { +public class BlobInfo implements Serializable { - static final Function FROM_PB_FUNCTION = - new Function() { - @Override - public BlobInfo apply(StorageObject pb) { - return BlobInfo.fromPb(pb); - } - }; - static final Function TO_PB_FUNCTION = + static final Function INFO_TO_PB_FUNCTION = new Function() { @Override public StorageObject apply(BlobInfo blobInfo) { return blobInfo.toPb(); } }; + private static final long serialVersionUID = 2228487739943277159L; private final BlobId blobId; private final String id; @@ -96,7 +90,110 @@ public Set> entrySet() { } } - public static final class Builder { + /** + * Builder for {@code BlobInfo}. + */ + public abstract static class Builder { + + /** + * Sets the blob identity. + */ + public abstract Builder blobId(BlobId blobId); + + abstract Builder id(String id); + + /** + * Sets the blob's data content type. + * + * @see Content-Type + */ + public abstract Builder contentType(String contentType); + + /** + * Sets the blob's data content disposition. + * + * @see Content-Disposition + */ + public abstract Builder contentDisposition(String contentDisposition); + + /** + * Sets the blob's data content language. + * + * @see Content-Language + */ + public abstract Builder contentLanguage(String contentLanguage); + + /** + * Sets the blob's data content encoding. + * + * @see Content-Encoding + */ + public abstract Builder contentEncoding(String contentEncoding); + + abstract Builder componentCount(Integer componentCount); + + /** + * Sets the blob's data cache control. + * + * @see Cache-Control + */ + public abstract Builder cacheControl(String cacheControl); + + /** + * Sets the blob's access control configuration. + * + * @see + * About Access Control Lists + */ + public abstract Builder acl(List acl); + + abstract Builder owner(Acl.Entity owner); + + abstract Builder size(Long size); + + abstract Builder etag(String etag); + + abstract Builder selfLink(String selfLink); + + /** + * Sets the MD5 hash of blob's data. MD5 value must be encoded in base64. + * + * @see + * Hashes and ETags: Best Practices + */ + public abstract Builder md5(String md5); + + /** + * Sets the CRC32C checksum of blob's data as described in + * RFC 4960, Appendix B; encoded in + * base64 in big-endian order. + * + * @see + * Hashes and ETags: Best Practices + */ + public abstract Builder crc32c(String crc32c); + + abstract Builder mediaLink(String mediaLink); + + /** + * Sets the blob's user provided metadata. + */ + public abstract Builder metadata(Map metadata); + + abstract Builder metageneration(Long metageneration); + + abstract Builder deleteTime(Long deleteTime); + + abstract Builder updateTime(Long updateTime); + + /** + * Creates a {@code BlobInfo} object. + */ + public abstract BlobInfo build(); + } + + static final class BuilderImpl extends Builder { private BlobId blobId; private String id; @@ -119,9 +216,11 @@ public static final class Builder { private Long deleteTime; private Long updateTime; - private Builder() {} + BuilderImpl(BlobId blobId) { + this.blobId = blobId; + } - private Builder(BlobInfo blobInfo) { + BuilderImpl(BlobInfo blobInfo) { blobId = blobInfo.blobId; id = blobInfo.id; cacheControl = blobInfo.cacheControl; @@ -144,168 +243,135 @@ private Builder(BlobInfo blobInfo) { updateTime = blobInfo.updateTime; } - /** - * Sets the blob identity. - */ + @Override public Builder blobId(BlobId blobId) { this.blobId = checkNotNull(blobId); return this; } + @Override Builder id(String id) { this.id = id; return this; } - /** - * Sets the blob's data content type. - * - * @see Content-Type - */ + @Override public Builder contentType(String contentType) { this.contentType = firstNonNull(contentType, Data.nullOf(String.class)); return this; } - /** - * Sets the blob's data content disposition. - * - * @see Content-Disposition - */ + @Override public Builder contentDisposition(String contentDisposition) { this.contentDisposition = firstNonNull(contentDisposition, Data.nullOf(String.class)); return this; } - /** - * Sets the blob's data content language. - * - * @see Content-Language - */ + @Override public Builder contentLanguage(String contentLanguage) { this.contentLanguage = firstNonNull(contentLanguage, Data.nullOf(String.class)); return this; } - /** - * Sets the blob's data content encoding. - * - * @see Content-Encoding - */ + @Override public Builder contentEncoding(String contentEncoding) { this.contentEncoding = firstNonNull(contentEncoding, Data.nullOf(String.class)); return this; } + @Override Builder componentCount(Integer componentCount) { this.componentCount = componentCount; return this; } - /** - * Sets the blob's data cache control. - * - * @see Cache-Control - */ + @Override public Builder cacheControl(String cacheControl) { this.cacheControl = firstNonNull(cacheControl, Data.nullOf(String.class)); return this; } - /** - * Sets the blob's access control configuration. - * - * @see - * About Access Control Lists - */ + @Override public Builder acl(List acl) { this.acl = acl != null ? ImmutableList.copyOf(acl) : null; return this; } + @Override Builder owner(Acl.Entity owner) { this.owner = owner; return this; } + @Override Builder size(Long size) { this.size = size; return this; } + @Override Builder etag(String etag) { this.etag = etag; return this; } + @Override Builder selfLink(String selfLink) { this.selfLink = selfLink; return this; } - /** - * Sets the MD5 hash of blob's data. MD5 value must be encoded in base64. - * - * @see - * Hashes and ETags: Best Practices - */ + @Override public Builder md5(String md5) { this.md5 = firstNonNull(md5, Data.nullOf(String.class)); return this; } - /** - * Sets the CRC32C checksum of blob's data as described in - * RFC 4960, Appendix B; encoded in - * base64 in big-endian order. - * - * @see - * Hashes and ETags: Best Practices - */ + @Override public Builder crc32c(String crc32c) { this.crc32c = firstNonNull(crc32c, Data.nullOf(String.class)); return this; } + @Override Builder mediaLink(String mediaLink) { this.mediaLink = mediaLink; return this; } - /** - * Sets the blob's user provided metadata. - */ + @Override public Builder metadata(Map metadata) { this.metadata = metadata != null ? new HashMap<>(metadata) : Data.>nullOf(ImmutableEmptyMap.class); return this; } + @Override Builder metageneration(Long metageneration) { this.metageneration = metageneration; return this; } + @Override Builder deleteTime(Long deleteTime) { this.deleteTime = deleteTime; return this; } + @Override Builder updateTime(Long updateTime) { this.updateTime = updateTime; return this; } - /** - * Creates a {@code BlobInfo} object. - */ + @Override public BlobInfo build() { checkNotNull(blobId); return new BlobInfo(this); } } - private BlobInfo(Builder builder) { + BlobInfo(BuilderImpl builder) { blobId = builder.blobId; id = builder.id; cacheControl = builder.cacheControl; @@ -526,7 +592,7 @@ public Long updateTime() { * Returns a builder for the current blob. */ public Builder toBuilder() { - return new Builder(this); + return new BuilderImpl(this); } @Override @@ -548,7 +614,8 @@ public int hashCode() { @Override public boolean equals(Object obj) { - return obj instanceof BlobInfo && Objects.equals(toPb(), ((BlobInfo) obj).toPb()); + return obj != null && obj.getClass().equals(BlobInfo.class) + && Objects.equals(toPb(), ((BlobInfo) obj).toPb()); } StorageObject toPb() { @@ -609,7 +676,7 @@ public static Builder builder(BucketInfo bucketInfo, String name) { * Returns a {@code BlobInfo} builder where blob identity is set using the provided values. */ public static Builder builder(String bucket, String name) { - return new Builder().blobId(BlobId.of(bucket, name)); + return builder(BlobId.of(bucket, name)); } /** @@ -623,11 +690,11 @@ public static Builder builder(BucketInfo bucketInfo, String name, Long generatio * Returns a {@code BlobInfo} builder where blob identity is set using the provided values. */ public static Builder builder(String bucket, String name, Long generation) { - return new Builder().blobId(BlobId.of(bucket, name, generation)); + return builder(BlobId.of(bucket, name, generation)); } public static Builder builder(BlobId blobId) { - return new Builder().blobId(blobId); + return new BuilderImpl(blobId); } static BlobInfo fromPb(StorageObject storageObject) { diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannel.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannel.java index 121f2eb63589..2b9643434ecc 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannel.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannel.java @@ -29,6 +29,7 @@ import java.io.IOException; import java.io.Serializable; import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; import java.util.Map; import java.util.Objects; import java.util.concurrent.Callable; @@ -55,7 +56,7 @@ class BlobReadChannel implements ReadChannel { private byte[] buffer; BlobReadChannel(StorageOptions serviceOptions, BlobId blob, - Map requestOptions) { + Map requestOptions) { this.serviceOptions = serviceOptions; this.blob = blob; this.requestOptions = requestOptions; @@ -91,9 +92,9 @@ public void close() { } } - private void validateOpen() throws IOException { + private void validateOpen() throws ClosedChannelException { if (!isOpen) { - throw new IOException("stream is closed"); + throw new ClosedChannelException(); } } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Bucket.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Bucket.java index 3acd3f5d79b9..d318626f4207 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Bucket.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Bucket.java @@ -23,13 +23,11 @@ import com.google.common.base.Function; import com.google.common.base.MoreObjects; -import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; +import com.google.common.collect.Sets; import com.google.gcloud.Page; -import com.google.gcloud.PageImpl; import com.google.gcloud.spi.StorageRpc; import com.google.gcloud.storage.Storage.BlobGetOption; -import com.google.gcloud.storage.Storage.BlobTargetOption; -import com.google.gcloud.storage.Storage.BlobWriteOption; import com.google.gcloud.storage.Storage.BucketTargetOption; import java.io.IOException; @@ -39,87 +37,25 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; -import java.util.Iterator; import java.util.List; import java.util.Objects; +import java.util.Set; /** * A Google cloud storage bucket. * *

    Objects of this class are immutable. Operations that modify the bucket like {@link #update} * return a new object. To get a {@code Bucket} object with the most recent information use - * {@link #reload}. + * {@link #reload}. {@code Bucket} adds a layer of service-related functionality over + * {@link BucketInfo}. *

    */ -public final class Bucket { +public final class Bucket extends BucketInfo { - private final Storage storage; - private final BucketInfo info; + private static final long serialVersionUID = 8574601739542252586L; - private static class BlobPageFetcher implements PageImpl.NextPageFetcher { - - private static final long serialVersionUID = 3221100177471323801L; - - private final StorageOptions options; - private final Page infoPage; - - BlobPageFetcher(StorageOptions options, Page infoPage) { - this.options = options; - this.infoPage = infoPage; - } - - @Override - public Page nextPage() { - Page nextInfoPage = infoPage.nextPage(); - return new PageImpl<>(new BlobPageFetcher(options, nextInfoPage), - nextInfoPage.nextPageCursor(), new LazyBlobIterable(options, nextInfoPage.values())); - } - } - - private static class LazyBlobIterable implements Iterable, Serializable { - - private static final long serialVersionUID = -3092290247725378832L; - - private final StorageOptions options; - private final Iterable infoIterable; - private transient Storage storage; - - public LazyBlobIterable(StorageOptions options, Iterable infoIterable) { - this.options = options; - this.infoIterable = infoIterable; - this.storage = options.service(); - } - - private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { - in.defaultReadObject(); - this.storage = options.service(); - } - - @Override - public Iterator iterator() { - return Iterators.transform(infoIterable.iterator(), new Function() { - @Override - public Blob apply(BlobInfo blobInfo) { - return new Blob(storage, blobInfo); - } - }); - } - - @Override - public int hashCode() { - return Objects.hash(options, infoIterable); - } - - @Override - public boolean equals(Object obj) { - if (!(obj instanceof LazyBlobIterable)) { - return false; - } - LazyBlobIterable other = (LazyBlobIterable) obj; - return Objects.equals(options, other.options) - && Objects.equals(infoIterable, other.infoIterable); - } - } + private final StorageOptions options; + private transient Storage storage; /** * Class for specifying bucket source options when {@code Bucket} methods are used. @@ -132,7 +68,7 @@ private BucketSourceOption(StorageRpc.Option rpcOption) { super(rpcOption, null); } - private Storage.BucketSourceOption toSourceOptions(BucketInfo bucketInfo) { + private Storage.BucketSourceOption toSourceOption(BucketInfo bucketInfo) { switch (rpcOption()) { case IF_METAGENERATION_MATCH: return Storage.BucketSourceOption.metagenerationMatch(bucketInfo.metageneration()); @@ -176,7 +112,7 @@ static Storage.BucketSourceOption[] toSourceOptions(BucketInfo bucketInfo, new Storage.BucketSourceOption[options.length]; int index = 0; for (BucketSourceOption option : options) { - convertedOptions[index++] = option.toSourceOptions(bucketInfo); + convertedOptions[index++] = option.toSourceOption(bucketInfo); } return convertedOptions; } @@ -193,37 +129,404 @@ static Storage.BucketGetOption[] toGetOptions(BucketInfo bucketInfo, } /** - * Constructs a {@code Bucket} object for the provided {@code BucketInfo}. The storage service is - * used to issue requests. - * - * @param storage the storage service used for issuing requests - * @param info bucket's info + * Class for specifying blob target options when {@code Bucket} methods are used. */ - public Bucket(Storage storage, BucketInfo info) { - this.storage = checkNotNull(storage); - this.info = checkNotNull(info); + public static class BlobTargetOption extends Option { + + private static final Function TO_ENUM = + new Function() { + @Override + public StorageRpc.Option apply(BlobTargetOption blobTargetOption) { + return blobTargetOption.rpcOption(); + } + }; + private static final long serialVersionUID = 8345296337342509425L; + + private BlobTargetOption(StorageRpc.Option rpcOption, Object value) { + super(rpcOption, value); + } + + private StorageRpc.Tuple toTargetOption(BlobInfo blobInfo) { + BlobId blobId = blobInfo.blobId(); + switch (rpcOption()) { + case PREDEFINED_ACL: + return StorageRpc.Tuple.of(blobInfo, + Storage.BlobTargetOption.predefinedAcl((Storage.PredefinedAcl) value())); + case IF_GENERATION_MATCH: + blobId = BlobId.of(blobId.bucket(), blobId.name(), (Long) value()); + return StorageRpc.Tuple.of(blobInfo.toBuilder().blobId(blobId).build(), + Storage.BlobTargetOption.generationMatch()); + case IF_GENERATION_NOT_MATCH: + blobId = BlobId.of(blobId.bucket(), blobId.name(), (Long) value()); + return StorageRpc.Tuple.of(blobInfo.toBuilder().blobId(blobId).build(), + Storage.BlobTargetOption.generationNotMatch()); + case IF_METAGENERATION_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().metageneration((Long) value()).build(), + Storage.BlobTargetOption.metagenerationMatch()); + case IF_METAGENERATION_NOT_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().metageneration((Long) value()).build(), + Storage.BlobTargetOption.metagenerationNotMatch()); + default: + throw new AssertionError("Unexpected enum value"); + } + } + + /** + * Returns an option for specifying blob's predefined ACL configuration. + */ + public static BlobTargetOption predefinedAcl(Storage.PredefinedAcl acl) { + return new BlobTargetOption(StorageRpc.Option.PREDEFINED_ACL, acl); + } + + /** + * Returns an option that causes an operation to succeed only if the target blob does not exist. + * This option can not be provided together with {@link #generationMatch(long)} or + * {@link #generationNotMatch(long)}. + */ + public static BlobTargetOption doesNotExist() { + return new BlobTargetOption(StorageRpc.Option.IF_GENERATION_MATCH, 0L); + } + + /** + * Returns an option for blob's data generation match. If this option is used the request will + * fail if generation does not match the provided value. This option can not be provided + * together with {@link #generationNotMatch(long)} or {@link #doesNotExist()}. + */ + public static BlobTargetOption generationMatch(long generation) { + return new BlobTargetOption(StorageRpc.Option.IF_GENERATION_MATCH, generation); + } + + /** + * Returns an option for blob's data generation mismatch. If this option is used the request + * will fail if blob's generation matches the provided value. This option can not be provided + * together with {@link #generationMatch(long)} or {@link #doesNotExist()}. + */ + public static BlobTargetOption generationNotMatch(long generation) { + return new BlobTargetOption(StorageRpc.Option.IF_GENERATION_NOT_MATCH, generation); + } + + /** + * Returns an option for blob's metageneration match. If this option is used the request will + * fail if metageneration does not match the provided value. This option can not be provided + * together with {@link #metagenerationNotMatch(long)}. + */ + public static BlobTargetOption metagenerationMatch(long metageneration) { + return new BlobTargetOption(StorageRpc.Option.IF_METAGENERATION_MATCH, metageneration); + } + + /** + * Returns an option for blob's metageneration mismatch. If this option is used the request will + * fail if metageneration matches the provided value. This option can not be provided together + * with {@link #metagenerationMatch(long)}. + */ + public static BlobTargetOption metagenerationNotMatch(long metageneration) { + return new BlobTargetOption(StorageRpc.Option.IF_METAGENERATION_NOT_MATCH, metageneration); + } + + static StorageRpc.Tuple toTargetOptions( + BlobInfo info, BlobTargetOption... options) { + Set optionSet = + Sets.immutableEnumSet(Lists.transform(Arrays.asList(options), TO_ENUM)); + checkArgument(!(optionSet.contains(StorageRpc.Option.IF_METAGENERATION_NOT_MATCH) + && optionSet.contains(StorageRpc.Option.IF_METAGENERATION_MATCH)), + "metagenerationMatch and metagenerationNotMatch options can not be both provided"); + checkArgument(!(optionSet.contains(StorageRpc.Option.IF_GENERATION_NOT_MATCH) + && optionSet.contains(StorageRpc.Option.IF_GENERATION_MATCH)), + "Only one option of generationMatch, doesNotExist or generationNotMatch can be provided"); + Storage.BlobTargetOption[] convertedOptions = new Storage.BlobTargetOption[options.length]; + BlobInfo targetInfo = info; + int index = 0; + for (BlobTargetOption option : options) { + StorageRpc.Tuple target = + option.toTargetOption(targetInfo); + targetInfo = target.x(); + convertedOptions[index++] = target.y(); + } + return StorageRpc.Tuple.of(targetInfo, convertedOptions); + } } /** - * Creates a {@code Bucket} object for the provided bucket name. Performs an RPC call to get the - * latest bucket information. - * - * @param storage the storage service used for issuing requests - * @param bucket bucket's name - * @param options blob get options - * @return the {@code Bucket} object or {@code null} if not found - * @throws StorageException upon failure + * Class for specifying blob write options when {@code Bucket} methods are used. */ - public static Bucket get(Storage storage, String bucket, Storage.BucketGetOption... options) { - BucketInfo info = storage.get(bucket, options); - return info != null ? new Bucket(storage, info) : null; + public static class BlobWriteOption implements Serializable { + + private static final Function TO_ENUM = + new Function() { + @Override + public Storage.BlobWriteOption.Option apply(BlobWriteOption blobWriteOption) { + return blobWriteOption.option; + } + }; + private static final long serialVersionUID = 4722190734541993114L; + + private final Storage.BlobWriteOption.Option option; + private final Object value; + + private StorageRpc.Tuple toWriteOption(BlobInfo blobInfo) { + BlobId blobId = blobInfo.blobId(); + switch (option) { + case PREDEFINED_ACL: + return StorageRpc.Tuple.of(blobInfo, + Storage.BlobWriteOption.predefinedAcl((Storage.PredefinedAcl) value)); + case IF_GENERATION_MATCH: + blobId = BlobId.of(blobId.bucket(), blobId.name(), (Long) value); + return StorageRpc.Tuple.of(blobInfo.toBuilder().blobId(blobId).build(), + Storage.BlobWriteOption.generationMatch()); + case IF_GENERATION_NOT_MATCH: + blobId = BlobId.of(blobId.bucket(), blobId.name(), (Long) value); + return StorageRpc.Tuple.of(blobInfo.toBuilder().blobId(blobId).build(), + Storage.BlobWriteOption.generationNotMatch()); + case IF_METAGENERATION_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().metageneration((Long) value).build(), + Storage.BlobWriteOption.metagenerationMatch()); + case IF_METAGENERATION_NOT_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().metageneration((Long) value).build(), + Storage.BlobWriteOption.metagenerationNotMatch()); + case IF_MD5_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().md5((String) value).build(), + Storage.BlobWriteOption.md5Match()); + case IF_CRC32C_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().crc32c((String) value).build(), + Storage.BlobWriteOption.crc32cMatch()); + default: + throw new AssertionError("Unexpected enum value"); + } + } + + private BlobWriteOption(Storage.BlobWriteOption.Option option, Object value) { + this.option = option; + this.value = value; + } + + @Override + public int hashCode() { + return Objects.hash(option, value); + } + + @Override + public boolean equals(Object obj) { + if (obj == null) { + return false; + } + if (!(obj instanceof BlobWriteOption)) { + return false; + } + final BlobWriteOption other = (BlobWriteOption) obj; + return this.option == other.option && Objects.equals(this.value, other.value); + } + + /** + * Returns an option for specifying blob's predefined ACL configuration. + */ + public static BlobWriteOption predefinedAcl(Storage.PredefinedAcl acl) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.PREDEFINED_ACL, acl); + } + + /** + * Returns an option that causes an operation to succeed only if the target blob does not exist. + * This option can not be provided together with {@link #generationMatch(long)} or + * {@link #generationNotMatch(long)}. + */ + public static BlobWriteOption doesNotExist() { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_GENERATION_MATCH, 0L); + } + + /** + * Returns an option for blob's data generation match. If this option is used the request will + * fail if generation does not match the provided value. This option can not be provided + * together with {@link #generationNotMatch(long)} or {@link #doesNotExist()}. + */ + public static BlobWriteOption generationMatch(long generation) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_GENERATION_MATCH, generation); + } + + /** + * Returns an option for blob's data generation mismatch. If this option is used the request + * will fail if generation matches the provided value. This option can not be provided + * together with {@link #generationMatch(long)} or {@link #doesNotExist()}. + */ + public static BlobWriteOption generationNotMatch(long generation) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_GENERATION_NOT_MATCH, + generation); + } + + /** + * Returns an option for blob's metageneration match. If this option is used the request will + * fail if metageneration does not match the provided value. This option can not be provided + * together with {@link #metagenerationNotMatch(long)}. + */ + public static BlobWriteOption metagenerationMatch(long metageneration) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_METAGENERATION_MATCH, + metageneration); + } + + /** + * Returns an option for blob's metageneration mismatch. If this option is used the request will + * fail if metageneration matches the provided value. This option can not be provided together + * with {@link #metagenerationMatch(long)}. + */ + public static BlobWriteOption metagenerationNotMatch(long metageneration) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_METAGENERATION_NOT_MATCH, + metageneration); + } + + /** + * Returns an option for blob's data MD5 hash match. If this option is used the request will + * fail if blobs' data MD5 hash does not match the provided value. + */ + public static BlobWriteOption md5Match(String md5) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_MD5_MATCH, md5); + } + + /** + * Returns an option for blob's data CRC32C checksum match. If this option is used the request + * will fail if blobs' data CRC32C checksum does not match the provided value. + */ + public static BlobWriteOption crc32cMatch(String crc32c) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_CRC32C_MATCH, crc32c); + } + + static StorageRpc.Tuple toWriteOptions( + BlobInfo info, BlobWriteOption... options) { + Set optionSet = + Sets.immutableEnumSet(Lists.transform(Arrays.asList(options), TO_ENUM)); + checkArgument(!(optionSet.contains(Storage.BlobWriteOption.Option.IF_METAGENERATION_NOT_MATCH) + && optionSet.contains(Storage.BlobWriteOption.Option.IF_METAGENERATION_MATCH)), + "metagenerationMatch and metagenerationNotMatch options can not be both provided"); + checkArgument(!(optionSet.contains(Storage.BlobWriteOption.Option.IF_GENERATION_NOT_MATCH) + && optionSet.contains(Storage.BlobWriteOption.Option.IF_GENERATION_MATCH)), + "Only one option of generationMatch, doesNotExist or generationNotMatch can be provided"); + Storage.BlobWriteOption[] convertedOptions = new Storage.BlobWriteOption[options.length]; + BlobInfo writeInfo = info; + int index = 0; + for (BlobWriteOption option : options) { + StorageRpc.Tuple write = option.toWriteOption(writeInfo); + writeInfo = write.x(); + convertedOptions[index++] = write.y(); + } + return StorageRpc.Tuple.of(writeInfo, convertedOptions); + } } /** - * Returns the bucket's information. + * Builder for {@code Bucket}. */ - public BucketInfo info() { - return info; + public static class Builder extends BucketInfo.Builder { + private final Storage storage; + private final BucketInfo.BuilderImpl infoBuilder; + + Builder(Bucket bucket) { + this.storage = bucket.storage; + this.infoBuilder = new BucketInfo.BuilderImpl(bucket); + } + + @Override + public Builder name(String name) { + infoBuilder.name(name); + return this; + } + + @Override + Builder id(String id) { + infoBuilder.id(id); + return this; + } + + @Override + Builder owner(Acl.Entity owner) { + infoBuilder.owner(owner); + return this; + } + + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + public Builder versioningEnabled(Boolean enable) { + infoBuilder.versioningEnabled(enable); + return this; + } + + @Override + public Builder indexPage(String indexPage) { + infoBuilder.indexPage(indexPage); + return this; + } + + @Override + public Builder notFoundPage(String notFoundPage) { + infoBuilder.notFoundPage(notFoundPage); + return this; + } + + @Override + public Builder deleteRules(Iterable rules) { + infoBuilder.deleteRules(rules); + return this; + } + + @Override + public Builder storageClass(String storageClass) { + infoBuilder.storageClass(storageClass); + return this; + } + + @Override + public Builder location(String location) { + infoBuilder.location(location); + return this; + } + + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; + } + + @Override + Builder createTime(Long createTime) { + infoBuilder.createTime(createTime); + return this; + } + + @Override + Builder metageneration(Long metageneration) { + infoBuilder.metageneration(metageneration); + return this; + } + + @Override + public Builder cors(Iterable cors) { + infoBuilder.cors(cors); + return this; + } + + @Override + public Builder acl(Iterable acl) { + infoBuilder.acl(acl); + return this; + } + + @Override + public Builder defaultAcl(Iterable acl) { + infoBuilder.defaultAcl(acl); + return this; + } + + @Override + public Bucket build() { + return new Bucket(storage, infoBuilder); + } + } + + Bucket(Storage storage, BucketInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.storage = checkNotNull(storage); + this.options = storage.options(); } /** @@ -234,9 +537,9 @@ public BucketInfo info() { */ public boolean exists(BucketSourceOption... options) { int length = options.length; - Storage.BucketGetOption[] getOptions = Arrays.copyOf(toGetOptions(info, options), length + 1); + Storage.BucketGetOption[] getOptions = Arrays.copyOf(toGetOptions(this, options), length + 1); getOptions[length] = Storage.BucketGetOption.fields(); - return storage.get(info.name(), getOptions) != null; + return storage.get(name(), getOptions) != null; } /** @@ -247,7 +550,7 @@ public boolean exists(BucketSourceOption... options) { * @throws StorageException upon failure */ public Bucket reload(BucketSourceOption... options) { - return Bucket.get(storage, info.name(), toGetOptions(info, options)); + return storage.get(name(), toGetOptions(this, options)); } /** @@ -255,16 +558,14 @@ public Bucket reload(BucketSourceOption... options) { * is returned. By default no checks are made on the metadata generation of the current bucket. * If you want to update the information only if the current bucket metadata are at their latest * version use the {@code metagenerationMatch} option: - * {@code bucket.update(newInfo, BucketTargetOption.metagenerationMatch())} + * {@code bucket.update(BucketTargetOption.metagenerationMatch())} * - * @param bucketInfo new bucket's information. Name must match the one of the current bucket * @param options update options * @return a {@code Bucket} object with updated information * @throws StorageException upon failure */ - public Bucket update(BucketInfo bucketInfo, BucketTargetOption... options) { - checkArgument(Objects.equals(bucketInfo.name(), info.name()), "Bucket name must match"); - return new Bucket(storage, storage.update(bucketInfo, options)); + public Bucket update(BucketTargetOption... options) { + return storage.update(this, options); } /** @@ -275,36 +576,33 @@ public Bucket update(BucketInfo bucketInfo, BucketTargetOption... options) { * @throws StorageException upon failure */ public boolean delete(BucketSourceOption... options) { - return storage.delete(info.name(), toSourceOptions(info, options)); + return storage.delete(name(), toSourceOptions(this, options)); } /** * Returns the paginated list of {@code Blob} in this bucket. - * + * * @param options options for listing blobs * @throws StorageException upon failure */ public Page list(Storage.BlobListOption... options) { - Page infoPage = storage.list(info.name(), options); - StorageOptions storageOptions = storage.options(); - return new PageImpl<>(new BlobPageFetcher(storageOptions, infoPage), infoPage.nextPageCursor(), - new LazyBlobIterable(storageOptions, infoPage.values())); + return storage.list(name(), options); } /** * Returns the requested blob in this bucket or {@code null} if not found. - * + * * @param blob name of the requested blob * @param options blob search options * @throws StorageException upon failure */ public Blob get(String blob, BlobGetOption... options) { - return new Blob(storage, storage.get(BlobId.of(info.name(), blob), options)); + return storage.get(BlobId.of(name(), blob), options); } /** * Returns a list of requested blobs in this bucket. Blobs that do not exist are null. - * + * * @param blobName1 first blob to get * @param blobName2 second blob to get * @param blobNames other blobs to get @@ -313,16 +611,16 @@ public Blob get(String blob, BlobGetOption... options) { */ public List get(String blobName1, String blobName2, String... blobNames) { BatchRequest.Builder batch = BatchRequest.builder(); - batch.get(info.name(), blobName1); - batch.get(info.name(), blobName2); + batch.get(name(), blobName1); + batch.get(name(), blobName2); for (String name : blobNames) { - batch.get(info.name(), name); + batch.get(name(), name); } List blobs = new ArrayList<>(blobNames.length); BatchResponse response = storage.submit(batch.build()); - for (BatchResponse.Result result : response.gets()) { + for (BatchResponse.Result result : response.gets()) { BlobInfo blobInfo = result.get(); - blobs.add(blobInfo != null ? new Blob(storage, blobInfo) : null); + blobs.add(blobInfo != null ? new Blob(storage, new BlobInfo.BuilderImpl(blobInfo)) : null); } return Collections.unmodifiableList(blobs); } @@ -332,7 +630,7 @@ public List get(String blobName1, String blobName2, String... blobNames) { * For large content, {@link Blob#writer(com.google.gcloud.storage.Storage.BlobWriteOption...)} * is recommended as it uses resumable upload. MD5 and CRC32C hashes of {@code content} are * computed and used for validating transferred data. - * + * * @param blob a blob name * @param content the blob content * @param contentType the blob content type. If {@code null} then @@ -342,16 +640,18 @@ public List get(String blobName1, String blobName2, String... blobNames) { * @throws StorageException upon failure */ public Blob create(String blob, byte[] content, String contentType, BlobTargetOption... options) { - BlobInfo blobInfo = BlobInfo.builder(BlobId.of(info.name(), blob)) + BlobInfo blobInfo = BlobInfo.builder(BlobId.of(name(), blob)) .contentType(MoreObjects.firstNonNull(contentType, Storage.DEFAULT_CONTENT_TYPE)).build(); - return new Blob(storage, storage.create(blobInfo, content, options)); + StorageRpc.Tuple target = + BlobTargetOption.toTargetOptions(blobInfo, options); + return storage.create(target.x(), content, target.y()); } /** * Creates a new blob in this bucket. Direct upload is used to upload {@code content}. * For large content, {@link Blob#writer(com.google.gcloud.storage.Storage.BlobWriteOption...)} * is recommended as it uses resumable upload. - * + * * @param blob a blob name * @param content the blob content as a stream * @param contentType the blob content type. If {@code null} then @@ -362,9 +662,11 @@ public Blob create(String blob, byte[] content, String contentType, BlobTargetOp */ public Blob create(String blob, InputStream content, String contentType, BlobWriteOption... options) { - BlobInfo blobInfo = BlobInfo.builder(BlobId.of(info.name(), blob)) + BlobInfo blobInfo = BlobInfo.builder(BlobId.of(name(), blob)) .contentType(MoreObjects.firstNonNull(contentType, Storage.DEFAULT_CONTENT_TYPE)).build(); - return new Blob(storage, storage.create(blobInfo, content, options)); + StorageRpc.Tuple write = + BlobWriteOption.toWriteOptions(blobInfo, options); + return storage.create(write.x(), content, write.y()); } /** @@ -373,4 +675,29 @@ public Blob create(String blob, InputStream content, String contentType, public Storage storage() { return storage; } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof Bucket && Objects.equals(toPb(), ((Bucket) obj).toPb()) + && Objects.equals(options, ((Bucket) obj).options); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); + } + + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.storage = options.service(); + } + + static Bucket fromPb(Storage storage, com.google.api.services.storage.model.Bucket bucketPb) { + return new Bucket(storage, new BucketInfo.BuilderImpl(BucketInfo.fromPb(bucketPb))); + } } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BucketInfo.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BucketInfo.java index 62fbf9c6521f..a1de1a07e03e 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BucketInfo.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BucketInfo.java @@ -16,8 +16,8 @@ package com.google.gcloud.storage; -import static com.google.api.client.repackaged.com.google.common.base.Preconditions.checkNotNull; import static com.google.common.base.MoreObjects.firstNonNull; +import static com.google.common.base.Preconditions.checkNotNull; import static com.google.common.collect.Lists.transform; import com.google.api.client.json.jackson2.JacksonFactory; @@ -48,7 +48,7 @@ * @see Concepts and * Terminology */ -public final class BucketInfo implements Serializable { +public class BucketInfo implements Serializable { static final Function FROM_PB_FUNCTION = new Function() { @@ -317,7 +317,99 @@ void populateCondition(Rule.Condition condition) { } } - public static final class Builder { + /** + * Builder for {@code BucketInfo}. + */ + public abstract static class Builder { + /** + * Sets the bucket's name. + */ + public abstract Builder name(String name); + + abstract Builder id(String id); + + abstract Builder owner(Acl.Entity owner); + + abstract Builder selfLink(String selfLink); + + /** + * Sets whether versioning should be enabled for this bucket. When set to true, versioning is + * fully enabled. + */ + public abstract Builder versioningEnabled(Boolean enable); + + /** + * Sets the bucket's website index page. Behaves as the bucket's directory index where missing + * blobs are treated as potential directories. + */ + public abstract Builder indexPage(String indexPage); + + /** + * Sets the custom object to return when a requested resource is not found. + */ + public abstract Builder notFoundPage(String notFoundPage); + + /** + * Sets the bucket's lifecycle configuration as a number of delete rules. + * + * @see Lifecycle Management + */ + public abstract Builder deleteRules(Iterable rules); + + /** + * Sets the bucket's storage class. This defines how blobs in the bucket are stored and + * determines the SLA and the cost of storage. A list of supported values is available + * here. + */ + public abstract Builder storageClass(String storageClass); + + /** + * Sets the bucket's location. Data for blobs in the bucket resides in physical storage within + * this region. A list of supported values is available + * here. + */ + public abstract Builder location(String location); + + abstract Builder etag(String etag); + + abstract Builder createTime(Long createTime); + + abstract Builder metageneration(Long metageneration); + + /** + * Sets the bucket's Cross-Origin Resource Sharing (CORS) configuration. + * + * @see + * Cross-Origin Resource Sharing (CORS) + */ + public abstract Builder cors(Iterable cors); + + /** + * Sets the bucket's access control configuration. + * + * @see + * About Access Control Lists + */ + public abstract Builder acl(Iterable acl); + + /** + * Sets the default access control configuration to apply to bucket's blobs when no other + * configuration is specified. + * + * @see + * About Access Control Lists + */ + public abstract Builder defaultAcl(Iterable acl); + + /** + * Creates a {@code BucketInfo} object. + */ + public abstract BucketInfo build(); + } + + static final class BuilderImpl extends Builder { private String id; private String name; @@ -336,9 +428,11 @@ public static final class Builder { private List acl; private List defaultAcl; - private Builder() {} + BuilderImpl(String name) { + this.name = name; + } - private Builder(BucketInfo bucketInfo) { + BuilderImpl(BucketInfo bucketInfo) { id = bucketInfo.id; name = bucketInfo.name; etag = bucketInfo.etag; @@ -357,144 +451,110 @@ private Builder(BucketInfo bucketInfo) { deleteRules = bucketInfo.deleteRules; } - /** - * Sets the bucket's name. - */ + @Override public Builder name(String name) { this.name = checkNotNull(name); return this; } + @Override Builder id(String id) { this.id = id; return this; } + @Override Builder owner(Acl.Entity owner) { this.owner = owner; return this; } + @Override Builder selfLink(String selfLink) { this.selfLink = selfLink; return this; } - /** - * Sets whether versioning should be enabled for this bucket. When set to true, versioning is - * fully enabled. - */ + @Override public Builder versioningEnabled(Boolean enable) { this.versioningEnabled = firstNonNull(enable, Data.nullOf(Boolean.class)); return this; } - /** - * Sets the bucket's website index page. Behaves as the bucket's directory index where missing - * blobs are treated as potential directories. - */ + @Override public Builder indexPage(String indexPage) { this.indexPage = indexPage; return this; } - /** - * Sets the custom object to return when a requested resource is not found. - */ + @Override public Builder notFoundPage(String notFoundPage) { this.notFoundPage = notFoundPage; return this; } - /** - * Sets the bucket's lifecycle configuration as a number of delete rules. - * - * @see Lifecycle Management - */ + @Override public Builder deleteRules(Iterable rules) { this.deleteRules = rules != null ? ImmutableList.copyOf(rules) : null; return this; } - /** - * Sets the bucket's storage class. This defines how blobs in the bucket are stored and - * determines the SLA and the cost of storage. A list of supported values is available - * here. - */ + @Override public Builder storageClass(String storageClass) { this.storageClass = storageClass; return this; } - /** - * Sets the bucket's location. Data for blobs in the bucket resides in physical storage within - * this region. A list of supported values is available - * here. - */ + @Override public Builder location(String location) { this.location = location; return this; } + @Override Builder etag(String etag) { this.etag = etag; return this; } + @Override Builder createTime(Long createTime) { this.createTime = createTime; return this; } + @Override Builder metageneration(Long metageneration) { this.metageneration = metageneration; return this; } - /** - * Sets the bucket's Cross-Origin Resource Sharing (CORS) configuration. - * - * @see - * Cross-Origin Resource Sharing (CORS) - */ + @Override public Builder cors(Iterable cors) { this.cors = cors != null ? ImmutableList.copyOf(cors) : null; return this; } - /** - * Sets the bucket's access control configuration. - * - * @see - * About Access Control Lists - */ + @Override public Builder acl(Iterable acl) { this.acl = acl != null ? ImmutableList.copyOf(acl) : null; return this; } - /** - * Sets the default access control configuration to apply to bucket's blobs when no other - * configuration is specified. - * - * @see - * About Access Control Lists - */ + @Override public Builder defaultAcl(Iterable acl) { this.defaultAcl = acl != null ? ImmutableList.copyOf(acl) : null; return this; } - /** - * Creates a {@code BucketInfo} object. - */ + @Override public BucketInfo build() { checkNotNull(name); return new BucketInfo(this); } } - private BucketInfo(Builder builder) { + BucketInfo(BuilderImpl builder) { id = builder.id; name = builder.name; etag = builder.etag; @@ -649,7 +709,7 @@ public List defaultAcl() { * Returns a builder for the current bucket. */ public Builder toBuilder() { - return new Builder(this); + return new BuilderImpl(this); } @Override @@ -659,7 +719,8 @@ public int hashCode() { @Override public boolean equals(Object obj) { - return obj instanceof BucketInfo && Objects.equals(toPb(), ((BucketInfo) obj).toPb()); + return obj != null && obj.getClass().equals(BucketInfo.class) + && Objects.equals(toPb(), ((BucketInfo) obj).toPb()); } @Override @@ -743,11 +804,11 @@ public static BucketInfo of(String name) { * Returns a {@code BucketInfo} builder where the bucket's name is set to the provided name. */ public static Builder builder(String name) { - return new Builder().name(name); + return new BuilderImpl(name); } static BucketInfo fromPb(com.google.api.services.storage.model.Bucket bucketPb) { - Builder builder = new Builder().name(bucketPb.getName()); + Builder builder = new BuilderImpl(bucketPb.getName()); if (bucketPb.getId() != null) { builder.id(bucketPb.getId()); } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/CopyWriter.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/CopyWriter.java index 1e5427a847d4..7eb91d0910a2 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/CopyWriter.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/CopyWriter.java @@ -57,9 +57,10 @@ public class CopyWriter implements Restorable { * is {@code false} will block until all pending chunks are copied. * *

    This method has the same effect of doing: - *

        {@code while (!copyWriter.isDone()) {
    -   *        copyWriter.copyChunk();
    -   *    }}
    +   * 
     {@code
    +   * while (!copyWriter.isDone()) {
    +   *    copyWriter.copyChunk();
    +   * }}
        * 
    * * @throws StorageException upon failure diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java index 272c5fdef223..98f3450b7f10 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java @@ -655,7 +655,7 @@ public static BucketListOption prefix(String prefix) { */ public static BucketListOption fields(BucketField... fields) { StringBuilder builder = new StringBuilder(); - builder.append("items(").append(BucketField.selector(fields)).append(")"); + builder.append("items(").append(BucketField.selector(fields)).append("),nextPageToken"); return new BucketListOption(StorageRpc.Option.FIELDS, builder.toString()); } } @@ -700,6 +700,15 @@ public static BlobListOption recursive(boolean recursive) { return new BlobListOption(StorageRpc.Option.DELIMITER, recursive); } + /** + * If set to {@code true}, lists all versions of a blob. The default is {@code false}. + * + * @see Object Versioning + */ + public static BlobListOption versions(boolean versions) { + return new BlobListOption(StorageRpc.Option.VERSIONS, versions); + } + /** * Returns an option to specify the blob's fields to be returned by the RPC call. If this option * is not provided all blob's fields are returned. {@code BlobListOption.fields}) can be used to @@ -708,7 +717,7 @@ public static BlobListOption recursive(boolean recursive) { */ public static BlobListOption fields(BlobField... fields) { StringBuilder builder = new StringBuilder(); - builder.append("items(").append(BlobField.selector(fields)).append(")"); + builder.append("items(").append(BlobField.selector(fields)).append("),nextPageToken"); return new BlobListOption(StorageRpc.Option.FIELDS, builder.toString()); } } @@ -1208,82 +1217,83 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx /** * Create a new bucket. * - * @return a complete bucket information + * @return a complete bucket * @throws StorageException upon failure */ - BucketInfo create(BucketInfo bucketInfo, BucketTargetOption... options); + Bucket create(BucketInfo bucketInfo, BucketTargetOption... options); /** * Create a new blob with no content. * - * @return a complete blob information + * @return a [@code Blob} with complete information * @throws StorageException upon failure */ - BlobInfo create(BlobInfo blobInfo, BlobTargetOption... options); + Blob create(BlobInfo blobInfo, BlobTargetOption... options); /** * Create a new blob. Direct upload is used to upload {@code content}. For large content, * {@link #writer} is recommended as it uses resumable upload. MD5 and CRC32C hashes of * {@code content} are computed and used for validating transferred data. * - * @return a complete blob information + * @return a [@code Blob} with complete information * @throws StorageException upon failure * @see Hashes and ETags */ - BlobInfo create(BlobInfo blobInfo, byte[] content, BlobTargetOption... options); + Blob create(BlobInfo blobInfo, byte[] content, BlobTargetOption... options); /** * Create a new blob. Direct upload is used to upload {@code content}. For large content, * {@link #writer} is recommended as it uses resumable upload. By default any md5 and crc32c * values in the given {@code blobInfo} are ignored unless requested via the - * {@code BlobWriteOption.md5Match} and {@code BlobWriteOption.crc32cMatch} options. + * {@code BlobWriteOption.md5Match} and {@code BlobWriteOption.crc32cMatch} options. The given + * input stream is closed upon success. * - * @return a complete blob information + * @return a [@code Blob} with complete information * @throws StorageException upon failure */ - BlobInfo create(BlobInfo blobInfo, InputStream content, BlobWriteOption... options); + Blob create(BlobInfo blobInfo, InputStream content, BlobWriteOption... options); /** * Return the requested bucket or {@code null} if not found. * * @throws StorageException upon failure */ - BucketInfo get(String bucket, BucketGetOption... options); + Bucket get(String bucket, BucketGetOption... options); /** * Return the requested blob or {@code null} if not found. * * @throws StorageException upon failure */ - BlobInfo get(String bucket, String blob, BlobGetOption... options); + Blob get(String bucket, String blob, BlobGetOption... options); /** * Return the requested blob or {@code null} if not found. * * @throws StorageException upon failure */ - BlobInfo get(BlobId blob, BlobGetOption... options); + Blob get(BlobId blob, BlobGetOption... options); /** * Return the requested blob or {@code null} if not found. * * @throws StorageException upon failure */ - BlobInfo get(BlobId blob); + Blob get(BlobId blob); /** * List the project's buckets. * * @throws StorageException upon failure */ - Page list(BucketListOption... options); + Page list(BucketListOption... options); /** * List the bucket's blobs. * * @throws StorageException upon failure */ - Page list(String bucket, BlobListOption... options); + Page list(String bucket, BlobListOption... options); /** * Update bucket information. @@ -1291,7 +1301,7 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * @return the updated bucket * @throws StorageException upon failure */ - BucketInfo update(BucketInfo bucketInfo, BucketTargetOption... options); + Bucket update(BucketInfo bucketInfo, BucketTargetOption... options); /** * Update blob information. Original metadata are merged with metadata in the provided @@ -1299,14 +1309,16 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * can be done by setting the provided {@code blobInfo}'s metadata to {@code null}. * *

    Example usage of replacing blob's metadata: - *

        {@code service.update(BlobInfo.builder("bucket", "name").metadata(null).build());}
    -   *    {@code service.update(BlobInfo.builder("bucket", "name").metadata(newMetadata).build());}
    +   * 
     {@code
    +   * service.update(BlobInfo.builder("bucket", "name").metadata(null).build());
    +   * service.update(BlobInfo.builder("bucket", "name").metadata(newMetadata).build());
    +   * }
        * 
    * * @return the updated blob * @throws StorageException upon failure */ - BlobInfo update(BlobInfo blobInfo, BlobTargetOption... options); + Blob update(BlobInfo blobInfo, BlobTargetOption... options); /** * Update blob information. Original metadata are merged with metadata in the provided @@ -1314,14 +1326,16 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * can be done by setting the provided {@code blobInfo}'s metadata to {@code null}. * *

    Example usage of replacing blob's metadata: - *

        {@code service.update(BlobInfo.builder("bucket", "name").metadata(null).build());}
    -   *    {@code service.update(BlobInfo.builder("bucket", "name").metadata(newMetadata).build());}
    +   * 
     {@code
    +   * service.update(BlobInfo.builder("bucket", "name").metadata(null).build());
    +   * service.update(BlobInfo.builder("bucket", "name").metadata(newMetadata).build());
    +   * }
        * 
    * * @return the updated blob * @throws StorageException upon failure */ - BlobInfo update(BlobInfo blobInfo); + Blob update(BlobInfo blobInfo); /** * Delete the requested bucket. @@ -1361,7 +1375,7 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * @return the composed blob * @throws StorageException upon failure */ - BlobInfo compose(ComposeRequest composeRequest); + Blob compose(ComposeRequest composeRequest); /** * Sends a copy request. Returns a {@link CopyWriter} object for the provided @@ -1372,14 +1386,15 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * might issue multiple RPC calls depending on blob's size. * *

    Example usage of copy: - *

        {@code BlobInfo blob = service.copy(copyRequest).result();}
    +   * 
     {@code BlobInfo blob = service.copy(copyRequest).result();}
        * 
    * To explicitly issue chunk copy requests use {@link CopyWriter#copyChunk()} instead: - *
        {@code CopyWriter copyWriter = service.copy(copyRequest);
    -   *    while (!copyWriter.isDone()) {
    -   *        copyWriter.copyChunk();
    -   *    }
    -   *    BlobInfo blob = copyWriter.result();
    +   * 
     {@code
    +   * CopyWriter copyWriter = service.copy(copyRequest);
    +   * while (!copyWriter.isDone()) {
    +   *     copyWriter.copyChunk();
    +   * }
    +   * BlobInfo blob = copyWriter.result();
        * }
        * 
    * @@ -1461,12 +1476,12 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * accessible blobs, but don't want to require users to explicitly log in. * *

    Example usage of creating a signed URL that is valid for 2 weeks: - *

       {@code
    -   *     service.signUrl(BlobInfo.builder("bucket", "name").build(), 14, TimeUnit.DAYS);
    +   * 
     {@code
    +   * service.signUrl(BlobInfo.builder("bucket", "name").build(), 14, TimeUnit.DAYS);
        * }
    * * @param blobInfo the blob associated with the signed URL - * @param duration time until the signed URL expires, expressed in {@code unit}. The finer + * @param duration time until the signed URL expires, expressed in {@code unit}. The finest * granularity supported is 1 second, finer granularities will be truncated * @param unit time unit of the {@code duration} parameter * @param options optional URL signing options @@ -1478,11 +1493,11 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * Gets the requested blobs. A batch request is used to perform this call. * * @param blobIds blobs to get - * @return an immutable list of {@code BlobInfo} objects. If a blob does not exist or access to it + * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it * has been denied the corresponding item in the list is {@code null}. * @throws StorageException upon failure */ - List get(BlobId... blobIds); + List get(BlobId... blobIds); /** * Updates the requested blobs. A batch request is used to perform this call. Original metadata @@ -1492,11 +1507,11 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * {@link #update(com.google.gcloud.storage.BlobInfo)} for a code example. * * @param blobInfos blobs to update - * @return an immutable list of {@code BlobInfo} objects. If a blob does not exist or access to it + * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it * has been denied the corresponding item in the list is {@code null}. * @throws StorageException upon failure */ - List update(BlobInfo... blobInfos); + List update(BlobInfo... blobInfos); /** * Deletes the requested blobs. A batch request is used to perform this call. diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageException.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageException.java index 0c952c9a65d6..ee85b80d6e13 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageException.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageException.java @@ -33,7 +33,7 @@ */ public class StorageException extends BaseServiceException { - // see: https://cloud.google.com/storage/docs/concepts-techniques#practices + // see: https://cloud.google.com/storage/docs/resumable-uploads-xml#practices private static final Set RETRYABLE_ERRORS = ImmutableSet.of( new Error(504, null), new Error(503, null), diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageImpl.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageImpl.java index b6a833f26ab4..de77cba021a1 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageImpl.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageImpl.java @@ -33,7 +33,6 @@ import com.google.api.services.storage.model.StorageObject; import com.google.auth.oauth2.ServiceAccountCredentials; import com.google.common.base.Function; -import com.google.common.base.Functions; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Iterables; @@ -78,6 +77,14 @@ final class StorageImpl extends BaseService implements Storage { private static final String EMPTY_BYTE_ARRAY_MD5 = "1B2M2Y8AsgTpgAmY7PhCfg=="; private static final String EMPTY_BYTE_ARRAY_CRC32C = "AAAAAA=="; + private static final Function, Boolean> DELETE_FUNCTION = + new Function, Boolean>() { + @Override + public Boolean apply(Tuple tuple) { + return tuple.y(); + } + }; + private final StorageRpc storageRpc; StorageImpl(StorageOptions options) { @@ -86,11 +93,11 @@ final class StorageImpl extends BaseService implements Storage { } @Override - public BucketInfo create(BucketInfo bucketInfo, BucketTargetOption... options) { + public Bucket create(BucketInfo bucketInfo, BucketTargetOption... options) { final com.google.api.services.storage.model.Bucket bucketPb = bucketInfo.toPb(); final Map optionsMap = optionMap(bucketInfo, options); try { - return BucketInfo.fromPb(runWithRetries( + return Bucket.fromPb(this, runWithRetries( new Callable() { @Override public com.google.api.services.storage.model.Bucket call() { @@ -103,7 +110,7 @@ public com.google.api.services.storage.model.Bucket call() { } @Override - public BlobInfo create(BlobInfo blobInfo, BlobTargetOption... options) { + public Blob create(BlobInfo blobInfo, BlobTargetOption... options) { BlobInfo updatedInfo = blobInfo.toBuilder() .md5(EMPTY_BYTE_ARRAY_MD5) .crc32c(EMPTY_BYTE_ARRAY_CRC32C) @@ -112,7 +119,7 @@ public BlobInfo create(BlobInfo blobInfo, BlobTargetOption... options) { } @Override - public BlobInfo create(BlobInfo blobInfo, byte[] content, BlobTargetOption... options) { + public Blob create(BlobInfo blobInfo, byte[] content, BlobTargetOption... options) { content = firstNonNull(content, EMPTY_BYTE_ARRAY); BlobInfo updatedInfo = blobInfo.toBuilder() .md5(BaseEncoding.base64().encode(Hashing.md5().hashBytes(content).asBytes())) @@ -123,16 +130,16 @@ public BlobInfo create(BlobInfo blobInfo, byte[] content, BlobTargetOption... op } @Override - public BlobInfo create(BlobInfo blobInfo, InputStream content, BlobWriteOption... options) { + public Blob create(BlobInfo blobInfo, InputStream content, BlobWriteOption... options) { Tuple targetOptions = BlobTargetOption.convert(blobInfo, options); return create(targetOptions.x(), content, targetOptions.y()); } - private BlobInfo create(BlobInfo info, final InputStream content, BlobTargetOption... options) { + private Blob create(BlobInfo info, final InputStream content, BlobTargetOption... options) { final StorageObject blobPb = info.toPb(); final Map optionsMap = optionMap(info, options); try { - return BlobInfo.fromPb(runWithRetries(new Callable() { + return Blob.fromPb(this, runWithRetries(new Callable() { @Override public StorageObject call() { return storageRpc.create(blobPb, @@ -145,7 +152,7 @@ public StorageObject call() { } @Override - public BucketInfo get(String bucket, BucketGetOption... options) { + public Bucket get(String bucket, BucketGetOption... options) { final com.google.api.services.storage.model.Bucket bucketPb = BucketInfo.of(bucket).toPb(); final Map optionsMap = optionMap(options); try { @@ -156,19 +163,19 @@ public com.google.api.services.storage.model.Bucket call() { return storageRpc.get(bucketPb, optionsMap); } }, options().retryParams(), EXCEPTION_HANDLER); - return answer == null ? null : BucketInfo.fromPb(answer); + return answer == null ? null : Bucket.fromPb(this, answer); } catch (RetryHelperException e) { throw StorageException.translateAndThrow(e); } } @Override - public BlobInfo get(String bucket, String blob, BlobGetOption... options) { + public Blob get(String bucket, String blob, BlobGetOption... options) { return get(BlobId.of(bucket, blob), options); } @Override - public BlobInfo get(BlobId blob, BlobGetOption... options) { + public Blob get(BlobId blob, BlobGetOption... options) { final StorageObject storedObject = blob.toPb(); final Map optionsMap = optionMap(blob, options); try { @@ -178,18 +185,18 @@ public StorageObject call() { return storageRpc.get(storedObject, optionsMap); } }, options().retryParams(), EXCEPTION_HANDLER); - return storageObject == null ? null : BlobInfo.fromPb(storageObject); + return storageObject == null ? null : Blob.fromPb(this, storageObject); } catch (RetryHelperException e) { throw StorageException.translateAndThrow(e); } } @Override - public BlobInfo get(BlobId blob) { + public Blob get(BlobId blob) { return get(blob, new BlobGetOption[0]); } - private static class BucketPageFetcher implements NextPageFetcher { + private static class BucketPageFetcher implements NextPageFetcher { private static final long serialVersionUID = 5850406828803613729L; private final Map requestOptions; @@ -204,12 +211,12 @@ private static class BucketPageFetcher implements NextPageFetcher { } @Override - public Page nextPage() { + public Page nextPage() { return listBuckets(serviceOptions, requestOptions); } } - private static class BlobPageFetcher implements NextPageFetcher { + private static class BlobPageFetcher implements NextPageFetcher { private static final long serialVersionUID = 81807334445874098L; private final Map requestOptions; @@ -225,22 +232,22 @@ private static class BlobPageFetcher implements NextPageFetcher { } @Override - public Page nextPage() { + public Page nextPage() { return listBlobs(bucket, serviceOptions, requestOptions); } } @Override - public Page list(BucketListOption... options) { + public Page list(BucketListOption... options) { return listBuckets(options(), optionMap(options)); } @Override - public Page list(final String bucket, BlobListOption... options) { + public Page list(final String bucket, BlobListOption... options) { return listBlobs(bucket, options(), optionMap(options)); } - private static Page listBuckets(final StorageOptions serviceOptions, + private static Page listBuckets(final StorageOptions serviceOptions, final Map optionsMap) { try { Tuple> result = runWithRetries( @@ -251,22 +258,23 @@ public Tuple> cal } }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); - Iterable buckets = - result.y() == null ? ImmutableList.of() : Iterables.transform(result.y(), - new Function() { + Iterable buckets = + result.y() == null ? ImmutableList.of() : Iterables.transform(result.y(), + new Function() { @Override - public BucketInfo apply(com.google.api.services.storage.model.Bucket bucketPb) { - return BucketInfo.fromPb(bucketPb); + public Bucket apply(com.google.api.services.storage.model.Bucket bucketPb) { + return Bucket.fromPb(serviceOptions.service(), bucketPb); } }); - return new PageImpl<>(new BucketPageFetcher(serviceOptions, cursor, optionsMap), cursor, + return new PageImpl<>( + new BucketPageFetcher(serviceOptions, cursor, optionsMap), cursor, buckets); } catch (RetryHelperException e) { throw StorageException.translateAndThrow(e); } } - private static Page listBlobs(final String bucket, + private static Page listBlobs(final String bucket, final StorageOptions serviceOptions, final Map optionsMap) { try { Tuple> result = runWithRetries( @@ -277,15 +285,17 @@ public Tuple> call() { } }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); - Iterable blobs = - result.y() == null ? ImmutableList.of() : Iterables.transform(result.y(), - new Function() { + Iterable blobs = + result.y() == null + ? ImmutableList.of() + : Iterables.transform(result.y(), new Function() { @Override - public BlobInfo apply(StorageObject storageObject) { - return BlobInfo.fromPb(storageObject); + public Blob apply(StorageObject storageObject) { + return Blob.fromPb(serviceOptions.service(), storageObject); } }); - return new PageImpl<>(new BlobPageFetcher(bucket, serviceOptions, cursor, optionsMap), + return new PageImpl<>( + new BlobPageFetcher(bucket, serviceOptions, cursor, optionsMap), cursor, blobs); } catch (RetryHelperException e) { @@ -294,11 +304,11 @@ public BlobInfo apply(StorageObject storageObject) { } @Override - public BucketInfo update(BucketInfo bucketInfo, BucketTargetOption... options) { + public Bucket update(BucketInfo bucketInfo, BucketTargetOption... options) { final com.google.api.services.storage.model.Bucket bucketPb = bucketInfo.toPb(); final Map optionsMap = optionMap(bucketInfo, options); try { - return BucketInfo.fromPb(runWithRetries( + return Bucket.fromPb(this, runWithRetries( new Callable() { @Override public com.google.api.services.storage.model.Bucket call() { @@ -311,11 +321,11 @@ public com.google.api.services.storage.model.Bucket call() { } @Override - public BlobInfo update(BlobInfo blobInfo, BlobTargetOption... options) { + public Blob update(BlobInfo blobInfo, BlobTargetOption... options) { final StorageObject storageObject = blobInfo.toPb(); final Map optionsMap = optionMap(blobInfo, options); try { - return BlobInfo.fromPb(runWithRetries(new Callable() { + return Blob.fromPb(this, runWithRetries(new Callable() { @Override public StorageObject call() { return storageRpc.patch(storageObject, optionsMap); @@ -327,7 +337,7 @@ public StorageObject call() { } @Override - public BlobInfo update(BlobInfo blobInfo) { + public Blob update(BlobInfo blobInfo) { return update(blobInfo, new BlobTargetOption[0]); } @@ -374,7 +384,7 @@ public boolean delete(BlobId blob) { } @Override - public BlobInfo compose(final ComposeRequest composeRequest) { + public Blob compose(final ComposeRequest composeRequest) { final List sources = Lists.newArrayListWithCapacity(composeRequest.sourceBlobs().size()); for (ComposeRequest.SourceBlob sourceBlob : composeRequest.sourceBlobs()) { @@ -386,7 +396,7 @@ public BlobInfo compose(final ComposeRequest composeRequest) { final Map targetOptions = optionMap(composeRequest.target().generation(), composeRequest.target().metageneration(), composeRequest.targetOptions()); try { - return BlobInfo.fromPb(runWithRetries(new Callable() { + return Blob.fromPb(this, runWithRetries(new Callable() { @Override public StorageObject call() { return storageRpc.compose(sources, target, targetOptions); @@ -468,18 +478,19 @@ public BatchResponse submit(BatchRequest batchRequest) { } StorageRpc.BatchResponse response = storageRpc.batch(new StorageRpc.BatchRequest(toDelete, toUpdate, toGet)); - List> deletes = transformBatchResult( - toDelete, response.deletes, Functions.identity()); - List> updates = transformBatchResult( - toUpdate, response.updates, BlobInfo.FROM_PB_FUNCTION); - List> gets = transformBatchResult( - toGet, response.gets, BlobInfo.FROM_PB_FUNCTION); + List> deletes = + transformBatchResult(toDelete, response.deletes, DELETE_FUNCTION); + List> updates = + transformBatchResult(toUpdate, response.updates, Blob.BLOB_FROM_PB_FUNCTION); + List> gets = + transformBatchResult(toGet, response.gets, Blob.BLOB_FROM_PB_FUNCTION); return new BatchResponse(deletes, updates, gets); } private List> transformBatchResult( Iterable>> request, - Map> results, Function transform) { + Map> results, + Function, O> transform) { List> response = Lists.newArrayListWithCapacity(results.size()); for (Tuple tuple : request) { Tuple result = results.get(tuple.x()); @@ -489,7 +500,8 @@ private List> transformBatch response.add(new BatchResponse.Result(exception)); } else { response.add(object != null - ? BatchResponse.Result.of(transform.apply(object)) : BatchResponse.Result.empty()); + ? BatchResponse.Result.of(transform.apply(Tuple.of((Storage) this, object))) + : BatchResponse.Result.empty()); } } return response; @@ -587,7 +599,7 @@ public URL signUrl(BlobInfo blobInfo, long duration, TimeUnit unit, SignUrlOptio } @Override - public List get(BlobId... blobIds) { + public List get(BlobId... blobIds) { BatchRequest.Builder requestBuilder = BatchRequest.builder(); for (BlobId blob : blobIds) { requestBuilder.get(blob); @@ -597,7 +609,7 @@ public List get(BlobId... blobIds) { } @Override - public List update(BlobInfo... blobInfos) { + public List update(BlobInfo... blobInfos) { BatchRequest.Builder requestBuilder = BatchRequest.builder(); for (BlobInfo blobInfo : blobInfos) { requestBuilder.update(blobInfo); diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/package-info.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/package-info.java index fda14ea2e808..181e63b08d0b 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/package-info.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/package-info.java @@ -17,23 +17,32 @@ /** * A client to Google Cloud Storage. * - *

    Here's a simple usage example for using gcloud-java from App/Compute Engine: + *

    Here's a simple usage example for using gcloud-java from App/Compute Engine. This example + * shows how to create a Storage blob. For the complete source code see + * + * CreateBlob.java. *

     {@code
      * Storage storage = StorageOptions.defaultInstance().service();
      * BlobId blobId = BlobId.of("bucket", "blob_name");
    - * Blob blob = Blob.get(storage, blobId);
    - * if (blob == null) {
    - *   BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
    - *   storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
    - * } else {
    - *   System.out.println("Updating content for " + blobId.name());
    + * BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
    + * Blob blob = storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
    + * }
    + *

    + * This second example shows how to update the blob's content if the blob exists. For the complete + * source code see + * + * UpdateBlob.java. + *

     {@code
    + * Storage storage = StorageOptions.defaultInstance().service();
    + * BlobId blobId = BlobId.of("bucket", "blob_name");
    + * Blob blob = storage.get(blobId);
    + * if (blob != null) {
      *   byte[] prevContent = blob.content();
      *   System.out.println(new String(prevContent, UTF_8));
      *   WritableByteChannel channel = blob.writer();
      *   channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
      *   channel.close();
      * }}
    - * *

    When using gcloud-java from outside of App/Compute Engine, you have to specify a * project ID and diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/RemoteGcsHelper.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/RemoteGcsHelper.java index 024aa04eba1b..1287ede746d5 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/RemoteGcsHelper.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/RemoteGcsHelper.java @@ -20,6 +20,7 @@ import com.google.gcloud.RetryParams; import com.google.gcloud.storage.BlobInfo; import com.google.gcloud.storage.Storage; +import com.google.gcloud.storage.Storage.BlobListOption; import com.google.gcloud.storage.StorageException; import com.google.gcloud.storage.StorageOptions; @@ -173,8 +174,8 @@ public DeleteBucketTask(Storage storage, String bucket) { @Override public Boolean call() { while (true) { - for (BlobInfo info : storage.list(bucket).values()) { - storage.delete(bucket, info.name()); + for (BlobInfo info : storage.list(bucket, BlobListOption.versions(true)).values()) { + storage.delete(info.blobId()); } try { storage.delete(bucket); diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BatchResponseTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BatchResponseTest.java index 5985329e0183..eb45b8b51271 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BatchResponseTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BatchResponseTest.java @@ -22,22 +22,35 @@ import com.google.common.collect.ImmutableList; import com.google.gcloud.storage.BatchResponse.Result; +import org.easymock.EasyMock; +import org.junit.Before; import org.junit.Test; import java.util.List; public class BatchResponseTest { - private static final BlobInfo BLOB_INFO_1 = BlobInfo.builder("b", "o1").build(); - private static final BlobInfo BLOB_INFO_2 = BlobInfo.builder("b", "o2").build(); - private static final BlobInfo BLOB_INFO_3 = BlobInfo.builder("b", "o3").build(); + private Storage mockStorage; + private Blob blob1; + private Blob blob2; + private Blob blob3; + + @Before + public void setUp() { + mockStorage = EasyMock.createMock(Storage.class); + EasyMock.expect(mockStorage.options()).andReturn(null).times(3); + EasyMock.replay(mockStorage); + blob1 = new Blob(mockStorage, new BlobInfo.BuilderImpl(BlobInfo.builder("b", "o1").build())); + blob2 = new Blob(mockStorage, new BlobInfo.BuilderImpl(BlobInfo.builder("b", "o2").build())); + blob3 = new Blob(mockStorage, new BlobInfo.BuilderImpl(BlobInfo.builder("b", "o3").build())); + } @Test public void testBatchResponse() { List> deletes = ImmutableList.of(Result.of(true), Result.of(false)); - List> updates = - ImmutableList.of(Result.of(BLOB_INFO_1), Result.of(BLOB_INFO_2)); - List> gets = ImmutableList.of(Result.of(BLOB_INFO_2), Result.of(BLOB_INFO_3)); + List> updates = + ImmutableList.of(Result.of(blob1), Result.of(blob2)); + List> gets = ImmutableList.of(Result.of(blob2), Result.of(blob3)); BatchResponse response = new BatchResponse(deletes, updates, gets); assertEquals(deletes, response.deletes()); assertEquals(updates, response.updates()); @@ -47,14 +60,13 @@ public void testBatchResponse() { @Test public void testEquals() { List> deletes = ImmutableList.of(Result.of(true), Result.of(false)); - List> updates = - ImmutableList.of(Result.of(BLOB_INFO_1), Result.of(BLOB_INFO_2)); - List> gets = ImmutableList.of(Result.of(BLOB_INFO_2), Result.of(BLOB_INFO_3)); + List> updates = + ImmutableList.of(Result.of(blob1), Result.of(blob2)); + List> gets = ImmutableList.of(Result.of(blob2), Result.of(blob3)); List> otherDeletes = ImmutableList.of(Result.of(false), Result.of(true)); - List> otherUpdates = - ImmutableList.of(Result.of(BLOB_INFO_2), Result.of(BLOB_INFO_3)); - List> otherGets = - ImmutableList.of(Result.of(BLOB_INFO_1), Result.of(BLOB_INFO_2)); + List> otherUpdates = ImmutableList.of(Result.of(blob2), Result.of(blob3)); + List> otherGets = + ImmutableList.of(Result.of(blob1), Result.of(blob2)); BatchResponse response = new BatchResponse(deletes, updates, gets); BatchResponse responseEquals = new BatchResponse(deletes, updates, gets); BatchResponse responseNotEquals1 = new BatchResponse(otherDeletes, updates, gets); diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobReadChannelTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobReadChannelTest.java index ffb37e8c5032..5dc947df51f8 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobReadChannelTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobReadChannelTest.java @@ -39,6 +39,7 @@ import java.io.IOException; import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; import java.util.Arrays; import java.util.Map; import java.util.Random; @@ -156,15 +157,15 @@ public void testClose() { } @Test - public void testReadClosed() { + public void testReadClosed() throws IOException { replay(storageRpcMock); reader = new BlobReadChannel(options, BLOB_ID, EMPTY_RPC_OPTIONS); reader.close(); try { ByteBuffer readBuffer = ByteBuffer.allocate(DEFAULT_CHUNK_SIZE); reader.read(readBuffer); - fail("Expected BlobReadChannel read to throw IOException"); - } catch (IOException ex) { + fail("Expected BlobReadChannel read to throw ClosedChannelException"); + } catch (ClosedChannelException ex) { // expected } } diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobTest.java index 586e7fd0fd39..c7508593f8c9 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobTest.java @@ -16,22 +16,28 @@ package com.google.gcloud.storage; +import static com.google.gcloud.storage.Acl.Project.ProjectRole.VIEWERS; +import static com.google.gcloud.storage.Acl.Role.READER; +import static com.google.gcloud.storage.Acl.Role.WRITER; import static org.easymock.EasyMock.capture; import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; +import static org.easymock.EasyMock.eq; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertArrayEquals; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; -import com.google.api.client.util.Lists; +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; import com.google.gcloud.ReadChannel; +import com.google.gcloud.storage.Acl.Project; +import com.google.gcloud.storage.Acl.User; import com.google.gcloud.storage.Storage.CopyRequest; import org.easymock.Capture; @@ -40,25 +46,65 @@ import org.junit.Test; import java.net.URL; -import java.util.Arrays; import java.util.List; +import java.util.Map; import java.util.concurrent.TimeUnit; public class BlobTest { + private static final List ACL = ImmutableList.of( + Acl.of(User.ofAllAuthenticatedUsers(), READER), Acl.of(new Project(VIEWERS, "p1"), WRITER)); + private static final Integer COMPONENT_COUNT = 2; + private static final String CONTENT_TYPE = "text/html"; + private static final String CACHE_CONTROL = "cache"; + private static final String CONTENT_DISPOSITION = "content-disposition"; + private static final String CONTENT_ENCODING = "UTF-8"; + private static final String CONTENT_LANGUAGE = "En"; + private static final String CRC32 = "0xFF00"; + private static final Long DELETE_TIME = System.currentTimeMillis(); + private static final String ETAG = "0xFF00"; + private static final Long GENERATION = 1L; + private static final String ID = "B/N:1"; + private static final String MD5 = "0xFF00"; + private static final String MEDIA_LINK = "http://media/b/n"; + private static final Map METADATA = ImmutableMap.of("n1", "v1", "n2", "v2"); + private static final Long META_GENERATION = 10L; + private static final User OWNER = new User("user@gmail.com"); + private static final String SELF_LINK = "http://storage/b/n"; + private static final Long SIZE = 1024L; + private static final Long UPDATE_TIME = DELETE_TIME - 1L; + private static final BlobInfo FULL_BLOB_INFO = BlobInfo.builder("b", "n", GENERATION) + .acl(ACL) + .componentCount(COMPONENT_COUNT) + .contentType(CONTENT_TYPE) + .cacheControl(CACHE_CONTROL) + .contentDisposition(CONTENT_DISPOSITION) + .contentEncoding(CONTENT_ENCODING) + .contentLanguage(CONTENT_LANGUAGE) + .crc32c(CRC32) + .deleteTime(DELETE_TIME) + .etag(ETAG) + .id(ID) + .md5(MD5) + .mediaLink(MEDIA_LINK) + .metadata(METADATA) + .metageneration(META_GENERATION) + .owner(OWNER) + .selfLink(SELF_LINK) + .size(SIZE) + .updateTime(UPDATE_TIME) + .build(); private static final BlobInfo BLOB_INFO = BlobInfo.builder("b", "n").metageneration(42L).build(); - private static final BlobId[] BLOB_ID_ARRAY = {BlobId.of("b1", "n1"), - BlobId.of("b2", "n2"), BlobId.of("b3", "n3")}; - private static final BlobInfo[] BLOB_INFO_ARRAY = {BlobInfo.builder("b1", "n1").build(), - BlobInfo.builder("b2", "n2").build(), BlobInfo.builder("b3", "n3").build()}; private Storage storage; private Blob blob; + private Blob expectedBlob; + private Storage serviceMockReturnsOptions = createMock(Storage.class); + private StorageOptions mockOptions = createMock(StorageOptions.class); @Before - public void setUp() throws Exception { + public void setUp() { storage = createStrictMock(Storage.class); - blob = new Blob(storage, BLOB_INFO); } @After @@ -66,91 +112,122 @@ public void tearDown() throws Exception { verify(storage); } - @Test - public void testInfo() throws Exception { - assertEquals(BLOB_INFO, blob.info()); - replay(storage); + private void initializeExpectedBlob(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); + expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(BLOB_INFO)); + } + + private void initializeBlob() { + blob = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO)); } @Test public void testExists_True() throws Exception { + initializeExpectedBlob(1); Storage.BlobGetOption[] expectedOptions = {Storage.BlobGetOption.fields()}; - expect(storage.get(BLOB_INFO.blobId(), expectedOptions)).andReturn(BLOB_INFO); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(expectedBlob.blobId(), expectedOptions)).andReturn(expectedBlob); replay(storage); + initializeBlob(); assertTrue(blob.exists()); } @Test public void testExists_False() throws Exception { Storage.BlobGetOption[] expectedOptions = {Storage.BlobGetOption.fields()}; + expect(storage.options()).andReturn(null); expect(storage.get(BLOB_INFO.blobId(), expectedOptions)).andReturn(null); replay(storage); + initializeBlob(); assertFalse(blob.exists()); } @Test public void testContent() throws Exception { + initializeExpectedBlob(2); byte[] content = {1, 2}; + expect(storage.options()).andReturn(mockOptions); expect(storage.readAllBytes(BLOB_INFO.blobId())).andReturn(content); replay(storage); + initializeBlob(); assertArrayEquals(content, blob.content()); } @Test public void testReload() throws Exception { - BlobInfo updatedInfo = BLOB_INFO.toBuilder().cacheControl("c").build(); - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(updatedInfo); + initializeExpectedBlob(2); + Blob expectedReloadedBlob = expectedBlob.toBuilder().cacheControl("c").build(); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])) + .andReturn(expectedReloadedBlob); replay(storage); + initializeBlob(); Blob updatedBlob = blob.reload(); - assertSame(storage, updatedBlob.storage()); - assertEquals(updatedInfo, updatedBlob.info()); + assertEquals(expectedReloadedBlob, updatedBlob); } @Test public void testReloadNull() throws Exception { + initializeExpectedBlob(1); + expect(storage.options()).andReturn(mockOptions); expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(null); replay(storage); - assertNull(blob.reload()); + initializeBlob(); + Blob reloadedBlob = blob.reload(); + assertNull(reloadedBlob); } @Test public void testReloadWithOptions() throws Exception { - BlobInfo updatedInfo = BLOB_INFO.toBuilder().cacheControl("c").build(); + initializeExpectedBlob(2); + Blob expectedReloadedBlob = expectedBlob.toBuilder().cacheControl("c").build(); Storage.BlobGetOption[] options = {Storage.BlobGetOption.metagenerationMatch(42L)}; - expect(storage.get(BLOB_INFO.blobId(), options)).andReturn(updatedInfo); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(BLOB_INFO.blobId(), options)).andReturn(expectedReloadedBlob); replay(storage); + initializeBlob(); Blob updatedBlob = blob.reload(Blob.BlobSourceOption.metagenerationMatch()); - assertSame(storage, updatedBlob.storage()); - assertEquals(updatedInfo, updatedBlob.info()); + assertEquals(expectedReloadedBlob, updatedBlob); } @Test public void testUpdate() throws Exception { - BlobInfo updatedInfo = BLOB_INFO.toBuilder().cacheControl("c").build(); - expect(storage.update(updatedInfo, new Storage.BlobTargetOption[0])).andReturn(updatedInfo); + initializeExpectedBlob(2); + Blob expectedUpdatedBlob = expectedBlob.toBuilder().cacheControl("c").build(); + expect(storage.options()).andReturn(mockOptions).times(2); + expect(storage.update(eq(expectedUpdatedBlob), new Storage.BlobTargetOption[0])) + .andReturn(expectedUpdatedBlob); replay(storage); - Blob updatedBlob = blob.update(updatedInfo); - assertSame(storage, blob.storage()); - assertEquals(updatedInfo, updatedBlob.info()); + initializeBlob(); + Blob updatedBlob = new Blob(storage, new BlobInfo.BuilderImpl(expectedUpdatedBlob)); + Blob actualUpdatedBlob = updatedBlob.update(); + assertEquals(expectedUpdatedBlob, actualUpdatedBlob); } @Test public void testDelete() throws Exception { + initializeExpectedBlob(2); + expect(storage.options()).andReturn(mockOptions); expect(storage.delete(BLOB_INFO.blobId(), new Storage.BlobSourceOption[0])).andReturn(true); replay(storage); + initializeBlob(); assertTrue(blob.delete()); } @Test public void testCopyToBucket() throws Exception { + initializeExpectedBlob(2); BlobInfo target = BlobInfo.builder(BlobId.of("bt", "n")).build(); CopyWriter copyWriter = createMock(CopyWriter.class); Capture capturedCopyRequest = Capture.newInstance(); + expect(storage.options()).andReturn(mockOptions); expect(storage.copy(capture(capturedCopyRequest))).andReturn(copyWriter); replay(storage); + initializeBlob(); CopyWriter returnedCopyWriter = blob.copyTo("bt"); assertEquals(copyWriter, returnedCopyWriter); - assertEquals(capturedCopyRequest.getValue().source(), blob.id()); + assertEquals(capturedCopyRequest.getValue().source(), blob.blobId()); assertEquals(capturedCopyRequest.getValue().target(), target); assertTrue(capturedCopyRequest.getValue().sourceOptions().isEmpty()); assertTrue(capturedCopyRequest.getValue().targetOptions().isEmpty()); @@ -158,14 +235,17 @@ public void testCopyToBucket() throws Exception { @Test public void testCopyTo() throws Exception { + initializeExpectedBlob(2); BlobInfo target = BlobInfo.builder(BlobId.of("bt", "nt")).build(); CopyWriter copyWriter = createMock(CopyWriter.class); Capture capturedCopyRequest = Capture.newInstance(); + expect(storage.options()).andReturn(mockOptions); expect(storage.copy(capture(capturedCopyRequest))).andReturn(copyWriter); replay(storage); + initializeBlob(); CopyWriter returnedCopyWriter = blob.copyTo("bt", "nt"); assertEquals(copyWriter, returnedCopyWriter); - assertEquals(capturedCopyRequest.getValue().source(), blob.id()); + assertEquals(capturedCopyRequest.getValue().source(), blob.blobId()); assertEquals(capturedCopyRequest.getValue().target(), target); assertTrue(capturedCopyRequest.getValue().sourceOptions().isEmpty()); assertTrue(capturedCopyRequest.getValue().targetOptions().isEmpty()); @@ -173,15 +253,18 @@ public void testCopyTo() throws Exception { @Test public void testCopyToBlobId() throws Exception { + initializeExpectedBlob(2); BlobId targetId = BlobId.of("bt", "nt"); CopyWriter copyWriter = createMock(CopyWriter.class); BlobInfo target = BlobInfo.builder(targetId).build(); Capture capturedCopyRequest = Capture.newInstance(); + expect(storage.options()).andReturn(mockOptions); expect(storage.copy(capture(capturedCopyRequest))).andReturn(copyWriter); replay(storage); + initializeBlob(); CopyWriter returnedCopyWriter = blob.copyTo(targetId); assertEquals(copyWriter, returnedCopyWriter); - assertEquals(capturedCopyRequest.getValue().source(), blob.id()); + assertEquals(capturedCopyRequest.getValue().source(), blob.blobId()); assertEquals(capturedCopyRequest.getValue().target(), target); assertTrue(capturedCopyRequest.getValue().sourceOptions().isEmpty()); assertTrue(capturedCopyRequest.getValue().targetOptions().isEmpty()); @@ -189,174 +272,93 @@ public void testCopyToBlobId() throws Exception { @Test public void testReader() throws Exception { + initializeExpectedBlob(2); ReadChannel channel = createMock(ReadChannel.class); + expect(storage.options()).andReturn(mockOptions); expect(storage.reader(BLOB_INFO.blobId())).andReturn(channel); replay(storage); + initializeBlob(); assertSame(channel, blob.reader()); } @Test public void testWriter() throws Exception { + initializeExpectedBlob(2); BlobWriteChannel channel = createMock(BlobWriteChannel.class); - expect(storage.writer(BLOB_INFO)).andReturn(channel); + expect(storage.options()).andReturn(mockOptions); + expect(storage.writer(eq(expectedBlob))).andReturn(channel); replay(storage); + initializeBlob(); assertSame(channel, blob.writer()); } @Test public void testSignUrl() throws Exception { + initializeExpectedBlob(2); URL url = new URL("http://localhost:123/bla"); - expect(storage.signUrl(BLOB_INFO, 100, TimeUnit.SECONDS)).andReturn(url); + expect(storage.options()).andReturn(mockOptions); + expect(storage.signUrl(expectedBlob, 100, TimeUnit.SECONDS)).andReturn(url); replay(storage); + initializeBlob(); assertEquals(url, blob.signUrl(100, TimeUnit.SECONDS)); } @Test - public void testGetSome() throws Exception { - List blobInfoList = Arrays.asList(BLOB_INFO_ARRAY); - expect(storage.get(BLOB_ID_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.get(storage, BLOB_ID_ARRAY[0], BLOB_ID_ARRAY[1], BLOB_ID_ARRAY[2]); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } - } - - @Test - public void testGetSomeList() throws Exception { - List blobInfoList = Arrays.asList(BLOB_INFO_ARRAY); - expect(storage.get(BLOB_ID_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.get(storage, Arrays.asList(BLOB_ID_ARRAY)); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } - } - - @Test - public void testGetSomeNull() throws Exception { - List blobInfoList = Arrays.asList(BLOB_INFO_ARRAY[0], null, BLOB_INFO_ARRAY[2]); - expect(storage.get(BLOB_ID_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.get(storage, BLOB_ID_ARRAY[0], BLOB_ID_ARRAY[1], BLOB_ID_ARRAY[2]); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - if (blobInfoList.get(i) != null) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } else { - assertNull(result.get(i)); - } - } - } - - @Test - public void testUpdateNone() throws Exception { - replay(storage); - assertTrue(Blob.update(storage).isEmpty()); - } - - @Test - public void testUpdateSome() throws Exception { - List blobInfoList = Lists.newArrayListWithCapacity(BLOB_ID_ARRAY.length); - for (BlobInfo info : BLOB_INFO_ARRAY) { - blobInfoList.add(info.toBuilder().contentType("content").build()); - } - expect(storage.update(BLOB_INFO_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.update(storage, BLOB_INFO_ARRAY); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } - } - - @Test - public void testUpdateSomeNull() throws Exception { - List blobInfoList = Arrays.asList( - BLOB_INFO_ARRAY[0].toBuilder().contentType("content").build(), null, - BLOB_INFO_ARRAY[2].toBuilder().contentType("content").build()); - expect(storage.update(BLOB_INFO_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.update(storage, BLOB_INFO_ARRAY); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - if (blobInfoList.get(i) != null) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } else { - assertNull(result.get(i)); - } - } - } - - @Test - public void testDeleteNone() throws Exception { - replay(storage); - assertTrue(Blob.delete(storage).isEmpty()); - } - - @Test - public void testDeleteSome() throws Exception { - List deleteResult = Arrays.asList(true, true, true); - expect(storage.delete(BLOB_ID_ARRAY)).andReturn(deleteResult); - replay(storage); - List result = Blob.delete(storage, BLOB_ID_ARRAY); - assertEquals(deleteResult.size(), result.size()); - for (int i = 0; i < deleteResult.size(); i++) { - assertEquals(deleteResult.get(i), result.get(i)); - } - } - - @Test - public void testGetFromString() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(BLOB_INFO); - replay(storage); - Blob loadedBlob = Blob.get(storage, BLOB_INFO.bucket(), BLOB_INFO.name()); - assertEquals(BLOB_INFO, loadedBlob.info()); - } - - @Test - public void testGetFromId() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(BLOB_INFO); - replay(storage); - Blob loadedBlob = Blob.get(storage, BLOB_INFO.blobId()); - assertNotNull(loadedBlob); - assertEquals(BLOB_INFO, loadedBlob.info()); - } - - @Test - public void testGetFromStringNull() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(null); - replay(storage); - assertNull(Blob.get(storage, BLOB_INFO.bucket(), BLOB_INFO.name())); - } - - @Test - public void testGetFromIdNull() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(null); - replay(storage); - assertNull(Blob.get(storage, BLOB_INFO.blobId())); - } - - @Test - public void testGetFromStringWithOptions() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), Storage.BlobGetOption.generationMatch(42L))) - .andReturn(BLOB_INFO); + public void testToBuilder() { + expect(storage.options()).andReturn(mockOptions).times(4); replay(storage); - Blob loadedBlob = Blob.get(storage, BLOB_INFO.bucket(), BLOB_INFO.name(), - Storage.BlobGetOption.generationMatch(42L)); - assertEquals(BLOB_INFO, loadedBlob.info()); + Blob fullBlob = new Blob(storage, new BlobInfo.BuilderImpl(FULL_BLOB_INFO)); + assertEquals(fullBlob, fullBlob.toBuilder().build()); + Blob simpleBlob = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO)); + assertEquals(simpleBlob, simpleBlob.toBuilder().build()); } @Test - public void testGetFromIdWithOptions() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), Storage.BlobGetOption.generationMatch(42L))) - .andReturn(BLOB_INFO); + public void testBuilder() { + initializeExpectedBlob(4); + expect(storage.options()).andReturn(mockOptions).times(2); replay(storage); - Blob loadedBlob = - Blob.get(storage, BLOB_INFO.blobId(), Storage.BlobGetOption.generationMatch(42L)); - assertNotNull(loadedBlob); - assertEquals(BLOB_INFO, loadedBlob.info()); + Blob.Builder builder = new Blob.Builder(new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO))); + Blob blob = builder.acl(ACL) + .componentCount(COMPONENT_COUNT) + .contentType(CONTENT_TYPE) + .cacheControl(CACHE_CONTROL) + .contentDisposition(CONTENT_DISPOSITION) + .contentEncoding(CONTENT_ENCODING) + .contentLanguage(CONTENT_LANGUAGE) + .crc32c(CRC32) + .deleteTime(DELETE_TIME) + .etag(ETAG) + .id(ID) + .md5(MD5) + .mediaLink(MEDIA_LINK) + .metadata(METADATA) + .metageneration(META_GENERATION) + .owner(OWNER) + .selfLink(SELF_LINK) + .size(SIZE) + .updateTime(UPDATE_TIME) + .build(); + assertEquals("b", blob.bucket()); + assertEquals("n", blob.name()); + assertEquals(ACL, blob.acl()); + assertEquals(COMPONENT_COUNT, blob.componentCount()); + assertEquals(CONTENT_TYPE, blob.contentType()); + assertEquals(CACHE_CONTROL, blob.cacheControl()); + assertEquals(CONTENT_DISPOSITION, blob.contentDisposition()); + assertEquals(CONTENT_ENCODING, blob.contentEncoding()); + assertEquals(CONTENT_LANGUAGE, blob.contentLanguage()); + assertEquals(CRC32, blob.crc32c()); + assertEquals(DELETE_TIME, blob.deleteTime()); + assertEquals(ETAG, blob.etag()); + assertEquals(ID, blob.id()); + assertEquals(MD5, blob.md5()); + assertEquals(MEDIA_LINK, blob.mediaLink()); + assertEquals(METADATA, blob.metadata()); + assertEquals(META_GENERATION, blob.metageneration()); + assertEquals(OWNER, blob.owner()); + assertEquals(SELF_LINK, blob.selfLink()); + assertEquals(SIZE, blob.size()); + assertEquals(UPDATE_TIME, blob.updateTime()); } } diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BucketTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BucketTest.java index 4e253033c6f2..236411e0c2d8 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BucketTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BucketTest.java @@ -16,27 +16,35 @@ package com.google.gcloud.storage; +import static com.google.gcloud.storage.Acl.Project.ProjectRole.VIEWERS; +import static com.google.gcloud.storage.Acl.Role.READER; +import static com.google.gcloud.storage.Acl.Role.WRITER; import static org.easymock.EasyMock.capture; +import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; import com.google.common.collect.ImmutableList; import com.google.gcloud.Page; import com.google.gcloud.PageImpl; +import com.google.gcloud.storage.Acl.Project; +import com.google.gcloud.storage.Acl.User; import com.google.gcloud.storage.BatchResponse.Result; +import com.google.gcloud.storage.BucketInfo.AgeDeleteRule; +import com.google.gcloud.storage.BucketInfo.DeleteRule; import org.easymock.Capture; import org.junit.After; import org.junit.Before; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.ExpectedException; import java.io.ByteArrayInputStream; import java.io.InputStream; @@ -48,20 +56,57 @@ public class BucketTest { + private static final List ACL = ImmutableList.of( + Acl.of(User.ofAllAuthenticatedUsers(), READER), Acl.of(new Project(VIEWERS, "p1"), WRITER)); + private static final String ETAG = "0xFF00"; + private static final String ID = "B/N:1"; + private static final Long META_GENERATION = 10L; + private static final User OWNER = new User("user@gmail.com"); + private static final String SELF_LINK = "http://storage/b/n"; + private static final Long CREATE_TIME = System.currentTimeMillis(); + private static final List CORS = Collections.singletonList(Cors.builder().build()); + private static final List DEFAULT_ACL = + Collections.singletonList(Acl.of(User.ofAllAuthenticatedUsers(), WRITER)); + private static final List DELETE_RULES = + Collections.singletonList(new AgeDeleteRule(5)); + private static final String INDEX_PAGE = "index.html"; + private static final String NOT_FOUND_PAGE = "error.html"; + private static final String LOCATION = "ASIA"; + private static final String STORAGE_CLASS = "STANDARD"; + private static final Boolean VERSIONING_ENABLED = true; + private static final BucketInfo FULL_BUCKET_INFO = BucketInfo.builder("b") + .acl(ACL) + .etag(ETAG) + .id(ID) + .metageneration(META_GENERATION) + .owner(OWNER) + .selfLink(SELF_LINK) + .cors(CORS) + .createTime(CREATE_TIME) + .defaultAcl(DEFAULT_ACL) + .deleteRules(DELETE_RULES) + .indexPage(INDEX_PAGE) + .notFoundPage(NOT_FOUND_PAGE) + .location(LOCATION) + .storageClass(STORAGE_CLASS) + .versioningEnabled(VERSIONING_ENABLED) + .build(); private static final BucketInfo BUCKET_INFO = BucketInfo.builder("b").metageneration(42L).build(); - private static final Iterable BLOB_INFO_RESULTS = ImmutableList.of( - BlobInfo.builder("b", "n1").build(), - BlobInfo.builder("b", "n2").build(), - BlobInfo.builder("b", "n3").build()); private static final String CONTENT_TYPE = "text/plain"; private Storage storage; + private Storage serviceMockReturnsOptions = createMock(Storage.class); + private StorageOptions mockOptions = createMock(StorageOptions.class); private Bucket bucket; + private Bucket expectedBucket; + private Iterable blobResults; + + @Rule + public ExpectedException thrown = ExpectedException.none(); @Before - public void setUp() throws Exception { + public void setUp() { storage = createStrictMock(Storage.class); - bucket = new Bucket(storage, BUCKET_INFO); } @After @@ -69,124 +114,165 @@ public void tearDown() throws Exception { verify(storage); } - @Test - public void testInfo() throws Exception { - assertEquals(BUCKET_INFO, bucket.info()); - replay(storage); + private void initializeExpectedBucket(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); + expectedBucket = new Bucket(serviceMockReturnsOptions, new BucketInfo.BuilderImpl(BUCKET_INFO)); + blobResults = ImmutableList.of( + new Blob(serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder("b", "n1").build())), + new Blob(serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder("b", "n2").build())), + new Blob(serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder("b", "n3").build()))); + } + + private void initializeBucket() { + bucket = new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO)); } @Test public void testExists_True() throws Exception { + initializeExpectedBucket(4); Storage.BucketGetOption[] expectedOptions = {Storage.BucketGetOption.fields()}; - expect(storage.get(BUCKET_INFO.name(), expectedOptions)).andReturn(BUCKET_INFO); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(BUCKET_INFO.name(), expectedOptions)).andReturn(expectedBucket); replay(storage); + initializeBucket(); assertTrue(bucket.exists()); } @Test public void testExists_False() throws Exception { + initializeExpectedBucket(4); Storage.BucketGetOption[] expectedOptions = {Storage.BucketGetOption.fields()}; + expect(storage.options()).andReturn(mockOptions); expect(storage.get(BUCKET_INFO.name(), expectedOptions)).andReturn(null); replay(storage); + initializeBucket(); assertFalse(bucket.exists()); } @Test public void testReload() throws Exception { + initializeExpectedBucket(5); BucketInfo updatedInfo = BUCKET_INFO.toBuilder().notFoundPage("p").build(); - expect(storage.get(updatedInfo.name())).andReturn(updatedInfo); + Bucket expectedUpdatedBucket = + new Bucket(serviceMockReturnsOptions, new BucketInfo.BuilderImpl(updatedInfo)); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(updatedInfo.name())).andReturn(expectedUpdatedBucket); replay(storage); + initializeBucket(); Bucket updatedBucket = bucket.reload(); - assertSame(storage, updatedBucket.storage()); - assertEquals(updatedInfo, updatedBucket.info()); + assertEquals(expectedUpdatedBucket, updatedBucket); } @Test public void testReloadNull() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); expect(storage.get(BUCKET_INFO.name())).andReturn(null); replay(storage); + initializeBucket(); assertNull(bucket.reload()); } @Test public void testReloadWithOptions() throws Exception { + initializeExpectedBucket(5); BucketInfo updatedInfo = BUCKET_INFO.toBuilder().notFoundPage("p").build(); + Bucket expectedUpdatedBucket = + new Bucket(serviceMockReturnsOptions, new BucketInfo.BuilderImpl(updatedInfo)); + expect(storage.options()).andReturn(mockOptions); expect(storage.get(updatedInfo.name(), Storage.BucketGetOption.metagenerationMatch(42L))) - .andReturn(updatedInfo); + .andReturn(expectedUpdatedBucket); replay(storage); + initializeBucket(); Bucket updatedBucket = bucket.reload(Bucket.BucketSourceOption.metagenerationMatch()); - assertSame(storage, updatedBucket.storage()); - assertEquals(updatedInfo, updatedBucket.info()); + assertEquals(expectedUpdatedBucket, updatedBucket); } @Test public void testUpdate() throws Exception { - BucketInfo updatedInfo = BUCKET_INFO.toBuilder().notFoundPage("p").build(); - expect(storage.update(updatedInfo)).andReturn(updatedInfo); + initializeExpectedBucket(5); + Bucket expectedUpdatedBucket = expectedBucket.toBuilder().notFoundPage("p").build(); + expect(storage.options()).andReturn(mockOptions).times(2); + expect(storage.update(expectedUpdatedBucket)).andReturn(expectedUpdatedBucket); replay(storage); - Bucket updatedBucket = bucket.update(updatedInfo); - assertSame(storage, bucket.storage()); - assertEquals(updatedInfo, updatedBucket.info()); + initializeBucket(); + Bucket updatedBucket = new Bucket(storage, new BucketInfo.BuilderImpl(expectedUpdatedBucket)); + Bucket actualUpdatedBucket = updatedBucket.update(); + assertEquals(expectedUpdatedBucket, actualUpdatedBucket); } @Test public void testDelete() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); expect(storage.delete(BUCKET_INFO.name())).andReturn(true); replay(storage); + initializeBucket(); assertTrue(bucket.delete()); } @Test public void testList() throws Exception { - StorageOptions storageOptions = createStrictMock(StorageOptions.class); - PageImpl blobInfoPage = new PageImpl<>(null, "c", BLOB_INFO_RESULTS); - expect(storage.list(BUCKET_INFO.name())).andReturn(blobInfoPage); - expect(storage.options()).andReturn(storageOptions); - expect(storageOptions.service()).andReturn(storage); - replay(storage, storageOptions); + initializeExpectedBucket(4); + PageImpl expectedBlobPage = new PageImpl<>(null, "c", blobResults); + expect(storage.options()).andReturn(mockOptions); + expect(storage.list(BUCKET_INFO.name())).andReturn(expectedBlobPage); + replay(storage); + initializeBucket(); Page blobPage = bucket.list(); - Iterator blobInfoIterator = blobInfoPage.values().iterator(); + Iterator blobInfoIterator = blobPage.values().iterator(); Iterator blobIterator = blobPage.values().iterator(); while (blobInfoIterator.hasNext() && blobIterator.hasNext()) { - assertEquals(blobInfoIterator.next(), blobIterator.next().info()); + assertEquals(blobInfoIterator.next(), blobIterator.next()); } assertFalse(blobInfoIterator.hasNext()); assertFalse(blobIterator.hasNext()); - assertEquals(blobInfoPage.nextPageCursor(), blobPage.nextPageCursor()); - verify(storageOptions); + assertEquals(expectedBlobPage.nextPageCursor(), blobPage.nextPageCursor()); } @Test public void testGet() throws Exception { - BlobInfo info = BlobInfo.builder("b", "n").build(); - expect(storage.get(BlobId.of(bucket.info().name(), "n"), new Storage.BlobGetOption[0])) - .andReturn(info); + initializeExpectedBucket(5); + Blob expectedBlob = new Blob( + serviceMockReturnsOptions, new BlobInfo.BuilderImpl(BlobInfo.builder("b", "n").build())); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(BlobId.of(expectedBucket.name(), "n"), new Storage.BlobGetOption[0])) + .andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.get("n"); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); } @Test public void testGetAll() throws Exception { + initializeExpectedBucket(4); Capture capturedBatchRequest = Capture.newInstance(); - List> batchResultList = new LinkedList<>(); - for (BlobInfo info : BLOB_INFO_RESULTS) { + List> batchResultList = new LinkedList<>(); + for (Blob info : blobResults) { batchResultList.add(new Result<>(info)); } BatchResponse response = new BatchResponse(Collections.>emptyList(), - Collections.>emptyList(), batchResultList); + Collections.>emptyList(), batchResultList); + expect(storage.options()).andReturn(mockOptions); expect(storage.submit(capture(capturedBatchRequest))).andReturn(response); + expect(storage.options()).andReturn(mockOptions).times(3); replay(storage); + initializeBucket(); List blobs = bucket.get("n1", "n2", "n3"); Set blobInfoSet = capturedBatchRequest.getValue().toGet().keySet(); assertEquals(batchResultList.size(), blobInfoSet.size()); - for (BlobInfo info : BLOB_INFO_RESULTS) { + for (BlobInfo info : blobResults) { assertTrue(blobInfoSet.contains(info.blobId())); } Iterator blobIterator = blobs.iterator(); - Iterator> batchResultIterator = response.gets().iterator(); + Iterator> batchResultIterator = response.gets().iterator(); while (batchResultIterator.hasNext() && blobIterator.hasNext()) { - assertEquals(batchResultIterator.next().get(), blobIterator.next().info()); + assertEquals(batchResultIterator.next().get(), blobIterator.next()); } assertFalse(batchResultIterator.hasNext()); assertFalse(blobIterator.hasNext()); @@ -194,70 +280,251 @@ public void testGetAll() throws Exception { @Test public void testCreate() throws Exception { + initializeExpectedBucket(5); BlobInfo info = BlobInfo.builder("b", "n").contentType(CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); byte[] content = {0xD, 0xE, 0xA, 0xD}; - expect(storage.create(info, content)).andReturn(info); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, content)).andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.create("n", content, CONTENT_TYPE); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); } @Test public void testCreateNullContentType() throws Exception { + initializeExpectedBucket(5); BlobInfo info = BlobInfo.builder("b", "n").contentType(Storage.DEFAULT_CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); byte[] content = {0xD, 0xE, 0xA, 0xD}; - expect(storage.create(info, content)).andReturn(info); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, content)).andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.create("n", content, null); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); + } + + @Test + public void testCreateWithOptions() throws Exception { + initializeExpectedBucket(5); + BlobInfo info = BlobInfo.builder(BlobId.of("b", "n", 42L)) + .contentType(CONTENT_TYPE) + .metageneration(24L) + .build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + Storage.PredefinedAcl acl = Storage.PredefinedAcl.ALL_AUTHENTICATED_USERS; + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, content, Storage.BlobTargetOption.generationMatch(), + Storage.BlobTargetOption.metagenerationMatch(), + Storage.BlobTargetOption.predefinedAcl(acl))).andReturn(expectedBlob); + replay(storage); + initializeBucket(); + Blob blob = bucket.create("n", content, CONTENT_TYPE, + Bucket.BlobTargetOption.generationMatch(42L), + Bucket.BlobTargetOption.metagenerationMatch(24L), + Bucket.BlobTargetOption.predefinedAcl(acl)); + assertEquals(expectedBlob, blob); + } + + @Test + public void testCreateNotExists() throws Exception { + initializeExpectedBucket(5); + BlobInfo info = BlobInfo.builder(BlobId.of("b", "n", 0L)).contentType(CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, content, Storage.BlobTargetOption.generationMatch())) + .andReturn(expectedBlob); + replay(storage); + initializeBucket(); + Blob blob = bucket.create("n", content, CONTENT_TYPE, Bucket.BlobTargetOption.doesNotExist()); + assertEquals(expectedBlob, blob); + } + + @Test + public void testCreateWithWrongGenerationOptions() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); + replay(storage); + initializeBucket(); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage( + "Only one option of generationMatch, doesNotExist or generationNotMatch can be provided"); + bucket.create("n", content, CONTENT_TYPE, Bucket.BlobTargetOption.generationMatch(42L), + Bucket.BlobTargetOption.generationNotMatch(24L)); + } + + @Test + public void testCreateWithWrongMetagenerationOptions() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); + replay(storage); + initializeBucket(); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage( + "metagenerationMatch and metagenerationNotMatch options can not be both provided"); + bucket.create("n", content, CONTENT_TYPE, Bucket.BlobTargetOption.metagenerationMatch(42L), + Bucket.BlobTargetOption.metagenerationNotMatch(24L)); } @Test public void testCreateFromStream() throws Exception { + initializeExpectedBucket(5); BlobInfo info = BlobInfo.builder("b", "n").contentType(CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); byte[] content = {0xD, 0xE, 0xA, 0xD}; InputStream streamContent = new ByteArrayInputStream(content); - expect(storage.create(info, streamContent)).andReturn(info); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, streamContent)).andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.create("n", streamContent, CONTENT_TYPE); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); } @Test public void testCreateFromStreamNullContentType() throws Exception { + initializeExpectedBucket(5); BlobInfo info = BlobInfo.builder("b", "n").contentType(Storage.DEFAULT_CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); byte[] content = {0xD, 0xE, 0xA, 0xD}; InputStream streamContent = new ByteArrayInputStream(content); - expect(storage.create(info, streamContent)).andReturn(info); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, streamContent)).andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.create("n", streamContent, null); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); } @Test - public void testStaticGet() throws Exception { - expect(storage.get(BUCKET_INFO.name())).andReturn(BUCKET_INFO); + public void testCreateFromStreamWithOptions() throws Exception { + initializeExpectedBucket(5); + BlobInfo info = BlobInfo.builder(BlobId.of("b", "n", 42L)) + .contentType(CONTENT_TYPE) + .metageneration(24L) + .crc32c("crc") + .md5("md5") + .build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + Storage.PredefinedAcl acl = Storage.PredefinedAcl.ALL_AUTHENTICATED_USERS; + InputStream streamContent = new ByteArrayInputStream(content); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, streamContent, Storage.BlobWriteOption.generationMatch(), + Storage.BlobWriteOption.metagenerationMatch(), Storage.BlobWriteOption.predefinedAcl(acl), + Storage.BlobWriteOption.crc32cMatch(), Storage.BlobWriteOption.md5Match())) + .andReturn(expectedBlob); replay(storage); - Bucket loadedBucket = Bucket.get(storage, BUCKET_INFO.name()); - assertNotNull(loadedBucket); - assertEquals(BUCKET_INFO, loadedBucket.info()); + initializeBucket(); + Blob blob = bucket.create("n", streamContent, CONTENT_TYPE, + Bucket.BlobWriteOption.generationMatch(42L), + Bucket.BlobWriteOption.metagenerationMatch(24L), Bucket.BlobWriteOption.predefinedAcl(acl), + Bucket.BlobWriteOption.crc32cMatch("crc"), Bucket.BlobWriteOption.md5Match("md5")); + assertEquals(expectedBlob, blob); } @Test - public void testStaticGetNull() throws Exception { - expect(storage.get(BUCKET_INFO.name())).andReturn(null); + public void testCreateFromStreamNotExists() throws Exception { + initializeExpectedBucket(5); + BlobInfo info = BlobInfo.builder(BlobId.of("b", "n", 0L)).contentType(CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + InputStream streamContent = new ByteArrayInputStream(content); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, streamContent, Storage.BlobWriteOption.generationMatch())) + .andReturn(expectedBlob); + replay(storage); + initializeBucket(); + Blob blob = + bucket.create("n", streamContent, CONTENT_TYPE, Bucket.BlobWriteOption.doesNotExist()); + assertEquals(expectedBlob, blob); + } + + @Test + public void testCreateFromStreamWithWrongGenerationOptions() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); + replay(storage); + initializeBucket(); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + InputStream streamContent = new ByteArrayInputStream(content); + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage( + "Only one option of generationMatch, doesNotExist or generationNotMatch can be provided"); + bucket.create("n", streamContent, CONTENT_TYPE, Bucket.BlobWriteOption.generationMatch(42L), + Bucket.BlobWriteOption.generationNotMatch(24L)); + } + + @Test + public void testCreateFromStreamWithWrongMetagenerationOptions() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); + replay(storage); + initializeBucket(); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + InputStream streamContent = new ByteArrayInputStream(content); + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage( + "metagenerationMatch and metagenerationNotMatch options can not be both provided"); + bucket.create("n", streamContent, CONTENT_TYPE, Bucket.BlobWriteOption.metagenerationMatch(42L), + Bucket.BlobWriteOption.metagenerationNotMatch(24L)); + } + + @Test + public void testToBuilder() { + expect(storage.options()).andReturn(mockOptions).times(4); replay(storage); - assertNull(Bucket.get(storage, BUCKET_INFO.name())); + Bucket fullBucket = new Bucket(storage, new BucketInfo.BuilderImpl(FULL_BUCKET_INFO)); + assertEquals(fullBucket, fullBucket.toBuilder().build()); + Bucket simpleBlob = new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO)); + assertEquals(simpleBlob, simpleBlob.toBuilder().build()); } @Test - public void testStaticGetWithOptions() throws Exception { - expect(storage.get(BUCKET_INFO.name(), Storage.BucketGetOption.fields())) - .andReturn(BUCKET_INFO); + public void testBuilder() { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions).times(4); replay(storage); - Bucket loadedBucket = - Bucket.get(storage, BUCKET_INFO.name(), Storage.BucketGetOption.fields()); - assertNotNull(loadedBucket); - assertEquals(BUCKET_INFO, loadedBucket.info()); + Bucket.Builder builder = + new Bucket.Builder(new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO))); + Bucket bucket = builder.acl(ACL) + .etag(ETAG) + .id(ID) + .metageneration(META_GENERATION) + .owner(OWNER) + .selfLink(SELF_LINK) + .cors(CORS) + .createTime(CREATE_TIME) + .defaultAcl(DEFAULT_ACL) + .deleteRules(DELETE_RULES) + .indexPage(INDEX_PAGE) + .notFoundPage(NOT_FOUND_PAGE) + .location(LOCATION) + .storageClass(STORAGE_CLASS) + .versioningEnabled(VERSIONING_ENABLED) + .build(); + assertEquals("b", bucket.name()); + assertEquals(ACL, bucket.acl()); + assertEquals(ETAG, bucket.etag()); + assertEquals(ID, bucket.id()); + assertEquals(META_GENERATION, bucket.metageneration()); + assertEquals(OWNER, bucket.owner()); + assertEquals(SELF_LINK, bucket.selfLink()); + assertEquals(CREATE_TIME, bucket.createTime()); + assertEquals(CORS, bucket.cors()); + assertEquals(DEFAULT_ACL, bucket.defaultAcl()); + assertEquals(DELETE_RULES, bucket.deleteRules()); + assertEquals(INDEX_PAGE, bucket.indexPage()); + assertEquals(NOT_FOUND_PAGE, bucket.notFoundPage()); + assertEquals(LOCATION, bucket.location()); + assertEquals(STORAGE_CLASS, bucket.storageClass()); + assertEquals(VERSIONING_ENABLED, bucket.versioningEnabled()); + assertEquals(storage.options(), bucket.storage().options()); } } diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/RemoteGcsHelperTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/RemoteGcsHelperTest.java index d06f004fe84c..154554a029fe 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/RemoteGcsHelperTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/RemoteGcsHelperTest.java @@ -21,9 +21,11 @@ import com.google.common.collect.ImmutableList; import com.google.gcloud.Page; +import com.google.gcloud.storage.Storage.BlobListOption; import com.google.gcloud.storage.testing.RemoteGcsHelper; import org.easymock.EasyMock; +import org.junit.Before; import org.junit.Rule; import org.junit.Test; import org.junit.rules.ExpectedException; @@ -66,43 +68,60 @@ public class RemoteGcsHelperTest { + " \"type\": \"service_account\"\n" + "}"; private static final InputStream JSON_KEY_STREAM = new ByteArrayInputStream(JSON_KEY.getBytes()); - private static final List BLOB_LIST = ImmutableList.of( - BlobInfo.builder(BUCKET_NAME, "n1").build(), - BlobInfo.builder(BUCKET_NAME, "n2").build()); private static final StorageException RETRYABLE_EXCEPTION = new StorageException(409, ""); private static final StorageException FATAL_EXCEPTION = new StorageException(500, ""); - private static final Page BLOB_PAGE = new Page() { - @Override - public String nextPageCursor() { - return "nextPageCursor"; - } - - @Override - public Page nextPage() { - return null; - } - - @Override - public Iterable values() { - return BLOB_LIST; - } - - @Override - public Iterator iterateAll() { - return BLOB_LIST.iterator(); - } - }; + private static Storage serviceMockReturnsOptions; + private List blobList; + private Page blobPage; @Rule public ExpectedException thrown = ExpectedException.none(); + @Before + public void setUp() { + serviceMockReturnsOptions = EasyMock.createMock(Storage.class); + EasyMock.expect(serviceMockReturnsOptions.options()) + .andReturn(EasyMock.createMock(StorageOptions.class)) + .times(2); + EasyMock.replay(serviceMockReturnsOptions); + blobList = ImmutableList.of( + new Blob( + serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder(BUCKET_NAME, "n1").build())), + new Blob( + serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder(BUCKET_NAME, "n2").build()))); + blobPage = new Page() { + @Override + public String nextPageCursor() { + return "nextPageCursor"; + } + + @Override + public Page nextPage() { + return null; + } + + @Override + public Iterable values() { + return blobList; + } + + @Override + public Iterator iterateAll() { + return blobList.iterator(); + } + }; + } + @Test public void testForceDelete() throws InterruptedException, ExecutionException { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true); + EasyMock.expect(storageMock.list(BUCKET_NAME, BlobListOption.versions(true))) + .andReturn(blobPage); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andReturn(true); EasyMock.replay(storageMock); @@ -113,9 +132,9 @@ public void testForceDelete() throws InterruptedException, ExecutionException { @Test public void testForceDeleteTimeout() throws InterruptedException, ExecutionException { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE).anyTimes(); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true).anyTimes(); + EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(blobPage).anyTimes(); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true).anyTimes(); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andThrow(RETRYABLE_EXCEPTION).anyTimes(); EasyMock.replay(storageMock); @@ -126,9 +145,10 @@ public void testForceDeleteTimeout() throws InterruptedException, ExecutionExcep @Test public void testForceDeleteFail() throws InterruptedException, ExecutionException { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true); + EasyMock.expect(storageMock.list(BUCKET_NAME, BlobListOption.versions(true))) + .andReturn(blobPage); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andThrow(FATAL_EXCEPTION); EasyMock.replay(storageMock); @@ -143,9 +163,10 @@ public void testForceDeleteFail() throws InterruptedException, ExecutionExceptio @Test public void testForceDeleteNoTimeout() { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true); + EasyMock.expect(storageMock.list(BUCKET_NAME, BlobListOption.versions(true))) + .andReturn(blobPage); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andReturn(true); EasyMock.replay(storageMock); @@ -156,9 +177,10 @@ public void testForceDeleteNoTimeout() { @Test public void testForceDeleteNoTimeoutFail() { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true); + EasyMock.expect(storageMock.list(BUCKET_NAME, BlobListOption.versions(true))) + .andReturn(blobPage); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andThrow(FATAL_EXCEPTION); EasyMock.replay(storageMock); diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/SerializationTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/SerializationTest.java index 8bef27cb0cd0..c9b957bb936a 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/SerializationTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/SerializationTest.java @@ -42,6 +42,7 @@ public class SerializationTest { + private static final Storage STORAGE = StorageOptions.builder().projectId("p").build().service(); private static final Acl.Domain ACL_DOMAIN = new Acl.Domain("domain"); private static final Acl.Group ACL_GROUP = new Acl.Group("group"); private static final Acl.Project ACL_PROJECT_ = new Acl.Project(ProjectRole.VIEWERS, "pid"); @@ -50,16 +51,18 @@ public class SerializationTest { private static final Acl ACL = Acl.of(ACL_DOMAIN, Acl.Role.OWNER); private static final BlobInfo BLOB_INFO = BlobInfo.builder("b", "n").build(); private static final BucketInfo BUCKET_INFO = BucketInfo.of("b"); + private static final Blob BLOB = new Blob(STORAGE, new BlobInfo.BuilderImpl(BLOB_INFO)); + private static final Bucket BUCKET = new Bucket(STORAGE, new BucketInfo.BuilderImpl(BUCKET_INFO)); private static final Cors.Origin ORIGIN = Cors.Origin.any(); private static final Cors CORS = Cors.builder().maxAgeSeconds(1).origins(Collections.singleton(ORIGIN)).build(); private static final BatchRequest BATCH_REQUEST = BatchRequest.builder().delete("B", "N").build(); private static final BatchResponse BATCH_RESPONSE = new BatchResponse( Collections.singletonList(BatchResponse.Result.of(true)), - Collections.>emptyList(), - Collections.>emptyList()); - private static final PageImpl PAGE_RESULT = new PageImpl<>( - null, "c", Collections.singletonList(BlobInfo.builder("b", "n").build())); + Collections.>emptyList(), + Collections.>emptyList()); + private static final PageImpl PAGE_RESULT = + new PageImpl<>(null, "c", Collections.singletonList(BLOB)); private static final Storage.BlobListOption BLOB_LIST_OPTIONS = Storage.BlobListOption.maxResults(100); private static final Storage.BlobSourceOption BLOB_SOURCE_OPTIONS = @@ -96,9 +99,9 @@ public void testServiceOptions() throws Exception { @Test public void testModelAndRequests() throws Exception { Serializable[] objects = {ACL_DOMAIN, ACL_GROUP, ACL_PROJECT_, ACL_USER, ACL_RAW, ACL, - BLOB_INFO, BUCKET_INFO, ORIGIN, CORS, BATCH_REQUEST, BATCH_RESPONSE, PAGE_RESULT, - BLOB_LIST_OPTIONS, BLOB_SOURCE_OPTIONS, BLOB_TARGET_OPTIONS, BUCKET_LIST_OPTIONS, - BUCKET_SOURCE_OPTIONS, BUCKET_TARGET_OPTIONS}; + BLOB_INFO, BLOB, BUCKET_INFO, BUCKET, ORIGIN, CORS, BATCH_REQUEST, BATCH_RESPONSE, + PAGE_RESULT, BLOB_LIST_OPTIONS, BLOB_SOURCE_OPTIONS, BLOB_TARGET_OPTIONS, + BUCKET_LIST_OPTIONS, BUCKET_SOURCE_OPTIONS, BUCKET_TARGET_OPTIONS}; for (Serializable obj : objects) { Object copy = serializeAndDeserialize(obj); assertEquals(obj, obj); diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/StorageImplTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/StorageImplTest.java index f32a51507857..612664de14ae 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/StorageImplTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/StorageImplTest.java @@ -200,11 +200,14 @@ public class StorageImplTest { Storage.BlobListOption.prefix("prefix"); private static final Storage.BlobListOption BLOB_LIST_FIELDS = Storage.BlobListOption.fields(Storage.BlobField.CONTENT_TYPE, Storage.BlobField.MD5HASH); + private static final Storage.BlobListOption BLOB_LIST_VERSIONS = + Storage.BlobListOption.versions(false); private static final Storage.BlobListOption BLOB_LIST_EMPTY_FIELDS = Storage.BlobListOption.fields(); private static final Map BLOB_LIST_OPTIONS = ImmutableMap.of( StorageRpc.Option.MAX_RESULTS, BLOB_LIST_MAX_RESULT.value(), - StorageRpc.Option.PREFIX, BLOB_LIST_PREFIX.value()); + StorageRpc.Option.PREFIX, BLOB_LIST_PREFIX.value(), + StorageRpc.Option.VERSIONS, BLOB_LIST_VERSIONS.value()); private static final String PRIVATE_KEY_STRING = "MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoG" + "BAL2xolH1zrISQ8+GzOV29BNjjzq4/HIP8Psd1+cZb81vDklSF+95wB250MSE0BDc81pvIMwj5OmIfLg1NY6uB" @@ -238,6 +241,9 @@ public long millis() { private StorageRpc storageRpcMock; private Storage storage; + private Blob expectedBlob1, expectedBlob2, expectedBlob3; + private Bucket expectedBucket1, expectedBucket2; + @Rule public ExpectedException thrown = ExpectedException.none(); @@ -272,10 +278,23 @@ public void tearDown() throws Exception { EasyMock.verify(rpcFactoryMock, storageRpcMock); } + private void initializeService() { + storage = options.service(); + initializeServiceDependentObjects(); + } + + private void initializeServiceDependentObjects() { + expectedBlob1 = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO1)); + expectedBlob2 = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO2)); + expectedBlob3 = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO3)); + expectedBucket1 = new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO1)); + expectedBucket2 = new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO2)); + } + @Test public void testGetOptions() { EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertSame(options, storage.options()); } @@ -284,9 +303,9 @@ public void testCreateBucket() { EasyMock.expect(storageRpcMock.create(BUCKET_INFO1.toPb(), EMPTY_RPC_OPTIONS)) .andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.create(BUCKET_INFO1); - assertEquals(BUCKET_INFO1.toPb(), bucket.toPb()); + initializeService(); + Bucket bucket = storage.create(BUCKET_INFO1); + assertEquals(expectedBucket1, bucket); } @Test @@ -294,10 +313,10 @@ public void testCreateBucketWithOptions() { EasyMock.expect(storageRpcMock.create(BUCKET_INFO1.toPb(), BUCKET_TARGET_OPTIONS)) .andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = + initializeService(); + Bucket bucket = storage.create(BUCKET_INFO1, BUCKET_TARGET_METAGENERATION, BUCKET_TARGET_PREDEFINED_ACL); - assertEquals(BUCKET_INFO1, bucket); + assertEquals(expectedBucket1, bucket); } @Test @@ -309,9 +328,9 @@ public void testCreateBlob() throws IOException { EasyMock.eq(EMPTY_RPC_OPTIONS))) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.create(BLOB_INFO1, BLOB_CONTENT); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.create(BLOB_INFO1, BLOB_CONTENT); + assertEquals(expectedBlob1, blob); ByteArrayInputStream byteStream = capturedStream.getValue(); byte[] streamBytes = new byte[BLOB_CONTENT.length]; assertEquals(BLOB_CONTENT.length, byteStream.read(streamBytes)); @@ -332,9 +351,9 @@ public void testCreateEmptyBlob() throws IOException { EasyMock.eq(EMPTY_RPC_OPTIONS))) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.create(BLOB_INFO1); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.create(BLOB_INFO1); + assertEquals(expectedBlob1, blob); ByteArrayInputStream byteStream = capturedStream.getValue(); byte[] streamBytes = new byte[BLOB_CONTENT.length]; assertEquals(-1, byteStream.read(streamBytes)); @@ -353,11 +372,11 @@ public void testCreateBlobWithOptions() throws IOException { EasyMock.eq(BLOB_TARGET_OPTIONS_CREATE))) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = + initializeService(); + Blob blob = storage.create(BLOB_INFO1, BLOB_CONTENT, BLOB_TARGET_METAGENERATION, BLOB_TARGET_NOT_EXIST, BLOB_TARGET_PREDEFINED_ACL); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); ByteArrayInputStream byteStream = capturedStream.getValue(); byte[] streamBytes = new byte[BLOB_CONTENT.length]; assertEquals(BLOB_CONTENT.length, byteStream.read(streamBytes)); @@ -374,9 +393,9 @@ public void testCreateBlobFromStream() { EasyMock.expect(storageRpcMock.create(infoWithoutHashes.toPb(), fileStream, EMPTY_RPC_OPTIONS)) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.create(infoWithHashes, fileStream); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.create(infoWithHashes, fileStream); + assertEquals(expectedBlob1, blob); } @Test @@ -384,9 +403,9 @@ public void testGetBucket() { EasyMock.expect(storageRpcMock.get(BucketInfo.of(BUCKET_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.get(BUCKET_NAME1); - assertEquals(BUCKET_INFO1, bucket); + initializeService(); + Bucket bucket = storage.get(BUCKET_NAME1); + assertEquals(expectedBucket1, bucket); } @Test @@ -394,9 +413,9 @@ public void testGetBucketWithOptions() { EasyMock.expect(storageRpcMock.get(BucketInfo.of(BUCKET_NAME1).toPb(), BUCKET_GET_OPTIONS)) .andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION); - assertEquals(BUCKET_INFO1, bucket); + initializeService(); + Bucket bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION); + assertEquals(expectedBucket1, bucket); } @Test @@ -405,8 +424,8 @@ public void testGetBucketWithSelectedFields() { EasyMock.expect(storageRpcMock.get(EasyMock.eq(BucketInfo.of(BUCKET_NAME1).toPb()), EasyMock.capture(capturedOptions))).andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION, BUCKET_GET_FIELDS); + initializeService(); + Bucket bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION, BUCKET_GET_FIELDS); assertEquals(BUCKET_GET_METAGENERATION.value(), capturedOptions.getValue().get(BUCKET_GET_METAGENERATION.rpcOption())); String selector = (String) capturedOptions.getValue().get(BLOB_GET_FIELDS.rpcOption()); @@ -423,8 +442,8 @@ public void testGetBucketWithEmptyFields() { EasyMock.expect(storageRpcMock.get(EasyMock.eq(BucketInfo.of(BUCKET_NAME1).toPb()), EasyMock.capture(capturedOptions))).andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION, + initializeService(); + Bucket bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION, BUCKET_GET_EMPTY_FIELDS); assertEquals(BUCKET_GET_METAGENERATION.value(), capturedOptions.getValue().get(BUCKET_GET_METAGENERATION.rpcOption())); @@ -440,9 +459,9 @@ public void testGetBlob() { storageRpcMock.get(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.get(BUCKET_NAME1, BLOB_NAME1); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.get(BUCKET_NAME1, BLOB_NAME1); + assertEquals(expectedBlob1, blob); } @Test @@ -451,10 +470,10 @@ public void testGetBlobWithOptions() { storageRpcMock.get(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), BLOB_GET_OPTIONS)) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = + initializeService(); + Blob blob = storage.get(BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, BLOB_GET_GENERATION); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); } @Test @@ -463,10 +482,10 @@ public void testGetBlobWithOptionsFromBlobId() { storageRpcMock.get(BLOB_INFO1.blobId().toPb(), BLOB_GET_OPTIONS)) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = + initializeService(); + Blob blob = storage.get(BLOB_INFO1.blobId(), BLOB_GET_METAGENERATION, BLOB_GET_GENERATION_FROM_BLOB_ID); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); } @Test @@ -475,8 +494,9 @@ public void testGetBlobWithSelectedFields() { EasyMock.expect(storageRpcMock.get(EasyMock.eq(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb()), EasyMock.capture(capturedOptions))).andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.get(BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, + initializeService(); + Blob blob = storage.get( + BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, BLOB_GET_GENERATION, BLOB_GET_FIELDS); assertEquals(BLOB_GET_METAGENERATION.value(), capturedOptions.getValue().get(BLOB_GET_METAGENERATION.rpcOption())); @@ -488,7 +508,7 @@ public void testGetBlobWithSelectedFields() { assertTrue(selector.contains("contentType")); assertTrue(selector.contains("crc32c")); assertEquals(30, selector.length()); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); } @Test @@ -497,8 +517,8 @@ public void testGetBlobWithEmptyFields() { EasyMock.expect(storageRpcMock.get(EasyMock.eq(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb()), EasyMock.capture(capturedOptions))).andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.get(BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, + initializeService(); + Blob blob = storage.get(BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, BLOB_GET_GENERATION, BLOB_GET_EMPTY_FIELDS); assertEquals(BLOB_GET_METAGENERATION.value(), capturedOptions.getValue().get(BLOB_GET_METAGENERATION.rpcOption())); @@ -508,21 +528,22 @@ public void testGetBlobWithEmptyFields() { assertTrue(selector.contains("bucket")); assertTrue(selector.contains("name")); assertEquals(11, selector.length()); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); } @Test public void testListBuckets() { String cursor = "cursor"; - ImmutableList bucketList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); + ImmutableList bucketInfoList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(bucketList, BucketInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(bucketInfoList, BucketInfo.TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(); + initializeService(); + ImmutableList bucketList = ImmutableList.of(expectedBucket1, expectedBucket2); + Page page = storage.list(); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test @@ -530,79 +551,84 @@ public void testListBucketsEmpty() { EasyMock.expect(storageRpcMock.list(EMPTY_RPC_OPTIONS)).andReturn( Tuple.>of(null, null)); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(); + initializeService(); + Page page = storage.list(); assertNull(page.nextPageCursor()); - assertArrayEquals(ImmutableList.of().toArray(), - Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(ImmutableList.of().toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test public void testListBucketsWithOptions() { String cursor = "cursor"; - ImmutableList bucketList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); + ImmutableList bucketInfoList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(bucketList, BucketInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(bucketInfoList, BucketInfo.TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(BUCKET_LIST_OPTIONS)).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_LIST_MAX_RESULT, BUCKET_LIST_PREFIX); + initializeService(); + ImmutableList bucketList = ImmutableList.of(expectedBucket1, expectedBucket2); + Page page = storage.list(BUCKET_LIST_MAX_RESULT, BUCKET_LIST_PREFIX); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test public void testListBucketsWithSelectedFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList bucketList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); + ImmutableList bucketInfoList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(bucketList, BucketInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(bucketInfoList, BucketInfo.TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(EasyMock.capture(capturedOptions))).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_LIST_FIELDS); + initializeService(); + ImmutableList bucketList = ImmutableList.of(expectedBucket1, expectedBucket2); + Page page = storage.list(BUCKET_LIST_FIELDS); String selector = (String) capturedOptions.getValue().get(BLOB_LIST_FIELDS.rpcOption()); assertTrue(selector.contains("items")); assertTrue(selector.contains("name")); assertTrue(selector.contains("acl")); assertTrue(selector.contains("location")); - assertEquals(24, selector.length()); + assertTrue(selector.contains("nextPageToken")); + assertEquals(38, selector.length()); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test public void testListBucketsWithEmptyFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList bucketList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); + ImmutableList bucketInfoList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(bucketList, BucketInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(bucketInfoList, BucketInfo.TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(EasyMock.capture(capturedOptions))).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_LIST_EMPTY_FIELDS); + initializeService(); + ImmutableList bucketList = ImmutableList.of(expectedBucket1, expectedBucket2); + Page page = storage.list(BUCKET_LIST_EMPTY_FIELDS); String selector = (String) capturedOptions.getValue().get(BLOB_LIST_FIELDS.rpcOption()); assertTrue(selector.contains("items")); assertTrue(selector.contains("name")); - assertEquals(11, selector.length()); + assertTrue(selector.contains("nextPageToken")); + assertEquals(25, selector.length()); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test public void testListBlobs() { String cursor = "cursor"; - ImmutableList blobList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); + ImmutableList blobInfoList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(blobList, BlobInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(blobInfoList, BlobInfo.INFO_TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(BUCKET_NAME1, EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_NAME1); + initializeService(); + ImmutableList blobList = ImmutableList.of(expectedBlob1, expectedBlob2); + Page page = storage.list(BUCKET_NAME1); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test @@ -611,40 +637,42 @@ public void testListBlobsEmpty() { .andReturn(Tuple.>of( null, null)); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_NAME1); + initializeService(); + Page page = storage.list(BUCKET_NAME1); assertNull(page.nextPageCursor()); - assertArrayEquals(ImmutableList.of().toArray(), - Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(ImmutableList.of().toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test public void testListBlobsWithOptions() { String cursor = "cursor"; - ImmutableList blobList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); + ImmutableList blobInfoList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(blobList, BlobInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(blobInfoList, BlobInfo.INFO_TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(BUCKET_NAME1, BLOB_LIST_OPTIONS)).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_NAME1, BLOB_LIST_MAX_RESULT, BLOB_LIST_PREFIX); + initializeService(); + ImmutableList blobList = ImmutableList.of(expectedBlob1, expectedBlob2); + Page page = + storage.list(BUCKET_NAME1, BLOB_LIST_MAX_RESULT, BLOB_LIST_PREFIX, BLOB_LIST_VERSIONS); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test public void testListBlobsWithSelectedFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList blobList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); + ImmutableList blobInfoList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(blobList, BlobInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(blobInfoList, BlobInfo.INFO_TO_PB_FUNCTION)); EasyMock.expect( storageRpcMock.list(EasyMock.eq(BUCKET_NAME1), EasyMock.capture(capturedOptions))) .andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = + initializeService(); + ImmutableList blobList = ImmutableList.of(expectedBlob1, expectedBlob2); + Page page = storage.list(BUCKET_NAME1, BLOB_LIST_MAX_RESULT, BLOB_LIST_PREFIX, BLOB_LIST_FIELDS); assertEquals(BLOB_LIST_MAX_RESULT.value(), capturedOptions.getValue().get(BLOB_LIST_MAX_RESULT.rpcOption())); @@ -656,24 +684,26 @@ public void testListBlobsWithSelectedFields() { assertTrue(selector.contains("name")); assertTrue(selector.contains("contentType")); assertTrue(selector.contains("md5Hash")); - assertEquals(38, selector.length()); + assertTrue(selector.contains("nextPageToken")); + assertEquals(52, selector.length()); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test public void testListBlobsWithEmptyFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList blobList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); + ImmutableList blobInfoList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(blobList, BlobInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(blobInfoList, BlobInfo.INFO_TO_PB_FUNCTION)); EasyMock.expect( storageRpcMock.list(EasyMock.eq(BUCKET_NAME1), EasyMock.capture(capturedOptions))) .andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = + initializeService(); + ImmutableList blobList = ImmutableList.of(expectedBlob1, expectedBlob2); + Page page = storage.list(BUCKET_NAME1, BLOB_LIST_MAX_RESULT, BLOB_LIST_PREFIX, BLOB_LIST_EMPTY_FIELDS); assertEquals(BLOB_LIST_MAX_RESULT.value(), capturedOptions.getValue().get(BLOB_LIST_MAX_RESULT.rpcOption())); @@ -683,9 +713,10 @@ public void testListBlobsWithEmptyFields() { assertTrue(selector.contains("items")); assertTrue(selector.contains("bucket")); assertTrue(selector.contains("name")); - assertEquals(18, selector.length()); + assertTrue(selector.contains("nextPageToken")); + assertEquals(32, selector.length()); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test @@ -694,9 +725,9 @@ public void testUpdateBucket() { EasyMock.expect(storageRpcMock.patch(updatedBucketInfo.toPb(), EMPTY_RPC_OPTIONS)) .andReturn(updatedBucketInfo.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.update(updatedBucketInfo); - assertEquals(updatedBucketInfo, bucket); + initializeService(); + Bucket bucket = storage.update(updatedBucketInfo); + assertEquals(new Bucket(storage, new BucketInfo.BuilderImpl(updatedBucketInfo)), bucket); } @Test @@ -705,11 +736,11 @@ public void testUpdateBucketWithOptions() { EasyMock.expect(storageRpcMock.patch(updatedBucketInfo.toPb(), BUCKET_TARGET_OPTIONS)) .andReturn(updatedBucketInfo.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = + initializeService(); + Bucket bucket = storage.update(updatedBucketInfo, BUCKET_TARGET_METAGENERATION, BUCKET_TARGET_PREDEFINED_ACL); - assertEquals(updatedBucketInfo, bucket); + assertEquals(new Bucket(storage, new BucketInfo.BuilderImpl(updatedBucketInfo)), bucket); } @Test @@ -718,9 +749,9 @@ public void testUpdateBlob() { EasyMock.expect(storageRpcMock.patch(updatedBlobInfo.toPb(), EMPTY_RPC_OPTIONS)) .andReturn(updatedBlobInfo.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.update(updatedBlobInfo); - assertEquals(updatedBlobInfo, blob); + initializeService(); + Blob blob = storage.update(updatedBlobInfo); + assertEquals(new Blob(storage, new BlobInfo.BuilderImpl(updatedBlobInfo)), blob); } @Test @@ -729,10 +760,10 @@ public void testUpdateBlobWithOptions() { EasyMock.expect(storageRpcMock.patch(updatedBlobInfo.toPb(), BLOB_TARGET_OPTIONS_UPDATE)) .andReturn(updatedBlobInfo.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = + initializeService(); + Blob blob = storage.update(updatedBlobInfo, BLOB_TARGET_METAGENERATION, BLOB_TARGET_PREDEFINED_ACL); - assertEquals(updatedBlobInfo, blob); + assertEquals(new Blob(storage, new BlobInfo.BuilderImpl(updatedBlobInfo)), blob); } @Test @@ -740,7 +771,7 @@ public void testDeleteBucket() { EasyMock.expect(storageRpcMock.delete(BucketInfo.of(BUCKET_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BUCKET_NAME1)); } @@ -750,7 +781,7 @@ public void testDeleteBucketWithOptions() { .expect(storageRpcMock.delete(BucketInfo.of(BUCKET_NAME1).toPb(), BUCKET_SOURCE_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BUCKET_NAME1, BUCKET_SOURCE_METAGENERATION)); } @@ -760,7 +791,7 @@ public void testDeleteBlob() { storageRpcMock.delete(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BUCKET_NAME1, BLOB_NAME1)); } @@ -770,7 +801,7 @@ public void testDeleteBlobWithOptions() { storageRpcMock.delete(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), BLOB_SOURCE_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BUCKET_NAME1, BLOB_NAME1, BLOB_SOURCE_GENERATION, BLOB_SOURCE_METAGENERATION)); } @@ -781,7 +812,7 @@ public void testDeleteBlobWithOptionsFromBlobId() { storageRpcMock.delete(BLOB_INFO1.blobId().toPb(), BLOB_SOURCE_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BLOB_INFO1.blobId(), BLOB_SOURCE_GENERATION_FROM_BLOB_ID, BLOB_SOURCE_METAGENERATION)); } @@ -795,9 +826,9 @@ public void testCompose() { EasyMock.expect(storageRpcMock.compose(ImmutableList.of(BLOB_INFO2.toPb(), BLOB_INFO3.toPb()), BLOB_INFO1.toPb(), EMPTY_RPC_OPTIONS)).andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.compose(req); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.compose(req); + assertEquals(expectedBlob1, blob); } @Test @@ -810,9 +841,9 @@ public void testComposeWithOptions() { EasyMock.expect(storageRpcMock.compose(ImmutableList.of(BLOB_INFO2.toPb(), BLOB_INFO3.toPb()), BLOB_INFO1.toPb(), BLOB_TARGET_OPTIONS_COMPOSE)).andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.compose(req); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.compose(req); + assertEquals(expectedBlob1, blob); } @Test @@ -824,7 +855,7 @@ public void testCopy() { false, "token", 21L); EasyMock.expect(storageRpcMock.openRewrite(rpcRequest)).andReturn(rpcResponse); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); CopyWriter writer = storage.copy(request); assertEquals(42L, writer.blobSize()); assertEquals(21L, writer.totalBytesCopied()); @@ -844,7 +875,7 @@ public void testCopyWithOptions() { false, "token", 21L); EasyMock.expect(storageRpcMock.openRewrite(rpcRequest)).andReturn(rpcResponse); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); CopyWriter writer = storage.copy(request); assertEquals(42L, writer.blobSize()); assertEquals(21L, writer.totalBytesCopied()); @@ -864,7 +895,7 @@ public void testCopyWithOptionsFromBlobId() { new StorageRpc.RewriteResponse(rpcRequest, null, 42L, false, "token", 21L); EasyMock.expect(storageRpcMock.openRewrite(rpcRequest)).andReturn(rpcResponse); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); CopyWriter writer = storage.copy(request); assertEquals(42L, writer.blobSize()); assertEquals(21L, writer.totalBytesCopied()); @@ -883,7 +914,7 @@ public void testCopyMultipleRequests() { EasyMock.expect(storageRpcMock.openRewrite(rpcRequest)).andReturn(rpcResponse1); EasyMock.expect(storageRpcMock.continueRewrite(rpcResponse1)).andReturn(rpcResponse2); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); CopyWriter writer = storage.copy(request); assertEquals(42L, writer.blobSize()); assertEquals(21L, writer.totalBytesCopied()); @@ -900,7 +931,7 @@ public void testReadAllBytes() { storageRpcMock.load(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(BLOB_CONTENT); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); byte[] readBytes = storage.readAllBytes(BUCKET_NAME1, BLOB_NAME1); assertArrayEquals(BLOB_CONTENT, readBytes); } @@ -911,7 +942,7 @@ public void testReadAllBytesWithOptions() { storageRpcMock.load(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), BLOB_SOURCE_OPTIONS)) .andReturn(BLOB_CONTENT); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); byte[] readBytes = storage.readAllBytes(BUCKET_NAME1, BLOB_NAME1, BLOB_SOURCE_GENERATION, BLOB_SOURCE_METAGENERATION); assertArrayEquals(BLOB_CONTENT, readBytes); @@ -923,7 +954,7 @@ public void testReadAllBytesWithOptionsFromBlobId() { storageRpcMock.load(BLOB_INFO1.blobId().toPb(), BLOB_SOURCE_OPTIONS)) .andReturn(BLOB_CONTENT); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); byte[] readBytes = storage.readAllBytes(BLOB_INFO1.blobId(), BLOB_SOURCE_GENERATION_FROM_BLOB_ID, BLOB_SOURCE_METAGENERATION); assertArrayEquals(BLOB_CONTENT, readBytes); @@ -970,11 +1001,10 @@ public Tuple apply(StorageObject f) { StorageRpc.BatchResponse res = new StorageRpc.BatchResponse(deleteResult, updateResult, getResult); - Capture capturedBatchRequest = Capture.newInstance(); EasyMock.expect(storageRpcMock.batch(EasyMock.capture(capturedBatchRequest))).andReturn(res); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); BatchResponse batchResponse = storage.submit(req); // Verify captured StorageRpc.BatchRequest @@ -1012,7 +1042,7 @@ public Tuple apply(StorageObject f) { @Test public void testReader() { EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); ReadChannel channel = storage.reader(BUCKET_NAME1, BLOB_NAME1); assertNotNull(channel); assertTrue(channel.isOpen()); @@ -1025,7 +1055,7 @@ public void testReaderWithOptions() throws IOException { storageRpcMock.read(BLOB_INFO2.toPb(), BLOB_SOURCE_OPTIONS, 0, DEFAULT_CHUNK_SIZE)) .andReturn(StorageRpc.Tuple.of("etag", result)); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); ReadChannel channel = storage.reader(BUCKET_NAME1, BLOB_NAME2, BLOB_SOURCE_GENERATION, BLOB_SOURCE_METAGENERATION); assertNotNull(channel); @@ -1040,7 +1070,7 @@ public void testReaderWithOptionsFromBlobId() throws IOException { storageRpcMock.read(BLOB_INFO1.blobId().toPb(), BLOB_SOURCE_OPTIONS, 0, DEFAULT_CHUNK_SIZE)) .andReturn(StorageRpc.Tuple.of("etag", result)); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); ReadChannel channel = storage.reader(BLOB_INFO1.blobId(), BLOB_SOURCE_GENERATION_FROM_BLOB_ID, BLOB_SOURCE_METAGENERATION); assertNotNull(channel); @@ -1056,7 +1086,7 @@ public void testWriter() { EasyMock.expect(storageRpcMock.open(infoWithoutHashes.toPb(), EMPTY_RPC_OPTIONS)) .andReturn("upload-id"); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); WriteChannel channel = storage.writer(infoWithHashes); assertNotNull(channel); assertTrue(channel.isOpen()); @@ -1068,7 +1098,7 @@ public void testWriterWithOptions() { EasyMock.expect(storageRpcMock.open(info.toPb(), BLOB_TARGET_OPTIONS_CREATE)) .andReturn("upload-id"); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); WriteChannel channel = storage.writer(info, BLOB_WRITE_METAGENERATION, BLOB_WRITE_NOT_EXIST, BLOB_WRITE_PREDEFINED_ACL, BLOB_WRITE_CRC2C, BLOB_WRITE_MD5_HASH); assertNotNull(channel); @@ -1154,12 +1184,11 @@ public Tuple apply(StorageObject f) { StorageRpc.BatchResponse res = new StorageRpc.BatchResponse(deleteResult, updateResult, getResult); - Capture capturedBatchRequest = Capture.newInstance(); EasyMock.expect(storageRpcMock.batch(EasyMock.capture(capturedBatchRequest))).andReturn(res); EasyMock.replay(storageRpcMock); - storage = options.service(); - List resultBlobs = storage.get(blobId1, blobId2); + initializeService(); + List resultBlobs = storage.get(blobId1, blobId2); // Verify captured StorageRpc.BatchRequest List>> capturedToGet = @@ -1197,12 +1226,11 @@ public Tuple apply(StorageObject f) { StorageRpc.BatchResponse res = new StorageRpc.BatchResponse(deleteResult, updateResult, getResult); - Capture capturedBatchRequest = Capture.newInstance(); EasyMock.expect(storageRpcMock.batch(EasyMock.capture(capturedBatchRequest))).andReturn(res); EasyMock.replay(storageRpcMock); - storage = options.service(); - List resultBlobs = storage.update(blobInfo1, blobInfo2); + initializeService(); + List resultBlobs = storage.update(blobInfo1, blobInfo2); // Verify captured StorageRpc.BatchRequest List>> capturedToUpdate = @@ -1243,7 +1271,7 @@ public Tuple apply(StorageObject f) { Capture capturedBatchRequest = Capture.newInstance(); EasyMock.expect(storageRpcMock.batch(EasyMock.capture(capturedBatchRequest))).andReturn(res); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); List deleteResults = storage.delete(blobInfo1.blobId(), blobInfo2.blobId()); // Verify captured StorageRpc.BatchRequest @@ -1270,8 +1298,9 @@ public void testRetryableException() { .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); storage = options.toBuilder().retryParams(RetryParams.defaultInstance()).build().service(); - BlobInfo readBlob = storage.get(blob); - assertEquals(BLOB_INFO1, readBlob); + initializeServiceDependentObjects(); + Blob readBlob = storage.get(blob); + assertEquals(expectedBlob1, readBlob); } @Test @@ -1282,6 +1311,7 @@ public void testNonRetryableException() { .andThrow(new StorageException(501, exceptionMessage)); EasyMock.replay(storageRpcMock); storage = options.toBuilder().retryParams(RetryParams.defaultInstance()).build().service(); + initializeServiceDependentObjects(); thrown.expect(StorageException.class); thrown.expectMessage(exceptionMessage); storage.get(blob); diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/ITStorageTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/it/ITStorageTest.java similarity index 80% rename from gcloud-java-storage/src/test/java/com/google/gcloud/storage/ITStorageTest.java rename to gcloud-java-storage/src/test/java/com/google/gcloud/storage/it/ITStorageTest.java index 63b9d739b686..8e954de57e68 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/ITStorageTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/it/ITStorageTest.java @@ -14,7 +14,7 @@ * limitations under the License. */ -package com.google.gcloud.storage; +package com.google.gcloud.storage.it; import static java.nio.charset.StandardCharsets.UTF_8; import static org.junit.Assert.assertArrayEquals; @@ -25,15 +25,27 @@ import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; -import com.google.api.client.util.Lists; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import com.google.common.collect.Lists; import com.google.gcloud.Page; import com.google.gcloud.ReadChannel; import com.google.gcloud.RestorableState; import com.google.gcloud.WriteChannel; +import com.google.gcloud.storage.BatchRequest; +import com.google.gcloud.storage.BatchResponse; +import com.google.gcloud.storage.Blob; +import com.google.gcloud.storage.BlobId; +import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.Bucket; +import com.google.gcloud.storage.BucketInfo; +import com.google.gcloud.storage.CopyWriter; +import com.google.gcloud.storage.HttpMethod; +import com.google.gcloud.storage.Storage; import com.google.gcloud.storage.Storage.BlobField; import com.google.gcloud.storage.Storage.BucketField; +import com.google.gcloud.storage.StorageException; import com.google.gcloud.storage.testing.RemoteGcsHelper; import org.junit.AfterClass; @@ -52,6 +64,7 @@ import java.util.List; import java.util.Map; import java.util.Random; +import java.util.Set; import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; import java.util.logging.Level; @@ -77,8 +90,9 @@ public static void beforeClass() { @AfterClass public static void afterClass() throws ExecutionException, InterruptedException { - if (storage != null && !RemoteGcsHelper.forceDelete(storage, BUCKET, 5, TimeUnit.SECONDS)) { - if (log.isLoggable(Level.WARNING)) { + if (storage != null) { + boolean wasDeleted = RemoteGcsHelper.forceDelete(storage, BUCKET, 5, TimeUnit.SECONDS); + if (!wasDeleted && log.isLoggable(Level.WARNING)) { log.log(Level.WARNING, "Deletion of bucket {0} timed out, bucket is not empty", BUCKET); } } @@ -86,7 +100,8 @@ public static void afterClass() throws ExecutionException, InterruptedException @Test(timeout = 5000) public void testListBuckets() throws InterruptedException { - Iterator bucketIterator = storage.list(Storage.BucketListOption.prefix(BUCKET), + Iterator bucketIterator = + storage.list(Storage.BucketListOption.prefix(BUCKET), Storage.BucketListOption.fields()).values().iterator(); while (!bucketIterator.hasNext()) { Thread.sleep(500); @@ -94,7 +109,7 @@ public void testListBuckets() throws InterruptedException { Storage.BucketListOption.fields()).values().iterator(); } while (bucketIterator.hasNext()) { - BucketInfo remoteBucket = bucketIterator.next(); + Bucket remoteBucket = bucketIterator.next(); assertTrue(remoteBucket.name().startsWith(BUCKET)); assertNull(remoteBucket.createTime()); assertNull(remoteBucket.selfLink()); @@ -103,7 +118,7 @@ public void testListBuckets() throws InterruptedException { @Test public void testGetBucketSelectedFields() { - BucketInfo remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields(BucketField.ID)); + Bucket remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields(BucketField.ID)); assertEquals(BUCKET, remoteBucket.name()); assertNull(remoteBucket.createTime()); assertNotNull(remoteBucket.id()); @@ -111,7 +126,7 @@ public void testGetBucketSelectedFields() { @Test public void testGetBucketAllSelectedFields() { - BucketInfo remoteBucket = storage.get(BUCKET, + Bucket remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields(BucketField.values())); assertEquals(BUCKET, remoteBucket.name()); assertNotNull(remoteBucket.createTime()); @@ -120,7 +135,7 @@ public void testGetBucketAllSelectedFields() { @Test public void testGetBucketEmptyFields() { - BucketInfo remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields()); + Bucket remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields()); assertEquals(BUCKET, remoteBucket.name()); assertNull(remoteBucket.createTime()); assertNull(remoteBucket.selfLink()); @@ -130,26 +145,26 @@ public void testGetBucketEmptyFields() { public void testCreateBlob() { String blobName = "test-create-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - BlobInfo remoteBlob = storage.create(blob, BLOB_BYTE_CONTENT); + Blob remoteBlob = storage.create(blob, BLOB_BYTE_CONTENT); assertNotNull(remoteBlob); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); byte[] readBytes = storage.readAllBytes(BUCKET, blobName); assertArrayEquals(BLOB_BYTE_CONTENT, readBytes); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testCreateEmptyBlob() { String blobName = "test-create-empty-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - BlobInfo remoteBlob = storage.create(blob); + Blob remoteBlob = storage.create(blob); assertNotNull(remoteBlob); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); byte[] readBytes = storage.readAllBytes(BUCKET, blobName); assertArrayEquals(new byte[0], readBytes); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -157,21 +172,22 @@ public void testCreateBlobStream() { String blobName = "test-create-blob-stream"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).contentType(CONTENT_TYPE).build(); ByteArrayInputStream stream = new ByteArrayInputStream(BLOB_STRING_CONTENT.getBytes(UTF_8)); - BlobInfo remoteBlob = storage.create(blob, stream); + Blob remoteBlob = storage.create(blob, stream); assertNotNull(remoteBlob); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); assertEquals(blob.contentType(), remoteBlob.contentType()); byte[] readBytes = storage.readAllBytes(BUCKET, blobName); assertEquals(BLOB_STRING_CONTENT, new String(readBytes, UTF_8)); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testCreateBlobFail() { String blobName = "test-create-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobInfo wrongGenerationBlob = BlobInfo.builder(BUCKET, blobName, -1L).build(); try { storage.create(wrongGenerationBlob, BLOB_BYTE_CONTENT, @@ -180,7 +196,7 @@ public void testCreateBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -204,10 +220,10 @@ public void testGetBlobEmptySelectedFields() { String blobName = "test-get-empty-selected-fields-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).contentType(CONTENT_TYPE).build(); assertNotNull(storage.create(blob)); - BlobInfo remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields()); + Blob remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields()); assertEquals(blob.blobId(), remoteBlob.blobId()); assertNull(remoteBlob.contentType()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -218,12 +234,12 @@ public void testGetBlobSelectedFields() { .metadata(ImmutableMap.of("k", "v")) .build(); assertNotNull(storage.create(blob)); - BlobInfo remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields( + Blob remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields( BlobField.METADATA)); assertEquals(blob.blobId(), remoteBlob.blobId()); assertEquals(ImmutableMap.of("k", "v"), remoteBlob.metadata()); assertNull(remoteBlob.contentType()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -234,21 +250,22 @@ public void testGetBlobAllSelectedFields() { .metadata(ImmutableMap.of("k", "v")) .build(); assertNotNull(storage.create(blob)); - BlobInfo remoteBlob = storage.get(blob.blobId(), + Blob remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields(BlobField.values())); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); assertEquals(ImmutableMap.of("k", "v"), remoteBlob.metadata()); assertNotNull(remoteBlob.id()); assertNotNull(remoteBlob.selfLink()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testGetBlobFail() { String blobName = "test-get-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobId wrongGenerationBlob = BlobId.of(BUCKET, blobName); try { storage.get(wrongGenerationBlob, Storage.BlobGetOption.generationMatch(-1)); @@ -256,17 +273,18 @@ public void testGetBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testGetBlobFailNonExistingGeneration() { String blobName = "test-get-blob-fail-non-existing-generation"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobId wrongGenerationBlob = BlobId.of(BUCKET, blobName, -1L); assertNull(storage.get(wrongGenerationBlob)); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -282,20 +300,22 @@ public void testListBlobsSelectedFields() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob1)); - assertNotNull(storage.create(blob2)); - Page page = storage.list(BUCKET, + Blob remoteBlob1 = storage.create(blob1); + Blob remoteBlob2 = storage.create(blob2); + assertNotNull(remoteBlob1); + assertNotNull(remoteBlob2); + Page page = storage.list(BUCKET, Storage.BlobListOption.prefix("test-list-blobs-selected-fields-blob"), Storage.BlobListOption.fields(BlobField.METADATA)); int index = 0; - for (BlobInfo remoteBlob : page.values()) { + for (Blob remoteBlob : page.values()) { assertEquals(BUCKET, remoteBlob.bucket()); assertEquals(blobNames[index++], remoteBlob.name()); assertEquals(metadata, remoteBlob.metadata()); assertNull(remoteBlob.contentType()); } - assertTrue(storage.delete(BUCKET, blobNames[0])); - assertTrue(storage.delete(BUCKET, blobNames[1])); + assertTrue(remoteBlob1.delete()); + assertTrue(remoteBlob2.delete()); } @Test @@ -308,32 +328,70 @@ public void testListBlobsEmptySelectedFields() { BlobInfo blob2 = BlobInfo.builder(BUCKET, blobNames[1]) .contentType(CONTENT_TYPE) .build(); - assertNotNull(storage.create(blob1)); - assertNotNull(storage.create(blob2)); - Page page = storage.list(BUCKET, + Blob remoteBlob1 = storage.create(blob1); + Blob remoteBlob2 = storage.create(blob2); + assertNotNull(remoteBlob1); + assertNotNull(remoteBlob2); + Page page = storage.list(BUCKET, Storage.BlobListOption.prefix("test-list-blobs-empty-selected-fields-blob"), Storage.BlobListOption.fields()); int index = 0; - for (BlobInfo remoteBlob : page.values()) { + for (Blob remoteBlob : page.values()) { assertEquals(BUCKET, remoteBlob.bucket()); assertEquals(blobNames[index++], remoteBlob.name()); assertNull(remoteBlob.contentType()); } - assertTrue(storage.delete(BUCKET, blobNames[0])); - assertTrue(storage.delete(BUCKET, blobNames[1])); + assertTrue(remoteBlob1.delete()); + assertTrue(remoteBlob2.delete()); + } + + @Test + public void testListBlobsVersioned() throws ExecutionException, InterruptedException { + String bucketName = RemoteGcsHelper.generateBucketName(); + Bucket bucket = storage.create(BucketInfo.builder(bucketName).versioningEnabled(true).build()); + try { + String[] blobNames = {"test-list-blobs-versioned-blob1", "test-list-blobs-versioned-blob2"}; + BlobInfo blob1 = BlobInfo.builder(bucket, blobNames[0]) + .contentType(CONTENT_TYPE) + .build(); + BlobInfo blob2 = BlobInfo.builder(bucket, blobNames[1]) + .contentType(CONTENT_TYPE) + .build(); + Blob remoteBlob1 = storage.create(blob1); + Blob remoteBlob2 = storage.create(blob2); + Blob remoteBlob3 = storage.create(blob2); + assertNotNull(remoteBlob1); + assertNotNull(remoteBlob2); + assertNotNull(remoteBlob3); + Page page = storage.list(bucketName, + Storage.BlobListOption.prefix("test-list-blobs-versioned-blob"), + Storage.BlobListOption.versions(true)); + Set blobSet = ImmutableSet.of(blobNames[0], blobNames[1]); + for (Blob remoteBlob : page.values()) { + assertEquals(bucketName, remoteBlob.bucket()); + assertTrue(blobSet.contains(remoteBlob.name())); + assertNotNull(remoteBlob.generation()); + } + assertTrue(remoteBlob1.delete()); + assertTrue(remoteBlob2.delete()); + assertTrue(remoteBlob3.delete()); + } finally { + RemoteGcsHelper.forceDelete(storage, bucketName, 5, TimeUnit.SECONDS); + } } @Test public void testUpdateBlob() { String blobName = "test-update-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); - BlobInfo updatedBlob = storage.update(blob.toBuilder().contentType(CONTENT_TYPE).build()); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); + Blob updatedBlob = remoteBlob.toBuilder().contentType(CONTENT_TYPE).build().update(); assertNotNull(updatedBlob); assertEquals(blob.name(), updatedBlob.name()); assertEquals(blob.bucket(), updatedBlob.bucket()); assertEquals(CONTENT_TYPE, updatedBlob.contentType()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(updatedBlob.delete()); } @Test @@ -345,15 +403,16 @@ public void testUpdateBlobReplaceMetadata() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob)); - BlobInfo updatedBlob = storage.update(blob.toBuilder().metadata(null).build()); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); + Blob updatedBlob = remoteBlob.toBuilder().metadata(null).build().update(); assertNotNull(updatedBlob); assertNull(updatedBlob.metadata()); - updatedBlob = storage.update(blob.toBuilder().metadata(newMetadata).build()); + updatedBlob = remoteBlob.toBuilder().metadata(newMetadata).build().update(); assertEquals(blob.name(), updatedBlob.name()); assertEquals(blob.bucket(), updatedBlob.bucket()); assertEquals(newMetadata, updatedBlob.metadata()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(updatedBlob.delete()); } @Test @@ -366,13 +425,14 @@ public void testUpdateBlobMergeMetadata() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob)); - BlobInfo updatedBlob = storage.update(blob.toBuilder().metadata(newMetadata).build()); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); + Blob updatedBlob = remoteBlob.toBuilder().metadata(newMetadata).build().update(); assertNotNull(updatedBlob); assertEquals(blob.name(), updatedBlob.name()); assertEquals(blob.bucket(), updatedBlob.bucket()); assertEquals(expectedMetadata, updatedBlob.metadata()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(updatedBlob.delete()); } @Test @@ -387,20 +447,22 @@ public void testUpdateBlobUnsetMetadata() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob)); - BlobInfo updatedBlob = storage.update(blob.toBuilder().metadata(newMetadata).build()); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); + Blob updatedBlob = remoteBlob.toBuilder().metadata(newMetadata).build().update(); assertNotNull(updatedBlob); assertEquals(blob.name(), updatedBlob.name()); assertEquals(blob.bucket(), updatedBlob.bucket()); assertEquals(expectedMetadata, updatedBlob.metadata()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(updatedBlob.delete()); } @Test public void testUpdateBlobFail() { String blobName = "test-update-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobInfo wrongGenerationBlob = BlobInfo.builder(BUCKET, blobName, -1L) .contentType(CONTENT_TYPE) .build(); @@ -410,13 +472,13 @@ public void testUpdateBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testDeleteNonExistingBlob() { String blobName = "test-delete-non-existing-blob"; - assertTrue(!storage.delete(BUCKET, blobName)); + assertFalse(storage.delete(BUCKET, blobName)); } @Test @@ -424,21 +486,22 @@ public void testDeleteBlobNonExistingGeneration() { String blobName = "test-delete-blob-non-existing-generation"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); assertNotNull(storage.create(blob)); - assertTrue(!storage.delete(BlobId.of(BUCKET, blobName, -1L))); + assertFalse(storage.delete(BlobId.of(BUCKET, blobName, -1L))); } @Test public void testDeleteBlobFail() { String blobName = "test-delete-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); try { storage.delete(BUCKET, blob.name(), Storage.BlobSourceOption.generationMatch(-1L)); fail("StorageException was expected"); } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blob.name())); + assertTrue(remoteBlob.delete()); } @Test @@ -447,24 +510,26 @@ public void testComposeBlob() { String sourceBlobName2 = "test-compose-blob-source-2"; BlobInfo sourceBlob1 = BlobInfo.builder(BUCKET, sourceBlobName1).build(); BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); - assertNotNull(storage.create(sourceBlob1, BLOB_BYTE_CONTENT)); - assertNotNull(storage.create(sourceBlob2, BLOB_BYTE_CONTENT)); + Blob remoteSourceBlob1 = storage.create(sourceBlob1, BLOB_BYTE_CONTENT); + Blob remoteSourceBlob2 = storage.create(sourceBlob2, BLOB_BYTE_CONTENT); + assertNotNull(remoteSourceBlob1); + assertNotNull(remoteSourceBlob2); String targetBlobName = "test-compose-blob-target"; BlobInfo targetBlob = BlobInfo.builder(BUCKET, targetBlobName).build(); Storage.ComposeRequest req = Storage.ComposeRequest.of(ImmutableList.of(sourceBlobName1, sourceBlobName2), targetBlob); - BlobInfo remoteBlob = storage.compose(req); - assertNotNull(remoteBlob); - assertEquals(targetBlob.name(), remoteBlob.name()); - assertEquals(targetBlob.bucket(), remoteBlob.bucket()); + Blob remoteTargetBlob = storage.compose(req); + assertNotNull(remoteTargetBlob); + assertEquals(targetBlob.name(), remoteTargetBlob.name()); + assertEquals(targetBlob.bucket(), remoteTargetBlob.bucket()); byte[] readBytes = storage.readAllBytes(BUCKET, targetBlobName); byte[] composedBytes = Arrays.copyOf(BLOB_BYTE_CONTENT, BLOB_BYTE_CONTENT.length * 2); System.arraycopy(BLOB_BYTE_CONTENT, 0, composedBytes, BLOB_BYTE_CONTENT.length, BLOB_BYTE_CONTENT.length); assertArrayEquals(composedBytes, readBytes); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); - assertTrue(storage.delete(BUCKET, targetBlobName)); + assertTrue(remoteSourceBlob1.delete()); + assertTrue(remoteSourceBlob2.delete()); + assertTrue(remoteTargetBlob.delete()); } @Test @@ -473,8 +538,10 @@ public void testComposeBlobFail() { String sourceBlobName2 = "test-compose-blob-fail-source-2"; BlobInfo sourceBlob1 = BlobInfo.builder(BUCKET, sourceBlobName1).build(); BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); - assertNotNull(storage.create(sourceBlob1)); - assertNotNull(storage.create(sourceBlob2)); + Blob remoteSourceBlob1 = storage.create(sourceBlob1); + Blob remoteSourceBlob2 = storage.create(sourceBlob2); + assertNotNull(remoteSourceBlob1); + assertNotNull(remoteSourceBlob2); String targetBlobName = "test-compose-blob-fail-target"; BlobInfo targetBlob = BlobInfo.builder(BUCKET, targetBlobName).build(); Storage.ComposeRequest req = Storage.ComposeRequest.builder() @@ -488,8 +555,8 @@ public void testComposeBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); + assertTrue(remoteSourceBlob1.delete()); + assertTrue(remoteSourceBlob2.delete()); } @Test @@ -501,7 +568,8 @@ public void testCopyBlob() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob, BLOB_BYTE_CONTENT)); + Blob remoteBlob = storage.create(blob, BLOB_BYTE_CONTENT); + assertNotNull(remoteBlob); String targetBlobName = "test-copy-blob-target"; Storage.CopyRequest req = Storage.CopyRequest.of(source, BlobId.of(BUCKET, targetBlobName)); CopyWriter copyWriter = storage.copy(req); @@ -510,7 +578,7 @@ public void testCopyBlob() { assertEquals(CONTENT_TYPE, copyWriter.result().contentType()); assertEquals(metadata, copyWriter.result().metadata()); assertTrue(copyWriter.isDone()); - assertTrue(storage.delete(BUCKET, sourceBlobName)); + assertTrue(remoteBlob.delete()); assertTrue(storage.delete(BUCKET, targetBlobName)); } @@ -518,7 +586,8 @@ public void testCopyBlob() { public void testCopyBlobUpdateMetadata() { String sourceBlobName = "test-copy-blob-update-metadata-source"; BlobId source = BlobId.of(BUCKET, sourceBlobName); - assertNotNull(storage.create(BlobInfo.builder(source).build(), BLOB_BYTE_CONTENT)); + Blob remoteSourceBlob = storage.create(BlobInfo.builder(source).build(), BLOB_BYTE_CONTENT); + assertNotNull(remoteSourceBlob); String targetBlobName = "test-copy-blob-update-metadata-target"; ImmutableMap metadata = ImmutableMap.of("k", "v"); BlobInfo target = BlobInfo.builder(BUCKET, targetBlobName) @@ -532,7 +601,7 @@ public void testCopyBlobUpdateMetadata() { assertEquals(CONTENT_TYPE, copyWriter.result().contentType()); assertEquals(metadata, copyWriter.result().metadata()); assertTrue(copyWriter.isDone()); - assertTrue(storage.delete(BUCKET, sourceBlobName)); + assertTrue(remoteSourceBlob.delete()); assertTrue(storage.delete(BUCKET, targetBlobName)); } @@ -540,7 +609,8 @@ public void testCopyBlobUpdateMetadata() { public void testCopyBlobFail() { String sourceBlobName = "test-copy-blob-source-fail"; BlobId source = BlobId.of(BUCKET, sourceBlobName, -1L); - assertNotNull(storage.create(BlobInfo.builder(source).build(), BLOB_BYTE_CONTENT)); + Blob remoteSourceBlob = storage.create(BlobInfo.builder(source).build(), BLOB_BYTE_CONTENT); + assertNotNull(remoteSourceBlob); String targetBlobName = "test-copy-blob-target-fail"; BlobInfo target = BlobInfo.builder(BUCKET, targetBlobName).contentType(CONTENT_TYPE).build(); Storage.CopyRequest req = Storage.CopyRequest.builder() @@ -565,7 +635,7 @@ public void testCopyBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, sourceBlobName)); + assertTrue(remoteSourceBlob.delete()); } @Test @@ -658,25 +728,26 @@ public void testBatchRequestManyDeletes() { } // Check updates - BlobInfo remoteUpdatedBlob2 = response.updates().get(0).get(); + Blob remoteUpdatedBlob2 = response.updates().get(0).get(); assertEquals(sourceBlob2.bucket(), remoteUpdatedBlob2.bucket()); assertEquals(sourceBlob2.name(), remoteUpdatedBlob2.name()); assertEquals(updatedBlob2.contentType(), remoteUpdatedBlob2.contentType()); // Check gets - BlobInfo remoteBlob1 = response.gets().get(0).get(); + Blob remoteBlob1 = response.gets().get(0).get(); assertEquals(sourceBlob1.bucket(), remoteBlob1.bucket()); assertEquals(sourceBlob1.name(), remoteBlob1.name()); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); + assertTrue(remoteBlob1.delete()); + assertTrue(remoteUpdatedBlob2.delete()); } @Test public void testBatchRequestFail() { String blobName = "test-batch-request-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobInfo updatedBlob = BlobInfo.builder(BUCKET, blobName, -1L).build(); BatchRequest batchRequest = BatchRequest.builder() .update(updatedBlob, Storage.BlobTargetOption.generationMatch()) @@ -696,7 +767,7 @@ public void testBatchRequestFail() { assertTrue(batchResponse.deletes().get(0).failed()); assertFalse(batchResponse.deletes().get(1).failed()); assertFalse(batchResponse.deletes().get(1).get()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -755,7 +826,8 @@ public void testReadAndWriteCaptureChannels() throws IOException { public void testReadChannelFail() throws IOException { String blobName = "test-read-channel-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); try (ReadChannel reader = storage.reader(blob.blobId(), Storage.BlobSourceOption.metagenerationMatch(-1L))) { reader.read(ByteBuffer.allocate(42)); @@ -778,7 +850,7 @@ public void testReadChannelFail() throws IOException { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -790,7 +862,7 @@ public void testReadChannelFailUpdatedGeneration() throws IOException { int blobSize = 2 * chunkSize; byte[] content = new byte[blobSize]; random.nextBytes(content); - BlobInfo remoteBlob = storage.create(blob, content); + Blob remoteBlob = storage.create(blob, content); assertNotNull(remoteBlob); assertEquals(blobSize, (long) remoteBlob.size()); try (ReadChannel reader = storage.reader(blob.blobId())) { @@ -834,9 +906,9 @@ public void testWriteChannelFail() throws IOException { public void testWriteChannelExistingBlob() throws IOException { String blobName = "test-write-channel-existing-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - BlobInfo remoteBlob = storage.create(blob); + storage.create(blob); byte[] stringBytes; - try (WriteChannel writer = storage.writer(remoteBlob)) { + try (WriteChannel writer = storage.writer(blob)) { stringBytes = BLOB_STRING_CONTENT.getBytes(UTF_8); writer.write(ByteBuffer.wrap(stringBytes)); } @@ -848,14 +920,15 @@ public void testWriteChannelExistingBlob() throws IOException { public void testGetSignedUrl() throws IOException { String blobName = "test-get-signed-url-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob, BLOB_BYTE_CONTENT)); + Blob remoteBlob = storage.create(blob, BLOB_BYTE_CONTENT); + assertNotNull(remoteBlob); URL url = storage.signUrl(blob, 1, TimeUnit.HOURS); URLConnection connection = url.openConnection(); byte[] readBytes = new byte[BLOB_BYTE_CONTENT.length]; try (InputStream responseStream = connection.getInputStream()) { assertEquals(BLOB_BYTE_CONTENT.length, responseStream.read(readBytes)); assertArrayEquals(BLOB_BYTE_CONTENT, readBytes); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } } @@ -869,11 +942,11 @@ public void testPostSignedUrl() throws IOException { URLConnection connection = url.openConnection(); connection.setDoOutput(true); connection.connect(); - BlobInfo remoteBlob = storage.get(BUCKET, blobName); + Blob remoteBlob = storage.get(BUCKET, blobName); assertNotNull(remoteBlob); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -884,13 +957,13 @@ public void testGetBlobs() { BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); assertNotNull(storage.create(sourceBlob1)); assertNotNull(storage.create(sourceBlob2)); - List remoteBlobs = storage.get(sourceBlob1.blobId(), sourceBlob2.blobId()); + List remoteBlobs = storage.get(sourceBlob1.blobId(), sourceBlob2.blobId()); assertEquals(sourceBlob1.bucket(), remoteBlobs.get(0).bucket()); assertEquals(sourceBlob1.name(), remoteBlobs.get(0).name()); assertEquals(sourceBlob2.bucket(), remoteBlobs.get(1).bucket()); assertEquals(sourceBlob2.name(), remoteBlobs.get(1).name()); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); + assertTrue(remoteBlobs.get(0).delete()); + assertTrue(remoteBlobs.get(1).delete()); } @Test @@ -900,11 +973,11 @@ public void testGetBlobsFail() { BlobInfo sourceBlob1 = BlobInfo.builder(BUCKET, sourceBlobName1).build(); BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); assertNotNull(storage.create(sourceBlob1)); - List remoteBlobs = storage.get(sourceBlob1.blobId(), sourceBlob2.blobId()); + List remoteBlobs = storage.get(sourceBlob1.blobId(), sourceBlob2.blobId()); assertEquals(sourceBlob1.bucket(), remoteBlobs.get(0).bucket()); assertEquals(sourceBlob1.name(), remoteBlobs.get(0).name()); assertNull(remoteBlobs.get(1)); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); + assertTrue(remoteBlobs.get(0).delete()); } @Test @@ -929,7 +1002,7 @@ public void testDeleteBlobsFail() { assertNotNull(storage.create(sourceBlob1)); List deleteStatus = storage.delete(sourceBlob1.blobId(), sourceBlob2.blobId()); assertTrue(deleteStatus.get(0)); - assertTrue(!deleteStatus.get(1)); + assertFalse(deleteStatus.get(1)); } @Test @@ -938,11 +1011,11 @@ public void testUpdateBlobs() { String sourceBlobName2 = "test-update-blobs-2"; BlobInfo sourceBlob1 = BlobInfo.builder(BUCKET, sourceBlobName1).build(); BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); - BlobInfo remoteBlob1 = storage.create(sourceBlob1); - BlobInfo remoteBlob2 = storage.create(sourceBlob2); + Blob remoteBlob1 = storage.create(sourceBlob1); + Blob remoteBlob2 = storage.create(sourceBlob2); assertNotNull(remoteBlob1); assertNotNull(remoteBlob2); - List updatedBlobs = storage.update( + List updatedBlobs = storage.update( remoteBlob1.toBuilder().contentType(CONTENT_TYPE).build(), remoteBlob2.toBuilder().contentType(CONTENT_TYPE).build()); assertEquals(sourceBlob1.bucket(), updatedBlobs.get(0).bucket()); @@ -951,8 +1024,8 @@ public void testUpdateBlobs() { assertEquals(sourceBlob2.bucket(), updatedBlobs.get(1).bucket()); assertEquals(sourceBlob2.name(), updatedBlobs.get(1).name()); assertEquals(CONTENT_TYPE, updatedBlobs.get(1).contentType()); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); + assertTrue(updatedBlobs.get(0).delete()); + assertTrue(updatedBlobs.get(1).delete()); } @Test @@ -963,13 +1036,13 @@ public void testUpdateBlobsFail() { BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); BlobInfo remoteBlob1 = storage.create(sourceBlob1); assertNotNull(remoteBlob1); - List updatedBlobs = storage.update( + List updatedBlobs = storage.update( remoteBlob1.toBuilder().contentType(CONTENT_TYPE).build(), sourceBlob2.toBuilder().contentType(CONTENT_TYPE).build()); assertEquals(sourceBlob1.bucket(), updatedBlobs.get(0).bucket()); assertEquals(sourceBlob1.name(), updatedBlobs.get(0).name()); assertEquals(CONTENT_TYPE, updatedBlobs.get(0).contentType()); assertNull(updatedBlobs.get(1)); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); + assertTrue(updatedBlobs.get(0).delete()); } } diff --git a/gcloud-java/README.md b/gcloud-java/README.md index 1f56696c3643..c51b4e8fe7bc 100644 --- a/gcloud-java/README.md +++ b/gcloud-java/README.md @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Platform][cloud-platform] services. [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs) @@ -25,16 +27,16 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java - 0.1.2 + 0.1.4 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java:0.1.2' +compile 'com.google.gcloud:gcloud-java:0.1.4' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.2" +libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.4" ``` Troubleshooting diff --git a/gcloud-java/pom.xml b/gcloud-java/pom.xml index 68dcdd661b6d..adfa716fe27b 100644 --- a/gcloud-java/pom.xml +++ b/gcloud-java/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java jar GCloud Java @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.3-SNAPSHOT + 0.1.5-SNAPSHOT diff --git a/pom.xml b/pom.xml index 7a435cc1dbae..880e2530be60 100644 --- a/pom.xml +++ b/pom.xml @@ -4,13 +4,31 @@ com.google.gcloud gcloud-java-pom pom - 0.1.3-SNAPSHOT + 0.1.5-SNAPSHOT GCloud Java https://github.com/GoogleCloudPlatform/gcloud-java Java idiomatic client for Google Cloud Platform services. + + derka + Martin Derka + derka@google.com + Google + + Developer + + + + ajaykannan + Ajay Kannan + ajaykannan@google.com + Google + + Developer + + ozarov Arie Ozarov @@ -20,6 +38,15 @@ Developer + + mziccardi + Marco Ziccardi + mziccardi@google.com + Google + + Developer + + Google @@ -68,6 +95,7 @@ gcloud-java gcloud-java-bigquery + gcloud-java-contrib gcloud-java-core gcloud-java-datastore gcloud-java-dns @@ -106,7 +134,7 @@ org.apache.maven.plugins maven-surefire-plugin - 2.18 + 2.19.1 @@ -117,7 +145,7 @@ org.apache.maven.plugins maven-enforcer-plugin - 1.4 + 1.4.1 enforce-maven @@ -159,7 +187,7 @@ org.apache.maven.plugins maven-failsafe-plugin - 2.18.1 + 2.19.1 @@ -191,7 +219,7 @@ maven-compiler-plugin - 3.2 + 3.5.1 1.7 1.7 @@ -202,7 +230,7 @@ org.apache.maven.plugins maven-source-plugin - 2.4 + 3.0.0 attach-sources @@ -242,7 +270,7 @@ org.sonatype.plugins nexus-staging-maven-plugin - 1.6.5 + 1.6.6 true sonatype-nexus-staging @@ -253,7 +281,7 @@ org.eluder.coveralls coveralls-maven-plugin - 3.1.0 + 4.1.0 ${basedir}/target/coverage.xml @@ -286,12 +314,12 @@ org.apache.maven.plugins maven-checkstyle-plugin - 2.16 + 2.17 com.puppycrawl.tools checkstyle - 6.8.1 + 6.15 @@ -310,7 +338,7 @@ org.apache.maven.plugins maven-project-info-reports-plugin - 2.8 + 2.8.1 @@ -367,12 +395,12 @@ org.apache.maven.plugins maven-surefire-report-plugin - 2.18.1 + 2.19.1 org.apache.maven.plugins maven-checkstyle-plugin - 2.16 + 2.17 checkstyle.xml false diff --git a/src/site/resources/index.html b/src/site/resources/index.html index 8f40cfbcd97e..0e0933e7b68c 100644 --- a/src/site/resources/index.html +++ b/src/site/resources/index.html @@ -178,8 +178,10 @@

    Examples

    • - SparkJava demo - Use gcloud-java with App Engine Managed VMs, Datastore, and SparkJava. + SparkJava demo - Uses gcloud-java with App Engine Managed VMs, Datastore, and SparkJava.
    • +
    • + Bookshelf - An App Engine app that manages a virtual bookshelf using gcloud-java libraries for Datastore and Storage.
    diff --git a/src/site/site.xml b/src/site/site.xml index 55047ce85c54..6279179eb389 100644 --- a/src/site/site.xml +++ b/src/site/site.xml @@ -20,7 +20,7 @@ org.apache.maven.skins maven-fluido-skin - 1.3.1 + 1.4 diff --git a/utilities/after_success.sh b/utilities/after_success.sh index 73f5fa95dec3..be7484806c46 100755 --- a/utilities/after_success.sh +++ b/utilities/after_success.sh @@ -1,5 +1,4 @@ #!/bin/bash -source ./utilities/integration_test_env.sh # This script is used by Travis-CI to publish artifacts (binary, sorce and javadoc jars) when releasing snapshots. # This script is referenced in .travis.yml. @@ -11,6 +10,7 @@ echo "Travis JDK version: " ${TRAVIS_JDK_VERSION} if [ "${TRAVIS_JDK_VERSION}" == "oraclejdk7" ]; then mvn clean cobertura:cobertura coveralls:report if [ "${TRAVIS_PULL_REQUEST}" == "false" -a "${TRAVIS_BRANCH}" == "master" ]; then + source ./utilities/integration_test_env.sh SITE_VERSION=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | grep -Ev '(^\[|\w+:)') if [ "${SITE_VERSION##*-}" != "SNAPSHOT" ]; then # Deploy site if not a SNAPSHOT diff --git a/utilities/verify.sh b/utilities/verify.sh index d98e4ab53513..b29ab8d8f747 100755 --- a/utilities/verify.sh +++ b/utilities/verify.sh @@ -1,10 +1,9 @@ #!/bin/bash -source ./utilities/integration_test_env.sh - # This script is used by Travis-CI to run tests. # This script is referenced in .travis.yml. if [ "${TRAVIS_PULL_REQUEST}" == "false" ]; then + source ./utilities/integration_test_env.sh # Get signing tools and API keyfile openssl aes-256-cbc -K $encrypted_631490ecae8f_key -iv $encrypted_631490ecae8f_iv -in target/travis/signing-tools.tar.enc -out $TRAVIS_BUILD_DIR/signing-tools.tar -d mkdir $TRAVIS_BUILD_DIR/signing-tools