Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

expand JSON test coverage; add datastore and cloud storage #443

Merged
merged 22 commits into from
Sep 8, 2017
Merged

expand JSON test coverage; add datastore and cloud storage #443

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
833b073
expand JSON test coverage
Aug 25, 2017
41e8f29
add test for equals
Aug 25, 2017
896ac8f
hamcrest
Aug 25, 2017
500ee93
redundant assert removed
Aug 25, 2017
eb8283a
fix loop
Aug 25, 2017
568fd0e
fix loop
Aug 25, 2017
653117f
regex versions
Aug 25, 2017
4de1ae1
don't make repo a corrdinate
Aug 25, 2017
6146737
type to packaging
Aug 25, 2017
4e6949f
verify group ID artifact ID
Aug 25, 2017
26e02d9
checkstyle fixes
Aug 25, 2017
a29e993
revert metadata for now
Aug 25, 2017
ec13151
code review
Aug 25, 2017
7ec06f6
add datastore api
Aug 25, 2017
6e23e11
toArray
Aug 25, 2017
fd5b606
GCS
Aug 26, 2017
867bd07
make transport an array
Aug 30, 2017
abaf735
make transport an array
Aug 30, 2017
7517ae3
launchStage
Aug 30, 2017
88a622e
pluralize transports
Aug 31, 2017
2c8e5d8
pluralize transports
Aug 31, 2017
05e0d09
add packages
Aug 31, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,18 @@
<version>2.7.17</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-core</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-library</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
65 changes: 58 additions & 7 deletions src/main/java/com/google/cloud/tools/libraries/libraries.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,24 +3,75 @@
"name": "Google BigQuery API",
"documentation": "https://cloud.google.com/bigquery/docs/reference/libraries",
"description":"Google's fully managed, petabyte scale, serverless data warehouse for analytics",
"transport": "http",
"transports": ["http"],
"icon":"https://cloud.google.com/_static/images/cloud/products/logos/svg/bigquery.svg",
"clients": [
{
"name":"Google Cloud Java Client for BigQuery",
"language":"java",
"site":"https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-install-java",
"apireference":"https://googlecloudplatform.github.io/google-cloud-java/0.22.0/apidocs/com/google/cloud/bigquery/package-summary.html",
"version" : "0.22.0",
"apireference":"https://googlecloudplatform.github.io/google-cloud-java/latest/apidocs/com/google/cloud/bigquery/package-summary.html",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be clearer as a language-specific field called javadoc?
Oh, actually we should have a javadoc field for programmatic access (e.g., Eclipse javadoc hovers) as well as a the more human-meaningful API starting point.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what would the difference be. For example, what javadoc URL would Eclipse need here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I try setting the Javadoc URL for a jar to this URL (right-click on a jar, like a bundle in the Plug-in Dependencies, and select Javadoc Location, and set as a URL), Eclipse shown an error:
screen shot 2017-08-28 at 4 32 01 pm

The java docs have a specific layout, as does the HTML. So Eclipse is able to find and parse out the appropriate Javadoc to be shown in a hover.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make that work, I think we first need to get google-cloud-java to publish individual javadoc for each API. The file it's looking for is at

https://googlecloudplatform.github.io/google-cloud-java/latest/apidocs/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably pull IDE readable source and javadoc jars from Maven Central rather than calling them out as a separate artifact.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to @elharo 's comment - then the javadoc and source will actually match the classes that the user is using.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CT for IntelliJ plans to expose these links to the end user so it needs human browsable URLs.

The links to the Maven source/javadoc jars don't need to be explicitly included here because they can be derived from the Maven coordinates.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good.

"infotip":"The com.google.cloud.bigquery packages",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be useful to encode in the metadata to enable UI discovery (e.g., a quick-fix on unresolved symbols)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you elaborate? I'm not sure exactly what you're proposing here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expressing this as:

"javaPackages":["com.google.cloud.bigquery", "com.google.cloud.bigquery.**"],

For example, PDE contributes a quick-fix to suggesting adding an Import-Package or Require-Bundle on an unresolved symbol that is found in a bundle:

screen shot 2017-08-28 at 4 22 18 pm

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be something different. This is human readable text for the infotip on the library.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to have this in a machine readable format too for the tooling reasons above.

Maybe it would help to add additional information here. For example, there are sometimes alpha versions of libraries that users may want to try. Or if this is the Google API Client library, we could add a pointer to the Google Client library?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. How do we tell Eclipse about our packages when adding a library?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd have a quick-fix on an unresolved symbol like PDE's quick-fix. Our quick-fix would iterate on the library definitions to see if it can find a match.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, ping on this request too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

"status" : "beta",
"launchStage" : "beta",
"packages" : ["com.google.cloud.bigquery"],
"source" : "https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-bigquery",
"languageLevel" : "1.7.0",
"mavenCoordinates" : {
"groupId" : "com.google.cloud",
"artifactId" : "google-cloud-bigquery",
"version" : "0.22.0-beta"
}
}
}
]
]
},
{
"name": "Google Cloud Datastore",
"documentation": "https://cloud.google.com/datastore/docs/reference/libraries",
"description":"Google Cloud Datastore is a NoSQL key-value database built for automatic scaling.",
"transports": ["http"],
"icon":"https://cloud.google.com/_static/images/cloud/products/logos/svg/datastore.svg",
"clients": [
{
"name":"Cloud Datastore Client Library for Java",
"language":"java",
"site":"https://cloud.google.com/datastore/docs/reference/libraries#client-libraries-install-java",
"apireference":"https://googlecloudplatform.github.io/google-cloud-java/latest/apidocs/index.html?com/google/cloud/datastore/package-summary.html",
"infotip":"The com.google.cloud.datastore packages",
"packages" : ["com.google.cloud.datastore"],
"launchStage" : "GA",
"source" : "https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-bigquery",
"languageLevel" : "1.7.0",
"mavenCoordinates" : {
"groupId" : "com.google.cloud",
"artifactId" : "google-cloud-datastore",
"version" : "1.4.0"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this version have to be updated with every google-cloud-java release?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course clients do have the option to ignore the version here and query Maven central for the latest release version. CT4E contains code to do this.

}
}
]
},
{
"name": "Google Cloud Storage",
"documentation": "https://cloud.google.com/storage/docs/reference/libraries",
"description":"Unified BLOB storage for developers and enterprises, from live data serving to data analytics/ML to data archiving.",
"transports": ["http"],
"icon":"https://cloud.google.com/_static/images/cloud/products/logos/svg/storage.svg",
"clients": [
{
"name":"Google Cloud Storage Client Library for Java",
"language":"java",
"site":"https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-java",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't the #using_the_client_library section be more appropriate, since the user will be using the consuming UI to automate the installation of this library?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I really wanted was the entire page, but that won't pick up Java as opposed to Python. This is the nearest link to the top.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me file a bug requesting a URL for the whole page.

"apireference" : "https://googlecloudplatform.github.io/google-cloud-java/latest/apidocs/index.html?com/google/cloud/storage/package-summary.html",
"packages" : ["com.google.cloud.storage"],
"infotip":"The com.google.cloud.storage packages for client side access to GCS",
"launchStage" : "GA",
"source" : "https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-storage",
"languageLevel" : "1.7.0",
"mavenCoordinates" : {
"groupId" : "com.google.cloud",
"artifactId" : "google-cloud-storage",
"version" : "1.4.0"
}
}
]
}
]
]
117 changes: 89 additions & 28 deletions src/test/java/com/google/cloud/tools/libraries/LibrariesTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,22 @@
package com.google.cloud.tools.libraries;
/*
* Copyright 2017 Google Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.google.cloud.tools.libraries;

import static org.hamcrest.collection.IsArrayContaining.hasItemInArray;

import java.io.File;
import java.io.FileInputStream;
Expand All @@ -7,25 +25,36 @@
import java.io.InputStream;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Iterator;
import java.util.HashMap;
import java.util.Map;

import javax.json.Json;
import javax.json.JsonArray;
import javax.json.JsonObject;
import javax.json.JsonReader;
import javax.json.JsonReaderFactory;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;

import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.xml.sax.SAXException;

import javax.json.Json;
import javax.json.JsonArray;
import javax.json.JsonObject;
import javax.json.JsonReader;
import javax.json.JsonReaderFactory;
import javax.json.JsonValue;

public class LibrariesTest {

private JsonObject[] apis;

@Before
public void parseJson() throws FileNotFoundException {
JsonReaderFactory factory = Json.createReaderFactory(null);
InputStream in =
new FileInputStream("src/main/java/com/google/cloud/tools/libraries/libraries.json");
JsonReader reader = factory.createReader(in);
apis = reader.readArray().toArray(new JsonObject[0]);
}

@Test
public void testWellFormed() throws ParserConfigurationException, SAXException, IOException {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
Expand All @@ -36,27 +65,59 @@ public void testWellFormed() throws ParserConfigurationException, SAXException,
}

@Test
public void testJson() throws FileNotFoundException, URISyntaxException {
JsonReaderFactory factory = Json.createReaderFactory(null);
InputStream in =
new FileInputStream("src/main/java/com/google/cloud/tools/libraries/libraries.json");
JsonReader reader = factory.createReader(in);
Iterator<JsonValue> apis = reader.readArray().iterator();
Assert.assertTrue(apis.hasNext());
for (JsonObject api = (JsonObject) apis.next(); apis.hasNext(); api = (JsonObject) apis.next()) {

new URI(api.getString("documentation"));
if (api.getString("icon") != null) {
new URI(api.getString("icon"));
public void testJson() throws URISyntaxException {
Assert.assertTrue(apis.length > 0);
for (int i = 0; i < apis.length; i++) {
assertApi(apis[i]);
}
}

private static final String[] statuses = {"early access", "alpha", "beta", "GA", "deprecated"};

private static void assertApi(JsonObject api) throws URISyntaxException {
Assert.assertFalse(api.getString("name").isEmpty());
Assert.assertFalse(api.getString("description").isEmpty());
String transports = api.getJsonArray("transports").getString(0);
Assert.assertTrue(transports + " is not a recognized transport",
"http".equals(transports) || "grpc".equals(transports));
new URI(api.getString("documentation"));
if (api.getString("icon") != null) {
new URI(api.getString("icon"));
}
JsonArray clients = api.getJsonArray("clients");
Assert.assertFalse(clients.isEmpty());
for (int i = 0; i < clients.size(); i++) {
JsonObject client = (JsonObject) clients.get(i);
String launchStage = client.getString("launchStage");
Assert.assertThat(statuses, hasItemInArray(launchStage));
new URI(client.getString("apireference"));
new URI(client.getString("site"));
Assert.assertTrue(client.getString("languageLevel").matches("1\\.\\d+\\.\\d+"));
Assert.assertFalse(client.getString("name").isEmpty());
Assert.assertNotNull(client.getJsonObject("mavenCoordinates"));
if (client.getString("source") != null) {
new URI(client.getString("source"));
}
}
}

@Test
public void testDuplicates() throws URISyntaxException {
Map<String, String> apiCoordinates = new HashMap<>();
for (int i = 0; i < apis.length; i++) {
JsonObject api = apis[i];
String name = api.getString("name");
if (apiCoordinates.containsKey(name)) {
Assert.fail(name + " is defined twice");
}
JsonArray clients = api.getJsonArray("clients");
Assert.assertFalse(clients.isEmpty());
for (int i = 0; i < clients.size(); i++) {
JsonObject client = (JsonObject) clients.get(i);
String status = client.getString("status");
Assert.assertTrue(status,
"beta".equals(status) || "alpha".equals(status) || "GA".equals(status));
JsonObject coordinates =
((JsonObject) api.getJsonArray("clients").get(0)).getJsonObject("mavenCoordinates");
String mavenCoordinates =
coordinates.getString("groupId") + ":" + coordinates.getString("artifactId");
if (apiCoordinates.containsValue(mavenCoordinates)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be ok to provide different versions of an API library, shouldn't it? E.g., Java libraries for BigQuery 1.0.0 vs 2.0.0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that would have a different group or artifact ID. I don't think we want both 1.3.0 and 1.4.0.

Assert.fail(mavenCoordinates + " is defined twice");
}
apiCoordinates.put(name, mavenCoordinates);
}
}

Expand Down