No obvious way to pass the project ID to the datastore options

[patflynn]

Hi there,

I was looking at the datastore example and the DatastoreServiceOptions class and didn't see an obvious way to configure the project ID other than using a system environment variable. Is that intentional or am I missing something?

thx.

[jgeewax]

For reference, we're having this same kind of discussion over on gcloud-python, where I think the way that gcloud-node and gcloud-ruby handle things (using a "client" object) is the cleanest way...

for example, in gcloud-node

"being explicit about the key and project id"

var gcloud = require('gcloud')({
  projectId: 'my-project',
  keyFilename: '/path/to/keyfile.json'
});

var dataset = gcloud.datastore.dataset();
var productKey = dataset.key(['Product', 123]);

dataset.get(productKey, function(err, entity) {
  console.log(err, entity);
});

"figure out the project id from the environment":

var gcloud = require('gcloud')({
  keyFilename: '/path/to/keyfile.json'
});

var dataset = gcloud.datastore.dataset();
var productKey = dataset.key(['Product', 123]);

dataset.get(productKey, function(err, entity) {
  console.log(err, entity);
});

"figure out the project id and the key file from the environment":

var gcloud = require('gcloud')
var dataset = gcloud.datastore.dataset();
var productKey = dataset.key(['Product', 123]);

dataset.get(productKey, function(err, entity) {
  console.log(err, entity);
});

[aozarov]

You can use DatastoreServiceOptions.Builder#dataset(...)
see this test as an example:
https://github.com/GoogleCloudPlatform/gcloud-java/blob/251ac2507c9222b3f1d7279b2edbbbeefe0c8497/src/test/java/com/google/gcloud/datastore/DatastoreServiceOptionsTest.java#L48

[aozarov]

Also, when not provided we are trying to detect it from:
(1) AppEngine environement
(2) GCE using Metadata service
(3) config property file using CLOUDSDK_CONFIG or config directory.

see: https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/src/main/java/com/google/gcloud/ServiceOptions.java#L174

[patflynn]

so dataset is projectId?

On Thu, May 7, 2015 at 6:50 PM, Arie [email protected] wrote:

You can use DatastoreServiceOptions.Builder#dataset(...)
see this test as an example:

https://github.com/GoogleCloudPlatform/gcloud-java/blob/251ac2507c9222b3f1d7279b2edbbbeefe0c8497/src/test/java/com/google/gcloud/datastore/DatastoreServiceOptionsTest.java#L48

—
Reply to this email directly or view it on GitHub
#45 (comment)
.

[jgeewax]

Currently Dataset ID == Project ID, however that will not always be the case.

How does a user specify a Project ID as well as a Dataset ID?

[aozarov]

No, but what we need for a datastore connection is a dataset and not projectid.
Today, as @jgeewax pointed out, there is a 1:1 relation between the two but in the future that will change and one project can be associated with more than 1 dataset.
Regardless, we can always derive the default dataset (if one was not supplied) from the projectid.
I think you are correct that the current state is confusing and I should probably rename the dataset in serviceoptions to projectid.

[jgeewax]

I agree, that's a great idea @aozarov .

Then once we expose multiple datasets (or databases) per project, we should explore the idea of specifying separate dataset IDs.

[aozarov]

Exactly. That was my plan.

[jboynes]

Dataset is a first-class concept in the Datastore API and that term in used in the underlying stub as well. Don't rename to projectId as that will just get to be confusing.

see
https://cloud.google.com/datastore/docs/apis/v1beta2/
https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/master/java/datastore/src/main/java/com/google/api/services/datastore/client/DatastoreOptions.java#L86

[aozarov]

Actually the termp "dataset" is going away (and likely to be replaced with "database").
Regardless of its future name, it is not globally unique but rather associated with a project.
Currently a project can only have one dataset/database.
My understanding is that in Datastore V1 it would be replaced in the APIs/docs with projectid.
When a project will support more than one dataset/database API will change to accept also a dataset/database name but the concept of a default one per project (if non provided) would still exist.
Therefore, for now, in gcloud-java I think renaming dataset with projectId is the right thing to do and in the future, when supported, we can add the database field. Adding @pcostell for any more comments.

[jboynes]

There's are many hypotheticals there which may not come to pass and then
being different will only confuse people. The existing Datastore API is
public and versioned so we should use the same terminology and semantics as
the version we are wrapping.

What is being identified here is a dataset and not a project. That
identifying a project implicitly identifies a dataset is a just a quirk of
the existing system.

As things stand today, dataset is a globally unique identifier. When the
Datastore API changes to use a compound identifier (e.g. project/database),
change the model then.

On Fri, May 8, 2015 at 9:36 AM, Arie [email protected] wrote:

Actually the termp "dataset" is going away (and likely to be replaced with
"database").
Regardless of its future name, it is not globally unique but rather
associated with a project.
Currently a project can only have one dataset/database.
My understanding is that in Datastore V1 it would be replaced in the
APIs/docs with projectid.
When a project will support more than one dataset/database API will change
to accept also a dataset/database name but the concept of a default one per
project (if non provided) would still exist.
Therefore, for now, in gcloud-java I think renaming dataset with projectId
is the right thing to do and in the future, when supported, we can add the
database field. Adding @pcostell https://github.com/pcostell for any
more comments.

—
Reply to this email directly or view it on GitHub
#45 (comment)
.

[patflynn]

How about updating the docs and example to make the relationship between
project id and dataset clear?

On Fri, May 8, 2015 at 1:07 PM, Jeremy Boynes [email protected]
wrote:

There's are many hypotheticals there which may not come to pass and then
being different will only confuse people. The existing Datastore API is
public and versioned so we should use the same terminology and semantics as
the version we are wrapping.

What is being identified here is a dataset and not a project. That
identifying a project implicitly identifies a dataset is a just a quirk of
the existing system.

As things stand today, dataset is a globally unique identifier. When the
Datastore API changes to use a compound identifier (e.g. project/database),
change the model then.

On Fri, May 8, 2015 at 9:36 AM, Arie [email protected] wrote:

Actually the termp "dataset" is going away (and likely to be replaced
with
"database").
Regardless of its future name, it is not globally unique but rather
associated with a project.
Currently a project can only have one dataset/database.
My understanding is that in Datastore V1 it would be replaced in the
APIs/docs with projectid.
When a project will support more than one dataset/database API will
change
to accept also a dataset/database name but the concept of a default one
per
project (if non provided) would still exist.
Therefore, for now, in gcloud-java I think renaming dataset with
projectId
is the right thing to do and in the future, when supported, we can add
the
database field. Adding @pcostell https://github.com/pcostell for any
more comments.

—
Reply to this email directly or view it on GitHub
<
#45 (comment)

.

—
Reply to this email directly or view it on GitHub
#45 (comment)
.

[aozarov]

I agree with @patflynn. I think we should make it clear that a dataset is associated with a project and every project has a default dataset (later on we can rename dataset to database).

@jboynes I think that the current state that used the name "dataset" as a globally unique identifier in their API is a mistake that is going to be changed soon and per the Datastore recommendation we should not follow the same mistake.

[pcostell]

The next version of the Datastore API will expose the currently named "dataset_id" as "project_id". We would recommend exposing it as such now. Eventually we'd like to have the notion of a database, which would exist in a project, but we have not finalized any plans for what that will look like in the API when released.

[aozarov]

per datastore advice we are now using project_id (which is consistent with the storage, and other planeed services).