Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create API for GCE Metadata #917

Closed
JustinBeckwith opened this issue Oct 27, 2015 · 26 comments
Closed

Create API for GCE Metadata #917

JustinBeckwith opened this issue Oct 27, 2015 · 26 comments
Assignees
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release.

Comments

@JustinBeckwith
Copy link
Contributor

There's a great metadata service for getting access to information on the VM. It would be great to have a structured API for this:
https://cloud.google.com/compute/docs/metadata?hl=en

@JustinBeckwith
Copy link
Contributor Author

@dhermes
Copy link
Contributor

dhermes commented Oct 27, 2015

We already use the metadata service to auto-detect the current project on GCE. As a first start we could move that functionality into a sub-package.

@tseaver @jgeewax Any preferences on what the subpackage should be named?

@stephenplusplus
Copy link
Contributor

So far, our APIs assume you're talking to the googleapis.com API, regardless of how you auth. We have recently landed support for Resource Manager and are working on Logging, when things get more complex. Specifically, a portion of our API-- creating a Project (RM) or a Sink (Logging)-- requires user auth, as opposed to from a service account. This can quickly confuse our users, since it's up to our docs to explain how, when, & where to use auth and run our code. So this method is another example where we have a method that may or may not work when actually called.

Are there other actions that can be performed only when a user is running on a GCE instance? Possibly enough that could warrant creating a new module that has a dream-suite of utilities for someone running inside GCE?

@JustinBeckwith
Copy link
Contributor Author

Not that I know of, but it's entirely possible. This will be a common thread though for the compute environment based APIs. For example, there are App Engine APIs we need to build veneers for that dig out App Engine specific information that only work when running inside of App Engine:

https://cloud.google.com/appengine/docs/python/refdocs/google.appengine.api.modules.modules

I think those use standard auth mechanisms though based on this:
https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1beta4/apps.modules/get

@stephenplusplus
Copy link
Contributor

There's a library that looks like it was built to be a GAE suite of tools:

This library will only work inside App Engine Managed VMs, or with the Managed VM SDK.

It doesn't look like it's been updated recently, so I'm not sure if it has any plans for expansion in sight. However, that seems like the right approach, as it's obvious to the user that their GAE app will benefit from this GAE-only toolkit. That could set the precedent as well for GCE, even if it starts as just a module with one function, "getMetadata". Heck, I'll write it!

// @jgeewax any thoughts?

@JustinBeckwith
Copy link
Contributor Author

Yeah - we really need to take appengine-nodejs down (we don't want people using that). It's talking to the App Engine v1 APIs with node, which was never really intended.

The App Engine APIs I'm talking about are available via a One Platform API:
https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1beta4/apps.modules/get

@stephenplusplus
Copy link
Contributor

The API seems pretty straightforward, so I wouldn't mind writing a library for it. Just to review, my proposal here is:

  1. Write a separately-installable GCE-env-only Node module.

    var env = require('gcloud-compute-engine');
    env.getMetadata(function(err, metadata) {});
  2. Write a separately-installable GAE-env-only Node module.

    var env = require('gcloud-appengine');
    env.getApps(function(err, apps) {});
    
    var app = env.app('my-site');
    app.getModules(function(err, modules) {});
    
    var appVersion1 = app.module('1');
    appVersion1.delete(function(err) {});

The whole point being, where confusion is possible, simply narrowing the scope of what a user can expect from a given library we offer.

@JustinBeckwith
Copy link
Contributor Author

Interesting - why separate these specifically into their own NPM modules? Splitting this up seems kind of confusing, unless we choose to split up the whole module.

Instead, I could see adding something like:

let gcloud = require('gcloud');
gcloud.appEngine.isAvailable(function(err, isAvailable) {
  if (!err && isAvailable) {
    gcloud.appEngine.getModules(function(err, modules) {
      console.log(modules);
    });
  } else {
    console.log('not running in app engine!');
});

That way I can write code to specifically handle the case where I'm not in a particular environment. Same for GCE:

let gcloud = require('gcloud');
gcloud.computeEngine.isAvailable(function(err, isAvailable) {
  if (!err && isAvailable) {
    gcloud.computeEngine.getMetadata(path, function(err, res) {
      console.log(res);
    });
  } else {
    console.log('not running in compute engine!');
  }
});

If isAvailable doesn't sound right, getExecutionEnvironment is another option:

let gcloud = require('gcloud');
gcloud.getExecutionEnvironment(function(err, env) {
  if (!err && env === gcloud.environments.appEngine) {    
      console.log("I'm in app engine!");
  } else {
      console.log('not running in app engine!');
  }
});

@JustinBeckwith
Copy link
Contributor Author

@jmdobry and @jonparrott interested in your thoughts here too.

@theacodes
Copy link

  • I don't want to create another appengine specific API, ever. They can just use environment variables for stuff like getting the current module.
  • getExecutionEnvironment is the 100% wrong way to do this, I think. You're either on GCP or not. Being on GAE shouldn't matter vs being on GKE or GCE. You're either in GCP or not.
  • The metadata API is incredibly useful, and having an API to do things like watch for changes would be extremely useful for users. I don't understand the desire to put it in a separate npm module as it's applicable to all of our hosting platforms. I suppose node.js doesn't have anything like Python's namespace packages.

@stephenplusplus
Copy link
Contributor

why separate these specifically into their own NPM modules?

One reason is simply to shrink the overhead of this repo. The bigger this library grows, it's simply harder to maintain and put out a new release. If we stabilize at 1.0, then catch a breaking change in one of (smallest possible number here) APIs we support, we have to force a major increment upgrade (+1.0) on all of our users to have them benefit from smaller releases with features and fixes.

That aside, I prefer this split to avoid confusion. I think a user should be able to install a library and have everything work. Then, if they need something more specific to their application, they can install another library. I mentioned earlier the Logging and Resource Manager API muddying things up a bit earlier, but in those cases, we at least give the user the majority of the API regardless of what environment they're in.

With these extra APIs for App and Compute Engine, I think the separation will make it more clear for a user to understand what to use given their environment. Also, it will help users who may only need the GAE/GCE functionality by offering that library as separately installable.

@stephenplusplus
Copy link
Contributor

I don't want to create another appengine specific API, ever.

Why is that?

You're either in GCP or not.

Basically, the user shouldn't need this behavior because they're the ones deploying their app, so they should know where it's at?

@theacodes
Copy link

Why is that?

Because from my point of view, App Engine is not a special snowflake in our hosting options.

Basically, the user shouldn't need this behavior because they're the ones deploying their app, so they should know where it's at?

I'm just saying that at most, we should have

var metadata = require('gcloud').metadata();

metadata.isAvailable(function(err) { ... });

Ideally, there should be little need to distinguish between GAE, GCE, and GKE. Even if that is desired, the metadata API is the wrong place to do it.

@stephenplusplus
Copy link
Contributor

Ideally, there should be little need to distinguish between GAE, GCE, and GKE

That makes sense to me. The idea of being able to stop/delete a module programmatically sounded cool, though.

But, if all we want is a "try to get metadata" method, this is fine with me?:

var gcloud = require('gcloud');

gcloud.getMetadata(function(err, metadata) {
  if (/*not in a place we can get metadata*/) {
    err.message = 'Not in a place we can get metadata.';
  }
});

@theacodes
Copy link

theacodes commented Nov 23, 2015

FYI, I believe the ideal API surface for the metadata server should look something like this:

var metadata = require('gcloud').metadata();

// get instance tags.
metadata.instance.tags(function(err, tags) {
  console.log(tags);
  // ['one', 'two', 'blue', 'shoe']
});

// Get project-level custom metadata.
metadata.project.custom(function(err, kv) {
  console.log(kv);
  // {color: "red", active: "yes"}
});

// watch for changes to the instance tags.
metadata.instance.tags.watch(function(err, tags) {
   //...
});

@theacodes
Copy link

That makes sense to me. The idea of being able to stop/delete a module programmatically sounded cool, though.

That should be done in the with the App Engine Admin API. :)

But, if all we want is a "try to get metadata" method, this is fine with me?:

Yep, or just have all of the methods of metadata do one of the following:

  • Give an error with a status code of 503.
  • Give a special error that indicates unavailability.
  • Give the generic error for socket timeout or connection failed.

FYI, IMO the timeout for hitting the metadata server should be set relatively low.

@stephenplusplus
Copy link
Contributor

I think your examples show more than I thought was involved with my "just try to get the metadata" understanding. I still would do this separate if it was my own project, but I can see the importance of the docs all being in one place.

// watch for changes to the instance tags.

I definitely wouldn't have thought of anything like that myself. Any other things we should support?

@stephenplusplus stephenplusplus added enhancement api: compute Issues related to the Compute Engine API. labels Nov 23, 2015
@theacodes
Copy link

That's a good start. Writing metadata is also a useful thing to have. See https://cloud.google.com/compute/docs/metadata

Let me know if there's anything I can do to help.

As a heads up, please use metadata.google.internal as the hostname. Using just metadata will break on kubernetes. ;) Feel free to cc me on any PRs for this.

@stephenplusplus
Copy link
Contributor

Definitely, thanks as always!

@JustinBeckwith
Copy link
Contributor Author

Hey on the App Engine API bit - we absolutely need to have an App Engine Management API available. There is an entire REST API surface of commands customers will use:
https://cloud.google.com/appengine/docs/admin-api/reference/rest/

When I say App Engine API - I don't mean things like mail and datastore, I'm talking about APIs that:

  • get all modules in an app
  • deploy a new module
  • start/stop versions
  • etc...

That merits an SDK in the same way VM creation and management on our platform needs a client library.

@thesandlord
Copy link

Is this being worked on? If not, I'm happy to submit a PR. I think the API surface @jonparrott described is a good place to start.

@stephenplusplus
Copy link
Contributor

Thanks for offering! Just for the sake of mentioning, I whipped up this quick API that has a basic, similar surface recently: https://www.npmjs.com/package/gcp-metadata. It's lacking features that @jonparrott originally proposed, such as a watcher. And it's missing all of the stuff @JustinBeckwith suggested.

Since this issue was originally discussed, we've modularized our repo and deprecated google-cloud, the package that bundled all of them. This discussion is now effectively:

  • Should we create and maintain a new module in this repository, i.e. @google-cloud/metadata?
  • What features should it have?

And if the answer to #­1 is 👍 , then a PR to get the ball rolling would be awesome 💃

@lukesneeringer for all of the answers.

@lukesneeringer
Copy link
Contributor

Another thing that has changed since the issue was discussed is that we can auto-generate GAPICs now. This sounds like a thing that should be auto-gen only so we do not have as high of a maintenance burden on it.

@stephenplusplus
Copy link
Contributor

This issue was moved to googleapis/nodejs-compute#12

@stephenplusplus
Copy link
Contributor

Re-opening, as this wasn't really an issue about the Compute API. Is there anything to do here? @alexander-fenster ?

@stephenplusplus stephenplusplus removed api: compute Issues related to the Compute Engine API. help wanted labels Feb 5, 2018
@JustinBeckwith
Copy link
Contributor Author

👋 I think gcp-metadata is covering this quite nicely. @stephenplusplus we can probably close this out :)

sofisl pushed a commit that referenced this issue Nov 11, 2022
…tion context (#917)

* feat: added option to configure the number of sentences in the suggestion context

PiperOrigin-RevId: 420101041

Source-Link: googleapis/googleapis@549b567

Source-Link: googleapis/googleapis-gen@7d095a8
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiN2QwOTVhOGM2YTYyMjNkZDRlZTc3MmMyODUxMTI0ODNjMmFmYjFjMCJ9

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this issue Nov 11, 2022
🤖 I have created a release *beep* *boop*
---


## [4.7.0](googleapis/nodejs-dialogflow@v4.6.0...v4.7.0) (2022-03-25)


### Features

* added export documentation method ([35a77f4](googleapis/nodejs-dialogflow@35a77f4))
* added option to configure the number of sentences in the suggestion context ([#917](googleapis/nodejs-dialogflow#917)) ([17aa7d6](googleapis/nodejs-dialogflow@17aa7d6))
* removed OPTIONAL for speech model variant docs: added more docs for speech model variant and improved docs format for participant ([#906](googleapis/nodejs-dialogflow#906)) ([7df7d27](googleapis/nodejs-dialogflow@7df7d27))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
sofisl pushed a commit that referenced this issue Jan 17, 2023
... chore: update gapic-generator-ruby to the latest commit chore: release gapic-generator-typescript 1.5.0

Committer: @miraleung
PiperOrigin-RevId: 380641501

Source-Link: googleapis/googleapis@076f7e9

Source-Link: googleapis/googleapis-gen@27e4c88
sofisl pushed a commit that referenced this issue Jan 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release.
Projects
None yet
Development

No branches or pull requests

7 participants