[Feature] gcloud-logging

[VikramTiwari]

Is there any plan to include logging api in this package? If so what is the expected timeline?

Link: https://cloud.google.com/logging/docs/api/gcloud-logging

[jgeewax]

PR's are welcome ! :)

Seriously though -- I think @stephenplusplus may be able to help answer....?

[stephenplusplus]

I think the next two APIs we want to implement are Resource Manager and logging. A rough guess is probably an ETA of early-Mid October for both. PRs definitely welcome to speed up the process :)

[VikramTiwari]

@jgeewax @stephenplusplus There's already some work going on with trace, I am guessing that they will also include logs in the lib once everything is stable on the core api end. Best will be to have that whole lib included in this project.

As for PRs, step 0 - forking done! 👍 :)

[stephenplusplus]

Awesome, keep us posted! Feel free to send a PR whenever you have something ready to discuss or get feedback on; don't worry about waiting until it's perfect. I'll tackle supporting Resource Manager first, but feel free to ping me for any help with logging.

Thanks for helping!

[stephenplusplus]

@VikramTiwari I'm getting started on this. I'll hopefully have a preliminary PR up for review in a couple days.

[VikramTiwari]

@stephenplusplus Aha! Nice. 😄 Sorry but i have been busy hence couldn't contribute anything.

[stephenplusplus]

No problem at all! Just make sure to leave some time to test it out for us :)

[VikramTiwari]

haha Alrighty! Committed for that. :)

[stephenplusplus]

@jgeewax running into "The caller does not have permission" API errors while testing my Logging implementation. Specifically, it's a call to https://cloud.google.com/logging/docs/api/ref/rest/v1beta3/projects.sinks/create -- I'm guessing service accounts aren't sufficient for this API?

[jgeewax]

@munangst : Does projects.sinks/create require 3LO? Or can a service account do that?

[munangst]

The issue is that sinks.create (and also sinks.delete, sinks.update, and logs.delete) requires the caller to be a project owner, and service accounts can't be owners. Our expectation was that typically these operations are infrequent (since they set up long-lived log export destinations) and users would perform them manually as an admin using gcloud or the Developers Console. However, we've gotten some recent feedback that users want to manage log sinks with service accounts (e.g., to allow automation to set up new projects) so we're reconsidering this. We're also planning to support Cloud IAM policies in the future so that you can manage permissions at a level finer than Project Viewer/Editor/Owner.

Can you explain a bit more about the use case? Is it feasible to pass the user's credentials for this operation instead of using a service account?

[stephenplusplus]

I can't come up with a specific use case, but in general our library make the upstream APIs available for any of our users to easily fit into theirs. We do run into these one-off/bootstrap procedures from time to time that are best done with a UI (Pub/Sub creating a topic, Storage creating a bucket). In this case, since our library has full support for Storage, and a possible sink can be a bucket, if a user did have a need for programmatic creation of a sink, we can simplify the connection between a new sink and a Bucket destination:

var gcloud = require('gcloud')({
  projectId: 'my-project-id',
  keyFilename: 'path-to-service-account-keyfile.json'
});

var storage = gcloud.storage();
var logging = gcloud.logging();

logging.createSink('new-sink-name', {
  destination: storage.bucket('logging')
}, function(err, sink) {});

Internally, that would grant the logging account as an owner on the bucket, create the sink, and give the user programmatic access to make further modifications to the sink (getting/setting metadata, deleting).

@jgeewax might be able to speak to specific use cases.

Is it feasible to pass the user's credentials for this operation instead of using a service account?

As far as I know, this would be pretty tricky. In production, a user would be using our library headlessly, so I'm not sure how the user auth process would begin. This could just be due to my still-maturing familiarity with all of the auth options, so if anyone can explain how we can easily get the credentials we need, please correct me.

[jgeewax]

Is it feasible to pass the user's credentials for this operation instead of using a service account?

It really should be...

For example, the resource manager API only works with user account credentials... so this should look like

$ gcloud auth login
$ node myscript.js

And myscript should use the user-creds .... Do we really not do that anywhere in our auth situation?

[stephenplusplus]

Do we really not do that anywhere in our auth situation?

We do via https://github.com/google/google-auth-library-nodejs. We get a token successfully this way, and pass it with the request. The response is still "The caller does not have permission" -- does this mean I'm getting a token that doesn't work? Or maybe my project/account needs to be whitelisted?

[jgeewax]

Or it's pulling credentials from the wrong place?

It should work just fine with your credentials coming from gcloud auth login

[stephenplusplus]

I had the error wrong, I'm getting "The caller does not have permission" with the service account and "Insufficient Permission" with gcloud auth login.

It seems to be picking them up from the right spot, because if I gcloud auth revoke, google-auth-library fails to get a token: "invalid_grant"

[jgeewax]

Can you execute the API request using the demo thing in the docs pages (where you flip the slider to login wiht OAuth2?

[stephenplusplus]

Yes.

[jgeewax]

Yea that's really weird then... @munangst? Any ideas?

[stephenplusplus]

Note that this is similar to #864 (comment)

[stephenplusplus]

@callmehiphop want to give this a shot to see if it's just me? Basically run the system test code from my PR for creating a sink. Make sure you override the auth for the test and provide only a projectId. If you need help getting set up let me know.

[stephenplusplus]

Is it possible falling back to the gcloud sdk token doesn't work here because it doesn't request the scopes needed for this API? (nothing about Logging or the all encompassing auth/cloud-platform scope). (It does request auth/cloud-platform)

This is from running locally on my development machine after gcloud auth login. Maybe running on GAE or GCE will have the correct scopes enabled by default? If that's the case, do we worry about the users in development mode?

[callmehiphop]

I also received a Insufficient Permission error

[stephenplusplus]

@jgeewax another whitelisting issue, maybe. My project ID is nth-circlet-705.

[jgeewax]

Just FYI, pinged the team, waiting to see what they say

[filipjs]

Yes, we need the "cloud-platform" scope to work.

[stephenplusplus]

Sorry for the late update, but I actually later confirmed auth/cloud-platform is included in the scopes requested by gcloud auth login.

[filipjs]

Did you mean "cloud" not "gcloud"?
I checked our backend, and it looks like those requests are not getting to us, meaning they are rejected earlier.

[stephenplusplus]

Yes, sorry the auth/cloud-platform scope is requested by gcloud auth login.

After some playing around, I was able to create a sink successfully. As you predicted, @filipjs, an intermediary request was getting the error. I'll look into that and open an issue for it if necessary, but it's not one that relates to Logging.

Thanks for your help!

[filipjs]

Great!
Also a note about that PR. Code there is creating a "project" sink, so it is probably a good idea to add some filter to it, otherwise it will match all entries from the whole project (see "filter" field name)
https://cloud.google.com/logging/docs/api/ref/rest/v1beta3/projects.logServices.sinks#LogSink
https://cloud.google.com/logging/docs/view/advanced_filters

We also have "log sinks" and "service sinks" but those are going away.

[stephenplusplus]

Good call, and thanks for checking it out! Is there an API update coming soon? If there are any docs available or a changelog you can send over, that'd be helpful.

Also, if you don't mind, I'll give you another ping when the PR is closer to completion, but feel free to stay involved as much as you can as it evolves.

[munangst]

Yes, we're working on some changes that will be released as v2 of the API.
We don't have any docs or changelog available yet because the new API
version hasn't been released yet -- contact us at
[email protected] if you're interested in testing the new
API before it's publicly launched. We'll also ping this thread when the new
API version is available.

The main difference relevant to this issue is that we're simplifying the
way log sinks are specified. We'll be deprecating log-level sinks
(projects.logs.sinks) and service-level sinks (projects.logServices.sinks)
and keeping only project-level sinks (projects.sinks). You can send logs
from only a specific log or a specific service to a given project-level
sink with the "filter" sink property (using the syntax documented on
https://cloud.google.com/logging/docs/view/advanced_filters). For example,
the log-level sink "/projects/my-project/logs/syslog/sinks/my-sink" in
v1beta3 is equivalent to the project-level sink
"/projects/my-project/sinks/my-sink" with a filter expression
[log="syslog"].

[munangst]

(Also, not to fear, we'll transparently migrate any pre-existing log-level or service-level sinks to project-level sinks at some point after the v2 API is released.)

[stephenplusplus]

To anyone interested, support for Logging in gcloud-node is ready for review. It would be great if you could check out our API docs and leave any comments on the PR: #865. Thanks!

[JustinBeckwith]

So this is based on the v1 logging API, right? So we don't need to worry about whitelisting?

[stephenplusplus]

All correct :)