Improve error handling and messaging for cloud-logging

[MichaelDoyle]

The general idea is, the first time we hit an auth error with telemetry related APIs, we log some helpful information about how to resolve it, which includes a gcloud command with your project and service account name filled in. After that, we don't log - but we could optionally log out telemetry that "would have" been sent.

Caveats

• Locally, it's a bit weird because you aren't running as a service account
• I was able to "impersonate" a gcp service account locally to try and test it, I have not yet done it in prod.
• Currently - I cannot get metrics to fail locally? even without permission?

Errors look like this:

Unable to send logs to Google Cloud: 7 PERMISSION_DENIED: Permission 'logging.logEntries.create' denied on resource (or it may not exist).

Add the role 'roles/logging.logWriter' to your Service Account in the IAM & Admin page on the Google Cloud console, or use the following command:

gcloud projects add-iam-policy-binding prompt-management-401514 \
    --member=serviceAccount:[email protected] \
    --role=roles/logging.logWriter

Unable to send telemetry to Google Cloud: batchWriteSpans error: 7 PERMISSION_DENIED: The caller does not have permission

Add the role 'roles/cloudtrace.agent' to your Service Account in the IAM & Admin page on the Google Cloud console, or use the following command:

gcloud projects add-iam-policy-binding prompt-management-401514 \
    --member=serviceAccount:[email protected] \
    --role=roles/cloudtrace.agent

TODO:

• Logging
• Metrics
• Tracing