Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Snowflake receiver doesn't seem to be collecting storage data #29750

Closed
luthes opened this issue Dec 12, 2023 · 2 comments
Closed

Snowflake receiver doesn't seem to be collecting storage data #29750

luthes opened this issue Dec 12, 2023 · 2 comments
Assignees
@shalper2
Labels
bug Something isn't working receiver/snowflake Snowflake receiver

Comments

@luthes
Copy link

luthes commented Dec 12, 2023 •
edited
Loading

Component(s)

receiver/snowflake

What happened?

Description

When using the Snowflake receiver, I noticed that we weren't getting the storage metrics I'd expect:
snowflake.storage.stage_bytes.total, snowflake.storage.storage_bytes.total or snowflake.storage.failsafe_bytes.total. It does appear that the query for this is running, as I can see the query in the query history. I don't, however, see this getting sent to my OTLP endpoint (New Relic, in this case), or even logged out using the debug exporter.

All other metrics are received, including optional metrics.

Steps to Reproduce

Docker container with config for Snowflake connector and

Expected Result

I should see logs in the debug exporter that it's finding metrics that it's querying data, or at least

Actual Result

No logs, no errors, no data in otlp endpoint.

Collector version

0.90.1

Environment information

Official Docker Container

OpenTelemetry Collector configuration

receivers:
  snowflake:
    username: ${SNOWFLAKE_USERNAME}
    password: ${SNOWFLAKE_PASSWORD}
    account: my-snowflake-account
    warehouse: my-warehouse
    collection_interval: 1m
    metrics:
      [...]
      # All other metrics are explicitly disabled, including default metrics.
      snowflake.storage.stage_bytes.total:
        enabled: true
      snowflake.storage.storage_bytes.total:
        enabled: true
      snowflake.storage.failsafe_bytes.total:
        enabled: true
     [...]
processors:
  batch:
exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200
  otlp:
    # New Relic Config
    [...]

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    metrics:
      receivers:
        - snowflake
      processors: [batch]
      exporters: [otlp, debug]

Log output

snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    info    [email protected]/telemetry.go:86 Setting up own telemetry...           
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    info    [email protected]/telemetry.go:203        Serving Prometheus metrics   {"address": ":8888", "level": "Basic"}                                                                                                 
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   [email protected]/extension.go:162      Beta component. May change in the future.     {"kind": "extension", "name": "zpages"}                                                                               
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   [email protected]/extension.go:162      Beta component. May change in the future.     {"kind": "extension", "name": "health_check"}                                                                         
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   [email protected]/exporter.go:273        Stable component.       {"kind": "exporter", "data_type": "metrics", "name": "otlp"}                                                                                
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    info    [email protected]/exporter.go:275        Development component. May change in the future.      {"kind": "exporter", "data_type": "metrics", "name": "debug"}                                                 
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   [email protected]/processor.go:287      Stable component.       {"kind ": "processor", "name": "batch", "pipeline": "metrics"}                                                                               
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   [email protected]/receiver.go:294        Alpha component. May change in the future.    {"kind": "receiver", "name": "snowflake", "data_type": "metrics"}                              snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   [email protected]/receiver.go:294        Alpha component. May change in the future.    {"kind": "receiver", "name": "snowflake", "data_type": "metrics"}                                snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    [email protected]/service.go:148  Starting otelcol-contrib...     {"Version": "0.90.1", "NumCPU": 20}                                                                                                         snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:34     Starting extensions...
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:37     Extension is starting...        {"kind": "extension", "name": "health_check"}                                                                                               
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    [email protected]/healthcheckextension.go:35 Starting health_check extension       {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/status/check","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    warn    [email protected]/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks      {"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:45     Extension started.      {"kind": "extension", "name": "health_check"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:37     Extension is starting...        {"kind": "extension", "name": "zpages"} 
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    [email protected]/zpagesextension.go:53   Registered zPages span processor on tracer provider   {"kind": "extension", "name": "zpages"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    [email protected]/zpagesextension.go:63   Registered Host's zPages      {"kind": "extension", "name": "zpages"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    [email protected]/zpagesextension.go:75   Starting zPages extension     {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":":55679"}}}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:45     Extension started.      {"kind": "extension", "name": "zpages"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel created     {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] original dial target is: "otlp.nr-data.net:4318"  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] parsed dial target is: {URL:{Scheme:otlp.nr-data.net Opaque:4318 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] fallback to scheme "passthrough"  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] parsed dial target is: {URL:{Scheme:passthrough Opaque: User: Host: Path:/otlp.nr-data.net:4318 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}     {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel authority set to "otlp.nr-data.net:4318"  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Resolver state updated: {
snowflake-otel-collector  |   "Addresses": [
snowflake-otel-collector  |     {
snowflake-otel-collector  |       "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector  |       "ServerName": "",
snowflake-otel-collector  |       "Attributes": null,
snowflake-otel-collector  |       "BalancerAttributes": null,
snowflake-otel-collector  |       "Metadata": null
snowflake-otel-collector  |     }
snowflake-otel-collector  |   ],
snowflake-otel-collector  |   "Endpoints": [
snowflake-otel-collector  |     {
snowflake-otel-collector  |       "Addresses": [
snowflake-otel-collector  |         {
snowflake-otel-collector  |           "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector  |           "ServerName": "",
snowflake-otel-collector  |           "Attributes": null,
snowflake-otel-collector  |           "BalancerAttributes": null,
snowflake-otel-collector  |           "Metadata": null
snowflake-otel-collector  |         }
snowflake-otel-collector  |       ],
snowflake-otel-collector  |       "Attributes": null
snowflake-otel-collector  |     }
snowflake-otel-collector  |   ],
snowflake-otel-collector  |   "ServiceConfig": null,
snowflake-otel-collector  |   "Attributes": null
snowflake-otel-collector  | } (resolver returned new addresses) {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel switches to new LB policy "pick_first"    {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [pick-first-lb 0xc002d81a10] Received new config {
snowflake-otel-collector  |   "shuffleAddressList": false
snowflake-otel-collector  | }, resolver state {
snowflake-otel-collector  |   "Addresses": [
snowflake-otel-collector  |     {
snowflake-otel-collector  |       "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector  |       "ServerName": "",
snowflake-otel-collector  |       "Attributes": null,
snowflake-otel-collector  |       "BalancerAttributes": null,
snowflake-otel-collector  |       "Metadata": null
snowflake-otel-collector  |     }
snowflake-otel-collector  |   ],
snowflake-otel-collector  |   "Endpoints": [
snowflake-otel-collector  |     {
snowflake-otel-collector  |       "Addresses": [
snowflake-otel-collector  |         {
snowflake-otel-collector  |           "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector  |           "ServerName": "",
snowflake-otel-collector  |           "Attributes": null,
snowflake-otel-collector  |           "BalancerAttributes": null,
snowflake-otel-collector  |           "Metadata": null
snowflake-otel-collector  |         }
snowflake-otel-collector  |       ],
snowflake-otel-collector  |       "Attributes": null
snowflake-otel-collector  |     }
snowflake-otel-collector  |   ],
snowflake-otel-collector  |   "ServiceConfig": null,
snowflake-otel-collector  |   "Attributes": null
snowflake-otel-collector  | }   {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel created  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel Connectivity change to CONNECTING {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING        {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel picks a new address "otlp.nr-data.net:4318" to connect   {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [pick-first-lb 0xc002d81a10] Received SubConn state update: 0xc002d81bc0, {ConnectivityState:CONNECTING ConnectionError:<nil>} {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    healthcheck/handler.go:132      Health Check state change       {"kind": "extension", "name": "health_check", "status": "ready"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    [email protected]/service.go:174  Everything is ready. Begin running and processing data.
snowflake-otel-collector  | 2023-12-12T00:08:07.507Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY     {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.508Z    info    zapgrpc/zapgrpc.go:178  [core] [pick-first-lb 0xc002d81a10] Received SubConn state update: 0xc002d81bc0, {ConnectivityState:READY ConnectionError:<nil>}      {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.508Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel Connectivity change to READY      {"grpc_log": true}

Additional context

I'm not sure if it matters, but the type in the client.go is of type int64, but it looks like the type returned from Snowflake might be of type float64 like some of the other types.

For example, this query (from the client.go):

storageMetricsQuery          = "select STORAGE_BYTES, STAGE_BYTES, FAILSAFE_BYTES from STORAGE_USAGE ORDER BY USAGE_DATE DESC LIMIT 1;"

returns the following:

STORAGE_BYTES	                STAGE_BYTES	        FAILSAFE_BYTES
123412341234123.000000	        123412341234.000000	12341234123412.000000

But frankly, I'm not sure if there is some type conversion happening somewhere that would impact this, or if it matters at all.

Just to be clear, these are the only three metrics that don't work. If I explicity enable all other metrics, or allow defaults with empty metric:, I see all of the other data points reflected in the OTLP endpoint.
@luthes luthes added bug Something isn't working needs triage New item requiring triage labels Dec 12, 2023
@github-actions github-actions bot added the receiver/snowflake Snowflake receiver label Dec 12, 2023
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot mentioned this issue Dec 12, 2023
Closed
95 tasks
@shalper2
Copy link
Contributor

@luthes hey! Thanks for opening the issue, I was able to reproduce using your config. Your intuition was right, there is a typing issue with the rows.Scan() (from database/sql) used in client.go.

@shalper2 shalper2 mentioned this issue Dec 12, 2023
Merged
@crobert-1 crobert-1 removed the needs triage New item requiring triage label Dec 12, 2023
@github-actions github-actions bot mentioned this issue Dec 19, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shalper2 shalper2

Labels
bug Something isn't working receiver/snowflake Snowflake receiver
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@luthes @crobert-1 @shalper2