Simple setup, healthcheck failure

[zachdaniel]

I followed the guide here: https://cloud.google.com/trace/docs/zipkin And used the startup script option for GCP. I've now sshd and I'm port forwarding locally for testing. However, when I check the healthcheck I get this:

{"status":"UP","zipkin":{"status":"UNKNOWN"},"diskSpace":{"status":"UP","total":10499674112,"free":8691965952,"threshold":10485760}}

Any ideas?
Thanks :D

[zachdaniel]

I have the same effect if run as a daemon set in my kubernetes cluster.

[zachdaniel]

I'm also using an erlang open tracing client that sends things to a zipkin instance, and (the documentation on google is really just horrendous around all of this) it looks like perhaps I'm missing an important piece here?

[zachdaniel]

I should also add that when I hook up to a normal instance of a zipkin server that I see trace data. Its only using this that I don't see trace data in stackdriver, and I'm not sure the best way to go about debugging it.

[codefromthecrypt]

the old image is replaced by the content here. does advice in #74 help? You should be running the zipkin-gcp image basically

[zachdaniel]

@adriancole trying it out now, I misread that issue originally, but I see now how it could be helpful!

[zachdaniel]

@adriancole I'm a bit confused. I'm not supposed to run both am I?

[codefromthecrypt]

I reworded that issue.. right zipkin-gcp replaces the old image. You don't need to and shouldn't run both.

…

On Tue, Apr 10, 2018 at 2:14 PM, Zach Daniel ***@***.***> wrote: @adriancole <https://github.com/adriancole> I'm a bit confused. I'm not supposed to run both am I? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#75 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD618X51lapdOC7s2bksJXPSmvglAXdks5tnE3PgaJpZM4TNoK2> .

[zachdaniel]

Awesome, thanks :D Now I see that the health check is normal, but I don't see any trace data in stackdriver. Any recommendations on debugging steps?

[zachdaniel]

I've got traces going into a local zipkin server, are there special rules about the tags I need, or about what values certain tags can have?

[codefromthecrypt]

I think I need to merge #73 what do you think?

[zachdaniel]

I can't say with little experience, but merge away and I'll try it out!

[jsw]

@zachdaniel I found /metrics helpful in verifying that it's sending traces to Stackdriver. Look for counter.zipkin_storage.stackdriver.sent. Also make sure your cluster has the trace.append oauth scope via gcloud container clusters describe ... | grep trace.append

[zachdaniel]

@jsw interesting, I don't see that key at all. I do see a bunch of spans as "dropped", so perhaps there is a problem with my spans?

[codefromthecrypt]

I'm releasing 0.3.2 which hopefully will become a docker image within an hour. meanwhile feel free to use https://gitter.im/openzipkin/zipkin for exploratory troubleshooting

…

On Tue, Apr 10, 2018 at 3:08 PM, Zach Daniel ***@***.***> wrote: @jsw <https://github.com/jsw> interesting, I don't see that key at all. I *do* see a bunch of spans as "dropped", so perhaps there is a problem with my spans? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#75 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD614RvbZaixmDl2ucrGQLyVppZSjqMks5tnFp0gaJpZM4TNoK2> .

[jsw]

@adriancole It looks like the counter.zipkin_storage.stackdriver.sent metric is only present in the gcr image? Wondering if it got lost in the zipkin-gcp image.

[codefromthecrypt]

That is not a normal metric. The drop metric covers lost for any reason. In other words if you are not seeing drop metrics the server accepted the spans. That doesnt necessarily mean they processed properly after getting to goog though.

[zachdaniel]

{  
   "mem":611627,
   "mem.free":300139,
   "processors":2,
   "instance.uptime":3305466,
   "uptime":3317785,
   "systemload.average":0.19,
   "heap.committed":540160,
   "heap.init":120832,
   "heap.used":240020,
   "heap":1703936,
   "nonheap.committed":73696,
   "nonheap.init":2496,
   "nonheap.used":71468,
   "nonheap":0,
   "threads.peak":19,
   "threads.daemon":12,
   "threads.totalStarted":29,
   "threads":19,
   "classes":9687,
   "classes.loaded":9687,
   "classes.unloaded":0,
   "gc.ps_scavenge.count":13,
   "gc.ps_scavenge.time":244,
   "gc.ps_marksweep.count":2,
   "gc.ps_marksweep.time":151,
   "gauge.zipkin_collector.message_spans.http":5.0,
   "gauge.zipkin_collector.message_bytes.http":4500.0,
   "counter.zipkin_collector.messages.http":7,
   "counter.zipkin_collector.spans_dropped.http":165,
   "counter.zipkin_collector.bytes.http":152759,
   "counter.zipkin_collector.spans.http":165,
   "counter.status.200.health":1,
   "gauge.response.star-star":21.0,
   "gauge.response.health":59.0,
   "counter.status.404.star-star":1
}

These are my metrics.

[codefromthecrypt]

Yep so if you set debug logging you can see maybe the reason. Look at the README for the args

[codefromthecrypt]

To get all do JAVA_OPTS=-Dlogging.level.zipkin=DEBUG -Dlogging.level.zipkin2=DEBUG

…

On Tue, 10 Apr 2018, 15:40 Adrian Cole, ***@***.***> wrote: Yep so if you set debug logging you can see maybe the reason. Look at the README for the args

[codefromthecrypt]

Reason it isnt logging failures by default is if something in your net has bad impl it would fill up disks. The counters tell yoi something is failing..

[zachdaniel]

@adriancole yeah, that makes sense! Thanks I'll try it out!

[zachdaniel]

🤦‍♂️ The trace API was not enabled.

[zachdaniel]

Now I'm getting PERMISSION_DENIED: The caller does not have permission, but I checked and those oauth scopes are granted on the cluster.

[zachdaniel]

I got everything worked out :D It was an issue w/ the service account's permissions, the oauth scope on the cluster was not enough, I also needed properly configure the service account to have write access to the Trace API

[zachdaniel]

I got everything worked out :D It was an issue w/ the service account's permissions, the oauth scope on the cluster was not enough, I also needed properly configure the service account to have write access to the Trace API