gem doesn't honor DISABLE_DATADOG_AGENT env var

[breathe]

We get extraneous output when running our ruby code in an environment with DISABLE_DATADOG_AGENT=true.

The agent processes won't be started (due to the presence of the env var) - but the gem still tries to log to the non-running agents and produces extraneous error log output ...

E, [2020-07-24T13:11:34.086726 #3] ERROR -- ddtrace: [ddtrace] (/app/vendor/bundle/ruby/2.4.0/gems/ddtrace-0.38.0/lib/ddtrace/transport/http/client.rb:35:in `rescue in send_request') Internal error during HTTP transport request. Cause: Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126) Location: /app/vendor/ruby-2.4.6/lib/ruby/2.4.0/net/http.rb:906:in `rescue in block in connect'
W, [2020-07-24T13:11:34.088445 #3]  WARN -- ddtrace: [ddtrace] DATADOG TRACER CONFIGURATION - {"date":"2020-07-24T13:11:34+00:00","os_name":"x86_64-pc-linux-gnu","version":"0.38.0","lang":"ruby","lang_version":"2.4.6","env":"production","enabled":true,"service":"itsacheckmate-production","agent_url":"http://127.0.0.1:8126?timeout=1","agent_error":"Datadog::Transport::InternalErrorResponse ok?: unsupported?:, not_found?:, client_error?:, server_error?:, internal_error?:true, payload:, error_type:Errno::ECONNREFUSED error:Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for \"127.0.0.1\" port 8126)","debug":false,"analytics_enabled":true,"tags":"env:production","runtime_metrics_enabled":false,"integrations_loaded":"[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]","vm":"ruby-2.4.6","partial_flushing_enabled":false,"priority_sampling_enabled":false,"health_metrics_enabled":false,"integration_rails_analytics_enabled":"","integration_rails_analytics_sample_rate":"1.0","integration_rails_service_name":"itsacheckmate-production","integration_rails_cache_service":"itsacheckmate-production-cache","integration_rails_controller_service":"itsacheckmate-production","integration_rails_database_service":"itsacheckmate-production-postgres","integration_rails_distributed_tracing":"true","integration_rails_exception_controller":"","integration_rails_middleware":"true","integration_rails_middleware_names":"false","integration_rails_template_base_path":"views/","integration_delayed_job_analytics_enabled":"false","integration_delayed_job_analytics_sample_rate":"1.0","integration_delayed_job_service_name":"delayed_job","integration_rest_client_analytics_enabled":"false","integration_rest_client_analytics_sample_rate":"1.0","integration_rest_client_service_name":"rest_client","integration_rest_client_distributed_tracing":"true","integration_rack_analytics_enabled":"","integration_rack_analytics_sample_rate":"1.0","integration_rack_service_name":"itsacheckmate-production","integration_rack_application":"#\u003cItsACheckmate::Application:0x000055d2e8f16010\u003e","integration_rack_distributed_tracing":"true","integration_rack_headers":"{:response=\u003e[\"Content-Type\", \"X-Request-ID\"]}","integration_rack_middleware_names":"false","integration_rack_quantize":"{}","integration_rack_request_queuing":"false","integration_rack_web_service_name":"web-server","integration_active_support_analytics_enabled":"false","integration_active_support_analytics_sample_rate":"1.0","integration_active_support_service_name":"","integration_active_support_cache_service":"itsacheckmate-production-cache","integration_action_pack_analytics_enabled":"","integration_action_pack_analytics_sample_rate":"1.0","integration_action_pack_service_name":"itsacheckmate-production","integration_action_pack_controller_service":"itsacheckmate-production","integration_action_pack_exception_controller":"","integration_action_view_analytics_enabled":"false","integration_action_view_analytics_sample_rate":"1.0","integration_action_view_service_name":"itsacheckmate-production","integration_action_view_template_base_path":"views/","integration_active_record_analytics_enabled":"false","integration_active_record_analytics_sample_rate":"1.0","integration_active_record_service_name":"itsacheckmate-production-postgres","integration_active_record_orm_service_name":""}
W, [2020-07-24T13:11:34.088533 #3]  WARN -- ddtrace: [ddtrace] DATADOG TRACER DIAGNOSTIC - Agent Error: Datadog::Transport::InternalErrorResponse ok?: unsupported?:, not_found?:, client_error?:, server_error?:, internal_error?:true, payload:, error_type:Errno::ECONNREFUSED error:Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126)

[ericmustin]

@breathe thanks for reaching out. So, I wasn't familiar with that env var off the top of my head but, looking into it, i believe that DISABLE_DATADOG_AGENT is a datadog-agent specific environment variable for Heroku.

|
| `DISABLE_DATADOG_AGENT`    \| *Optional.* When set, the Datadog Agent does not run.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
…

from: https://docs.datadoghq.com/agent/basic_agent_usage/heroku/#configuration

The datadog-agent is the daemon that this ddtrace gem ruby tracing clients emits traces to. The error you're seeing would be expected. The tracing client is flushing traces to a port that it expects the datadog-agent to be listening on, but since it is disabled, it is not.

If you want to disable the ddtrace tracing client in your application, currently the only available way to do so is by configuring the configuration block in application code, so something like (in a rails app)

# config/initializers/datadog-tracer.rb

Datadog.configure do |c|
  c.tracer.enabled = false
end

from: https://github.com/DataDog/dd-trace-rb/blob/master/docs/GettingStarted.md#tracer-settings

We recently merged in support for an env var to disable the ruby tracing client (DD_TRACE_ENABLED) but it hasn't been released yet, so in your application code is the only way at the moment. I can update this issue when a version that supports DD_TRACE_ENABLED env var is released. In the interim, I suppose you could also do something like

# config/initializers/datadog-tracer.rb

Datadog.configure do |c|
  if ENV['DISABLE_DATADOG_AGENT'] == 'true'
    c.tracer.enabled = false
  else
    c.trace.enabled = true
  end
end

Does that help? Also, always feel free to reach out to support if there's more detailed environment specific information or logs that may contain sensitive information which need to be shared.

[breathe]

awesome thank you @ericmustin !

I'm using code like this to ensure gem honors both these env vars and seems to work in our environments

Datadog.configure do |c|
  if ENV["DISABLE_DATADOG_AGENT"] == "true" || ENV["DD_TRACE_ENABLED"] == "false"
    c.tracer.enabled = false
  else
    c.tracer.enabled = true
  end

  c.use :rails, service_name: "my_service"
  c.use :delayed_job
  c.use :rest_client
end

Please feel free to close this ticket now or close later when the support for DD_TRACE_ENABLED is released as you see fit!

Appreciate the help!

[marcotc]

We'll ship DD_TRACE_ENABLED (merged in #1115) in the next release. I'm tagging this issue so we can update it and inform users when the release goes out.

[marcotc]

We've just released 0.39.0, which adds support for the DD_TRACE_ENABLED environment variable.
I'm closing this issue, but let us know if you have any further questions about it.