Use OkHttp for xray sampling requests.

[anuraaga]

Now that OTLP exporters have migrated from the grpc library to okhttp and have a default dependency on it, there is no good reason to use HTTPUrlConnection here as almost all users will have okhttp on the classpath.

The objective of this PR is to use a more modern HTTP library. Coincidentally it will also remove spans generated for sampling requests which we wanted.

Fixes #133 (works around probably a bug in HTTPUrlConnection instrumentation in the javaagent)

[anuraaga]

For the instrumentation members that will probably read this I should mention that @wangzlei will file an issue with the stacktrace he is seeing in HTTPUrlConnection instrumentation, as applied to the sampler (I suspect it's a more broad-reaching corner case of some sort).

[wangzlei]

Environment: Amazon EC2, Amazon Linux 2 amd64, OpenJDK 11. To be noted, the issue does not happen in Mac, though in Mac we see another issue #133

docker-compose.yml

version: "3.6"
services:
  otel:
    image: otel/opentelemetry-collector-contrib:latest
    command: --config /config/collector3.yml
    volumes:
      - /home/ec2-user:/config
    environment:
      AWS_REGION: us-west-2
    ports:
      - '4317:4317'
      - '13133:13133'
      - '2000:2000'
  app:
    image: myapp:latest
    command: java -Dotel.javaagent.debug=true -javaagent:./aws-opentelemetry-agent.jar -jar ./build/libs/spring-boot-0.0.1-SNAPSHOT.jar Application
    environment:
      OTEL_SERVICE_NAME: MyEC2App
      OTEL_TRACES_SAMPLER: xray
      OTEL_TRACES_SAMPLER_ARG: 'endpoint=http://otel:2000'
      OTEL_TRACES_EXPORTER: otlp
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel:4317
    ports:
      - '8080:8080'
    depends_on:
      - otel

Collector config

extensions:
  awsproxy:
    endpoint: 0.0.0.0:2000

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  logging:
    loglevel: debug
  awsxray:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, awsxray]
  extensions: [awsproxy]
  telemetry:
    logs:
      level: "debug"

XRay sampler fetches SamplingRules failed:

[otel.javaagent 2021-11-23 02:02:52:862 +0000] [xray-rules-poller] DEBUG io.opentelemetry.sdk.internal.JavaVersionSpecific - Using the APIs optimized for: Java 9+
[otel.javaagent 2021-11-23 02:02:52:870 +0000] [xray-rules-poller] DEBUG io.opentelemetry.javaagent.bootstrap.ExceptionLogger - Failed to handle exception in instrumentation for sun.net.www.protocol.http.HttpURLConnection on io.opentelemetry.javaagent.bootstrap.AgentClassLoader@1324409e
java.lang.IllegalStateException: Already connected
	at java.base/sun.net.www.protocol.http.HttpURLConnection.setRequestProperty(HttpURLConnection.java:3104)
	at io.opentelemetry.javaagent.instrumentation.httpurlconnection.HeadersInjectAdapter.set(HeadersInjectAdapter.java:17)
	at io.opentelemetry.javaagent.instrumentation.httpurlconnection.HeadersInjectAdapter.set(HeadersInjectAdapter.java:11)
	at io.opentelemetry.javaagent.shaded.io.opentelemetry.extension.aws.AwsXrayPropagator.inject(AwsXrayPropagator.java:157)
	at io.opentelemetry.javaagent.shaded.io.opentelemetry.context.propagation.MultiTextMapPropagator.inject(MultiTextMapPropagator.java:52)
	at io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.ClientInstrumenter.start(ClientInstrumenter.java:27)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1493)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3069)
	at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
	at io.opentelemetry.contrib.awsxray.JdkHttpClient.fetchString(JdkHttpClient.java:83)
	at io.opentelemetry.contrib.awsxray.XraySamplerClient.executeJsonRequest(XraySamplerClient.java:85)
	at io.opentelemetry.contrib.awsxray.XraySamplerClient.getSamplingRules(XraySamplerClient.java:68)
	at io.opentelemetry.contrib.awsxray.AwsXrayRemoteSampler.getAndUpdateSampler(AwsXrayRemoteSampler.java:123)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Thread.java:834)
[otel.javaagent 2021-11-23 02:02:52:873 +0000] [xray-rules-poller] DEBUG io.opentelemetry.contrib.awsxray.JdkHttpClient - JdkHttpClient fetch string failed.
java.net.SocketTimeoutException: Read timed out
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:746)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
	at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
	at io.opentelemetry.contrib.awsxray.JdkHttpClient.fetchString(JdkHttpClient.java:83)
	at io.opentelemetry.contrib.awsxray.XraySamplerClient.executeJsonRequest(XraySamplerClient.java:85)
	at io.opentelemetry.contrib.awsxray.XraySamplerClient.getSamplingRules(XraySamplerClient.java:68)
	at io.opentelemetry.contrib.awsxray.AwsXrayRemoteSampler.getAndUpdateSampler(AwsXrayRemoteSampler.java:123)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Thread.java:834)
[otel.javaagent 2021-11-23 02:02:52:875 +0000] [xray-rules-poller] DEBUG io.opentelemetry.contrib.awsxray.AwsXrayRemoteSampler - Failed to update sampler
java.io.UncheckedIOException: Failed to deserialize response.
	at io.opentelemetry.contrib.awsxray.XraySamplerClient.executeJsonRequest(XraySamplerClient.java:90)
	at io.opentelemetry.contrib.awsxray.XraySamplerClient.getSamplingRules(XraySamplerClient.java:68)
	at io.opentelemetry.contrib.awsxray.AwsXrayRemoteSampler.getAndUpdateSampler(AwsXrayRemoteSampler.java:123)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
 at [Source: (String)""; line: 1, column: 0]
	at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4766)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4668)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3630)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3598)
	at io.opentelemetry.contrib.awsxray.XraySamplerClient.executeJsonRequest(XraySamplerClient.java:88)
	... 8 more

Collector error logs:

otel_1  | 2021-11-23T02:02:50.749Z	debug	[email protected]/server.go:91	Received request on X-Ray receiver TCP proxy server	{"kind": "extension", "name": "awsproxy", "URL": "/GetSamplingRules"}
otel_1  | 2021-11-23T02:02:52.823Z	error	[email protected]/server.go:116	Unable to sign request	{"kind": "extension", "name": "awsproxy", "error": "RequestCanceled: request context canceled\ncaused by: context canceled"}
otel_1  | github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/proxy.NewServer.func1
otel_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/[email protected]/server.go:116
otel_1  | net/http/httputil.(*ReverseProxy).ServeHTTP
otel_1  | 	net/http/httputil/reverseproxy.go:251
otel_1  | net/http.serverHandler.ServeHTTP
otel_1  | 	net/http/server.go:2878
otel_1  | net/http.(*conn).serve
otel_1  | 	net/http/server.go:1929
otel_1  | 2021/11/23 02:02:52 http: proxy error: context canceled