prometheusexporter error: failed to convert metric latency: "service.name" is not a valid label name for metric

[samariei]

Describe the bug
Hi,

I assume something is broken in v0.22 between prometheusexporter and spanmetricsprocessor. Also, I saw that prometheusexporter was reimplemented recently (#2393) while spanmetricsprocessor wasn’t. This made me think it might be a problem with prometheusexporter. Even more, in v0.21 both are working fine.

Steps to reproduce

1. Have a manually instrumented micro-service that export spans using Otlp Exporter to a custom build otel-collector v0.22.
2. Pass a trace through the instrumented micro-service to export the span to Collector.
3. At this step the span is displayed in collector logging exporter.
4. Go to Collector's Prometheus exporter /metrics endpoint (in my case http://localhost:8889/metrics)
5. No metrics are displayed.

What did you expect to see?
Prometheus span metrics

What did you see instead?
Blank page in browser and ERRORs in collector's logs

2021-03-13T19:37:50.489+0200    INFO    [email protected]/processor.go:174   Consuming trace data    {"component_kind": "processor", "component_type": "spanmetrics", "component_name": "spanmetrics"}
2021-03-13T19:37:50.489+0200    DEBUG   prometheusexporter/accumulator.go:81    accumulating metric: calls      {"component_kind": "exporter", "component_type": "prometheus", "component_name": "prometheus"}
2021-03-13T19:37:50.489+0200    DEBUG   prometheusexporter/accumulator.go:81    accumulating metric: latency    {"component_kind": "exporter", "component_type": "prometheus", "component_name": "prometheus"}
2021-03-13T19:37:52.105+0200    INFO    loggingexporter/logging_exporter.go:327 TracesExporter  {"#spans": 1}
2021-03-13T19:37:52.105+0200    DEBUG   loggingexporter/logging_exporter.go:366 ResourceSpans #0
Resource labels:
     -> service.name: STRING(mock)
InstrumentationLibrarySpans #0
InstrumentationLibrary my.test.mock 
Span #0
    Trace ID       : 2de6b97890aa1b06b491a747b9a8c569
    Parent ID      : b4a4c24a93dc05b4
    ID             : ce5f0f09b93c6365
    Name           : POST /test
    Kind           : SPAN_KIND_SERVER
    Start time     : 2021-03-13 17:37:50.441111796 +0000 UTC
    End time       : 2021-03-13 17:37:50.442 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message : 
Attributes:
     -> http.method: STRING(POST)

2021-03-13T19:38:24.249+0200    DEBUG   prometheusexporter/collector.go:252     collect called  {"component_kind": "exporter", "component_type": "prometheus", "component_name": "prometheus"}
2021-03-13T19:38:24.249+0200    DEBUG   prometheusexporter/accumulator.go:327   Accumulator collect called      {"component_kind": "exporter", "component_type": "prometheus", "component_name": "prometheus"}
2021-03-13T19:38:24.249+0200    ERROR   prometheusexporter/collector.go:259     failed to convert metric calls: "status.code" is not a valid label name for metric "promtest_calls"     {"component_kind": "exporter", "component_type": "prometheus", "component_name": "prometheus"}
go.opentelemetry.io/collector/exporter/prometheusexporter.(*collector).Collect
        /Users/myuser/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/prometheusexporter/collector.go:259
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        /Users/myuser/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:448
2021-03-13T19:38:24.249+0200    ERROR   prometheusexporter/collector.go:259     failed to convert metric latency: "service.name" is not a valid label name for metric "promtest_latency"        {"component_kind": "exporter", "component_type": "prometheus", "component_name": "prometheus"}
go.opentelemetry.io/collector/exporter/prometheusexporter.(*collector).Collect
        /Users/myuser/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/prometheusexporter/collector.go:259
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        /Users/myuser/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:448

What version did you use?
Version: 0.22

What config did you use?
Config:

  otlp:
    protocols:
      grpc:

exporters:
  logging:
    loglevel: debug
  prometheus:
    endpoint: "localhost:8889"
    namespace: promtest

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  spanmetrics:
    metrics_exporter: prometheus
    dimensions:
    - name: location.city
      default: "none"
    - name: location.country
      default: "none"
extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [spanmetrics, batch]
      exporters: [logging]
    metrics:
      receivers: [otlp]
      exporters: [logging, prometheus]

Environment
OS: MacOS 10.13.6
Compiler: go1.15.7 darwin/amd64

Additional context
I've managed to do a "fix" by sanitising the key names here

labels.ForEach(func(k string, v string) {
		keys = append(keys, sanitise(k))
		values = append(values, v)
	})

What I'm not sure of is if they should have been already sanitised at this point by spanmetricsprocessor