Error level dpanic invalid key "v2beta2","stacktrace

[frpataciertocom]

Hi,

I have deployed OTEL without issues, then I removed the installation to perform some tests, and after when I redeployed OTEL now I get the error below.

1. I have installed like this: https://aws-otel.github.io/docs/getting-started/adot-eks-add-on/installation
2. Then applied this to try to get traces in X-Ray: https://aws-otel.github.io/docs/getting-started/adot-eks-add-on/config-xray

I got nothing in X-ray and when I checked the logs and I found this "dpanic" invalid key":"v2beta2.

Any ideas?, thank you.

"level":"info","ts":1663256456.4467094,"logger":"instrumentation-upgrade","msg":"no instances to upgrade"}
{"level":"info","ts":1663256456.4467492,"logger":"collector-upgrade","msg":"no instances to upgrade"}
{"level":"dpanic","ts":1663256456.548566,"logger":"config","msg":"non-string key argument passed to logging, ignoring all later arguments","invalid key":"v2beta2","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/config.(*Config).StartAutoDetect\n\t/workspace/internal/config/main.go:93\nmain.addDependencies.func1\n\t/workspace/main.go:244\nsigs.k8s.io/controller-runtime/pkg/manager.RunnableFunc.Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/manager.go:307\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/runnable_group.go:219"}
{"level":"info","ts":1663256456.5485582,"logger":"config","msg":"In Autodetect, Set HPA version to ["}
{"level":"info","ts":1663256456.8421454,"msg":"Starting workers","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","worker count":1}
{"level":"dpanic","ts":1663256461.561525,"logger":"config","msg":"non-string key argument passed to logging, ignoring all later arguments","invalid key":"v2beta2","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/config.(*Config).periodicAutoDetect\n\t/workspace/internal/config/main.go:103"}
{"level":"info","ts":1663256461.5615194,"logger":"config","msg":"In Autodetect, Set HPA version to ["}
{"level":"dpanic","ts":1663256466.5549068,"logger":"config","msg":"non-string key argument passed to logging, ignoring all later arguments","invalid key":"v2beta2","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/config.(*Config).periodicAutoDetect\n\t/workspace/internal/config/main.go:103"}
{"level":"info","ts":1663256466.5548995,"logger":"config","msg":"In Autodetect, Set HPA version to ["}
{"level":"dpanic","ts":1663256471.5632463,"logger":"config","msg":"non-string key argument passed to logging, ignoring all later arguments","invalid key":"v2beta2","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/config.(*Config).periodicAutoDetect\n\t/workspace/internal/config/main.go:103"}
{"level":"info","ts":1663256471.5632415,"logger":"config","msg":"In Autodetect, Set HPA version to ["}
{"level":"dpanic","ts":1663256476.5548413,"logger":"config","msg":"non-string key argument passed to logging, ignoring all later arguments","invalid key":"v2beta2","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/config.(*Config).periodicAutoDetect\n\t/workspace/internal/config/main.go:103"}

[pavolloffay]

@kevinearls was this fixed in 19c186d ?

[pavolloffay]

@frpataciertocom please tell us what version of OTEL operator are you running.

[kevinearls]

@pavolloffay @frpataciertocom Yes, this has been fixed.

[frpataciertocom]

Hi @pavolloffay

Hope that you are doing fine.

I wonder if this is the OTEL version?,

aws eks describe-addon --addon-name adot --cluster-name XXXX 
{
    "addon": {
        "addonName": "adot",
        "clusterName": "XXXX",
        "status": "ACTIVE",
        "addonVersion": "v0.58.0-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:xx-xxxx-x:xxxxxxxxxxxx:addon/zzzzzzzzzzz/adot/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "createdAt": "2022-09-15T12:06:05.710000+02:00",
        "modifiedAt": "2022-09-15T12:06:47.662000+02:00",
        "tags": {}
    }
}

When I decided to install this, I have used the steps from this website https://aws-otel.github.io/docs/getting-started/adot-eks-add-on/installation, so I didn't have any chance to pick a version, I just did aws eks create-addon --addon-name adot --cluster-name <your_cluster_name> , then in the yaml I changed the region and executed kubectl apply -f.

BTW if needed my Kubernetes version: Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.11-eks-18ef993"

ACCORDING TO THIS eksctl command below, that version is okay, however I can see here https://github.com/open-telemetry/opentelemetry-operator that there is a v0.60.0 version available.

aws eks describe-addon-versions \
>     --addon-name adot \
>     --kubernetes-version 1.22 \
>     --query "addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]" \
>     --output text
v0.58.0-eksbuild.1
True
v0.56.0-eksbuild.1
False
v0.51.0-eksbuild.1
False
v0.45.0-eksbuild.1
False

Thank you.

[frpataciertocom]

Hi,

I just wanted to let you know that I have deployed OTEL in a different cluster and I got right the same issue.

Please let me know if you need some more details to understand this, thank you.

[pavolloffay]

0.59.0 upward should have the fix.

Could you please upgrade and let us know/close the issue if it works?

[frpataciertocom]

Hi @pavolloffay

Probably I would need some advise because the update command below doesn't work, I tried with 0.59.0 and 0.60.0 too, please see below.

Then I went to AWS console for EKS and I can see that the latest version available is 0.58.0, please see image attached.

I believe I did everything I could with the documentation available.

Is there anything else I can do to have this working?, cheers.

Command below as per AWS docs: https://docs.aws.amazon.com/eks/latest/userguide/adot-manage.html#adot-update

aws eks update-addon \
>     --cluster-name xxx \
>     --addon-name adot \
>     --addon-version 0.59.0 \
>     --resolve-conflicts OVERWRITE

An error occurred (InvalidParameterException) when calling the UpdateAddon operation: Addon version specified=0.59.0 is not supported for addon=adot

REPEATING FOR VERSION 0.60.0

An error occurred (InvalidParameterException) when calling the UpdateAddon operation: Addon version specified=0.60.0 is not supported for addon=adot

[pavolloffay]

Note that this is an upstream community project, not AWS.

Probably @Aneurysm9 could help you.

[frpataciertocom]

@pavolloffay Honestly I have no idea where to go from here, Thank you.

[xxfroxx]

Hi @Aneurysm9 , Any chance you can advise on this one?, cheers.

[alex88]

@frpataciertocom @xxfroxx you'd have to wait for aws to update to 0.59.0 and make that version available

[Aneurysm9]

The next ADOT collector release is in progress and a release of the EKS managed add-on using v0.61.0 of the Operator will follow. I can't give a firm date at this time, and it is dependent on #1131 landing and the release happening, but would anticipate sometime next week.

[pavolloffay]

I am closing this as the issue was fixed, please open ticket against AWS repo to track their release with the fix.

[frpataciertocom]

I am closing this as the issue was fixed, please open ticket against AWS repo to track their release with the fix.

Fine to me Brother, thank you.