memcached dns+ addresses resolve from CNAME

[oneacl]

Describe the bug

With cortex I could use the memcached addresses dns+memcached.example.com configuration (although the documentation was clearly saying that it has to resolve to A/AAAA so maybe it was an unknown bug?)

This no longer works with mimir.

To Reproduce

Configure mimir with:
bucket_store:
metadata_cache:
backend: memcached
memcached:
addresses: dns+memcached.example.com

where memcached.example.com is a CNAME to someserver.example.com and someserver.example.com IN A 192.0.2.1

The mimir output will return:
name=metadata-cacche msg="failed to resolve addresses for memcached" addresses=dns+memcached.example.com err="2 errors: lookup IP address - invalid A or AAA response record [..] IN CNAME somserver.example.com

Additional Context

Unfortunately due to my infrastructure is deployed I can only have a CNAME for the memcached servers (and no SRV records either).

Would there be a workaround here for me to try?

Thank you.

[pracucci]

Thanks for your report!

I did a brief investigation and I've been able to reproduce it. The change has been introduced in #142 where memcached client DNS resolver moved from golang one to miekgdns.

To better understand your use case and properly fix it, could you show me the output of dig memcached.example.com (replacing memcached.example.com with the the actual host and then redacting any sensitive data), please?

[oneacl]

Thanks for the quick reply and also figuring out where it was introduced.

The redacted output looks like this:

;; ANSWER SECTION:
memcached.example.com. 10 IN CNAME cluster-abc.example.com
cluster-abc.example.com. 60 IN A 192.0.2.1
cluster-abc.example.com. 60 IN A 192.0.2.2

In my environment everything except memcached1.example.com is dynamic (including the cluster1.example.com name) - hence I need to use the cname.

I'm also expecting the resolver to obey the TTLs as quite frequently it will change to (along the lines of)

;; ANSWER SECTION:
memcached.example.com 10 IN CNAME cluster-def.example.com
cluster-def.example.com 60 IN A 192.0.2.3
cluster-def.example.com. 60 IN A 192.0.2.4

Thanks again.

[pracucci]

Thanks a lot for your reply. I've opened a Thanos PR that should fix it:
thanos-io/thanos#5271

[oneacl]

Thank you too for the prompt fix, really appreciate it.