[Feature] Support tailscale serve with HTTPS

[teleclimber]

Use case

Tailscale serve is very useful for exposing a server in your tailnet. For those of us who use Tailscale to expose servers either privately with other users or globally using Funnel, this feature is borderline magical. I'd love to see Headscale support it.

Description

A complete description of the ts serve is here: https://tailscale.com/kb/1242/tailscale-serve

Contribution

• I can write the design doc for this feature
• I can contribute this feature

How can it be implemented?

Honestly I don't know how much is involved here, but I'm willing to try and have a look.

[teleclimber]

Some clarifications:

$ tailscale serve --bg --http 80 http://localhost:3003

Works as expected. It gives me a http URL of the form http://<my-machine>.<my-username>.<my headscale-domain> that I can punch into my browser and that gets me a response from the small server I have running locally on :3003.

Where it goes wrong is if I don't include the --http 80, it defaults to https, and that's where the tailscale CLI prints this error:

error enabling https feature: error 404 Not Found: 404 page not found:

So basically it's the https part that I want to try to enable. Any hints on where to start would be greatly appreciated.

[teleclimber]

I spent some time going over the Tailscale client code to see what needs to happen.

Since the serve feature already works for HTTP, the missing piece mostly involves getting and using a TLS certificate for the right domain.

It is clear from the docs and the code that Tailscale fully expects to be involved in provisioning a certificate for that node. See https://tailscale.com/kb/1153/enabling-https

Additional fact: DNS-01 is the only LetsEncrypt challenge that the tailscale client can solve. See this line.

The following options are ruled out unless Tailscale make changes to their clients:

• Using a wildcard certificate is not possible. There is currently no way to tell tailscale serve to use that cert, AFAIK.
• Doing an HTTP-01 challenge, which would be easier to implement than DNS-01, is not possible unless that challenge is implemented on the client side too.

With that out of the way the only path forwards is to have Headscale implement DNS-01. I know of two approaches to this:

• Make API calls to DNS name servers to set records as needed. Thanks to Caddy server there is precedent and plenty of Go Code for this.
• Embed something like acme-dns into headscale.

I'd be interested to know maintainer's thoughts on this at this point. Thanks.

[Hypnotist1148]

This would also be the first step to have stuff like funnel working!

[bentemple]

I fully support this. Tried to setup a serve for a client yesterday, as I wanted to use a tailscale sidecar to expose a service, but I can't connect to it except through http. Would be so cool if it would work with https. Especially since I'm using a domain name with hsts so I have no choice but to connect via IP to use HTTP (I'd much prefer magicDNS of course)

[ananthb]

I can pitch in code for this as well @teleclimber. Would love to see this feature on Headscale.

[pavanbuzz]

I spent some time going over the Tailscale client code to see what needs to happen.

Since the serve feature already works for HTTP, the missing piece mostly involves getting and using a TLS certificate for the right domain.

It is clear from the docs and the code that Tailscale fully expects to be involved in provisioning a certificate for that node. See https://tailscale.com/kb/1153/enabling-https

Additional fact: DNS-01 is the only LetsEncrypt challenge that the tailscale client can solve. See this line.

The following options are ruled out unless Tailscale make changes to their clients:

• Using a wildcard certificate is not possible. There is currently no way to tell tailscale serve to use that cert, AFAIK.
• Doing an HTTP-01 challenge, which would be easier to implement than DNS-01, is not possible unless that challenge is implemented on the client side too.

With that out of the way the only path forwards is to have Headscale implement DNS-01. I know of two approaches to this:

• Make API calls to DNS name servers to set records as needed. Thanks to Caddy server there is precedent and plenty of Go Code for this.
• Embed something like acme-dns into headscale.

I'd be interested to know maintainer's thoughts on this at this point. Thanks.

@teleclimber Tracing it further, tailscale already creates the acme challenge record (key,value). It then calls the control plane to SetDNS at #L472.

If we trace this call, noiseClient sends a POST request to the control server at api endpoint /machine/set-dns with NodeKey and Body SetDNSRequest.

So if we handle this endpoint in Headscale by adding a new TXT record into the corresponding provider (like cloudflare, digitaloceans, etc). Then tailscale serve would be able to obtain a TLS certificate.

We need a way to let users configure their dns provider. Traefik uses environment variables/secret files to configure. Also they use a library like lego to handle DNS challenge. But in this case, we just need to add a new record to the provider.

[ananthb]

@pavanbuzz since the new beta release changes Node Magic DNS names to <node>.<your-domain> instead of <node>.<user>.<your-domain>, we could also solve HTTP-01 or TLS-ALPN-01 challenges.

Users can point *. to their headscale instance via DNS.

The advantage being that we don't need to support upstream DNS APIs.

[pavanbuzz]

@pavanbuzz since the new beta release changes Node Magic DNS names to <node>.<your-domain> instead of <node>.<user>.<your-domain>, we could also solve HTTP-01 or TLS-ALPN-01 challenges.

@ananthb I understand. I think the recent beta changes for Magic DNS to <node>.<your-domain> will be similar to the one tailscale offers <node>.<ts-name>.ts.net. Headscale already handles HTTP-01 or TLS-ALPN-01 challenges to get a TLS certificate for headscale instance.

Users can point *. to their headscale instance via DNS.

The advantage being that we don't need to support upstream DNS APIs.

It's definitely less maintenance. Does it mean then headscale stores all the certificates/keys? How will the node that is requesting a TLS be able to get a certificate then? This approach is similar to caddy-tailscale (Tailscale plugin for caddy). Please correct me, if am mistaken.

Tailscale cli/api requests a TLS certificate on the node where serve/funnel is invoked and saves the certificate/key in that node. In order to get tailscale serve work natively with TLS, Headscale should handle /machine/set-dns endpoint and create a corresponding dns entry in the authoritative dns server for that domain. I believe this is how Tailscale is accomplishing this feature.

This functionality can be extended to accomplish Funnel feature, but it requires atleast one node in the tailnet which can receive public traffic and do a tcp routing based on server name indication. This way TLS traffic is not decrypted by that public facing node.

[ananthb]

@pavanbuzz I get it now. With the DNS challenge, the node requesting the cert can fetch it directly from an ACME issuer.

Letting the node handle its own secret material is infinitely better.

[ananthb]

As @teleclimber pointed out earlier, we could embed a DNS server inside the headscale server and make it authoritative for a domain.

[pavanbuzz]

As @teleclimber pointed out earlier, we could embed a DNS server inside the headscale server and make it authoritative for a domain.

@ananthb That is a good idea!

I am leaning towards using other provider for DNS due to several reasons (including but not limited to),

• Not too familiar with setting up authoritative DNS server
• Require a minimum of 2 NS
• Security, performance and maintenance of DNS server

I found that using lego's Challenge.PreSolve will be able to add a TXT record. Since the ts-node is going to check for propagation and obtaining a certificate, headscale server has to just insert the record and return a response.

Because lego has support for multiple dns providers, it allows users to choose any DNS servers (including custom DNS servers). Values (or file reference) can be passed-in via Environment variables. More info can be found in their docs. We can take a similar approach to Traefik's dnsChallenge (although, headscale will not obtain the certificate, but just add DNS record and clean it up later).

[ananthb]

Leaning on lego for challenge providers sounds promising.

[pavanbuzz]

I found that using lego's Challenge.PreSolve will be able to add a TXT record. Since the ts-node is going to check for propagation and obtaining a certificate, headscale server has to just insert the record and return a response.

I did a bit more analysis and found it shouldn't be Challenge.PreSolve, but instead Provider.Present. However there is an issue using Present method. Based on many provider's implementation, it hashes the value before adding an entry into the dns provider. Tailscale client already sends a hashed value (solved the challenge). So this won't work. There is an issue opened in their repo to allow adding the value without hashing. Until this functionality is added, we cannot use lego to achieve this feature.

Libdns seems promising and can do what we require. But they are still WIP. Would it be okay to add that dependency in headscale?

@ananthb Your idea of DNS server would be better solution as well. Do you have any suggestion for embedded dns? This would require a detailed instruction of how to get headscale to become an authoritative server for a subdomain.

Any idea how do we proceed?

[ananthb]

It was @teleclimber's idea to embed an authoritative DNS server in headscale. They've even linked to one we can use.

But, the more I think about it, the less this sounds like a good idea. Headscale cannot be hosted in a high availability configuration, so any DNS server hosted by it will suffer from reliability issues.

That Lego issue is moving frustratingly slowly for what should be a straightforward change.

I haven't looked at libdns in depth, but it looks promising.

We should be able to write our own interface similar to the lego one that we are unable to use currently.

I'd strongly recommend against hosting our own DNS. There be dragons.

[pavanbuzz]

@teleclimber I think your idea to use libdns would be the way to go. I can help coding this feature. Kindly let me know if you have started working on this. Would be happy to help.

[teleclimber]

Hi everyone, I'm happy to see some enthusiasm for adding this feature to headscale. Thanks for all the comments.

One concern I have about setting records on a third party DNS provider like cloudflare etc.. is that many of them do not offer granular control over what the API key allows. On Porkbun (one that I have used) it's all or nothing. If I make an API key and allow it for the domain, that API key can be used to change the A records. Not great. Cloudflare appears to be the same, according to this: https://developers.cloudflare.com/fundamentals/api/reference/permissions/#zone-permissions

Some DNS providers don't even offer any API access to change DNS records. So going with 3rd party nameservers implies that Headscale users may have to change their domain's nameservers to use one that has an API. Hopefully that's not too high a burden.

Of course, if headscale is the authoritative nameserver, then all the same issues apply: a security issue in Headscale could allow someone to change your A record. And of course you have to change the nameserver to your headscale. So it's the same burden in the end.

Between the two options I think setting records on a third party DNS authoritative name server is easier to implement, easier to set up for the user, and likely more reliable. The risks in case of a security issue are about the same.

@pavanbuzz I haven't started working on this yet. If we agree that we should go with 3rd party and something like libdns, I would like to hear from maintainers whether they would accept libdns as a dependency. Also I need to dig into libdns and headscale code a lot more.

[pavanbuzz]

Hi everyone, I'm happy to see some enthusiasm for adding this feature to headscale. Thanks for all the comments.

One concern I have about setting records on a third party DNS provider like cloudflare etc.. is that many of them do not offer granular control over what the API key allows. On Porkbun (one that I have used) it's all or nothing. If I make an API key and allow it for the domain, that API key can be used to change the A records. Not great. Cloudflare appears to be the same, according to this: https://developers.cloudflare.com/fundamentals/api/reference/permissions/#zone-permissions

@teleclimber - Thanks for this info. I didn't know that other providers did not provide granular control. Though Cloudflare provides creation of api token for specific resource (domain). This token can edit DNS records only for this zone (step-6 gives instruction as to how to select this). I am using this setup currently.

But you are right, its a security implication that needs to be carefully considered.

Of course, if headscale is the authoritative nameserver, then all the same issues apply: a security issue in Headscale could allow someone to change your A record. And of course you have to change the nameserver to your headscale. So it's the same burden in the end.

I don't think it will be an issue. We could design a solution similar to joohoi/acme-dns, by using a DNS server as an authority server for a subdomain, instead of the main domain. This way, only DNS requests for this subdomain and subdomains of this subdomain will be served. This is achieved by creating a NS record for the subdomain on the main DNS provider along with a A record that points to the DNS server (info provided in dns-records section of joohoi/acme-dns).

Note - We might not be able to use joohoi/acme-dns. It requires a way to add a CNAME redirection (ACME magic) into the main DNS provider for each challenge (goes back to the same issue of security).

Let me try to explain the above logic with an example. If the main domain is example.com & hs.example.com is the subdomain. This new DNS server will become an authoritative server for hs.example.com & *.hs.example.com. If this server is ever compromised, impact is restricted only to the hs.example.com and *.hs.example.com.

Note - I think remediation could also be as easy as users logging into their main DNS provider and disabling the NS & A record for this subdomain.

There are two main hurdles for users that I can think of.

1. Users have to change their dns_config.base_domain to a subdomain like hs.example.com. So all the hostnames for magic_dns will become myhost.hs.example.com.
2. Create a NS & A record (one-time setup) in their main DNS provider.

There are few of things for implementing this feature -
We require a dns server that provides a mechanism (API/RPC/etc) to create/update/delete (TXT,CNAME,A,AAAA) records.

• Leverage existing DNS servers like CoreDNS with file/redis/grpc plugin. I can't think of any small dns servers other than CoreDNS that we can leverage.
  Pros

  • Faster feature implementation
  • Less code maintenance
  • Can be used for both acme_challenge (TXT) & Funnel DNS (CNAME/A/AAAA) records out of the box
  • With grpc plugin used by CoreDNS, we could request it to fetch data from Headscale. Headscale could save the details (like TXT, CNAME, A, AAAA) in the DB (sqlite/postgres).

  Cons

  • Feature would rely explicitly on users to setup the stack properly. Especially if we plan to use CoreDNS along with redis plugin.
  • Detailed guide and on-going support to questions related to setting up this stack like the community maintained guides.

• Write a micro-dns embedded within Headscale server explicitly for acme_challenges and DNS requests for Funnel.
  Pros

  • Everything works out of the box without additional setup required from users

  Cons

  • More code & more maintenance

Between the two options I think setting records on a third party DNS authoritative name server is easier to implement, easier to set up for the user, and likely more reliable. The risks in case of a security issue are about the same.

I tried libdns, and its actually pretty easy. Though the custom dns server is more secure, it also means more work.

@teleclimber / @ananthb - let me know what do think. Hope I didn't confuse.

[ananthb]

As to the question of DNS zone security, the blast radius is the same whether headscale can manipulate a third-party hosted zone or whether its hosting the zone.

Self-hosting reliable DNS means at least two servers for failover and a whole other can of worms besides.

My vote is resoundingly for third-party DNS server support.

[teleclimber]

Let me try to explain the above logic with an example. If the main domain is example.com & hs.example.com is the subdomain. This new DNS server will become an authoritative server for hs.example.com & *.hs.example.com. If this server is ever compromised, impact is restricted only to the hs.example.com and *.hs.example.com.

Yes this is how I was imagining we would do things. I may have been too loose with terminology, using "domain" instead of subdomain and zone. Sorry for the confusion.

My vote is resoundingly for third-party DNS server support.

Yes I think that's where I'm at as well.

Note that nothing prevents headscale from supporting other options down the line.

[mitchellkellett]

I've been quietly following this in the background. I've previously taken a look at jsiebens/ionscale, and I can see that they are using libdns for their implementation of Serve. Looks like that might be the way to go for now at least.

[pavanbuzz]

If we all agree with libdns for now, should we involve the maintainers now? We can embed dns (with limited scope for acme challenge and funnel dns response) implemented later.

Sequence diagram with external DNS server using libdns

%%{init: {'sequence': {'rightAngles': true}} }%%

sequenceDiagram
title TLS certificate flow with Cloudflare/Other DNS Provider

participant node as ts-node
participant hs as Headscale Server
participant le as Let's Encrypt
participant dns as Cloudflare DNS Server

node->>+le: AuthorizeOrder with `DNS-01` Challenge
le-->>-node: Challenge value

node->>+hs: /machine/set-dns

hs->>+dns: using libdns to save _acme-challenge.subdomain.example.com

dns-->>-hs: Saved

hs-->>-node: OK

node->>+le: Challenge accepted

le->>+dns: lookup _acme-challenge.subdomain.example.com

dns-->>-le: TXT record

le->>-le: validate response

node->>+le: get status
le-->>-node: challenge verified
node->>+le: CreateOrderCert
le->>-node: Certificate

Loading

Sequence diagram with embedded DNS server in Headscale (future implementation - if maintainers are okay)

%%{init: {'sequence': {'rightAngles': true}} }%%

sequenceDiagram
title TLS certificate flow with Headscale as DNS server

participant node as ts-node
participant hs as Headscale Server
participant le as Let's Encrypt

node->>+le: AuthorizeOrder with `DNS-01` Challenge
le-->>-node: Challenge value

node->>+hs: /machine/set-dns
hs->>hs: save _acme-challenge.subdomain.hs.example.com
hs-->>-node: OK


node->>+le: Challenge accepted
critical DNS lookup on port 53/other port if redirected using rinetd

le->>+hs: lookup _acme-challenge.subdomain.hs.example.com

hs->>hs: fetch _acme-challenge.subdomain.hs.example.com from db
hs-->>-le: TXT record

end


le->>-le: validate response

node->>+le: get status
le-->>-node: challenge verified
node->>+le: CreateOrderCert
le-->>-node: Certificate

Loading

[teleclimber]

Nice diagrams @pavanbuzz . You're well ahead of me on this, I haven't had much time to dive in. If you want to take the lead on this I wouldn't be offended.

[pavanbuzz]

Nice diagrams @pavanbuzz . You're well ahead of me on this, I haven't had much time to dive in. If you want to take the lead on this I wouldn't be offended.

@teleclimber thanks! This is my first experience with Go. So might take a bit longer, but i will get this up and running.

[pavanbuzz]

@juanfont / @kradalby - Our objective is to incorporate the tailscale serve feature into the Headscale server. To achieve this, @teleclimber has proposed two options detailed below:

Since the serve feature already works for HTTP, the missing piece mostly involves getting and using a TLS certificate for the right domain.

It is clear from the docs and the code that Tailscale fully expects to be involved in provisioning a certificate for that node. See https://tailscale.com/kb/1153/enabling-https

Additional fact: DNS-01 is the only LetsEncrypt challenge that the tailscale client can solve. See this line.

The following options are ruled out unless Tailscale make changes to their clients:

• Using a wildcard certificate is not possible. There is currently no way to tell tailscale serve to use that cert, AFAIK.
• Doing an HTTP-01 challenge, which would be easier to implement than DNS-01, is not possible unless that challenge is implemented on the client side too.

With that out of the way the only path forwards is to have Headscale implement DNS-01. I know of two approaches to this:

• Make API calls to DNS name servers to set records as needed. Thanks to Caddy server there is precedent and plenty of Go Code for this.
• Embed something like acme-dns into headscale.

I'd be interested to know maintainer's thoughts on this at this point. Thanks.

We believe that leveraging libdns would be the optimal approach, given its compatibility with various external DNS providers such as Cloudflare. This choice also sets the stage for the future integration of the Tailscale Funnel feature.

Corresponding sequence diagrams can be found here #1921 (comment) .

We would like to get your opinion so we can move forward with the implementation.

[kradalby]

I think serve is quite attainable, while funnel is less realistic, but happy for someone to work towards it.

I think the work should be split into dns+serve standalone, and then potentially funnel in the future.

My main concern with all user contributed code is outlined in our contribution guidelines.

I'm positive to someone contributing it, but we will not accept it if we find that it is likely going to cause us a large burden now that we have other things to do. We would eventually aim to get to this ourselves, but not sure when that would be.

Summarised, it needs to be:

• Very well tested (integration mostly for this I would assume)
• Well documented, code + docs
• Any external dependencies need to be vetted.

[pavanbuzz]

I think serve is quite attainable, while funnel is less realistic, but happy for someone to work towards it.

I believe I have an idea on how to achieve this. Though this would require building a separate funnel ingress server like a derp server and should be self-hosted separately by users. Funnel can be dealt later once Serve is implemented.

I think the work should be split into dns+serve standalone, and then potentially funnel in the future.

I don't understand this part. Do you mean separate PRs for dns+serve standalone?

I'm positive to someone contributing it, but we will not accept it if we find that it is likely going to cause us a large burden now that we have other things to do. We would eventually aim to get to this ourselves, but not sure when that would be.

I understand the concern. And would stick to the contribution guidelines.

Summarised, it needs to be:

• Very well tested (integration mostly for this I would assume)

I am not sure how we can test the part where the DNS records are updated. But i think other unit tests & integration tests for other things are doable.

• Any external dependencies need to be vetted.

@kradalby This is where we would like your opinion as well , whether the PR would be accepted if we use libdns for updating DNS records.

[kradalby]

I don't understand this part. Do you mean separate PRs for dns+serve standalone?

Do what is needed for serve, and just dont start on funnel, I would be comfortable with giving a thumbs up for serve, but not funnel.

I am not sure how we can test the part where the DNS records are updated. But i think other unit tests & integration tests for other things are doable.

Yes, I think the logic of what and how it is set should be tested, but not necessarily the upstream.

@kradalby This is where we would like your opinion as well , whether the PR would be accepted if we use libdns for updating DNS records.

libdns looks fine, I think it is the one I looked at last time this came up.

A nice exercise for using libdns would be to replace/add to the current configuration and logic to set up headscale itself with HTTPS, the config is old and yankee and could use some love and nicer configuration.

[ananthb]

Funnel definitely needs more from the community than I think we can ask of it/ourselves for now.

I'm also comfortable pitching in on serve. @pavanbuzz we can work together on this if that works for you.

[pavanbuzz]

Funnel definitely needs more from the community than I think we can ask of it/ourselves for now.

I'm also comfortable pitching in on serve. @pavanbuzz we can work together on this if that works for you.

That would be great @ananthb, lets have a chat to see how we can split the work and get started!

[xzzpig]

Perhaps I misunderstood, but I am puzzled as to why we insist on automatically obtaining SSL certificates?
I think for Headscale, what is needed to serve HTTPS should be similar to other regular HTTP servers such as nginx. Headscale administrators should provide SSL certificates, and what Headscale needs to do is to find suitable certificates from the SSL certificates provided by users to serve HTTPS.
As for the certificate content, the administrator can provide a generic domain name certificate (*.) or multiple specific domain name certificates.
At the same time, for the convenience of automation, Headscale can trigger external programs to automatically apply for certificates by configuring webhooks or provide API to provide certificates that need to be applied for, allowing external programs to automatically poll and apply for certificates.

[pavanbuzz]

@pavanbuzz do you want to get started?

Hey @ananthb , I will not be work on it for a while as am busy with other work. Please feel free to get started with this feature.

[madejackson]

Perhaps I misunderstood, but I am puzzled as to why we insist on automatically obtaining SSL certificates? I think for Headscale, what is needed to serve HTTPS should be similar to other regular HTTP servers such as nginx. Headscale administrators should provide SSL certificates, and what Headscale needs to do is to find suitable certificates from the SSL certificates provided by users to serve HTTPS. As for the certificate content, the administrator can provide a generic domain name certificate (*.) or multiple specific domain name certificates. At the same time, for the convenience of automation, Headscale can trigger external programs to automatically apply for certificates by configuring webhooks or provide API to provide certificates that need to be applied for, allowing external programs to automatically poll and apply for certificates.

When you read the earlier comments in this issue, you'll realize that it has been established already how tailscale serve works on the client side: It requests an SSL-Cert via DNS Challenge. From headscales perspective, this is given and cannot be changed. Headscale has no control over the clients functionality so it can only respond to the client's requests accordingly.

For the client to be able to do a DNS-Challenge, it relies on the coordination server (headscale) to enter in the necessary DNS-entries in to the authorative DNS-server. This is the part that has to be implemented into headscale.

[p3lim]

I am not sure how we can test the part where the DNS records are updated.

Could support RFC2136 and set up a minimal BIND9 server to facilitate DNS-01 during the tests, if libdns (if it ends up using that) supports this method.

[almereyda]

For running a self-hosted DNS API for integration testing, libdns also has a PowerDNS provider, next to its older RFC2136 provider the more up to date DNS UPDATE provider.

This way a PowerDNS container could be spun up and a zone served. Together with a custom ACME endpoint, such as one offered by a Smallstep CA, and its certificate loaded into the test system's trust store, plus a reverse proxy and ACME client (Caddy, Traefik, etc.), it is possible to build a complete DNS-01 ACME test environment. I would be happy to provide example boilerplate for such a setup, when needed.

[mitchellkellett]

Could take a look at how the folks over at DNSControl do it as well, they integrate with a number of providers and have tests for each one.

[e-zk]

Maybe I'm confusing things but I am able to run tailscale serve on my windows machine to expose a local service to my Headscale tailnet and it just works.

Edit: I see now this discussion here is more about serve defaulting to https and how to support that my bad.

[imft-debug]

@e-zk Tailscale serve works but for headscale the opensource version of it doesnt allow serve to expose internal services to the internet. Try web server to expose the website hosted on interal pods / docker to expose to internet as well

[kradalby]

ah, I was under the understanding that serve didnt work, and from what I understand from the last comment, is this issue actually trying to implement Tailscale Funnel? If so thats cool, should we rename this issue and consolidate other questions about funnel?

[mitchellkellett]

Last time I tried to use serve (about a month ago) I had issues and had to use --http <port> flag.

My understanding of this issue is that serve by default will still want certificates for HTTPS, and this will resolve it.

[teleclimber]

serve works but not over HTTPS. The problem is headscale is not wired to fetch TLS certs, which is what the tailscale CLI expects. This issue is about adding support for automatic TLS cert acquisition.

$ tailscale serve

..doesn't work because by default tailscale tries to use :443 and get a TLS cert. So it fails.

$ tailscale serve --bg --http 80 http://localhost:3003

..works because no cert is being fetched.

Funnel support is not part of this issue, but funnel support will be much easier when this issue is resolved.

[korpa]

Is anyone working on this feature?

[minhnqn]

Is this feature similar to caddy?

[nom3ad]

Is anyone working on this feature?

I've been having a working setup on my local fork for quite some time. That codebase is a mess due to personal customizations I made, which doesn't make any sense to be in upstream. I think I even have a setup for "funnel" that's extremely hacky and very close to working.
I can spend sometime over this weekend to pull out "serve" related code and put a draft PR for others to try out.
FWIW, my approach to DNS update was to allow headscale to run a command, I did something like follows.

serve:
	enable: true
	set-dns-command: "/path/to/route53-update-script"

[korpa]

... I can spend sometime over this weekend to pull out "serve" related code and put a draft PR for others to try out. ...

That would be great

[ananthb]

I'm available to test and to clean up code.

[korpa]

I started to look into this and have a POC with all my personal values hardcoded into the code which already works. This helped me to understand, that tailscaled does the heavy-lifting of cert creation. Headscale "just" has to forward the DNS challenge to the DNS provider.

The conclusion from previous comments was to use libdns. But this means very much work to add all DNS providers as every API needs different credentials and config values. With all that knowledge and @nom3ad's solution to use a script I was thinking of an hybrid solution: Use libdns for the 5 to 10 most common DNS providers and as an alternative to provide the script/command solution. Then anyone who uses a not so common DNS provider could create a small binary with libdns or even create a shell script with an API call via curl.

What do you think?

[nom3ad]

FWIW, I have placed a draft PR to showcase how I did it this on my setup.
In my case, it's Route53 and I have a simple AWS CLI wrapper to set DNS.

[korpa]

@nom3ad Thank you very much. I had a look at your code. In general my code looks similar except of the exec part.

While cleaning up my code and testing I encountered a few things which are not so easy to cover with the "command" solution:

• If anything goes wrong, the TXT record lives forever in DNS and prevents to set a new record. This could be solved that the command removes all TXT records for that entry. But still the TXT record stays in there very long and clutters the zone.
• After the record was set it could take a couple of seconds before it can be found by Lets-Encrypt. Therefore we should test every couple of seconds if the record is accessible before returning a success to tailscaled. In case of a timeout or error we have to remove the TXT entry
• After everything went ok, we want to remove the TXT record as well to unclutter the zone. As we do not get informed by tailscaled after it got an cert (or ran into an error e.g. Lets-Encrypt rate limit - don't ask how I know :) ), I added a sleep of 30s before removing the record again.

All in all this could be done by a script as well, but it is not straight forward.

[ananthb]

The command solution seems brittle to me. I would much rather keep this in Go if at all possible. If headscale starts depending on external commands then this breaks the current model where everything that an installation needs is inside the single binary.

[renne]

LEGO seems to have the most extensive support for DNS providers (unfortunately I wasn't able to persuade the IETF ACME working group to make an already existing standard like RFC2136 mandatory).
Are the problems with the LEGO-API solved meanwhile?

PowerDNS seems to be a good choice as it is the only authoritative DNS server supported in Proxmox Virtual Environment Software Defined Network zones.

[nblock]

The command solution seems brittle to me. I would much rather keep this in Go if at all possible.

DNS setups are site-specific and I doubt that a single library can cope with all kinds of DNS configurations. Some kind of custom command/script support - be it in headscale or in a library itself - may be needed to support such custom setups.

[ananthb]

I personally cannot acquiesce to a solution that is built on the hope that a reliable command with wide ranging DNS configuration support will materialise. Headscale requires very specific configuration to DNS records and I cannot see a universal way to do that on the back of a command runner.

I'm also not sure that a solution involving shelling out to arbitrary commands will be merged.

We agreed on an implementation direction with the maintainer of this project earlier in this same thread. The quick summary is that we planned to implement serve using libdns.

[ananthb]

@nom3ad I will work on adapting your PR to use libdns.