Kubernetes deployment and security concerns

[darioAnongba]

Hi, thank you for this example implementation of a Lightning Service Provider Daemon and for open sourcing it.

I've recently deployed LSPD successfully in Kubernetes infra along with lightningd (helm charts can be found here https://github.com/bitcoin-numeraire/k8s-infra, note that it is a work in progress). Here are the pain points I faced in case in case someone needs it:

Containerization

The main point that I want to raise is regarding containerization, which in turn greatly impacts security. @JssDWt explained that it is currently not possible to use lightningd grpc API so lspd is forced to use lightning-rpc by being deployed in the same container as lightningd.

This breaks the principle of "1 process per container" of Kubernetes, which greatly decreases separation of concerns, ease of deployment and monitoring (can't see lightningd logs):

• I had to implement init containers to download lspd and lspd_cln_plugin and place them in their correct directory for lightningd to use
• I had to change lightningd entrypoint for a custom one where I can run both lightningd (with --daemon) and lspd
• Liveness and readiness are difficult to implement as we will have to check both services are running and then restart the whole container if one is failing. Currently we have no checks for lspd.

The obvious improvement that comes to mind is to encapsulate lightningd and lspd in its own image (by implementing a Dockerfile) guaranteeing that lspd_cln_plugin and lspd are built in the same env as lightningd with a custom entrypoint. it would then just be a matter of pulling the lightningd-lspd image and running it.

Security

More importantly is that lspd, exposed to the internet, is running on the same container as the lightning node, which is in control of the funds. The attack surface is thus immense as lspd has 100% access to the lightning node, its hsm_secret, config, everything. This is of course not acceptable for anything else than a PoC.

Even if the hsm_secret is encrypted (something I recommend you add to your lspd-install.sh: https://docs.corelightning.org/docs/hsm-secret#encrypt-hsm-secret), lspd has access to the running node so it has power to withdraw all the funds.

In the future, IMO the perfect fix is to completely separate lspd from lightningd so that lightningd is in total control of what lspd can do by issuing a very specific rune for lspd, a variety of network policies could be added as well.

NODES

I didn't really understand the NODES env var but it felt like nodes are not part of the configuration of lspd but seemed more like data to be managed as CRUD. Something like lspd add-node or calls to POST lspd/api/nodes, GET /nodes/{id}, etc. It was painful to create this ENV var. Specifying the EXTERNAL_IP seemed weird and a chicken and egg, seems that you first need to run lspd and then add nodes to it?

Versions

• bitcoind v26
• lightningd v24.02
• lspd (waiting for official release after this is fixed: Incompatlble binaries: GLIBC version #205)

[JssDWt]

I made an issue for replacing the client with a grpc version: #209

[JssDWt]

@darioAnongba I've been looking at replacing the client that lspd uses with a grpc client. In order to connect to the cln grpc interface, you'll need to have access to the certficates, namely ca.pem, client.pem and client-key.pem. I thought we could have a configuration option where you either point to a file path, or include the raw certificate. This would allow the grpc server to run on another machine. I'm not exactly sure yet about the rotating times for these certificates. Thinking about the moment these certificates rotate, lspd will be unable to connect to the grpc interface, which would be something to mitigate.

Thinking about your docker deployment, maybe it's possible to have lspd run in a different container than lightningd, but share the lightning-rpc file in a volume, to make it readable by the lspd container? Because ideally when working with certificates, you would also have access to the certificate files themselves for when they become invalid and new ones have to be created

[darioAnongba]

Hi @JssDWt. In general I guess the certificates would be generated outside and injected into the lightningd directory by mounting a volume. That volume can be shared with the lspd container as well. In Kubernetes, using cert-manager and/or secrets will work.

It is also possible to mount the lightning-rpc into a directory in the lspd container. LSPD will act as a sidecar container.

But ideally, the concept of "runes" seems ideal for lspd if it can work: https://docs.corelightning.org/docs/commando