Open question - Using dotnet-monitor as part of an ephemeral / debug container

[LarsBingBong]

So from v1.23+ and stable in Kubernetes v1.25. We have ephemeral containers. I've tried to get the dotnet-monitor to work with ephemeral containers. So far haven't been able to.

I've even tried manually calling the RAW Kubernetes API with good 'ol cURL. To get the volumeMount ( diag / emptyDir ) mounted ( succeeding ). However, not getting much further from there.
The application itself - running as a Kubernetes Deployment - into which I'm trying to inject the dotnet-monitor as an ephemeral / debug container, IS a dotnet v6 application.

I'm just seeing:

 2022-09-13T20:47:36.343180509+02:00 debugger {"Timestamp":"2022-09-13T18:47:36.3357295Z","EventId":60,"LogLevel":"Information","Category":"Microsoft.Diagnostics.Tools.Monitor.Startup","Message":"Tell us about your experience with dotnet monitor: https: │
│ //aka.ms/dotnet-monitor-survey","State":{"Message":"Tell us about your experience with dotnet monitor: https://aka.ms/dotnet-monitor-survey","link":"https://aka.ms/dotnet-monitor-survey","{OriginalFormat}":"Tell us about your experience with dotnet mon │
│ itor: {link}"},"Scopes":[]}                                                                                                                                                                                                                                  │
│ 2022-09-13T20:47:36.347892568+02:00 debugger {"Timestamp":"2022-09-13T18:47:36.3475070Z","EventId":13,"LogLevel":"Warning","Category":"Microsoft.Diagnostics.Tools.Monitor.Startup","Message":"WARNING: Authentication has been disabled. This can pose a se │
│ curity risk and is not intended for production environments.","State":{"Message":"WARNING: Authentication has been disabled. This can pose a security risk and is not intended for production environments.","{OriginalFormat}":"WARNING: Authentication has │
│  been disabled. This can pose a security risk and is not intended for production environments."},"Scopes":[]}                                                                                                                                                │
│ 2022-09-13T20:47:36.425506454+02:00 debugger {"Timestamp":"2022-09-13T18:47:36.4250628Z","EventId":0,"LogLevel":"Warning","Category":"Microsoft.AspNetCore.Server.Kestrel","Message":"Overriding address(es) \u0027http://localhost:52323\u0027. Binding to  │
│ endpoints defined via IConfiguration and/or UseKestrel() instead.","State":{"Message":"Overriding address(es) \u0027http://localhost:52323\u0027. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.","addresses":"http://localhos │
│ t:52323","{OriginalFormat}":"Overriding address(es) \u0027{addresses}\u0027. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead."},"Scopes":[]}                                                                                     │
│ 2022-09-13T20:47:36.436582163+02:00 debugger {"Timestamp":"2022-09-13T18:47:36.4362351Z","EventId":14,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Now listening on: http://localhost:52323","State":{"Message":"Now listenin │
│ g on: http://localhost:52323","address":"http://localhost:52323","{OriginalFormat}":"Now listening on: {address}"},"Scopes":[]}                                                                                                                              │
│ 2022-09-13T20:47:36.436602806+02:00 debugger {"Timestamp":"2022-09-13T18:47:36.4362966Z","EventId":14,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Now listening on: http://:52325","State":{"Message":"Now listening on: htt │
│ p://:52325","address":"http://:52325","{OriginalFormat}":"Now listening on: {address}"},"Scopes":[]}                                                                                                                                                         │
│ 2022-09-13T20:47:36.451610048+02:00 debugger {"Timestamp":"2022-09-13T18:47:36.4513444Z","EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Application started. Press Ctrl\u002BC to shut down.","State":{"Message":"A │
│ pplication started. Press Ctrl\u002BC to shut down.","{OriginalFormat}":"Application started. Press Ctrl\u002BC to shut down."},"Scopes":[]}                                                                                                                 │
│ 2022-09-13T20:47:36.451630551+02:00 debugger {"Timestamp":"2022-09-13T18:47:36.4514439Z","EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Hosting environment: Production","State":{"Message":"Hosting environment: P │
│ roduction","envName":"Production","{OriginalFormat}":"Hosting environment: {envName}"},"Scopes":[]}                                                                                                                                                          │
│ 2022-09-13T20:47:36.451656232+02:00 debugger {"Timestamp":"2022-09-13T18:47:36.4514642Z","EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Content root path: /app/.store/dotnet-monitor/6.2.2/dotnet-monitor/6.2.2/to │
│ ols/net6.0/any/","State":{"Message":"Content root path: /app/.store/dotnet-monitor/6.2.2/dotnet-monitor/6.2.2/tools/net6.0/any/","contentRoot":"/app/.store/dotnet-monitor/6.2.2/dotnet-monitor/6.2.2/tools/net6.0/any/","{OriginalFormat}":"Content root pa │
│ th: {contentRoot}"},"Scopes":[]}                                                                                                                                                                                                                             │
│ 2022-09-13T20:49:46.240066838+02:00 reservation-api The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.                                                                  │
│ 2022-09-13T20:49:46.240129835+02:00 reservation-api DOTNET_DiagnosticPorts="/diag/port-$(PODNAME).sock"                                                                                                                                                      │
│ 2022-09-13T20:49:46.240140342+02:00 reservation-api DOTNET_DefaultDiagnosticPortSuspend=0 

Any advice or simply stating that this is not a supported use-case and why will be greatly appreciated.

Thank you

[LarsBingBong]

Info from:

• https://iximiuz.com/en/posts/kubernetes-ephemeral-containers/
• https://iximiuz.com/en/posts/kubernetes-api-call-simple-http-client/
• https://github.com/dotnet/dotnet-monitor/blob/main/documentation/kubernetes.md
• The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port #1958
• "System.Net.Sockets.SocketException (98): Address in use" exception during rolling deployment #1827 (comment)

used

[jander-msft]

We haven't explicitly tested ephemeral container scenarios but this is something I can look into.

One thing that does stand out from your logs is:

│ 2022-09-13T20:49:46.240129835+02:00 reservation-api DOTNET_DiagnosticPorts="/diag/port-$(PODNAME).sock"  

This suggests that $(PODNAME) wasn't replaced correctly when you applied the deployment. Unless the dotnet-monitor ephemeral container was applied with that exact name (/diag/port-$(PODNAME).sock) then the target application is not going to find it. Check what the realized path of the socket file is from the perspective of the dotnet-monitor container and make sure the DOTNET_DiagnosticPorts environment variable on the application container is resolved appropriately.

[LarsBingBong]

Yeah sorry @jander-msft - you're correct /diag/port-$(PODNAME).sock looks weird. I have tried with just /diag/port.sock and still had the same experience. Reading some of the linked dotnet-monitor issues made me try the $(PODNAME) sock replacement method. But, the way I deployed the ephemeral container it wasn't properly substituted with the actual value.

I would really appreciate if you had a look at this.

---

Also, by using:

        env:
        - name: ASPNETCORE_URLS
          value: http://+:80
        - name: DOTNET_DiagnosticPorts
          value: /diag/port.sock

The container waits until a debugger is attached. Is there a way to prep. a dotnet related container workload/service so that it's not waiting .... e.g. of a waiting situation below.

2022-09-13T20:49:46.240066838+02:00 reservation-api The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.                                                                  │
│ 2022-09-13T20:49:46.240129835+02:00 reservation-api DOTNET_DiagnosticPorts="/diag/port.sock"                                                                                                                                                      │
│ 2022-09-13T20:49:46.240140342+02:00 reservation-api DOTNET_DefaultDiagnosticPortSuspend=0 

So that one can use ephemeral containers and attach to a dotnet workload on a needs be basis/ad-hoc

Thank you very munchies

[LarsBingBong]

So we tried going another route. So not use ephemeral containers. That works. When we use ephemeral containers I'm not sure that the needed diag volumeMount is mounted at the correct time. As a port.sock file never appears in the /diag/ folder.

This is the cURL we've tried when injecting the ephemeral container.

curl https://192.168.23.18:6443/api/v1/namespaces/spas/pods/{POD_NAME}/ephemeralcontainers \
    --cacert ~/.kcli/clusters/some-cluster/auth/ca.crt \
    --cert ~/.kcli/clusters/some-cluster/auth/client.crt \
    --key ~/.kcli/clusters/some-cluster/auth/client.key \
    -XPATCH \
    -H 'Content-Type: application/strategic-merge-patch+json' \
    -d '
{
    "spec":
    {
        "ephemeralContainers":
        [
            {
                "name": "debugger",
                "args": [
                    "--no-auth"
                ],
                "env": [{
                    "name": "DOTNETMONITOR_DiagnosticPort__ConnectionMode",
                    "value": "connect"
                },
                {
                    "name": "DOTNETMONITOR_DiagnosticPort__EndpointName",
                    "value": "/diag/port.sock"
                },
                {
                    "name": "DOTNETMONITOR_Storage__DumpTempFolder",
                    "value": "/diag/dumps"
                }],
                "image": "mcr.microsoft.com/dotnet/monitor",
                "targetContainerName": "api",
                "securityContext": {
                    "runAsUser": 1000
                },
                "stdin": true,
                "tty": true,
                "volumeMounts": [{
                    "emptyDir": "{}",
                    "mountPath": "/diag",
                    "name": "diagvol"
                }]
            }
        ]
    }
}'

N.B. {POD_NAME} was updated before use with the name of an existing Pod

---

Any ideas and thanks.

[jander-msft]

So we tried going another route. So not use ephemeral containers. That works. When we use ephemeral containers I'm not sure that the needed diag volumeMount is mounted at the correct time. As a port.sock file never appears in the /diag/ folder.

This is the cURL we've tried when injecting the ephemeral container.

curl https://192.168.23.18:6443/api/v1/namespaces/spas/pods/{POD_NAME}/ephemeralcontainers \
    --cacert ~/.kcli/clusters/some-cluster/auth/ca.crt \
    --cert ~/.kcli/clusters/some-cluster/auth/client.crt \
    --key ~/.kcli/clusters/some-cluster/auth/client.key \
    -XPATCH \
    -H 'Content-Type: application/strategic-merge-patch+json' \
    -d '
{
    "spec":
    {
        "ephemeralContainers":
        [
            {
                "name": "debugger",
                "args": [
                    "--no-auth"
                ],
                "env": [{
                    "name": "DOTNETMONITOR_DiagnosticPort__ConnectionMode",
                    "value": "connect"
                },
                {
                    "name": "DOTNETMONITOR_DiagnosticPort__EndpointName",
                    "value": "/diag/port.sock"
                },
                {
                    "name": "DOTNETMONITOR_Storage__DumpTempFolder",
                    "value": "/diag/dumps"
                }],
                "image": "mcr.microsoft.com/dotnet/monitor",
                "targetContainerName": "api",
                "securityContext": {
                    "runAsUser": 1000
                },
                "stdin": true,
                "tty": true,
                "volumeMounts": [{
                    "emptyDir": "{}",
                    "mountPath": "/diag",
                    "name": "diagvol"
                }]
            }
        ]
    }
}'

N.B. {POD_NAME} was updated before use with the name of an existing Pod

Any ideas and thanks.

DOTNETMONITOR_DiagnosticPort__ConnectionMode needs to be set to Listen. Setting it to Connect will prevent dotnet-monitor from creating the server Unix Domain Socket specified in the DOTNETMONITOR_DiagnosticPort__EndpointName environment variable. Alternatively, just set DOTNETMONITOR_DiagnosticPort to /diag/port.dock (the sub properties don't need to be specified), which has dotnet-monitor create the server Unix domain socket.

[jander-msft]

I'm going to close this out due to lack of activity and I believe I've answered your last question of why the port.sock file was not appearing in the /diag directory.

[LarsBingBong]

Yes sorry @jander-msft for being unresponsive.