how to handle clean updates of dotnetcore agent

[ruben-herold]

hi,

we have the situation that we have running multiple dotnet services on the same system with different users and groups.
We start them via systemd like this:

# [email protected]
[Unit]
Description=DotNetStarter Service: %i
AssertPathExists=/home/%i
AssertPathExists=/etc/dotnetservices/%i-ports.conf

[Service]
EnvironmentFile=/etc/dotnetservices/%i-ports.conf
Environment=DOTNET_CLI_TELEMETRY_OPTOUT=1
Environment=ASPNETCORE_ENVIRONMENT=Production

## AppDynamics
Environment=LD_LIBRARY_PATH=/opt/appdynamics/dotnet

Environment=CORECLR_PROFILER={XXXXxxXX-xxxx-xxxx-xxxx-xxxxxxxxxxxx}
Environment=CORECLR_ENABLE_PROFILING=1
Environment=CORECLR_PROFILER_PATH=/opt/appdynamics/dotnet/libappdprofiler.so
Environment=ENABLE_NLOG=true

## Configure connection to the controller
Environment=APPDYNAMICS_CONTROLLER_HOST_NAME=XXXXX.saas.appdynamics.com
Environment=APPDYNAMICS_CONTROLLER_PORT=443
Environment=APPDYNAMICS_CONTROLLER_SSL_ENABLED=true
Environment=APPDYNAMICS_AGENT_ACCOUNT_NAME=XXXX
Environment=APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY=XXXXX

## Configure application identity in AppDynamics
Environment=APPDYNAMICS_AGENT_APPLICATION_NAME=%i
Environment=APPDYNAMICS_AGENT_TIER_NAME=%i
Environment=APPDYNAMICS_AGENT_NODE_NAME=%i.hostname.org


WorkingDirectory=/home/%i
User=%i
PermissionsStartOnly=true
Type=simple
Restart=on-failure
PrivateTmp=1
#Restart=no
ExecStartPre=/usr/sbin/usermod -aG %i msuser
ExecStart=/usr/bin/dotnet /home/%i/%i.dll --urls http://*:${port}
[Install]
WantedBy=multi-user.target

We also must use PrivateTmp cause your agent will create a directory /tmp/appd/dotnet with the files Agent_0.log and Profiler_0.log.
This will only work with the first service, the second service on the machine will fail because it can't write to these files.

This files will also never cleaned up.....

We did the last updates manual and replaced the files in /opt/appdynamics/dotnet. Direct after replace all services died with segmentation faults. Like:

[Mi Mär 15 11:34:24 2023] .NET ThreadPool[64541]: segfault at 1af290001 ip 00000001af290001 sp 00007f67d57f8228 error 14 in dotnet[55f03b783000+c000]
[Mi Mär 15 11:34:24 2023] .NET ThreadPool[100795]: segfault at 1af290001 ip 00000001af290001 sp 00007f263effbe08 error 14 in dotnet[5628f98c1000+c000]
[Mi Mär 15 11:34:25 2023] .NET ThreadPool[101577]: segfault at 1af290001 ip 00000001af290001 sp 00007f25afff5cf8 error 14 in dotnet[55b78076d000+c000]
[Mi Mär 15 11:34:25 2023] .NET ThreadPool[95181]: segfault at 1af290001 ip 00000001af290001 sp 00007fc5ab7e4cf8 error 14 in dotnet[556e68eee000+c000]
[Mi Mär 15 11:34:26 2023] traps: .NET ThreadPool[102722] trap invalid opcode ip:7f20620ca920 sp:7f1ac87ee8f8 error:0 in libappdprofiler_glibc.so[7f2061e7e000+53d000]
[Mi Mär 15 11:34:27 2023] dotnet[104290]: segfault at 8 ip 00007fcac96dc9bc sp 00007fcac5eaab30 error 6 in libcoreclr.so[7fcac959b000+253000]
[Mi Mär 15 11:34:27 2023] dotnet[27438]: segfault at 8 ip 00007f530ff759bc sp 00007f530c743b30 error 6 in libcoreclr.so[7f530fe34000+253000]
[Mi Mär 15 11:34:30 2023] .NET ThreadPool[97466]: segfault at 1af290001 ip 00000001af290001 sp 00007f0037ffde08 error 14 in dotnet[55fe89f3d000+c000]
[Mi Mär 15 11:34:30 2023] dotnet[104640]: segfault at 8 ip 00007f494ca209bc sp 00007f49491eeb30 error 6 in libcoreclr.so[7f494c8df000+253000]
[Mi Mär 15 11:34:31 2023] traps: .NET ThreadPool[80018] trap invalid opcode ip:7fa4c4a50920 sp:7f9f48ff78f8 error:0 in libappdprofiler_glibc.so[7fa4c4804000+53d000]
[Mi Mär 15 11:34:31 2023] dotnet[30441]: segfault at ffffffffffffff8e ip 00007f17bd52a731 sp 00007f17bd292640 error 7 in libappdprofiler_glibc.so[7f17bd294000+53d000]
[Mi Mär 15 11:34:35 2023] dotnet[66715]: segfault at ffffffffffffff8e ip 00007f3b145e0731 sp 00007f3b14348640 error 7 in libappdprofiler_glibc.so[7f3b1434a000+53d000]
[Mi Mär 15 11:34:35 2023] traps: .NET ThreadPool[98449] trap invalid opcode ip:7fbdc80b0920 sp:7fb857ffd528 error:0 in libappdprofiler_glibc.so[7fbdc7e64000+53d000]
[Mi Mär 15 11:34:38 2023] dotnet[72277]: segfault at ffffffffffffff8e ip 00007f9f37c0c731 sp 00007f9f37974640 error 7 in libappdprofiler_glibc.so[7f9f37976000+53d000]
[Mi Mär 15 11:34:40 2023] dotnet[72278]: segfault at ffffffffffffff8e ip 00007f0b281b8731 sp 00007f0b27f20640 error 7 in libappdprofiler_glibc.so[7f0b27f22000+53d000]
[Mi Mär 15 11:34:41 2023] dotnet[91130]: segfault at ffffffffffffff8e ip 00007f1e73a45731 sp 00007f1e737ad640 error 7 in libappdprofiler_glibc.so[7f1e737af000+53d000]
[Mi Mär 15 11:34:53 2023] dotnet[99955]: segfault at 8 ip 00007f776af309bc sp 00007f77676feb30 error 6 in libcoreclr.so[7f776adef000+253000]
[Mi Mär 15 11:34:53 2023] dotnet[107392]: segfault at 8 ip 00007f222587d9bc sp 00007f222204bb30 error 6 in libcoreclr.so[7f222573c000+253000]
[Mi Mär 15 11:34:56 2023] dotnet[100490]: segfault at 8 ip 00007efcc5d849bc sp 00007efcc2552b30 error 6 in libcoreclr.so[7efcc5c43000+253000]
[Mi Mär 15 11:34:57 2023] dotnet[23263]: segfault at 8 ip 00007fcb34b829bc sp 00007fcb31350b30 error 6 in libcoreclr.so[7fcb34a41000+253000]
[Mi Mär 15 11:35:02 2023] dotnet[17298]: segfault at 8 ip 00007f361d4659bc sp 00007f3619c33b30 error 6 in libcoreclr.so[7f361d324000+253000]
[Mi Mär 15 11:35:04 2023] dotnet[11572]: segfault at ffffffffffffff8e ip 00007fd8728aa731 sp 00007fd872612640 error 7 in libappdprofiler_glibc.so[7fd872614000+53d000]
[Mi Mär 15 11:35:11 2023] dotnet[73712]: segfault at ffffffffffffff8e ip 00007f342fe42731 sp 00007f342fbaa640 error 7 in libappdprofiler_glibc.so[7f342fbac000+53d000]
[Mi Mär 15 11:35:14 2023] dotnet[102958]: segfault at ffffffffffffff8e ip 00007fd362903731 sp 00007fd36266b640 error 7 in libappdprofiler_glibc.so[7fd36266d000+53d000]
[Mi Mär 15 11:35:24 2023] dotnet[59844]: segfault at ffffffffffffff8e ip 00007f01da51d731 sp 00007f01da285640 error 7 in libappdprofiler_glibc.so[7f01da287000+53d000]

As far as I under stud your role did the same, it replaces the files in the dotnetcore agent directory.

For me, it looks like that I must take all services on the system out of the loadbalancer and stopp all services before updating the agent and restart all services after update. Or is there a better way?