Log all services to memory

[felipecrs]

• All logs can be accessed from /dev/shm/logs/<service>, e.g. /dev/shm/logs/go2rtc/current.
• NGINX now logs to stdout as well (like in Logs in UI #4562)
• Frigate service isn't set in CMD anymore as this allows us to log it as well.
  • S6-Overlay does not seem to support customizing logging for CMD
• Refs Logs in UI #4562 (comment)
• Is depended on by Logs in UI #4562

[netlify]

✅ Deploy Preview for frigate-docs canceled.

Name	Link
🔨 Latest commit	40046e9
🔍 Latest deploy log	https://app.netlify.com/sites/frigate-docs/deploys/638e652983476f0008aa14f2

[NickM-27]

Nice! That should be good

[felipecrs]

@NickM-27 the current behavior, which mimics s6-overlay defaults, is to rotate the current log every time it reaches 1MB, and keep at most 20 archive files.

As we are logging to memory, we should perhaps consider decreasing the number of archives. Perhaps 1 is enough?

https://skarnet.org/software/s6/s6-log.html

[NickM-27]

Yeah I agree that's better archive, might be worth increasing the size a bit to 10MB or so so there's a decent history for users to view.

[felipecrs]

Ok, I also agree.

[felipecrs]

Done.

[felipecrs]

On my tests, this is working. If you could give it a spin as well before merging, it would be nice.

[NickM-27]

On my tests, this is working. If you could give it a spin as well before merging, it would be nice.

Sure, I can do it tomorrow night or Monday morning, I'm out right now

[felipecrs]

There's another thing to take into consideration here. Since Frigate isn't run as CMD anymore, s6-overlay now apply some more generic rules to it. Most importantly:

• S6_SERVICES_GRACETIME (default = 3000): How long (in milliseconds) s6 should wait,
  at shutdown time, for services declared in /etc/services.d to die before proceeding
  with the rest of the shutdown.
• S6_KILL_GRACETIME (default = 3000): How long (in milliseconds) s6 should wait, at the end of
  the shutdown procedure when all the processes have received a TERM signal, for them to die
  before sending a KILL signal to make sure they're dead.

So, do you think 3 seconds is enough for the Frigate process to exit gracefully or should we give it more time?

[NickM-27]

I'd say definitely give it more time, don't want it to get killed mid storage maintenance or something.

[felipecrs]

Ok, I think there's a bug in s6-overlay which will affect us:

• s6-overlay exits as soon as the first service exits despite S6_SERVICES_GRACETIME just-containers/s6-overlay#503

Let's wait for the closure of it before merging this PR.

[NickM-27]

Ok, I think there's a bug in s6-overlay which will affect us:

• S6_SERVICES_GRACETIME exits as soon as the first service exits just-containers/s6-overlay#503

Let's wait for the closure of it before merging this PR.

is it this PR that causes that issue or the update to s6 v3?

[felipecrs]

That's a good question. I got to test.

My assumption is that it's this PR, since I'm switching Frigate from CMD to a service.

[felipecrs]

I tested here, the same issue happens when using CMD (which means this PR is NOT introducing this issue). Now I'll test now with S6 v2.

[felipecrs]

Ok, same issue with s6-overlay v2 which means it was always there. This may actually be the reason for multiple random database corruptions that we found people reporting (including me).

Therefore, this PR can be merged as it brings no additional issues.

[NickM-27]

how does this work with the dev container? are we still able to start / stop frigate python process as expected and it isn't started automatically?

[felipecrs]

@NickM-27 Nothing should change for the devcontainer because of:

https://github.com/blakeblackshear/frigate/pull/4587/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R195-R196

[NickM-27]

Thanks @felipecrs

I'm seeing an error, when I try to view the contents of the log files from the frigate python process it does not have appropriate permissions, the python process will need to be able to read the log file contents.

[felipecrs]

Hm... in the devcontainer Python runs as vscode. Maybe that's why. But I think it was supposed to have access anyway. I will do some testing tonight, but if you identify some fix feel free to amend this PR.

[NickM-27]

Hm... in the devcontainer Python runs as vscode. Maybe that's why. But I think it was supposed to have access anyway. I will do some testing tonight, but if you identify some fix feel free to amend this PR.

that's a fair point, I haven't tested in a full container if python will have access

[felipecrs]

Another thing, have in mind that, since the Frigate process is not being supervised in the devcontainer, there won't be any logs for it.

Perhaps we should have a fake Frigate service running in the devcontainer just printing some dummy text, so that the log files gets actually created. What do you think?

[NickM-27]

Another thing, have in mind that, since the Frigate process is not being supervised in the devcontainer, there won't be any logs for it.

Perhaps we should have a fake Frigate service running in the devcontainer just printing some dummy text, so that the log files gets actually created. What do you think?

Yeah I am aware of that, I'm not sure it's necessary since the logs will be visible in the console (so it isn't needed) and the logs being gone won't cause any real issues. Happy to hear other's opinions though

[felipecrs]

@NickM-27 check my last commit. It should fix all issues now.

[NickM-27]

Thanks that works great

[NickM-27]

@felipecrs I don't think frigate is being started with env variables with this PR

or at least the frigate config can not see any FRIGATE_... variables

[NickM-27]

When I run printenv in the container I can see the env variables, but it seems that inside the frigate python process it is unable to see the variables

[felipecrs]

Thanks. Should work now.