persistent log for PAI services

[fanyangCS]

What would you like to be added:
persistent log for key PAI services like RM, NameNode, etc.

Why is this needed:
the log will be useful for debug.

Without this feature, how does the current module work：
right now, when a pai service restarts, the log is gone. hard to debug

Components that may involve changes:
PAI key services.

[fanyangCS]

partially resolved by #2244

[squirrelsc]

@scarlett2018 recently, there are some wired issues in OpenPAI, as there is no log, so it's hard to trouble shooting. It should persistent log of all pods near realtime.

[xudifsd]

We have two options:

• centralized log storage like fluentd
• persist to local and define a mechanism to view, maybe nginx services in each node serving log view request

[squirrelsc]

We have two options:

• centralized log storage like fluentd
• persist to local and define a mechanism to view, maybe nginx services in each node serving log view request

It's better to centralize logs. It don't need to deploy Nginx services on host.

[xudifsd]

@squirrelsc they have pros and cons, centralized will need fluentd service deployed on every hosts and elastic search deployed and a lot of space dedicated to the service. IMO, The local host is much more light weighted.

[squirrelsc]

@squirrelsc they have pros and cons, centralized will need fluentd service deployed on every hosts and elastic search deployed and a lot of space dedicated to the service. IMO, The local host is much more light weighted.

we can just put all logs on HDFS, no elasticSearch is needed. If there is any alert, we have pod name, and it can be used to find log files.
For fluentd, how is log files of user container uploaded? Can it be used to upload logs of OpenPAI service?

[fanyangCS]

closed and tracked in #4992.