[develop] Log the process ID of the currently running Slurm Resume Program

[EddyMM]

Description of changes

• The Slurm Resume Program outputs logs in the following format:
  %(asctime)s - [%(name)s:%(funcName)s] - %(levelname)s - %(message)s
• Due to the possibility of multiple Resume Programs running concurrently, we want to include the Process ID of the currently running Slurm Resume Program.
• The following format will be used to include the Process ID
  %(asctime)s - %(process)d - [%(name)s:%(funcName)s] - %(levelname)s - %(message)s

Sample Logs:

2023-10-24 09:27:24,339 - 27807 - [slurm_plugin.resume:main] - INFO - ResumeProgram startup.
2023-10-24 09:27:24,339 - 27807 - [slurm_plugin.resume:_get_config] - INFO - Reading /etc/parallelcluster/slurm_plugin/parallelcluster_slurm_resume.conf
2023-10-24 09:27:24,341 - 27807 - [slurm_plugin.resume:main] - INFO - ResumeProgram config: SlurmResumeConfig(...)
2023-10-24 09:27:24,341 - 27807 - [slurm_plugin.resume:_get_slurm_resume] - INFO - Slurm Resume File content: {...}
2023-10-24 09:27:24,345 - 27807 - [slurm_plugin.common:is_clustermgtd_heartbeat_valid] - INFO - Latest heartbeat from clustermgtd: 2023-10-24 09:27:15.503146+00:00
2023-10-24 09:27:24,345 - 27807 - [slurm_plugin.resume:_resume] - INFO - Launching EC2 instances for the following Slurm nodes: queue-0-dy-compute-resource-0-[1-10]
2023-10-24 09:27:24,361 - 27807 - [slurm_plugin.resume:_resume] - INFO - Current state of Slurm nodes to resume: [('queue-0-dy-compute-resource-0-1', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-2', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-3', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-4', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-5', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-6', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-7', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-8', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-9', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-10', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP')]
2023-10-24 09:27:24,387 - 27807 - [botocore.credentials:load] - INFO - Found credentials from IAM Role: ...
2023-10-24 09:27:24,428 - 27807 - [slurm_plugin.instance_manager:_launch_instances] - INFO - Launching best-effort instances for nodes (x10) ['queue-0-dy-compute-resource-0-1', 'queue-0-dy-compute-resource-0-2', 'queue-0-dy-compute-resource-0-3', 'queue-0-dy-compute-resource-0-4', 'queue-0-dy-compute-resource-0-5', 'queue-0-dy-compute-resource-0-6', 'queue-0-dy-compute-resource-0-7', 'queue-0-dy-compute-resource-0-8', 'queue-0-dy-compute-resource-0-9', 'queue-0-dy-compute-resource-0-10']
2023-10-24 09:27:24,428 - 27807 - [slurm_plugin.fleet_manager:create_fleet] - INFO - Launching instances with create_fleet API. Parameters: {...}

...

2023-10-24 09:27:24,345 - 27807 - [slurm_plugin.common:is_clustermgtd_heartbeat_valid] - INFO - Latest heartbeat from clustermgtd: 2023-10-24 09:27:15.503146+00:00
2023-10-24 09:27:24,345 - 27807 - [slurm_plugin.resume:_resume] - INFO - Launching EC2 instances for the following Slurm nodes: queue-0-dy-compute-resource-0-[1-10]
2023-10-24 09:27:24,361 - 27807 - [slurm_plugin.resume:_resume] - INFO - Current state of Slurm nodes to resume: [('queue-0-dy-compute-resource-0-1', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-2', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-3', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-4', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-5', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-6', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-7', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-8', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-9', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP'), ('queue-0-dy-compute-resource-0-10', 'ALLOCATED+CLOUD+NOT_RESPONDING+POWERING_UP')]
2023-10-24 09:27:24,387 - 27807 - [botocore.credentials:load] - INFO - Found credentials from IAM Role: ...
2023-10-24 09:27:24,428 - 27807 - [slurm_plugin.instance_manager:_launch_instances] - INFO - Launching best-effort instances for nodes (x10) ['queue-0-dy-compute-resource-0-1', 'queue-0-dy-compute-resource-0-2', 'queue-0-dy-compute-resource-0-3', 'queue-0-dy-compute-resource-0-4', 'queue-0-dy-compute-resource-0-5', 'queue-0-dy-compute-resource-0-6', 'queue-0-dy-compute-resource-0-7', 'queue-0-dy-compute-resource-0-8', 'queue-0-dy-compute-resource-0-9', 'queue-0-dy-compute-resource-0-10']
2023-10-24 09:27:24,428 - 27807 - [slurm_plugin.fleet_manager:create_fleet] - INFO - Launching instances with create_fleet API. Parameters: {...}
2023-10-24 09:27:28,037 - 27807 - [slurm_plugin.fleet_manager:launch_ec2_instances] - INFO - Launched the following instances (x10) [...]

...

2023-10-24 09:27:28,637 - 27807 - [slurm_plugin.resume:main] - INFO - ResumeProgram finished.

Tests

• Manually tested by submitting jobs and inspecting the slurm resume logs

References

• LogRecord attributes / %(process)d

Checklist

• Make sure you are pointing to the right branch.
• If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (9385fc6) 90.04% compared to head (d2dee33) 90.04%.

❗ Current head d2dee33 differs from pull request most recent head 1e4bf4e. Consider uploading reports for the commit 1e4bf4e to get more accurate results

Additional details and impacted files

@@           Coverage Diff            @@
##           develop     #583   +/-   ##
========================================
  Coverage    90.04%   90.04%           
========================================
  Files           16       16           
  Lines         2703     2703           
========================================
  Hits          2434     2434           
  Misses         269      269           

Flag	Coverage Δ
unittests	90.04% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
src/slurm_plugin/resume.py	80.99% <ø> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.