3.6.8.29

[PalNilsson]

• Redirecting stdout/stderr to files for trace service curls
  • This could prevent thread deadlocks in the standard python subprocess.communicate() function in case of overwhelming amount of stdout/stderr. The subprocess.communicate() function is no longer used, which also means that the internal timeout capability in subprocess can no longer be used and had to be reimplemented by a threading timer which sets the relevant error code if necessary
  • The 'last’ output from curl is stored in trace_curl_last.stdout/stderr, and gets appended to trace_curl.stdout/stderr
  • The trace_curl_last.stdout/stderr files are searched for any curl errors (curl command always returns 0 exit code even when there was an error, so the output has to be processed)
  • A failed rucio trace curl operation is now reported with job metrics
    • Example: rucioTraceError=N
  • Increased connection timeout from 20s to 100s to be in line with panda server curl operations (where we don’t see any problems)
  • Related JIRA ticket: https://its.cern.ch/jira/browse/ATLASPANDA-835
• Reporting prmon read_bytes/total_input_size with job metrics (‘readbyterate’)
  • Information to be used for optimizing brokerage
  • Requested by J. Elmsheuser, R. Walker
• Extended usage of psutil
  • Job monitoring is now using psutil to discover prmon pid
  • If psutil is not available (e.g. as is the case on marenostrum), the code falls back to old ps command usage
• Added protection against expired job objects in job_monitor loop
  • Reported by W. Guan/Z. Yang
• Updated GitHub Action workflows
  • Unit tests and flake8 are are now independent workflows
  • Moved to latest flake8 version 6.1.0 for flake8 verification
  • All tests are run for python versions 3.8, 3.9, 3.10 and 3.11
• Tested pilot running under python versions 3.9.18 and 3.11.5
  • Grid jobs are currently running under python version 3.9.14 but will soon switch to 3.9.18 to be in line with user tools (like rucio)
  • Python version 3.11.5 will be the default version on EL9
  • Requested by A. De Silva