Add --maxmem_profile to all runTheMatrix workflows to log mamimum memory allocated

[gartung]

addresses cms-sw/framework-team#829
requires cms-sw/cmssw#44001

[cmsbuild]

A new Pull Request was created by @gartung for branch master.

@cmsbuild, @aandvalenzuela, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[gartung]

@makortel

[makortel]

addresses makortel/framework#673

(this PR is more about cms-sw/framework-team#829)

[makortel]

I think there are some test flavors for which this monitoring would be harmful (like UBSAN, ASAN, threading etc). Perhaps the best way to get started would be to limit to the default testing set on x86 (we could then expand after gaining more experience).

[cmsbuild]

Pull request #2185 was updated.

[cmsbuild]

REMINDER @rappoccio, @sextonkennedy, @antoniovilela: This PR was tested with cms-sw/cmssw#44001, please check if they should be merged together

[antoniovilela]

cms-sw/cmssw#44001 has been merged.

[makortel]

Maybe we wait for Shahzad to get back to decide how exactly to continue (unless @aandvalenzuela @iarspider would know on the top of their head what exactly would need to be done to achieve #2185 (comment)).

[smuzaffar]

@gartung , is --maxmem_profile supported by all cmssw release cycles? If not then I would suggest to run runTheMatrix.py --help to see if this opt is supported or not. I n case a cmssw release cycle does not support it then just mark the commit status successfull with message Memory profile not run due to lack of support

[makortel]

is --maxmem_profile supported by all cmssw release cycles?

It is not. The option was added in 14_1_X by cms-sw/cmssw#44001 .

Patrick is on a leave, so someone else would have to continue from here. I could do that, but I'd need guidance how to achieve #2185 (comment) .

[smuzaffar]

So remind me what we want to do with this change? from the change it looks like we want to run extra relvals with --maxmem_profile option e.g. one can say enable maxmem_profile and then bot should run normal relval + all normal relvals with --maxmem_profile .. right?

[makortel]

I want all normal relval on x86 (only) to be run with --maxmem_profile (i.e. no additional "test flavor"). Any extra workflows could include the --maxmem_profile too, but for other test flavors (non-x86 architectures, different IB flavor, extra tests) for now.

[smuzaffar]

@gartung @makortel I have opened #2202 to enable memory profiling for default workflows for production IB/arch only

[makortel]

I think we can close this PR in favor of #2202

[makortel]

@cmsbuild, please close