simplify micro arch selection; set default micro arch to x86-64-v2 #9166

smuzaffar · 2024-04-29T05:40:20Z

This PR simplify the selection of default micro archtitecture which is now defined in cmsdist/microarch_flags.file file. Every package which needs to be enable micro-instraction should include it and pass %{selected_microarch} to the its build system.

This also updates the default micro arch to x86-64-v2 ( previously it was sse3)

cmsbuild · 2024-04-29T05:40:45Z

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_14_1_X/master.

@aandvalenzuela, @cmsbuild, @smuzaffar, @iarspider can you please review it and eventually sign? Thanks.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2024-04-29T05:40:46Z

cms-bot internal usage

cmsbuild · 2024-04-29T05:42:15Z

Pull request #9166 was updated.

smuzaffar · 2024-04-29T06:20:01Z

test parameters:

full_cmssw = true

smuzaffar · 2024-04-29T06:20:23Z

please test for CMSSW_14_1_MULTIARCHS_X/el8_amd64_gcc12

cmsbuild · 2024-04-29T07:04:21Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39135/summary.html
COMMIT: 56d942a
CMSSW: CMSSW_14_1_MULTIARCHS_X_2024-04-28-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9166/39135/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ CFLAGS='-fPIC -O3 -DUSE_MMAP -DUNALIGNED_OK -D_LARGEFILE64_SOURCE=1 -mx86-64-v2'
+ ./configure --prefix=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/84956533166d4eef7141f08703457f8b/opt/cmssw/el8_amd64_gcc12/external/zlib/1.2.11-84956533166d4eef7141f08703457f8b
Checking for gcc...
Compiler error reporting is too harsh for ./configure (perhaps remove -Werror).
** ./configure aborting.
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.wuboyp (%build)


RPM build errors:
line 36: It's not recommended to have unversioned Obsoletes: Obsoletes: external+zlib+1.2.11-84956533166d4eef7141f08703457f8b
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.wuboyp (%build)

smuzaffar · 2024-04-29T07:19:29Z

please test for CMSSW_14_1_MULTIARCHS_X/el8_amd64_gcc12

cmsbuild · 2024-04-29T07:19:38Z

Pull request #9166 was updated.

smuzaffar · 2024-04-29T19:03:14Z

please test for CMSSW_14_1_MULTIARCHS_X/el8_amd64_gcc12

smuzaffar · 2024-04-30T19:58:17Z

please test for CMSSW_14_1_MULTIARCHS_X/el8_amd64_gcc12

cmsbuild · 2024-04-30T19:58:25Z

Pull request #9166 was updated.

smuzaffar · 2024-05-01T08:23:02Z

please test for CMSSW_14_1_MULTIARCHS_X/el8_amd64_gcc12

cmsbuild · 2024-05-01T10:06:53Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39193/summary.html
COMMIT: d32a009
CMSSW: CMSSW_14_1_MULTIARCHS_X_2024-04-30-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9166/39193/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ perl -p -i -e 's|\@([a-zA-Z0-9_-]*)\@|$ENV{$1}|g' /pool/condor/dir_2920405/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/190a2a634f77e746b58b37be4193442d/opt/cmssw/el8_amd64_gcc12/cms/coral-tool-conf/10.0-190a2a634f77e746b58b37be4193442d/tools/selected/x11.xml
+ echo '  Copied x11.xml'
Copied x11.xml
+ '[' -d /etc/scram.d ']'
/pool/condor/dir_2920405/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Rm5yHV: line 62: syntax error near unexpected token `<<<'
error: Bad exit status from /pool/condor/dir_2920405/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Rm5yHV (%install)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: cms+coral-tool-conf+10.0-190a2a634f77e746b58b37be4193442d
Bad exit status from /pool/condor/dir_2920405/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Rm5yHV (%install)

smuzaffar · 2024-05-01T18:05:07Z

please test for CMSSW_14_1_MULTIARCHS_X/el8_amd64_gcc12

smuzaffar · 2024-05-02T08:45:43Z

please test

cmsbuild · 2024-05-02T17:10:15Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39197/summary.html
COMMIT: b7bf40b
CMSSW: CMSSW_14_1_MULTIARCHS_X_2024-04-30-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9166/39197/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Fix unittest link flags for extra micro-arch #9170

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39197/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39197/git-merge-result

Comparison Summary

Summary:

You potentially removed 274 lines from the logs
Reco comparison results: 70759 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3326929
DQMHistoTests: Total failures: 305847
DQMHistoTests: Total nulls: 268
DQMHistoTests: Total successes: 3020794
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -2.0850000000000004 KiB( 47 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.586 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 13034.0 ): -2.906 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 140.043,... ): -0.016 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 140.063 ): -0.070 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.044 ): 0.086 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.046 ): 0.016 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 250202.181 ): -0.836 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.469 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 7.3 ): 1.774 KiB SiStrip/MechanicalView
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: found differences in 18 / 46 workflows

cmsbuild · 2024-05-02T18:36:13Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39204/summary.html
COMMIT: b7bf40b
CMSSW: CMSSW_14_1_X_2024-05-01-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9166/39204/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 241 lines from the logs
Reco comparison results: 2151 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3331548
DQMHistoTests: Total failures: 1979
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3329549
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

smuzaffar · 2024-05-02T19:28:20Z

please test

cmsbuild · 2024-05-03T04:08:54Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39220/summary.html
COMMIT: b7bf40b
CMSSW: CMSSW_14_1_X_2024-05-02-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9166/39220/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39220/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39220/git-merge-result

Comparison Summary

Summary:

You potentially removed 226 lines from the logs
Reco comparison results: 2145 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3331548
DQMHistoTests: Total failures: 1991
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3329537
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

smuzaffar · 2024-05-03T07:17:41Z

please test for slc7_amd64_gcc12

smuzaffar · 2024-05-03T07:18:10Z

please test for el9_amd64_gcc12

cmsbuild · 2024-05-04T07:58:47Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39224/summary.html
COMMIT: b7bf40b
CMSSW: CMSSW_14_1_X_2024-05-02-2300/el9_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9166/39224/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39224/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39224/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test testHGCalMapFileParser had ERRORS

Comparison Summary

Summary:

You potentially removed 217 lines from the logs
Reco comparison results: 56443 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3331548
DQMHistoTests: Total failures: 146376
DQMHistoTests: Total nulls: 392
DQMHistoTests: Total successes: 3184760
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1007.6629999999999 KiB( 47 files compared)
DQMHistoSizes: changed ( 10224.0,... ): 34.812 KiB CTPPS/TimingDiamond
DQMHistoSizes: changed ( 10224.0 ): -0.903 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 11634.0,... ): 69.500 KiB CTPPS/TimingDiamond
DQMHistoSizes: changed ( 13034.0 ): 0.862 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 141.044 ): -0.012 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.046 ): 0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 250202.181 ): -0.288 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.064 KiB SiStrip/MechanicalView
Checked 202 log files, 165 edm output root files, 48 DQM output files

cmsbuild · 2024-05-04T09:05:13Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39223/summary.html
COMMIT: b7bf40b
CMSSW: CMSSW_14_1_X_2024-05-02-2300/slc7_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9166/39223/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39223/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aa37cc/39223/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test testHGCalMapFileParser had ERRORS

Comparison Summary

Summary:

You potentially removed 148 lines from the logs
Reco comparison results: 62220 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3331548
DQMHistoTests: Total failures: 222502
DQMHistoTests: Total nulls: 411
DQMHistoTests: Total successes: 3108615
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1010.516 KiB( 47 files compared)
DQMHistoSizes: changed ( 10224.0,... ): 34.812 KiB CTPPS/TimingDiamond
DQMHistoSizes: changed ( 10224.0 ): 0.308 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 11634.0,... ): 69.500 KiB CTPPS/TimingDiamond
DQMHistoSizes: changed ( 13034.0 ): 2.381 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 141.042 ): 0.023 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.044 ): -0.023 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.046 ): 0.008 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 250202.181 ): -0.182 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.064 KiB SiStrip/MechanicalView
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: found differences in 14 / 46 workflows

smuzaffar · 2024-05-08T07:31:44Z

+externals

looks good, tested via #9156 for el8/el9 and slc7

cmsbuild · 2024-05-08T07:32:06Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_14_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @sextonkennedy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels Apr 29, 2024

simplify micro arch selection; set default micro arch to x86-64-v2

56d942a

smuzaffar force-pushed the microarch-x86-64-v2 branch from 2293273 to 56d942a Compare April 29, 2024 05:41

cmsbuild added tests-started and removed tests-pending labels Apr 29, 2024

cmsbuild added tests-rejected and removed tests-started labels Apr 29, 2024

Update microarch_flags.file: fix default microarch flag

d32a009

cmsbuild added tests-started and removed tests-rejected labels Apr 29, 2024

smuzaffar mentioned this pull request Apr 29, 2024

Upload and reuse packages from the global object store cms-sw/pkgtools#206

Merged

cmsbuild added the requires-external label Apr 29, 2024

smuzaffar force-pushed the microarch-x86-64-v2 branch from d649c04 to d32a009 Compare May 1, 2024 08:22

cmsbuild removed the tests-started label May 1, 2024

cmsbuild added tests-started and removed tests-pending labels May 1, 2024

cmsbuild removed the requires-external label May 2, 2024

cmsbuild added tests-approved and removed tests-started labels May 2, 2024

cmsbuild added tests-started and removed tests-approved labels May 2, 2024

cmsbuild added tests-approved and removed tests-started labels May 3, 2024

smuzaffar mentioned this pull request May 4, 2024

Update several py3 packages following dependabot alerts #9156

Merged

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels May 8, 2024

smuzaffar merged commit 9d01795 into IB/CMSSW_14_1_X/master May 8, 2024
26 of 28 checks passed

smuzaffar deleted the microarch-x86-64-v2 branch May 8, 2024 22:54

cmsbuild mentioned this pull request May 9, 2024

Update of Phase-2 GMT Emulator cms-sw/cmssw#44498

Merged

smuzaffar mentioned this pull request Sep 24, 2024

[TF] Do not use mcpu/mtune native if CMS arch_build_flags are not set #9434

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

simplify micro arch selection; set default micro arch to x86-64-v2 #9166

simplify micro arch selection; set default micro arch to x86-64-v2 #9166

smuzaffar commented Apr 29, 2024 •

edited

Loading

cmsbuild commented Apr 29, 2024

cmsbuild commented Apr 29, 2024 •

edited

Loading

cmsbuild commented Apr 29, 2024

smuzaffar commented Apr 29, 2024 •

edited

Loading

smuzaffar commented Apr 29, 2024

cmsbuild commented Apr 29, 2024

smuzaffar commented Apr 29, 2024

cmsbuild commented Apr 29, 2024

smuzaffar commented Apr 29, 2024

smuzaffar commented Apr 30, 2024

cmsbuild commented Apr 30, 2024

smuzaffar commented May 1, 2024

cmsbuild commented May 1, 2024

smuzaffar commented May 1, 2024

smuzaffar commented May 2, 2024

cmsbuild commented May 2, 2024

cmsbuild commented May 2, 2024

smuzaffar commented May 2, 2024

cmsbuild commented May 3, 2024

smuzaffar commented May 3, 2024

smuzaffar commented May 3, 2024

cmsbuild commented May 4, 2024

cmsbuild commented May 4, 2024

smuzaffar commented May 8, 2024

cmsbuild commented May 8, 2024

simplify micro arch selection; set default micro arch to x86-64-v2 #9166

simplify micro arch selection; set default micro arch to x86-64-v2 #9166

Conversation

smuzaffar commented Apr 29, 2024 • edited Loading

cmsbuild commented Apr 29, 2024

cmsbuild commented Apr 29, 2024 • edited Loading

cmsbuild commented Apr 29, 2024

smuzaffar commented Apr 29, 2024 • edited Loading

smuzaffar commented Apr 29, 2024

cmsbuild commented Apr 29, 2024

External Build

smuzaffar commented Apr 29, 2024

cmsbuild commented Apr 29, 2024

smuzaffar commented Apr 29, 2024

smuzaffar commented Apr 30, 2024

cmsbuild commented Apr 30, 2024

smuzaffar commented May 1, 2024

cmsbuild commented May 1, 2024

External Build

smuzaffar commented May 1, 2024

smuzaffar commented May 2, 2024

cmsbuild commented May 2, 2024

Comparison Summary

cmsbuild commented May 2, 2024

Comparison Summary

smuzaffar commented May 2, 2024

cmsbuild commented May 3, 2024

Comparison Summary

smuzaffar commented May 3, 2024

smuzaffar commented May 3, 2024

cmsbuild commented May 4, 2024

Unit Tests

Comparison Summary

cmsbuild commented May 4, 2024

Unit Tests

Comparison Summary

smuzaffar commented May 8, 2024

cmsbuild commented May 8, 2024

smuzaffar commented Apr 29, 2024 •

edited

Loading

cmsbuild commented Apr 29, 2024 •

edited

Loading

smuzaffar commented Apr 29, 2024 •

edited

Loading