Non-reproducibility in TriggerResults in 24834.0, 24834.911, and 25034.999

[makortel]

It seems like there is a recent non-reproducibility in the TriggerResults comparisons of the step2 of the workflows 24834.0, 24834.911, and 25034.999 (all of them use D98). Here is an example from #43758 (comment)

Here is another occurrence #43738 (comment)

[makortel]

assign upgrade, hlt

[cmsbuild]

New categories assigned: upgrade,hlt

@Martin-Grunewald,@mmusich,@srimanob,@subirsarkar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel Matti Kortelainen.

@antoniovilela, @smuzaffar, @rappoccio, @makortel, @sextonkennedy, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

Visible e.g. in

• CMSSW_14_0_X_2024-01-24-1100
  • Revert back to chi-square fit instead of likelihood fit #43782 (comment)
  • Added signed GSF track variable collections to the HLT producer: 14_0 #43774 (comment)
• CMSSW_14_0_X_2024-01-23-2300
  • Split testConfigDP into smaller parts #43776 (comment)

but not in CMSSW_14_0_X_2024-01-22-1100 (#43737 (comment))

[mmusich]

Given the pattern of merges between CMSSW_14_0_X_2024-01-22-100 (not visible) and CMSSW_14_0_X_2024-01-23-2300 (visible), see the gitHub diff of commits merged, it looks plausible that the non-reproducibility could come from #43738 (merged in CMSSW_14_0_X_2024-01-23-2300). Incidentally it was also noticed in the PR tests themselves: #43738 (comment).
@SohamBhattacharya @rovere FYI.

[makortel]

Is anyone looking into this non-reproducibility?

[dan131riley]

Possibly related to #43723? The fix for that issue in #43852 was just merged, so will be interesting to see if it improves the reproducibility.

[makortel]

Possibly related to #43723? The fix for that issue in #43852 was just merged, so will be interesting to see if it improves the reproducibility.

The random differences are still there.

[makortel]

Is anyone looking into this non-reproducibility?

[mmusich]

Is anyone looking into this non-reproducibility?

while I don't think anyone looked at this explicitly, I don't seem to be able to find recent instances (at least since ~ August 2024).
Perhaps it got fixed in one of the recent phase-2 HLT menus updates?