[Performance] Investigate performance regression caused by Onyx Connection Manager

[fabioh8010]

It seems that the connection manager caused a performance regression during E2E tests, specifically the Open Chat Finder Page TTI test. Although it's a minor difference (~10%) we need to investigate what can be the cause of the regression and look for a fix (E/App, Onyx or both).

Context: https://expensify.slack.com/archives/C05LX9D6E07/p1724690575748209

[melvin-bot]

Triggered auto assignment to @muttmuure (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[fabioh8010]

Update: Started investigation but not entirely focused on this at the moment, I will have something more concrete next week.

[mountiny]

Making this weekly as this is a valid regression that was noted by the E2E but due to the nature of the onyx changes its hard to identify the root cause, we will most likely have to analyze traces. Additionally it might be that some concurrent improvements already negated the slowdown

[fabioh8010]

Update: Still focused on other tasks with more priority at the moment.

[melvin-bot]

@fabioh8010 @mountiny @muttmuure this issue was created 2 weeks ago. Are we close to a solution? Let's make sure we're treating this as a top priority. Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

[mountiny]

@fabioh8010 have you been able to find anything promising?

[fabioh8010]

Hey @mountiny Sorry I couldn't go back to this last week, I will focus on it this week

[fabioh8010]

Update: Resuming investigations on this issue today.

[fabioh8010]

I decided to run again the E2E tests where the baseline is the main branch using an Onyx version prior to the Connection Manager (v2.0.64 as it was introduced in v2.0.65), and the delta is the main branch using an Onyx version after the Connection Manager (v2.0.67, I decided to not use v2.0.68 as it includes a change that could potentially interfere with the results).

Here's the results of all E2E tests done with 60 runs (the same amount we do in the CI):

❇️ Performance comparison results:

➡️ Significant changes to duration

➡️ Meaningless changes to duration

• App start nativeLaunch: 64.482 ms → 63.155 ms (-1.327 ms, -2.1%)
• App start nativeLaunchEnd_To_appCreationStart: 116.467 ms → 117.949 ms (+1.482 ms, +1.3%)
• App start appCreation: 134.780 ms → 131.862 ms (-2.918 ms, -2.2%)
• App start contentAppeared_To_screenTTI: 635.627 ms → 671.438 ms (+35.811 ms, +5.6%)
• App start runJsBundle: 279.898 ms → 277.431 ms (-2.467 ms, -0.9%)
• App start regularAppStart: 0.012 ms → 0.011 ms (-0.000 ms, -3.8%)
• App start TTI: 1478.303 ms → 1460.940 ms (-17.363 ms, -1.2%)
• App start appCreationEnd_To_contentAppeared: 528.250 ms → 490.383 ms (-37.867 ms, -7.2%)
• App start time (CPU): 97.044 % → 98.512 % (+1.468 %, +1.5%)
• App start time (FPS): 52.897 FPS → 52.240 FPS (-0.657 FPS, +1.2%)
• App start time (RAM): 337.577 MB → 339.487 MB (+1.910 MB, +0.6%)
• App start time (CPU/JS): 52.320 % → 50.991 % (-1.329 %, -2.5%)
• App start time (CPU/UI): 25.342 % → 26.190 % (+0.848 %, +3.3%)

---

• Load Search Options: 56.050 ms → 55.712 ms (-0.339 ms, -0.6%)
• Open Chat Finder Page TTI: 686.347 ms → 714.931 ms (+28.584 ms, +4.2%)
• Open chat finder page TTI (CPU): 87.141 % → 87.219 % (+0.079 %, ±0.0%)
• Open chat finder page TTI (FPS): 54.954 FPS → 54.702 FPS (-0.252 FPS, ±0.0%)
• Open chat finder page TTI (RAM): 363.076 MB → 363.774 MB (+0.698 MB, ±0.0%)
• Open chat finder page TTI (CPU/JS): 40.467 % → 39.023 % (-1.443 %, -3.6%)
• Open chat finder page TTI (CPU/UI): 20.835 % → 21.120 % (+0.285 %, +1.4%)

---

• Composer typing rerender count: 1.000 renders → 1.000 renders
• Message sent: 264.658 ms → 262.824 ms (-1.834 ms, -0.7%)
• Report typing (CPU): 55.025 % → 54.164 % (-0.861 %, -1.6%)
• Report typing (FPS): 57.733 FPS → 57.972 FPS (+0.239 FPS, ±0.0%)
• Report typing (RAM): 344.565 MB → 341.349 MB (-3.216 MB, -0.9%)
• Report typing (CPU/JS): 27.500 % → 26.257 % (-1.243 %, -4.5%)
• Report typing (CPU/UI): 14.662 % → 15.037 % (+0.375 %, +2.6%)

---

• Chat opening: 76.156 ms → 84.754 ms (+8.598 ms, +11.3%)
• Chat TTI: 235.453 ms → 246.231 ms (+10.777 ms, +4.6%)
• Chat opening (CPU): 89.242 % → 88.766 % (-0.476 %, -0.5%)
• Chat opening (FPS): 51.857 FPS → 51.657 FPS (-0.200 FPS, ±0.0%)
• Chat opening (RAM): 324.835 MB → 322.705 MB (-2.130 MB, -0.7%)
• Chat opening (CPU/JS): 40.969 % → 40.454 % (-0.515 %, -1.3%)
• Chat opening (CPU/UI): 22.955 % → 23.317 % (+0.362 %, +1.6%)

---

• Comment linking: 126.162 ms → 131.498 ms (+5.336 ms, +4.2%)
• Linking (CPU): 104.760 % → 102.541 % (-2.219 %, -2.1%)
• Linking (FPS): 53.579 FPS → 53.281 FPS (-0.299 FPS, +0.6%)
• Linking (RAM): 337.579 MB → 329.615 MB (-7.964 MB, -2.4%)
• Linking (CPU/JS): 55.760 % → 50.975 % (-4.785 %, -8.6%)
• Linking (CPU/UI): 22.498 % → 23.517 % (+1.018 %, +4.5%)

---

As you can see we have slightly increases and decreases in different parts of the tests, all of them marked as insignificant ones.

These are my findings for now, but we can have an idea that the connection manager didn't impact greatly the open chat TTI as we first thought.

[mountiny]

@fabioh8010 thanks for the follow up, to confirm, you are still digging into the open chat tti

[fabioh8010]

@mountiny Yes I'm still going to analyse the flow separately 👍🏻 I'm just giving more priority to other tasks I have at the moment

[mountiny]

no problem at all, just wanted to make sure we are on the same page 🙌

[fabioh8010]

Update: Still on hold while I have other tasks with higher priority

[mountiny]

@fabioh8010 any updates?

If not, I think we can close this one and just focus on making any other improvements based on active traces

[fabioh8010]

@mountiny Yes I think it's better for now, I have been quite unavailable to work on this as I have been fixing Onyx bugs that we are discovering during useOnyx migration. If we have other significant regressions we can re-open this.