e2e: graceful envoy shutdown

[guydc]

What this PR does / why we need it:

• Update the upgrade suite to use its own namespace, gatewayclass, envoyproxy, gateway and backend.
  • The new gatewayclass will have 2 replicas and 15s drain shutdown, which are required for this test to succeed. Increasing the replica count and shutdown drain time for existing gateways causes certain tests like local rate limit to fail and prolongs the suite.
  • The need for another suite is explained in e2e: add e2e test for MergeGateways feature #2665, as only a single gatewayclass is currently supported in conformance suites
• Adds a test that rolls out the proxy deployment while running load:
  • Restart is achieved by patching EnvoyProxy resource with a custom restart annotation.
  • The shutdown manager should ensure that connections are drained gracefully and not closed forcibly. New connections are gradually loadbalanced to the active proxy envoy instance.
• Small fixes to existing backend upgrade test and moving common parts to utils
• Minor adjustments of the EG upgrade test to the new suite's resources

Future improvements:

• Leverage the HA EnvoyProxy with Shutdown manager as part of the EG upgrade test, and check for hitless upgrades.
• Move the backend upgrade test into this suite, and restart the upgrade "infra-backend". The current test restarts a common backend, in a way that can affect other tests in the suite.
• When all limitations are resolved, we can consider merging this suite back into the main suite.

Which issue(s) this PR fixes:
Fixes #2610

[guydc]

/retest

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 64.45%. Comparing base (b5f4a07) to head (32fea88).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2839   +/-   ##
=======================================
  Coverage   64.45%   64.45%           
=======================================
  Files         122      122           
  Lines       21039    21039           
=======================================
  Hits        13561    13561           
  Misses       6630     6630           
  Partials      848      848           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[guydc]

/retest

[guydc]

blocked on: #2665 (comment)

[Xunzhuo]

/retest

[guydc]

/retest

[guydc]

/retest

[arkodg]

LGTM thanks !