[Backport 2.x] [Refactor] Remove bloat due to unnecessary setup in test and add retry for potential flaky behavior due to timeout

[Backport 2.x] [Refactor] Remove bloat due to unnecessary setup in test and add retry for potential flaky behavior due to timeout #2926