BWC Rolling upgrade tests are failing

[vibrantvarun]

The BWC Rolling Upgrade Tests are failing on Branch 2.x for BWC version <= 2.10 . This issue keeps a track for the fix.

[vibrantvarun]

Error Coming from RestClient while running testAgainstOneThirdUpgradedCluster

org.opensearch.neuralsearch.bwc.SemanticSearchIT > testTextEmbeddingProcessor_E2EFlow FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:40629], URI [/neuralsearch-bwc-testtextembeddingprocessor_e2eflow/_search?size=1], status line [HTTP/1.1 500 Internal Server Error]
    {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"unexpected byte [0x48]"}],"type":"illegal_state_exception","reason":"unexpected byte [0x48]"},"status":500}
        at __randomizedtesting.SeedInfo.seed([AEF0E47C79E934E5:BE6CB232349B64BC]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:396)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:357)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:343)
        at app//org.opensearch.neuralsearch.bwc.SemanticSearchIT.validateTestIndexOnUpgrade(SemanticSearchIT.java:77)
        at app//org.opensearch.neuralsearch.bwc.SemanticSearchIT.testTextEmbeddingProcessor_E2EFlow(SemanticSearchIT.java:46)
REPRODUCE WITH: ./gradlew ':qa:rolling-upgrade:testAgainstOneThirdUpgradedCluster' --tests "org.opensearch.neuralsearch.bwc.SemanticSearchIT.testTextEmbeddingProcessor_E2EFlow" -Dtests.seed=AEF0E47C79E934E5 -Dtests.security.manager=false -Dtests.locale=ms-MY -Dtests.timezone=Asia/Sakhalin -Druntime.java=11

[vibrantvarun]

We are tracking the fix of issue over here