Fix issue with finishing handshake in ssl driver (#30580)

This is fixing an issue that has come up in some builds. In some
scenarios I see an assertion failure that we are trying to move to
application mode when we are not in handshake mode. What I think is
happening is that we are in handshake mode and have received the
completed handshake message AND an application message. While reading in
handshake mode we switch to application mode. However, there is still
data to be consumed so we attempt to continue to read in handshake mode.
This leads to us attempting to move to application mode again throwing
an assertion.

This commit fixes this by immediatly exiting the handshake mode read
method if we are not longer in handshake mode. Additionally if we swap
modes during a read we attempt to read with the new mode to see if there
is data that needs to be handled.

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/transport/nio/SSLDriver.java

@@ -113,7 +113,13 @@ public ByteBuffer getNetworkReadBuffer() {
     }
 
     public void read(InboundChannelBuffer buffer) throws SSLException {
-        currentMode.read(buffer);
+        Mode modePriorToRead;
+        do {
+            modePriorToRead = currentMode;
+            currentMode.read(buffer);
+            // If we switched modes we want to read again as there might be unhandled bytes that need to be
+            // handled by the new mode.
+        } while (modePriorToRead != currentMode);
     }
 
     public boolean readyForApplicationWrites() {
@@ -365,8 +371,9 @@ public void read(InboundChannelBuffer buffer) throws SSLException {
                 try {
                     SSLEngineResult result = unwrap(buffer);
                     handshakeStatus = result.getHandshakeStatus();
-                    continueUnwrap = result.bytesConsumed() > 0;
                     handshake();
+                    // If we are done handshaking we should exit the handshake read
+                    continueUnwrap = result.bytesConsumed() > 0 && currentMode.isHandshake();
                 } catch (SSLException e) {
                     closingInternal();
                     throw e;