feat: handle 'session not found' errors

README.md

@@ -236,6 +236,68 @@ In order to improve the performance of the first connection per request, we use
 By default, laravel-spanner uses [Filesystem Cache Adapter](https://symfony.com/doc/current/components/cache/adapters/filesystem_adapter.html) as the caching pool. If you want to use your own caching pool, you can extend ServiceProvider and inject it into the constructor of `Colopl\Spanner\Connection`.
 
 
+
+### 'Session not found' exception handling
+There are a few cases when a 'Session not found' error can
+[happen](https://cloud.google.com/spanner/docs/sessions#handle_deleted_sessions):
+
+ - Scripts that idle too long - for example, a [Laravel queue worker](https://laravel.com/docs/9.x/queues#the-queue-work-command)
+or anything that doesn't call Spanner frequently enough (more than once an hour).
+ - The session is more than 28 days old.
+ - Some random flukes on Google's side.
+
+The errors can be handled by one of the supported modes:
+
+- **MAINTAIN_SESSION_POOL** - When the 'session not found' error is encountered, the library tries to disconnect,
+[maintain a session pool](https://github.com/googleapis/google-cloud-php/blob/077810260b58f5de8a3bbdfd999a5e9a48f71a7f/Spanner/src/Session/CacheSessionPool.php#L864)
+(to remove outdated sessions), reconnect, and then try querying again.
+```php
+        'spanner' => [
+            'driver' => 'spanner',
+        ...
+   	        'sessionNotFoundErrorMode' => 'MAINTAIN_SESSION_POOL',
+        ]
+```
+
+- **CLEAR_SESSION_POOL** (default) - The **MAINTAIN_SESSION_POOL** mode is tried first. If the error still happens, then
+the [clearing of the session pool](https://github.com/googleapis/google-cloud-php/blob/077810260b58f5de8a3bbdfd999a5e9a48f71a7f/Spanner/src/Session/CacheSessionPool.php#L465)
+is enforced and the query is tried once again.
+As a consequence of session pool clearing, all processes that share the current session pool will be forced
+to use the new session on the next call. The mode is enabled by default, but you can enable it explicitly via congifuration:
+```php
+        'spanner' => [
+            'driver' => 'spanner',
+        ...
+       	    'sessionNotFoundErrorMode' => 'CLEAR_SESSION_POOL'
+        ]
+```
+
+- none - The QueryException is raised and the client code is free to handle it by itself.:
+```php
+        'spanner' => [
+            'driver' => 'spanner',
+        ...
+   	        'sessionNotFoundErrorMode' => false,
+        ]
+```
+
+Please note, that [`getDatabaseContext()->execute(...)->rows()`](https://github.com/googleapis/google-cloud-php/blob/077810260b58f5de8a3bbdfd999a5e9a48f71a7f/Spanner/src/Result.php#L175)
+returns a `/Generator` object, which only accesses Spanner when iterated. That affects `cursor()`
+and `cursorWithTimestampBound()` functions and many low-level calls. So you might still
+get `Google\Cloud\Core\Exception\NotFoundException` when trying to resolve cursor.
+To avoid that, please run cursor* functions inside
+[explicit transactions](#transactions) so statements will repeat on error.
+
+```php
+$conn->transaction(function () use ($conn) {
+    $cursor = $conn->cursor('SELECT ...');
+
+    foearch ($cursor as $value) { ...
+});
+```
+
+
+
 ### Laravel Tinker
 You can use [Laravel Tinker](https://github.com/laravel/tinker) with commands such as `php artisan tinker`.
 But your session may hang when accessing Cloud Spanner. This is known gRPC issue that occurs when PHP forks a process.

phpstan.neon

@@ -16,8 +16,6 @@ parameters:
           path: src/Colopl/Spanner/Connection.php
         - message: '#^Method Colopl\\Spanner\\Connection::getSpannerDatabase\(\) should return Google\\Cloud\\Spanner\\Database but returns Google\\Cloud\\Spanner\\Database\|null.$#'
           path: src/Colopl/Spanner/Connection.php
-        - message: '#^Method Colopl\\Spanner\\Connection::transaction\(\) should return T but returns mixed\.$#'
-          path: src/Colopl/Spanner/Connection.php
         - message: '#^Cannot cast mixed to int\.$#'
           path: src/Colopl/Spanner/Eloquent/Model.php
         - message: '#^Method Colopl\\Spanner\\Query\\Builder::insertGetId\(\) should return int but return statement is missing\.$#'

src/Colopl/Spanner/Concerns/ManagesSessionPool.php

@@ -81,7 +81,8 @@ public function listSessions(): Collection
     {
         $databaseName = $this->getSpannerDatabase()->name();
         $response = (new ProtobufSpannerClient())->listSessions($databaseName);
-        return collect($response->iterateAllElements())->map(function (ProtobufSpannerSession $session) {
+        return collect($response->iterateAllElements())->map(function ($session) {
+            assert($session instanceof ProtobufSpannerSession);
             return new Session($session);
         });
     }

src/Colopl/Spanner/Concerns/ManagesStaleReads.php

@@ -56,7 +56,9 @@ public function cursorWithTimestampBound($query, $bindings = [], TimestampBoundI
      */
     public function selectWithTimestampBound($query, $bindings = [], TimestampBoundInterface $timestampBound = null): array
     {
-        return iterator_to_array($this->cursorWithTimestampBound($query, $bindings, $timestampBound));
+        return $this->sessionNotFoundWrapper(function () use ($query, $bindings, $timestampBound) {
+            return iterator_to_array($this->cursorWithTimestampBound($query, $bindings, $timestampBound));
+        });
     }
 
     /**
@@ -68,7 +70,11 @@ public function selectWithTimestampBound($query, $bindings = [], TimestampBoundI
      */
     public function selectOneWithTimestampBound($query, $bindings = [], TimestampBoundInterface $timestampBound = null): ?array
     {
-        return $this->cursorWithTimestampBound($query, $bindings, $timestampBound)->current();
+        $result = $this->sessionNotFoundWrapper(function () use ($query, $bindings, $timestampBound) {
+            return $this->cursorWithTimestampBound($query, $bindings, $timestampBound)->current();
+        });
+        assert(is_null($result) || is_array($result));
+        return $result;
     }
 }
 

src/Colopl/Spanner/Concerns/ManagesTransactions.php

@@ -20,6 +20,7 @@
 use Closure;
 use Exception;
 use Google\Cloud\Core\Exception\AbortedException;
+use Google\Cloud\Core\Exception\NotFoundException;
 use Google\Cloud\Spanner\Database;
 use Google\Cloud\Spanner\Transaction;
 use Throwable;
@@ -44,29 +45,31 @@ trait ManagesTransactions
      */
     public function transaction(Closure $callback, $attempts = Database::MAX_RETRIES)
     {
-        // Since Cloud Spanner does not support nested transactions,
-        // we use Laravel's transaction management for nested transactions only.
-        if ($this->transactions > 0) {
-            return parent::transaction($callback, $attempts);
-        }
-
-        $return = $this->getSpannerDatabase()->runTransaction(function (Transaction $tx) use ($callback) {
-            try {
-                $this->currentTransaction = $tx;
-                $this->transactions++;
-                $this->fireConnectionEvent('beganTransaction');
-                $result = $callback($this);
-                $this->performSpannerCommit();
-                return $result;
-            } catch (Throwable $e) {
-                $this->rollBack();
-                throw $e;
+		return $this->sessionNotFoundWrapper(function () use ($callback, $attempts) {
+            // Since Cloud Spanner does not support nested transactions,
+            // we use Laravel's transaction management for nested transactions only.
+            if ($this->transactions > 0) {
+                return parent::transaction($callback, $attempts);
             }
-        }, ['maxRetries' => $attempts - 1]);
-
-        $this->fireConnectionEvent('committed');
 
-        return $return;
+			$return = $this->getSpannerDatabase()->runTransaction(function (Transaction $tx) use ($callback) {
+				try {
+					$this->currentTransaction = $tx;
+					$this->transactions++;
+					$this->fireConnectionEvent('beganTransaction');
+					$result = $callback($this);
+					$this->performSpannerCommit();
+					return $result;
+				} catch (Throwable $e) {
+					$this->rollBack();
+					throw $e;
+				}
+			}, ['maxRetries' => $attempts - 1]);
+
+			$this->fireConnectionEvent('committed');
+
+			return $return;
+		});
     }
 
     /**
@@ -158,7 +161,14 @@ protected function performRollBack($toLevel)
 
         if ($this->currentTransaction !== null) {
             if ($this->currentTransaction->state() === Transaction::STATE_ACTIVE) {
-                $this->currentTransaction->rollBack();
+                try {
+                    $this->currentTransaction->rollBack();
+                } catch (NotFoundException $e) {
+                    // ignore session not found error
+                    if (empty($this->getSessionNotFoundMode()) || !$this->causedBySessionNotFound($e)) {
+                        throw $e;
+                    }
+                }
             }
             $this->currentTransaction = null;
         }

src/Colopl/Spanner/Connection.php

@@ -30,13 +30,15 @@
 use Generator;
 use Google\Cloud\Core\Exception\AbortedException;
 use Google\Cloud\Core\Exception\GoogleException;
+use Google\Cloud\Core\Exception\NotFoundException;
 use Google\Cloud\Spanner\Database;
 use Google\Cloud\Spanner\Session\SessionPoolInterface;
 use Google\Cloud\Spanner\SpannerClient;
 use Google\Cloud\Spanner\Transaction;
 use Illuminate\Contracts\Support\Arrayable;
 use Illuminate\Database\Connection as BaseConnection;
 use Illuminate\Database\QueryException;
+use InvalidArgumentException;
 use Psr\Cache\CacheItemPoolInterface;
 use RuntimeException;
 use Throwable;
@@ -82,6 +84,21 @@ class Connection extends BaseConnection
      */
     protected $sessionPool;
 
+    /**
+     * Try to maintain session pool on 'session not found' error
+     */
+    public const MAINTAIN_SESSION_POOL = 'MAINTAIN_SESSION_POOL';
+
+    /**
+     * Try to maintain and then clear session pool on 'session not found' error
+     */
+    public const CLEAR_SESSION_POOL = 'CLEAR_SESSION_POOL';
+
+    /**
+     * Used to detect specific exception
+     */
+    public const SESSION_NOT_FOUND_CONDITION = 'Session does not exist';
+
     /**
      * @param string $instanceId instance ID
      * @param string $databaseName
@@ -425,7 +442,9 @@ protected function runQueryCallback($query, $bindings, Closure $callback)
         [$query, $bindings] = $this->parameterizer->parameterizeQuery($query, $bindings);
 
         try {
-            $result = $callback($query, $bindings);
+            $result = $this->sessionNotFoundWrapper(function () use ($query, $bindings, $callback) {
+                return $callback($query, $bindings);
+            });
         }
 
         // AbortedExceptions are expected to be thrown upstream by the Google Client Library upstream,
@@ -443,4 +462,83 @@ protected function runQueryCallback($query, $bindings, Closure $callback)
 
         return $result;
     }
+
+    /**
+     * Returns current mode
+     *
+     * @return string
+     */
+    protected function getSessionNotFoundMode()
+    {
+        return $this->config['sessionNotFoundErrorMode'] ?? self::CLEAR_SESSION_POOL;
+    }
+
+    /**
+     * Handle "session not found" errors
+     *
+     * @template T
+     * @param  Closure(): T $callback
+     * @return T
+     * @throws InvalidArgumentException|NotFoundException|AbortedException
+     */
+    protected function sessionNotFoundWrapper(Closure $callback)
+    {
+        $handlerMode = $this->getSessionNotFoundMode();
+        if (empty($handlerMode) || $this->sessionPool === null) {
+            // skip handlers
+            return $callback();
+        }
+
+        if (!in_array($handlerMode, [
+                self::MAINTAIN_SESSION_POOL,
+                self::CLEAR_SESSION_POOL,
+            ])
+        ) {
+            throw new InvalidArgumentException("Unsupported sessionNotFoundErrorMode [{$handlerMode}].");
+        }
+        try {
+            return $callback();
+        } catch (NotFoundException $e) {
+            // ensure if this really error with session
+            if ($this->causedBySessionNotFound($e)) {
+                if ($this->inTransaction()) {
+                    // if we inside transaction then throw abort exception
+                    throw new AbortedException(self::SESSION_NOT_FOUND_CONDITION, $e->getCode(), $e);
+                }
+                $this->disconnect();
+                // clear expired sessions, manually deleted sessions still raise error
+                $this->maintainSessionPool();
+                $this->reconnect();
+                try {
+                    return $callback();
+                } catch (NotFoundException $e) {
+                    if ($handlerMode == self::CLEAR_SESSION_POOL && $this->causedBySessionNotFound($e)) {
+                        $this->disconnect();
+                        // forcefully clearing sessions, might affect parallel processes
+                        // also cleared sessions are still accounted toward spanner limit - 10k sessions per node
+                        $this->clearSessionPool();
+                        $this->reconnect();
+                        return $callback();
+                    } else {
+                        throw $e;
+                    }
+                }
+            } else {
+                throw $e;
+            }
+        }
+    }
+
+    /**
+     * Check if this is "session not found" error
+     *
+     * @param  Throwable  $e
+     * @return boolean
+     */
+    public function causedBySessionNotFound(Throwable $e): bool
+    {
+        return ($e instanceof NotFoundException)
+            && strpos($e->getMessage(), self::SESSION_NOT_FOUND_CONDITION) !== false;
+    }
+
 }