feat: add support for Directed Read options

[arpan14]

Customers can set the option at two level a) per RPC by passing Options.directedRead or b) global setting in SpannerOptions. The option is not applicable for RW transactions. If customer sets the setting for a RW operation, they will receive an exception.

Sample Usage

DatabaseClient client =
        spanner.getDatabaseClient(DatabaseId.of(TEST_PROJECT, TEST_INSTANCE, TEST_DATABASE));
    try (ResultSet resultSet =
        client.singleUse().executeQuery(SELECT1, Options.directedRead(DIRECTED_READ_OPTIONS1))) {
      while (resultSet.next()) {}
    }

Spanner spannerWithDirectedReadOptions =
        spanner
            .getOptions()
            .toBuilder()
            .setDirectedReadOption(DIRECTED_READ_OPTIONS2)
            .build()
            .getService();
    DatabaseClient client =
        spannerWithDirectedReadOptions.getDatabaseClient(
            DatabaseId.of(TEST_PROJECT, TEST_INSTANCE, TEST_DATABASE));
    TransactionRunner runner = client.readWriteTransaction();
    runner.run(
        transaction -> {
          try (ResultSet resultSet = transaction.executeQuery(SELECT1)) {
            while (resultSet.next()) {}
          }
          return null;
        });

[olavloite]

Will this fail if someone tries to run this integration test on an instance config that does not have a read-only replica in us-west1? If so; could we handle that more gracefully? (Could we for example read the possible values from the actual instance config, and set one of those here?)

[harshachinta]

If the provided replica is not available in that instance config, then the request will silently ignore the setting.

[arpan14]

@olavloite Based on what @harshachinta mentioned, it should be ok with the way the tests are currently written? Or do you think we can further enhance something? I think its fine, since over here we don't have a way to assert if backend used the setting or ignored it.

[olavloite]

It actually makes me worry a bit more, but more generally about the feature than about the test :-)

The fact that an invalid value is silently ignored means that the user does not get any feedback about whether the feature is being used or not. If a user makes a typo for the location, then there is no feedback loop that will indicate that to the user, which again can make for very difficult debugging.
Could we file a request with the feature owner that there is some kind of feedback from the server about which replica was actually used to serve the read? Or do we already have that information in some of the metadata that we receive?

[arpan14]

@mayurkale22 Can you take a look here and answer this query? I think this is another reason why we should return the feedback in our backend response (similar to what we were discussing last evening)

[mayurkale22]

If the provided replica is not available in that instance config, then the request will silently ignore the setting.

In the earlier design, the plan was to ensure that the location and/or replica type specified in the directed reads settings does indeed belong to the instance configuration used, otherwise, return an INVALID_ARGUMENT error to indicate users that they are using the incorrect setting/location. However, this means that if the currently specified location(s) from the source instance configuration does not exist in the destination instance configuration, this would result in requests errors during the instance move/migration process. That's a poor user experience.

To support friction-free instance migration, we will eliminate the above constraint. So basically, if the location(s) provided in directed reads settings are not found in the instance configuration, the Spanner will fallback to default routing mechanism by sending the requests to the nearest available replica and hence this will ensure that the directed read requests do not fail during/after the instance migration. To tell users about incorrect location in the directed reads, we plan to expose a new metric. The metric can be used to understand misconfigured settings.

Could we file a request with the feature owner that there is some kind of feedback from the server about which replica was actually used to serve the read?

We plan to expose this info in the metric for now. In the future we can expose in the API response itself.