Skip to content

Commit

Permalink
[#1190] feat(server-common): Support custom filters (#1191)
Browse files Browse the repository at this point in the history 
### What changes were proposed in this pull request?
Add the support of custom filters referring to the Spark.

### Why are the changes needed?
Some filters can be configured by users, the needs of users are
different. Some filters are implemented by other libs.
For example, Jetty has implemented the CORS filter, so we supported
custom filters to satisfy the needs of users.

Fix: #1190

### Does this PR introduce _any_ user-facing change?

Yes, I have added the document.

### How was this patch tested?
test by the browser console
I test Jetty's cors filter.

Configuration 
```
gravitino.server.webserver.customFilters=org.eclipse.jetty.servlets.CrossOriginFilter
gravitino.server.webserver.org.eclipse.jetty.servlets.CrossOriginFilter.param.allowedOrigins=*
```

Browser console code
```
var http = new XMLHttpRequest();
var url = 'http://localhost:8090/api/version';
http.onreadystatechange = (e) => {
    console.log(http.responseText)
}
http.open("GET", url);
http.send();
```

---------

Co-authored-by: Heng Qin <[email protected]>
  • Loading branch information
@qqqttt123
qqqttt123 and Heng Qin authored Dec 21, 2023
1 parent 972747e commit 18feb15
Show file tree
Hide file tree
Showing 7 changed files with 85 additions and 18 deletions.
1 change: 1 addition & 0 deletions .../src/main/java/com/datastrato/gravitino/catalog/lakehouse/iceberg/IcebergRESTService.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ protected void configure() {

Servlet servlet = new ServletContainer(config);
server.addServlet(servlet, "/iceberg/*");
server.addCustomFilters("/iceberg/*");
server.addFilter(new AuthenticationFilter(), "/iceberg/*");
}

Expand Down
28 changes: 16 additions & 12 deletions docs/gravitino-server-config.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,22 @@ The `gravitino.conf` file lists the configuration items in the following table.

### Gravitino HTTP Server configuration

| Configuration item | Description | Default value | Required | Since version |
|------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|----------|---------------|
| `gravitino.server.webserver.host` | The host of Gravitino server. | `0.0.0.0` | No | 0.1.0 |
| `gravitino.server.webserver.httpPort` | The port on which the Gravitino server listens for incoming connections. | `8090` | No | 0.1.0 |
| `gravitino.server.webserver.minThreads` | The minimum number of threads in the thread pool used by Jetty webserver. `minThreads` is 8 if the value is less than 8. | `Math.max(Math.min(Runtime.getRuntime().availableProcessors() * 2, 100), 8)` | No | 0.2.0 |
| `gravitino.server.webserver.maxThreads` | The maximum number of threads in the thread pool used by Jetty webserver. `maxThreads` is 8 if the value is less than 8, and `maxThreads` must be great or equal to `minThreads`. | `Math.max(Runtime.getRuntime().availableProcessors() * 4, 400)` | No | 0.1.0 |
| `gravitino.server.webserver.threadPoolWorkQueueSize` | The size of the queue in the thread pool used by Jetty webserver. | `100` | No | 0.1.0 |
| `gravitino.server.webserver.stopTimeout` | Time in milliseconds to gracefully shutdown the Jetty webserver, for more, please see `org.eclipse.jetty.server.Server#setStopTimeout`. | `30000` | No | 0.2.0 |
| `gravitino.server.webserver.idleTimeout` | The timeout in milliseconds of idle connections. | `30000` | No | 0.2.0 |
| `gravitino.server.webserver.requestHeaderSize` | Maximum size of HTTP requests. | `131072` | No | 0.1.0 |
| `gravitino.server.webserver.responseHeaderSize` | Maximum size of HTTP responses. | `131072` | No | 0.1.0 |
| `gravitino.server.shutdown.timeout` | Time in milliseconds to gracefully shutdown of the Gravitino webserver. | `3000` | No | 0.2.0 |
| Configuration item | Description | Default value | Required | Since version |
|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|----------|---------------|
| `gravitino.server.webserver.host` | The host of Gravitino server. | `0.0.0.0` | No | 0.1.0 |
| `gravitino.server.webserver.httpPort` | The port on which the Gravitino server listens for incoming connections. | `8090` | No | 0.1.0 |
| `gravitino.server.webserver.minThreads` | The minimum number of threads in the thread pool used by Jetty webserver. `minThreads` is 8 if the value is less than 8. | `Math.max(Math.min(Runtime.getRuntime().availableProcessors() * 2, 100), 8)` | No | 0.2.0 |
| `gravitino.server.webserver.maxThreads` | The maximum number of threads in the thread pool used by Jetty webserver. `maxThreads` is 8 if the value is less than 8, and `maxThreads` must be great or equal to `minThreads`. | `Math.max(Runtime.getRuntime().availableProcessors() * 4, 400)` | No | 0.1.0 |
| `gravitino.server.webserver.threadPoolWorkQueueSize` | The size of the queue in the thread pool used by Jetty webserver. | `100` | No | 0.1.0 |
| `gravitino.server.webserver.stopTimeout` | Time in milliseconds to gracefully shutdown the Jetty webserver, for more, please see `org.eclipse.jetty.server.Server#setStopTimeout`. | `30000` | No | 0.2.0 |
| `gravitino.server.webserver.idleTimeout` | The timeout in milliseconds of idle connections. | `30000` | No | 0.2.0 |
| `gravitino.server.webserver.requestHeaderSize` | Maximum size of HTTP requests. | `131072` | No | 0.1.0 |
| `gravitino.server.webserver.responseHeaderSize` | Maximum size of HTTP responses. | `131072` | No | 0.1.0 |
| `gravitino.server.shutdown.timeout` | Time in milliseconds to gracefully shutdown of the Gravitino webserver. | `3000` | No | 0.2.0 |
| `gravitino.server.webserver.customFilters` | Comma separated list of filter class names to apply to the APIs. | (none) | No | 0.4.0 |

The filter in the customFilters should be a standard javax servlet Filter.
Filter parameters can also be specified in the configuration, by setting config entries of the form `gravitino.server.webserver.<class name of filter>.param.<param name>=<value>`

### Storage configuration

Expand Down
5 changes: 5 additions & 0 deletions docs/iceberg-rest-service.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ Deploy the Gravitino server to the `GRAVITINO_HOME` directory. You can find the
| `gravitino.auxService.iceberg-rest.idleTimeout` | The timeout in ms of idle connections. | `30000` | No | 0.2.0 |
| `gravitino.auxService.iceberg-rest.requestHeaderSize` | The maximum size of an HTTP request. | `131072` | No | 0.2.0 |
| `gravitino.auxService.iceberg-rest.responseHeaderSize` | The maximum size of an HTTP response. | `131072` | No | 0.2.0 |
| `gravitino.auxService.iceberg-rest.customFilters` | Comma separated list of filter class names to apply to the APIs. | (none) | No | 0.4.0 |


The filter in the customFilters should be a standard javax servlet Filter.
Filter parameters can also be specified in the configuration, by setting config entries of the form `gravitino.auxService.iceberg-rest.<class name of filter>.param.<param name>=<value>`

:::caution
You must set `gravitino.auxService.iceberg-rest.httpPort` explicitly, like `9001`.
Expand Down
20 changes: 18 additions & 2 deletions server-common/src/main/java/com/datastrato/gravitino/server/web/JettyServer.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import java.nio.file.Paths;
import java.security.PrivilegedAction;
import java.util.EnumSet;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.LinkedBlockingQueue;
Expand Down Expand Up @@ -103,7 +104,7 @@ public synchronized void initialize(
StringUtils.isNotBlank(serverConfig.getTrustStorePath()),
"If enables the authentication of the client, must set trustStorePath");
Preconditions.checkArgument(
StringUtils.isNotBlank(serverConfig.getTrustStorePasword()),
StringUtils.isNotBlank(serverConfig.getTrustStorePassword()),
"If enables the authentication of the client, must set trustStorePassword");
}
ServerConnector httpsConnector =
Expand All @@ -122,7 +123,7 @@ public synchronized void initialize(
serverConfig.getSupportedAlgorithms(),
serverConfig.isEnableClientAuth(),
serverConfig.getTrustStorePath(),
serverConfig.getTrustStorePasword(),
serverConfig.getTrustStorePassword(),
serverConfig.getTrustStoreType());
server.addConnector(httpsConnector);
} else {
Expand Down Expand Up @@ -433,4 +434,19 @@ public Thread run() {
public ThreadPool getThreadPool() {
return server.getThreadPool();
}

public void addCustomFilters(String pathSpec) {
for (String filterName : serverConfig.getCustomFilters()) {
if (StringUtils.isBlank(filterName)) {
continue;
}
FilterHolder filterHolder = new FilterHolder();
filterHolder.setClassName(filterName);
for (Map.Entry<String, String> entry :
serverConfig.getAllWithPrefix(String.format("%s.param.", filterName)).entrySet()) {
filterHolder.setInitParameter(entry.getKey(), entry.getValue());
}
servletContextHandler.addFilter(filterHolder, pathSpec, EnumSet.allOf(DispatcherType.class));
}
}
}
30 changes: 26 additions & 4 deletions server-common/src/main/java/com/datastrato/gravitino/server/web/JettyServerConfig.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,13 @@ public final class JettyServerConfig {
.stringConf()
.createWithDefault("JKS");

public static final ConfigEntry<Optional<String>> CUSTOM_FILTERS =
new ConfigBuilder("customFilters")
.doc("Comma separated list of filter class names to apply to the APIs")
.version("0.4.0")
.stringConf()
.createWithOptional();

private final String host;

private final int httpPort;
Expand Down Expand Up @@ -199,7 +206,9 @@ public final class JettyServerConfig {
private final Set<String> enableCipherAlgorithms;
private final boolean enableClientAuth;
private final String trustStorePath;
private final String trustStorePasword;
private final String trustStorePassword;

private final Set<String> customFilters;
private final String trustStoreType;
private final Config internalConfig;

Expand Down Expand Up @@ -245,8 +254,13 @@ private JettyServerConfig(Map<String, String> configs) {
Sets.newHashSet(internalConfig.get(ENABLE_CIPHER_ALGORITHMS).split(SPLITTER)));
this.enableClientAuth = internalConfig.get(ENABLE_CLIENT_AUTH);
this.trustStorePath = internalConfig.get(SSL_TRUST_STORE_PATH);
this.trustStorePasword = internalConfig.get(SSL_TRUST_STORE_PASSWORD);
this.trustStorePassword = internalConfig.get(SSL_TRUST_STORE_PASSWORD);
this.trustStoreType = internalConfig.get(SSL_TRUST_STORE_TYPE);
this.customFilters =
internalConfig
.get(CUSTOM_FILTERS)
.map(filters -> Collections.unmodifiableSet(Sets.newHashSet(filters.split(SPLITTER))))
.orElse(Collections.emptySet());
}

public static JettyServerConfig fromConfig(Config config, String prefix) {
Expand Down Expand Up @@ -330,14 +344,18 @@ public String getTrustStorePath() {
return trustStorePath;
}

public String getTrustStorePasword() {
return trustStorePasword;
public String getTrustStorePassword() {
return trustStorePassword;
}

public String getTrustStoreType() {
return trustStoreType;
}

public Map<String, String> getAllWithPrefix(String prefix) {
return internalConfig.getConfigsWithPrefix(prefix);
}

private SSLContext getDefaultSSLContext() {
try {
return SSLContext.getDefault();
Expand Down Expand Up @@ -366,6 +384,10 @@ public Set<String> getSupportedAlgorithms() {
return supportedAlgorithms;
}

public Set<String> getCustomFilters() {
return customFilters;
}

@VisibleForTesting
Set<String> getSupportedCipherSuites() {
SSLContext context =
Expand Down
18 changes: 18 additions & 0 deletions server-common/src/test/java/com/datastrato/gravitino/server/web/TestJettyServerConfig.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
package com.datastrato.gravitino.server.web;

import com.datastrato.gravitino.Config;
import com.datastrato.gravitino.config.ConfigBuilder;
import com.google.common.collect.Sets;
import java.util.Collections;
import java.util.Optional;
Expand Down Expand Up @@ -44,4 +45,21 @@ public void testCipherAlgorithms() {
Assertions.assertIterableEquals(
Sets.newHashSet(algorithm), jettyServerConfig.getSupportedAlgorithms());
}

@Test
public void testCustomFilters() {
Config emptyconfig = new Config() {};
JettyServerConfig jettyServerConfig = JettyServerConfig.fromConfig(emptyconfig, "");
Assertions.assertTrue(jettyServerConfig.getCustomFilters().isEmpty());

Config somethingConfig = new Config() {};
somethingConfig.set(JettyServerConfig.CUSTOM_FILTERS, Optional.of("1,2"));
somethingConfig.set(new ConfigBuilder("1.1").stringConf(), "test");
somethingConfig.set(new ConfigBuilder("1.2").stringConf(), "test");
jettyServerConfig = JettyServerConfig.fromConfig(somethingConfig, "");
Assertions.assertIterableEquals(
Sets.newHashSet("1", "2"), jettyServerConfig.getCustomFilters());
Assertions.assertTrue(jettyServerConfig.getAllWithPrefix("2.").isEmpty());
Assertions.assertEquals(2, jettyServerConfig.getAllWithPrefix("1.").size());
}
}
1 change: 1 addition & 0 deletions server/src/main/java/com/datastrato/gravitino/server/GravitinoServer.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ protected void configure() {
server.addServlet(servlet, "/api/*");
Servlet configServlet = new ConfigServlet(serverConfig);
server.addServlet(configServlet, "/configs");
server.addCustomFilters("/api/*");
server.addFilter(new VersioningFilter(), "/api/*");
server.addFilter(new AuthenticationFilter(), "/api/*");
}
Expand Down

0 comments on commit 18feb15

Please sign in to comment.