Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#1190] feat(server-common): Support custom filters #1191

Merged
merged 9 commits into from
Dec 21, 2023
+85 −18
Merged

[#1190] feat(server-common): Support custom filters #1191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
bc09888
[ISSUE-1190]feat(server-common): Support custom filter
Dec 18, 2023
18faf4d
fix
Dec 18, 2023
b17e332
Merge remote-tracking branch 'upstream/main' into ISSUE-42
Dec 18, 2023
d5272f5
fix
Dec 19, 2023
55184a6
[MINOR] fix(docs): Modify the required value for the document
Dec 20, 2023
239e55d
fix
Dec 20, 2023
9a9c107
Revert "[MINOR] fix(docs): Modify the required value for the document"
Dec 20, 2023
831feee
fixed
Dec 20, 2023
d72c39d
fix
Dec 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .../src/main/java/com/datastrato/gravitino/catalog/lakehouse/iceberg/IcebergRESTService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ protected void configure() {

Servlet servlet = new ServletContainer(config);
server.addServlet(servlet, "/iceberg/*");
server.addCustomFilters("/iceberg/*");
server.addFilter(new AuthenticationFilter(), "/iceberg/*");
}

Expand Down
28 changes: 16 additions & 12 deletions docs/gravitino-server-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,22 @@ The `gravitino.conf` file lists the configuration items in the following table.

### Gravitino HTTP Server configuration

| Configuration item | Description | Default value | Required | Since version |
|------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|----------|---------------|
| `gravitino.server.webserver.host` | The host of Gravitino server. | `0.0.0.0` | No | 0.1.0 |
| `gravitino.server.webserver.httpPort` | The port on which the Gravitino server listens for incoming connections. | `8090` | No | 0.1.0 |
| `gravitino.server.webserver.minThreads` | The minimum number of threads in the thread pool used by Jetty webserver. `minThreads` is 8 if the value is less than 8. | `Math.max(Math.min(Runtime.getRuntime().availableProcessors() * 2, 100), 8)` | No | 0.2.0 |
| `gravitino.server.webserver.maxThreads` | The maximum number of threads in the thread pool used by Jetty webserver. `maxThreads` is 8 if the value is less than 8, and `maxThreads` must be great or equal to `minThreads`. | `Math.max(Runtime.getRuntime().availableProcessors() * 4, 400)` | No | 0.1.0 |
| `gravitino.server.webserver.threadPoolWorkQueueSize` | The size of the queue in the thread pool used by Jetty webserver. | `100` | No | 0.1.0 |
| `gravitino.server.webserver.stopTimeout` | Time in milliseconds to gracefully shutdown the Jetty webserver, for more, please see `org.eclipse.jetty.server.Server#setStopTimeout`. | `30000` | No | 0.2.0 |
| `gravitino.server.webserver.idleTimeout` | The timeout in milliseconds of idle connections. | `30000` | No | 0.2.0 |
| `gravitino.server.webserver.requestHeaderSize` | Maximum size of HTTP requests. | `131072` | No | 0.1.0 |
| `gravitino.server.webserver.responseHeaderSize` | Maximum size of HTTP responses. | `131072` | No | 0.1.0 |
| `gravitino.server.shutdown.timeout` | Time in milliseconds to gracefully shutdown of the Gravitino webserver. | `3000` | No | 0.2.0 |
| Configuration item | Description | Default value | Required | Since version |
|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|----------|---------------|
| `gravitino.server.webserver.host` | The host of Gravitino server. | `0.0.0.0` | No | 0.1.0 |
| `gravitino.server.webserver.httpPort` | The port on which the Gravitino server listens for incoming connections. | `8090` | No | 0.1.0 |
| `gravitino.server.webserver.minThreads` | The minimum number of threads in the thread pool used by Jetty webserver. `minThreads` is 8 if the value is less than 8. | `Math.max(Math.min(Runtime.getRuntime().availableProcessors() * 2, 100), 8)` | No | 0.2.0 |
| `gravitino.server.webserver.maxThreads` | The maximum number of threads in the thread pool used by Jetty webserver. `maxThreads` is 8 if the value is less than 8, and `maxThreads` must be great or equal to `minThreads`. | `Math.max(Runtime.getRuntime().availableProcessors() * 4, 400)` | No | 0.1.0 |
| `gravitino.server.webserver.threadPoolWorkQueueSize` | The size of the queue in the thread pool used by Jetty webserver. | `100` | No | 0.1.0 |
| `gravitino.server.webserver.stopTimeout` | Time in milliseconds to gracefully shutdown the Jetty webserver, for more, please see `org.eclipse.jetty.server.Server#setStopTimeout`. | `30000` | No | 0.2.0 |
| `gravitino.server.webserver.idleTimeout` | The timeout in milliseconds of idle connections. | `30000` | No | 0.2.0 |
| `gravitino.server.webserver.requestHeaderSize` | Maximum size of HTTP requests. | `131072` | No | 0.1.0 |
| `gravitino.server.webserver.responseHeaderSize` | Maximum size of HTTP responses. | `131072` | No | 0.1.0 |
| `gravitino.server.shutdown.timeout` | Time in milliseconds to gracefully shutdown of the Gravitino webserver. | `3000` | No | 0.2.0 |
| `gravitino.server.webserver.customFilters` | Comma separated list of filter class names to apply to the APIs. | (none) | No | 0.4.0 |

The filter in the customFilters should be a standard javax servlet Filter.
Filter parameters can also be specified in the configuration, by setting config entries of the form `gravitino.server.webserver.<class name of filter>.param.<param name>=<value>`

### Storage configuration

Expand Down
5 changes: 5 additions & 0 deletions docs/iceberg-rest-service.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ Deploy the Gravitino server to the `GRAVITINO_HOME` directory. You can find the
| `gravitino.auxService.iceberg-rest.idleTimeout` | The timeout in ms of idle connections. | `30000` | No | 0.2.0 |
| `gravitino.auxService.iceberg-rest.requestHeaderSize` | The maximum size of an HTTP request. | `131072` | No | 0.2.0 |
| `gravitino.auxService.iceberg-rest.responseHeaderSize` | The maximum size of an HTTP response. | `131072` | No | 0.2.0 |
| `gravitino.auxService.iceberg-rest.customFilters` | Comma separated list of filter class names to apply to the APIs. | (none) | No | 0.4.0 |


The filter in the customFilters should be a standard javax servlet Filter.
Filter parameters can also be specified in the configuration, by setting config entries of the form `gravitino.auxService.iceberg-rest.<class name of filter>.param.<param name>=<value>`

:::caution
You must set `gravitino.auxService.iceberg-rest.httpPort` explicitly, like `9001`.
Expand Down
20 changes: 18 additions & 2 deletions server-common/src/main/java/com/datastrato/gravitino/server/web/JettyServer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import java.security.AccessController;
import java.security.PrivilegedAction;
import java.util.EnumSet;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.LinkedBlockingQueue;
Expand Down Expand Up @@ -104,7 +105,7 @@ public synchronized void initialize(
StringUtils.isNotBlank(serverConfig.getTrustStorePath()),
"If enables the authentication of the client, must set trustStorePath");
Preconditions.checkArgument(
StringUtils.isNotBlank(serverConfig.getTrustStorePasword()),
StringUtils.isNotBlank(serverConfig.getTrustStorePassword()),
"If enables the authentication of the client, must set trustStorePassword");
}
ServerConnector httpsConnector =
Expand All @@ -123,7 +124,7 @@ public synchronized void initialize(
serverConfig.getSupportedAlgorithms(),
serverConfig.isEnableClientAuth(),
serverConfig.getTrustStorePath(),
serverConfig.getTrustStorePasword(),
serverConfig.getTrustStorePassword(),
serverConfig.getTrustStoreType());
server.addConnector(httpsConnector);
} else {
Expand Down Expand Up @@ -433,4 +434,19 @@ public Thread run() {
public ThreadPool getThreadPool() {
return server.getThreadPool();
}

public void addCustomFilters(String pathSpec) {
for (String filterName : serverConfig.getCustomFilters()) {
if (StringUtils.isBlank(filterName)) {
continue;
}
FilterHolder filterHolder = new FilterHolder();
filterHolder.setClassName(filterName);
for (Map.Entry<String, String> entry :
serverConfig.getAllWithPrefix(String.format("%s.param.", filterName)).entrySet()) {
filterHolder.setInitParameter(entry.getKey(), entry.getValue());
}
servletContextHandler.addFilter(filterHolder, pathSpec, EnumSet.allOf(DispatcherType.class));
}
}
}
30 changes: 26 additions & 4 deletions server-common/src/main/java/com/datastrato/gravitino/server/web/JettyServerConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,13 @@ public final class JettyServerConfig {
.stringConf()
.createWithDefault("JKS");

public static final ConfigEntry<Optional<String>> CUSTOM_FILTERS =
new ConfigBuilder("customFilters")
.doc("Comma separated list of filter class names to apply to the APIs")
.version("0.4.0")
.stringConf()
.createWithOptional();

private final String host;

private final int httpPort;
Expand Down Expand Up @@ -199,7 +206,9 @@ public final class JettyServerConfig {
private final Set<String> enableCipherAlgorithms;
private final boolean enableClientAuth;
private final String trustStorePath;
private final String trustStorePasword;
private final String trustStorePassword;

private final Set<String> customFilters;
private final String trustStoreType;
private final Config internalConfig;

Expand Down Expand Up @@ -245,8 +254,13 @@ private JettyServerConfig(Map<String, String> configs) {
Sets.newHashSet(internalConfig.get(ENABLE_CIPHER_ALGORITHMS).split(SPLITTER)));
this.enableClientAuth = internalConfig.get(ENABLE_CLIENT_AUTH);
this.trustStorePath = internalConfig.get(SSL_TRUST_STORE_PATH);
this.trustStorePasword = internalConfig.get(SSL_TRUST_STORE_PASSWORD);
this.trustStorePassword = internalConfig.get(SSL_TRUST_STORE_PASSWORD);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An unrelated typo. Just fix by the way.

this.trustStoreType = internalConfig.get(SSL_TRUST_STORE_TYPE);
this.customFilters =
internalConfig
.get(CUSTOM_FILTERS)
.map(filters -> Collections.unmodifiableSet(Sets.newHashSet(filters.split(SPLITTER))))
.orElse(Collections.emptySet());
}

public static JettyServerConfig fromConfig(Config config, String prefix) {
Expand Down Expand Up @@ -330,14 +344,18 @@ public String getTrustStorePath() {
return trustStorePath;
}

public String getTrustStorePasword() {
return trustStorePasword;
public String getTrustStorePassword() {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An unrelated typo. Just fix by the way.

return trustStorePassword;
}

public String getTrustStoreType() {
return trustStoreType;
}

public Map<String, String> getAllWithPrefix(String prefix) {
return internalConfig.getConfigsWithPrefix(prefix);
}

private SSLContext getDefaultSSLContext() {
try {
return SSLContext.getDefault();
Expand Down Expand Up @@ -366,6 +384,10 @@ public Set<String> getSupportedAlgorithms() {
return supportedAlgorithms;
}

public Set<String> getCustomFilters() {
return customFilters;
}

@VisibleForTesting
Set<String> getSupportedCipherSuites() {
SSLContext context =
Expand Down
18 changes: 18 additions & 0 deletions server-common/src/test/java/com/datastrato/gravitino/server/web/TestJettyServerConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
package com.datastrato.gravitino.server.web;

import com.datastrato.gravitino.Config;
import com.datastrato.gravitino.config.ConfigBuilder;
import com.google.common.collect.Sets;
import java.util.Collections;
import java.util.Optional;
Expand Down Expand Up @@ -44,4 +45,21 @@ public void testCipherAlgorithms() {
Assertions.assertIterableEquals(
Sets.newHashSet(algorithm), jettyServerConfig.getSupportedAlgorithms());
}

@Test
public void testCustomFilters() {
Config emptyconfig = new Config() {};
JettyServerConfig jettyServerConfig = JettyServerConfig.fromConfig(emptyconfig, "");
Assertions.assertTrue(jettyServerConfig.getCustomFilters().isEmpty());

Config somethingConfig = new Config() {};
somethingConfig.set(JettyServerConfig.CUSTOM_FILTERS, Optional.of("1,2"));
somethingConfig.set(new ConfigBuilder("1.1").stringConf(), "test");
somethingConfig.set(new ConfigBuilder("1.2").stringConf(), "test");
jettyServerConfig = JettyServerConfig.fromConfig(somethingConfig, "");
Assertions.assertIterableEquals(
Sets.newHashSet("1", "2"), jettyServerConfig.getCustomFilters());
Assertions.assertTrue(jettyServerConfig.getAllWithPrefix("2.").isEmpty());
Assertions.assertEquals(2, jettyServerConfig.getAllWithPrefix("1.").size());
}
}
1 change: 1 addition & 0 deletions server/src/main/java/com/datastrato/gravitino/server/GravitinoServer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ protected void configure() {
server.addServlet(servlet, "/api/*");
Servlet configServlet = new ConfigServlet(serverConfig);
server.addServlet(configServlet, "/configs");
server.addCustomFilters("/api/*");
server.addFilter(new VersioningFilter(), "/api/*");
server.addFilter(new AuthenticationFilter(), "/api/*");
}
Expand Down
Loading