Skip to content

Commit

Permalink
[apache#3910] feat (trino-connector): Support load plugin in Trino De…
Browse files Browse the repository at this point in the history 
…velopmentServer startup mode (apache#3911)

### What changes were proposed in this pull request?

(Please outline the changes and how this PR fixes the issue.)

Support load plugin in Trino DevelopmentServer startup mode

Fix: apache#3910

### Does this PR introduce _any_ user-facing change?

Update docs

### How was this patch tested?

Manually test
  • Loading branch information
@diqiu50
diqiu50 authored Jul 1, 2024
1 parent c43d9f5 commit ff94f02
Show file tree
Hide file tree
Showing 11 changed files with 221 additions and 68 deletions.
2 changes: 0 additions & 2 deletions docs/trino-connector/configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,5 @@ This software is licensed under the Apache License version 2."
| gravitino.metalake | string | (none) | The `gravitino.metalake` defines which metalake in Gravitino server the Trino connector uses. Trino connector should set it at start, the value of `gravitino.metalake` needs to be a valid name, Trino connector can detect and load the metalake with catalogs, schemas and tables once created and keep in sync. | Yes | 0.2.0 |
| gravitino.uri | string | http://localhost:8090 | The `gravitino.uri` defines the connection URL of the Gravitino server, the default value is `http://localhost:8090`. Trino connector can detect and connect to Gravitino server once it is ready, no need to start Gravitino server beforehand. | No | 0.2.0 |
| gravitino.simplify-catalog-names | boolean | true | The `gravitino.simplify-catalog-names` setting omits the metalake prefix from catalog names when set to true. | NO | 0.5.0 |
| trino.jdbc.uri | string | jdbc:trino://localhost:8080 | The jdbc uri of current Trino server. | NO | 0.5.1 |
| trino.catalog.store | string | etc/catalog | The directory that stores the catalog configuration of current Trino. | NO | 0.5.1 |
| trino.jdbc.user | string | admin | The jdbc user name of current Trino. | NO | 0.5.1 |
| trino.jdbc.password | string | (none) | The jdbc password of current Trino. | NO | 0.5.1 |
2 changes: 1 addition & 1 deletion docs/trino-connector/development.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ then you can see the `gravitino-trino-connecor` source files and directories in

<artifactId>trino-gravitino</artifactId>
<packaging>trino-plugin</packaging>
<description>Trino - Graviton Connector</description>
<description>Trino - Gravitino Connector</description>

<properties>
<air.main.basedir>${project.parent.basedir}</air.main.basedir>
Expand Down
5 changes: 1 addition & 4 deletions docs/trino-connector/installation.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,13 @@ Please refer to the [Deploying Trino documentation](https://trino.io/docs/curren
Normally, the directory location is `Trino-server-<version>/plugin`, and the directory contains other catalogs used by Trino.
3. Add Trino JVM arguments `-Dlog4j.configurationFile=file:////etc/trino/log4j2.properties` to enable logging for the Gravitino connector.
4. Update Trino coordinator configuration.
You need to set `catalog.management=dynamic` and `catalog.store=file`
The config location is `Trino-server-<version>/etc/config.properteis`, and the contents like:
You need to set `catalog.management=dynamic`, The config location is `Trino-server-<version>/etc/config.properteis`, and the contents like:

```text
coordinator=true
node-scheduler.include-coordinator=true
http-server.http.port=8080
catalog.management=dynamic
catalog.store=file
discovery.uri=http://0.0.0.0:8080
```

Expand Down Expand Up @@ -87,7 +85,6 @@ node-scheduler.include-coordinator=true
http-server.http.port=8080
discovery.uri=http://localhost:8080
catalog.management=dynamic
catalog.store=file
```

### Configuring the Gravitino connector
Expand Down
1 change: 0 additions & 1 deletion docs/trino-connector/requirements.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,3 @@ To install and deploy the Gravitino connector, The following environmental setup
- Ensure that all nodes running Trino can access the real catalogs resources, such as Hive, Iceberg, MySQL, PostgreSQL, etc.
- Ensure that you have installed the following connectors in Trino: Hive, Iceberg, MySQL, PostgreSQL.
- Ensure that you have set the `catalog.management` to `dynamic` in the Trino coordinator configuration.
- Ensure that you have set the `catalog.store` to `file` in the Trino coordinator configuration.
2 changes: 2 additions & 0 deletions gradle/libs.versions.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ jersey = "2.41"
mockito = "4.11.0"
airlift-json = "237"
airlift-log = "231"
airlift-resolver = "1.6"
airlift-units = "1.8"
hive2 = "2.3.9"
hadoop2 = "2.10.2"
Expand Down Expand Up @@ -121,6 +122,7 @@ hadoop3-client = { group = "org.apache.hadoop", name = "hadoop-client", version.
hadoop3-mapreduce-client-core = { group = "org.apache.hadoop", name = "hadoop-mapreduce-client-core", version.ref = "hadoop3"}
hadoop3-minicluster = { group = "org.apache.hadoop", name = "hadoop-minicluster", version.ref = "hadoop-minikdc"}
airlift-json = { group = "io.airlift", name = "json", version.ref = "airlift-json"}
airlift-resolver = { group = "io.airlift.resolver", name = "resolver", version.ref = "airlift-resolver"}
airlift-units = { group = "io.airlift", name = "units", version.ref = "airlift-units"}
airlift-log = { group = "io.airlift", name = "log", version.ref = "airlift-log"}
httpclient5 = { group = "org.apache.httpcomponents.client5", name = "httpclient5", version.ref = "httpclient5" }
Expand Down
1 change: 1 addition & 0 deletions trino-connector/build.gradle.kts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ dependencies {
implementation(libs.commons.collections4)
implementation(libs.commons.lang3)
implementation(libs.trino.jdbc)
compileOnly(libs.airlift.resolver)
compileOnly(libs.trino.spi) {
exclude("org.apache.logging.log4j")
}
Expand Down
114 changes: 96 additions & 18 deletions trino-connector/src/main/java/com/datastrato/gravitino/trino/connector/GravitinoConfig.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,42 @@
import static com.datastrato.gravitino.trino.connector.GravitinoErrorCode.GRAVITINO_MISSING_CONFIG;

import io.trino.spi.TrinoException;
import java.io.FileInputStream;
import java.io.IOException;
import java.net.URI;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.logging.log4j.util.Strings;
import java.util.Properties;
import org.apache.commons.lang3.StringUtils;

public class GravitinoConfig {

// Trino config keys
private static final String TRINO_DISCOVERY_URI = "discovery.uri";
private static final String TRINO_CATALOG_CONFIG_DIR = "catalog.config-dir";
public static final String TRINO_PLUGIN_BUNDLES = "plugin.bundles";
public static final String TRINO_CATALOG_STORE = "catalog.store";
public static final String TRINO_CATALOG_MANAGEMENT = "catalog.management";

// Trino config default value
private static final String TRINO_CATALOG_CONFIG_DIR_DEFAULT_VALUE = "etc/catalog";
public static final String TRINO_CATALOG_STORE_DEFAULT_VALUE = "file";
public static final String TRINO_CATALOG_MANAGEMENT_DEFAULT_VALUE = "static";

// The trino configuration of etc/config.properties
public static final TrinoConfig trinoConfig = new TrinoConfig();

// Gravitino config keys
public static final String GRAVITINO_DYNAMIC_CONNECTOR = "__gravitino.dynamic.connector";
public static final String GRAVITINO_DYNAMIC_CONNECTOR_CATALOG_CONFIG =
"__gravitino.dynamic.connector.catalog.config";
private static final Map<String, ConfigEntry> CONFIG_DEFINITIONS = new HashMap<>();

private static final Map<String, ConfigEntry> CONFIG_DEFINITIONS = new HashMap<>();
private final Map<String, String> config;

// Gravitino config entity
private static final ConfigEntry GRAVITINO_URI =
new ConfigEntry(
"gravitino.uri", "The uri of the gravitino web server", "http://localhost:8090", false);
Expand All @@ -36,17 +57,6 @@ public class GravitinoConfig {
"true",
false);

private static final ConfigEntry TRINO_JDBC_URI =
new ConfigEntry(
"trino.jdbc.uri", "The jdbc uri of Trino server", "jdbc:trino://localhost:8080", false);

private static final ConfigEntry TRINO_CATALOG_STORE =
new ConfigEntry(
"trino.catalog.store",
"The directory stored the catalog configuration of Trino",
"etc/catalog",
false);

private static final ConfigEntry TRINO_JDBC_USER =
new ConfigEntry("trino.jdbc.user", "The jdbc user name of Trino", "admin", false);

Expand Down Expand Up @@ -98,12 +108,30 @@ public String getCatalogConfig() {
return config.get(GRAVITINO_DYNAMIC_CONNECTOR_CATALOG_CONFIG);
}

public String getTrinoURI() {
return config.getOrDefault(TRINO_JDBC_URI.key, TRINO_JDBC_URI.defaultValue);
public String getTrinoJdbcURI() {
String uriString = "";
if (config.containsKey(TRINO_DISCOVERY_URI)) {
uriString = config.get(TRINO_DISCOVERY_URI);
} else {
uriString = trinoConfig.getProperty(TRINO_DISCOVERY_URI);
}
try {
URI trinoURI = new URI(uriString);
return String.format("jdbc:trino://%s:%s", trinoURI.getHost(), trinoURI.getPort());
} catch (Exception e) {
throw new TrinoException(
GRAVITINO_MISSING_CONFIG,
"The Trino configuration of `discovery.uri` = " + uriString + " is not correct");
}
}

public String getCatalogStoreDirectory() {
return config.getOrDefault(TRINO_CATALOG_STORE.key, TRINO_CATALOG_STORE.defaultValue);
public String getCatalogConfigDirectory() {
if (config.containsKey(TRINO_CATALOG_CONFIG_DIR)) {
return config.get(TRINO_CATALOG_CONFIG_DIR);
} else {
return trinoConfig.getProperty(
TRINO_CATALOG_CONFIG_DIR, TRINO_CATALOG_CONFIG_DIR_DEFAULT_VALUE);
}
}

public String getTrinoUser() {
Expand All @@ -122,7 +150,7 @@ public String toCatalogConfig() {
stringList.add(String.format("\"%s\"='%s'", entry.getKey(), value));
}
}
return Strings.join(stringList, ',');
return StringUtils.join(stringList, ',');
}

static class ConfigEntry {
Expand All @@ -140,4 +168,54 @@ static class ConfigEntry {
CONFIG_DEFINITIONS.put(key, this);
}
}

static class TrinoConfig {

private final Properties properties;

public TrinoConfig() {
this.properties = new Properties();
try {
String configFileName = System.getProperty("config");
if (StringUtils.isEmpty(configFileName)) {
return;
}

properties.load(new FileInputStream(configFileName));
if (properties.containsKey(TRINO_CATALOG_STORE)
&& !properties
.getProperty(TRINO_CATALOG_STORE)
.equals(TRINO_CATALOG_STORE_DEFAULT_VALUE)) {
throw new TrinoException(
GRAVITINO_MISSING_CONFIG,
"Gravitino connector works only at catalog.store = file mode");
}

if (!properties.containsKey(TRINO_CATALOG_MANAGEMENT)
|| TRINO_CATALOG_MANAGEMENT_DEFAULT_VALUE.equals(
properties.getProperty(TRINO_CATALOG_MANAGEMENT))) {
throw new TrinoException(
GRAVITINO_MISSING_CONFIG,
"Gravitino connector works only at catalog.management = static mode");
}
} catch (IOException e) {
throw new TrinoException(
GRAVITINO_MISSING_CONFIG,
"Missing the Trino config file, please verify the jvm args '-Dconfig'");
}
}

String getProperty(String key) {
return properties.getProperty(key);
}

String getProperty(String key, String defaultValue) {

return properties.getProperty(key, defaultValue);
}

boolean contains(String key) {
return properties.containsKey(key);
}
}
}
Loading

0 comments on commit ff94f02

Please sign in to comment.