Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize file configuration #5399

Merged
merged 12 commits into from
Aug 3, 2023
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ dependencyCheck {
"annotationProcessor",
"animalsniffer",
"spotless-1972451482", // spotless-1972451482 is a weird configuration that's only added in jaeger-proto
"js2p",
"jmhAnnotationProcessor",
"jmhCompileClasspath",
"jmhRuntimeClasspath",
Expand Down
109 changes: 109 additions & 0 deletions sdk-extensions/incubator/build.gradle.kts
Original file line number Diff line number Diff line change
@@ -1,8 +1,13 @@
import de.undercouch.gradle.tasks.download.Download

plugins {
id("otel.java-conventions")
id("otel.publish-conventions")

id("otel.animalsniffer-conventions")

id("de.undercouch.download")
id("org.jsonschema2pojo")
}

// SDK modules that are still being developed.
Expand All @@ -19,8 +24,112 @@ dependencies {
implementation(project(":sdk-extensions:autoconfigure-spi"))
implementation("org.snakeyaml:snakeyaml-engine")

// io.opentelemetry.sdk.extension.incubator.fileconfig
implementation("com.fasterxml.jackson.core:jackson-databind")
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this eventually make jackson a hard dependency for the sdk or just this extension?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jackson-databind gets CVEs very often, I'd sleep better if we don't include it in the SDK (at some point); would jackson-jr be able to handle the generated POJOs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would likely eventually be merged into the autoconfigure module, rather than the SDK itself.

I can investigate jackson-jr.

Also, there's the more hands on option of doing something like we do with YAML view configuration, where we use snake YAML to parse to a generic type, then extract the fields manually. Maybe we use jackson-databind / jackson-jr to prototype, and consider switching to the manual method when its looking more complete.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering this is for configuration, which is likely only done at startup I feel we should go with a potentially less performant option and not use jackson in exchange for fewer dependencies and CVE exposure.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's less of the performance I'm worried about and more the additional complexity to manually do the parsing that jackson does automatically. Additionally, there is likely to be a fair amount of churn in the early versions of the configuration schema - I'm happy enough writing manual parsing code once, but won't sign up to rewrite the manual parsing code for each of the early iterations.

implementation("org.yaml:snakeyaml:1.31")

testImplementation(project(":sdk:testing"))
testImplementation(project(":sdk-extensions:autoconfigure"))

testImplementation("com.google.guava:guava-testlib")
}

// The following tasks download the JSON Schema files from open-telemetry/opentelemetry-configuration and generate classes from the type definitions which are used with jackson-databind to parse JSON / YAML to the configuration schema.
// The sequence of tasks is:
// 1. downloadConfigurationSchema - download configuration schema from open-telemetry/opentelemetry-configuration
// 2. unzipConfigurationSchema - unzip the configuration schema archive contents to $buildDir/configuration/
// 3. generateJsonSchema2Pojo - generate java POJOs from the configuration schema
// 4. jsonSchema2PojoPostProcessing - perform various post processing on the generated POJOs, e.g. replace javax.annotation.processing.Generated with javax.annotation.Generated, add @SuppressWarning("rawtypes") annotation
// 5. overwriteJs2p - overwrite original generated classes with versions containing updated @Generated annotation
// 6. deleteJs2pTmp - delete tmp directory
// ... proceed with normal sourcesJar, compileJava, etc

// TODO(jack-berg): update ref to be released version when available
val configurationRef = "2107dbb6f2a6c99fe2f55d550796ee7e2286fd1d"
val configurationRepoZip = "https://github.com/open-telemetry/opentelemetry-configuration/archive/$configurationRef.zip"

val downloadConfigurationSchema by tasks.registering(Download::class) {
src(configurationRepoZip)
dest("$buildDir/configuration/opentelemetry-configuration.zip")
overwrite(false)
}

val unzipConfigurationSchema by tasks.registering(Copy::class) {
dependsOn(downloadConfigurationSchema)

from(zipTree(downloadConfigurationSchema.get().dest))
eachFile(closureOf<FileCopyDetails> {
// Remove the top level folder "/opentelemetry-configuration-$configurationRef"
val pathParts = path.split("/")
path = pathParts.subList(1, pathParts.size).joinToString("/")
})
into("$buildDir/configuration/")
}

jsonSchema2Pojo {
sourceFiles = setOf(file("$buildDir/configuration/schema"))
targetDirectory = file("$buildDir/generated/sources/js2p/java/main")
targetPackage = "io.opentelemetry.sdk.extension.incubator.fileconfig.internal.model"

// Clear old source files to avoid contaminated source dir when updating
removeOldOutput = true

// Prefer builders to setters
includeSetters = false
generateBuilders = true

// Use title field to generate class name, instead of default which is based on filename / propertynames
useTitleAsClassname = true

// Force java 9+ @Generated annotation, since java 8 @Generated annotation isn't detected by
// jsonSchema2Pojo and annotation is skipped altogether
targetVersion = "1.9"
}

val generateJsonSchema2Pojo = tasks.getByName("generateJsonSchema2Pojo")
generateJsonSchema2Pojo.dependsOn(unzipConfigurationSchema)

val jsonSchema2PojoPostProcessing by tasks.registering(Copy::class) {
dependsOn(generateJsonSchema2Pojo)

from("$buildDir/generated/sources/js2p")
into("$buildDir/generated/sources/js2p-tmp")
filter {
it
// Replace java 9+ @Generated annotation with java 8 version
.replace("import javax.annotation.processing.Generated", "import javax.annotation.Generated")
// Add @SuppressWarnings("rawtypes") annotation to address raw types used in jsonschema2pojo builders
.replace("@Generated(\"jsonschema2pojo\")", "@Generated(\"jsonschema2pojo\")\n@SuppressWarnings(\"rawtypes\")")
}
}
val overwriteJs2p by tasks.registering(Copy::class) {
dependsOn(jsonSchema2PojoPostProcessing)

from("$buildDir/generated/sources/js2p-tmp")
into("$buildDir/generated/sources/js2p")
}
val deleteJs2pTmp by tasks.registering(Delete::class) {
dependsOn(overwriteJs2p)

delete("$buildDir/generated/sources/js2p-tmp/")
}

tasks.getByName("compileJava").dependsOn(deleteJs2pTmp)
tasks.getByName("sourcesJar").dependsOn(deleteJs2pTmp)

// Exclude jsonschema2pojo generated sources from checkstyle
tasks.named<Checkstyle>("checkstyleMain") {
exclude("**/fileconfig/internal/model/**")
}

tasks {
withType<Test>().configureEach {
environment(
mapOf(
// Expose the kitchen sink example file to tests
"CONFIG_EXAMPLE_DIR" to "$buildDir/configuration/examples/"
)
)
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
/*
* Copyright The OpenTelemetry Authors
* SPDX-License-Identifier: Apache-2.0
*/

package io.opentelemetry.sdk.extension.incubator.fileconfig;

import com.fasterxml.jackson.databind.ObjectMapper;
import io.opentelemetry.sdk.extension.incubator.fileconfig.internal.model.OpenTelemetryConfiguration;
import java.io.InputStream;
import org.yaml.snakeyaml.Yaml;

class ConfigurationReader {

private static final ObjectMapper MAPPER = new ObjectMapper();
private static final Yaml YAML = new Yaml();

private ConfigurationReader() {}

/** Parse the {@code configuration} YAML and return the {@link OpenTelemetryConfiguration}. */
static OpenTelemetryConfiguration parse(InputStream configuration) {
Object yamlObj = YAML.load(configuration);
return MAPPER.convertValue(yamlObj, OpenTelemetryConfiguration.class);
}
}
Loading