Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for ElasticSearch alias to read spans from (Resolves #68) #86

Merged
merged 1 commit into from
May 24, 2022
+20 −1
Merged

Add support for ElasticSearch alias to read spans from (Resolves #68) #86

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ Elasticsearch is used when `STORAGE=elasticsearch`.
* `ES_INDEX_PREFIX`: index prefix of Jaeger indices. By default unset.
* `ES_TIME_RANGE`: How far in the past the job should look to for spans, the maximum and default is `24h`.
Any value accepted by [date-math](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#date-math) can be used here, but the anchor is always `now`.
* `ES_USE_ALIASES`: Set to true to use index alias names to read from and write to.
Usually required when using rollover indices.

Example usage:

Expand Down
19 changes: 18 additions & 1 deletion ...c/main/java/io/jaegertracing/spark/dependencies/elastic/ElasticsearchDependenciesJob.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ public static final class Builder {
Boolean nodesWanOnly = Boolean.parseBoolean(Utils.getEnv("ES_NODES_WAN_ONLY", "false"));
String indexPrefix = Utils.getEnv("ES_INDEX_PREFIX", null);
String spanRange = Utils.getEnv("ES_TIME_RANGE", "24h");
Boolean useAliases = Boolean.parseBoolean(Utils.getEnv("ES_USE_ALIASES", "false"));

final Map<String, String> sparkProperties = new LinkedHashMap<>();

Expand Down Expand Up @@ -166,6 +167,7 @@ private static String getSystemPropertyAsFileResource(String key) {
private final SparkConf conf;
private final String indexPrefix;
private final String spanRange;
private final Boolean useAliases;

ElasticsearchDependenciesJob(Builder builder) {
this.day = builder.day;
Expand Down Expand Up @@ -195,6 +197,7 @@ private static String getSystemPropertyAsFileResource(String key) {
}
this.indexPrefix = builder.indexPrefix;
this.spanRange = builder.spanRange;
this.useAliases = builder.useAliases;
}

/**
Expand All @@ -209,7 +212,21 @@ private static String prefix(String prefix) {
}

public void run(String peerServiceTag) {
run(indexDate("jaeger-span"), indexDate("jaeger-dependencies") ,peerServiceTag);

String[] readIndices;
String[] writeIndex;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: writeIndices since it's an array?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True but also not really intuitive then.

The data type is an array, just because the indexDate helper does return an array (

spark-dependencies/jaeger-spark-dependencies-elasticsearch/src/main/java/io/jaegertracing/spark/dependencies/elastic/ElasticsearchDependenciesJob.java

Line 215 in a41f173

String[] indexDate(String index) {
). But this is only for read indices.

I gladly clean this up by either splitting the helpers for read and write or by removing (breaking) compatibility with the old prefixBefore19 format, but it's changing more than the feature I wanted to add. Just let me know that you'd rather like

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, that makes sense, given there'd only be one write index.


// use alias indices common when using index rollover
if (this.useAliases) {
readIndices = new String[]{prefix(indexPrefix) + "jaeger-span-read", prefixBefore19(indexPrefix) + "jaeger-span-read"};
writeIndex = new String[] {prefix(indexPrefix) + "jaeger-dependencies-write", prefixBefore19(indexPrefix) + "jaeger-dependencies-write"};
}
else {
readIndices = indexDate("jaeger-span");
writeIndex = indexDate("jaeger-dependencies");
}

run(readIndices, writeIndex, peerServiceTag);
}

String[] indexDate(String index) {
Expand Down