LOG-1195: Add Openshift Ingest Plugin for ES #2122

lukas-vlcek · 2021-05-18T12:42:53Z

Description

Configure Elasticsearch to install Openshift Ingest Plugin.

/cc @ewolinetz
/assign @jcantrill

Links

JIRA: https://issues.redhat.com/browse/LOG-1195

jcantrill

/hold

lukas-vlcek · 2021-05-18T13:07:38Z

/retest

lukas-vlcek · 2021-05-18T13:26:04Z

/retest

elasticsearch/fetch-artifacts-koji.yaml

lukas-vlcek · 2021-05-19T14:40:01Z

@jcantrill @ewolinetz I added default pipeline definition and initialization on ES startup. Also I defined the default pipeline in appropriate index templates. Please have a look again.

Also notice change wrt the initialization process:

0100 is for ingest pipeline initialization (they need to happen before index templates are pushed in)
0200 is for index templates initialization now due to ^^

Q: Please let me know if I need to add the new folder ingest_pipelines on this line here:

origin-aggregated-logging/elasticsearch/install.sh

Line 52 in ddbb5d0

chmod -R u+w,g+w ${HOME}/sgconfig ${ES_HOME}/index_templates

elasticsearch/ci-env.sh

elasticsearch/init/0100-seed-pipelines

elasticsearch/fetch-artifacts-koji.yaml

ewolinetz · 2021-05-19T16:05:20Z

Q: Please let me know if I need to add the new folder ingest_pipelines on this line here:

we shouldn't need to because we aren't updating the file at all. no need to do writes there.

lukas-vlcek · 2021-05-20T12:47:44Z

/test cluster-logging-operator-e2e

elasticsearch/Dockerfile

elasticsearch/fetch-artifacts-koji.yaml

jcantrill · 2021-06-07T15:43:11Z

elasticsearch/ci-env.sh

@@ -4,14 +4,19 @@ set -o xtrace

 PROMETHEUS_EXPORTER_URL=${PROMETHEUS_EXPORTER_URL:-$MAVEN_REPO_URL/org/elasticsearch/plugin/prometheus/prometheus-exporter/$PROMETHEUS_EXPORTER_VER/prometheus-exporter-$PROMETHEUS_EXPORTER_VER.zip}
 OPENDISTRO_URL=${OPENDISTRO_URL:-$MAVEN_REPO_URL/com/amazon/opendistroforelasticsearch/opendistro_security/$OPENDISTRO_VER/opendistro_security-$OPENDISTRO_VER.zip}
+INGEST_PLUGIN_URL=${INGEST_PLUGIN_URL:-$MAVEN_REPO_URL/org/elasticsearch/plugin/ingest/openshift/$INGEST_PLUGIN_VER/openshift_ingest-plugin-$INGEST_PLUGIN_VER.zip}


If we never intend to pubblish this to an official maven repo then this line should simply default to use the URL manipulation that you are doing for CI and/or pulling it straight from the Viaq repo

This plugin is not generalized enough atm. It makes not sense to push it into any public mvn repo.

correct. that is exactly my point. we never intend to publish to maven so this change is incorrect (e.g. $MAVEN_REPO_URL)

ewolinetz · 2021-06-09T14:04:03Z

@lukas-vlcek can we also fix the run.sh file to properly pick up the log4j2.properties file?

lukas-vlcek · 2021-06-09T19:09:45Z

@ewolinetz I think we can (though I would prefer separate PR).

vimalk78 · 2021-06-10T15:43:27Z

/retest

vimalk78 · 2021-06-11T14:31:50Z

/retest

vimalk78 · 2021-06-14T09:58:57Z

/retest

elasticsearch/fetch-artifacts-koji.yaml

vimalk78 · 2021-06-17T09:30:49Z

/retest

vimalk78 · 2021-06-17T09:35:35Z

/retest

vimalk78 · 2021-06-18T07:37:44Z

/retest

jcantrill · 2021-06-18T14:47:40Z

elasticsearch/Dockerfile.in

@@ -57,6 +58,7 @@ ENV ES_PATH_CONF=/etc/elasticsearch/ \
    JAVA_HOME=/usr/lib/jvm/jre \
    NODE_QUORUM=1 \
    PROMETHEUS_EXPORTER_VER=6.8.1.0-redhat-00001 \
+    INGEST_PLUGIN_VER=6.8.1.0-redhat-00001 \


6.8.1.0-redhat-00002

ref: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1638435

jcantrill · 2021-06-18T14:48:37Z

elasticsearch/fetch-artifacts-koji.yaml

@@ -1,3 +1,4 @@
 - nvr: org.elasticsearch-elasticsearch-6.8.1.redhat_00006-1
 - nvr: com.amazon.opendistroforelasticsearch-opendistro_security-0.10.0.4_redhat_00001-1
 - nvr: org.elasticsearch.plugin.prometheus-prometheus-exporter-6.8.1.0_redhat_00001-1
+- nvr: org.elasticsearch.plugin.ingest-openshift-ingest-plugin-6.8.1.0_redhat_00001-1


org.elasticsearch.plugin.ingest-openshift-ingest-plugin-6.8.1.0_redhat_00002-1

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1638435

jcantrill · 2021-06-18T14:50:04Z

elasticsearch/ci-env.sh


 if [[ "${OPENSHIFT_CI:-}" == "true" ]]; then
    # This flag is set during CI runs. If no ARG was passed in,
    # default to maven.org.
    export ES_ARCHIVE_URL=https://github.com/openshift/origin-aggregated-logging/releases/download/elasticsearch-oss-$ES_VER/elasticsearch-oss-$ES_VER.zip
    export OPENDISTRO_URL=https://github.com/openshift/origin-aggregated-logging/releases/download/opendistro_security-$OPENDISTRO_VER/opendistro_security-$OPENDISTRO_VER.zip

+    # TODO[lvlcek]: We need to download official releases from ViaQ repo. Getting it from lukas-vlcek is temporary.
+    INGEST_PLUGIN_VER=$(echo $INGEST_PLUGIN_VER | cut -d'-' -f1)
+    export INGEST_PLUGIN_URL=https://github.com/lukas-vlcek/elasticsearch-openshift-ingest-plugin/releases/download/v$INGEST_PLUGIN_VER/openshift-ingest-plugin-$INGEST_PLUGIN_VER.zip


Shouldnt we "correct" the upstream url to point to a release published here https://github.com/ViaQ/elasticsearch-openshift-ingest-plugin

Ok, I see what you mean @jcantrill
The thing is that we need to decide and control who, how and when cuts new releases for https://github.com/ViaQ/elasticsearch-openshift-ingest-plugin/

At this point it is just manual process. I created some tags in my clone for which I manually built releases and uploaded them to https://github.com/lukas-vlcek/elasticsearch-openshift-ingest-plugin/releases
(^^ This is what @vimalk78 used for his tests btw...)

Now, there is no process that is syncing tags between my clone and official ViaQ upstream.
We can either create tags manually or we can adopt similar approach that is used in prometheus-plugin – which means that whenever there is a commit that includes this property file without -SNAPSHOT suffix here ...
https://github.com/vvanholl/elasticsearch-prometheus-exporter/blob/master/gradle.properties#L3

... then a new tag is created automatically and a new release is built and uploaded automatically as well. See https://github.com/vvanholl/elasticsearch-prometheus-exporter/blob/master/.github/workflows/ci.yml#L39

We should decide how we want to control this. I would prefer fully automated approach (i.e. the samy way the prometheus plugin does it), but I am OK if you decide to do it differently.

//cc @ewolinetz

BTW, I do not think I have privs to create tags on ViaQ repo (the upstream). Which means if we decide we will create and upload release manually then I believe we need to ask @ewolinetz to do this.

jcantrill · 2021-06-18T14:51:06Z

elasticsearch/ci-env.sh

@@ -4,14 +4,19 @@ set -o xtrace

 PROMETHEUS_EXPORTER_URL=${PROMETHEUS_EXPORTER_URL:-$MAVEN_REPO_URL/org/elasticsearch/plugin/prometheus/prometheus-exporter/$PROMETHEUS_EXPORTER_VER/prometheus-exporter-$PROMETHEUS_EXPORTER_VER.zip}
 OPENDISTRO_URL=${OPENDISTRO_URL:-$MAVEN_REPO_URL/com/amazon/opendistroforelasticsearch/opendistro_security/$OPENDISTRO_VER/opendistro_security-$OPENDISTRO_VER.zip}
+INGEST_PLUGIN_URL=${INGEST_PLUGIN_URL:-$MAVEN_REPO_URL/org/elasticsearch/plugin/ingest/openshift/$INGEST_PLUGIN_VER/openshift_ingest-plugin-$INGEST_PLUGIN_VER.zip}


correct. that is exactly my point. we never intend to publish to maven so this change is incorrect (e.g. $MAVEN_REPO_URL)

Add default openshift_schema pipeline definiton for all [app-, infra-, audit-] indices.

lukas-vlcek · 2021-06-21T18:47:20Z

/test cluster-logging-operator-e2e

jcantrill · 2021-06-21T19:03:35Z

/approve

jcantrill · 2021-06-21T19:15:46Z

/hold cancel

ewolinetz · 2021-06-21T20:57:19Z

/lgtm

openshift-ci · 2021-06-21T20:57:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ewolinetz, jcantrill, lukas-vlcek

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ewolinetz,jcantrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lukas-vlcek · 2021-06-21T22:51:07Z

/test cluster-logging-operator-e2e

openshift-bot · 2021-06-22T02:07:30Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-22T02:33:30Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-22T05:06:51Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-22T05:18:50Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci bot assigned jcantrill May 18, 2021

openshift-ci bot requested a review from ewolinetz May 18, 2021 12:42

openshift-ci bot added the midstream/Dockerfile label May 18, 2021

jcantrill reviewed May 18, 2021

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 18, 2021

lukas-vlcek force-pushed the LOG-1195 branch from 6bd66b9 to 0e1d688 Compare May 18, 2021 13:07

lukas-vlcek force-pushed the LOG-1195 branch from 0e1d688 to 3899770 Compare May 18, 2021 13:25

jcantrill reviewed May 18, 2021

View reviewed changes

elasticsearch/fetch-artifacts-koji.yaml Outdated Show resolved Hide resolved

lukas-vlcek force-pushed the LOG-1195 branch 2 times, most recently from 30f67fe to d914dfa Compare May 19, 2021 14:38

lukas-vlcek mentioned this pull request May 19, 2021

Define default_pipeline processor for [app,infra,audit] indices. ViaQ/elasticsearch-templates#127

Open

ewolinetz reviewed May 19, 2021

View reviewed changes

elasticsearch/ci-env.sh Outdated Show resolved Hide resolved

ewolinetz reviewed May 19, 2021

View reviewed changes

elasticsearch/init/0100-seed-pipelines Outdated Show resolved Hide resolved

ewolinetz reviewed May 19, 2021

View reviewed changes

elasticsearch/fetch-artifacts-koji.yaml Outdated Show resolved Hide resolved

lukas-vlcek force-pushed the LOG-1195 branch 2 times, most recently from 495350a to 0494b79 Compare May 19, 2021 16:18

lukas-vlcek force-pushed the LOG-1195 branch 2 times, most recently from 191cfd7 to 75cf970 Compare June 1, 2021 16:09

jcantrill requested changes Jun 7, 2021

View reviewed changes

lukas-vlcek force-pushed the LOG-1195 branch from 75cf970 to 08a9f2f Compare June 10, 2021 11:38

lukas-vlcek force-pushed the LOG-1195 branch from 08a9f2f to 05f5872 Compare June 14, 2021 12:11

lukas-vlcek commented Jun 14, 2021

View reviewed changes

elasticsearch/fetch-artifacts-koji.yaml Outdated Show resolved Hide resolved

lukas-vlcek force-pushed the LOG-1195 branch from 05f5872 to 9c44c39 Compare June 14, 2021 13:53

jcantrill reviewed Jun 18, 2021

View reviewed changes

lukas-vlcek force-pushed the LOG-1195 branch 3 times, most recently from 7acaf74 to a0cfcac Compare June 21, 2021 15:00

LOG-1195: Add Openshift Ingest Plugin for ES

67cce9e

Add default openshift_schema pipeline definiton for all [app-, infra-, audit-] indices.

lukas-vlcek force-pushed the LOG-1195 branch from a0cfcac to 67cce9e Compare June 21, 2021 15:12

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 21, 2021

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 21, 2021

openshift-ci bot assigned ewolinetz Jun 21, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 21, 2021

openshift-merge-robot merged commit f9e5da9 into openshift:master Jun 22, 2021

lukas-vlcek deleted the LOG-1195 branch June 22, 2021 12:42

lukas-vlcek mentioned this pull request Jun 24, 2021

LOG-1195: Add Openshift Ingest Plugin for ES #2134

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LOG-1195: Add Openshift Ingest Plugin for ES #2122

LOG-1195: Add Openshift Ingest Plugin for ES #2122

lukas-vlcek commented May 18, 2021

jcantrill left a comment

lukas-vlcek commented May 18, 2021

lukas-vlcek commented May 18, 2021

lukas-vlcek commented May 19, 2021 •

edited

Loading

ewolinetz commented May 19, 2021

lukas-vlcek commented May 20, 2021

jcantrill Jun 7, 2021

lukas-vlcek Jun 9, 2021

jcantrill Jun 18, 2021

ewolinetz commented Jun 9, 2021

lukas-vlcek commented Jun 9, 2021

vimalk78 commented Jun 10, 2021

vimalk78 commented Jun 11, 2021

vimalk78 commented Jun 14, 2021

vimalk78 commented Jun 17, 2021

vimalk78 commented Jun 17, 2021

vimalk78 commented Jun 18, 2021

jcantrill Jun 18, 2021

jcantrill Jun 18, 2021

jcantrill Jun 18, 2021

lukas-vlcek Jun 21, 2021

lukas-vlcek Jun 21, 2021

jcantrill Jun 18, 2021

lukas-vlcek commented Jun 21, 2021

jcantrill commented Jun 21, 2021

jcantrill commented Jun 21, 2021

ewolinetz commented Jun 21, 2021

openshift-ci bot commented Jun 21, 2021

lukas-vlcek commented Jun 21, 2021

openshift-bot commented Jun 22, 2021

openshift-bot commented Jun 22, 2021

openshift-bot commented Jun 22, 2021

openshift-bot commented Jun 22, 2021

LOG-1195: Add Openshift Ingest Plugin for ES #2122

LOG-1195: Add Openshift Ingest Plugin for ES #2122

Conversation

lukas-vlcek commented May 18, 2021

Description

Links

jcantrill left a comment

Choose a reason for hiding this comment

lukas-vlcek commented May 18, 2021

lukas-vlcek commented May 18, 2021

lukas-vlcek commented May 19, 2021 • edited Loading

ewolinetz commented May 19, 2021

lukas-vlcek commented May 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ewolinetz commented Jun 9, 2021

lukas-vlcek commented Jun 9, 2021

vimalk78 commented Jun 10, 2021

vimalk78 commented Jun 11, 2021

vimalk78 commented Jun 14, 2021

vimalk78 commented Jun 17, 2021

vimalk78 commented Jun 17, 2021

vimalk78 commented Jun 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukas-vlcek commented Jun 21, 2021

jcantrill commented Jun 21, 2021

jcantrill commented Jun 21, 2021

ewolinetz commented Jun 21, 2021

openshift-ci bot commented Jun 21, 2021

lukas-vlcek commented Jun 21, 2021

openshift-bot commented Jun 22, 2021

openshift-bot commented Jun 22, 2021

openshift-bot commented Jun 22, 2021

openshift-bot commented Jun 22, 2021

lukas-vlcek commented May 19, 2021 •

edited

Loading