[AUTOCUT] Distribution Build Failed for skills-2.12.0

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
The distribution build for skills has failed for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9319/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
The distribution build for skills has failed for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9325/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[opensearch-ci-bot]

Closing the issue as the distribution build for skills has passed for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9328/consoleFull

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
skills failed during the distribution build for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9350/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
skills failed during the distribution build for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9353/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
skills failed during the distribution build for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9352/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[gaiksaya]

Failure here:

> Could not resolve all files for configuration ':zipArchive'.
   > Could not find org.opensearch.plugin:opensearch-job-scheduler:2.12.0.0.
     Searched in the following locations:
       - file:/home/ci-runner/.m2/repository/org/opensearch/plugin/opensearch-job-scheduler/2.12.0.0/opensearch-job-scheduler-2.12.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/opensearch-job-scheduler/2.12.0.0/opensearch-job-scheduler-2.12.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/plugin/opensearch-job-scheduler/2.12.0.0/opensearch-job-scheduler-2.12.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-job-scheduler/2.12.0.0/opensearch-job-scheduler-2.12.0.0.pom
     Required by:
         project :
   > Could not find org.opensearch.plugin:opensearch-anomaly-detection:2.12.0.0.
     Searched in the following locations:
       - file:/home/ci-runner/.m2/repository/org/opensearch/plugin/opensearch-anomaly-detection/2.12.0.0/opensearch-anomaly-detection-2.12.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/opensearch-anomaly-detection/2.12.0.0/opensearch-anomaly-detection-2.12.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/plugin/opensearch-anomaly-detection/2.12.0.0/opensearch-anomaly-detection-2.12.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-anomaly-detection/2.12.0.0/opensearch-anomaly-detection-2.12.0.0.pom
     Required by:
         project :
   > Could not find org.opensearch.plugin:opensearch-sql-plugin:2.12.0.0.
     Searched in the following locations:
       - file:/home/ci-runner/.m2/repository/org/opensearch/plugin/opensearch-sql-plugin/2.12.0.0/opensearch-sql-plugin-2.12.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/opensearch-sql-plugin/2.12.0.0/opensearch-sql-plugin-2.12.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/plugin/opensearch-sql-plugin/2.12.0.0/opensearch-sql-plugin-2.12.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-sql-plugin/2.12.0.0/opensearch-sql-plugin-2.12.0.0.pom
     Required by:
         project :
   > Could not find org.opensearch.plugin:neural-search:2.12.0.0.
     Searched in the following locations:
       - file:/home/ci-runner/.m2/repository/org/opensearch/plugin/neural-search/2.12.0.0/neural-search-2.12.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/neural-search/2.12.0.0/neural-search-2.12.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/plugin/neural-search/2.12.0.0/neural-search-2.12.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/neural-search/2.12.0.0/neural-search-2.12.0.0.pom
     Required by:
         project :
   > Could not find org.opensearch.plugin:alerting:2.12.0.0.
     Searched in the following locations:
       - file:/home/ci-runner/.m2/repository/org/opensearch/plugin/alerting/2.12.0.0/alerting-2.12.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/alerting/2.12.0.0/alerting-2.12.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/plugin/alerting/2.12.0.0/alerting-2.12.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/alerting/2.12.0.0/alerting-2.12.0.0.pom
     Required by:
         project :

I believe this is because plugins are not publishing to maven local via build.sh scripts.

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
skills failed during the distribution build for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9356/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[gaiksaya]

Looks like skills repo is also running flow blown integ-test on assemble task:

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.integTest.NeuralSparseSearchToolIT.testNeuralSparseSearchToolInFlowAgent_withIllegalIndexField_thenThrowException" -Dtests.seed=26F47D2387E68520 -Dtests.security.manager=false -Dtests.locale=hu-HU -Dtests.timezone=Pacific/Tahiti -Druntime.java=21

org.opensearch.integTest.NeuralSparseSearchToolIT > testNeuralSparseSearchToolInFlowAgent_withIllegalIndexField_thenThrowException FAILED
    java.lang.AssertionError: 
    Expected: (a string containing "no such index [test_index2]" and a string containing "IndexNotFoundException")
         but: a string containing "IndexNotFoundException" was "method [POST], host [http://127.0.0.1:43387], URI [/_plugins/_ml/agents/eX7Fg40BtVoYv3-Sa4xb/_execute], status line [HTTP/1.1 404 Not Found]
    {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [test_index2]","index":"test_index2","resource.id":"test_index2","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [test_index2]","index":"test_index2","resource.id":"test_index2","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}"
        at __randomizedtesting.SeedInfo.seed([26F47D2387E68520:5084A50F8F4BC3CD]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
        at org.opensearch.integTest.NeuralSparseSearchToolIT.testNeuralSparseSearchToolInFlowAgent_withIllegalIndexField_thenThrowException(NeuralSparseSearchToolIT.java:157)

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.integTest.NeuralSparseSearchToolIT.testNeuralSparseSearchToolInFlowAgent_withIllegalModelIdField_thenThrowException" -Dtests.seed=26F47D2387E68520 -Dtests.security.manager=false -Dtests.locale=hu-HU -Dtests.timezone=Pacific/Tahiti -Druntime.java=21

https://build.ci.opensearch.org/blue/rest/organizations/jenkins/pipelines/distribution-build-opensearch/runs/9356/nodes/69/steps/455/log/?start=0

cc: @zane-neo @peterzhuamazon @prudhvigodithi

[prudhvigodithi]

Hey @owaiskazi19, looks to me like the build failure issue posted above is the same what we saw for flow-framework addressed with this PR https://github.com/opensearch-project/flow-framework/pull/418/files, can you take a look and fix it please?
Thank you

Adding @minalsha @dbwiddis

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
skills failed during the distribution build for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9357/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[owaiskazi19]

Hey @owaiskazi19, looks to me like the build failure issue posted above is the same what we saw for flow-framework addressed with this PR https://github.com/opensearch-project/flow-framework/pull/418/files, can you take a look and fix it please? Thank you

Adding @minalsha @dbwiddis

@prudhvigodithi Skills repo is dependent on these plugins. I see ml-commons publishToMavenLocal here and AD does too. Why don't we see the error for ml-commons but rest of the plugins using zipArchive?

[prudhvigodithi]

@owaiskazi19 from the code I see skills repo is dependent the external plugin zip's not jars, the publishToMavenLocal only publishes the jars to maven local but not the zips, to publish the zips the maven local the task publishPluginZipPublicationToMavenLocal should be explicitly called by the plugin.

[owaiskazi19]

@owaiskazi19 from the code I see skills repo is dependent the external plugin zip's not jars, the publishToMavenLocal only publishes the jars to maven local but not the zips, to publish the zips the maven local the task publishPluginZipPublicationToMavenLocal should be explicitly called by the plugin.

Does ml-commons calls publishPluginZipPublicationToMavenLocal?

[prudhvigodithi]

@owaiskazi19 I would suggest to explore few options, AFAIK running publishToMavenLocal publishes all publications to maven local which is called by ml-commons. It wont publish if explicitly denied which i done by job-scheduler. It also wont publish when neither of publishToMavenLocal or publishPluginZipPublicationToMavenLocal are called.
Thanks

[owaiskazi19]

PR is merged and backported. Issue should be resolved now @prudhvigodithi

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
skills failed during the distribution build for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9358/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[opensearch-ci-bot]

Closing the issue as the distribution build for skills has passed for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9359/display/redirect

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
skills failed during the distribution build for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9367/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[opensearch-ci-bot]

Received Error: Error building skills, retry with: ./build.sh manifests/2.12.0/opensearch-2.12.0.yml --component skills.
skills failed during the distribution build for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9369/display/redirect.
The failed build stage will be marked as unstable(!). Please see ./build.sh step for more details

[opensearch-ci-bot]

Closing the issue as the distribution build for skills has passed for version: 2.12.0.
Please see build log at https://build.ci.opensearch.org/job/distribution-build-opensearch/9373/display/redirect