Temporary adding Apache Lucene repositories for snapshots #3042

reta · 2022-04-22T13:19:19Z

Signed-off-by: Andriy Redko [email protected]

Description

Temporary adding Apache Lucene repositories for snapshots

Issues Resolved

Fixes build issues (after #2924)

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

reta · 2022-04-22T13:20:00Z

gradle/code-coverage.gradle

@@ -11,6 +11,13 @@ apply plugin: 'jacoco'
 repositories {
  mavenCentral()
  gradlePluginPortal()
+  // TODO: Find the way to use the repositories from RepositoriesSetupPlugin


@nknize @dblock I don't have better solution at the moment but this fix unblocks the builds

nknize

👍 I have no issue with this for now. Thanks for looking!

opensearch-ci-bot · 2022-04-22T13:33:00Z

❌ Gradle Check failure 6a875fe0b282b6e5fb1785e4b154a58ebe85dc15
Log 4714

Reports 4714

reta · 2022-04-22T13:57:19Z

x Gradle Check failure 6a875fe Log 4714

Reports 4714

For the record, org.apache.lucene:lucene-analysis-smartcn:9.2.0-snapshot-f4f1f70 is not considered to be a 'snapshot' by Maven (it does not end with snapshot)

opensearch-ci-bot · 2022-04-22T14:24:40Z

❌ Gradle Check failure dfd54c6e9c7b17c3e3fd071978cc2f66172a177c
Log 4715

Reports 4715

reta · 2022-04-22T14:27:40Z

x Gradle Check failure dfd54c6 Log 4715

Reports 4715

Hm ....

Tests with failures:

org.opensearch.snapshots.SharedClusterSnapshotRestoreIT.testDataFileCorruptionDuringRestore
org.opensearch.recovery.TruncatedRecoveryIT.testCancelRecoveryAndResume
org.opensearch.search.query.SearchQueryIT.testAllDocsQueryString
org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testRestoreShrinkIndex

reta · 2022-04-22T14:27:53Z

start gradle check

dblock · 2022-04-22T14:32:25Z

start gradle check

dblock · 2022-04-22T14:33:06Z

@peterzhuamazon could you please check that @reta can do gradle check starts too? I think it's a separate permissions list

reta · 2022-04-22T14:37:18Z

@peterzhuamazon could you please check that @reta can do gradle check starts too? I think it's a separate permissions list

I can, thanks for checking @dblock !

Signed-off-by: Andriy Redko <[email protected]>

opensearch-ci-bot · 2022-04-22T15:56:42Z

❌ Gradle Check failure dfd54c6e9c7b17c3e3fd071978cc2f66172a177c
Log 4716

Reports 4716

opensearch-ci-bot · 2022-04-22T15:56:52Z

❌ Gradle Check failure dfd54c6e9c7b17c3e3fd071978cc2f66172a177c
Log 4717

Reports 4717

opensearch-ci-bot · 2022-04-22T16:17:48Z

❌ Gradle Check failure ef05839
Log 4719

Reports 4719

reta · 2022-04-22T16:22:27Z

x Gradle Check failure ef05839 Log 4719

Reports 4719

It does not look good, seem like we have introduce instability somewhere ...

Tests with failures:
 - org.opensearch.action.search.TransportSearchIT.testCircuitBreakerReduceFail
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValuedFieldOrderedBySingleValueSubAggregationAscAndTermsAsc
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValuedFieldOrderedBySubAggregationAscMultiHierarchyLevels
 - org.opensearch.search.aggregations.metrics.HDRPercentilesIT.testOrderBySubAggregation
 - org.opensearch.indexing.IndexActionIT.testAutoGenerateIdNoDuplicates
 - org.opensearch.indices.IndicesRequestCacheIT.testCacheAggs
 - org.opensearch.snapshots.CloneSnapshotIT.testShardClone
 - org.opensearch.search.functionscore.FunctionScoreFieldValueIT.testFieldValueFactor
 - org.opensearch.search.basic.TransportSearchFailuresIT.testFailedSearchWithWrongQuery
 - org.opensearch.recovery.TruncatedRecoveryIT.testCancelRecoveryAndResume
 - org.opensearch.search.functionscore.FunctionScoreFieldValueIT.testFieldValueFactorExplain

2616 tests completed, 11 failed, 7 skipped

@tushar-kharbanda72 I think we have uncovered edge cases in #2639, not sure where it is coming from


    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=925, name=opensearch[node_s0][transport_worker][T#2], state=RUNNABLE, group=TGRP-FunctionScoreFieldValueIT]

        Caused by:
        java.lang.AssertionError: No thread should be marked active when task finishes
            at __randomizedtesting.SeedInfo.seed([93880727DE8A03B5]:0)

  2> أبر 23, 2022 1:44:35 ص com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
  2> WARNING: Uncaught exception in thread: Thread[opensearch[node_t1][search][T#2],5,TGRP-TruncatedRecoveryIT]
  2> java.lang.AssertionError: No thread should be marked active when task finishes
  2> 	at __randomizedtesting.SeedInfo.seed([93880727DE8A03B5]:0)
  2> 	at org.opensearch.tasks.TaskResourceTrackingService.stopTracking(TaskResourceTrackingService.java:117)
  2> 	at org.opensearch.tasks.TaskManager.unregister(TaskManager.java:214)

reta · 2022-04-22T16:26:30Z

start gradle check

opensearch-ci-bot · 2022-04-22T17:04:26Z

❌ Gradle Check failure ef05839
Log 4720

Reports 4720

reta · 2022-04-22T17:13:55Z

x Gradle Check failure ef05839 Log 4720

Reports 4720

Uh ...

Tests with failures:
 - org.opensearch.search.aggregations.bucket.terms.StringTermsIT.testSingleValuedFieldOrderedBySingleBucketSubAggregationAsc
 - org.opensearch.index.FinalPipelineIT.testFinalPipelineOfNewDestinationIsInvoked
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testOtherDocCount
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testMultiValuedFieldWithValueScript
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testMultiValuedFieldWithValueScriptNotUnique
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testScriptCaching
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testPartiallyUnmapped
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValuedFieldOrderedBySingleValueSubAggregationAscWithTermsSubAgg
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValueFieldWithPartitionedFiltering
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValuedFieldOrderedBySingleBucketSubAggregationAsc
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValuedFieldOrderedBySubAggregationAscMultiHierarchyLevels
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValuedFieldOrderedByMultiValueSubAggregationAsc
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValuedFieldOrderedByMultiValueSubAggregationDesc
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValuedFieldOrderedByMultiValueExtendedStatsAsc
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testPartiallyUnmappedWithFormat
 - org.opensearch.search.aggregations.bucket.LongTermsIT.testSingleValuedFieldWithValueScript
 - org.opensearch.search.fields.SearchFieldsIT.testStoredFields
 - org.opensearch.routing.AliasRoutingIT.testAliasSearchRouting
 - org.opensearch.search.aggregations.bucket.LongTermsIT.classMethod
 - org.opensearch.search.aggregations.bucket.MinDocCountIT.testLongScriptCountAsc
 - org.opensearch.search.aggregations.bucket.MinDocCountIT.testDoubleTermAsc
 - org.opensearch.search.aggregations.bucket.MinDocCountIT.testStringCountAscWithInclude
 - org.opensearch.search.profile.query.QueryProfilerIT.testProfileQuery
 - org.opensearch.search.profile.query.QueryProfilerIT.testPhrase
 - org.opensearch.search.profile.query.QueryProfilerIT.testSimpleMatch
 - org.opensearch.search.profile.query.QueryProfilerIT.testDisMaxRange
 - org.opensearch.search.profile.query.QueryProfilerIT.testNoProfile
 - org.opensearch.search.profile.query.QueryProfilerIT.testCollapsingBool

2> REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.profile.query.QueryProfilerIT.testProfileQuery" -Dtests.seed=2E6E80CA89F5582E -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=et -Dtests.timezone=Asia/Harbin -Druntime.java=17
  2> MasterNotDiscoveredException[null]
        at app//org.opensearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:275)
        at app//org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:369)
        at app//org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:287)
        at app//org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:692)
        at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:741)
        at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at [email protected]/java.lang.Thread.run(Thread.java:833)

    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1030, name=opensearch[node_s1][transport_worker][T#2], state=RUNNABLE, group=TGRP-QueryProfilerIT]

        Caused by:
        java.lang.AssertionError: No thread should be marked active when task finishes
            at __randomizedtesting.SeedInfo.seed([2E6E80CA89F5582E]:0)
            at org.opensearch.tasks.TaskResourceTrackingService.stopTracking(TaskResourceTrackingService.java:117)

reta · 2022-04-22T17:15:20Z

@dblock @nknize I cannot get green build :( thinking about rolling back #2639

reta · 2022-04-22T17:15:29Z

start gradle check

opensearch-ci-bot · 2022-04-22T17:45:05Z

❌ Gradle Check failure ef05839
Log 4722

Reports 4722

reta · 2022-04-22T17:49:52Z

start gradle check

dblock · 2022-04-22T18:01:54Z

Are you saying we introduced this break in #2639? Paging @tushar-kharbanda72, let's see if we can fix this before we rollback?

reta · 2022-04-22T18:05:01Z

Are you saying we introduced this break in #2639? Paging @tushar-kharbanda72, let's see if we can fix this before we rollback?

It looks like it, many exceptions in logs (java.lang.AssertionError: No thread should be marked active when task finishes):


    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1030, name=opensearch[node_s1][transport_worker][T#2], state=RUNNABLE, group=TGRP-QueryProfilerIT]

        Caused by:
        java.lang.AssertionError: No thread should be marked active when task finishes
            at __randomizedtesting.SeedInfo.seed([2E6E80CA89F5582E]:0)
            at org.opensearch.tasks.TaskResourceTrackingService.stopTracking(TaskResourceTrackingService.java:117)

opensearch-ci-bot · 2022-04-22T18:23:37Z

❌ Gradle Check failure ef05839
Log 4723

Reports 4723

reta · 2022-04-22T18:31:38Z

start gradle check

dblock · 2022-04-22T18:36:24Z

I am ok reverting that PR if it helps @reta.

#3046

reta · 2022-04-22T18:45:19Z

@dblock we may deadlock here - we need this guy to unblock the Lucene snapshots issue, the #3046 may fail because of that

opensearch-ci-bot · 2022-04-22T19:07:29Z

✅ Gradle Check success ef05839
Log 4724

Reports 4724

nknize · 2022-04-22T19:17:33Z

Just catching up on this.... I see a gradle success w/ no commits or rebases since the last massive failure. Care to summarize?

reta · 2022-04-22T19:19:26Z

Just catching up on this.... I see a gradle success w/ no commits or rebases since the last massive failure. Care to summarize?

Yeah, the builds are super unstable, in report, a lot of exceptions like #3042 (comment) pop up

nknize · 2022-04-22T19:21:21Z

Yeah, the builds are super unstable, in report

What do you mean by "in report"? Do you mean this as in the general "build report" or are you referring to the added code coverage report?

reta · 2022-04-22T19:22:12Z

Yeah, the builds are super unstable, in report

What do you mean by "in report"? Do you mean this as in the general "build report" or are you referring to the added code coverage report?

Oops, sorry, in build logs and build reports

Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit f006afa)

) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit f006afa) Co-authored-by: Andriy Redko <[email protected]>

reta requested a review from a team as a code owner April 22, 2022 13:19

reta commented Apr 22, 2022

View reviewed changes

nknize approved these changes Apr 22, 2022

View reviewed changes

reta force-pushed the fix.build branch from 6a875fe to dfd54c6 Compare April 22, 2022 13:56

dblock approved these changes Apr 22, 2022

View reviewed changes

This was referenced Apr 22, 2022

Build: do not ship jdk when no-jdk option set #3039

Merged

Adds the replication type index setting, alongside a formal notion of feature flags #3037

Merged

Temporary adding Apache Lucene repositories for snapshots

ef05839

Signed-off-by: Andriy Redko <[email protected]>

reta force-pushed the fix.build branch from dfd54c6 to ef05839 Compare April 22, 2022 15:43

reta added the backport 2.x Backport to 2.x branch label Apr 22, 2022

nknize mentioned this pull request Apr 22, 2022

Revert "Support task resource tracking in OpenSearch" #3046

Merged

reta merged commit f006afa into opensearch-project:main Apr 22, 2022

opensearch-trigger-bot bot pushed a commit that referenced this pull request Apr 22, 2022

Temporary adding Apache Lucene repositories for snapshots (#3042)

7fbf256

Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit f006afa)

opensearch-trigger-bot bot mentioned this pull request Apr 22, 2022

[Backport 2.x] Temporary adding Apache Lucene repositories for snapshots #3047

Merged

nknize pushed a commit that referenced this pull request Apr 22, 2022

Temporary adding Apache Lucene repositories for snapshots (#3042) (#3047

c7e8b6b

) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit f006afa) Co-authored-by: Andriy Redko <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Temporary adding Apache Lucene repositories for snapshots #3042

Temporary adding Apache Lucene repositories for snapshots #3042

reta commented Apr 22, 2022

reta Apr 22, 2022

nknize left a comment

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

reta commented Apr 22, 2022

dblock commented Apr 22, 2022

dblock commented Apr 22, 2022

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022 •

edited

Loading

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

reta commented Apr 22, 2022

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

dblock commented Apr 22, 2022

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

dblock commented Apr 22, 2022 •

edited

Loading

reta commented Apr 22, 2022 •

edited

Loading

opensearch-ci-bot commented Apr 22, 2022

nknize commented Apr 22, 2022

reta commented Apr 22, 2022

nknize commented Apr 22, 2022

reta commented Apr 22, 2022

Temporary adding Apache Lucene repositories for snapshots #3042

Temporary adding Apache Lucene repositories for snapshots #3042

Conversation

reta commented Apr 22, 2022

Description

Issues Resolved

Check List

reta Apr 22, 2022

Choose a reason for hiding this comment

nknize left a comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

reta commented Apr 22, 2022

dblock commented Apr 22, 2022

dblock commented Apr 22, 2022

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022 • edited Loading

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

reta commented Apr 22, 2022

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

dblock commented Apr 22, 2022

reta commented Apr 22, 2022

opensearch-ci-bot commented Apr 22, 2022

reta commented Apr 22, 2022

dblock commented Apr 22, 2022 • edited Loading

reta commented Apr 22, 2022 • edited Loading

opensearch-ci-bot commented Apr 22, 2022

nknize commented Apr 22, 2022

reta commented Apr 22, 2022

nknize commented Apr 22, 2022

reta commented Apr 22, 2022

reta commented Apr 22, 2022 •

edited

Loading

dblock commented Apr 22, 2022 •

edited

Loading

reta commented Apr 22, 2022 •

edited

Loading