feat(e2e): move e2e tests #328
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeqo
force-pushed
the
e2e-tests
branch
3 times, most recently
from
6c75023 to
c59042b
Compare
jeqo
force-pushed
the
e2e-tests
branch
2 times, most recently
from
8498791 to
2a18510
Compare
jeqo
force-pushed
the
e2e-tests
branch
6 times, most recently
from
d5220ef to
a58c3e7
Compare
jeqo
force-pushed
the
e2e-tests
branch
8 times, most recently
from
d956bc1 to
c846b28
Compare
AnatolyPopov
requested changes
Aug 22, 2023
e2e/src/test/java/io/aiven/kafka/tieredstorage/e2e/internal/TestUtils.java
Outdated
Show resolved
Hide resolved
@ivanyu we could do this, but I'd prefer to keep the same JDK version across modules to avoid confusion. We can migrate back to records once JDK 11 is dropped at some point (🤞🏽), wdyt? |
jeqo
commented
Aug 23, 2023
Comment on lines
+521
to
+529
| @Test | ||
| @Disabled("https://github.com/aiven/kafka/issues/35") | ||
| @Order(5) | ||
| void topicDelete() throws Exception { |
There was a problem hiding this comment.
topic delete with tiered storage is still flaky. Disabling it in the meantime, we can troubleshoot this after switching to upstream.
There was a problem hiding this comment.
[2023-08-23 00:32:07,465] ERROR [RemoteLogManager=1 partition=4OkS_EQaSVKfGYZVj2w9bg:topic1-0] Error occurred while copying log segments of partition: 4OkS_EQaSVKfGYZVj2w9bg:topic1-0 (kafka.log.remote.RemoteLogManager$RLMTask)
java.lang.IllegalStateException: This instance is in invalid state, initialized: false close: false
at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed(TopicBasedRemoteLogMetadataManager.java:501)
at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.highestOffsetForEpoch(TopicBasedRemoteLogMetadataManager.java:224)
at kafka.log.remote.RemoteLogManager.$anonfun$findHighestRemoteOffset$2(RemoteLogManager.scala:612)
at kafka.log.remote.RemoteLogManager.$anonfun$findHighestRemoteOffset$2$adapted(RemoteLogManager.scala:609)
at scala.Option.foreach(Option.scala:437)
at kafka.log.remote.RemoteLogManager.$anonfun$findHighestRemoteOffset$1(RemoteLogManager.scala:609)
at kafka.log.remote.RemoteLogManager.$anonfun$findHighestRemoteOffset$1$adapted(RemoteLogManager.scala:608)
at scala.Option.foreach(Option.scala:437)
at kafka.log.remote.RemoteLogManager.findHighestRemoteOffset(RemoteLogManager.scala:608)
at kafka.log.remote.RemoteLogManager$RLMTask.maybeUpdateReadOffset$1(RemoteLogManager.scala:394)
at kafka.log.remote.RemoteLogManager$RLMTask.copyLogSegmentsToRemote(RemoteLogManager.scala:399)
at kafka.log.remote.RemoteLogManager$RLMTask.run(RemoteLogManager.scala:576)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[2023-08-23 00:32:07,469] ERROR [RemoteLogManager=1 partition=4OkS_EQaSVKfGYZVj2w9bg:topic1-0] Error while cleaning up log segments for partition: 4OkS_EQaSVKfGYZVj2w9bg:topic1-0 (kafka.log.remote.RemoteLogManager$RLMTask)
java.lang.IllegalStateException: This instance is in invalid state, initialized: false close: false
at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed(TopicBasedRemoteLogMetadataManager.java:501)
at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.listRemoteLogSegments(TopicBasedRemoteLogMetadataManager.java:240)
at kafka.log.remote.RemoteLogManager$RLMTask.handleExpiredRemoteLogSegments(RemoteLogManager.scala:505)
at kafka.log.remote.RemoteLogManager$RLMTask.run(RemoteLogManager.scala:580)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[2023-08-23 00:32:07,467] INFO [Broker id=1] Finished LeaderAndIsr request in 303ms correlationId 1 from controller 1 for 1 partitions (state.change.logger)
e6036b8 to
6803f71
Compare
e2e/src/test/java/io/aiven/kafka/tieredstorage/e2e/LocalSystemSingleBrokerTest.java
Outdated
Show resolved
Hide resolved
jeqo
force-pushed
the
e2e-tests
branch
3 times, most recently
from
70bca21 to
6ac5c7d
Compare
jeqo
commented
Aug 24, 2023
Comment on lines
+521
to
+520
| await() | ||
| .between(Duration.ofSeconds(1), Duration.ofSeconds(30)) | ||
| .pollInterval(Duration.ofSeconds(1)) | ||
| .until(() -> assertNoTopicDataOnTierStorage(t0p0.topic(), t0p0.topicId())); |
There was a problem hiding this comment.
This was the last bit that seemed to be flaky. Before it was only checked once and there was no guarantee that files were removed. Now, it's checked for a bit. Builds seem to be more stable now.
| * | ||
| * <p>Needed to allow running {@code SingleBrokerTest#stopKafka} after all tests. | ||
| */ | ||
| public static class FailFastTestWatcher implements TestWatcher { |
There was a problem hiding this comment.
Can we instead implement ExecutionCondition and make failed a field of this class to avoid constant checking?
There was a problem hiding this comment.
Thanks. Looking at the ExecutionCondition lifecycle, I guess we need both: watcher to set the flag when a test fails, and condition to do not run a test if failed = true.
There was a problem hiding this comment.
Exactly, that's what I was trying to say but used the wrong wording sorry.
There was a problem hiding this comment.
No worries, changes are applied in the latest commit. Have a look. Thanks!
e2e/src/test/java/io/aiven/kafka/tieredstorage/e2e/SingleBrokerTest.java
Show resolved
Hide resolved
e2e/src/test/java/io/aiven/kafka/tieredstorage/e2e/LocalSystemSingleBrokerTest.java
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See commits for details on changes.
Depends on